bug: copyTo for local build path fails for node when running against 10 node network #1286
Description
When Alex is running a 10 node performance test, about once a night he will get an error on the copyTo for one of the 10 nodes.
It looks like this is happening in the NodeCommandTasks._uploadPlatformSoftware, where it is copying the local build path jar to the network node and runs one instance for each node and then sets it to concurrent. Instead we want to make it concurrent but have max concurrent calls set to 4 (with a constant and a env override).
one such error:
{"label":"","level":"INFO","message":"copyTo[network-node8-0,42496c42-cdd3-4765-b20c-5c791052ed71]: [srcPath=/home/runner/_work/performance-analysis-automation/performance-analysis-automation/hedera-services/hedera-node/data, destDir=/opt/hgcapp/services-hedera/HapiApp2.0]","ms":"+5ms","timestamp":"2025-02-04T03:11:21.100Z","traceId":"2fff5b45-aa3e-49e0-b59a-9b4a3d46d053"}
{"label":"","level":"DEBUG","message":"copyTo[network-node8-0,42496c42-cdd3-4765-b20c-5c791052ed71]: received message","ms":"+55ms","timestamp":"2025-02-04T03:11:22.662Z","traceId":"2fff5b45-aa3e-49e0-b59a-9b4a3d46d053"}
{"label":"","level":"WARN","message":"copyTo[network-node8-0,42496c42-cdd3-4765-b20c-5c791052ed71]: failed with code=1006, reason=","ms":"+0ms","timestamp":"2025-02-04T03:11:54.693Z","traceId":"2fff5b45-aa3e-49e0-b59a-9b4a3d46d053"}
{"cause":{},"label":"","level":"ERROR","message":"Error in setting up nodes: copyTo[network-node8-0,42496c42-cdd3-4765-b20c-5c791052ed71]: failed with code=1006, reason= copyTo[network-node8-0,42496c42-cdd3-4765-b20c-5c791052ed71]: failed with code=1006, reason=","meta":{},"ms":"+2ms","name":"SoloError","stack":"SoloError: copyTo[network-node8-0,42496c42-cdd3-4765-b20c-5c791052ed71]: failed with code=1006, reason=\n at K8.exitWithError (file:///home/runner/_work/performance-analysis-automation/performance-analysis-automation/solo/dist/src/core/k8.js:364:36)\n at WebSocket.<anonymous> (file:///home/runner/_work/performance-analysis-automation/performance-analysis-automation/solo/dist/src/core/k8.js:460:41)\n at WebSocket.emit (node:events:519:28)\n at WebSocket.emitClose (/home/runner/_work/performance-analysis-automation/performance-analysis-automation/solo/node_modules/ws/lib/websocket.js:272:10)\n at Receiver.receiverOnFinish (/home/runner/_work/performance-analysis-automation/performance-analysis-automation/solo/node_modules/ws/lib/websocket.js:1209:20)\n at Receiver.emit (node:events:519:28)\n at finish (node:internal/streams/writable:955:10)\n at node:internal/streams/writable:936:13\n at process.processTicksAndRejections (node:internal/process/task_queues:90:21)","timestamp":"2025-02-04T03:11:54.695Z","traceId":"2fff5b45-aa3e-49e0-b59a-9b4a3d46d053"}
{"cause":{"cause":{},"meta":{},"name":"SoloError"},"label":"","level":"ERROR","message":"Error in setting up nodes: copyTo[network-node8-0,42496c42-cdd3-4765-b20c-5c791052ed71]: failed with code=1006, reason= Error in setting up nodes: copyTo[network-node8-0,42496c42-cdd3-4765-b20c-5c791052ed71]: failed with code=1006, reason=","meta":{},"ms":"+11s","name":"SoloError","stack":"SoloError: Error in setting up nodes: copyTo[network-node8-0,42496c42-cdd3-4765-b20c-5c791052ed71]: failed with code=1006, reason=\n at file:///home/runner/_work/performance-analysis-automation/performance-analysis-automation/solo/dist/src/commands/base.js:166:23\n at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n at async NodeCommandHandlers.setup (file:///home/runner/_work/performance-analysis-automation/performance-analysis-automation/solo/dist/src/commands/node/handlers.js:576:9)","timestamp":"2025-02-04T03:12:05.282Z","traceId":"2fff5b45-aa3e-49e0-b59a-9b4a3d46d053"}