SSL verification error

[mshannon-sil]

When uploading checkpoints to the S3 bucket, the following error is occurring: botocore.exceptions.SSLError: SSL validation failed for [file to be uploaded] EOF occurred in violation of protocol (_ssl.c:2426). This error appears about 70 seconds past the initial upload timestamp. The read timeout is currently set to 600 seconds, but the connect timeout is only set to 60 seconds by default, so that may be the cause.

[ddaspit]

Is this error causing problems?

[mshannon-sil]

Yes, oftentimes experiments are failing after 10 retries.

[Enkidu93]

I just saw this on a production job as well: 2025-01-28 21:02:18,573 - clearml.storage - ERROR - Failed uploading: SSL validation failed for ... occurred in violation of protocol (_ssl.c:2406)

[mshannon-sil]

After talking with TechOps, It looks like there may be an issue with the network's upload speeds. I submitted a ticket to IT services yesterday to see if they can resolve the issue on their end, and I'm currently waiting for a reply.

[mshannon-sil]

Research has switched over to using the MinIO bucket on our server rather than AWS, so we shouldn't run into this issue anymore.