S3 GetObjectCommand leaks sockets if the body is never read / add doc links for streaming responses re: socket exhaustion

[modelbitjason]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

If you send a GetObjectCommand but only read the returned metadata and never read the body , the S3Client eventually runs out of sockets and gets stuck forever.

I realize the solution is to use HeadObjectCommand here, but it'd be nice to call this out in the docs someplace. The example for GetObjectCommand doesn't read the body and thus will lead to leaking connections.

Once the 50 sockets have been used, the S3Client object is dead and will never work again, even after the OS cleans up the stale sockets (By looking at netstat output).

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

3.637.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v20.14.0

Reproduction Steps

const client = new S3Client({
      region: getEnv("DATA_REGION"),
      requestHandler: new NodeHttpHandler({ socketTimeout: 25000, connectionTimeout: 5000 }),
      maxAttempts: 3,
    })

for(let i=0;i<=50;i++){
  const s3objCmd = new GetObjectCommand({
    Bucket: awsBucket,
    Key: s3Key,
  })
  const s3resp = await client.send(s3objCmd)
  doSomethingWithMetadata(s3resp.Metadata)
}

Observed Behavior

Without a timeout the 51st GetObjectCommand hangs forever. With a timeout, you get an error after the socketTimeout triggers.

Expected Behavior

I expected to continue getting S3 objects or there to be a note in the documentation that I should always calls .Body() on the response to drain the socket.

I would expect the client itself to garbage collect sockets after the timeout or after the OS closes them.

I would also expect the client to throw an OutOfSockets error instead of hanging forever or hitting a connection timeout (When it won't ever connect because it is out of sockets)

Possible Solution

The easiest fix is a note in the documentation but I'd appreciate also a different error when out of sockets in the client.

I'm not sure about the expected behavior with regards to draining the socket -- It's nice that I found this typo because of the errors -- I don't want the body here and downloading in the background would be a waste of bytes.

But, it took a long time to figure out what was wrong because the inner workings of the SDK are not really clear. I don't see any examples re-using a client and wasn't sure if that was even allowed. It was only when I found the default maxSockets parameter on the client that everything fell into place.

Additional Information/Context

No response

[sayhimatt]

Similar issue discussed in #4332 and #3560

[modelbitjason]

It's exactly the issues as in #3560 -- If you don't fully drain the socket, it gets lost forever.

How much effort to get a note put in the documentation page for S3 GetObjectCommand ?

The listed example will leak and cause this exact bug because it never consumes the body.
https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/client/s3/command/GetObjectCommand/

[kuhe]

Our current guidance on managing sockets and streams is in main/supplemental-docs/CLIENTS.md#nodejs-requesthandler under "A note on socket exhaustion", and in main/supplemental-docs/performance/parallel-workloads-node-js.md

The relevant portion is

Because keepAlive is defaulted to true, if you acquire a streaming response, such as S3::getObject's Body field, you must read the stream to completion in order for the socket to close naturally.

You can also destroy the stream in Node.js with e.g. (await s3.getObject(...)).Body.destroy() if it is a Node.js Readable stream. Specifically in the case of S3, if you don't need the object body, consider whether the object metadata can be retrieved with another operation such as HeadObject or GetObjectMetadata.

regarding socket exhaustion expected behavior

The SDK does emit a warning when we detect that socket usage it at capacity and a backlog of enqueued requests exists.
You did not encounter this warning because your enqueued request count was too low for us to conclusively determine socket exhaustion.

There are reasons that we cannot detect that you've entered a state of blocking socket exhaustion:

I would expect the client itself to garbage collect sockets after the timeout or after the OS closes them.

We cannot assume that an unread stream was forgotten by the user, even after some time including up to the point of socket timeout.

I would also expect the client to throw an OutOfSockets error

Sockets are a natural batching/queuing mechanism. Running out of sockets cannot be assumed to be an error. There may be 300 requests enqueued and 50 sockets, in which case you'd be "out of sockets", but they would free up once the 6 batches of 50 requests have completed.

documentation

However, we can add more documentation pointers to operations which contain stream responses.

• for streaming response operations, add links to the socket exhaustion and streaming responses guidance docs.

[modelbitjason]

Thanks!

Those are both clear and helpful docs! I wasn't able to find a link to them from the main reference page, if it's there, maybe it could be more clear? https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/introduction/. I like that they call out client re-use, that was something I wasn't sure was okay.

In particular though, I think a note on GetObject would be nice regarding what you said -- It wouldn't have stopped my problem, I intended to use HeadObject but I think my LSP 'fixed' it for me, but I'd have seen it when I was trying to find out what was wrong.

re: batching, I was confused by getting a connection timeout after socketTimeout seconds. Maybe this is a node issue not a smithy one, but seeing Socket timed out without establishing a connection within 5000 ms is different to me than Socket not available within 5000ms -- I was initially thinking network error, not programmer error.

And re: other issues about exhaustion, I understand it's hard to come up with any default behavior, hence I mostly would have liked to saved some time debugging, I don't know that the behavior is wrong, I was just confused by the errors.

That said, I was surprised the sockets never freed up, even 10-15 minutes with no use -- Maybe this is because I'm still using the deprecated socketTimeout instead of requestTimeout? But I would expect that my client should be reset at some point of making no requests through it given there is a timeout passed in.