refactor: Update the response queue in the server to reuse response slots

[pskiran1]

What does the PR do?

The current response queue allocates memory for each response.
This PR aims to enhance the response queue by reusing response slots across multiple responses within the same request once they have been written (completed) to the network. This may help reduce active memory utilization.

• In the PopResponse() function, we clear the response content and return it to the reusable pool.
• In the AllocateResponse() function, we check for any available responses in the reusable pool; if present, we use it, otherwise, we allocate a new response.
• Introduces a configurable threshold(--grpc-max-response-pool-size option) to limit the number of active response protobuf allocations in the gRPC response queue.

Test cases:
L0_decoupled:

• Added a test case to verify the behavior of --grpc-max-response-pool-size in decoupled mode. It runs the existing test cases with and without this flag

L0_memory:

• Added a test case to evaluate memory utilization when running the server with different values for --grpc-max-response-pool-size (1, 25, and 50) and also a scenario where the flag is not set, to compare memory usage across different configurations.
• We run a for loop in the script to test different pool sizes. We start the server with/without pool size, start monitoring the server's memory usage, run the client, and then terminate the server.

Checklist

• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging ref.
• All template sections are filled out.
• Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

• build
• ci
• docs
• feat
• fix
• perf
• refactor
• revert
• style
• test

Related PRs:

Where should the reviewer start?

Test plan:

• CI Pipeline ID: 24179262

Caveats:

Background

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• closes GitHub issue: #xxx

[pskiran1]

The PR was automatically closed on forced rebase with the main, reopened with a new commit.

[indrajit96]

Can you add a test plan in the description on what we are trying to test here?
Why have we set --grpc-max-response-pool-size only to 1?
Also can we add a test to confirm memory footprint decreses with using --grpc-max-response-pool-size VS not using --grpc-max-response-pool-size ?

[pskiran1]

Also can we add a test to confirm memory footprint decreses with using --grpc-max-response-pool-size VS not using --grpc-max-response-pool-size ?

I have added a new test case in L0_memory to evaluate memory utilization when running the server with different values for --grpc-max-response-pool-size (1, 25, and 50), as well as without this flag.

Why have we set --grpc-max-response-pool-size only to 1?

Regarding setting --grpc-max-response-pool-size to 1, I included this specific test to evaluate the lowest possible value. And, running the decoupled model tests takes a long time, with additional pool sizes it is leading to timeouts. The new test case in L0_memory covers different values.

[pskiran1]

Also, updated description with higher level details about tests. Please let me know if we are missing something.

[indrajit96]

Can we also update docs for --grpc-max-response-pool-size

[pskiran1]

Can we also update docs for --grpc-max-response-pool-size

I have updated the documentation to include details about both the --grpc-infer-allocation-pool-size and --grpc-max-response-pool-size options. I request @tanmayv25, and @indrajit96, please review and confirm if the content is correct. Thank you.

[tanmayv25]

Nice work @pskiran1 !