Skip to content

[Java] ApiClient: support deserializing from InputStream instead of String to bypass 2GB Java String limit #21115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

klin1344
Copy link
Contributor

@klin1344 klin1344 commented Apr 22, 2025

PR checklist

  • Read the contribution guidelines.
  • Pull Request title clearly describes the work in the pull request and Pull Request description provides details about how to validate the work. Missing information here may result in delayed response from the community.
  • Run the following to build the project and update samples:
    ./mvnw clean package || exit
    ./bin/generate-samples.sh ./bin/configs/*.yaml || exit
    ./bin/utils/export_docs_generators.sh || exit
    
    (For Windows users, please run the script in Git BASH)
    Commit all changed files.
    This is important, as CI jobs will verify all generator outputs of your HEAD commit as it would merge with master.
    These must match the expectations made by your contribution.
    You may regenerate an individual generator by passing the relevant config(s) as an argument to the script, for example ./bin/generate-samples.sh bin/configs/java*.
    IMPORTANT: Do NOT purge/delete any folders/files (e.g. tests) when regenerating the samples as manually written tests may be removed.
  • File the PR against the correct branch: master (upcoming 7.x.0 minor release - breaking changes with fallbacks), 8.0.x (breaking changes without fallbacks)
  • If your PR is targeting a particular programming language, @mention the technical committee members, so they are more likely to review the pull request.

__

Overview

Fixes issue kubernetes-client/java#4020 .

Makes two changes:

  1. In ApiClient, if we are deserializing JSON, get the response body as an InputStream instead of String for deserialization.
  2. In JSON, add a new deserialize() method in which accepts InputStream.

These two changes enable the ApiClient to bypass the 2GB String length limit in Java and deserialize large responses greater than 2GB into Java objects.

Root Cause

When the response is too large, we get an error like this:

java.lang.IllegalArgumentException: byteCount: 2293832235
  at okio.Buffer.readString(Buffer.kt:306)
  at okio.Buffer.readString(Buffer.kt:302)
  at okio.RealBufferedSource.readString(RealBufferedSource.kt:96)
  at okhttp3.ResponseBody.string(ResponseBody.kt:187)
  at io.kubernetes.client.openapi.ApiClient.deserialize(ApiClient.java:758)
  at io.kubernetes.client.openapi.ApiClient.handleResponse(ApiClient.java:978)
  at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:905)
  at io.kubernetes.client.informer.SharedInformerFactory$1.list(SharedInformerFactory.java:271)
  at io.kubernetes.client.informer.cache.ReflectorRunnable.run(ReflectorRunnable.java:91)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  at java.lang.Thread.run(Thread.java:1583)

This issue occurs because the response body is converted to a String here in the ApiClient, which hits this validation in okhttp:

require(byteCount >= 0 && byteCount <= Integer.MAX_VALUE) { "byteCount: $byteCount" }

because of Java's String max length limit of Integer.MAX_VALUE which is 2^31 - 1.

@wing328
Copy link
Member

wing328 commented Apr 22, 2025

https://github.com/OpenAPITools/openapi-generator/actions/runs/14587151272/job/40918080042?pr=21115

please update bin/utils/test_file_list.yaml according with new sha256 as the test file has been updated as part of this PR

@klin1344
Copy link
Contributor Author

https://github.com/OpenAPITools/openapi-generator/actions/runs/14587151272/job/40918080042?pr=21115

please update bin/utils/test_file_list.yaml according with new sha256 as the test file has been updated as part of this PR

@wing328 ah I missed that, thank you! I have updated the test_file_list.yaml with the new SHA256 now.

@@ -155,7 +155,7 @@ defmodule {{moduleName}}.Connection do
of the function call, will be set as a bearer token in the
`authorization` header.
- `scopes`: a list of Strings represenging OAuth2 scopes.
- `options`: a keyword list of OpenAPIPetstore.Connection.options.
- `options`: a keyword list of {{moduleName}}.Connection.options.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like a rebase failed and some other changes not authored by you are also included.

given that only a few elixir related changes (already merged into master) are included, we will accept these this time.

@wing328 wing328 merged commit b844d8d into OpenAPITools:master Apr 23, 2025
102 checks passed
@wing328 wing328 added this to the 7.13.0 milestone Apr 23, 2025
@klin1344 klin1344 deleted the klin1344/ApiClient-deserialize-inputstream branch April 23, 2025 15:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants