Skip to content

fix: add runtime hints for storage #2001

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Aug 4, 2023
Merged

fix: add runtime hints for storage #2001

merged 16 commits into from
Aug 4, 2023

Conversation

zhumin8
Copy link
Contributor

@zhumin8 zhumin8 commented Jul 5, 2023

Builds on top of CI setup in #1933. Will rebase once its merged in.

This pr:

  • adds runtime hint to storage/integration/aot/StorageIntegrationRuntimeHint.java.
  • adds storage module to spring-native profile in pom.xml
  • adds storage sample modules & resources to spring-cloud-gcp-samples/pom.xml to run sample test in ci.
  • for sample test, point to buckets within the native ci project.

CI:

  • sample test should be included in ci that runs with this pr.
  • module it test is not included in ci. Below are local test results. you can verify locally by running mvn -Pspring-native,\!default clean test . (adding this ci in fix: add runtime hints for vision #1991)
com.google.cloud.spring.storage.it.GoogleStorageIntegrationTests > createWithContent() SUCCESSFUL

com.google.cloud.spring.storage.it.GoogleStorageIntegrationTests > createAndWriteTest() SUCCESSFUL


Test run finished after 2186 ms
[         3 containers found      ]
[         0 containers skipped    ]
[         3 containers started    ]
[         0 containers aborted    ]
[         3 containers successful ]
[         0 containers failed     ]
[         2 tests found           ]
[         0 tests skipped         ]
[         2 tests started         ]
[         0 tests aborted         ]
[         2 tests successful      ]
[         0 tests failed          ]

[UPDATE] The reflection configuration is added in here because the FileUtils class in Spring Integration was calling com.google.cloud.storage.Blob[] (a child class of BlobInfo) reflectively using getClass(). The FileUtils class is invoked by AbstractInboundFileSynchronizer which is a parent of GcsInboundFileSynchronizer The stacktrace that helped with this discovery:

Caused by: java.lang.IllegalArgumentException: Class com.google.cloud.storage.Blob[] is instantiated reflectively but was never registered.Register the class by adding "unsafeAllocated" for the class in reflect-config.json.
    at org.graalvm.nativeimage.builder/com.oracle.svm.core.graal.snippets.SubstrateAllocationSnippets.arrayHubErrorStub(SubstrateAllocationSnippets.java:351)
    at [email protected]/java.util.stream.SpinedBuffer.asArray(SpinedBuffer.java:214)
    at [email protected]/java.util.stream.Nodes$SpinedNodeBuilder.asArray(Nodes.java:1296)
    at [email protected]/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:617)
    at org.springframework.integration.file.support.FileUtils.purgeUnwantedElements(FileUtils.java:61)
    at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.transferFilesFromRemoteToLocal(AbstractInboundFileSynchronizer.java:358)
    at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.lambda$synchronizeToLocalDirectory$0(AbstractInboundFileSynchronizer.java:342)
    at org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:452)
    ... 23 more

@zhumin8 zhumin8 marked this pull request as ready for review July 6, 2023 19:16
@zhumin8 zhumin8 requested a review from a team as a code owner July 6, 2023 19:16
@zhumin8 zhumin8 requested review from burkedavison and mpeddada1 July 6, 2023 19:16
if (fileName.isPresent()) {
GoogleStorageResource resource =
new GoogleStorageResource(
this.storage, String.format("gs://%s/%s", bucketName, fileName.get()));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider a private Resource getResource(Optional<String> filename) function to reduce the duplication in this method and writeGcs.

  @GetMapping(value = "/")
  public String readGcsFile(@RequestParam("filename") Optional<String> fileName)
      throws IOException {
    Resource resource = getResource(fileName);
    return StreamUtils.copyToString(resource.getInputStream(), Charset.defaultCharset()) + "\n";
  }

@mpeddada1 mpeddada1 requested a review from burkedavison July 19, 2023 15:18
Copy link
Member

@burkedavison burkedavison left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Check shortened fixedDelay
  • Refactor get() call to have in-scope isPresent() call.
  • Keep or replace aot.factories

@sonarqubecloud
Copy link

sonarqubecloud bot commented Aug 3, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

100.0% 100.0% Coverage
0.0% 0.0% Duplication

@mpeddada1 mpeddada1 merged commit 793c798 into main Aug 4, 2023
@mpeddada1 mpeddada1 deleted the storage-hints branch August 4, 2023 19:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants