🎉 Destination GCS: fix connection check

[tuliren]

What

• Resolves Investigate GCS destination integration test failure #9631.
• The check operation of the GCS destination uses native S3 TransferManager to test the multipart upload. However, in the sync operation, we use StreamTransferManager to do the job. It looks like the former requires two more permissions than the latter: multipartUpload.list and multipartUpload.listAll.
• This PR fixes this issue by using the StreamTransferManager to test the multipart upload. In this way, the check reflects what actually happens in the sync. In this way, users do not need to grant the client unnecessary permissions just to pass the connection check.
• The same check is applied to both S3 and GCS.

🚨 User Impact 🚨

• None expected.

Pre-merge Checklist

Updating a connector

Community member or Airbyter

• Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• Credentials added to Github CI. Instructions.
• /test connector=connectors/<name> command is passing.
• New Connector version released on Dockerhub by running the /publish command described here
• After the new connector version is published, connector version bumped in the seed directory as described here
• Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

[tuliren]

/test connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1834837170
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1834837170
No Python unittests run

[tuliren]

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1834837268
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1834837268
No Python unittests run

[tuliren]

/publish connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1834870216
❌ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1834870216

[tuliren]

/publish connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1834870467
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1834870467

[tuliren]

S3 version is bumped in #10256. I will revert its version bump here, and do it in a separate PR.

[edgao]

should we just replace all usages of this method, or is it possible that the staging warehouse destinations do actually need those extra permissions? E.g.

airbyte/airbyte-integrations/connectors/destination-snowflake/src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeCopyS3Destination.java

Line 40 in ae2a87e

S3Destination.attemptS3WriteAndDelete(getS3DestinationConfig(config), "");

[tuliren]

I think it would depend on the use case, i.e. how the warehouse uses the S3 or other staging bucket. Since there is no complaint from the community about any permission issue for those destinations, we have not looked into them yet.