Source S3: Add virtual-hosted-style option (#19006)

* add virtual-hosted-style option for S3 source

* update s3 version

* auto-bump connector version

Co-authored-by: Vincent Koc <[email protected]>
Co-authored-by: Octavia Squidington III <[email protected]>

Author: 3 people

airbyte-config/init/src/main/resources/seed/source_definitions.yaml

@@ -1127,7 +1127,7 @@
 - name: S3
   sourceDefinitionId: 69589781-7828-43c5-9f63-8925b1c1ccc2
   dockerRepository: airbyte/source-s3
-  dockerImageTag: 0.1.25
+  dockerImageTag: 0.1.26
   documentationUrl: https://docs.airbyte.com/integrations/sources/s3
   icon: s3.svg
   sourceType: file

airbyte-config/init/src/main/resources/seed/source_specs.yaml

@@ -10761,7 +10761,7 @@
     supportsNormalization: false
     supportsDBT: false
     supported_destination_sync_modes: []
-- dockerImage: "airbyte/source-s3:0.1.25"
+- dockerImage: "airbyte/source-s3:0.1.26"
   spec:
     documentationUrl: "https://docs.airbyte.com/integrations/sources/s3"
     changelogUrl: "https://docs.airbyte.com/integrations/sources/s3"

airbyte-integrations/connectors/source-s3/Dockerfile

@@ -17,5 +17,5 @@ COPY source_s3 ./source_s3
 ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
 ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
 
-LABEL io.airbyte.version=0.1.25
+LABEL io.airbyte.version=0.1.26
 LABEL io.airbyte.name=airbyte/source-s3

airbyte-integrations/connectors/source-s3/integration_tests/config_minio.json

@@ -6,7 +6,7 @@
     "aws_access_key_id": "123456",
     "aws_secret_access_key": "123456key",
     "path_prefix": "",
-    "endpoint": "http://10.0.56.135:9000"
+    "endpoint": "http://10.0.229.255:9000"
   },
   "format": {
     "filetype": "csv"

airbyte-integrations/connectors/source-s3/source_s3/s3_utils.py

@@ -48,6 +48,7 @@ def _get_s3_client_args(provider: dict, config: Config) -> dict:
         client_kv_args["endpoint_url"] = endpoint
         client_kv_args["use_ssl"] = provider.get("use_ssl", True)
         client_kv_args["verify"] = provider.get("verify_ssl_cert", True)
+        client_kv_args["config"] = Config(s3={"addressing_style": provider.get("addressing_style", "auto")})
 
     return client_kv_args
 

docs/integrations/sources/s3.md

@@ -205,6 +205,7 @@ The Jsonl parser uses pyarrow hence,only the line-delimited JSON format is suppo
 
 | Version | Date       | Pull Request                                                                                                    | Subject                                                                                 |
 |:--------|:-----------|:----------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------|
+| 0.1.26  | 2022-11-08 | [19006](https://github.com/airbytehq/airbyte/pull/19006)                                                        | Add virtual-hosted-style option                      |
 | 0.1.24  | 2022-10-28 | [18602](https://github.com/airbytehq/airbyte/pull/18602)                                                        | Wrap errors into AirbyteTracedException pointing to a problem file                      |
 | 0.1.23  | 2022-10-10 | [17991](https://github.com/airbytehq/airbyte/pull/17991)                                                        | Fix pyarrow to JSON schema type conversion for arrays                                   |
 | 0.1.23  | 2022-10-10 | [17800](https://github.com/airbytehq/airbyte/pull/17800)                                                        | Deleted `use_ssl` and `verify_ssl_cert` flags and hardcoded to `True`                   |