Description
Connector Name
source-s3
Connector Version
4.13.4
What step the error happened?
Configuring a new connector
Relevant information
I have Airbyte OSS running on GKE and I'm trying to setup an S3 source, but I can't make the authentication work. I need to use role based authentication but it fails saying it can't locate the credentials. I'm configuring all AWS Access Key ID, AWS Secret Access Key and AWS Role ARN.
I've tested the credentials with a small python script and it works. I've also cloned this repo and tested/debugged the connector code locally and it looks like it is not passing the configured AWS Access Key ID and AWS Secret Access Key to the boto session.
I've done this small change here and the connector passes the check command:
I'm not sure how this is supposed to be configured or to work.
I suspect these are preconfigured in anything running on AWS, but needs to be passed/configured somehow when running outside of it (usually done with the aws cli and/or profiles locally).
I guess one option will be to set these as env vars on the pods, so boto can find them and use them in the default session, but that doesn't seem right just to configure the S3 connector.
Am I configuring something wrong?
I'm happy to submit a PR if my change makes sense.
Relevant log output
Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/file_based/availability_strategy/default_file_based_availability_strategy.py", line 100, in _check_list_files file = next(iter(stream.get_files())) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/airbyte/integration_code/source_s3/v4/stream_reader.py", line 137, in get_matching_files s3 = self.s3_client ^^^^^^^^^^^^^^ File "/airbyte/integration_code/source_s3/v4/stream_reader.py", line 75, in s3_client self._s3_client = self._get_iam_s3_client(client_kv_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/airbyte/integration_code/source_s3/v4/stream_reader.py", line 122, in _get_iam_s3_client metadata=refresh(), ^^^^^^^^^ File "/airbyte/integration_code/source_s3/v4/stream_reader.py", line 108, in refresh role = client.assume_role( ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 570, in _api_call return self._make_api_call(operation_name, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/context.py", line 124, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 1013, in _make_api_call http, parsed_response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 1037, in _make_request return self._endpoint.make_request(operation_model, request_dict) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/endpoint.py", line 119, in make_request return self._send_request(request_dict, operation_model) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/endpoint.py", line 196, in _send_request request = self.create_request(request_dict, operation_model) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/endpoint.py", line 132, in create_request self._event_emitter.emit( File "/usr/local/lib/python3.11/site-packages/botocore/hooks.py", line 412, in emit return self._emitter.emit(aliased_event_name, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/hooks.py", line 256, in emit return self._emit(event_name, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/hooks.py", line 239, in _emit response = handler(**kwargs) ^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/signers.py", line 106, in handler return self.sign(operation_name, request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/signers.py", line 198, in sign auth.add_auth(request) File "/usr/local/lib/python3.11/site-packages/botocore/auth.py", line 424, in add_auth raise NoCredentialsError() botocore.exceptions.NoCredentialsError: Unable to locate credentials The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/file_based/availability_strategy/default_file_based_availability_strategy.py", line 79, in check_availability_and_parsability file = self._check_list_files(stream) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/file_based/availability_strategy/default_file_based_availability_strategy.py", line 106, in _check_list_files raise CheckAvailabilityError( airbyte_cdk.sources.file_based.exceptions.CheckAvailabilityError: Error listing files. Please check the credentials provided in the config and verify that they provide permission to list files. Contact Support if you need assistance. stream=run_outputContribute
- Yes, I want to contribute