[source-dynamodb] connector does not work in AWS cross-account setting with Role based authentication

[noamst-monday]

Connector Name

source-dynamodb

Connector Version

0.3.9

What step the error happened?

Configuring a new connector

Relevant information

Hello

I have the following setup:

Airbyte is hosted in AWS account A.
DynamoDB tables are hosted in AWS account B.

I am trying to set up a dynamodb source in airbyte, using role based authentication.
The role belongs to account A, and has permissions to perform dynamodb actions on account B.

Account B tables have a policy which allows account A role to access them, according to the documentation.
https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-cross-account-access-to-amazon-dynamodb.html#configure-cross-account-access-to-amazon-dynamodb-summary

When trying to set up the source, the first action taken is trying to list the tables in the same account, which is not the desired behaviour, since I'm trying to sync from a different account. This also fails, since the role in account A does not have permissions to list tables in account A, and it's irrelevant since the tables are in account B.

I also tried to set up AWS PrivateLink for DynamoDB, hoping it would "force" the client to perform the actions in the appropriate account, but to no avail.

As a possible solution, I would like to enter the full source ARN, so that the SDK tries to query the correct table in the correct account.
This is the suggested solution in the example provided by AWS:
https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-cross-account-access-to-amazon-dynamodb.html#configure-cross-account-access-to-amazon-dynamodb-additional

Thank you!

Relevant log output

2025-05-28 14:04:29 info INFO i.a.i.s.d.DynamodbUtils(createDynamoDbClient):44 Using Role Based Access
2025-05-28 14:04:29 error ERROR i.a.i.s.d.DynamodbSource(check):64 Error while listing Dynamodb tables with reason: software.amazon.awssdk.services.dynamodb.model.DynamoDbException: User: arn:aws:sts::account_A:assumed-role/AccountARole/aws-sdk-java-** is not authorized to perform: dynamodb:ListTables on resource: arn:aws:dynamodb:ap-southeast-2:account_A:table/* because no identity-based policy allows the dynamodb:ListTables action (Service: DynamoDb, Status Code: 400, Request ID:***)

Contribute

• Yes, I want to contribute

[BrettNWoods]

I've come across this one as well. From my understanding, this connector calls a ListTables operation which is used to pull out the list of tables that the role has access to. The connector then feeds these table names into the ScanPaginator to pull out the data.

The problem with this and what we want to do (get a role in account A to access a table in Account B) is that the ListTables API only returns tables from the current project. If we were to pass in a table ARN, we would have to skip this ListTables operation and just directly pass in the table ARNs that were provided. We could do this either in the listTables() function itself but I am not sure if that is the most elegant solution.

Happy to contribute for this one.