Implement data staging from DynamoDB

[victorskl]

Context:

OrcaVault objective is to do data warehouse need of our upstream microservices system, OrcaBus.

We now have few microservices; that are leveraging DynamoDB as their transactional database records.

We need to extract and load these DynamoDB data sources into OrcaVault warehouse staging area.

Prerequisite:

#60

Approach:

We strategically choose AWS Glue as such ETL data staging requirement purpose.

Study AWS Glue developer guide. The following as starter pointers.

• We want to use DynamoDB as source and, Aurora PostgreSQL as target for connection. The target database is orcavault and, schema can be any one of ods or tsa depends on data loading strategy choice.
• https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect-dynamodb-home.html
• Study CDC with DynamoDB stream and see whether it fits for the use case purpose discussed here?

[victorskl]

Alternate via FDW

Investigate

• https://wiki.postgresql.org/wiki/Foreign_data_wrappers
• https://github.com/pgspider/dynamodb_fdw

Note though, since we are using "managed" Aurora RDBMS cluster, not every PostgreSQL extension available online is viable; it may be barred/limited by AWS Aurora managed solution.

https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.Extensions.html