🎉 New Destination: AWS Datalake

[kattos-aws]

What

This destination connector allows for ingestion of data into an AWS data lake.

How

This destination connector takes advantage of the new governed tables features to better integrate the data into the larger AWS data ecosystem (ex: Glue, Athena, Redshift, ...)

Recommended reading order

1. docs/integrations/aws-datalake.md
2. airbyte-integrations/connectors/destination-aws-datalake/destination_aws_datalake/destination.py
3. airbyte-integrations/connectors/destination-aws-datalake/destination_aws_datalake/airbyte_helper.py
4. airbyte-integrations/connectors/destination-aws-datalake/destination_aws_datalake/aws_helpers.py
5. ... the rest :)

🚨 User Impact 🚨

None

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

• Community member? Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • docs/SUMMARY.md
  • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • docs/integrations/README.md
  • airbyte-integrations/builds.md
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• Credentials added to Github CI. Instructions.
• /test connector=connectors/<name> command is passing
• New Connector version released on Dockerhub by running the /publish command described here
• After the connector is published, connector added to connector index as described here
• Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

[CLAassistant]

All committers have signed the CLA.

[marcosmarxm]

Thanks @kattos-aws can you sign the CLA and share the test output from the connector? There is a implementation in python and some files in Java, can you explain the implementation you made?

[kattos-aws]

Thanks @kattos-aws can you sign the CLA and share the test output from the connector? There is a implementation in python and some files in Java, can you explain the implementation you made?

I did sign the CLA:

Not sure what is going on here, any idea?

Here is the log for the Destination Acceptance Test suite:
airbyte-integration-tests.log.gz

The connector itself is in Python, but as there were no Destination Acceptance Tests in Python, I had to write a bit of Java code to be able to run the DAT.

Hope that makes sense, feel free to reach out if you need more info. Also available for a call if need be!

[marcosmarxm]

@kattos-aws can you share the output from the acceptance tests?

[kattos-aws]

@marcosmarxm Sorry I realize my comment was really not clear. The logs I attached in my previous comment are actually the output of the acceptance tests. I did not implement specific integration tests.

I ran the tests as indicated in the generated README.md:

To run acceptance and custom integration tests:

```bash
  ./gradlew :airbyte-integrations:connectors:destination-aws-datalake:integrationTest
```

If you search for the string AwsDatalakeDestinationAcceptanceTest in the logs, you will find the section for this connector.

I hope that clears that out! If not, feel free to reach out.

Thanks!

[sherifnada]

@alafanechere @marcosmarxm would either of you have bandwidth to review this PR from Stephane, our friend at AWS?

[marcosmarxm]

thanks @kattos-aws there is a small change in the spec I'll do tomorrow and then I'll publish the new connector!

[kattos-aws]

Great news @marcosmarxm , thanks!