New Destination: Apache Iceberg

[kainoa21]

Tell us about the new integration you’d like to have

Apache Iceberg is an ideal destination as it represents an open source, file based table format.

Describe the context around this new integration

Iceberg powers our file based data warehouse at Netflix and we would like to be able to leverage the Airbyte ecosystem of source connectors to ease the burden of data engineers writing bespoke ingestion pipelines.

Describe the alternative you are considering or using

Bespoke data ingestion pipelines (primarily python) and Spark data sources and data frames.

┆Issue is synchronized with this Asana task by Unito

[sherifnada]

@kainoa21 thanks for the suggestion! What data scale do you anticipate moving into Iceberg?

[kainoa21]

Mostly this would be for the long tail of external or vendor product data ingestion. Examples are Zendesk, Anaplan, Jira and the like. So not massive datasets.

[v-kessler]

any progress here?

[jaceksan]

This seems to relate to #6745.
Once anyone prepares a class responsible for creating Iceberg tables, it should be reusable for both, right?

[Leibnizhu]

I'm working on it.