Skip to content

Release connectors with namespace change. #2990

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Apr 21, 2021
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,6 @@
"destinationDefinitionId": "22f6c74f-5699-40ff-833c-4a879ea40133",
"name": "BigQuery",
"dockerRepository": "airbyte/destination-bigquery",
"dockerImageTag": "0.2.3",
"dockerImageTag": "0.3.0",
"documentationUrl": "https://docs.airbyte.io/integrations/destinations/bigquery"
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,6 @@
"destinationDefinitionId": "25c5221d-dce2-4163-ade9-739ef790f503",
"name": "Postgres",
"dockerRepository": "airbyte/destination-postgres",
"dockerImageTag": "0.2.4",
"dockerImageTag": "0.3.0",
"documentationUrl": "https://docs.airbyte.io/integrations/destinations/postgres"
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,6 @@
"destinationDefinitionId": "424892c4-daac-4491-b35d-c6688ba547ba",
"name": "Snowflake",
"dockerRepository": "airbyte/destination-snowflake",
"dockerImageTag": "0.2.3",
"dockerImageTag": "0.3.0",
"documentationUrl": "https://docs.airbyte.io/integrations/destinations/snowflake"
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,6 @@
"destinationDefinitionId": "f7a7d195-377f-cf5b-70a5-be6b819019dc",
"name": "Redshift",
"dockerRepository": "airbyte/destination-redshift",
"dockerImageTag": "0.2.6",
"dockerImageTag": "0.3.0",
"documentationUrl": "https://docs.airbyte.io/integrations/destinations/redshift"
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,6 @@
"sourceDefinitionId": "435bb9a5-7887-4809-aa58-28c27df0d7ad",
"name": "MySQL",
"dockerRepository": "airbyte/source-mysql",
"dockerImageTag": "0.2.5",
"dockerImageTag": "0.3.0",
"documentationUrl": "https://docs.airbyte.io/integrations/sources/mysql"
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,6 @@
"sourceDefinitionId": "b39a7370-74c3-45a6-ac3a-380d48520a83",
"name": "Oracle",
"dockerRepository": "airbyte/source-oracle",
"dockerImageTag": "0.2.0",
"dockerImageTag": "0.3.0",
"documentationUrl": "https://hub.docker.com/r/airbyte/source-oracle"
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,6 @@
"sourceDefinitionId": "b5ea17b1-f170-46dc-bc31-cc744ca984c1",
"name": "Microsoft SQL Server (MSSQL)",
"dockerRepository": "airbyte/source-mssql",
"dockerImageTag": "0.2.3",
"dockerImageTag": "0.3.0",
"documentationUrl": "https://hub.docker.com/r/airbyte/source-mssql"
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,6 @@
"sourceDefinitionId": "decd338e-5647-4c0b-adf4-da0e75f5a750",
"name": "Postgres",
"dockerRepository": "airbyte/source-postgres",
"dockerImageTag": "0.2.7",
"dockerImageTag": "0.3.0",
"documentationUrl": "https://hub.docker.com/r/airbyte/source-postgres"
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,6 @@
"sourceDefinitionId": "e87ffa8e-a3b5-f69c-9076-6011339de1f6",
"name": "Redshift",
"dockerRepository": "airbyte/source-redshift",
"dockerImageTag": "0.2.3",
"dockerImageTag": "0.3.0",
"documentationUrl": "https://hub.docker.com/repository/docker/airbyte/source-redshift"
}
Original file line number Diff line number Diff line change
Expand Up @@ -11,22 +11,22 @@
- destinationDefinitionId: 25c5221d-dce2-4163-ade9-739ef790f503
name: Postgres
dockerRepository: airbyte/destination-postgres
dockerImageTag: 0.2.4
dockerImageTag: 0.3.0
documentationUrl: https://docs.airbyte.io/integrations/destinations/postgres
- destinationDefinitionId: 22f6c74f-5699-40ff-833c-4a879ea40133
name: BigQuery
dockerRepository: airbyte/destination-bigquery
dockerImageTag: 0.2.3
dockerImageTag: 0.3.0
documentationUrl: https://docs.airbyte.io/integrations/destinations/bigquery
- destinationDefinitionId: 424892c4-daac-4491-b35d-c6688ba547ba
name: Snowflake
dockerRepository: airbyte/destination-snowflake
dockerImageTag: 0.2.3
dockerImageTag: 0.3.0
documentationUrl: https://docs.airbyte.io/integrations/destinations/snowflake
- destinationDefinitionId: f7a7d195-377f-cf5b-70a5-be6b819019dc
name: Redshift
dockerRepository: airbyte/destination-redshift
dockerImageTag: 0.2.6
dockerImageTag: 0.3.0
documentationUrl: https://docs.airbyte.io/integrations/destinations/redshift
- destinationDefinitionId: af7c921e-5892-4ff2-b6c1-4a5ab258fb7e
name: MeiliSearch
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,12 @@
- sourceDefinitionId: b5ea17b1-f170-46dc-bc31-cc744ca984c1
name: Microsoft SQL Server (MSSQL)
dockerRepository: airbyte/source-mssql
dockerImageTag: 0.2.3
dockerImageTag: 0.3.0
documentationUrl: https://hub.docker.com/r/airbyte/source-mssql
- sourceDefinitionId: decd338e-5647-4c0b-adf4-da0e75f5a750
name: Postgres
dockerRepository: airbyte/source-postgres
dockerImageTag: 0.2.7
dockerImageTag: 0.3.0
documentationUrl: https://hub.docker.com/r/airbyte/source-postgres
- sourceDefinitionId: cd42861b-01fc-4658-a8ab-5d11d0510f01
name: Recurly
Expand All @@ -51,7 +51,7 @@
- sourceDefinitionId: 435bb9a5-7887-4809-aa58-28c27df0d7ad
name: MySQL
dockerRepository: airbyte/source-mysql
dockerImageTag: 0.2.5
dockerImageTag: 0.3.0
documentationUrl: https://docs.airbyte.io/integrations/sources/mysql
- sourceDefinitionId: 2470e835-feaf-4db6-96f3-70fd645acc77
name: Salesforce
Expand Down Expand Up @@ -96,7 +96,7 @@
- sourceDefinitionId: e87ffa8e-a3b5-f69c-9076-6011339de1f6
name: Redshift
dockerRepository: airbyte/source-redshift
dockerImageTag: 0.2.3
dockerImageTag: 0.3.0
documentationUrl: https://hub.docker.com/repository/docker/airbyte/source-redshift
- sourceDefinitionId: 932e6363-d006-4464-a9f5-102b82e07c06
name: Twilio
Expand Down Expand Up @@ -212,7 +212,7 @@
- sourceDefinitionId: b39a7370-74c3-45a6-ac3a-380d48520a83
name: Oracle
dockerRepository: airbyte/source-oracle
dockerImageTag: 0.2.0
dockerImageTag: 0.3.0
documentationUrl: https://hub.docker.com/r/airbyte/source-oracle
- sourceDefinitionId: c8630570-086d-4a40-99ae-ea5b18673071
name: Zendesk Talk
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,5 @@ COPY build/distributions/${APPLICATION}*.tar ${APPLICATION}.tar

RUN tar xf ${APPLICATION}.tar --strip-components=1

LABEL io.airbyte.version=0.2.3
LABEL io.airbyte.version=0.3.0
LABEL io.airbyte.name=airbyte/destination-bigquery
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,5 @@ COPY build/distributions/${APPLICATION}*.tar ${APPLICATION}.tar

RUN tar xf ${APPLICATION}.tar --strip-components=1

LABEL io.airbyte.version=0.2.2
LABEL io.airbyte.version=0.3.0
LABEL io.airbyte.name=airbyte/destination-jdbc
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,5 @@ COPY build/distributions/${APPLICATION}*.tar ${APPLICATION}.tar

RUN tar xf ${APPLICATION}.tar --strip-components=1

LABEL io.airbyte.version=0.2.4
LABEL io.airbyte.version=0.3.0
LABEL io.airbyte.name=airbyte/destination-postgres
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,5 @@ COPY build/distributions/${APPLICATION}*.tar ${APPLICATION}.tar

RUN tar xf ${APPLICATION}.tar --strip-components=1

LABEL io.airbyte.version=0.2.6
LABEL io.airbyte.version=0.3.0
LABEL io.airbyte.name=airbyte/destination-redshift
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,5 @@ COPY build/distributions/${APPLICATION}*.tar ${APPLICATION}.tar

RUN tar xf ${APPLICATION}.tar --strip-components=1

LABEL io.airbyte.version=0.2.4
LABEL io.airbyte.version=0.3.0
LABEL io.airbyte.name=airbyte/destination-snowflake
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-jdbc/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,5 @@ COPY build/distributions/${APPLICATION}*.tar ${APPLICATION}.tar

RUN tar xf ${APPLICATION}.tar --strip-components=1

LABEL io.airbyte.version=0.2.0
LABEL io.airbyte.version=0.3.0
LABEL io.airbyte.name=airbyte/source-jdbc
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-mssql/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,5 @@ COPY build/distributions/${APPLICATION}*.tar ${APPLICATION}.tar

RUN tar xf ${APPLICATION}.tar --strip-components=1

LABEL io.airbyte.version=0.2.3
LABEL io.airbyte.version=0.3.0
LABEL io.airbyte.name=airbyte/source-mssql
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-mysql/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,6 @@ COPY build/distributions/${APPLICATION}*.tar ${APPLICATION}.tar

RUN tar xf ${APPLICATION}.tar --strip-components=1

LABEL io.airbyte.version=0.2.5
LABEL io.airbyte.version=0.3.0

LABEL io.airbyte.name=airbyte/source-mysql
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-oracle/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,5 @@ COPY build/distributions/${APPLICATION}*.tar ${APPLICATION}.tar

RUN tar xf ${APPLICATION}.tar --strip-components=1

LABEL io.airbyte.version=0.2.0
LABEL io.airbyte.version=0.3.0
LABEL io.airbyte.name=airbyte/source-oracle
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,5 @@ COPY build/distributions/${APPLICATION}*.tar ${APPLICATION}.tar

RUN tar xf ${APPLICATION}.tar --strip-components=1

LABEL io.airbyte.version=0.2.7
LABEL io.airbyte.version=0.3.0
LABEL io.airbyte.name=airbyte/source-postgres
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-redshift/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,5 @@ COPY build/distributions/${APPLICATION}*.tar ${APPLICATION}.tar

RUN tar xf ${APPLICATION}.tar --strip-components=1

LABEL io.airbyte.version=0.2.3
LABEL io.airbyte.version=0.3.0
LABEL io.airbyte.name=airbyte/source-redshift
47 changes: 47 additions & 0 deletions docs/architecture/namespaces.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Namespaces

## What is a Namespace?
Technical systems often group their underlying data into namespaces with each namespace's data isolated from another namespace. This isolation allow
for better organisation and flexibility, leading to better usability.

An example of namespace is a RDMS's `schema` concept. Some common use cases for schemas are enforcing permissiong, segregating test and production data
and general data organisation.

## Syncing
The Airbyte protocol supports namespaces and allows Sources to define namespaces, and Destinations to write to a various namespaces.

A sync conducted with a Source and Destination that both support namespaces will have its data replicated into the source-defined namespace. For such syncs,
data is replicated into the Destination in a layout matching the Source.

All of this is automatic, and requires no additional user configuration.

If the Source does not support namespaces, the data will be replicated into the Destination's default namespace.

If the Destination does not support namespaces, the [namespace field](https://github.com/airbytehq/airbyte/blob/master/airbyte-protocol/models/src/main/resources/airbyte_protocol/airbyte_protocol.yaml#L64) is ignored.

## Requirements
* Both Source and Destiation connectors need to support namespaces.
* Relevant Source and Destination connectors need to be at least version `0.3.0` or later.
* Airbyte version `0.21.0-alpha` or later.

## Current Support
### Sources
* MSSQL
* MYSQL
* Oracle DB
* Postgres
* Redshift

### Destination
* BigQuery
* Postgres
* Snowflake
* Redshift

## Coming Soon
* Ability to prefix namespaces.
* Ability to configure custom namespaces.
* Ability to toggle namespaces.
* Please [create a ticket](https://github.com/airbytehq/airbyte/issues/new/choose) if other namespace features come to mind!

We welcome you to join our [Slack](https://airbyte.io/community/) if you have any other questions!
3 changes: 3 additions & 0 deletions docs/faq/transformation-and-schemas.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,6 @@ Airbyte continues to sync data using the configured schema until that schema is

For now, the schema can only be updated manually in the UI \(by clicking "Update Schema" in the settings page for the connection\). When a schema is updated Airbyte will re-sync all data for that source using the new schema.

## **How does Airbyte handle namespaces (or schemas for the DB-inclined)?**

Airbyte respects source-defined namespaces when syncing data with a namespace-supported destination. See [this](../architecture/namespaces.md) for more details.
1 change: 1 addition & 0 deletions docs/integrations/destinations/bigquery.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ Each stream will be output into its own table in BigQuery. Each table will conta
| :--- | :--- | :--- |
| Full Refresh Sync | Yes | |
| Incremental - Append Sync | Yes | |
| Namespaces | Yes | |

## Getting started

Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/local-csv.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ Each stream will be output into its own file. Each file will contain 3 columns:
| :--- | :--- |
| Full Refresh Sync | Yes |
| Incremental - Append Sync | Yes |
| Namespaces | No | |

#### Performance considerations

Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/local-json.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ Each stream will be output into its own file. Each file will a collections of `j
| :--- | :--- |
| Full Refresh Sync | Yes |
| Incremental - Append Sync | Yes |
| Namespaces | No | |

#### Performance considerations

Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/meilisearch.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ Each stream will be output into its own index in MeiliSearch. Each table will be
| :--- | :--- | :--- |
| Full Refresh Sync | Yes | |
| Incremental - Append Sync | Yes | |
| Namespaces | No | |

## Getting started

Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/postgres.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Each stream will be output into its own table in Postgres. Each table will conta
| :--- | :--- | :--- |
| Full Refresh Sync | Yes | |
| Incremental - Append Sync | Yes | |
| Namespaces | Yes | |

## Getting started

Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/redshift.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ Each stream will be output into its own raw table in Redshift. Each table will c
| :--- | :--- | :--- |
| Full Refresh Sync | Yes | |
| Incremental - Append Sync | Yes | |
| Namespaces | Yes | |

#### Target Database

Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/snowflake.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ Each stream will be output into its own table in Snowflake. Each table will cont
| :--- | :--- | :--- |
| Full Refresh Sync | Yes | |
| Incremental - Append Sync | Yes | |
| Namespaces | Yes | |

## Getting started

Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/appstore.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ Note that depending on the credentials you enter, you may only be able to sync s
| :--- | :--- | :--- |
| Full Refresh Sync | no | |
| Incremental Sync | yes | |
| Namespaces | No | |

### Performance considerations

Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/braintree.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ This Source is capable of syncing the following core Streams:
| :--- | :--- | :--- |
| Full Refresh Sync | Yes | |
| Incremental Sync | Yes | |
| Namespaces | No | |

### Performance considerations

Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/drift.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ If there are more endpoints you'd like Airbyte to support, please [create an iss
| Incremental Sync | Coming soon |
| Replicate Incremental Deletes | Coming soon |
| SSL connection | Yes |
| Namespaces | No |

### Performance considerations

Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/exchangeratesapi.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ Currencies are `number` and the date is a `string`.
| :--- | :--- |
| Full Refresh Sync | Yes |
| Incremental - Append Sync | Yes |
| Namespaces | No |

### Getting started

Expand Down
3 changes: 2 additions & 1 deletion docs/integrations/sources/facebook-marketing.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,9 @@ For more information, see the [Facebook Insights API documentation. ](https://de

| Feature | Supported?\(Yes/No\) | Notes |
| :--- | :--- | :--- |
| Full Refresh Sync | Yes | |
| Full Refresh Sync | Yes |
| Incremental Sync | Yes | except AdCreatives |
| Namespaces | No |

### Rate Limiting & Performance Considerations

Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/file.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ At this time, this source produces only a single stream for the target file as i
| Replicate Incremental Deletes | No |
| Replicate Folders \(multiple Files\) | No |
| Replicate Glob Patterns \(multiple Files\) | No |
| Namespaces | No |

How do we rate the functionalities below?

Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/freshdesk.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ If there are more endpoints you'd like Airbyte to support, please [create an iss
| Full Refresh Sync | Yes |
| Incremental Sync | Yes |
| SSL connection | Yes |
| Namespaces | No |

### Performance considerations

Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/github.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ This connector outputs the following streams:
| Incremental - Append Sync | Yes |
| Replicate Incremental Deletes | Coming soon |
| SSL connection | Yes |
| Namespaces | No |

### Performance considerations

Expand Down
Loading