Description
Tell us about the problem you're trying to solve
We want to create secure-only versions of connectors which can be used in Airbyte cloud. The idea is that these connectors inherently prevent certain insecure connections such as:
- Connecting to a database over the public internet without encryption
- Reading data from the local filesystem where airbyte is running the workload
- etc..
Currently the above two examples are the clear patterns. So, this epic tracks a few things:
- forking DB connectors to only allow encrypted access (TLS)
- forking the File connector to disallow pulling files from the local filesystem #6538
- Customize which connectors are shown in Cloud #6334 (this ticket is already being handled by the cloud/platform team)
TLS-only DBs
We want to do the same for all of our DBs. Here are the DBs for which we want to perform this change, in priority order:
Must have for Airbyte Cloud launch
- Create a secure-only Postgres Destination #6418
- Create a secure-only Postgres Source #6419
- Create a secure-only MySQL source #6420
- Create a secure-only MSSQL Source #6421
- Create a secure-only Oracle Source #6422
- Create a secure-only MySQL destination #6423
Nice-to-have for Airbyte Cloud launch
- Create a secure-only Oracle destination #6713
- Create a secure-only MongoDb source #6715
- Create a secure-only MongoDb destination #6714
- Create a secure-only Clickhouse Source #6716
- Create a secure-only CockroachDb source #6717
- Create a secure-only Db2 source #6718
- Create a secure-only MSSQL destination #6712
- Redshift Source: create secure-only version #6973
- Redshift Destination: create secure-only version #6974
This PR contains an example of modifying a DB to only allow using TLS connections. It does the following:
- Create a new connector
source-<name>-strict-encrypt
based on the current connector - modify the connector's spec to hide any options which allows the user to disable TLS
- change the connector to enable TLS by default if the TLS option is not specified
important note 1: until the above example PR is merged, please only focus on sources in the above list. The reason is that we are still working out how normalization will be handled for destinations. I will update here once we do.
important note 2: note that to perform this task for any given connector, it must first support TLS as described in #5628
Non-DB connectors
We also want to fork the file connector to disallow using local files:
- Local file
This is a nice to have for Airbyte Cloud.
As always we need to be mindful of the difference in changing sources & destinations. Destinations might also require changing something about normalization. See the reference PR for hints on the difference.
To find which DBs have TLS supported already see this epic: #5628
Follow up work
You'll notice that forking connectors introduces a really unpleasant dependency: every time we update the non-secure connector we have to also bump and publish the secure version. As a follow up, we need to find a way to make this process more seamless. We want updates to propagate downstream as needed.