Skip to content

[EPIC] Disable insecure settings in cloud connectors #6408

Closed
@sherifnada

Description

@sherifnada

Tell us about the problem you're trying to solve

We want to create secure-only versions of connectors which can be used in Airbyte cloud. The idea is that these connectors inherently prevent certain insecure connections such as:

  • Connecting to a database over the public internet without encryption
  • Reading data from the local filesystem where airbyte is running the workload
  • etc..

Currently the above two examples are the clear patterns. So, this epic tracks a few things:

TLS-only DBs

We want to do the same for all of our DBs. Here are the DBs for which we want to perform this change, in priority order:

Must have for Airbyte Cloud launch

Nice-to-have for Airbyte Cloud launch

This PR contains an example of modifying a DB to only allow using TLS connections. It does the following:

  1. Create a new connector source-<name>-strict-encrypt based on the current connector
  2. modify the connector's spec to hide any options which allows the user to disable TLS
  3. change the connector to enable TLS by default if the TLS option is not specified

important note 1: until the above example PR is merged, please only focus on sources in the above list. The reason is that we are still working out how normalization will be handled for destinations. I will update here once we do.
important note 2: note that to perform this task for any given connector, it must first support TLS as described in #5628

Non-DB connectors

We also want to fork the file connector to disallow using local files:

  • Local file
    This is a nice to have for Airbyte Cloud.

As always we need to be mindful of the difference in changing sources & destinations. Destinations might also require changing something about normalization. See the reference PR for hints on the difference.

To find which DBs have TLS supported already see this epic: #5628

Follow up work

You'll notice that forking connectors introduces a really unpleasant dependency: every time we update the non-secure connector we have to also bump and publish the secure version. As a follow up, we need to find a way to make this process more seamless. We want updates to propagate downstream as needed.

Metadata

Metadata

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions