From 7cd545fc5872fa74ac24515e1129f56c1b6c1865 Mon Sep 17 00:00:00 2001 From: tealjulia Date: Tue, 3 Sep 2024 15:47:11 -0400 Subject: [PATCH 1/2] remove lingering mentions of dbt core --- .../on-kubernetes-via-helm.md | 7 --- docs/understanding-airbyte/operations.md | 55 ------------------- 2 files changed, 62 deletions(-) delete mode 100644 docs/understanding-airbyte/operations.md diff --git a/docs/deploying-airbyte/on-kubernetes-via-helm.md b/docs/deploying-airbyte/on-kubernetes-via-helm.md index e8fc93457ef72..cff290b8922cb 100644 --- a/docs/deploying-airbyte/on-kubernetes-via-helm.md +++ b/docs/deploying-airbyte/on-kubernetes-via-helm.md @@ -17,13 +17,6 @@ Kubernetes cluster on AWS. Follow [this guide](on-restack.md) to get started. :::note -Airbyte running on Self-Hosted Kubernetes doesn't support DBT Transformations. Please refer to -[#5901](https://github.com/airbytehq/airbyte/issues/5091) - -::: - -:::note - Airbyte Kubernetes Community Edition does not support basic auth by default. To enable basic auth, consider adding a reverse proxy in front of Airbyte. diff --git a/docs/understanding-airbyte/operations.md b/docs/understanding-airbyte/operations.md deleted file mode 100644 index 50ce80e2c2c8b..0000000000000 --- a/docs/understanding-airbyte/operations.md +++ /dev/null @@ -1,55 +0,0 @@ -# Operations - -Airbyte [connections](/using-airbyte/core-concepts/sync-modes/) support configuring additional transformations that execute after the sync. Useful applications could be: - -- Customized normalization to better fit the requirements of your own business context. -- Business transformations from a technical data representation into a more logical and business oriented data structure. This can facilitate usage by end-users, non-technical operators, and executives looking to generate Business Intelligence dashboards and reports. -- Data Quality, performance optimization, alerting and monitoring, etc. -- Integration with other tools from your data stack \(orchestration, data visualization, etc.\) - -## Supported Operations - -### dbt transformations - -#### - git repository url: - -A url to a git repository to \(shallow\) clone the latest dbt project code from. - -The project versioned in the repository is expected to: - -- be a valid dbt package with a `dbt_project.yml` file at its root. -- have a `dbt_project.yml` with a "profile" name declared as described [here](https://docs.getdbt.com/dbt-cli/configure-your-profile). - -When using the dbt CLI, dbt checks your `profiles.yml` file for a profile with the same name. A profile contains all the details required to connect to your data warehouse. This file generally lives outside of your dbt project to avoid sensitive credentials being checked in to version control. Therefore, a `profiles.yml` will be generated according to the configured destination from the Airbyte UI. - -Note that if you prefer to use your own `profiles.yml` stored in the git repository or in the Docker image, then you can specify an override with `--profiles-dir=` in the dbt CLI arguments. - -#### - git repository branch \(optional\): - -The name of the branch to use when cloning the git repository. If left empty, git will use the default branch of your repository. - -#### - docker image: - -A Docker image and tag to run dbt commands from. The Docker image should have `/bin/bash` and `dbt` installed for this operation type to work. - -A typical value for this field would be for example: `fishtownanalytics/dbt:1.0.0` from [dbt dockerhub](https://hub.docker.com/r/fishtownanalytics/dbt/tags?page=1&ordering=last_updated). - -This field lets you configure the version of dbt that your custom dbt project requires and the loading of additional software and packages necessary for your transformations \(other than your dbt `packages.yml` file\). - -#### - dbt cli arguments - -This operation type is aimed at running the dbt cli. - -A typical value for this field would be "run" and the actual command invoked would as a result be: `dbt run` in the docker container. - -One thing to consider is that dbt allows for vast configuration of the run command, for example, allowing you to select a subset of models. You can find the [dbt reference docs](https://docs.getdbt.com/reference/dbt-commands) which describes this set of available commands and options. - -## Future Operations - -- Docker/Script operations: Execute a generic script in a custom Docker container. -- Webhook operations: Trigger API or hooks from other providers. -- Airflow operations: To use a specialized orchestration tool that lets you schedule and manage more advanced/complex sequences of operations in your sync workflow. - -## Going Further - -In the meantime, please feel free to react, comment, and share your thoughts/use cases with us. We would be glad to hear your feedback and ideas as they will help shape the next set of features and our roadmap for the future. You can head to our GitHub and participate in the corresponding issue or discussions. Thank you! From 692d9a9241f3dd117ec928dbb9f1b4f204d841a8 Mon Sep 17 00:00:00 2001 From: tealjulia Date: Tue, 3 Sep 2024 15:51:22 -0400 Subject: [PATCH 2/2] fix sidebar.js --- docusaurus/sidebars.js | 1 - 1 file changed, 1 deletion(-) diff --git a/docusaurus/sidebars.js b/docusaurus/sidebars.js index 3a5d46a2c1cb1..62962795c9cff 100644 --- a/docusaurus/sidebars.js +++ b/docusaurus/sidebars.js @@ -495,7 +495,6 @@ const understandingAirbyte = { "understanding-airbyte/database-data-catalog", "understanding-airbyte/beginners-guide-to-catalog", "understanding-airbyte/supported-data-types", - "understanding-airbyte/operations", "understanding-airbyte/secrets", "understanding-airbyte/cdc", "understanding-airbyte/resumability",