-
Notifications
You must be signed in to change notification settings - Fork 4.5k
docs: Connector Builder contribution flow text-based docs #44616
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
3b57f56
revamp the index connector development page
natikgadzhi 11d2192
misc small cleanup
natikgadzhi f27dbcb
Builder contrib flow instead of export
natikgadzhi 986593e
low-code docs update
natikgadzhi 592f81b
Update docs/connector-development/schema-reference.md
natikgadzhi File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -3,11 +3,7 @@ | |
Airbyte’s low-code framework enables you to build source connectors for REST APIs via a [connector builder UI](https://docs.airbyte.com/connector-development/connector-builder-ui/overview) or by modifying boilerplate YAML files via terminal or text editor. | ||
|
||
:::info | ||
Developer updates will be announced via our #help-connector-development Slack channel. If you are using the CDK, please join to stay up to date on changes and issues. | ||
::: | ||
|
||
:::note | ||
The low-code framework is in **beta**, which means that while it will be backwards compatible, it’s still in active development. Share feedback and requests with us on our [Slack channel](https://slack.airbyte.com/) or email us at [[email protected]](mailto:[email protected]) | ||
Developer updates will be announced via our [#help-connector-development Slack channel](https://airbytehq.slack.com/archives/C027KKE4BCZ). If you are using the CDK, please join to stay up to date on changes and issues. | ||
::: | ||
|
||
## Why low-code? | ||
|
@@ -65,7 +61,7 @@ If the answer to all questions is yes, you can use the low-code framework to bui | |
## Prerequisites | ||
|
||
- An API key for the source you want to build a connector for | ||
- Python >= 3.9 | ||
- Python >= 3.10 | ||
- Docker | ||
|
||
## Overview of the process | ||
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,7 +1,5 @@ | ||
# Getting Started | ||
|
||
:warning: This framework is in **alpha**. It is still in active development and may include backward-incompatible changes. Please share feedback and requests directly with us at [email protected] :warning: | ||
|
||
## Summary | ||
|
||
Throughout this tutorial, we'll walk you through the creation of an Airbyte source to read and extract data from an HTTP API. | ||
|
@@ -41,7 +39,7 @@ This can be done by signing up for the Free tier plan on [Exchange Rates Data AP | |
## Requirements | ||
|
||
- An Exchange Rates API key | ||
- Python >= 3.9 | ||
- Python >= 3.10 | ||
- [Poetry](https://python-poetry.org/) | ||
- Docker must be running | ||
- [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#L1) CLI | ||
|
49 changes: 17 additions & 32 deletions
49
docs/connector-development/connector-builder-ui/overview.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,56 +1,41 @@ | ||
# Connector Builder Intro | ||
|
||
Connector Builder is a no-code tool that’s part of the Airbyte UI. It provides an intuitive user interface on top of the [low-code YAML format](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/yaml-overview) and lets you develop a connector to use in data syncs without ever needing to leave your Airbyte workspace. Connector Builder offers the most straightforward method for building and maintaining connectors. | ||
|
||
We recommend that you determine whether the connector you want can be built with the Connector Builder before looking at the Low-Code CDK or Python CDK. Our [compatibility guide](./connector-builder-compatibility.md) can help you decide if Connector Builder is the right tool to use. | ||
Connector Builder is a no-code tool that’s part of the Airbyte UI. | ||
It provides an intuitive user interface on top of the [low-code YAML format](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/yaml-overview) and lets you develop a connector to use in data syncs without ever needing to leave your Airbyte workspace. | ||
Connector Builder offers the most straightforward method for building, contributing, and maintaining connectors. | ||
|
||
## When should I use Connector Builder? | ||
|
||
First, check if the API you want to use has an available connector in the [catalog](../../integrations). If you find it there, you can use it as is. If you need to update an existing connector, see the guide for updates. | ||
First, check if the API you want to use has an available connector in the [catalog](../../integrations). If you find it there, you can use it as is. If you need to update an existing connector, see the guide for updates. | ||
|
||
Generally, you can build a connector with the Connector Builder if you want to connect to an HTTP API that returns a collection of records as JSON and has fixed endpoints. For more detailed information on requirements, refer to the [compatibility guide](./connector-builder-compatibility.md). | ||
You can build a connector with the Connector Builder if you want to connect to an HTTP API that returns a collection of records as JSON and has fixed endpoints. For more detailed information on requirements, refer to the [compatibility guide](./connector-builder-compatibility.md). | ||
|
||
## Getting started | ||
|
||
The high-level process for using Connector Builder is as follows: | ||
|
||
1. Access Connector Builder in the Airbyte web app by selecting "Builder" in the left-hand sidebar. | ||
2. Iterate on your low-code connector by providing details for global configuration and user inputs. User inputs are the variables your connector will ask an end-user to provide when they configure a connector for use in a connection. | ||
3. Once the connector is ready, publish it. This makes it available in your local workspace | ||
1. Access Connector Builder in the Airbyte web app by selecting "Builder" in the left-hand sidebar | ||
2. Iterate on the connector by providing details for global configuration and user inputs, and streams | ||
3. Once the connector is ready, publish it to your workspace, or contribute it to Airbyte catalog | ||
4. Configure a Source based on the released connector | ||
5. Use the Source in a connection to sync data | ||
|
||
The concept pages in this section of the docs share more details related to the following topics: [authentication](./authentication.md), [record processing](./record-processing.mdx), [pagination](./pagination.md), [incremental sync](./incremental-sync.md), [partitioning](./partitioning.md), and [error handling](./error-handling.md). | ||
The concept pages in this section of the docs share more details related to the following topics: [authentication](./authentication.md), [record processing](./record-processing.mdx), [pagination](./pagination.md), [incremental sync](./incremental-sync.md), [partitioning](./partitioning.md), and [error handling](./error-handling.md). | ||
|
||
:::tip | ||
Do not hardcode things like API keys or passwords while configuring a connector in the builder. They will be used, but not saved, during development when you provide them as Testing Values. For use in production, these should be passed in as user inputs after publishing the connector to the workspace, when you configure a source using your connector. | ||
Do not hardcode things like API keys or passwords while configuring a connector in the builder. They will be used, but not saved, during development when you provide them as Testing Values. For use in production, these should be passed in as user inputs after publishing the connector to the workspace, when you configure a source using your connector. | ||
|
||
Follow [the tutorial](./tutorial.mdx) for an example of what this looks like in practice. | ||
::: | ||
|
||
## Exporting the connector | ||
|
||
:::info | ||
If you choose to contribute your connector to the Airbyte connector catalog, making it publicly available outside of your workspace, you'll need to export it and go through the process of submitting it for review. | ||
::: | ||
|
||
Connector Builder leverages the [low-code CDK](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/yaml-overview) under the hood, turning all configurations into the YAML format. Typically, it's not necessary to interact with the YAML representation. However, you can export the connector YAML into a file and build a docker image containing the connector which can be shared more widely: | ||
|
||
1. Use Connector Builder to iterate on your low-code connector | ||
2. Export the YAML into a low-code connector module on your local machine | ||
3. Build the connector's Docker image | ||
4. Use the built connector image in Airbyte | ||
|
||
Once you're done iterating on your connector in the UI, you'll need to export the low-code YAML representation of the connector to your local filesystem into a connector module. This YAML can be downloaded by clicking the `Download Config` button in the bottom-left. | ||
|
||
Create a low-code connector module using the connector generator (see [this YAML tutorial for an example](../config-based/tutorial/1-create-source.md)) using the name you'd like to use for your connector. After creating the connector, overwrite the contents of `airbyte-integrations/connectors/source-<connector name>/source_<connector name>/manifest.yaml` with the YAML you created in the UI. | ||
## Contributing the connector | ||
|
||
Follow the instructions in the connector README to build the Docker image. Typically this will be something like `docker build . -t airbyte/source-<name>:<version>`. | ||
If you'd like to share your connector with other Airbyte users, you can contribute it to Airbyte's GitHub repository right from the Builder. | ||
|
||
From this point on your connector is a regular low-code CDK connector. It can now be distributed as a docker image and be made part of the regular Airbyte connector catalog. For more information, read the [overview page for the publishing process](/connector-development/#publishing-a-connector). | ||
1. Click "Publish" chevron -> "Contribute to Marketplace" | ||
2. Fill out the form: add the connector description, and provide your GitHub PAT (Personal Access Token) to create a pull request | ||
3. Click "Contribute" to submit the connector to the Airbyte catalog | ||
|
||
:::note | ||
Connector Builder UI is in beta, which means it’s still in active development and may include backward-incompatible changes. Share feedback and requests with us on our Slack channel or email us at [email protected] | ||
Reviews typically take under a week. | ||
|
||
Developer updates will be announced via our #help-connector-development Slack channel. If you are using the CDK, please join to stay up to date on changes and issues. | ||
::: | ||
You can also export the YAML manifest file for your connector and share it with others. The manifest file contains all the information about the connector, including the global configuration, streams, and user inputs. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
are we committing to this timeline?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not entirely true — aspirational, but I would be happy if anyone asks me and cites this doc.