You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: airbyte-cdk/python/README.md
+97-65
Original file line number
Diff line number
Diff line change
@@ -1,57 +1,79 @@
1
-
# Connector Development Kit \(Python\)
1
+
# Airbyte Python CDK and Low-Code CDK
2
2
3
-
The Airbyte Python CDK is a framework for rapidly developing production-grade Airbyte connectors.The CDK currently offers helpers specific for creating Airbyte source connectors for:
3
+
Airbyte Python CDK is a framework for building Airbyte API Source Connectors. It provides a set of
4
+
classes and helpers that make it easy to build a connector against an HTTP API (REST, GraphQL, etc),
5
+
or a generic Python source connector.
4
6
5
-
- HTTP APIs \(REST APIs, GraphQL, etc..\)
6
-
- Generic Python sources \(anything not covered by the above\)
7
+
## Usage
7
8
8
-
The CDK provides an improved developer experience by providing basic implementation structure and abstracting away low-level glue boilerplate.
9
+
If you're looking to build a connector, we highly recommend that you
10
+
[start with the Connector Builder](https://docs.airbyte.com/connector-development/connector-builder-ui/overview).
11
+
It should be enough for 90% connectors out there. For more flexible and complex connectors, use the
12
+
[low-code CDK and `SourceDeclarativeManifest`](https://docs.airbyte.com/connector-development/config-based/low-code-cdk-overview).
9
13
10
-
This document is a general introduction to the CDK. Readers should have basic familiarity with the [Airbyte Specification](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/) before proceeding.
14
+
If that doesn't work, then consider building on top of the
Before you can start working on this project, you will need to have Poetry installed on your system. Please follow the instructions below to install Poetry:
19
-
20
-
1. Open your terminal or command prompt.
21
-
2. Install Poetry using the recommended installation method:
19
+
To get started on a Python CDK based connector or a low-code connector, you can generate a connector
cd airbyte-integrations/connector-templates/generator
25
+
./generate.sh
25
26
```
26
27
27
-
Alternatively, you can use `pip` to install Poetry:
28
-
29
-
```bash
30
-
pip install --user poetry
31
-
```
32
-
33
-
3. After the installation is complete, close and reopen your terminal to ensure the newly installed `poetry` command is available in your system's PATH.
34
-
35
-
For more detailed instructions and alternative installation methods, please refer to the official Poetry documentation: https://python-poetry.org/docs/#installation
36
-
37
-
### Concepts & Documentation
38
-
39
-
See the [concepts docs](docs/concepts/) for a tour through what the API offers.
-`destinations`. Basic Destination connector support! If you're building a Destination connector in
53
+
Python, try that. Some of our vector DB destinations like `destination-pinecone` are using that
54
+
code.
55
+
-`models` expose `airbyte_protocol.models` as a part of `airbyte_cdk` package.
56
+
-`sources/concurrent_source` is the Concurrent CDK implementation. It supports reading data from
57
+
streams concurrently per slice / partition, useful for connectors with high throughput and high
58
+
number of records.
59
+
-`sources/declarative` is the low-code CDK. It works on top of Airbyte Python CDK, but provides a
60
+
declarative manifest language to define streams, operations, etc. This makes it easier to build
61
+
connectors without writing Python code.
62
+
-`sources/file_based` is the CDK for file-based sources. Examples include S3, Azure, GCS, etc.
63
+
-`sources/singer` is a singer tap source adapter. Deprecated.
52
64
53
65
## Contributing
54
66
67
+
Thank you for being interested in contributing to Airbyte Python CDK! Here are some guidelines to
68
+
get you started:
69
+
70
+
- We adhere to the [code of conduct](/CODE_OF_CONDUCT.md).
71
+
- You can contribute by reporting bugs, posting github discussions, opening issues, improving [documentation](/docs/), and
72
+
submitting pull requests with bugfixes and new features alike.
73
+
- If you're changing the code, please add unit tests for your change.
74
+
- When submitting issues or PRs, please add a small reproduction project. Using the changes in your
75
+
connector and providing that connector code as an example (or a satellite PR) helps!
76
+
55
77
### First time setup
56
78
57
79
Install the project dependencies and development tools:
@@ -62,61 +84,58 @@ poetry install --all-extras
62
84
63
85
Installing all extras is required to run the full suite of unit tests.
64
86
65
-
#### Iteration
87
+
#### Running tests locally
66
88
67
89
- Iterate on the CDK code locally
68
-
- Run tests via `poetry run poe unit-test-with-cov`, or `python -m pytest -s unit_tests` if you want to pass pytest options.
69
-
- Run `poetry run poe check-local` to lint all code, type-check modified code, and run unit tests with coverage in one command.
90
+
- Run tests via `poetry run poe unit-test-with-cov`, or `python -m pytest -s unit_tests` if you want
91
+
to pass pytest options.
92
+
- Run `poetry run poe check-local` to lint all code, type-check modified code, and run unit tests
93
+
with coverage in one command.
70
94
71
95
To see all available scripts, run `poetry run poe`.
72
96
73
97
##### Autogenerated files
74
98
75
-
If the iteration you are working on includes changes to the models or the connector generator, you might want to regenerate them. In order to do that, you can run:
99
+
Low-code CDK models are generated from `sources/declarative/declarative_component_schema.yaml`. If
100
+
the iteration you are working on includes changes to the models or the connector generator, you
101
+
might want to regenerate them. In order to do that, you can run:
76
102
77
103
```bash
78
104
poetry run poe build
79
105
```
80
106
81
-
This will generate the code generator docker image and the component manifest files based on the schemas and templates.
107
+
This will generate the code generator docker image and the component manifest files based on the
108
+
schemas and templates.
82
109
83
110
#### Testing
84
111
85
-
All tests are located in the `unit_tests` directory. Run `poetry run poe unit-test-with-cov` to run them. This also presents a test coverage report. For faster iteration with no coverage report and more options, `python -m pytest -s unit_tests` is a good place to start.
112
+
All tests are located in the `unit_tests` directory. Run `poetry run poe unit-test-with-cov` to run
113
+
them. This also presents a test coverage report. For faster iteration with no coverage report and
114
+
more options, `python -m pytest -s unit_tests` is a good place to start.
86
115
87
116
#### Building and testing a connector with your local CDK
88
117
89
-
When developing a new feature in the CDK, you may find it helpful to run a connector that uses that new feature. You can test this in one of two ways:
118
+
When developing a new feature in the CDK, you may find it helpful to run a connector that uses that
119
+
new feature. You can test this in one of two ways:
90
120
91
121
- Running a connector locally
92
122
- Building and running a source via Docker
93
123
94
124
##### Installing your local CDK into a local Python connector
95
125
96
-
In order to get a local Python connector running your local CDK, do the following.
97
-
98
-
First, make sure you have your connector's virtual environment active:
99
-
100
-
```bash
101
-
# from the `airbyte/airbyte-integrations/connectors/<connector-directory>` directory
102
-
source .venv/bin/activate
103
-
104
-
# if you haven't installed dependencies for your connector already
105
-
pip install -e .
106
-
```
126
+
Open the connector's `pyproject.toml` file and replace the line with `airbyte_cdk` with the
127
+
following:
107
128
108
-
Then, navigate to the CDK and install it in editable mode:
You should see that `pip` has uninstalled the version of `airbyte-cdk` defined by your connector's `setup.py` and installed your local CDK. Any changes you make will be immediately reflected in your editor, so long as your editor's interpreter is set to your connector's virtual environment.
133
+
Then, running `poetry update` should reinstall `airbyte_cdk` from your local working directory.
116
134
117
135
##### Building a Python connector in Docker with your local CDK installed
118
136
119
-
_Pre-requisite: Install the [`airbyte-ci` CLI](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md)_
There may be a time when you do not have access to the API (either because you don't have the credentials, network access, etc...) You will probably still want to do end-to-end testing at least once. In order to do so, you can emulate the server you would be reaching using a server stubbing tool.
164
+
There may be a time when you do not have access to the API (either because you don't have the
165
+
credentials, network access, etc...) You will probably still want to do end-to-end testing at least
166
+
once. In order to do so, you can emulate the server you would be reaching using a server stubbing
167
+
tool.
143
168
144
-
For example, using [mockserver](https://www.mock-server.com/), you can set up an expectation file like this:
169
+
For example, using [mockserver](https://www.mock-server.com/), you can set up an expectation file
170
+
like this:
145
171
146
172
```json
147
173
{
@@ -155,13 +181,19 @@ For example, using [mockserver](https://www.mock-server.com/), you can set up an
155
181
}
156
182
```
157
183
158
-
Assuming this file has been created at `secrets/mock_server_config/expectations.json`, running the following command will allow to match any requests on path `/data` to return the response defined in the expectation file:
184
+
Assuming this file has been created at `secrets/mock_server_config/expectations.json`, running the
185
+
following command will allow to match any requests on path `/data` to return the response defined in
HTTP requests to `localhost:8113/data` should now return the body defined in the expectations file. To test this, the implementer either has to change the code which defines the base URL for Python source or update the `url_base` from low-code. With the Connector Builder running in docker, you will have to use domain `host.docker.internal` instead of `localhost` as the requests are executed within docker.
192
+
HTTP requests to `localhost:8113/data` should now return the body defined in the expectations file.
193
+
To test this, the implementer either has to change the code which defines the base URL for Python
194
+
source or update the `url_base` from low-code. With the Connector Builder running in docker, you
195
+
will have to use domain `host.docker.internal` instead of `localhost` as the requests are executed
- Requires the keys `__injected_declarative_manifest` and `__command` in its config, where `__injected_declarative_manifest` is a JSON manifest and `__command` is one of the commands handled by the ConnectorBuilderHandler (`stream_read` or `resolve_manifest`), i.e.
15
-
```
15
+
16
+
- Requires the keys `__injected_declarative_manifest` and `__command` in its config, where
17
+
`__injected_declarative_manifest` is a JSON manifest and `__command` is one of the commands
18
+
handled by the ConnectorBuilderHandler (`stream_read` or `resolve_manifest`), i.e.
19
+
20
+
```json
16
21
{
17
22
"config": <normal config>,
18
23
"__injected_declarative_manifest": {...},
19
24
"__command": <"resolve_manifest" | "test_read">
20
25
}
21
26
```
22
-
*See [ConnectionSpecification](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#actor-specification) for details on the `"config"` key if needed.
- When the `__command` is `resolve_manifest`, the argument to `catalog` should be an empty string.
25
-
- The config can optionally contain an object under the `__test_read_config` key which can define custom test read limits with `max_records`, `max_slices`, and `max_pages_per_slice` properties. All custom limits are optional; a default will be used for any limit that is not provided.
33
+
- The config can optionally contain an object under the `__test_read_config` key which can define
34
+
custom test read limits with `max_records`, `max_slices`, and `max_pages_per_slice` properties.
35
+
All custom limits are optional; a default will be used for any limit that is not provided.
26
36
27
37
### Locally running the docker image
28
38
29
39
#### Build
30
40
31
41
First, make sure you build the latest Docker image:
0 commit comments