You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#### Build & Activate Virtual Environment and install dependencies
14
-
From this connector directory, create a virtual environment:
15
-
```
16
-
python3 -m venv .venv
17
-
```
18
13
19
-
This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your
20
-
development environment of choice. To activate it from the terminal, run:
21
-
```
22
-
source .venv/bin/activate
23
-
pip install -r requirements.txt
24
-
```
25
-
If you are in an IDE, follow your IDE's instructions to activate the virtualenv.
26
-
27
-
Once you finished with installing python requirements:
28
-
```
29
-
deactivate
14
+
### Installing the connector
15
+
From this connector directory, run:
16
+
```bash
17
+
poetry install --with dev
30
18
```
31
19
32
-
Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is
33
-
used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`.
34
-
If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything
35
-
should work as you expect.
36
20
37
-
####Create credentials
38
-
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/shopify)
39
-
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_shopify/spec.json` file.
21
+
### Create credentials
22
+
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/shopify)
23
+
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_shopify/spec.yaml` file.
40
24
Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
41
-
See `integration_tests/sample_config.json` for a sample config file.
25
+
See `sample_files/sample_config.json` for a sample config file.
42
26
43
-
**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source shopify test creds`
44
-
and place them into `secrets/config.json`.
45
27
46
28
### Locally running the connector
47
-
Use your .venv inside your connector in order to proceed:
2. Run the following command to build the docker image:
65
45
```bash
66
46
airbyte-ci connectors --name=source-shopify build
67
47
```
68
-
Once the command is done, you will find your connector image in your local docker registry: `airbyte/source-shopify:dev`.
69
-
70
-
##### Customizing our build process
71
-
When contributing on our connector you might need to customize the build process to add a system dependency or set an env var.
72
-
You can customize our build process by adding a `build_customization.py` module to your connector.
73
-
This module should contain a `pre_connector_install` and `post_connector_install` async function that will mutate the base image and the connector container respectively.
74
-
It will be imported at runtime by our build process and the functions will be called if they exist.
75
-
76
-
Here is an example of a `build_customization.py` module:
77
-
```python
78
-
from__future__import annotations
79
-
80
-
from typing importTYPE_CHECKING
81
48
82
-
ifTYPE_CHECKING:
83
-
# Feel free to check the dagger documentation for more information on the Container object and its methods.
This connector is built using our dynamic built process in `airbyte-ci`.
97
-
The base image used to build it is defined within the metadata.yaml file under the `connectorBuildOptions`.
98
-
The build logic is defined using [Dagger](https://dagger.io/)[here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/pipelines/builds/python_connectors.py).
99
-
It does not rely on a Dockerfile.
100
-
101
-
If you would like to patch our connector and build your own a simple approach would be to:
102
-
103
-
1. Create your own Dockerfile based on the latest version of the connector image.
104
-
```Dockerfile
105
-
FROM airbyte/source-shopify:latest
106
-
107
-
COPY . ./airbyte/integration_code
108
-
RUN pip install ./airbyte/integration_code
109
-
110
-
# The entrypoint and default env vars are already set in the base image
You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
134
63
```bash
135
64
airbyte-ci connectors --name=source-shopify test
136
65
```
137
66
138
67
### Customizing acceptance Tests
139
-
Customize `acceptance-test-config.yml` file to configure tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
68
+
Customize `acceptance-test-config.yml` file to configure acceptance tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
140
69
If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
141
70
142
-
## Dependency Management
143
-
All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.
144
-
We split dependencies between two groups, dependencies that are:
145
-
* required for your connector to work need to go to `MAIN_REQUIREMENTS` list.
146
-
* required for the testing need to go to `TEST_REQUIREMENTS` list
71
+
### Dependency Management
72
+
All of your dependencies should be managed via Poetry.
73
+
To add a new dependency, run:
74
+
```bash
75
+
poetry add <package-name>
76
+
```
147
77
148
-
### Publishing a new version of the connector
78
+
Please commit the changes to `pyproject.toml` and `poetry.lock` files.
79
+
80
+
## Publishing a new version of the connector
149
81
You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
150
82
1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-shopify test`
151
-
2. Bump the connector version in `metadata.yaml`: increment the `dockerImageTag` value. Please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors).
83
+
2. Bump the connector version (please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors)):
84
+
- bump the `dockerImageTag` value in in `metadata.yaml`
85
+
- bump the `version` value in `pyproject.toml`
152
86
3. Make sure the `metadata.yaml` content is up to date.
153
-
4. Make the connector documentation and its changelog is up to date (`docs/integrations/sources/shopify.md`).
87
+
4. Make sure the connector documentation and its changelog is up to date (`docs/integrations/sources/shopify.md`).
154
88
5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
155
89
6. Pat yourself on the back for being an awesome contributor.
156
90
7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
157
-
91
+
8. Once your PR is merged, the new version of the connector will be automatically published to Docker Hub and our connector registry.
0 commit comments