Skip to content

Commit 9d3dd2f

Browse files
olivermeyergl-pix
olivermeyer
authored and
gl-pix
committed
🎉 New source: Dixa (#4358)
1 parent f1304fa commit 9d3dd2f

26 files changed

+1182
-2
lines changed

‎airbyte-integrations/builds.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,9 @@
1313

1414
AWS CloudTrail [![source-aws-cloudtrail](https://img.shields.io/endpoint?url=https%3A%2F%2Fstatus-api.airbyte.io%2Ftests%2Fsummary%2Fsource-aws-cloudtrail%2Fbadge.json)](https://status-api.airbyte.io/tests/summary/source-aws-cloudtrail)
1515

16-
Braintree [![source-braintree-singer](https://img.shields.io/endpoint?url=https%3A%2F%2Fstatus-api.airbyte.io%2Ftests%2Fsummary%2Fsource-braintree-singer%2Fbadge.json)](https://status-api.airbyte.io/tests/summary/source-braintree-singer)
16+
Braintree [![source-braintree-singer](https://img.shields.io/endpoint?url=https%3A%2F%2Fstatus-api.airbyte.io%2Ftests%2Fsummary%2Fsource-braintree-singer%2Fbadge.json)](https://status-api.airbyte.io/tests/summary/source-braintree-singer)
17+
18+
Dixa [![source-dixa](https://img.shields.io/endpoint?url=https%3A%2F%2Fstatus-api.airbyte.io%2Ftests%2Fsummary%2Fsource-dixa%2Fbadge.json)](https://status-api.airbyte.io/tests/summary/source-dixa)
1719

1820
Drift [![source-drift](https://img.shields.io/endpoint?url=https%3A%2F%2Fstatus-api.airbyte.io%2Ftests%2Fsummary%2Fsource-drift%2Fbadge.json)](https://status-api.airbyte.io/tests/summary/source-drift)
1921

‎airbyte-integrations/connectors/source-dixa/.dockerignore

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
*
2+
!Dockerfile
3+
!Dockerfile.test
4+
!main.py
5+
!source_dixa
6+
!setup.py
7+
!secrets

‎airbyte-integrations/connectors/source-dixa/Dockerfile

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
FROM python:3.7-slim
2+
3+
# Bash is installed for more convenient debugging.
4+
RUN apt-get update && apt-get install -y bash && rm -rf /var/lib/apt/lists/*
5+
6+
WORKDIR /airbyte/integration_code
7+
COPY source_dixa ./source_dixa
8+
COPY main.py ./
9+
COPY setup.py ./
10+
RUN pip install .
11+
12+
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
13+
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
14+
15+
LABEL io.airbyte.version=0.1.0
16+
LABEL io.airbyte.name=airbyte/source-dixa

‎airbyte-integrations/connectors/source-dixa/README.md

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
# Dixa Source
2+
3+
### DISCLAIMER
4+
<!-- TODO acceptance tests are disabled in CI pending a Dixa Sandbox: https://github.com/airbytehq/airbyte/issues/4667 -->
5+
This source is currently not running CI pending the creation of a sandbox account, tracked [here](https://github.com/airbytehq/airbyte/issues/4667).
6+
### END DISCLAIMER
7+
8+
This is the repository for the Dixa source connector, written in Python.
9+
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/dixa).
10+
11+
## Local development
12+
13+
### Prerequisites
14+
**To iterate on this connector, make sure to complete this prerequisites section.**
15+
16+
#### Minimum Python version required `= 3.7.0`
17+
18+
#### Build & Activate Virtual Environment and install dependencies
19+
From this connector directory, create a virtual environment:
20+
```
21+
python -m venv .venv
22+
```
23+
24+
This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your
25+
development environment of choice. To activate it from the terminal, run:
26+
```
27+
source .venv/bin/activate
28+
pip install -r requirements.txt
29+
```
30+
If you are in an IDE, follow your IDE's instructions to activate the virtualenv.
31+
32+
Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is
33+
used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`.
34+
If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything
35+
should work as you expect.
36+
37+
#### Building via Gradle
38+
You can also build the connector in Gradle. This is typically used in CI and not needed for your development workflow.
39+
40+
To build using Gradle, from the Airbyte repository root, run:
41+
```
42+
./gradlew :airbyte-integrations:connectors:source-dixa:build
43+
```
44+
45+
#### Create credentials
46+
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/dixa)
47+
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_dixa/spec.json` file.
48+
Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
49+
See `integration_tests/sample_config.json` for a sample config file.
50+
51+
**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source dixa test creds`
52+
and place them into `secrets/config.json`.
53+
54+
### Locally running the connector
55+
```
56+
python main.py spec
57+
python main.py check --config secrets/config.json
58+
python main.py discover --config secrets/config.json
59+
python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
60+
```
61+
62+
### Locally running the connector docker image
63+
64+
#### Build
65+
First, make sure you build the latest Docker image:
66+
```
67+
docker build . -t airbyte/source-dixa:dev
68+
```
69+
70+
You can also build the connector image via Gradle:
71+
```
72+
./gradlew :airbyte-integrations:connectors:source-dixa:airbyteDocker
73+
```
74+
When building via Gradle, the docker image name and tag, respectively, are the values of the `io.airbyte.name` and `io.airbyte.version` `LABEL`s in
75+
the Dockerfile.
76+
77+
#### Run
78+
Then run any of the connector commands as follows:
79+
```
80+
docker run --rm airbyte/source-dixa:dev spec
81+
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-dixa:dev check --config /secrets/config.json
82+
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-dixa:dev discover --config /secrets/config.json
83+
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-dixa:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
84+
```
85+
## Testing
86+
Make sure to familiarize yourself with [pytest test discovery](https://docs.pytest.org/en/latest/goodpractices.html#test-discovery) to know how your test files and methods should be named.
87+
First install test dependencies into your virtual environment:
88+
```
89+
pip install .[tests]
90+
```
91+
### Unit Tests
92+
To run unit tests locally, from the connector directory run:
93+
```
94+
python -m pytest unit_tests
95+
```
96+
97+
### Integration Tests
98+
There are two types of integration tests: Acceptance Tests (Airbyte's test suite for all source connectors) and custom integration tests (which are specific to this connector).
99+
#### Custom Integration tests
100+
Place custom tests inside `integration_tests/` folder, then, from the connector root, run
101+
```
102+
python -m pytest integration_tests
103+
```
104+
#### Acceptance Tests
105+
Customize `acceptance-test-config.yml` file to configure tests. See [Source Acceptance Tests](source-acceptance-tests.md) for more information.
106+
If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
107+
To run your integration tests with acceptance tests, from the connector root, run
108+
```
109+
python -m pytest integration_tests -p integration_tests.acceptance
110+
```
111+
To run your integration tests with docker
112+
113+
### Using gradle to run tests
114+
All commands should be run from airbyte project root.
115+
To run unit tests:
116+
```
117+
./gradlew :airbyte-integrations:connectors:source-dixa:unitTest
118+
```
119+
To run acceptance and custom integration tests:
120+
```
121+
./gradlew :airbyte-integrations:connectors:source-dixa:integrationTest
122+
```
123+
124+
## Dependency Management
125+
All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.
126+
We split dependencies between two groups, dependencies that are:
127+
* required for your connector to work need to go to `MAIN_REQUIREMENTS` list.
128+
* required for the testing need to go to `TEST_REQUIREMENTS` list
129+
130+
### Publishing a new version of the connector
131+
You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
132+
1. Make sure your changes are passing unit and integration tests.
133+
1. Bump the connector version in `Dockerfile` -- just increment the value of the `LABEL io.airbyte.version` appropriately (we use [SemVer](https://semver.org/)).
134+
1. Create a Pull Request.
135+
1. Pat yourself on the back for being an awesome contributor.
136+
1. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.

‎airbyte-integrations/connectors/source-dixa/acceptance-test-config.yml

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# See [Source Acceptance Tests](https://docs.airbyte.io/contributing-to-airbyte/building-new-connector/source-acceptance-tests.md)
2+
# for more information about how to configure these tests
3+
connector_image: airbyte/source-dixa:dev
4+
tests:
5+
spec:
6+
- spec_path: "source_dixa/spec.json"
7+
connection:
8+
- config_path: "secrets/config.json"
9+
status: "succeed"
10+
- config_path: "integration_tests/invalid_config.json"
11+
status: "failed"
12+
discovery:
13+
- config_path: "secrets/config.json"
14+
basic_read:
15+
- config_path: "secrets/config.json"
16+
configured_catalog_path: "integration_tests/configured_catalog.json"
17+
validate_output_from_all_streams: yes
18+
# TODO uncomment this block to specify that the tests should assert the connector outputs the records provided in the input file a file
19+
# expect_records:
20+
# path: "integration_tests/expected_records.txt"
21+
# extra_fields: no
22+
# exact_order: no
23+
# extra_records: yes
24+
incremental:
25+
- config_path: "secrets/config.json"
26+
configured_catalog_path: "integration_tests/configured_catalog.json"
27+
# future_state_path: "integration_tests/abnormal_state.json"
28+
# We skip the full_refresh test because of unusual behaviour in the Dixa API.
29+
# We observed cases where a record was updated without the updated_at value changing.
30+
# See the thread below for further information:
31+
# https://airbytehq.slack.com/archives/C01VDDEGL7M/p1625319909273300
32+
# full_refresh:
33+
# - config_path: "secrets/config.json"
34+
# configured_catalog_path: "integration_tests/configured_catalog.json"

‎airbyte-integrations/connectors/source-dixa/acceptance-test-docker.sh

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
#!/usr/bin/env sh
2+
docker run --rm -it \
3+
-v /var/run/docker.sock:/var/run/docker.sock \
4+
-v /tmp:/tmp \
5+
-v $(pwd):/test_input \
6+
airbyte/source-acceptance-test \
7+
--acceptance-test-config /test_input

‎airbyte-integrations/connectors/source-dixa/build.gradle

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
plugins {
2+
id 'airbyte-python'
3+
id 'airbyte-docker'
4+
// TODO acceptance tests are disabled in CI pending a Dixa Sandbox: https://github.com/airbytehq/airbyte/issues/4667
5+
// id 'airbyte-source-acceptance-test'
6+
}
7+
8+
airbytePython {
9+
moduleDirectory 'source_dixa'
10+
}
11+
12+
// TODO acceptance tests are disabled in CI pending a Dixa Sandbox: https://github.com/airbytehq/airbyte/issues/4667
13+
// no-op integration test task
14+
task("integrationTest")
15+
16+
dependencies {
17+
implementation files(project(':airbyte-integrations:bases:source-acceptance-test').airbyteDocker.outputs)
18+
implementation files(project(':airbyte-integrations:bases:base-python').airbyteDocker.outputs)
19+
}

‎airbyte-integrations/connectors/source-dixa/integration_tests/__init__.py

Whitespace-only changes.

‎airbyte-integrations/connectors/source-dixa/integration_tests/abnormal_state.json

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"conversation_export": {
3+
"updated_at": 9999999999999
4+
}
5+
}

‎airbyte-integrations/connectors/source-dixa/integration_tests/acceptance.py

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
#
2+
# MIT License
3+
#
4+
# Copyright (c) 2020 Airbyte
5+
#
6+
# Permission is hereby granted, free of charge, to any person obtaining a copy
7+
# of this software and associated documentation files (the "Software"), to deal
8+
# in the Software without restriction, including without limitation the rights
9+
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
10+
# copies of the Software, and to permit persons to whom the Software is
11+
# furnished to do so, subject to the following conditions:
12+
#
13+
# The above copyright notice and this permission notice shall be included in all
14+
# copies or substantial portions of the Software.
15+
#
16+
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
17+
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
18+
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
19+
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
20+
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
21+
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
22+
# SOFTWARE.
23+
#
24+
25+
26+
import pytest
27+
28+
pytest_plugins = ("source_acceptance_test.plugin",)
29+
30+
31+
@pytest.fixture(scope="session", autouse=True)
32+
def connector_setup():
33+
""" This fixture is a placeholder for external resources that acceptance test might require."""
34+
# TODO: setup test dependencies if needed. otherwise remove the TODO comments
35+
yield
36+
# TODO: clean up test dependencies

‎airbyte-integrations/connectors/source-dixa/integration_tests/catalog.json

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
{
2+
"streams": [
3+
{
4+
"name": "TODO fix this file",
5+
"supported_sync_modes": ["full_refresh", "incremental"],
6+
"source_defined_cursor": true,
7+
"default_cursor_field": "column1",
8+
"json_schema": {
9+
"$schema": "http://json-schema.org/draft-07/schema#",
10+
"type": "object",
11+
"properties": {
12+
"column1": {
13+
"type": "string"
14+
},
15+
"column2": {
16+
"type": "number"
17+
}
18+
}
19+
}
20+
},
21+
{
22+
"name": "table1",
23+
"supported_sync_modes": ["full_refresh", "incremental"],
24+
"source_defined_cursor": false,
25+
"json_schema": {
26+
"$schema": "http://json-schema.org/draft-07/schema#",
27+
"type": "object",
28+
"properties": {
29+
"column1": {
30+
"type": "string"
31+
},
32+
"column2": {
33+
"type": "number"
34+
}
35+
}
36+
}
37+
}
38+
]
39+
}

0 commit comments

Comments
 (0)