Skip to content

Commit 27b33d8

Browse files
jhammarstedtmarcosmarxmoctavia-squidington-iii
authored
🎉 New Source: DataScope [low-code cdk] (#18725)
* adding datascope connector * fixing tests * fixing readme * cleanup * adding incremental streams * correct tests * fixing non incremental streams * dropping incremental for some streams * update date cursor for incremental * rename parameters * correct integrqation test files * remove files * update start date example * add datascope to source def * add eof * auto-bump connector version Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]>
1 parent 0d795d5 commit 27b33d8

29 files changed

+718
-0
lines changed

‎airbyte-config/init/src/main/resources/seed/source_definitions.yaml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -314,6 +314,13 @@
314314
documentationUrl: https://docs.airbyte.com/integrations/sources/datadog
315315
sourceType: api
316316
releaseStage: alpha
317+
- name: Datascope
318+
sourceDefinitionId: 8e1ae2d2-4790-44d3-9d83-75b3fc3940ff
319+
dockerRepository: airbyte/source-datascope
320+
dockerImageTag: 0.1.0
321+
documentationUrl: https://docs.airbyte.com/integrations/sources/datascope
322+
sourceType: api
323+
releaseStage: alpha
317324
- name: Delighted
318325
sourceDefinitionId: cc88c43f-6f53-4e8a-8c4d-b284baaf9635
319326
dockerRepository: airbyte/source-delighted

‎airbyte-config/init/src/main/resources/seed/source_specs.yaml

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2697,6 +2697,33 @@
26972697
supportsNormalization: false
26982698
supportsDBT: false
26992699
supported_destination_sync_modes: []
2700+
- dockerImage: "airbyte/source-datascope:0.1.0"
2701+
spec:
2702+
documentationUrl: "https://docs.airbyte.com/integrations/sources/datascope"
2703+
connectionSpecification:
2704+
$schema: "http://json-schema.org/draft-07/schema#"
2705+
title: "Datascope Spec"
2706+
type: "object"
2707+
required:
2708+
- "api_key"
2709+
- "start_date"
2710+
additionalProperties: true
2711+
properties:
2712+
start_date:
2713+
title: "Start Date"
2714+
type: "string"
2715+
description: "Start date for the data to be replicated"
2716+
examples:
2717+
- "dd/mm/YYYY HH:MM"
2718+
pattern: "^[0-9]{2}/[0-9]{2}/[0-9]{4} [0-9]{2}:[0-9]{2}$"
2719+
api_key:
2720+
title: "Authorization"
2721+
type: "string"
2722+
description: "API Key"
2723+
airbyte_secret: true
2724+
supportsNormalization: false
2725+
supportsDBT: false
2726+
supported_destination_sync_modes: []
27002727
- dockerImage: "airbyte/source-delighted:0.1.4"
27012728
spec:
27022729
documentationUrl: "https://docsurl.com"

‎airbyte-integrations/connectors/source-datascope/.dockerignore

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
*
2+
!Dockerfile
3+
!main.py
4+
!source_datascope
5+
!setup.py
6+
!secrets

‎airbyte-integrations/connectors/source-datascope/BOOTSTRAP.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# DataScope
2+
DataScope is a mobile solution that helps you collect data offline, manage field teams, and share business insights. Use the intuitive Form Builder to create your forms, and then analyze the data you've collected via powerful and personalized dashboards.
3+
4+
The streams implemented allows you to pull data from the following DataScope objects:
5+
- Locations
6+
- Answers
7+
- Lists
8+
- Notifications
9+
10+
For more information about the DataScope API, see the [DataScope API documentation](https://dscope.github.io/docs/).

‎airbyte-integrations/connectors/source-datascope/Dockerfile

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
FROM python:3.9.11-alpine3.15 as base
2+
3+
# build and load all requirements
4+
FROM base as builder
5+
WORKDIR /airbyte/integration_code
6+
7+
# upgrade pip to the latest version
8+
RUN apk --no-cache upgrade \
9+
&& pip install --upgrade pip \
10+
&& apk --no-cache add tzdata build-base
11+
12+
13+
COPY setup.py ./
14+
# install necessary packages to a temporary folder
15+
RUN pip install --prefix=/install .
16+
17+
# build a clean environment
18+
FROM base
19+
WORKDIR /airbyte/integration_code
20+
21+
# copy all loaded and built libraries to a pure basic image
22+
COPY --from=builder /install /usr/local
23+
# add default timezone settings
24+
COPY --from=builder /usr/share/zoneinfo/Etc/UTC /etc/localtime
25+
RUN echo "Etc/UTC" > /etc/timezone
26+
27+
# bash is installed for more convenient debugging.
28+
RUN apk --no-cache add bash
29+
30+
# copy payload code only
31+
COPY main.py ./
32+
COPY source_datascope ./source_datascope
33+
34+
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
35+
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
36+
37+
LABEL io.airbyte.version=0.1.0
38+
LABEL io.airbyte.name=airbyte/source-datascope

‎airbyte-integrations/connectors/source-datascope/README.md

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
# Datascope Source
2+
3+
This is the repository for the Datascope configuration based source connector.
4+
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/datascope).
5+
6+
## Local development
7+
8+
#### Building via Gradle
9+
You can also build the connector in Gradle. This is typically used in CI and not needed for your development workflow.
10+
11+
To build using Gradle, from the Airbyte repository root, run:
12+
```
13+
./gradlew :airbyte-integrations:connectors:source-datascope:build
14+
```
15+
16+
#### Create credentials
17+
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/datascope)
18+
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_datascope/spec.yaml` file.
19+
Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
20+
See `integration_tests/sample_config.json` for a sample config file.
21+
22+
**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source datascope test creds`
23+
and place them into `secrets/config.json`.
24+
25+
### Locally running the connector docker image
26+
27+
#### Build
28+
First, make sure you build the latest Docker image:
29+
```
30+
docker build . -t airbyte/source-datascope:dev
31+
```
32+
33+
You can also build the connector image via Gradle:
34+
```
35+
./gradlew :airbyte-integrations:connectors:source-datascope:airbyteDocker
36+
```
37+
When building via Gradle, the docker image name and tag, respectively, are the values of the `io.airbyte.name` and `io.airbyte.version` `LABEL`s in
38+
the Dockerfile.
39+
40+
#### Run
41+
Then run any of the connector commands as follows:
42+
```
43+
docker run --rm airbyte/source-datascope:dev spec
44+
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-datascope:dev check --config /secrets/config.json
45+
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-datascope:dev discover --config /secrets/config.json
46+
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-datascope:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
47+
```
48+
## Testing
49+
50+
#### Acceptance Tests
51+
Customize `acceptance-test-config.yml` file to configure tests. See [Source Acceptance Tests](https://docs.airbyte.io/connector-development/testing-connectors/source-acceptance-tests-reference) for more information.
52+
If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
53+
54+
To run your integration tests with docker
55+
56+
### Using gradle to run tests
57+
All commands should be run from airbyte project root.
58+
To run unit tests:
59+
```
60+
./gradlew :airbyte-integrations:connectors:source-datascope:unitTest
61+
```
62+
To run acceptance and custom integration tests:
63+
```
64+
./gradlew :airbyte-integrations:connectors:source-datascope:integrationTest
65+
```
66+
67+
## Dependency Management
68+
All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.
69+
We split dependencies between two groups, dependencies that are:
70+
* required for your connector to work need to go to `MAIN_REQUIREMENTS` list.
71+
* required for the testing need to go to `TEST_REQUIREMENTS` list
72+
73+
### Publishing a new version of the connector
74+
You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
75+
1. Make sure your changes are passing unit and integration tests.
76+
1. Bump the connector version in `Dockerfile` -- just increment the value of the `LABEL io.airbyte.version` appropriately (we use [SemVer](https://semver.org/)).
77+
1. Create a Pull Request.
78+
1. Pat yourself on the back for being an awesome contributor.
79+
1. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.

‎airbyte-integrations/connectors/source-datascope/__init__.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
#
2+
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
3+
#

‎airbyte-integrations/connectors/source-datascope/acceptance-test-config.yml

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# See [Source Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/source-acceptance-tests-reference)
2+
# for more information about how to configure these tests
3+
connector_image: airbyte/source-datascope:dev
4+
tests:
5+
spec:
6+
- spec_path: "source_datascope/spec.yaml"
7+
connection:
8+
- config_path: "secrets/config.json"
9+
status: "succeed"
10+
- config_path: "integration_tests/invalid_config.json"
11+
status: "failed"
12+
discovery:
13+
- config_path: "secrets/config.json"
14+
basic_read:
15+
- config_path: "secrets/config.json"
16+
configured_catalog_path: "integration_tests/configured_catalog.json"
17+
empty_streams: ["notifications", "lists"]
18+
incremental:
19+
- config_path: "secrets/config.json"
20+
configured_catalog_path: "integration_tests/configured_catalog.json"
21+
future_state_path: "integration_tests/abnormal_state.json"
22+
full_refresh:
23+
- config_path: "secrets/config.json"
24+
configured_catalog_path: "integration_tests/configured_catalog.json"

‎airbyte-integrations/connectors/source-datascope/acceptance-test-docker.sh

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
#!/usr/bin/env sh
2+
3+
# Build latest connector image
4+
docker build . -t $(cat acceptance-test-config.yml | grep "connector_image" | head -n 1 | cut -d: -f2-)
5+
6+
# Pull latest acctest image
7+
docker pull airbyte/source-acceptance-test:latest
8+
9+
# Run
10+
docker run --rm -it \
11+
-v /var/run/docker.sock:/var/run/docker.sock \
12+
-v /tmp:/tmp \
13+
-v $(pwd):/test_input \
14+
airbyte/source-acceptance-test \
15+
--acceptance-test-config /test_input
16+

‎airbyte-integrations/connectors/source-datascope/build.gradle

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
plugins {
2+
id 'airbyte-python'
3+
id 'airbyte-docker'
4+
id 'airbyte-source-acceptance-test'
5+
}
6+
7+
airbytePython {
8+
moduleDirectory 'source_datascope'
9+
}

‎airbyte-integrations/connectors/source-datascope/integration_tests/__init__.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
#
2+
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
3+
#

‎airbyte-integrations/connectors/source-datascope/integration_tests/abnormal_state.json

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"answers": {
3+
"created_at": "01/01/9999 00:00"
4+
}
5+
}

‎airbyte-integrations/connectors/source-datascope/integration_tests/acceptance.py

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
#
2+
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
3+
#
4+
5+
6+
import pytest
7+
8+
pytest_plugins = ("source_acceptance_test.plugin",)
9+
10+
11+
@pytest.fixture(scope="session", autouse=True)
12+
def connector_setup():
13+
"""This fixture is a placeholder for external resources that acceptance test might require."""
14+
yield

‎airbyte-integrations/connectors/source-datascope/integration_tests/configured_catalog.json

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
{
2+
"streams": [
3+
{
4+
"stream": {
5+
"name": "locations",
6+
"json_schema": {},
7+
"source_defined_cursor": true,
8+
"source_defined_primary_key": [["id"]],
9+
"supported_sync_modes": ["full_refresh"]
10+
},
11+
"sync_mode": "incremental",
12+
"destination_sync_mode": "append"
13+
},
14+
{
15+
"stream": {
16+
"name": "answers",
17+
"json_schema": {},
18+
"supported_sync_modes": [
19+
"full_refresh",
20+
"incremental"]},
21+
"sync_mode": "incremental",
22+
"destination_sync_mode": "overwrite"
23+
},
24+
{
25+
"stream": {
26+
"name": "lists",
27+
"json_schema": {},
28+
"supported_sync_modes": ["full_refresh"]
29+
},
30+
"sync_mode": "full_refresh",
31+
"destination_sync_mode": "overwrite"
32+
},
33+
{
34+
"stream": {
35+
"name": "notifications",
36+
"json_schema": {},
37+
"supported_sync_modes": ["full_refresh"]
38+
},
39+
"sync_mode": "full_refresh",
40+
"destination_sync_mode": "overwrite"
41+
}
42+
]
43+
}

‎airbyte-integrations/connectors/source-datascope/integration_tests/invalid_config.json

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
{
2+
"api_key": "abctestconfig",
3+
"start_date": "2019-01-01T00:00:00Z"
4+
}

‎airbyte-integrations/connectors/source-datascope/integration_tests/sample_config.json

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
{
2+
"api_key": "test_key",
3+
"start_date": "2022-10-30 00:00"
4+
}

‎airbyte-integrations/connectors/source-datascope/integration_tests/sample_state.json

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"answers": {
3+
"created_at": "01/01/2000 00:00"
4+
}
5+
}

‎airbyte-integrations/connectors/source-datascope/main.py

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
#
2+
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
3+
#
4+
5+
6+
import sys
7+
8+
from airbyte_cdk.entrypoint import launch
9+
from source_datascope import SourceDatascope
10+
11+
if __name__ == "__main__":
12+
source = SourceDatascope()
13+
launch(source, sys.argv[1:])

‎airbyte-integrations/connectors/source-datascope/requirements.txt

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
-e ../../bases/source-acceptance-test
2+
-e .

‎airbyte-integrations/connectors/source-datascope/setup.py

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
#
2+
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
3+
#
4+
5+
6+
from setuptools import find_packages, setup
7+
8+
MAIN_REQUIREMENTS = [
9+
"airbyte-cdk~=0.1",
10+
]
11+
12+
TEST_REQUIREMENTS = [
13+
"pytest~=6.1",
14+
"pytest-mock~=3.6.1",
15+
"source-acceptance-test",
16+
]
17+
18+
setup(
19+
name="source_datascope",
20+
description="Source implementation for Datascope.",
21+
author="Airbyte",
22+
author_email="[email protected]",
23+
packages=find_packages(),
24+
install_requires=MAIN_REQUIREMENTS,
25+
package_data={"": ["*.json", "*.yaml", "schemas/*.json", "schemas/shared/*.json"]},
26+
extras_require={
27+
"tests": TEST_REQUIREMENTS,
28+
},
29+
)

‎airbyte-integrations/connectors/source-datascope/source_datascope/__init__.py

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
#
2+
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
3+
#
4+
5+
6+
from .source import SourceDatascope
7+
8+
__all__ = ["SourceDatascope"]

0 commit comments

Comments
 (0)