Skip to content

Update CDK & pytest versions for Vector DB destinations #37333

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 10 additions & 26 deletions airbyte-integrations/connectors/destination-astra/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,24 +10,11 @@ For information about how to use this connector within Airbyte, see [the documen

#### Minimum Python version required `= 3.9.0`

#### Activate Virtual Environment and install dependencies
From this connector directory, create a virtual environment:
```
python -m venv .venv
```

This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your
development environment of choice. To activate it from the terminal, run:
```
source .venv/bin/activate
pip install -r requirements.txt
### Installing the connector
From this connector directory, run:
```bash
poetry install --with dev
```
If you are in an IDE, follow your IDE's instructions to activate the virtualenv.

Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is
used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`.
If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything
should work as you expect.


#### Create credentials
Expand All @@ -43,8 +30,7 @@ and place them into `secrets/config.json`.
```
python main.py spec
python main.py check --config secrets/config.json
python main.py discover --config secrets/config.json
python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
python main.py write --config secrets/config.json --catalog integration_tests/configured_catalog.json
```

### Locally running the connector docker image
Expand Down Expand Up @@ -118,25 +104,23 @@ docker run --rm airbyte/destination-astra:dev spec
docker run --rm -v $(pwd)/secrets:/secrets airbyte/destination-astra:dev check --config /secrets/config.json
# messages.jsonl is a file containing line-separated JSON representing AirbyteMessages
cat messages.jsonl | docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/destination-astra:dev write --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
```


## Testing
Make sure to familiarize yourself with [pytest test discovery](https://docs.pytest.org/en/latest/goodpractices.html#test-discovery) to know how your test files and methods should be named.
First install test dependencies into your virtual environment:
```
pip install .[tests]
```

### Unit Tests
To run unit tests locally, from the connector directory run:
```
python -m pytest unit_tests
poetry run pytest -s unit_tests
```

### Integration Tests
There are two types of integration tests: Acceptance Tests (Airbyte's test suite for all destination connectors) and custom integration tests (which are specific to this connector).
#### Custom Integration tests
Place custom tests inside `integration_tests/` folder, then, from the connector root, run
```
python -m pytest integration_tests
poetry run pytest -s integration_tests
```
#### Acceptance Tests
Coming soon:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ data:
connectorSubtype: database
connectorType: destination
definitionId: 042ce96f-1158-4662-9543-e2ff015be97a
dockerImageTag: 0.1.1
dockerImageTag: 0.1.2
dockerRepository: airbyte/destination-astra
githubIssueLabel: destination-astra
icon: astra.svg
Expand Down
2,923 changes: 2,923 additions & 0 deletions airbyte-integrations/connectors/destination-astra/poetry.lock

Large diffs are not rendered by default.

29 changes: 29 additions & 0 deletions airbyte-integrations/connectors/destination-astra/pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
[build-system]
requires = [ "poetry-core>=1.0.0",]
build-backend = "poetry.core.masonry.api"

[tool.poetry]
name = "airbyte-destination-astra"
version = "0.1.2"
description = "Airbyte destination implementation for Astra DB."
authors = ["Airbyte <[email protected]>"]
license = "MIT"
readme = "README.md"
documentation = "https://docs.airbyte.com/integrations/destinations/astra"
homepage = "https://airbyte.com"
repository = "https://github.com/airbytehq/airbyte"

[[tool.poetry.packages]]
include = "destination_astra"

[tool.poetry.dependencies]
python = "^3.9,<3.12"
airbyte-cdk = {version = "0.81.6", extras = ["vector-db-based"]}

[tool.poetry.group.dev.dependencies]
pytest = "^7.2"
ruff = "^0.3.2"
mypy = "^1.9.0"

[tool.poetry.scripts]
destination-astra = "destination_astra.run:run"

This file was deleted.

23 changes: 0 additions & 23 deletions airbyte-integrations/connectors/destination-astra/setup.py

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,5 @@
!Dockerfile
!main.py
!destination_chroma
!setup.py
!pyproject.toml
!poetry.lock
45 changes: 0 additions & 45 deletions airbyte-integrations/connectors/destination-chroma/Dockerfile

This file was deleted.

36 changes: 17 additions & 19 deletions airbyte-integrations/connectors/destination-chroma/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,24 +10,11 @@ For information about how to use this connector within Airbyte, see [the documen

#### Minimum Python version required `= 3.7.0`

#### Build & Activate Virtual Environment and install dependencies
From this connector directory, create a virtual environment:
```
python -m venv .venv
```

This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your
development environment of choice. To activate it from the terminal, run:
```
source .venv/bin/activate
pip install -r requirements.txt
### Installing the connector
From this connector directory, run:
```bash
poetry install --with dev
```
If you are in an IDE, follow your IDE's instructions to activate the virtualenv.

Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is
used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`.
If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything
should work as you expect.

#### Create credentials
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/destinations/chroma)
Expand All @@ -42,8 +29,7 @@ and place them into `secrets/config.json`.
```
python main.py spec
python main.py check --config secrets/config.json
python main.py discover --config secrets/config.json
python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
python main.py write --config secrets/config.json --catalog integration_tests/configured_catalog.json
```

### Locally running the connector docker image
Expand Down Expand Up @@ -77,6 +63,18 @@ You can run our full test suite locally using [`airbyte-ci`](https://github.com/
airbyte-ci connectors --name=destination-chroma test
```

### Unit Tests
To run unit tests locally, from the connector directory run:
```
poetry run pytest -s unit_tests
```

### Integration Tests
To run integration tests locally, make sure you have a secrets/config.json as explained above, and then run:
```
poetry run pytest -s integration_tests
```

### Customizing acceptance Tests
Customize `acceptance-test-config.yml` file to configure tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,12 @@ data:
enabled: true
cloud:
enabled: false
connectorBuildOptions:
baseImage: docker.io/airbyte/python-connector-base:1.1.0@sha256:bd98f6505c6764b1b5f99d3aedc23dfc9e9af631a62533f60eb32b1d3dbab20c
connectorSubtype: vectorstore
connectorType: destination
definitionId: 0b75218b-f702-4a28-85ac-34d3d84c0fc2
dockerImageTag: 0.0.9
dockerImageTag: 0.0.10
dockerRepository: airbyte/destination-chroma
githubIssueLabel: destination-chroma
icon: chroma.svg
Expand Down
Loading
Loading