Skip to content

Commit 15a3272

Browse files
authored
[schema_generator] Use Poetry for schema_generator (#37455)
1 parent 097709a commit 15a3272

File tree

5 files changed

+1310
-37
lines changed

5 files changed

+1310
-37
lines changed

tools/schema_generator/README.md

+13-8
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,26 @@
11
# Schema Generator
2+
23
Util for generating catalog schema from a connector `read` command output.
34

45
## Prerequisites
56

67
To use this tool you first need to:
78

89
- Define all your streams.
9-
- Create schema files for each stream, containing only `{}` (valid json files). See [this doc section](https://docs.airbyte.com/connector-development/cdk-python/schemas#static-schemas) for instructions on how to name these files.
10-
- Build you container docker image.
10+
- Create schema files for each stream, containing only `{}` (valid json files). See
11+
[this doc section](https://docs.airbyte.com/connector-development/cdk-python/schemas#static-schemas)
12+
for instructions on how to name these files.
1113

12-
Going through all the steps above should enable you to run the `read` command of your connector using the docker image, which is the input for this tool.
14+
Going through all the steps above should enable you to run the `read` command of your connector
15+
using the docker image, which is the input for this tool.
1316

1417
## Usage
1518

1619
First install the tools in it's own virtual environment:
1720

1821
```bash
1922
$ cd tools/schema_generator # assumes you are starting from the root of the Airbyte project.
20-
$ python -m venv .venv # Create a virtual environment in the .venv directory
21-
$ source .venv/bin/activate # enable the venv
22-
$ pip install -r requirements.txt
23+
$ poetry install
2324
```
2425

2526
To use a connector's `run` command we first need a `ConfiguredAirbyteCatalog`:
@@ -28,11 +29,15 @@ To use a connector's `run` command we first need a `ConfiguredAirbyteCatalog`:
2829
$ ../../airbyte-integrations/connectors/<your-connector> # you need to use the tool at the root folder of a connector
2930
$ docker run --rm -v $(pwd)/secrets:/secrets airbyte/<your-connector-image-name>:dev discover --config /secrets/config.json | schema_generator --configure-catalog
3031
```
31-
This will created the file **configured_catalog.json** in the **integration_tests** folder in the current working directory.
32+
33+
This will created the file **configured_catalog.json** in the **integration_tests** folder in the
34+
current working directory.
3235

3336
Now you're all set to run the following command and generate your schemas:
3437

3538
```bash
3639
$ docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/<your-connector-image-name>:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json | schema_generator --infer-schemas
3740
```
38-
Which will create schema files for all streams and place them in the **schemas** folder in the current working directory.
41+
42+
Which will create schema files for all streams and place them in the **schemas** folder in the
43+
current working directory.

0 commit comments

Comments
 (0)