update doc to reference poetry

Author: alafanechere

docs/connector-development/config-based/tutorial/0-getting-started.md

@@ -42,6 +42,7 @@ This can be done by signing up for the Free tier plan on [Exchange Rates Data AP
 
 - An Exchange Rates API key
 - Python >= 3.9
+- [Poetry](https://python-poetry.org/)
 - Docker must be running
 - NodeJS
 - [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#L1) CLI

docs/connector-development/config-based/tutorial/2-install-dependencies.md

@@ -1,25 +1,17 @@
 # Step 2: Install dependencies
 
-Let's create a python virtual environment for our source.
-You can do this by executing the following commands from the root of the Airbyte repository.
-
-The command below assume that `python` points to a version of python >=3.9.0. On some systems, `python` points to a Python2 installation and `python3` points to Python3.
-If this is the case on your machine, substitute the `python` commands with `python3`.
-The subsequent `python` invocations will use the virtual environment created for the connector.
 
 ```bash
 cd ../../connectors/source-exchange-rates-tutorial
-python -m venv .venv
-source .venv/bin/activate
-pip install -r requirements.txt
+poetry install
 ```
 
 These steps create an initial python environment, and install the dependencies required to run an API Source connector.
 
 Let's verify everything works as expected by running the Airbyte `spec` operation:
 
 ```bash
-python main.py spec
+poetry run source-exchange-rates-tutorial spec
 ```
 
 You should see an output similar to the one below:

docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md

@@ -200,7 +200,7 @@ spec:
 We can now run the `check` operation, which verifies the connector can connect to the API source.
 
 ```bash
-python main.py check --config secrets/config.json
+poetry run source-exchange-rates-tutorial check --config secrets/config.json
 ```
 
 which should now succeed with logs similar to:

docs/connector-development/config-based/tutorial/4-reading-data.md

@@ -44,7 +44,7 @@ As an alternative to storing the stream's data schema to the `schemas/` director
 Reading from the source can be done by running the `read` operation
 
 ```bash
-python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
+poetry run source-exchange-rates-tutorial read --config secrets/config.json --catalog integration_tests/configured_catalog.json
 ```
 
 The logs should show that 1 record was read from the stream.
@@ -57,7 +57,7 @@ The logs should show that 1 record was read from the stream.
 The `--debug` flag can be set to print out debug information, including the outgoing request and its associated response
 
 ```bash
-python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json --debug
+poetry run source-exchange-rates-tutorial read --config secrets/config.json --catalog integration_tests/configured_catalog.json --debug
 ```
 
 ## Next steps

docs/connector-development/config-based/tutorial/5-incremental-reads.md

@@ -76,7 +76,7 @@ definitions:
 You can test these changes by executing the `read` operation:
 
 ```bash
-python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
+poetry run source-exchange-rates-tutorial read --config secrets/config.json --catalog integration_tests/configured_catalog.json
 ```
 
 By reading the output record, you should see that we read historical data instead of the latest exchange rate.
@@ -240,7 +240,7 @@ spec:
 Running the `read` operation will now read all data for all days between start_date and now:
 
 ```bash
-python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
+poetry run source-exchange-rates-tutorial read --config secrets/config.json --catalog integration_tests/configured_catalog.json
 ```
 
 The operation should now output more than one record:
@@ -295,7 +295,7 @@ We can simulate incremental syncs by creating a state file containing the last s
 Running the `read` operation will now only read data for dates later than the given state:
 
 ```bash
-python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json --state integration_tests/sample_state.json
+poetry run source-exchange-rates-tutorial read --config secrets/config.json --catalog integration_tests/configured_catalog.json --state integration_tests/sample_state.json
 ```
 
 There shouldn't be any data read if the state is today's date:

docs/connector-development/config-based/tutorial/6-testing.md

@@ -30,7 +30,7 @@ and `integration_tests/abnormal_state.json` with
 You can run the [acceptance tests](https://github.com/airbytehq/airbyte/blob/master/docs/connector-development/testing-connectors/connector-acceptance-tests-reference.md#L1) with the following commands using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#L1):
 
 ```bash
-airbyte-ci connectors --use-remote-secrets=false --name source-exchange-rates-tutorial test
+airbyte-ci connectors --use-remote-secrets=false --name source-exchange-rates-tutorial test --only-step=acceptance
 ```
 
 ## Next steps:

docs/connector-development/testing-connectors/README.md

@@ -15,8 +15,8 @@ Unit and integration tests can be run directly from the connector code.
 
 Using `pytest` for Python connectors:
 ```bash
-python -m pytest unit_tests/
-python -m pytest integration_tests/
+poetry run pytest tests/unit_tests/
+poetry run pytest tests/integration_tests/
 ```
 
 Using `gradle` for Java connectors:

docs/connector-development/tutorials/building-a-python-source.md

@@ -64,18 +64,16 @@ $ ./generate.sh
 
 Select the `python` template and then input the name of your connector. For this walk through we will refer to our source as `example-python`
 
-### Step 2: Build the newly generated source
+### Step 2: Install the newly generated source
 
-Build the source by running:
+Install the source by running:
 
 ```bash
 cd airbyte-integrations/connectors/source-<name>
-python -m venv .venv # Create a virtual environment in the .venv directory
-source .venv/bin/activate # enable the venv
-pip install -r requirements.txt
+poetry install
 ```
 
-This step sets up the initial python environment. **All** subsequent `python` or `pip` commands assume you have activated your virtual environment.
+This step sets up the initial python environment. 
 
 ### Step 3: Set up your Airbyte development environment
 
@@ -112,10 +110,10 @@ You'll notice in your source's directory that there is a python file called `mai
 
 ```bash
 # from airbyte-integrations/connectors/source-<source-name>
-python main.py spec
-python main.py check --config secrets/config.json
-python main.py discover --config secrets/config.json
-python main.py read --config secrets/config.json --catalog sample_files/configured_catalog.json
+poetry run source-<source-name> spec
+poetry run source-<source-name> check --config secrets/config.json
+poetry run source-<source-name> discover --config secrets/config.json
+poetry run source-<source-name> read --config secrets/config.json --catalog sample_files/configured_catalog.json
 ```
 
 The nice thing about this approach is that you can iterate completely within in python. The downside is that you are not quite running your source as it will actually be run by Airbyte. Specifically you're not running it from within the docker container that will house it.
@@ -182,7 +180,7 @@ The nice thing about this approach is that you are running your source exactly a
 During development of your connector, you can enable the printing of detailed debug information during a sync by specifying the `--debug` flag. This will allow you to get a better picture of what is happening during each step of your sync.
 
 ```bash
-python main.py read --config secrets/config.json --catalog sample_files/configured_catalog.json --debug
+poetry run source-<source-name> read --config secrets/config.json --catalog sample_files/configured_catalog.json --debug
 ```
 
 In addition to the preset CDK debug statements, you can also emit custom debug information from your connector by introducing your own debug statements:
@@ -233,7 +231,8 @@ As described in the template code, this method takes in the same config object a
 
 The Connector Acceptance Tests are a set of tests that run against all sources. These tests are run in the Airbyte CI to prevent regressions. They also can help you sanity check that your source works as expected. The following [article](../testing-connectors/connector-acceptance-tests-reference.md) explains Connector Acceptance Tests and how to run them.
 
-You can run the tests using `./gradlew :airbyte-integrations:connectors:source-<source-name>:integrationTest`. Make sure to run this command from the Airbyte repository root.
+You can run the tests using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
+`airbyte-ci connectors --name source-<source-name> test --only-step=acceptance`
 
 :::info
 In some rare cases we make exceptions and allow a source to not need to pass all the standard tests. If for some reason you think your source cannot reasonably pass one of the tests cases, reach out to us on github or slack, and we can determine whether there's a change we can make so that the test will pass or if we should skip that test for your source.
@@ -245,15 +244,15 @@ The connector acceptance tests are meant to cover the basic functionality of a s
 
 #### Unit Tests
 
-Add any relevant unit tests to the `unit_tests` directory. Unit tests should _not_ depend on any secrets.
+Add any relevant unit tests to the `tests/unit_tests` directory. Unit tests should _not_ depend on any secrets.
 
-You can run the tests using `python -m pytest -s unit_tests`
+You can run the tests using `poetry run pytest tests/unit_tests`
 
 #### Integration Tests
 
 Place any integration tests in the `integration_tests` directory such that they can be [discovered by pytest](https://docs.pytest.org/en/6.2.x/goodpractices.html#conventions-for-python-test-discovery).
 
-Run integration tests using `python -m pytest -s integration_tests`.
+You can run the tests using `poetry run pytest tests/integration_tests`
 
 ### Step 10: Update the `README.md`
 

docs/connector-development/tutorials/cdk-speedrun.md

@@ -11,6 +11,7 @@ If you are a visual learner and want to see a video version of this guide going
 ## Dependencies
 
 1. Python >= 3.9
+2. [Poetry](https://python-poetry.org/)
 2. Docker
 3. NodeJS
 
@@ -30,9 +31,7 @@ Select the `Python HTTP API Source` and name it `python-http-example`.
 
 ```bash
 cd ../../connectors/source-python-http-example
-python -m venv .venv # Create a virtual environment in the .venv directory
-source .venv/bin/activate
-pip install -r requirements.txt
+poetry install
 ```
 
 ### Define Connector Inputs
@@ -169,7 +168,7 @@ This file defines your output schema for every endpoint that you want to impleme
 Test your discover function. You should receive a fairly large JSON object in return.
 
 ```bash
-python main.py discover --config sample_files/config.json
+poetry run source-python-http-example discover --config sample_files/config.json
 ```
 
 Note that our discover function is using the `pokemon_name` config variable passed in from the `Pokemon` stream when we set it in the `__init__` function.
@@ -226,7 +225,7 @@ We now need a catalog that defines all of our streams. We only have one stream:
 Let's read some data.
 
 ```bash
-python main.py read --config sample_files/config.json --catalog sample_files/configured_catalog.json
+poetry run source-python-http-example read --config sample_files/config.json --catalog sample_files/configured_catalog.json
 ```
 
 If all goes well, containerize it so you can use it in the UI:

docs/connector-development/tutorials/cdk-tutorial-python-http/connection-checking.md

@@ -46,17 +46,17 @@ Let's test out this implementation by creating two objects: a valid and an inval
 mkdir sample_files
 echo '{"start_date": "2022-04-01", "base": "USD", "apikey": <your_apikey>}'  > secrets/config.json
 echo '{"start_date": "2022-04-01", "base": "BTC", "apikey": <your_apikey>}'  > secrets/invalid_config.json
-python main.py check --config secrets/config.json
-python main.py check --config secrets/invalid_config.json
+poetry run source-python-http-example check --config secrets/config.json
+poetry run source-python-http-example check --config secrets/invalid_config.json
 ```
 
 You should see output like the following:
 
 ```text
-> python main.py check --config secrets/config.json
+> poetry run source-python-http-example check --config secrets/config.json
 {"type": "CONNECTION_STATUS", "connectionStatus": {"status": "SUCCEEDED"}}
 
-> python main.py check --config secrets/invalid_config.json
+> poetry run source-python-http-example check --config secrets/invalid_config.json
 {"type": "CONNECTION_STATUS", "connectionStatus": {"status": "FAILED", "message": "Input currency BTC is invalid. Please input one of the following currencies: {'DKK', 'USD', 'CZK', 'BGN', 'JPY'}"}}
 ```
 

docs/connector-development/tutorials/cdk-tutorial-python-http/declare-schema.md

@@ -63,7 +63,7 @@ Having created this stream in code, we'll put a file `exchange_rates.json` in th
 With `.json` schema file in place, let's see if the connector can now find this schema and produce a valid catalog:
 
 ```text
-python main.py discover --config secrets/config.json # this is not a mistake, the schema file is found by naming snake_case naming convention as specified above
+poetry run source-python-http-example discover --config secrets/config.json # this is not a mistake, the schema file is found by naming snake_case naming convention as specified above
 ```
 
 you should see some output like:

docs/connector-development/tutorials/cdk-tutorial-python-http/getting-started.md

@@ -7,6 +7,7 @@ This is a step-by-step guide for how to create an Airbyte source in Python to re
 ## Requirements
 
 * Python >= 3.9
+* [Poetry](https://python-poetry.org/)
 * Docker
 * NodeJS \(only used to generate the connector\). We'll remove the NodeJS dependency soon.
 

docs/connector-development/tutorials/cdk-tutorial-python-http/install-dependencies.md

@@ -4,17 +4,14 @@ Now that you've generated the module, let's navigate to its directory and instal
 
 ```text
 cd ../../connectors/source-<name>
-python -m venv .venv # Create a virtual environment in the .venv directory
-source .venv/bin/activate # enable the venv
-pip install -r requirements.txt
+poetry install
 ```
 
-This step sets up the initial python environment. **All** subsequent `python` or `pip` commands assume you have activated your virtual environment.
 
 Let's verify everything is working as intended. Run:
 
 ```text
-python main.py spec
+poetry run source-<name> spec
 ```
 
 You should see some output:
@@ -49,10 +46,10 @@ You'll notice in your source's directory that there is a python file called `mai
 
 ```text
 # from airbyte-integrations/connectors/source-<name>
-python main.py spec
-python main.py check --config secrets/config.json
-python main.py discover --config secrets/config.json
-python main.py read --config secrets/config.json --catalog sample_files/configured_catalog.json
+poetry run source-<name> spec
+poetry run source-<name> check --config secrets/config.json
+poetry run source-<name> discover --config secrets/config.json
+poetry run source-<name> read --config secrets/config.json --catalog sample_files/configured_catalog.json
 ```
 
 The nice thing about this approach is that you can iterate completely within python. The downside is that you are not quite running your source as it will actually be run by Airbyte. Specifically, you're not running it from within the docker container that will house it.

docs/connector-development/tutorials/cdk-tutorial-python-http/read-data.md

@@ -108,7 +108,7 @@ We're now ready to query the API!
 To do this, we'll need a [ConfiguredCatalog](../../../understanding-airbyte/beginners-guide-to-catalog.md). We've prepared one [here](https://github.com/airbytehq/airbyte/blob/master/docs/connector-development/tutorials/cdk-tutorial-python-http/configured_catalog.json) -- download this and place it in `sample_files/configured_catalog.json`. Then run:
 
 ```text
- python main.py read --config secrets/config.json --catalog sample_files/configured_catalog.json
+ poetry run source-<name> --config secrets/config.json --catalog sample_files/configured_catalog.json
 ```
 
 you should see some output lines, one of which is a record from the API:
@@ -240,17 +240,17 @@ We should now have a working implementation of incremental sync!
 Let's try it out:
 
 ```text
-python main.py read --config secrets/config.json --catalog sample_files/configured_catalog.json
+poetry run source-<name> --config secrets/config.json --catalog sample_files/configured_catalog.json
 ```
 
 You should see a bunch of `RECORD` messages and `STATE` messages. To verify that incremental sync is working, pass the input state back to the connector and run it again:
 
 ```text
 # Save the latest state to sample_files/state.json
-python main.py read --config secrets/config.json --catalog sample_files/configured_catalog.json | grep STATE | tail -n 1 | jq .state.data > sample_files/state.json
+poetry run source-<name> --config secrets/config.json --catalog sample_files/configured_catalog.json | grep STATE | tail -n 1 | jq .state.data > sample_files/state.json
 
 # Run a read operation with the latest state message
-python main.py read --config secrets/config.json --catalog sample_files/configured_catalog.json --state sample_files/state.json
+poetry run source-<name> --config secrets/config.json --catalog sample_files/configured_catalog.json --state sample_files/state.json
 ```
 
 You should see that only the record from the last date is being synced! This is acceptable behavior, since Airbyte requires at-least-once delivery of records, so repeating the last record twice is OK.

docs/connector-development/tutorials/cdk-tutorial-python-http/test-your-connector.md

@@ -2,15 +2,15 @@
 
 ## Unit Tests
 
-Add any relevant unit tests to the `unit_tests` directory. Unit tests should **not** depend on any secrets.
+Add any relevant unit tests to the `tests/unit_tests` directory. Unit tests should **not** depend on any secrets.
 
-You can run the tests using `python -m pytest -s unit_tests`.
+You can run the tests using `poetry run pytest tests/unit_tests`.
 
 ## Integration Tests
 
 Place any integration tests in the `integration_tests` directory such that they can be [discovered by pytest](https://docs.pytest.org/en/6.2.x/goodpractices.html#conventions-for-python-test-discovery).
 
-Run integration tests using `python -m pytest -s integration_tests`.
+You can run the tests using `poetry run pytest tests/integration_tests`.
 
 More information on integration testing can be found on [the Testing Connectors doc](https://docs.airbyte.com/connector-development/testing-connectors/#running-integration-tests).