update doc to reference poetry (#35414)

Author: alafanechere

docs/connector-development/config-based/tutorial/0-getting-started.md

@@ -42,6 +42,7 @@ This can be done by signing up for the Free tier plan on [Exchange Rates Data AP
 
 - An Exchange Rates API key
 - Python >= 3.9
+- [Poetry](https://python-poetry.org/)
 - Docker must be running
 - NodeJS
 - [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#L1) CLI

docs/connector-development/config-based/tutorial/2-install-dependencies.md

@@ -1,25 +1,17 @@
 # Step 2: Install dependencies
 
-Let's create a python virtual environment for our source.
-You can do this by executing the following commands from the root of the Airbyte repository.
-
-The command below assume that `python` points to a version of python >=3.9.0. On some systems, `python` points to a Python2 installation and `python3` points to Python3.
-If this is the case on your machine, substitute the `python` commands with `python3`.
-The subsequent `python` invocations will use the virtual environment created for the connector.
 
 ```bash
 cd ../../connectors/source-exchange-rates-tutorial
-python -m venv .venv
-source .venv/bin/activate
-pip install -r requirements.txt
+poetry install
 ```
 
 These steps create an initial python environment, and install the dependencies required to run an API Source connector.
 
 Let's verify everything works as expected by running the Airbyte `spec` operation:
 
 ```bash
-python main.py spec
+poetry run source-exchange-rates-tutorial spec
 ```
 
 You should see an output similar to the one below:

docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md

@@ -200,7 +200,7 @@ spec:
 We can now run the `check` operation, which verifies the connector can connect to the API source.
 
 ```bash
-python main.py check --config secrets/config.json
+poetry run source-exchange-rates-tutorial check --config secrets/config.json
 ```
 
 which should now succeed with logs similar to:

docs/connector-development/config-based/tutorial/4-reading-data.md

@@ -44,7 +44,7 @@ As an alternative to storing the stream's data schema to the `schemas/` director
 Reading from the source can be done by running the `read` operation
 
 ```bash
-python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
+poetry run source-exchange-rates-tutorial read --config secrets/config.json --catalog integration_tests/configured_catalog.json
 ```
 
 The logs should show that 1 record was read from the stream.
@@ -57,7 +57,7 @@ The logs should show that 1 record was read from the stream.
 The `--debug` flag can be set to print out debug information, including the outgoing request and its associated response
 
 ```bash
-python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json --debug
+poetry run source-exchange-rates-tutorial read --config secrets/config.json --catalog integration_tests/configured_catalog.json --debug
 ```
 
 ## Next steps

docs/connector-development/config-based/tutorial/5-incremental-reads.md

@@ -76,7 +76,7 @@ definitions:
 You can test these changes by executing the `read` operation:
 
 ```bash
-python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
+poetry run source-exchange-rates-tutorial read --config secrets/config.json --catalog integration_tests/configured_catalog.json
 ```
 
 By reading the output record, you should see that we read historical data instead of the latest exchange rate.
@@ -240,7 +240,7 @@ spec:
 Running the `read` operation will now read all data for all days between start_date and now:
 
 ```bash
-python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
+poetry run source-exchange-rates-tutorial read --config secrets/config.json --catalog integration_tests/configured_catalog.json
 ```
 
 The operation should now output more than one record:
@@ -295,7 +295,7 @@ We can simulate incremental syncs by creating a state file containing the last s
 Running the `read` operation will now only read data for dates later than the given state:
 
 ```bash
-python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json --state integration_tests/sample_state.json
+poetry run source-exchange-rates-tutorial read --config secrets/config.json --catalog integration_tests/configured_catalog.json --state integration_tests/sample_state.json
 ```
 
 There shouldn't be any data read if the state is today's date:

docs/connector-development/config-based/tutorial/6-testing.md

@@ -30,7 +30,7 @@ and `integration_tests/abnormal_state.json` with
 You can run the [acceptance tests](https://github.com/airbytehq/airbyte/blob/master/docs/connector-development/testing-connectors/connector-acceptance-tests-reference.md#L1) with the following commands using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#L1):
 
 ```bash
-airbyte-ci connectors --use-remote-secrets=false --name source-exchange-rates-tutorial test
+airbyte-ci connectors --use-remote-secrets=false --name source-exchange-rates-tutorial test --only-step=acceptance
 ```
 
 ## Next steps:

docs/connector-development/testing-connectors/README.md

@@ -1,34 +1,32 @@
 # Testing Connectors
 
-## Our testing pyramid
-Multiple tests suites compose the Airbyte connector testing pyramid:
-Connector specific tests declared in the connector code directory:
-* Unit tests
-* Integration tests
-
-Tests common to all connectors:
-* [QA checks](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/connector_ops/connector_ops/qa_checks.py)
-* [Connector Acceptance tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference/)
+Multiple tests suites compose the Airbyte connector testing pyramid
 
-## Running tests
-Unit and integration tests can be run directly from the connector code.
+## Common to all connectors
+* [Connectors QA checks](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/connector_ops/connector_ops/qa_checks.py)
+* [Connector Acceptance tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference/)
 
-Using `pytest` for Python connectors:
+## Connector specific tests
+### 🐍 Python connectors
+We use `pytest` to run unit and integration tests:
 ```bash
-python -m pytest unit_tests/
-python -m pytest integration_tests/
+# From connector directory
+poetry run pytest
 ```
 
-Using `gradle` for Java connectors:
+### ☕ Java connectors
+We run Java connector tests with gradle.
 
 ```bash
+# Unit tests
 ./gradlew :airbyte-integrations:connectors:source-postgres:test
+# Integration tests
 ./gradlew :airbyte-integrations:connectors:source-postgres:integrationTestJava
 ```
 
 Please note that according to the test implementation you might have to provide connector configurations as a `config.json` file in a `.secrets` folder in the connector code directory.
 
-
+## 🤖 CI
 If you want to run the global test suite, exactly like what is run in CI, you should install [`airbyte-ci` CLI](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md) and use the following command:
 
 ```bash
@@ -39,6 +37,5 @@ This will run all the tests for the connector, including the QA checks and the C
 Connector Acceptance tests require connector configuration to be provided as a `config.json` file in a `.secrets` folder in the connector code directory.
 
 
-## Tests on pull requests
 Our CI infrastructure runs the connector tests with [`airbyte-ci` CLI](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md). Connectors tests are automatically and remotely triggered on your branch according to the changes made in your branch.
 **Passing tests are required to merge a connector pull request.**

docs/connector-development/tutorials/building-a-python-source.md

@@ -64,18 +64,16 @@ $ ./generate.sh
 
 Select the `python` template and then input the name of your connector. For this walk through we will refer to our source as `example-python`
 
-### Step 2: Build the newly generated source
+### Step 2: Install the newly generated source
 
-Build the source by running:
+Install the source by running:
 
 ```bash
 cd airbyte-integrations/connectors/source-<name>
-python -m venv .venv # Create a virtual environment in the .venv directory
-source .venv/bin/activate # enable the venv
-pip install -r requirements.txt
+poetry install
 ```
 
-This step sets up the initial python environment. **All** subsequent `python` or `pip` commands assume you have activated your virtual environment.
+This step sets up the initial python environment. 
 
 ### Step 3: Set up your Airbyte development environment
 
@@ -112,10 +110,10 @@ You'll notice in your source's directory that there is a python file called `mai
 
 ```bash
 # from airbyte-integrations/connectors/source-<source-name>
-python main.py spec
-python main.py check --config secrets/config.json
-python main.py discover --config secrets/config.json
-python main.py read --config secrets/config.json --catalog sample_files/configured_catalog.json
+poetry run source-<source-name> spec
+poetry run source-<source-name> check --config secrets/config.json
+poetry run source-<source-name> discover --config secrets/config.json
+poetry run source-<source-name> read --config secrets/config.json --catalog sample_files/configured_catalog.json
 ```
 
 The nice thing about this approach is that you can iterate completely within in python. The downside is that you are not quite running your source as it will actually be run by Airbyte. Specifically you're not running it from within the docker container that will house it.
@@ -182,7 +180,7 @@ The nice thing about this approach is that you are running your source exactly a
 During development of your connector, you can enable the printing of detailed debug information during a sync by specifying the `--debug` flag. This will allow you to get a better picture of what is happening during each step of your sync.
 
 ```bash
-python main.py read --config secrets/config.json --catalog sample_files/configured_catalog.json --debug
+poetry run source-<source-name> read --config secrets/config.json --catalog sample_files/configured_catalog.json --debug
 ```
 
 In addition to the preset CDK debug statements, you can also emit custom debug information from your connector by introducing your own debug statements:
@@ -233,7 +231,8 @@ As described in the template code, this method takes in the same config object a
 
 The Connector Acceptance Tests are a set of tests that run against all sources. These tests are run in the Airbyte CI to prevent regressions. They also can help you sanity check that your source works as expected. The following [article](../testing-connectors/connector-acceptance-tests-reference.md) explains Connector Acceptance Tests and how to run them.
 
-You can run the tests using `./gradlew :airbyte-integrations:connectors:source-<source-name>:integrationTest`. Make sure to run this command from the Airbyte repository root.
+You can run the tests using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
+`airbyte-ci connectors --name source-<source-name> test --only-step=acceptance`
 
 :::info
 In some rare cases we make exceptions and allow a source to not need to pass all the standard tests. If for some reason you think your source cannot reasonably pass one of the tests cases, reach out to us on github or slack, and we can determine whether there's a change we can make so that the test will pass or if we should skip that test for your source.
@@ -245,15 +244,15 @@ The connector acceptance tests are meant to cover the basic functionality of a s
 
 #### Unit Tests
 
-Add any relevant unit tests to the `unit_tests` directory. Unit tests should _not_ depend on any secrets.
+Add any relevant unit tests to the `tests/unit_tests` directory. Unit tests should _not_ depend on any secrets.
 
-You can run the tests using `python -m pytest -s unit_tests`
+You can run the tests using `poetry run pytest tests/unit_tests`
 
 #### Integration Tests
 
 Place any integration tests in the `integration_tests` directory such that they can be [discovered by pytest](https://docs.pytest.org/en/6.2.x/goodpractices.html#conventions-for-python-test-discovery).
 
-Run integration tests using `python -m pytest -s integration_tests`.
+You can run the tests using `poetry run pytest tests/integration_tests`
 
 ### Step 10: Update the `README.md`
 

docs/connector-development/tutorials/cdk-speedrun.md

@@ -11,6 +11,7 @@ If you are a visual learner and want to see a video version of this guide going
 ## Dependencies
 
 1. Python >= 3.9
+2. [Poetry](https://python-poetry.org/)
 2. Docker
 3. NodeJS
 
@@ -30,9 +31,7 @@ Select the `Python HTTP API Source` and name it `python-http-example`.
 
 ```bash
 cd ../../connectors/source-python-http-example
-python -m venv .venv # Create a virtual environment in the .venv directory
-source .venv/bin/activate
-pip install -r requirements.txt
+poetry install
 ```
 
 ### Define Connector Inputs
@@ -169,7 +168,7 @@ This file defines your output schema for every endpoint that you want to impleme
 Test your discover function. You should receive a fairly large JSON object in return.
 
 ```bash
-python main.py discover --config sample_files/config.json
+poetry run source-python-http-example discover --config sample_files/config.json
 ```
 
 Note that our discover function is using the `pokemon_name` config variable passed in from the `Pokemon` stream when we set it in the `__init__` function.
@@ -226,7 +225,7 @@ We now need a catalog that defines all of our streams. We only have one stream:
 Let's read some data.
 
 ```bash
-python main.py read --config sample_files/config.json --catalog sample_files/configured_catalog.json
+poetry run source-python-http-example read --config sample_files/config.json --catalog sample_files/configured_catalog.json
 ```
 
 If all goes well, containerize it so you can use it in the UI:

docs/connector-development/tutorials/cdk-tutorial-python-http/connection-checking.md

@@ -42,21 +42,21 @@ Note: in a real implementation you should write code to connect to the API to va
 
 Let's test out this implementation by creating two objects: a valid and an invalid config and attempt to give them as input to the connector. For this section, you will need to take the API access key generated earlier and add it to both configs. Because these configs contain secrets, we recommend storing configs which contain secrets in `secrets/config.json` because the `secrets` directory is gitignored by default.
 
-```text
+```bash
 mkdir sample_files
 echo '{"start_date": "2022-04-01", "base": "USD", "apikey": <your_apikey>}'  > secrets/config.json
 echo '{"start_date": "2022-04-01", "base": "BTC", "apikey": <your_apikey>}'  > secrets/invalid_config.json
-python main.py check --config secrets/config.json
-python main.py check --config secrets/invalid_config.json
+poetry run source-python-http-example check --config secrets/config.json
+poetry run source-python-http-example check --config secrets/invalid_config.json
 ```
 
 You should see output like the following:
 
-```text
-> python main.py check --config secrets/config.json
+```bash
+> poetry run source-python-http-example check --config secrets/config.json
 {"type": "CONNECTION_STATUS", "connectionStatus": {"status": "SUCCEEDED"}}
 
-> python main.py check --config secrets/invalid_config.json
+> poetry run source-python-http-example check --config secrets/invalid_config.json
 {"type": "CONNECTION_STATUS", "connectionStatus": {"status": "FAILED", "message": "Input currency BTC is invalid. Please input one of the following currencies: {'DKK', 'USD', 'CZK', 'BGN', 'JPY'}"}}
 ```
 

docs/connector-development/tutorials/cdk-tutorial-python-http/declare-schema.md

@@ -63,7 +63,7 @@ Having created this stream in code, we'll put a file `exchange_rates.json` in th
 With `.json` schema file in place, let's see if the connector can now find this schema and produce a valid catalog:
 
 ```text
-python main.py discover --config secrets/config.json # this is not a mistake, the schema file is found by naming snake_case naming convention as specified above
+poetry run source-python-http-example discover --config secrets/config.json # this is not a mistake, the schema file is found by naming snake_case naming convention as specified above
 ```
 
 you should see some output like:

docs/connector-development/tutorials/cdk-tutorial-python-http/getting-started.md

@@ -7,8 +7,8 @@ This is a step-by-step guide for how to create an Airbyte source in Python to re
 ## Requirements
 
 * Python >= 3.9
+* [Poetry](https://python-poetry.org/)
 * Docker
-* NodeJS \(only used to generate the connector\). We'll remove the NodeJS dependency soon.
 
 All the commands below assume that `python` points to a version of python >=3.9.0. On some systems, `python` points to a Python2 installation and `python3` points to Python3. If this is the case on your machine, substitute all `python` commands in this guide with `python3`.
 

docs/connector-development/tutorials/cdk-tutorial-python-http/install-dependencies.md

@@ -2,19 +2,16 @@
 
 Now that you've generated the module, let's navigate to its directory and install dependencies:
 
-```text
+```bash
 cd ../../connectors/source-<name>
-python -m venv .venv # Create a virtual environment in the .venv directory
-source .venv/bin/activate # enable the venv
-pip install -r requirements.txt
+poetry install
 ```
 
-This step sets up the initial python environment. **All** subsequent `python` or `pip` commands assume you have activated your virtual environment.
 
 Let's verify everything is working as intended. Run:
 
-```text
-python main.py spec
+```bash
+poetry run source-<name> spec
 ```
 
 You should see some output:
@@ -25,7 +22,6 @@ You should see some output:
 
 We just ran Airbyte Protocol's `spec` command! We'll talk more about this later, but this is a simple sanity check to make sure everything is wired up correctly.
 
-Note that the `main.py` file is a simple script that makes it easy to run your connector. Its invocation format is `python main.py <command> [args]`. See the module's generated `README.md` for the commands it supports.
 
 ## Notes on iteration cycle
 
@@ -47,12 +43,12 @@ There are two ways we recommend iterating on a source. Consider using whichever
 
 You'll notice in your source's directory that there is a python file called `main.py`. This file exists as convenience for development. You run it to test that your source works:
 
-```text
+```bash
 # from airbyte-integrations/connectors/source-<name>
-python main.py spec
-python main.py check --config secrets/config.json
-python main.py discover --config secrets/config.json
-python main.py read --config secrets/config.json --catalog sample_files/configured_catalog.json
+poetry run source-<name> spec
+poetry run source-<name> check --config secrets/config.json
+poetry run source-<name> discover --config secrets/config.json
+poetry run source-<name> read --config secrets/config.json --catalog sample_files/configured_catalog.json
 ```
 
 The nice thing about this approach is that you can iterate completely within python. The downside is that you are not quite running your source as it will actually be run by Airbyte. Specifically, you're not running it from within the docker container that will house it.
@@ -61,7 +57,7 @@ The nice thing about this approach is that you can iterate completely within pyt
 
 If you want to run your source exactly as it will be run by Airbyte \(i.e. within a docker container\), you can use the following commands from the connector module directory \(`airbyte-integrations/connectors/source-python-http-example`\):
 
-```text
+```bash
 # First build the container
 docker build . -t airbyte/source-<name>:dev