Skip to content

airbyte-ci: Switch to prod pypi #34606

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Jan 30, 2024
Merged

airbyte-ci: Switch to prod pypi #34606

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
ec99e79
switch to prod pypi
Jan 29, 2024
278fe01
use default instead of explicitly setting in every instance
Jan 29, 2024
e6c6e04
add comment
Jan 29, 2024
d172818
introduce options for python registry and url
Jan 29, 2024
e79e421
check and fail with descriptive error message
Jan 29, 2024
ef5f5a7
test with pokeapi
Jan 29, 2024
55ac27b
fix publish action
Jan 29, 2024
58e4005
Merge remote-tracking branch 'origin/master' into flash1293/pypi-prod
Jan 29, 2024
b954efb
dummy change
Jan 29, 2024
0e70342
format
Jan 29, 2024
97a9702
revert pokeapi changes
Jan 29, 2024
d3867ce
fix command
Jan 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/actions/run-airbyte-ci/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,10 @@ inputs:
python_registry_token:
description: "Python registry API token to publish python package"
required: false
python_registry_url:
description: "Python registry URL to publish python package"
default: "https://upload.pypi.org/legacy/"
required: false

runs:
using: "composite"
Expand Down Expand Up @@ -135,6 +139,7 @@ runs:
PRODUCTION: ${{ inputs.production }}
PULL_REQUEST_NUMBER: ${{ github.event.pull_request.number }}
PYTHON_REGISTRY_TOKEN: ${{ inputs.python_registry_token }}
PYTHON_REGISTRY_URL: ${{ inputs.python_registry_url }}
S3_BUILD_CACHE_ACCESS_KEY_ID: ${{ inputs.s3_build_cache_access_key_id }}
S3_BUILD_CACHE_SECRET_KEY: ${{ inputs.s3_build_cache_secret_key }}
SENTRY_DSN: ${{ inputs.sentry_dsn }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/publish-airbyte-lib-command-manually.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,5 +53,5 @@ jobs:
s3_build_cache_access_key_id: ${{ secrets.SELF_RUNNER_AWS_ACCESS_KEY_ID }}
s3_build_cache_secret_key: ${{ secrets.SELF_RUNNER_AWS_SECRET_ACCESS_KEY }}
tailscale_auth_key: ${{ secrets.TAILSCALE_AUTH_KEY }}
subcommand: 'poetry --package-path=airbyte-lib publish --registry-url="https://test.pypi.org/legacy/"'
subcommand: "poetry --package-path=airbyte-lib publish"
python_registry_token: ${{ secrets.PYPI_TOKEN }}
47 changes: 30 additions & 17 deletions airbyte-ci/connectors/pipelines/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -385,16 +385,18 @@ Publish all connectors modified in the head commit: `airbyte-ci connectors --mod

### Options

| Option | Required | Default | Mapped environment variable | Description |
| ------------------------------------ | -------- | --------------- | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--pre-release/--main-release` | False | `--pre-release` | | Whether to publish the pre-release or the main release version of a connector. Defaults to pre-release. For main release you have to set the credentials to interact with the GCS bucket. |
| `--spec-cache-gcs-credentials` | False | | `SPEC_CACHE_GCS_CREDENTIALS` | The service account key to upload files to the GCS bucket hosting spec cache. |
| `--spec-cache-bucket-name` | False | | `SPEC_CACHE_BUCKET_NAME` | The name of the GCS bucket where specs will be cached. |
| `--metadata-service-gcs-credentials` | False | | `METADATA_SERVICE_GCS_CREDENTIALS` | The service account key to upload files to the GCS bucket hosting the metadata files. |
| `--metadata-service-bucket-name` | False | | `METADATA_SERVICE_BUCKET_NAME` | The name of the GCS bucket where metadata files will be uploaded. |
| `--slack-webhook` | False | | `SLACK_WEBHOOK` | The Slack webhook URL to send notifications to. |
| `--slack-channel` | False | | `SLACK_CHANNEL` | The Slack channel name to send notifications to. |
| `--ci-requirements` | False | | | Output the CI requirements as a JSON payload. It is used to determine the CI runner to use. |
| Option | Required | Default | Mapped environment variable | Description |
| ------------------------------------ | -------- | ------------------------------- | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--pre-release/--main-release` | False | `--pre-release` | | Whether to publish the pre-release or the main release version of a connector. Defaults to pre-release. For main release you have to set the credentials to interact with the GCS bucket. |
| `--spec-cache-gcs-credentials` | False | | `SPEC_CACHE_GCS_CREDENTIALS` | The service account key to upload files to the GCS bucket hosting spec cache. |
| `--spec-cache-bucket-name` | False | | `SPEC_CACHE_BUCKET_NAME` | The name of the GCS bucket where specs will be cached. |
| `--metadata-service-gcs-credentials` | False | | `METADATA_SERVICE_GCS_CREDENTIALS` | The service account key to upload files to the GCS bucket hosting the metadata files. |
| `--metadata-service-bucket-name` | False | | `METADATA_SERVICE_BUCKET_NAME` | The name of the GCS bucket where metadata files will be uploaded. |
| `--slack-webhook` | False | | `SLACK_WEBHOOK` | The Slack webhook URL to send notifications to. |
| `--slack-channel` | False | | `SLACK_CHANNEL` | The Slack channel name to send notifications to. |
| `--ci-requirements` | False | | | Output the CI requirements as a JSON payload. It is used to determine the CI runner to use. |
| `--python-registry-token` | False | | `PYTHON_REGISTRY_TOKEN` | The API token to authenticate with the registry. For pypi, the `pypi-` prefix needs to be specified |
| `--python-registry-url` | False | https://upload.pypi.org/legacy/ | `PYTHON_REGISTRY_URL` | The python registry to publish to. Defaults to main pypi |


I've added an empty "Default" column, and you can fill in the default values as needed.
Expand All @@ -406,14 +408,24 @@ flowchart TD
validate[Validate the metadata file]
check[Check if the connector image already exists]
build[Build the connector image for all platform variants]
publish_to_python_registry[Push the connector image to the python registry if enabled]
upload_spec[Upload connector spec to the spec cache bucket]
push[Push the connector image from DockerHub, with platform variants]
pull[Pull the connector image from DockerHub to check SPEC can be run and the image layers are healthy]
upload_metadata[Upload its metadata file to the metadata service bucket]

validate-->check-->build-->upload_spec-->push-->pull-->upload_metadata
validate-->check-->build-->upload_spec-->publish_to_python_registry-->push-->pull-->upload_metadata
```

#### Python registry publishing

If `remoteRegistries.pypi.enabled` in the connector metadata is set to `true`, the connector will be published to the python registry.
To do so, the `--python-registry-token` and `--python-registry-url` options are used to authenticate with the registry and publish the connector.
If the current version of the connector is already published to the registry, the publish will be skipped.

On a pre-release, the connector will be published as a `.dev<N>` version.


### <a id="connectors-bump_version"></a>`connectors bump_version` command

Bump the version of the selected connectors.
Expand Down Expand Up @@ -534,12 +546,12 @@ For poetry packages, the package name and version can be taken from the `pyproje

#### Options

| Option | Required | Default | Mapped environment variable | Description |
| ------------------------- | -------- | ----------------------- | --------------------------- | -------------------------------------------------------------------------------------------------------- |
| `--publish-name` | False | | | The name of the package. Not required for poetry packages that define it in the `pyproject.toml` file |
| `--publish-version` | False | | | The version of the package. Not required for poetry packages that define it in the `pyproject.toml` file |
| `--python-registry-token` | True | | PYTHON_REGISTRY_TOKEN | The API token to authenticate with the registry. For pypi, the `pypi-` prefix needs to be specified |
| `--registry-url` | False | https://pypi.org/simple | | The python registry to publish to. Defaults to main pypi |
| Option | Required | Default | Mapped environment variable | Description |
| ------------------------- | -------- | ------------------------------- | --------------------------- | -------------------------------------------------------------------------------------------------------- |
| `--publish-name` | False | | | The name of the package. Not required for poetry packages that define it in the `pyproject.toml` file |
| `--publish-version` | False | | | The version of the package. Not required for poetry packages that define it in the `pyproject.toml` file |
| `--python-registry-token` | True | | PYTHON_REGISTRY_TOKEN | The API token to authenticate with the registry. For pypi, the `pypi-` prefix needs to be specified |
| `--python-registry-url` | False | https://upload.pypi.org/legacy/ | PYTHON_REGISTRY_URL | The python registry to publish to. Defaults to main pypi |

### <a id="metadata-validate-command-subgroup"></a>`metadata` command subgroup

Expand Down Expand Up @@ -597,6 +609,7 @@ E.G.: running `pytest` on a specific test folder:

| Version | PR | Description |
| ------- | ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| 3.9.0 | [#34316](https://github.com/airbytehq/airbyte/pull/34316) | Allow configuration of python registry URL via environment variable. |
| 3.8.0 | [#34316](https://github.com/airbytehq/airbyte/pull/34316) | Expose Dagger engine image name in `--ci-requirements` and add `--ci-requirements` to the `airbyte-ci` root command group. |
| 3.7.3 | [#34560](https://github.com/airbytehq/airbyte/pull/34560) | Simplify Gradle task execution framework by removing local maven repo support. |
| 3.7.2 | [#34555](https://github.com/airbytehq/airbyte/pull/34555) | Override secret masking in some very specific special cases. |
Expand Down
20 changes: 19 additions & 1 deletion airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/publish/commands.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
from pipelines.cli.click_decorators import click_ci_requirements_option
from pipelines.cli.confirm_prompt import confirm
from pipelines.cli.dagger_pipeline_command import DaggerPipelineCommand
from pipelines.consts import ContextState
from pipelines.consts import DEFAULT_PYTHON_PACKAGE_REGISTRY_URL, ContextState
from pipelines.helpers.utils import fail_if_missing_docker_hub_creds


Expand Down Expand Up @@ -59,6 +59,20 @@
envvar="SLACK_CHANNEL",
default="#connector-publish-updates",
)
@click.option(
"--python-registry-token",
help="Access token for python registry",
type=click.STRING,
required=True,
envvar="PYTHON_REGISTRY_TOKEN",
)
@click.option(
"--python-registry-url",
help="Which python registry registry to publish to. If not set, the default pypi is used. For test pypi, use https://test.pypi.org/legacy/",
type=click.STRING,
default=DEFAULT_PYTHON_PACKAGE_REGISTRY_URL,
envvar="PYTHON_REGISTRY_URL",
)
@click.pass_context
async def publish(
ctx: click.Context,
Expand All @@ -69,6 +83,8 @@ async def publish(
metadata_service_gcs_credentials: str,
slack_webhook: str,
slack_channel: str,
python_registry_token: str,
python_registry_url: str,
) -> bool:
ctx.obj["spec_cache_gcs_credentials"] = spec_cache_gcs_credentials
ctx.obj["spec_cache_bucket_name"] = spec_cache_bucket_name
Expand Down Expand Up @@ -109,6 +125,8 @@ async def publish(
s3_build_cache_access_key_id=ctx.obj.get("s3_build_cache_access_key_id"),
s3_build_cache_secret_key=ctx.obj.get("s3_build_cache_secret_key"),
use_local_cdk=ctx.obj.get("use_local_cdk"),
python_registry_token=python_registry_token,
python_registry_url=python_registry_url,
)
for connector in ctx.obj["selected_connectors_with_modified_files"]
]
Expand Down
4 changes: 4 additions & 0 deletions airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/publish/context.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,12 +46,16 @@ def __init__(
s3_build_cache_access_key_id: Optional[str] = None,
s3_build_cache_secret_key: Optional[str] = None,
use_local_cdk: bool = False,
python_registry_token: Optional[str] = None,
python_registry_url: Optional[str] = None,
Comment on lines +49 to +50
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a validation on init that these keyword arguments are not None when self.connector.language in [ConnectorLanguage.Python, ConnectorLanguage.LowCode]?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added a check to the step - if the python publish logic is hit, it will fail with a descriptive error in case the registry or token isn't set properly.

) -> None:
self.pre_release = pre_release
self.spec_cache_bucket_name = spec_cache_bucket_name
self.metadata_bucket_name = metadata_bucket_name
self.spec_cache_gcs_credentials = sanitize_gcs_credentials(spec_cache_gcs_credentials)
self.metadata_service_gcs_credentials = sanitize_gcs_credentials(metadata_service_gcs_credentials)
self.python_registry_token = python_registry_token
self.python_registry_url = python_registry_url
pipeline_name = f"Publish {connector.technical_name}"
pipeline_name = pipeline_name + " (pre-release)" if pre_release else pipeline_name

Expand Down
7 changes: 4 additions & 3 deletions airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/poetry/publish/commands.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,10 +47,11 @@ def _validate_python_version(_ctx: dict, _param: dict, value: Optional[str]) ->
envvar="PYTHON_REGISTRY_TOKEN",
)
@click.option(
"--registry-url",
"--python-registry-url",
help="Which registry to publish to. If not set, the default pypi is used. For test pypi, use https://test.pypi.org/legacy/",
type=click.STRING,
default=DEFAULT_PYTHON_PACKAGE_REGISTRY_URL,
envvar="PYTHON_REGISTRY_URL",
)
@click.option(
"--publish-name",
Expand All @@ -69,7 +70,7 @@ async def publish(
ctx: click.Context,
click_pipeline_context: ClickPipelineContext,
python_registry_token: str,
registry_url: str,
python_registry_url: str,
publish_name: Optional[str],
publish_version: Optional[str],
) -> bool:
Expand All @@ -85,7 +86,7 @@ async def publish(
ci_context=ctx.obj.get("ci_context"),
ci_gcs_credentials=ctx.obj["ci_gcs_credentials"],
python_registry_token=python_registry_token,
registry=registry_url,
registry=python_registry_url,
package_path=ctx.obj["package_path"],
package_name=publish_name,
version=publish_version,
Expand Down
8 changes: 6 additions & 2 deletions airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/steps/python_registry.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,9 +116,11 @@ async def _poetry_publish(self, package_dir_to_publish: Directory) -> StepResult
.with_directory("package", package_dir_to_publish)
.with_workdir("package")
.with_new_file(PYPROJECT_TOML_FILE_PATH, contents=tomli_w.dumps(contents))
# Make sure these steps are always executed and not cached as they are triggering a side-effect (calling the registry)
# Env var setting needs to be in this block as well to make sure a change of the env var will be propagated correctly
.with_env_variable("CACHEBUSTER", str(uuid.uuid4()))
.with_exec(["poetry", "config", "repositories.mypypi", self.context.registry])
.with_exec(sh_dash_c(["poetry config pypi-token.mypypi $PYTHON_REGISTRY_TOKEN"]))
.with_env_variable("CACHEBUSTER", str(uuid.uuid4()))
.with_exec(sh_dash_c(["poetry publish --build --repository mypypi -vvv --no-interaction"]))
)

Expand Down Expand Up @@ -156,9 +158,11 @@ async def _pip_publish(self, package_dir_to_publish: Directory) -> StepResult:
.with_new_file("setup.cfg", contents=setup_cfg)
.with_exec(["pip", "install", "--upgrade", "setuptools", "wheel"])
.with_exec(["python", SETUP_PY_FILE_PATH, "sdist", "bdist_wheel"])
# Make sure these steps are always executed and not cached as they are triggering a side-effect (calling the registry)
# Env var setting needs to be in this block as well to make sure a change of the env var will be propagated correctly
.with_env_variable("CACHEBUSTER", str(uuid.uuid4()))
.with_secret_variable("TWINE_USERNAME", pypi_username)
.with_secret_variable("TWINE_PASSWORD", pypi_password)
.with_env_variable("CACHEBUSTER", str(uuid.uuid4()))
.with_exec(["twine", "upload", "--verbose", "--repository-url", self.context.registry, "dist/*"])
)

Expand Down
2 changes: 1 addition & 1 deletion airbyte-ci/connectors/pipelines/pipelines/consts.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@
POETRY_CACHE_PATH = "/root/.cache/pypoetry"
STORAGE_DRIVER = "fuse-overlayfs"
SETUP_PY_FILE_PATH = "setup.py"
DEFAULT_PYTHON_PACKAGE_REGISTRY_URL = "https://pypi.org/simple"
DEFAULT_PYTHON_PACKAGE_REGISTRY_URL = "https://upload.pypi.org/legacy/"


class CIContext(str, Enum):
Expand Down
4 changes: 2 additions & 2 deletions airbyte-ci/connectors/pipelines/pipelines/models/contexts/python_registry_publish.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,8 +86,8 @@ async def from_publish_connector_context(
version = f"{version}.dev{release_candidate_tag}"

pypi_context = cls(
python_registry_token=os.environ["PYTHON_REGISTRY_TOKEN"],
registry="https://test.pypi.org/legacy/", # TODO: go live
python_registry_token=connector_context.python_registry_token,
registry=connector_context.python_registry_url,
package_path=str(connector_context.connector.code_directory),
package_name=current_metadata["remoteRegistries"]["pypi"]["packageName"],
version=version,
Expand Down
Loading