Skip to content

Commit d1334be

Browse files
author
Joe Reuter
authored
airbyte-ci: Switch to prod pypi (#34606)
1 parent b37bde8 commit d1334be

File tree

12 files changed

+95
-40
lines changed

12 files changed

+95
-40
lines changed

‎.github/actions/run-airbyte-ci/action.yml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,10 @@ inputs:
7979
python_registry_token:
8080
description: "Python registry API token to publish python package"
8181
required: false
82+
python_registry_url:
83+
description: "Python registry URL to publish python package"
84+
default: "https://upload.pypi.org/legacy/"
85+
required: false
8286

8387
runs:
8488
using: "composite"
@@ -135,6 +139,7 @@ runs:
135139
PRODUCTION: ${{ inputs.production }}
136140
PULL_REQUEST_NUMBER: ${{ github.event.pull_request.number }}
137141
PYTHON_REGISTRY_TOKEN: ${{ inputs.python_registry_token }}
142+
PYTHON_REGISTRY_URL: ${{ inputs.python_registry_url }}
138143
S3_BUILD_CACHE_ACCESS_KEY_ID: ${{ inputs.s3_build_cache_access_key_id }}
139144
S3_BUILD_CACHE_SECRET_KEY: ${{ inputs.s3_build_cache_secret_key }}
140145
SENTRY_DSN: ${{ inputs.sentry_dsn }}

‎.github/workflows/publish-airbyte-lib-command-manually.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ jobs:
3737
uses: actions/checkout@v3
3838
- name: Publish
3939
id: publish-airbyte-lib
40-
uses: ./.github/actions/run-dagger-pipeline
40+
uses: ./.github/actions/run-airbyte-ci
4141
with:
4242
context: "manual"
4343
dagger_cloud_token: ${{ secrets.DAGGER_CLOUD_TOKEN }}
@@ -53,5 +53,5 @@ jobs:
5353
s3_build_cache_access_key_id: ${{ secrets.SELF_RUNNER_AWS_ACCESS_KEY_ID }}
5454
s3_build_cache_secret_key: ${{ secrets.SELF_RUNNER_AWS_SECRET_ACCESS_KEY }}
5555
tailscale_auth_key: ${{ secrets.TAILSCALE_AUTH_KEY }}
56-
subcommand: 'poetry --package-path=airbyte-lib publish --registry-url="https://test.pypi.org/legacy/"'
56+
subcommand: "poetry --package-path=airbyte-lib publish"
5757
python_registry_token: ${{ secrets.PYPI_TOKEN }}

‎airbyte-ci/connectors/pipelines/README.md

Lines changed: 30 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -385,16 +385,18 @@ Publish all connectors modified in the head commit: `airbyte-ci connectors --mod
385385

386386
### Options
387387

388-
| Option | Required | Default | Mapped environment variable | Description |
389-
| ------------------------------------ | -------- | --------------- | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
390-
| `--pre-release/--main-release` | False | `--pre-release` | | Whether to publish the pre-release or the main release version of a connector. Defaults to pre-release. For main release you have to set the credentials to interact with the GCS bucket. |
391-
| `--spec-cache-gcs-credentials` | False | | `SPEC_CACHE_GCS_CREDENTIALS` | The service account key to upload files to the GCS bucket hosting spec cache. |
392-
| `--spec-cache-bucket-name` | False | | `SPEC_CACHE_BUCKET_NAME` | The name of the GCS bucket where specs will be cached. |
393-
| `--metadata-service-gcs-credentials` | False | | `METADATA_SERVICE_GCS_CREDENTIALS` | The service account key to upload files to the GCS bucket hosting the metadata files. |
394-
| `--metadata-service-bucket-name` | False | | `METADATA_SERVICE_BUCKET_NAME` | The name of the GCS bucket where metadata files will be uploaded. |
395-
| `--slack-webhook` | False | | `SLACK_WEBHOOK` | The Slack webhook URL to send notifications to. |
396-
| `--slack-channel` | False | | `SLACK_CHANNEL` | The Slack channel name to send notifications to. |
397-
| `--ci-requirements` | False | | | Output the CI requirements as a JSON payload. It is used to determine the CI runner to use. |
388+
| Option | Required | Default | Mapped environment variable | Description |
389+
| ------------------------------------ | -------- | ------------------------------- | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
390+
| `--pre-release/--main-release` | False | `--pre-release` | | Whether to publish the pre-release or the main release version of a connector. Defaults to pre-release. For main release you have to set the credentials to interact with the GCS bucket. |
391+
| `--spec-cache-gcs-credentials` | False | | `SPEC_CACHE_GCS_CREDENTIALS` | The service account key to upload files to the GCS bucket hosting spec cache. |
392+
| `--spec-cache-bucket-name` | False | | `SPEC_CACHE_BUCKET_NAME` | The name of the GCS bucket where specs will be cached. |
393+
| `--metadata-service-gcs-credentials` | False | | `METADATA_SERVICE_GCS_CREDENTIALS` | The service account key to upload files to the GCS bucket hosting the metadata files. |
394+
| `--metadata-service-bucket-name` | False | | `METADATA_SERVICE_BUCKET_NAME` | The name of the GCS bucket where metadata files will be uploaded. |
395+
| `--slack-webhook` | False | | `SLACK_WEBHOOK` | The Slack webhook URL to send notifications to. |
396+
| `--slack-channel` | False | | `SLACK_CHANNEL` | The Slack channel name to send notifications to. |
397+
| `--ci-requirements` | False | | | Output the CI requirements as a JSON payload. It is used to determine the CI runner to use. |
398+
| `--python-registry-token` | False | | `PYTHON_REGISTRY_TOKEN` | The API token to authenticate with the registry. For pypi, the `pypi-` prefix needs to be specified |
399+
| `--python-registry-url` | False | https://upload.pypi.org/legacy/ | `PYTHON_REGISTRY_URL` | The python registry to publish to. Defaults to main pypi |
398400

399401

400402
I've added an empty "Default" column, and you can fill in the default values as needed.
@@ -406,14 +408,24 @@ flowchart TD
406408
validate[Validate the metadata file]
407409
check[Check if the connector image already exists]
408410
build[Build the connector image for all platform variants]
411+
publish_to_python_registry[Push the connector image to the python registry if enabled]
409412
upload_spec[Upload connector spec to the spec cache bucket]
410413
push[Push the connector image from DockerHub, with platform variants]
411414
pull[Pull the connector image from DockerHub to check SPEC can be run and the image layers are healthy]
412415
upload_metadata[Upload its metadata file to the metadata service bucket]
413416
414-
validate-->check-->build-->upload_spec-->push-->pull-->upload_metadata
417+
validate-->check-->build-->upload_spec-->publish_to_python_registry-->push-->pull-->upload_metadata
415418
```
416419

420+
#### Python registry publishing
421+
422+
If `remoteRegistries.pypi.enabled` in the connector metadata is set to `true`, the connector will be published to the python registry.
423+
To do so, the `--python-registry-token` and `--python-registry-url` options are used to authenticate with the registry and publish the connector.
424+
If the current version of the connector is already published to the registry, the publish will be skipped.
425+
426+
On a pre-release, the connector will be published as a `.dev<N>` version.
427+
428+
417429
### <a id="connectors-bump_version"></a>`connectors bump_version` command
418430

419431
Bump the version of the selected connectors.
@@ -534,12 +546,12 @@ For poetry packages, the package name and version can be taken from the `pyproje
534546

535547
#### Options
536548

537-
| Option | Required | Default | Mapped environment variable | Description |
538-
| ------------------------- | -------- | ----------------------- | --------------------------- | -------------------------------------------------------------------------------------------------------- |
539-
| `--publish-name` | False | | | The name of the package. Not required for poetry packages that define it in the `pyproject.toml` file |
540-
| `--publish-version` | False | | | The version of the package. Not required for poetry packages that define it in the `pyproject.toml` file |
541-
| `--python-registry-token` | True | | PYTHON_REGISTRY_TOKEN | The API token to authenticate with the registry. For pypi, the `pypi-` prefix needs to be specified |
542-
| `--registry-url` | False | https://pypi.org/simple | | The python registry to publish to. Defaults to main pypi |
549+
| Option | Required | Default | Mapped environment variable | Description |
550+
| ------------------------- | -------- | ------------------------------- | --------------------------- | -------------------------------------------------------------------------------------------------------- |
551+
| `--publish-name` | False | | | The name of the package. Not required for poetry packages that define it in the `pyproject.toml` file |
552+
| `--publish-version` | False | | | The version of the package. Not required for poetry packages that define it in the `pyproject.toml` file |
553+
| `--python-registry-token` | True | | PYTHON_REGISTRY_TOKEN | The API token to authenticate with the registry. For pypi, the `pypi-` prefix needs to be specified |
554+
| `--python-registry-url` | False | https://upload.pypi.org/legacy/ | PYTHON_REGISTRY_URL | The python registry to publish to. Defaults to main pypi |
543555

544556
### <a id="metadata-validate-command-subgroup"></a>`metadata` command subgroup
545557

@@ -597,6 +609,7 @@ E.G.: running `pytest` on a specific test folder:
597609

598610
| Version | PR | Description |
599611
| ------- | ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
612+
| 3.9.0 | [#34606](https://github.com/airbytehq/airbyte/pull/34606) | Allow configuration of python registry URL via environment variable. |
600613
| 3.8.1 | [#34607](https://github.com/airbytehq/airbyte/pull/34607) | Improve gradle dependency cache volume protection. |
601614
| 3.8.0 | [#34316](https://github.com/airbytehq/airbyte/pull/34316) | Expose Dagger engine image name in `--ci-requirements` and add `--ci-requirements` to the `airbyte-ci` root command group. |
602615
| 3.7.3 | [#34560](https://github.com/airbytehq/airbyte/pull/34560) | Simplify Gradle task execution framework by removing local maven repo support. |

‎airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/publish/commands.py

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
from pipelines.cli.click_decorators import click_ci_requirements_option
1212
from pipelines.cli.confirm_prompt import confirm
1313
from pipelines.cli.dagger_pipeline_command import DaggerPipelineCommand
14-
from pipelines.consts import ContextState
14+
from pipelines.consts import DEFAULT_PYTHON_PACKAGE_REGISTRY_URL, ContextState
1515
from pipelines.helpers.utils import fail_if_missing_docker_hub_creds
1616

1717

@@ -59,6 +59,19 @@
5959
envvar="SLACK_CHANNEL",
6060
default="#connector-publish-updates",
6161
)
62+
@click.option(
63+
"--python-registry-token",
64+
help="Access token for python registry",
65+
type=click.STRING,
66+
envvar="PYTHON_REGISTRY_TOKEN",
67+
)
68+
@click.option(
69+
"--python-registry-url",
70+
help="Which python registry registry to publish to. If not set, the default pypi is used. For test pypi, use https://test.pypi.org/legacy/",
71+
type=click.STRING,
72+
default=DEFAULT_PYTHON_PACKAGE_REGISTRY_URL,
73+
envvar="PYTHON_REGISTRY_URL",
74+
)
6275
@click.pass_context
6376
async def publish(
6477
ctx: click.Context,
@@ -69,6 +82,8 @@ async def publish(
6982
metadata_service_gcs_credentials: str,
7083
slack_webhook: str,
7184
slack_channel: str,
85+
python_registry_token: str,
86+
python_registry_url: str,
7287
) -> bool:
7388
ctx.obj["spec_cache_gcs_credentials"] = spec_cache_gcs_credentials
7489
ctx.obj["spec_cache_bucket_name"] = spec_cache_bucket_name
@@ -109,6 +124,8 @@ async def publish(
109124
s3_build_cache_access_key_id=ctx.obj.get("s3_build_cache_access_key_id"),
110125
s3_build_cache_secret_key=ctx.obj.get("s3_build_cache_secret_key"),
111126
use_local_cdk=ctx.obj.get("use_local_cdk"),
127+
python_registry_token=python_registry_token,
128+
python_registry_url=python_registry_url,
112129
)
113130
for connector in ctx.obj["selected_connectors_with_modified_files"]
114131
]

‎airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/publish/context.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,12 +46,16 @@ def __init__(
4646
s3_build_cache_access_key_id: Optional[str] = None,
4747
s3_build_cache_secret_key: Optional[str] = None,
4848
use_local_cdk: bool = False,
49+
python_registry_token: Optional[str] = None,
50+
python_registry_url: Optional[str] = None,
4951
) -> None:
5052
self.pre_release = pre_release
5153
self.spec_cache_bucket_name = spec_cache_bucket_name
5254
self.metadata_bucket_name = metadata_bucket_name
5355
self.spec_cache_gcs_credentials = sanitize_gcs_credentials(spec_cache_gcs_credentials)
5456
self.metadata_service_gcs_credentials = sanitize_gcs_credentials(metadata_service_gcs_credentials)
57+
self.python_registry_token = python_registry_token
58+
self.python_registry_url = python_registry_url
5559
pipeline_name = f"Publish {connector.technical_name}"
5660
pipeline_name = pipeline_name + " (pre-release)" if pre_release else pipeline_name
5761

‎airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/publish/pipeline.py

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -352,6 +352,16 @@ async def _run_python_registry_publish_pipeline(context: PublishConnectorContext
352352
if not python_registry_context:
353353
return results, False
354354

355+
if not context.python_registry_token or not context.python_registry_url:
356+
# If the python registry token or url are not set, we can't publish to the python registry - stop the pipeline.
357+
return [
358+
StepResult(
359+
PublishToPythonRegistry(python_registry_context),
360+
status=StepStatus.FAILURE,
361+
stderr="Pypi publishing is enabled, but python registry token or url are not set.",
362+
)
363+
], True
364+
355365
check_python_registry_package_exists_results = await CheckPythonRegistryPackageDoesNotExist(python_registry_context).run()
356366
results.append(check_python_registry_package_exists_results)
357367
if check_python_registry_package_exists_results.status is StepStatus.SKIPPED:

‎airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/poetry/publish/commands.py

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -47,10 +47,11 @@ def _validate_python_version(_ctx: dict, _param: dict, value: Optional[str]) ->
4747
envvar="PYTHON_REGISTRY_TOKEN",
4848
)
4949
@click.option(
50-
"--registry-url",
50+
"--python-registry-url",
5151
help="Which registry to publish to. If not set, the default pypi is used. For test pypi, use https://test.pypi.org/legacy/",
5252
type=click.STRING,
5353
default=DEFAULT_PYTHON_PACKAGE_REGISTRY_URL,
54+
envvar="PYTHON_REGISTRY_URL",
5455
)
5556
@click.option(
5657
"--publish-name",
@@ -69,7 +70,7 @@ async def publish(
6970
ctx: click.Context,
7071
click_pipeline_context: ClickPipelineContext,
7172
python_registry_token: str,
72-
registry_url: str,
73+
python_registry_url: str,
7374
publish_name: Optional[str],
7475
publish_version: Optional[str],
7576
) -> bool:
@@ -85,7 +86,7 @@ async def publish(
8586
ci_context=ctx.obj.get("ci_context"),
8687
ci_gcs_credentials=ctx.obj["ci_gcs_credentials"],
8788
python_registry_token=python_registry_token,
88-
registry=registry_url,
89+
registry=python_registry_url,
8990
package_path=ctx.obj["package_path"],
9091
package_name=publish_name,
9192
version=publish_version,

0 commit comments

Comments
 (0)