Skip to content

Commit 619d58a

Browse files
alafanecherealafanechere
committed
airbyte-ci: multi arch build
1 parent d97a399 commit 619d58a

File tree

12 files changed

+111
-67
lines changed

12 files changed

+111
-67
lines changed

‎.github/workflows/publish_connectors.yml

+5
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,10 @@ on:
1818
type: string
1919
default: ci-runner-connector-publish-large-dagger-0-6-4
2020
required: true
21+
airbyte-ci-binary-url:
22+
description: "URL to airbyte-ci binary"
23+
required: false
24+
default: https://connectors.airbyte.com/airbyte-ci/releases/ubuntu/latest/airbyte-ci
2125
jobs:
2226
publish_connectors:
2327
name: Publish connectors
@@ -62,6 +66,7 @@ jobs:
6266
s3_build_cache_access_key_id: ${{ secrets.SELF_RUNNER_AWS_ACCESS_KEY_ID }}
6367
s3_build_cache_secret_key: ${{ secrets.SELF_RUNNER_AWS_SECRET_ACCESS_KEY }}
6468
subcommand: "connectors ${{ github.event.inputs.connectors-options }} publish ${{ github.event.inputs.publish-options }}"
69+
airbyte_ci_binary_url: ${{ github.event.inputs.airbyte-ci-binary-url }}
6570

6671
set-instatus-incident-on-failure:
6772
name: Create Instatus Incident on Failure

‎airbyte-ci/connectors/pipelines/README.md

+22-12
Original file line numberDiff line numberDiff line change
@@ -124,17 +124,17 @@ At this point you can run `airbyte-ci` commands.
124124

125125
#### Options
126126

127-
| Option | Default value | Mapped environment variable | Description |
128-
| ------------------------------------------ | ---------------------------------------------------------------------------------------------- | ----------------------------- | ------------------------------------------------------------------------------------------- |
129-
| `--enable-dagger-run/--disable-dagger-run` | `--enable-dagger-run`` | | Disables the Dagger terminal UI. | | |
130-
| `--is-local/--is-ci` | `--is-local` | | Determines the environment in which the CLI runs: local environment or CI environment. |
131-
| `--git-branch` | The checked out git branch name | `CI_GIT_BRANCH` | The git branch on which the pipelines will run. |
132-
| `--git-revision` | The current branch head | `CI_GIT_REVISION` | The commit hash on which the pipelines will run. |
133-
| `--diffed-branch` | `origin/master` | | Branch to which the git diff will happen to detect new or modified files. |
134-
| `--gha-workflow-run-id` | | | GHA CI only - The run id of the GitHub action workflow |
135-
| `--ci-context` | `manual` | | The current CI context: `manual` for manual run, `pull_request`, `nightly_builds`, `master` |
136-
| `--pipeline-start-timestamp` | Current epoch time | `CI_PIPELINE_START_TIMESTAMP` | Start time of the pipeline as epoch time. Used for pipeline run duration computation. |
137-
| `--show-dagger-logs/--hide-dagger-logs` | `--hide-dagger-logs` | | Flag to show or hide the dagger logs. |
127+
| Option | Default value | Mapped environment variable | Description |
128+
| ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- | ------------------------------------------------------------------------------------------- |
129+
| `--enable-dagger-run/--disable-dagger-run` | `--enable-dagger-run`` | | Disables the Dagger terminal UI. | | | | |
130+
| `--is-local/--is-ci` | `--is-local` | | Determines the environment in which the CLI runs: local environment or CI environment. |
131+
| `--git-branch` | The checked out git branch name | `CI_GIT_BRANCH` | The git branch on which the pipelines will run. |
132+
| `--git-revision` | The current branch head | `CI_GIT_REVISION` | The commit hash on which the pipelines will run. |
133+
| `--diffed-branch` | `origin/master` | | Branch to which the git diff will happen to detect new or modified files. |
134+
| `--gha-workflow-run-id` | | | GHA CI only - The run id of the GitHub action workflow |
135+
| `--ci-context` | `manual` | | The current CI context: `manual` for manual run, `pull_request`, `nightly_builds`, `master` |
136+
| `--pipeline-start-timestamp` | Current epoch time | `CI_PIPELINE_START_TIMESTAMP` | Start time of the pipeline as epoch time. Used for pipeline run duration computation. |
137+
| `--show-dagger-logs/--hide-dagger-logs` | `--hide-dagger-logs` | | Flag to show or hide the dagger logs. |
138138

139139
### <a id="connectors-command-subgroup"></a>`connectors` command subgroup
140140

@@ -254,6 +254,9 @@ It's mainly purposed for local use.
254254
Build a single connector:
255255
`airbyte-ci connectors --name=source-pokeapi build`
256256

257+
Build a single connector for multiple architectures:
258+
`airbyte-ci connectors --name=source-pokeapi build --architecture=linux/amd64 --architecture=linux/arm64`
259+
257260
Build multiple connectors:
258261
`airbyte-ci connectors --name=source-pokeapi --name=source-bigquery build`
259262

@@ -290,11 +293,17 @@ flowchart TD
290293
distTar-->connector
291294
normalization--"if supports normalization"-->connector
292295
293-
load[Load to docker host with :dev tag, current platform]
296+
load[Load to docker host with :dev tag]
294297
spec[Get spec]
295298
connector-->spec--"if success"-->load
296299
```
297300

301+
### Options
302+
303+
| Option | Multiple | Default value | Description |
304+
| --------------------- | -------- | -------------- | ----------------------------------------------------------------- |
305+
| `--architecture`/`-a` | True | Local platform | Defines for which architecture the connector image will be built. |
306+
298307
### <a id="connectors-publish-command"></a>`connectors publish` command
299308
Run a publish pipeline for one or multiple connectors.
300309
It's mainly purposed for CI use to release a connector update.
@@ -433,6 +442,7 @@ This command runs the Python tests for a airbyte-ci poetry package.
433442
## Changelog
434443
| Version | PR | Description |
435444
| ------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
445+
| 2.8.0 | [#32816](https://github.com/airbytehq/airbyte/pull/32816) | Add `--architecture` option to connector build. |
436446
| 2.7.0 | [#31930](https://github.com/airbytehq/airbyte/pull/31930) | Merge airbyte-ci-internal into airbyte-ci |
437447
| 2.6.0 | [#31831](https://github.com/airbytehq/airbyte/pull/31831) | Add `airbyte-ci format` commands, remove connector-specific formatting check |
438448
| 2.5.9 | [#32427](https://github.com/airbytehq/airbyte/pull/32427) | Re-enable caching for source-postgres |

‎airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/build_image/commands.py

+16-2
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,15 @@
22
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
33
#
44

5+
from typing import List
6+
57
import asyncclick as click
8+
import dagger
69
from pipelines.airbyte_ci.connectors.build_image.steps import run_connector_build_pipeline
710
from pipelines.airbyte_ci.connectors.context import ConnectorContext
811
from pipelines.airbyte_ci.connectors.pipeline import run_connectors_pipelines
912
from pipelines.cli.dagger_pipeline_command import DaggerPipelineCommand
13+
from pipelines.consts import BUILD_PLATFORMS, LOCAL_BUILD_PLATFORM
1014

1115

1216
@click.command(cls=DaggerPipelineCommand, help="Build all images for the selected connectors.")
@@ -17,10 +21,19 @@
1721
default=False,
1822
type=bool,
1923
)
24+
@click.option(
25+
"-a",
26+
"--architecture",
27+
"build_architectures",
28+
help="Architecture for which to build the connector image. If not specified, the image will be built for the local architecture.",
29+
multiple=True,
30+
default=[LOCAL_BUILD_PLATFORM],
31+
type=click.Choice(BUILD_PLATFORMS, case_sensitive=True),
32+
)
2033
@click.pass_context
21-
async def build(ctx: click.Context, use_host_gradle_dist_tar: bool) -> bool:
34+
async def build(ctx: click.Context, use_host_gradle_dist_tar: bool, build_architectures: List[str]) -> bool:
2235
"""Runs a build pipeline for the selected connectors."""
23-
36+
build_platforms = [dagger.Platform(architecture) for architecture in build_architectures]
2437
connectors_contexts = [
2538
ConnectorContext(
2639
pipeline_name=f"Build connector {connector.technical_name}",
@@ -53,6 +66,7 @@ async def build(ctx: click.Context, use_host_gradle_dist_tar: bool) -> bool:
5366
ctx.obj["concurrency"],
5467
ctx.obj["dagger_logs_path"],
5568
ctx.obj["execute_timeout"],
69+
build_platforms,
5670
)
5771

5872
return True

‎airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/build_image/steps/__init__.py

+14-11
Original file line numberDiff line numberDiff line change
@@ -5,17 +5,16 @@
55

66
from __future__ import annotations
77

8-
import platform
8+
from typing import List
99

1010
import anyio
11+
import dagger
1112
from connector_ops.utils import ConnectorLanguage
12-
from pipelines.models.steps import StepResult
13-
from pipelines.airbyte_ci.connectors.build_image.steps import python_connectors
13+
from pipelines.airbyte_ci.connectors.build_image.steps import java_connectors, python_connectors
1414
from pipelines.airbyte_ci.connectors.build_image.steps.common import LoadContainerToLocalDockerHost, StepStatus
15-
from pipelines.consts import LOCAL_BUILD_PLATFORM
16-
from pipelines.airbyte_ci.connectors.build_image.steps import java_connectors
1715
from pipelines.airbyte_ci.connectors.context import ConnectorContext
1816
from pipelines.airbyte_ci.connectors.reports import ConnectorReport
17+
from pipelines.models.steps import StepResult
1918

2019

2120
class NoBuildStepForLanguageError(Exception):
@@ -29,29 +28,33 @@ class NoBuildStepForLanguageError(Exception):
2928
}
3029

3130

32-
async def run_connector_build(context: ConnectorContext) -> StepResult:
31+
async def run_connector_build(context: ConnectorContext, build_platforms: List[dagger.Platform]) -> StepResult:
3332
"""Run a build pipeline for a single connector."""
3433
if context.connector.language not in LANGUAGE_BUILD_CONNECTOR_MAPPING:
3534
raise NoBuildStepForLanguageError(f"No build step for connector language {context.connector.language}.")
36-
return await LANGUAGE_BUILD_CONNECTOR_MAPPING[context.connector.language](context)
35+
return await LANGUAGE_BUILD_CONNECTOR_MAPPING[context.connector.language](context, build_platforms)
3736

3837

39-
async def run_connector_build_pipeline(context: ConnectorContext, semaphore: anyio.Semaphore) -> ConnectorReport:
38+
async def run_connector_build_pipeline(
39+
context: ConnectorContext, semaphore: anyio.Semaphore, build_platforms: List[dagger.Platform]
40+
) -> ConnectorReport:
4041
"""Run a build pipeline for a single connector.
4142
4243
Args:
4344
context (ConnectorContext): The initialized connector context.
44-
45+
semaphore (anyio.Semaphore): The semaphore to use to limit the number of concurrent builds.
46+
build_platforms (List[dagger.Platform]): The platforms for which to build the connector.
4547
Returns:
4648
ConnectorReport: The reports holding builds results.
4749
"""
4850
step_results = []
4951
async with semaphore:
5052
async with context:
51-
build_result = await run_connector_build(context)
53+
build_result = await run_connector_build(context, build_platforms)
54+
per_platform_built_containers = build_result.output_artifact
5255
step_results.append(build_result)
5356
if context.is_local and build_result.status is StepStatus.SUCCESS:
54-
load_image_result = await LoadContainerToLocalDockerHost(context, LOCAL_BUILD_PLATFORM, build_result.output_artifact).run()
57+
load_image_result = await LoadContainerToLocalDockerHost(context, per_platform_built_containers).run()
5558
step_results.append(load_image_result)
5659
context.report = ConnectorReport(context, step_results, name="BUILD RESULTS")
5760
return context.report

‎airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/build_image/steps/common.py

+19-10
Original file line numberDiff line numberDiff line change
@@ -2,14 +2,15 @@
22
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
33
#
44

5+
import json
56
from abc import ABC
67
from typing import List, Tuple
78

89
import docker
910
from dagger import Container, ExecError, Platform, QueryError
1011
from pipelines.airbyte_ci.connectors.context import ConnectorContext
1112
from pipelines.consts import BUILD_PLATFORMS
12-
from pipelines.helpers.utils import export_container_to_tarball
13+
from pipelines.helpers.utils import export_containers_to_tarball
1314
from pipelines.models.steps import Step, StepResult, StepStatus
1415

1516

@@ -58,26 +59,34 @@ async def _build_connector(self, platform: Platform, *args) -> Container:
5859
class LoadContainerToLocalDockerHost(Step):
5960
IMAGE_TAG = "dev"
6061

61-
def __init__(self, context: ConnectorContext, platform: Platform, containers: dict[Platform, Container]) -> None:
62+
def __init__(self, context: ConnectorContext, containers: dict[Platform, Container]) -> None:
6263
super().__init__(context)
63-
self.platform = platform
64-
self.container = containers[platform]
64+
self.containers = containers
6565

6666
@property
6767
def title(self):
68-
return f"Load {self.image_name}:{self.IMAGE_TAG} for platform {self.platform} to the local docker host."
68+
return f"Load {self.image_name}:{self.IMAGE_TAG} to the local docker host."
6969

7070
@property
7171
def image_name(self) -> Tuple:
7272
return f"airbyte/{self.context.connector.technical_name}"
7373

7474
async def _run(self) -> StepResult:
75-
_, exported_tarball_path = await export_container_to_tarball(self.context, self.container)
75+
container_variants = list(self.containers.values())
76+
_, exported_tar_path = await export_containers_to_tarball(self.context, container_variants)
7677
client = docker.from_env()
7778
try:
78-
with open(exported_tarball_path, "rb") as tarball_content:
79-
new_image = client.images.load(tarball_content.read())[0]
80-
new_image.tag(self.image_name, tag=self.IMAGE_TAG)
81-
return StepResult(self, StepStatus.SUCCESS)
79+
response = client.api.import_image_from_file(str(exported_tar_path), repository=self.image_name, tag=self.IMAGE_TAG)
80+
try:
81+
image_sha = json.loads(response)["status"]
82+
except KeyError:
83+
return StepResult(
84+
self,
85+
StepStatus.FAILURE,
86+
stderr=f"Failed to load image {self.image_name}:{self.IMAGE_TAG} to your Docker host: {response}",
87+
)
88+
return StepResult(
89+
self, StepStatus.SUCCESS, stdout=f"Loaded image {self.image_name}:{self.IMAGE_TAG} to your Docker host ({image_sha})."
90+
)
8291
except ConnectionError:
8392
return StepResult(self, StepStatus.FAILURE, stderr="The connection to the local docker host failed.")

‎airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/build_image/steps/java_connectors.py

+6-6
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,13 @@
22
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
33
#
44

5-
from typing import List, Optional, Tuple, Union
65

7-
from dagger import Container, Directory, ExecError, File, Host, Platform, QueryError
6+
from typing import List
7+
8+
from dagger import Container, Directory, File, Platform, QueryError
89
from pipelines.airbyte_ci.connectors.build_image.steps.common import BuildConnectorImagesBase
910
from pipelines.airbyte_ci.connectors.context import ConnectorContext
1011
from pipelines.airbyte_ci.steps.gradle import GradleTask
11-
from pipelines.consts import LOCAL_BUILD_PLATFORM
1212
from pipelines.dagger.containers import java
1313
from pipelines.models.steps import StepResult, StepStatus
1414

@@ -49,21 +49,21 @@ async def _build_connector(self, platform: Platform, dist_tar: File) -> Containe
4949
return await java.with_airbyte_java_connector(self.context, dist_tar, platform)
5050

5151

52-
async def run_connector_build(context: ConnectorContext) -> StepResult:
52+
async def run_connector_build(context: ConnectorContext, build_platforms: List[Platform]) -> StepResult:
5353
"""Create the java connector distribution tar file and build the connector image."""
5454

5555
if context.use_host_gradle_dist_tar and context.is_local:
5656
# Special case: use a local dist tar to speed up local development.
5757
dist_dir = await context.dagger_client.host().directory(dist_tar_directory_path(context), include=["*.tar"])
5858
# Speed things up by only building for the local platform.
59-
return await BuildConnectorImages(context, LOCAL_BUILD_PLATFORM).run(dist_dir)
59+
return await BuildConnectorImages(context, *build_platforms).run(dist_dir)
6060

6161
# Default case: distribution tar is built by the dagger pipeline.
6262
build_connector_tar_result = await BuildConnectorDistributionTar(context).run()
6363
if build_connector_tar_result.status is not StepStatus.SUCCESS:
6464
return build_connector_tar_result
6565
dist_dir = await build_connector_tar_result.output_artifact.directory(dist_tar_directory_path(context))
66-
return await BuildConnectorImages(context).run(dist_dir)
66+
return await BuildConnectorImages(context, *build_platforms).run(dist_dir)
6767

6868

6969
def dist_tar_directory_path(context: ConnectorContext) -> str:

0 commit comments

Comments
 (0)