connector base image: declare the base image package and implement (#30303)

Co-authored-by: alafanechere <[email protected]>

Author: alafanechere

.github/workflows/airbyte-ci-tests.yml

@@ -32,3 +32,15 @@ jobs:
           sentry_dsn: ${{ secrets.SENTRY_AIRBYTE_CI_DSN }}
           github_token: ${{ secrets.GH_PAT_MAINTENANCE_OCTAVIA }}
           subcommand: "test airbyte-ci/connectors/pipelines"
+      - name: Run airbyte-ci/connectors/base_images tests
+        id: run-airbyte-ci-connectors-base-images-tests
+        uses: ./.github/actions/run-dagger-pipeline
+        with:
+          context: "pull_request"
+          docker_hub_password: ${{ secrets.DOCKER_HUB_PASSWORD }}
+          docker_hub_username: ${{ secrets.DOCKER_HUB_USERNAME }}
+          gcp_gsm_credentials: ${{ secrets.GCP_GSM_CREDENTIALS }}
+          gcs_credentials: ${{ secrets.METADATA_SERVICE_PROD_GCS_CREDENTIALS }}
+          sentry_dsn: ${{ secrets.SENTRY_AIRBYTE_CI_DSN }}
+          github_token: ${{ secrets.GH_PAT_MAINTENANCE_OCTAVIA }}
+          subcommand: "test airbyte-ci/connectors/base_images"

.pre-commit-config.yaml

@@ -14,7 +14,7 @@ repos:
       - id: black
         args: ["--config", "pyproject.toml"]
   - repo: https://github.com/timothycrosley/isort
-    rev: 5.10.1
+    rev: 5.12.0
     hooks:
       - id: isort
         args:

airbyte-ci/connectors/base_images/README.md

@@ -0,0 +1,80 @@
+# airbyte-connectors-base-images
+
+This python package contains the base images used by Airbyte connectors.
+It is intended to be used as a python library.
+Our connector build pipeline ([`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#L1)) **will** use this library to build the connector images.
+Our base images are declared in code, using the [Dagger Python SDK](https://dagger-io.readthedocs.io/en/sdk-python-v0.6.4/).
+
+
+
+## Where are the Dockerfiles?
+Our base images are not declared using Dockerfiles.
+They are declared in code using the [Dagger Python SDK](https://dagger-io.readthedocs.io/en/sdk-python-v0.6.4/).
+We prefer this approach because it allows us to interact with base images container as code: we can use python to declare the base images and use the full power of the language to build and test them.
+However, we do artificially generate Dockerfiles for debugging and documentation purposes.
+
+
+
+### Example for `airbyte/python-connector-base`:
+```dockerfile
+FROM docker.io/python:3.9.18-slim-bookworm@sha256:44b7f161ed03f85e96d423b9916cdc8cb0509fb970fd643bdbc9896d49e1cad0
+RUN ln -snf /usr/share/zoneinfo/Etc/UTC /etc/localtime
+RUN pip install --upgrade pip==23.2.1
+ENV POETRY_VIRTUALENVS_CREATE=false
+ENV POETRY_VIRTUALENVS_IN_PROJECT=false
+ENV POETRY_NO_INTERACTION=1
+RUN pip install poetry==1.6.1
+```
+
+
+
+## Base images
+
+
+### `airbyte/python-connector-base`
+
+| Version | Published | Docker Image Address | Changelog | 
+|---------|-----------|--------------|-----------|
+|  1.0.0 | ✅| docker.io/airbyte/python-connector-base:1.0.0@sha256:dd17e347fbda94f7c3abff539be298a65af2d7fc27a307d89297df1081a45c27 | Initial release: based on Python 3.9.18, on slim-bookworm system, with pip==23.2.1 and poetry==1.6.1 |
+
+
+## How to release a new base image version (example for Python)
+
+### Requirements
+* [Docker](https://docs.docker.com/get-docker/)
+* [Poetry](https://python-poetry.org/docs/#installation)
+* Dockerhub logins
+
+### Steps
+1. `poetry install`
+2. Open  `base_images/python/bases.py`.
+3. Make changes to the `AirbytePythonConnectorBaseImage`, you're likely going to change the `get_container` method to change the base image.
+4. Implement the `container` property which must return a `dagger.Container` object.
+5. **Recommended**: Add new sanity checks to `run_sanity_check` to confirm that the new version is working as expected.
+6. Cut a new base image version by running `poetry run generate-release`. You'll need your DockerHub credentials.
+
+It will:
+  - Prompt you to pick which base image you'd like to publish.
+  - Prompt you for a major/minor/patch/pre-release version bump.
+  - Prompt you for  a changelog message.
+  - Run the sanity checks on the new version.
+  - Optional: Publish the new version to DockerHub.
+  - Regenerate the docs and the registry json file.
+7. Commit and push your changes.
+8. Create a PR and ask for a review from the Connector Operations team.
+
+**Please note that if you don't publish your image while cutting the new version you can publish it later with `poetry run publish <repository> <version>`.**
+No connector will use the new base image version until its metadata is updated to use it.
+If you're not fully confident with the new base image version please:
+  - please publish it as a pre-release version
+  - try out the new version on a couple of connectors
+  - cut a new version with a major/minor/patch bump and publish it
+  - This steps can happen in different PRs.
+
+
+## Running tests locally
+```bash
+poetry run pytest
+# Static typing checks
+poetry run mypy base_images --check-untyped-defs
+```

airbyte-ci/connectors/base_images/base_images/__init__.py

@@ -0,0 +1,7 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+from rich.console import Console
+
+console = Console()

airbyte-ci/connectors/base_images/base_images/bases.py

@@ -0,0 +1,101 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+"""This module declares common (abstract) classes and methods used by all base images."""
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from typing import final
+
+import dagger
+import semver
+
+from .published_image import PublishedImage
+
+
+class AirbyteConnectorBaseImage(ABC):
+    """An abstract class that represents an Airbyte base image.
+    Please do not declare any Dagger with_exec instruction in this class as in the abstract class context we have no guarantee about the underlying system used in the base image.
+    """
+
+    @final
+    def __init__(self, dagger_client: dagger.Client, version: semver.VersionInfo):
+        """Initializes the Airbyte base image.
+
+        Args:
+            dagger_client (dagger.Client): The dagger client used to build the base image.
+            version (semver.VersionInfo): The version of the base image.
+        """
+        self.dagger_client = dagger_client
+        self.version = version
+
+    # INSTANCE PROPERTIES:
+
+    @property
+    def name_with_tag(self) -> str:
+        """Returns the full name of the Airbyte base image, with its tag.
+
+        Returns:
+            str: The full name of the Airbyte base image, with its tag.
+        """
+        return f"{self.repository}:{self.version}"
+
+    # MANDATORY SUBCLASSES ATTRIBUTES / PROPERTIES:
+
+    @property
+    @abstractmethod
+    def root_image(self) -> PublishedImage:
+        """Returns the base image used to build the Airbyte base image.
+
+        Raises:
+            NotImplementedError: Raised if a subclass does not define a 'root_image' attribute.
+
+        Returns:
+            PublishedImage: The base image used to build the Airbyte base image.
+        """
+        raise NotImplementedError("Subclasses must define a 'root_image' attribute.")
+
+    @property
+    @abstractmethod
+    def repository(self) -> str:
+        """This is the name of the repository where the image will be hosted.
+        e.g: airbyte/python-connector-base
+
+        Raises:
+            NotImplementedError: Raised if a subclass does not define an 'repository' attribute.
+
+        Returns:
+            str: The repository name where the image will be hosted.
+        """
+        raise NotImplementedError("Subclasses must define an 'repository' attribute.")
+
+    # MANDATORY SUBCLASSES METHODS:
+
+    @abstractmethod
+    def get_container(self, platform: dagger.Platform) -> dagger.Container:
+        """Returns the container of the Airbyte connector base image."""
+        raise NotImplementedError("Subclasses must define a 'get_container' method.")
+
+    @abstractmethod
+    async def run_sanity_checks(self, platform: dagger.Platform):
+        """Runs sanity checks on the base image container.
+        This method is called before image publication.
+
+        Args:
+            base_image_version (AirbyteConnectorBaseImage): The base image version on which the sanity checks should run.
+
+        Raises:
+            SanityCheckError: Raised if a sanity check fails.
+        """
+        raise NotImplementedError("Subclasses must define a 'run_sanity_checks' method.")
+
+    # INSTANCE METHODS:
+    @final
+    def get_base_container(self, platform: dagger.Platform) -> dagger.Container:
+        """Returns a container using the base image. This container is used to build the Airbyte base image.
+
+        Returns:
+            dagger.Container: The container using the base python image.
+        """
+        return self.dagger_client.pipeline(self.name_with_tag).container(platform=platform).from_(self.root_image.address)