Skip to content

Commit 93b4046

Browse files
alafanecherejatinyadav-cc
authored andcommitted
source-faker: adopt our base image (airbytehq#35172)
1 parent e57dfaa commit 93b4046

File tree

4 files changed

+65
-50
lines changed

4 files changed

+65
-50
lines changed

airbyte-integrations/connectors/source-faker/Dockerfile

Lines changed: 0 additions & 38 deletions
This file was deleted.

airbyte-integrations/connectors/source-faker/README.md

Lines changed: 57 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -56,19 +56,70 @@ python main.py read --config secrets/config.json --catalog integration_tests/con
5656
### Locally running the connector docker image
5757

5858

59-
#### Build
60-
**Via [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md) (recommended):**
59+
60+
61+
#### Use `airbyte-ci` to build your connector
62+
The Airbyte way of building this connector is to use our `airbyte-ci` tool.
63+
You can follow install instructions [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#L1).
64+
Then running the following command will build your connector:
65+
6166
```bash
62-
airbyte-ci connectors --name=source-faker build
67+
airbyte-ci connectors --name source-faker build
68+
```
69+
Once the command is done, you will find your connector image in your local docker registry: `airbyte/source-faker:dev`.
70+
71+
##### Customizing our build process
72+
When contributing on our connector you might need to customize the build process to add a system dependency or set an env var.
73+
You can customize our build process by adding a `build_customization.py` module to your connector.
74+
This module should contain a `pre_connector_install` and `post_connector_install` async function that will mutate the base image and the connector container respectively.
75+
It will be imported at runtime by our build process and the functions will be called if they exist.
76+
77+
Here is an example of a `build_customization.py` module:
78+
```python
79+
from __future__ import annotations
80+
81+
from typing import TYPE_CHECKING
82+
83+
if TYPE_CHECKING:
84+
# Feel free to check the dagger documentation for more information on the Container object and its methods.
85+
# https://dagger-io.readthedocs.io/en/sdk-python-v0.6.4/
86+
from dagger import Container
87+
88+
89+
async def pre_connector_install(base_image_container: Container) -> Container:
90+
return await base_image_container.with_env_variable("MY_PRE_BUILD_ENV_VAR", "my_pre_build_env_var_value")
91+
92+
async def post_connector_install(connector_container: Container) -> Container:
93+
return await connector_container.with_env_variable("MY_POST_BUILD_ENV_VAR", "my_post_build_env_var_value")
6394
```
6495

65-
An image will be built with the tag `airbyte/source-faker:dev`.
96+
#### Build your own connector image
97+
This connector is built using our dynamic built process in `airbyte-ci`.
98+
The base image used to build it is defined within the metadata.yaml file under the `connectorBuildOptions`.
99+
The build logic is defined using [Dagger](https://dagger.io/) [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/pipelines/builds/python_connectors.py).
100+
It does not rely on a Dockerfile.
101+
102+
If you would like to patch our connector and build your own a simple approach would be to:
103+
104+
1. Create your own Dockerfile based on the latest version of the connector image.
105+
```Dockerfile
106+
FROM airbyte/source-faker:latest
107+
108+
COPY . ./airbyte/integration_code
109+
RUN pip install ./airbyte/integration_code
66110

67-
**Via `docker build`:**
111+
# The entrypoint and default env vars are already set in the base image
112+
# ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
113+
# ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
114+
```
115+
Please use this as an example. This is not optimized.
116+
117+
2. Build your image:
68118
```bash
69119
docker build -t airbyte/source-faker:dev .
120+
# Running the spec command against your patched connector
121+
docker run airbyte/source-faker:dev spec
70122
```
71-
72123
#### Run
73124

74125
Then run any of the connector commands as follows:
@@ -108,4 +159,3 @@ You've checked out the repo, implemented a million dollar feature, and you're re
108159
5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
109160
6. Pat yourself on the back for being an awesome contributor.
110161
7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
111-

airbyte-integrations/connectors/source-faker/metadata.yaml

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,12 @@ data:
44
sl: 100
55
allowedHosts:
66
hosts: []
7+
connectorBuildOptions:
8+
baseImage: docker.io/airbyte/python-connector-base:1.2.0@sha256:c22a9d97464b69d6ef01898edf3f8612dc11614f05a84984451dde195f337db9
79
connectorSubtype: api
810
connectorType: source
911
definitionId: dfd88b22-b603-4c3d-aad7-3701784586b1
10-
dockerImageTag: 6.0.0
12+
dockerImageTag: 6.0.1
1113
dockerRepository: airbyte/source-faker
1214
documentationUrl: https://docs.airbyte.com/integrations/sources/faker
1315
githubIssueLabel: source-faker
@@ -33,6 +35,10 @@ data:
3335
6.0.0:
3436
message: Declare 'id' columns as primary keys.
3537
upgradeDeadline: "2024-04-01"
38+
remoteRegistries:
39+
pypi:
40+
enabled: true
41+
packageName: airbyte-source-faker
3642
resourceRequirements:
3743
jobSpecific:
3844
- jobType: sync
@@ -45,10 +51,6 @@ data:
4551
- products
4652
- purchases
4753
supportLevel: community
48-
remoteRegistries:
49-
pypi:
50-
enabled: true
51-
packageName: airbyte-source-faker
5254
tags:
5355
- language:python
5456
metadataSpecVersion: "1.0"

docs/integrations/sources/faker.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,7 @@ None!
101101

102102
| Version | Date | Pull Request | Subject |
103103
| :------ | :--------- | :-------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------- |
104+
| 6.0.1 | 2024-02-12 | [TBD](https://github.com/airbytehq/airbyte/pull/TBD) | Base image migration: remove Dockerfile and use the python-connector-base image |
104105
| 6.0.0 | 2024-01-30 | [34644](https://github.com/airbytehq/airbyte/pull/34644) | Declare 'id' columns as primary keys. |
105106
| 5.0.2 | 2024-01-17 | [34344](https://github.com/airbytehq/airbyte/pull/34344) | Ensure unique state messages |
106107
| 5.0.1 | 2023-01-08 | [34033](https://github.com/airbytehq/airbyte/pull/34033) | Add standard entrypoints for usage with AirbyteLib |

0 commit comments

Comments
 (0)