Skip to content

Commit 202c3bc

Browse files
btkcodedevsajarin
authored and
nataly
committed
🎉New Source: Google Webfonts [low-code cdk] (#18496)
* New source: Google webfonts * chore: Add Docs * chore: update changelog * chore: resolved given comments for PR * chore: unwanted files removed * fix: generate and add source definitions Co-authored-by: sajarin <[email protected]>
1 parent ee0c2e2 commit 202c3bc

File tree

26 files changed

+662
-0
lines changed

26 files changed

+662
-0
lines changed

‎airbyte-config/init/src/main/resources/seed/source_definitions.yaml

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -480,6 +480,14 @@
480480
icon: google-sheets.svg
481481
sourceType: file
482482
releaseStage: generally_available
483+
- name: Google Webfonts
484+
sourceDefinitionId: a68fbcde-b465-4ab3-b2a6-b0590a875835
485+
dockerRepository: airbyte/source-google-webfonts
486+
dockerImageTag: 0.1.0
487+
documentationUrl: https://docs.airbyte.com/integrations/sources/google-webfonts
488+
icon: googleworkpace.svg
489+
sourceType: api
490+
releaseStage: alpha
483491
- name: Google Workspace Admin Reports
484492
sourceDefinitionId: ed9dfefa-1bbc-419d-8c5e-4d78f0ef6734
485493
dockerRepository: airbyte/source-google-workspace-admin-reports

‎airbyte-config/init/src/main/resources/seed/source_specs.yaml

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4763,6 +4763,34 @@
47634763
- - "client_secret"
47644764
oauthFlowOutputParameters:
47654765
- - "refresh_token"
4766+
- dockerImage: "airbyte/source-google-webfonts:0.1.0"
4767+
spec:
4768+
documentationUrl: "https://docs.airbyte.com/integrations/sources/google-webfonts"
4769+
connectionSpecification:
4770+
$schema: "http://json-schema.org/draft-07/schema#"
4771+
title: "Google Webfonts Spec"
4772+
type: "object"
4773+
required:
4774+
- "api_key"
4775+
additionalProperties: true
4776+
properties:
4777+
api_key:
4778+
type: "string"
4779+
description: "API key is required to access google apis, For getting your's\
4780+
\ goto google console and generate api key for Webfonts"
4781+
airbyte_secret: true
4782+
sort:
4783+
type: "string"
4784+
description: "Optional, to find how to sort"
4785+
prettyPrint:
4786+
type: "string"
4787+
description: "Optional, boolean type"
4788+
alt:
4789+
type: "string"
4790+
description: "Optional, Available params- json, media, proto"
4791+
supportsNormalization: false
4792+
supportsDBT: false
4793+
supported_destination_sync_modes: []
47664794
- dockerImage: "airbyte/source-google-workspace-admin-reports:0.1.8"
47674795
spec:
47684796
documentationUrl: "https://docs.airbyte.com/integrations/sources/google-workspace-admin-reports"

‎airbyte-integrations/connectors/source-google-webfonts/.dockerignore

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
*
2+
!Dockerfile
3+
!main.py
4+
!source_google_webfonts
5+
!setup.py
6+
!secrets

‎airbyte-integrations/connectors/source-google-webfonts/Dockerfile

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
FROM python:3.9.11-alpine3.15 as base
2+
3+
# build and load all requirements
4+
FROM base as builder
5+
WORKDIR /airbyte/integration_code
6+
7+
# upgrade pip to the latest version
8+
RUN apk --no-cache upgrade \
9+
&& pip install --upgrade pip \
10+
&& apk --no-cache add tzdata build-base
11+
12+
13+
COPY setup.py ./
14+
# install necessary packages to a temporary folder
15+
RUN pip install --prefix=/install .
16+
17+
# build a clean environment
18+
FROM base
19+
WORKDIR /airbyte/integration_code
20+
21+
# copy all loaded and built libraries to a pure basic image
22+
COPY --from=builder /install /usr/local
23+
# add default timezone settings
24+
COPY --from=builder /usr/share/zoneinfo/Etc/UTC /etc/localtime
25+
RUN echo "Etc/UTC" > /etc/timezone
26+
27+
# bash is installed for more convenient debugging.
28+
RUN apk --no-cache add bash
29+
30+
# copy payload code only
31+
COPY main.py ./
32+
COPY source_google_webfonts ./source_google_webfonts
33+
34+
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
35+
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
36+
37+
LABEL io.airbyte.version=0.1.0
38+
LABEL io.airbyte.name=airbyte/source-google-webfonts

‎airbyte-integrations/connectors/source-google-webfonts/README.md

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
# Google Webfonts Source
2+
3+
This is the repository for the Google Webfonts configuration based source connector.
4+
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/google-webfonts).
5+
6+
## Local development
7+
8+
### Prerequisites
9+
**To iterate on this connector, make sure to complete this prerequisites section.**
10+
11+
#### Minimum Python version required `= 3.9.0`
12+
13+
#### Build & Activate Virtual Environment and install dependencies
14+
From this connector directory, create a virtual environment:
15+
```
16+
python -m venv .venv
17+
```
18+
19+
This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your
20+
development environment of choice. To activate it from the terminal, run:
21+
```
22+
source .venv/bin/activate
23+
pip install -r requirements.txt
24+
```
25+
If you are in an IDE, follow your IDE's instructions to activate the virtualenv.
26+
27+
Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is
28+
used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`.
29+
If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything
30+
should work as you expect.
31+
32+
#### Building via Gradle
33+
You can also build the connector in Gradle. This is typically used in CI and not needed for your development workflow.
34+
35+
To build using Gradle, from the Airbyte repository root, run:
36+
```
37+
./gradlew :airbyte-integrations:connectors:source-google-webfonts:build
38+
```
39+
40+
#### Create credentials
41+
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/google-webfonts)
42+
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_google_webfonts/spec.yaml` file.
43+
Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
44+
See `integration_tests/sample_config.json` for a sample config file.
45+
46+
**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source google-webfonts test creds`
47+
and place them into `secrets/config.json`.
48+
49+
### Locally running the connector docker image
50+
51+
#### Build
52+
First, make sure you build the latest Docker image:
53+
```
54+
docker build . -t airbyte/source-google-webfonts:dev
55+
```
56+
57+
You can also build the connector image via Gradle:
58+
```
59+
./gradlew :airbyte-integrations:connectors:source-google-webfonts:airbyteDocker
60+
```
61+
When building via Gradle, the docker image name and tag, respectively, are the values of the `io.airbyte.name` and `io.airbyte.version` `LABEL`s in
62+
the Dockerfile.
63+
64+
#### Run
65+
Then run any of the connector commands as follows:
66+
```
67+
docker run --rm airbyte/source-google-webfonts:dev spec
68+
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-google-webfonts:dev check --config /secrets/config.json
69+
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-google-webfonts:dev discover --config /secrets/config.json
70+
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-google-webfonts:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
71+
```
72+
## Testing
73+
74+
#### Acceptance Tests
75+
Customize `acceptance-test-config.yml` file to configure tests. See [Source Acceptance Tests](https://docs.airbyte.io/connector-development/testing-connectors/source-acceptance-tests-reference) for more information.
76+
If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
77+
78+
To run your integration tests with docker
79+
80+
### Using gradle to run tests
81+
All commands should be run from airbyte project root.
82+
To run unit tests:
83+
```
84+
./gradlew :airbyte-integrations:connectors:source-google-webfonts:unitTest
85+
```
86+
To run acceptance and custom integration tests:
87+
```
88+
./gradlew :airbyte-integrations:connectors:source-google-webfonts:integrationTest
89+
```
90+
91+
## Dependency Management
92+
All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.
93+
We split dependencies between two groups, dependencies that are:
94+
* required for your connector to work need to go to `MAIN_REQUIREMENTS` list.
95+
* required for the testing need to go to `TEST_REQUIREMENTS` list
96+
97+
### Publishing a new version of the connector
98+
You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
99+
1. Make sure your changes are passing unit and integration tests.
100+
1. Bump the connector version in `Dockerfile` -- just increment the value of the `LABEL io.airbyte.version` appropriately (we use [SemVer](https://semver.org/)).
101+
1. Create a Pull Request.
102+
1. Pat yourself on the back for being an awesome contributor.
103+
1. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.

‎airbyte-integrations/connectors/source-google-webfonts/__init__.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
#
2+
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
3+
#

‎airbyte-integrations/connectors/source-google-webfonts/acceptance-test-config.yml

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# See [Source Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/source-acceptance-tests-reference)
2+
# for more information about how to configure these tests
3+
connector_image: airbyte/source-google-webfonts:dev
4+
tests:
5+
spec:
6+
- spec_path: "source_google_webfonts/spec.yaml"
7+
connection:
8+
- config_path: "secrets/config.json"
9+
status: "succeed"
10+
- config_path: "integration_tests/invalid_config.json"
11+
status: "failed"
12+
discovery:
13+
- config_path: "secrets/config.json"
14+
basic_read:
15+
- config_path: "secrets/config.json"
16+
configured_catalog_path: "integration_tests/configured_catalog.json"
17+
empty_streams: []
18+
full_refresh:
19+
- config_path: "secrets/config.json"
20+
configured_catalog_path: "integration_tests/configured_catalog.json"

‎airbyte-integrations/connectors/source-google-webfonts/acceptance-test-docker.sh

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
#!/usr/bin/env sh
2+
3+
# Build latest connector image
4+
docker build . -t $(cat acceptance-test-config.yml | grep "connector_image" | head -n 1 | cut -d: -f2-)
5+
6+
# Pull latest acctest image
7+
docker pull airbyte/source-acceptance-test:latest
8+
9+
# Run
10+
docker run --rm -it \
11+
-v /var/run/docker.sock:/var/run/docker.sock \
12+
-v /tmp:/tmp \
13+
-v $(pwd):/test_input \
14+
airbyte/source-acceptance-test \
15+
--acceptance-test-config /test_input
16+

‎airbyte-integrations/connectors/source-google-webfonts/bootstrap.md

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Google-webfonts
2+
3+
The connector uses the v1 API documented here: https://developers.google.com/fonts/docs/developer_api . It is
4+
straightforward HTTP REST API with API authentication.
5+
6+
## API key
7+
8+
Api key is mandate for this connector to work, It could be generated by a gmail account for free at https://console.cloud.google.com/apis/dashboard.
9+
Just pass the generated API key and optional parameters for establishing the connection. Example:123
10+
11+
## Implementation details
12+
13+
## Setup guide
14+
15+
### Step 1: Set up Google-webfonts connection
16+
17+
- Generate an API key (Example: 12345)
18+
- Params (If specific info is needed)
19+
- Available params
20+
- sort: SORT_UNDEFINED, ALPHA, DATE, STYLE, TRENDING, POPULARITY
21+
- alt: json, media or proto
22+
- prettyPrint: boolean
23+
24+
## Step 2: Generate schema for the endpoint
25+
26+
### Custom schema is generated and tested with different IDs
27+
28+
## Step 3: Spec, Secrets, and connector yaml files are configured with reference to the Airbyte documentation.
29+
30+
## In a nutshell:
31+
32+
1. Navigate to the Airbyte Open Source dashboard.
33+
2. Set the name for your source.
34+
3. Enter your `api_key`.
35+
5. Enter your config params if needed. (Optional)
36+
6. Click **Set up source**.
37+
38+
* We use only GET methods, towards the webfonts endpoints which is straightforward

‎airbyte-integrations/connectors/source-google-webfonts/build.gradle

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
plugins {
2+
id 'airbyte-python'
3+
id 'airbyte-docker'
4+
id 'airbyte-source-acceptance-test'
5+
}
6+
7+
airbytePython {
8+
moduleDirectory 'source_google_webfonts'
9+
}

0 commit comments

Comments
 (0)