Skip to content

Commit 684fbff

Browse files
lmossmangirardasherifnadabrianjlai
authored
Enable the Connector Builder server in the platform (#19269)
* init * bad copy/paste * move to top level * Revert "move to top level" This reverts commit aca3534. * attempt to wire up connector builder frontend to server * copy from octaviacli * fix connection to builder server * update * delete * Update * delete python-version * Revert "delete python-version" This reverts commit f9258a7. * setup python * install python * rename * kube stuff * Install python * missing kube file * rename * Update files * Update bumpversion * install python * try with different entrypoint * rename container * point to docker-compose.yaml file * derp * copy acceptance_test.sh * copy from acceptance tests * delete cruft * update * remove application env * reset * reset to master * update * skip comprehensive incremental tests * Revert "skip comprehensive incremental tests" This reverts commit 9cee657. * reset to master * remove cruft * readme * readme * save * buncha stuff * it works * improvements * comment * codesnippet * delete superfluous steps * update port to 8003 * reset to master * Update publish docker * move openapi spec to airbyte-connector-builder * point to openapi spec * dont expose the connector builder to localhost * reset FE components to master * Don't deploy the connector-builder * Revert "Don't deploy the connector-builder" This reverts commit 3d15749. * Revert "Revert "Don't deploy the connector-builder"" This reverts commit beac3d4. * comment out more things related to connector builder server * more attempts at removing the connector builder * comment out more things * Update airbyte-connector-builder/README.md Co-authored-by: Brian Lai <[email protected]> * fixes * rename gradle task * save * copyright header * Apply suggestions from code review Co-authored-by: Lake Mossman <[email protected]> * Update airbyte-webapp/src/config/configProviders.ts Co-authored-by: Lake Mossman <[email protected]> * update * rename * indent * Revert "move openapi spec to airbyte-connector-builder" This reverts commit 57dda04. * Revert "rename" This reverts commit b2d802b. * Revert "Revert "rename"" This reverts commit 91db24f. * point to wrong file in case it fixes the build * point to right openapi file * Revert "Revert "move openapi spec to airbyte-connector-builder"" This reverts commit e46a837. * point to moved file * fix path * Update from master * newline * Add failing test * Revert "Add failing test" This reverts commit ed9fea0. * comment * update commented requires * Add a comment * comments * save * formatting is for losers anyways * pyproject * toml * toml * saved * nice * anothaone * fmt * fixmypy * fix autogen notice header location * improvecomment * save working branch * fmt * delete license and disable caching * removeyaml * fix merge and uncomment flakeCheck * undo css change * fix typo * format * enable CORS * move template to backend server * try adding connector builder server to 'generate-docker' task to see if it fixes CI * try adding buildDockerImage task * try adding gradle.properties * Revert "try adding buildDockerImage task" This reverts commit 2102009. * Revert "try adding connector builder server to 'generate-docker' task to see if it fixes CI" This reverts commit 40b5ae0. * process detail in apiOverride * fix gradle * fix scrollbars * Run connector builder server tests and static analysis from a docker container (#19495) * Revert "try adding buildDockerImage task" This reverts commit 2102009. * Revert "try adding connector builder server to 'generate-docker' task to see if it fixes CI" This reverts commit 40b5ae0. * missing file * build airbyte-connector-builder-server image * Add . * comment * build connector builder server as part of octavia_cli * build connector builder too * patch * missing file * comment * comment * run unit tests in a container * Run integration tests * run * move to platform build * format * rootdir * get the path from the project * Update * pyproject * delete a bunch of code * parameterize * run tests * update * rename * comment * reset * uncomment * dont format * run everything except isort * delete dead code * Revert "delete dead code" This reverts commit b11df71. * comment * Update module * reset to master * mount in home * commit formatted files * try running format * Better comment * reset * Revert "reset" This reverts commit 20808f8. * Update buildSrc/src/main/groovy/airbyte-python-docker.gradle Co-authored-by: Sherif A. Nada <[email protected]> * comment * Add everything except build and hidden directories * symlink * Update airbyte-connector-builder-server/run_format.sh Co-authored-by: Sherif A. Nada <[email protected]> * Update airbyte-connector-builder-server/run_tests.sh Co-authored-by: Sherif A. Nada <[email protected]> Co-authored-by: Sherif A. Nada <[email protected]> Co-authored-by: Alexandre Girard <[email protected]> Co-authored-by: sherifnada <[email protected]> Co-authored-by: Brian Lai <[email protected]>
1 parent a1da5fc commit 684fbff

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

42 files changed

+185
-258
lines changed

.env

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=0.35.15.001
5757
### AIRBYTE SERVICES ###
5858
TEMPORAL_HOST=airbyte-temporal:7233
5959
INTERNAL_API_HOST=airbyte-server:8001
60-
#CONNECTOR_BUILDER_API_HOST=airbyte-connector-builder-server:80 #FIXME: Uncomment this when enabling the connector-builder
60+
CONNECTOR_BUILDER_API_HOST=airbyte-connector-builder-server:80
6161
WEBAPP_URL=http://localhost:8000/
6262
# Although not present as an env var, required for webapp configuration.
6363
API_URL=/api/v1/

.env.dev

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ HACK_LOCAL_ROOT_PARENT=/tmp
2323
WEBAPP_URL=http://localhost:8000/
2424
API_URL=/api/v1/
2525
INTERNAL_API_HOST=airbyte-server:8001
26-
#CONNECTOR_BUILDER_API_HOST=airbyte-connector-builder-server:8003 #FIXME: Uncomment this when enabling the connector-builder
26+
CONNECTOR_BUILDER_API_HOST=airbyte-connector-builder-server:80
2727
SYNC_JOB_MAX_ATTEMPTS=3
2828
SYNC_JOB_MAX_TIMEOUT_DAYS=3
2929
WORKERS_MICRONAUT_ENVIRONMENTS=control-plane

airbyte-connector-builder-server/build.gradle

Lines changed: 20 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,15 +2,15 @@ import org.openapitools.generator.gradle.plugin.tasks.GenerateTask
22

33
plugins {
44
id "org.openapi.generator" version "5.3.1"
5-
id 'airbyte-python'
5+
id 'airbyte-python-docker'
66
id 'airbyte-docker'
77
}
88

9-
airbytePython {
9+
airbytePythonDocker {
1010
moduleDirectory 'connector_builder'
1111
}
1212

13-
task generateOpenApiPythonServer(type: GenerateTask){
13+
task generateOpenApiPythonServer(type: GenerateTask) {
1414
outputs.upToDateWhen { false }
1515

1616
def generatedCodeDir = "$buildDir/airbyte_connector_builder_server"
@@ -42,4 +42,20 @@ task generateOpenApiPythonServer(type: GenerateTask){
4242
}
4343

4444
project.build.dependsOn(generateOpenApiPythonServer)
45-
45+
46+
// java modules such as airbyte-server can use copyGeneratedTar to copy the files to the docker image
47+
// We cannot do this here because we don't generate a tar file
48+
// Instead, we copy the files into the build directory so they can be copied to the docker container
49+
task prepareBuild(type: Copy) {
50+
from layout.projectDirectory.file(".")
51+
exclude '.*'
52+
exclude 'build'
53+
54+
55+
into layout.buildDirectory.dir("docker")
56+
}
57+
58+
tasks.named("buildDockerImage") {
59+
dependsOn prepareBuild
60+
dependsOn copyDocker
61+
}

airbyte-connector-builder-server/connector_builder/entrypoint.py

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,14 +2,24 @@
22
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
33
#
44

5+
from fastapi import FastAPI
6+
from fastapi.middleware.cors import CORSMiddleware
7+
58
from connector_builder.generated.apis.default_api_interface import initialize_router
69
from connector_builder.impl.default_api import DefaultApiImpl
7-
from fastapi import FastAPI
810

911
app = FastAPI(
1012
title="Connector Builder Server API",
1113
description="Connector Builder Server API ",
1214
version="1.0.0",
1315
)
1416

17+
app.add_middleware(
18+
CORSMiddleware,
19+
allow_origins=["*"],
20+
allow_credentials=True,
21+
allow_methods=["*"],
22+
allow_headers=["*"],
23+
)
24+
1525
app.include_router(initialize_router(DefaultApiImpl()))

airbyte-connector-builder-server/connector_builder/generated/apis/default_api_interface.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
#
22
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
33
#
4+
# This file was auto-generated from Airbyte's custom OpenAPI templates. Do not edit it manually.
5+
# coding: utf-8
46

57
import inspect
68
from abc import ABC, abstractmethod

airbyte-connector-builder-server/connector_builder/generated/models/extra_models.py

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,3 @@
1-
#
2-
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
3-
#
4-
51
# coding: utf-8
62

73
from pydantic import BaseModel

airbyte-connector-builder-server/connector_builder/generated/models/http_request.py

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,3 @@
1-
#
2-
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
3-
#
4-
51
# coding: utf-8
62

73
from __future__ import annotations

airbyte-connector-builder-server/connector_builder/generated/models/http_response.py

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,3 @@
1-
#
2-
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
3-
#
4-
51
# coding: utf-8
62

73
from __future__ import annotations

airbyte-connector-builder-server/connector_builder/generated/models/invalid_input_exception_info.py

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,3 @@
1-
#
2-
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
3-
#
4-
51
# coding: utf-8
62

73
from __future__ import annotations

airbyte-connector-builder-server/connector_builder/generated/models/invalid_input_property.py

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,3 @@
1-
#
2-
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
3-
#
4-
51
# coding: utf-8
62

73
from __future__ import annotations

airbyte-connector-builder-server/connector_builder/generated/models/known_exception_info.py

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,3 @@
1-
#
2-
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
3-
#
4-
51
# coding: utf-8
62

73
from __future__ import annotations

airbyte-connector-builder-server/connector_builder/generated/models/stream_read.py

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,3 @@
1-
#
2-
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
3-
#
4-
51
# coding: utf-8
62

73
from __future__ import annotations

airbyte-connector-builder-server/connector_builder/generated/models/stream_read_pages.py

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,3 @@
1-
#
2-
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
3-
#
4-
51
# coding: utf-8
62

73
from __future__ import annotations

airbyte-connector-builder-server/connector_builder/generated/models/stream_read_request_body.py

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,3 @@
1-
#
2-
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
3-
#
4-
51
# coding: utf-8
62

73
from __future__ import annotations

airbyte-connector-builder-server/connector_builder/generated/models/stream_read_slice_descriptor.py

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,3 @@
1-
#
2-
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
3-
#
4-
51
# coding: utf-8
62

73
from __future__ import annotations

airbyte-connector-builder-server/connector_builder/generated/models/stream_read_slices.py

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,3 @@
1-
#
2-
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
3-
#
4-
51
# coding: utf-8
62

73
from __future__ import annotations

airbyte-connector-builder-server/connector_builder/generated/models/streams_list_read.py

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,3 @@
1-
#
2-
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
3-
#
4-
51
# coding: utf-8
62

73
from __future__ import annotations

airbyte-connector-builder-server/connector_builder/generated/models/streams_list_read_streams.py

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,3 @@
1-
#
2-
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
3-
#
4-
51
# coding: utf-8
62

73
from __future__ import annotations

airbyte-connector-builder-server/connector_builder/generated/models/streams_list_request_body.py

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,3 @@
1-
#
2-
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
3-
#
4-
51
# coding: utf-8
62

73
from __future__ import annotations

airbyte-connector-builder-server/connector_builder/impl/default_api.py

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,9 @@
99
from urllib.parse import parse_qs, urljoin, urlparse
1010

1111
from airbyte_cdk.models import AirbyteLogMessage, AirbyteMessage, Type
12+
from fastapi import Body, HTTPException
13+
from jsonschema import ValidationError
14+
1215
from connector_builder.generated.apis.default_api_interface import DefaultApi
1316
from connector_builder.generated.models.http_request import HttpRequest
1417
from connector_builder.generated.models.http_response import HttpResponse
@@ -20,15 +23,13 @@
2023
from connector_builder.generated.models.streams_list_read_streams import StreamsListReadStreams
2124
from connector_builder.generated.models.streams_list_request_body import StreamsListRequestBody
2225
from connector_builder.impl.low_code_cdk_adapter import LowCodeSourceAdapter
23-
from fastapi import Body, HTTPException
2426

2527

2628
class DefaultApiImpl(DefaultApi):
2729
logger = logging.getLogger("airbyte.connector-builder")
2830

2931
async def get_manifest_template(self) -> str:
3032
return """version: "0.1.0"
31-
3233
definitions:
3334
selector:
3435
extractor:
@@ -112,7 +113,7 @@ async def read_stream(self, stream_read_request_body: StreamReadRequestBody = Bo
112113
log_messages = []
113114
try:
114115
for message_group in self._get_message_groups(
115-
adapter.read_stream(stream_read_request_body.stream, stream_read_request_body.config)
116+
adapter.read_stream(stream_read_request_body.stream, stream_read_request_body.config)
116117
):
117118
if isinstance(message_group, AirbyteLogMessage):
118119
log_messages.append({"message": message_group.message})
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
dockerImageName=connector-builder-server
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
#
2+
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
3+
#
4+
5+
6+
def test_test():
7+
assert True
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
cd $1
2+
3+
# Pasted from https://github.com/airbytehq/airbyte/blob/master/buildSrc/src/main/groovy/airbyte-python.gradle#L85-L96
4+
pip install 'mccabe==0.6.1'
5+
pip install 'flake8==4.0.1'
6+
pip install 'pyproject-flake8==0.0.1a2'
7+
pip install 'black==22.3.0'
8+
pip install 'mypy==0.930'
9+
pip install 'isort==5.6.4'
10+
pip install 'pytest==6.1.2'
11+
pip install 'coverage[toml]==6.3.1'
12+
13+
# Format and static analysis
14+
# FIXME: isort formats python files differently from gradlew format
15+
python -m isort --settings-file=pyproject.toml ./
16+
python -m isort --settings-file=pyproject.toml --diff --quiet ./
17+
python -m black --config pyproject.toml ./
18+
python -m black --config pyproject.toml ./ --diff --quiet
19+
python -m pflake8 --config pyproject.toml ./
20+
python -m pflake8 --config pyproject.toml ./ --diff --quiet
21+
python -m mypy --config pyproject.toml ./
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
cd $1
2+
3+
# Install dependencies
4+
pip install -e .
5+
pip install -e '.[main]'
6+
pip install -e '.[tests]'
7+
8+
# Run the tests
9+
python -m coverage run -m pytest unit_tests -c pytest.ini
10+
python -m coverage run -m pytest integration_tests -c pytest.ini

airbyte-connector-builder-server/setup.py

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,14 @@
4444
install_requires=["airbyte-cdk~=0.8", "fastapi", "uvicorn"],
4545
python_requires=">=3.9.11",
4646
extras_require={
47-
"tests": ["MyPy~=0.812", "pytest~=6.2.5", "pytest-cov", "pytest-mock", "pytest-recording", "requests-mock", "pre-commit"],
47+
"tests": [
48+
"MyPy~=0.812",
49+
"pytest~=6.2.5",
50+
"pytest-cov",
51+
"pytest-mock",
52+
"pytest-recording",
53+
"requests-mock",
54+
"pre-commit",
55+
],
4856
},
4957
)

airbyte-connector-builder-server/unit_tests/connector_builder/impl/test_default_api.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@
88

99
import pytest
1010
from airbyte_cdk.models import AirbyteLogMessage, AirbyteMessage, AirbyteRecordMessage, Level, Type
11+
from fastapi import HTTPException
12+
1113
from connector_builder.generated.models.http_request import HttpRequest
1214
from connector_builder.generated.models.http_response import HttpResponse
1315
from connector_builder.generated.models.stream_read import StreamRead
@@ -17,7 +19,6 @@
1719
from connector_builder.generated.models.streams_list_read_streams import StreamsListReadStreams
1820
from connector_builder.generated.models.streams_list_request_body import StreamsListRequestBody
1921
from connector_builder.impl.default_api import DefaultApiImpl
20-
from fastapi import HTTPException
2122

2223
MANIFEST = {
2324
"version": "0.1.0",

airbyte-connector-builder-server/unit_tests/connector_builder/impl/test_low_code_cdk_adapter.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
from airbyte_cdk.sources.declarative.declarative_stream import DeclarativeStream
1313
from airbyte_cdk.sources.declarative.parsers.undefined_reference_exception import UndefinedReferenceException
1414
from airbyte_cdk.sources.streams.http import HttpStream
15+
1516
from connector_builder.impl.low_code_cdk_adapter import LowCodeSourceAdapter
1617

1718

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
#
2+
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
3+
#
4+
5+
6+
def test_test():
7+
assert True

airbyte-webapp/.env

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,4 @@ REACT_APP_FULL_STORY_ORG=13AXQ4
33
REACT_APP_SENTRY_DSN=
44
REACT_APP_INTERCOM_APP_ID=nj1oam7s
55
REACT_APP_OSANO=16A0CTTE7vE8m1Qif/67beec9b-e563-4736-bdb4-4fe4adc39d48
6-
#REACT_APP_CONNECTOR_BUILDER_API=/connector-builder-api/ #FIXME: Uncomment this when enabling the connector builder
6+
REACT_APP_CONNECTOR_BUILDER_API=/connector-builder-api/

airbyte-webapp/nginx/default.conf.template

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@ upstream api-server {
22
server $INTERNAL_API_HOST;
33
}
44

5-
# upstream connector-builder-server { #FIXME: Uncomment this block when enabling the connector-builder
6-
# server $CONNECTOR_BUILDER_API_HOST;
7-
# }
5+
upstream connector-builder-server {
6+
server $CONNECTOR_BUILDER_API_HOST;
7+
}
88

99
server {
1010
listen 80;
@@ -45,10 +45,10 @@ server {
4545
proxy_pass http://api-server/api/;
4646
}
4747

48-
#location /connector-builder-api/ { #FIXME: Uncomment this block when enabling the connector-builder
49-
# fastcgi_read_timeout 1h;
50-
# proxy_read_timeout 1h;
51-
# client_max_body_size 200M;
52-
# proxy_pass http://connector-builder-server/;
53-
#}
48+
location /connector-builder-api/ {
49+
fastcgi_read_timeout 1h;
50+
proxy_read_timeout 1h;
51+
client_max_body_size 200M;
52+
proxy_pass http://connector-builder-server/;
53+
}
5454
}

airbyte-webapp/src/components/StreamTestingPanel/LogsDisplay.module.scss

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
display: flex;
77
flex-direction: column;
88
height: 100%;
9+
overflow-y: hidden;
910
}
1011

1112
.header {

airbyte-webapp/src/components/StreamTestingPanel/StreamTester.module.scss

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
flex-direction: column;
88
align-items: center;
99
gap: variables.$spacing-lg;
10+
min-height: 0;
1011
}
1112

1213
.resizablePanelsContainer {

airbyte-webapp/src/config/configProviders.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ const windowConfigProvider: ConfigProvider = async () => {
2222
const envConfigProvider: ConfigProvider = async () => {
2323
return {
2424
apiUrl: process.env.REACT_APP_API_URL,
25-
// connectorBuilderUrl: process.env.REACT_APP_CONNECTOR_BUILDER_API, #FIXME: Uncomment this when enabling the connector-builder
25+
connectorBuilderUrl: process.env.REACT_APP_CONNECTOR_BUILDER_API,
2626
integrationUrl: process.env.REACT_APP_INTEGRATION_DOCS_URLS,
2727
segment: {
2828
token: process.env.REACT_APP_SEGMENT_TOKEN,

0 commit comments

Comments
 (0)