Skip to content

source-faker to Beta + Fix AllowedHosts Checks #22117

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Jan 31, 2023
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -508,7 +508,7 @@
documentationUrl: https://docs.airbyte.com/integrations/sources/faker
icon: faker.svg
sourceType: api
releaseStage: alpha
releaseStage: beta
allowedHosts:
hosts: []
suggestedStreams:
Expand Down
3 changes: 0 additions & 3 deletions airbyte-integrations/connectors/source-faker/CHANGELOG.md

This file was deleted.

2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-faker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -34,5 +34,5 @@ COPY source_faker ./source_faker
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=2.0.0
LABEL io.airbyte.version=2.0.1
LABEL io.airbyte.name=airbyte/source-faker
1 change: 1 addition & 0 deletions docs/integrations/sources/faker.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ None!

| Version | Date | Pull Request | Subject |
| :------ | :--------- | :-------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------- |
| 2.0.1 | 2022-01-30 | [22117](https://github.com/airbytehq/airbyte/pull/22117) | `soruce-faker` goes beta |
| 2.0.0 | 2022-12-14 | [20492](https://github.com/airbytehq/airbyte/pull/20492) and [20741](https://github.com/airbytehq/airbyte/pull/20741) | Decouple stream states for better parallelism |
| 1.0.0 | 2022-11-28 | [19490](https://github.com/airbytehq/airbyte/pull/19490) | Faker uses the CDK; rename streams to be lower-case (breaking), add determinism to random purchases, and rename |
| 0.2.1 | 2022-10-14 | [19197](https://github.com/airbytehq/airbyte/pull/19197) | Emit `AirbyteEstimateTraceMessage` |
Expand Down
2 changes: 1 addition & 1 deletion tools/ci_connector_ops/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,4 @@ source .venv/bin/activate
pip install -e . # assuming you are in the ./tools/ci_connector_ops directory
```

pip will make binaries for all the commands in setup.py, so you can run `allowed-hosts-check` directly from the virtual-env
pip will make binaries for all the commands in setup.py, so you can run `allowed-hosts-checks` directly from the virtual-env
22 changes: 10 additions & 12 deletions tools/ci_connector_ops/ci_connector_ops/allowed_hosts_checks.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,22 +10,20 @@

RELEASE_STAGES_TO_CHECK = ["generally_available", "beta"]

def get_connectors_missing_allowed_hosts() -> List[str]:
connectors_missing_allowed_hosts = []
changed_connector_names = utils.get_changed_connector_names()

for connector_name in changed_connector_names:
connector_release_stage = utils.get_connector_release_stage(connector_name)
if connector_release_stage in RELEASE_STAGES_TO_CHECK:
missing = not connector_has_allowed_hosts(connector_name)
def get_connectors_missing_allowed_hosts() -> List[utils.Connector]:
connectors_missing_allowed_hosts: List[utils.Connector] = []
changed_connectors = utils.get_changed_connectors()

for connector in changed_connectors:
if connector.release_stage in RELEASE_STAGES_TO_CHECK:
missing = not connector_has_allowed_hosts(connector)
if missing:
connectors_missing_allowed_hosts.append(connector_name)
connectors_missing_allowed_hosts.append(connector)

return connectors_missing_allowed_hosts

def connector_has_allowed_hosts(connector_name: str) -> bool:
definition = utils.get_connector_definition(connector_name)
return definition.get("allowedHosts") is not None
def connector_has_allowed_hosts(connector: utils.Connector) -> bool:
return connector.allowed_hosts is not None


def check_allowed_hosts():
Expand Down
16 changes: 12 additions & 4 deletions tools/ci_connector_ops/ci_connector_ops/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
from dataclasses import dataclass
import logging
from pathlib import Path
from typing import Dict, Optional, Set, Tuple
from typing import Dict, Optional, Set, Tuple, List
import os
import git
import requests
Expand Down Expand Up @@ -108,15 +108,15 @@ def icon_path(self) -> Path:
@property
def code_directory(self) -> Path:
return Path(f"./airbyte-integrations/connectors/{self.technical_name}")

@property
def version(self) -> str:
with open(self.code_directory / "Dockerfile") as f:
for line in f:
if "io.airbyte.version" in line:
return line.split("=")[1].strip()
raise ConnectorVersionNotFound("""
Could not find the connector version from its Dockerfile.
Could not find the connector version from its Dockerfile.
The io.airbyte.version tag is missing.
""")

Expand All @@ -138,7 +138,15 @@ def definition(self) -> Optional[dict]:

@property
def release_stage(self) -> Optional[str]:
return self.definition["releaseStage"] if self.definition else None
return self.definition["releaseStage"] if self.definition and self.definition.get('releaseStage') else None

@property
def allowed_hosts(self) -> Optional[List[str]]:
return self.definition["allowedHosts"] if self.definition and self.definition.get('allowedHosts') else None

@property
def suggested_streams(self) -> Optional[List[str]]:
return self.definition["suggestedStreams"] if self.definition and self.definition.get('suggestedStreams') else None

@property
def acceptance_test_config_path(self) -> Path:
Expand Down