Skip to content

Commit bf719b8

Browse files
alafanecherealafanechere
authored
connectors-ci: disable dependency scanning (#29033)
1 parent 549e36f commit bf719b8

File tree

8 files changed

+109
-58
lines changed

8 files changed

+109
-58
lines changed

‎airbyte-ci/connectors/pipelines/README.md

Lines changed: 16 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,7 @@ At this point you can run `airbyte-ci` commands from the root of the repository.
9696
#### Options
9797

9898
| Option | Default value | Mapped environment variable | Description |
99-
|-----------------------------------------|---------------------------------|-------------------------------|---------------------------------------------------------------------------------------------|
99+
| --------------------------------------- | ------------------------------- | ----------------------------- | ------------------------------------------------------------------------------------------- |
100100
| `--no-tui` | | | Disables the Dagger terminal UI. |
101101
| `--is-local/--is-ci` | `--is-local` | | Determines the environment in which the CLI runs: local environment or CI environment. |
102102
| `--git-branch` | The checked out git branch name | `CI_GIT_BRANCH` | The git branch on which the pipelines will run. |
@@ -115,16 +115,16 @@ Available commands:
115115
* `airbyte-ci connectors publish`: Publish a connector to Airbyte's DockerHub.
116116

117117
#### Options
118-
| Option | Multiple | Default value | Description |
119-
|------------------------|----------|---------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
120-
| `--use-remote-secrets` | False | True | If True, connectors configuration will be pulled from Google Secret Manager. Requires the GCP_GSM_CREDENTIALS environment variable to be set with a service account with permission to read GSM secrets. If False the connector configuration will be read from the local connector `secrets` folder. |
121-
| `--name` | True | | Select a specific connector for which the pipeline will run. Can be used multiple time to select multiple connectors. The expected name is the connector technical name. e.g. `source-pokeapi` |
122-
| `--release-stage` | True | | Select connectors with a specific release stage: `alpha`, `beta`, `generally_available`. Can be used multiple times to select multiple release stages. |
123-
| `--language` | True | | Select connectors with a specific language: `python`, `low-code`, `java`. Can be used multiple times to select multiple languages. |
124-
| `--modified` | False | False | Run the pipeline on only the modified connectors on the branch or previous commit (depends on the pipeline implementation). |
125-
| `--concurrency` | False | 5 | Control the number of connector pipelines that can run in parallel. Useful to speed up pipelines or control their resource usage. |
126-
| `--metadata-change-only/--not-metadata-change-only` | False | `--not-metadata-change-only` | Only run the pipeline on connectors with changes on their metadata.yaml file. |
127-
118+
| Option | Multiple | Default value | Description |
119+
| -------------------------------------------------------------- | -------- | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
120+
| `--use-remote-secrets` | False | True | If True, connectors configuration will be pulled from Google Secret Manager. Requires the GCP_GSM_CREDENTIALS environment variable to be set with a service account with permission to read GSM secrets. If False the connector configuration will be read from the local connector `secrets` folder. |
121+
| `--name` | True | | Select a specific connector for which the pipeline will run. Can be used multiple time to select multiple connectors. The expected name is the connector technical name. e.g. `source-pokeapi` |
122+
| `--release-stage` | True | | Select connectors with a specific release stage: `alpha`, `beta`, `generally_available`. Can be used multiple times to select multiple release stages. |
123+
| `--language` | True | | Select connectors with a specific language: `python`, `low-code`, `java`. Can be used multiple times to select multiple languages. |
124+
| `--modified` | False | False | Run the pipeline on only the modified connectors on the branch or previous commit (depends on the pipeline implementation). |
125+
| `--concurrency` | False | 5 | Control the number of connector pipelines that can run in parallel. Useful to speed up pipelines or control their resource usage. |
126+
| `--metadata-change-only/--not-metadata-change-only` | False | `--not-metadata-change-only` | Only run the pipeline on connectors with changes on their metadata.yaml file. |
127+
| `--enable-dependency-scanning / --disable-dependency-scanning` | False | ` --disable-dependency-scanning` | When enabled the dependency scanning will be performed to detect the connectors to select according to a dependency change. |
128128

129129
### <a id="connectors-list-command"></a>`connectors list` command
130130
Retrieve the list of connectors satisfying the provided filters.
@@ -285,7 +285,7 @@ Publish all connectors modified in the head commit: `airbyte-ci connectors --mod
285285
### Options
286286

287287
| Option | Required | Default | Mapped environment variable | Description |
288-
|--------------------------------------|----------|-----------------|------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
288+
| ------------------------------------ | -------- | --------------- | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
289289
| `--pre-release/--main-release` | False | `--pre-release` | | Whether to publish the pre-release or the main release version of a connector. Defaults to pre-release. For main release you have to set the credentials to interact with the GCS bucket. |
290290
| `--docker-hub-username` | True | | `DOCKER_HUB_USERNAME` | Your username to connect to DockerHub. |
291291
| `--docker-hub-password` | True | | `DOCKER_HUB_PASSWORD` | Your password to connect to DockerHub. |
@@ -329,7 +329,7 @@ Validate all `metadata.yaml` files in the repo:
329329

330330
#### Options
331331
| Option | Default | Description |
332-
|--------------------|--------------|----------------------------------------------------------------------------------------------------------------------------|
332+
| ------------------ | ------------ | -------------------------------------------------------------------------------------------------------------------------- |
333333
| `--modified/--all` | `--modified` | Flag to run validation of `metadata.yaml` files on the modified files in the head commit or all the `metadata.yaml` files. |
334334

335335
### <a id="metadata-upload-command"></a>`metadata upload` command
@@ -341,7 +341,7 @@ Upload all the `metadata.yaml` files to a GCS bucket:
341341

342342
#### Options
343343
| Option | Required | Default | Mapped environment variable | Description |
344-
|---------------------|----------|--------------|-----------------------------|--------------------------------------------------------------------------------------------------------------------------|
344+
| ------------------- | -------- | ------------ | --------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
345345
| `--gcs-credentials` | True | | `GCS_CREDENTIALS` | Service account credentials in JSON format with permission to get and upload on the GCS bucket |
346346
| `--modified/--all` | True | `--modified` | | Flag to upload the modified `metadata.yaml` files in the head commit or all the `metadata.yaml` files to a GCS bucket. |
347347

@@ -379,7 +379,8 @@ This command runs the Python tests for a airbyte-ci poetry package.
379379
## Changelog
380380

381381
| Version | PR | Description |
382-
|---------|-----------------------------------------------------------|----------------------------------------------------------------------------------------------|
382+
| ------- | --------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
383+
| 0.4.3 | [#29033](https://github.com/airbytehq/airbyte/pull/29033) | Disable dependency scanning for Java connectors. |
383384
| 0.4.2 | [#29030](https://github.com/airbytehq/airbyte/pull/29030) | Make report path always have the same prefix: `airbyte-ci/`. |
384385
| 0.4.1 | [#28855](https://github.com/airbytehq/airbyte/pull/28855) | Improve the selected connectors detection for connectors commands. |
385386
| 0.4.0 | [#28947](https://github.com/airbytehq/airbyte/pull/28947) | Show Dagger Cloud run URLs in CI |

‎airbyte-ci/connectors/pipelines/pipelines/commands/groups/connectors.py

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,7 @@ def get_selected_connectors_with_modified_files(
5757
modified: bool,
5858
metadata_changes_only: bool,
5959
modified_files: Set[Path],
60+
enable_dependency_scanning: bool = False,
6061
) -> List[ConnectorWithModifiedFiles]:
6162
"""Get the connectors that match the selected criteria.
6263
@@ -67,6 +68,7 @@ def get_selected_connectors_with_modified_files(
6768
modified (bool): Whether to select the modified connectors.
6869
metadata_changes_only (bool): Whether to select only the connectors with metadata changes.
6970
modified_files (Set[Path]): The modified files.
71+
enable_dependency_scanning (bool): Whether to enable the dependency scanning.
7072
Returns:
7173
List[ConnectorWithModifiedFiles]: The connectors that match the selected criteria.
7274
"""
@@ -75,7 +77,9 @@ def get_selected_connectors_with_modified_files(
7577
main_logger.info("--metadata-changes-only overrides --modified")
7678
modified = True
7779

78-
selected_modified_connectors = get_modified_connectors(modified_files) if modified else set()
80+
selected_modified_connectors = (
81+
get_modified_connectors(modified_files, ALL_CONNECTORS, enable_dependency_scanning) if modified else set()
82+
)
7983
selected_connectors_by_name = {c for c in ALL_CONNECTORS if c.technical_name in selected_names}
8084
selected_connectors_by_release_stage = {connector for connector in ALL_CONNECTORS if connector.release_stage in selected_release_stages}
8185
selected_connectors_by_language = {connector for connector in ALL_CONNECTORS if connector.language in selected_languages}
@@ -139,6 +143,12 @@ def get_selected_connectors_with_modified_files(
139143
default=None,
140144
type=int,
141145
)
146+
@click.option(
147+
"--enable-dependency-scanning/--disable-dependency-scanning",
148+
help="When enabled, the dependency scanning will be performed to detect the connectors to test according to a dependency change.",
149+
default=False,
150+
type=bool,
151+
)
142152
@click.pass_context
143153
def connectors(
144154
ctx: click.Context,
@@ -150,6 +160,7 @@ def connectors(
150160
metadata_changes_only: bool,
151161
concurrency: int,
152162
execute_timeout: int,
163+
enable_dependency_scanning: bool,
153164
):
154165
"""Group all the connectors-ci command."""
155166
validate_environment(ctx.obj["is_local"], use_remote_secrets)
@@ -159,7 +170,7 @@ def connectors(
159170
ctx.obj["concurrency"] = concurrency
160171
ctx.obj["execute_timeout"] = execute_timeout
161172
ctx.obj["selected_connectors_with_modified_files"] = get_selected_connectors_with_modified_files(
162-
names, release_stages, languages, modified, metadata_changes_only, ctx.obj["modified_files"]
173+
names, release_stages, languages, modified, metadata_changes_only, ctx.obj["modified_files"], enable_dependency_scanning
163174
)
164175
log_selected_connectors(ctx.obj["selected_connectors_with_modified_files"])
165176

0 commit comments

Comments
 (0)