metadata-service: add connectorTestSuitesOptions

[alafanechere]

What

Closes https://github.com/airbytehq/airbyte-internal-issues/issues/7548

We want to introduce new metadata fields for connectors to:

• Configure which test suite will run in CI
• Explicitely map test credentials in secret store to a connector test suite

How

• Introduce a new top level connectorTestSuitesOptions to configure test suite runs.

How will this be used?

This will help in conditionally running specific test suites according to the content of the connector metadata file.

Examples

• A source Python connector on which we want to run unitTests, integrationTests (without secrets) and acceptanceTests (with secrets):

# in metadata.yaml
connectorTestSuitesOptions:
  - suite: unitTests
  - suite: integrationTests
  - suite: acceptanceTests
    testSecrets:
      - name: SECRET_FOR_MY_AWESOME_SOURCE # The actual secret name in the secret store
        fileName: config.yaml # name of the file airbyte-ci will create after pulling the secret from the secret store
        secretStore:
          type: GSM
          alias: airbyte-internal-secret-store # This is to hide the actual address of the secret store, a mapping between alias and address will be stored as a secret value in GHA

• A source java connector on which we want to run unit, integration and acceptance tests without secrets:

connectorTestSuitesOptions:
  - suite: unitTests
  - suite: integrationTests
  - suite: acceptanceTests

User Impact

None as these new fields are not used ATM.

Follow up work

• https://github.com/airbytehq/airbyte-internal-issues/issues/7549
• https://github.com/airbytehq/airbyte-internal-issues/issues/7551
• https://github.com/airbytehq/airbyte-internal-issues/issues/7552

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		May 13, 2024 2:41pm

[alafanechere]

• metadata-service: add connectorTestSuitesOptions #38116 👈
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @alafanechere and the rest of your teammates on Graphite

[bnchrch]

Just one thought on string enum vs model enum I'd love your thoughts on

[bnchrch]

❓ Given that

1. We will likely have different configuration options for each test
2. We may have different configuration values for the same option between tests
3. We want these test suites to be discoverable

What do you think of a format that

1. encourages connector devs to use connectorTestSuitesOptions.suite. integrationTests.enabled = false to disable a test suite
2. provides settings globally and per suite?

example:

connectorTestSuitesOptions:
  suite:
    unitTests:
      enabled: true
    integrationTests:
      enabled: false
    acceptanceTests:
      enabled: true
      override:
        secrets:
          fileName: config_this.yaml
  settings:
    secrets:
      - name: SECRET_FOR_MY_AWESOME_SOURCE
        fileName: config.yaml
        secretStore:
          type: GSM
          alias: airbyte-internal-secret-store

[alafanechere]

@bnchrch I think I like the enabled field. But I think I prefer keeping per test suites secrets as it explicitly declares test suite secrets requirements.
If we'd go with your suggestion airbyte-ci would have to eagerly mount the secrets declared in settings to each test suites even if they're not required. I'd prefer to keep things as explicit as possible and have a way to assess which test suite needs which secret... Does it sound reasonable?

[bnchrch]

Sounds very reasonable! Approved!

[bnchrch]

[bnchrch]

Sounds very reasonable! Approved!