connectors-ci: unique secret name at session level

[alafanechere]

What

Relates to #27867 #28617
According to the nightly tests results it appears that connectors spec validation tests often fails transiently, on different connectors.
My hypothesis is that the wrong config/secret is mounted to the CAT container here
After investigation it's probably related to a namespace issue when we instantiate Secret object after GSM download::
Dagger derives secret ID from their name and not their content, so far we've named secrets according to the config file name, which is not unique and often config.json.
It means that when we call context.dagger_client.set_secret(secret_filename, secret_plaintext) in a concurrent context we can overwrite a connector secret with a wrong value as dagger_client is a pipeline wide object, not a context specific instance.

How

Make secret name unique by concatenating connector name and file name:

unique_secret_name = f"{context.connector.technical_name}_{secret_file}"
connector_secrets[secret_file] = context.dagger_client.set_secret(unique_secret_name, secret_plaintext)

Tests

Running a nightly build from this branch

[alafanechere]

formatting change

[erohmensing]

Is there still the chance that 2 concurrent runs for the same connector (one which modified the spec) could have this issue?

E.g. the new 1s1t big query branch and the regular bigquery branch?

[alafanechere]

Is there still the chance that 2 concurrent runs for the same connector (one which modified the spec) could have this issue?

@erohmensing By concurrent I mean concurrent within the same test pipeline execution. In our current paradigm a connector can't be tested twice is the same pipeline so I think we're safe.

Two concurrent connector test running on a different branch share a different dagger session/client, the Dagger team told me the Secret ID are session specific.

[erohmensing]

Gotcha, so this is only a problem for changes with dependencies which kick off multiple test runs in the same session. Thanks for the clarification!

[alafanechere]

Gotcha, so this is only a problem for changes with dependencies which kick off multiple test runs in the same session. Thanks for the clarification!

Or specifically nightly builds where we trigger tons of concurrent test execution in a single pipeline.

[erohmensing]

so long as the connector_secrets dict is per connector, looks good! otherwise i'd use the namespaced name as the key instead of just secret_file

[alafanechere]

so long as the connector_secrets dict is per connector, looks good! otherwise i'd use the namespaced name as the key instead of just secret_file

Yep I could have done it but it's not useful as this dict instance is per connector and it would mean "denamespacing" the key downstream when we need to actually name the secret file correctly as expected by CAT.

[alafanechere]

Merging as the nightly build workflow, even if not finished, look healthy so far, and I'd like to see the result on the real tomorrow's nightly build.