Skip to content

octavia-cli: enable transformations #12727

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
Jun 7, 2022

Conversation

alafanechere
Copy link
Contributor

@alafanechere alafanechere commented May 10, 2022

What

We want to allow users to set up normalization / custom transformation on their connection managed with octavia-cli.

How

octavia generate:

octavia apply (changes inoctavia-cli/octavia_cli/apply/resources.py):

  • Retrieve source/destination definition spec to check if normalization is supported on the destination.
  • Management of operations on a connection is simpler using the WebBackendConnection* endpoints and models. I changed the resources.py code make use of these endpoints in model for connection management. This required setting some nullable fields on these models in airbyte-api/src/main/openapi/config.yaml
  • Deserialize raw configuration operations into API models with Connection._deserialize_operations method.

I also updated unit and integrations tests to test this new feature extensively.

@github-actions github-actions bot added area/api Related to the api area/platform issues related to the platform labels May 10, 2022
@alafanechere alafanechere temporarily deployed to more-secrets May 10, 2022 15:16 Inactive
@alafanechere alafanechere temporarily deployed to more-secrets May 10, 2022 15:16 Inactive
@alafanechere alafanechere temporarily deployed to more-secrets May 10, 2022 16:26 Inactive
@alafanechere alafanechere temporarily deployed to more-secrets May 10, 2022 16:26 Inactive
@alafanechere alafanechere temporarily deployed to more-secrets May 10, 2022 17:34 Inactive
@alafanechere alafanechere temporarily deployed to more-secrets May 10, 2022 17:34 Inactive
@alafanechere alafanechere temporarily deployed to more-secrets May 11, 2022 09:28 Inactive
@alafanechere alafanechere temporarily deployed to more-secrets May 11, 2022 09:28 Inactive
@alafanechere alafanechere temporarily deployed to more-secrets May 11, 2022 16:43 Inactive
@alafanechere alafanechere temporarily deployed to more-secrets May 11, 2022 16:43 Inactive
@alafanechere alafanechere temporarily deployed to more-secrets May 12, 2022 06:40 Inactive
@alafanechere alafanechere temporarily deployed to more-secrets May 12, 2022 06:40 Inactive
@alafanechere alafanechere temporarily deployed to more-secrets May 12, 2022 07:04 Inactive
@alafanechere alafanechere temporarily deployed to more-secrets May 12, 2022 07:05 Inactive
@alafanechere alafanechere force-pushed the augustin/octavia-cli/enable-transformations branch from ae5cd24 to f99b67e Compare May 12, 2022 07:28
@alafanechere alafanechere temporarily deployed to more-secrets May 12, 2022 07:31 Inactive
@alafanechere alafanechere temporarily deployed to more-secrets May 12, 2022 07:31 Inactive
@alafanechere alafanechere temporarily deployed to more-secrets May 12, 2022 07:37 Inactive
@alafanechere alafanechere temporarily deployed to more-secrets May 12, 2022 07:37 Inactive
@alafanechere alafanechere marked this pull request as ready for review May 12, 2022 07:40
@alafanechere alafanechere requested a review from lmossman May 12, 2022 07:41
@alafanechere alafanechere temporarily deployed to more-secrets May 12, 2022 08:03 Inactive
@alafanechere alafanechere temporarily deployed to more-secrets May 12, 2022 08:03 Inactive
@alafanechere alafanechere temporarily deployed to more-secrets May 16, 2022 21:26 Inactive
@alafanechere alafanechere temporarily deployed to more-secrets May 16, 2022 21:26 Inactive
@alafanechere alafanechere temporarily deployed to more-secrets May 19, 2022 17:08 Inactive
@alafanechere alafanechere temporarily deployed to more-secrets May 19, 2022 17:08 Inactive
@alafanechere alafanechere temporarily deployed to more-secrets May 19, 2022 17:19 Inactive
@alafanechere alafanechere temporarily deployed to more-secrets May 19, 2022 17:19 Inactive
@alafanechere alafanechere requested a review from lmossman May 19, 2022 17:22
@alafanechere alafanechere temporarily deployed to more-secrets June 1, 2022 09:31 Inactive
@alafanechere alafanechere temporarily deployed to more-secrets June 3, 2022 10:05 Inactive
Copy link
Contributor

@lmossman lmossman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a couple questions about things that may be able to be removed now that null values are excluded in the API. Otherwise LGTM!

Comment on lines 659 to 666
if "operations" in comparable:
for operation in comparable["operations"]:
for k in self.remote_operation_level_keys_to_filter_out:
operation.pop(k)
if "dbt" in operation["operator_configuration"] and operation["operator_configuration"]["dbt"] is None:
operation["operator_configuration"].pop("dbt")
if "normalization" in operation["operator_configuration"] and operation["operator_configuration"]["normalization"] is None:
operation["operator_configuration"].pop("normalization")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alafanechere I just wanted to double check: now that we have made the API change to exclude null values, is this logic still necessary?

@alafanechere alafanechere temporarily deployed to more-secrets June 7, 2022 16:59 Inactive
@github-actions github-actions bot removed area/api Related to the api area/platform issues related to the platform labels Jun 7, 2022
@alafanechere alafanechere merged commit f614187 into master Jun 7, 2022
@alafanechere alafanechere deleted the augustin/octavia-cli/enable-transformations branch June 7, 2022 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

🐙 octavia-cli: create normalization operation if it does not exists
3 participants