Allow Async Google Operators to accept `impersonation_chain` and `delegate_to` parameters for authentication #703

bharanidharan14 · 2022-10-06T14:32:19Z

Google operators ignore authentication parameters impersonation_chain which supports connecting with google cloud API without any authentication key by using impersonation chain authentication. These parameters are supported by their sync version operators, So now made a fix to the async Operators to support impersonation_chain and delegate_to other type authentication params.

Closes: #651

codecov · 2022-10-07T09:22:04Z

Codecov Report

Base: 98.50% // Head: 98.46% // Decreases project coverage by -0.03% ⚠️

Coverage data is based on head (3a8d5fa) compared to base (e9864f2).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #703      +/-   ##
==========================================
- Coverage   98.50%   98.46%   -0.04%     
==========================================
  Files          87       87              
  Lines        4746     4753       +7     
==========================================
+ Hits         4675     4680       +5     
- Misses         71       73       +2

Impacted Files	Coverage Δ
...stronomer/providers/google/cloud/hooks/bigquery.py	`99.09% <100.00%> (+0.01%)`	⬆️
...nomer/providers/google/cloud/operators/bigquery.py	`100.00% <100.00%> (ø)`
...nomer/providers/google/cloud/operators/dataproc.py	`98.82% <100.00%> (ø)`
...onomer/providers/google/cloud/triggers/bigquery.py	`93.90% <100.00%> (+0.06%)`	⬆️
...onomer/providers/google/cloud/triggers/dataproc.py	`100.00% <100.00%> (ø)`
...mer/providers/amazon/aws/hooks/redshift_cluster.py	`95.94% <0.00%> (-2.71%)`	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

phanikumv

need the integration test DAG to contain examples of using the impersonation_chain and delegate_to params

pankajastro

Just interested to know how we are testing this.

phanikumv

why are you changing snowflake provider in this PR? It looks like your PR changes for Snowflake came into this PR - which needs to be corrected

bharanidharan14 · 2022-11-30T10:13:59Z

why are you changing snowflake provider in this PR? It looks like your PR changes for Snowflake came into this PR - which needs to be corrected

Yes snowflake changes are not related to these PR changes, reverting it back

bharanidharan14 · 2022-11-30T11:55:22Z

why are you changing snowflake provider in this PR? It looks like your PR changes for Snowflake came into this PR - which needs to be corrected

@phanikumv can you review now

astronomer/providers/google/cloud/example_dags/example_bigquery_impersonation_key.py

astronomer/providers/google/cloud/operators/bigquery.py

bharanidharan14 · 2022-12-01T10:19:35Z

@rajaths010494 @pankajkoti can you review this PR

astronomer/providers/google/cloud/example_dags/example_bigquery_impersonation_key.py

.circleci/integration-tests/master_dag.py

astronomer/providers/google/cloud/example_dags/example_bigquery_impersonation_key.py

phanikumv

please add example dag for dataproc operator as well as you changed the dataproc operator

bharanidharan14 changed the title ~~Add Async Google Operator with impersonation_chain param~~ Add Async Google Operator and trigger to accept impersonation_chain and delegate_to param Oct 6, 2022

bharanidharan14 force-pushed the gcs-authentication-651 branch from 6580114 to e54d8c3 Compare October 6, 2022 14:37

bharanidharan14 changed the title ~~Add Async Google Operator and trigger to accept impersonation_chain and delegate_to param~~ Add Async Google Operator and trigger to accept impersonation_chain and delegate_to param Oct 7, 2022

bharanidharan14 marked this pull request as ready for review October 7, 2022 09:25

bharanidharan14 requested review from phanikumv, rajaths010494, pankajastro and pankajkoti as code owners October 7, 2022 09:25

bharanidharan14 changed the title ~~Add Async Google Operator and trigger to accept impersonation_chain and delegate_to param~~ [WIP]: Add Async Google Operator and trigger to accept impersonation_chain and delegate_to param Oct 7, 2022

phanikumv requested changes Oct 11, 2022

View reviewed changes

bharanidharan14 force-pushed the gcs-authentication-651 branch from 5982e54 to ab64979 Compare October 21, 2022 06:20

pankajastro reviewed Oct 21, 2022

View reviewed changes

bharanidharan14 force-pushed the gcs-authentication-651 branch from ab64979 to 6723b41 Compare November 10, 2022 06:45

bharanidharan14 force-pushed the gcs-authentication-651 branch 2 times, most recently from 73ee033 to 5ffce3f Compare November 29, 2022 09:50

bharanidharan14 changed the title ~~[WIP]: Add Async Google Operator and trigger to accept impersonation_chain and delegate_to param~~ Add Async Google Operator and trigger to accept impersonation_chain and delegate_to param Nov 29, 2022

bharanidharan14 force-pushed the gcs-authentication-651 branch 2 times, most recently from da34334 to 316283a Compare November 30, 2022 06:30

phanikumv requested changes Nov 30, 2022

View reviewed changes

bharanidharan14 force-pushed the gcs-authentication-651 branch from 5f46287 to 7896998 Compare November 30, 2022 11:28

bharanidharan14 requested review from pankajastro and phanikumv November 30, 2022 11:53

phanikumv reviewed Nov 30, 2022

View reviewed changes

astronomer/providers/google/cloud/example_dags/example_bigquery_impersonation_key.py Outdated Show resolved Hide resolved

astronomer/providers/google/cloud/operators/bigquery.py Show resolved Hide resolved

bharanidharan14 requested a review from phanikumv December 1, 2022 06:06

bharanidharan14 force-pushed the gcs-authentication-651 branch from 44e8e6c to e5cef8e Compare December 1, 2022 11:21

Add Async Google Operator with impersonation_chain param

523b9f7

bharanidharan14 added 6 commits December 1, 2022 21:13

Fix BigQueryCheckTrigger

15e4c8f

Fix test case

3064196

Add example dag

3faeb71

Fix mypy issue

9141e35

Add delegate

e994f63

Add example to master dag

a0bc1f8

bharanidharan14 force-pushed the gcs-authentication-651 branch from 896b0c1 to a0bc1f8 Compare December 1, 2022 15:43

rajaths010494 reviewed Dec 2, 2022

View reviewed changes

astronomer/providers/google/cloud/example_dags/example_bigquery_impersonation_key.py Outdated Show resolved Hide resolved

rajaths010494 approved these changes Dec 2, 2022

View reviewed changes

Change sync to async for insert query operator

8b03682

phanikumv reviewed Dec 2, 2022

View reviewed changes

phanikumv changed the title ~~Add Async Google Operator and trigger to accept impersonation_chain and delegate_to param~~ Allow Async Google Operators to accept impersonation_chain and delegate_to parameters for authentication Dec 2, 2022

Fix dag id, doc string

77aaad4

phanikumv requested changes Dec 2, 2022

View reviewed changes

bharanidharan14 added 3 commits December 2, 2022 19:04

Add DataProc operator example with impersonation authentication

00854a0

Fix Example Dag

c32381d

Pass connection id to trigger

3a8d5fa

phanikumv approved these changes Dec 2, 2022

View reviewed changes

bharanidharan14 merged commit 9853f6c into main Dec 2, 2022

bharanidharan14 deleted the gcs-authentication-651 branch December 2, 2022 14:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Allow Async Google Operators to accept `impersonation_chain` and `delegate_to` parameters for authentication #703

Allow Async Google Operators to accept `impersonation_chain` and `delegate_to` parameters for authentication #703

Uh oh!

bharanidharan14 commented Oct 6, 2022 •

edited

Loading

Uh oh!

codecov bot commented Oct 7, 2022 •

edited

Loading

Uh oh!

phanikumv left a comment

Uh oh!

pankajastro left a comment

Uh oh!

phanikumv left a comment •

edited

Loading

Uh oh!

bharanidharan14 commented Nov 30, 2022

Uh oh!

bharanidharan14 commented Nov 30, 2022

Uh oh!

Uh oh!

Uh oh!

bharanidharan14 commented Dec 1, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

phanikumv left a comment •

edited

Loading

Uh oh!

Uh oh!

Allow Async Google Operators to accept impersonation_chain and delegate_to parameters for authentication #703

Allow Async Google Operators to accept impersonation_chain and delegate_to parameters for authentication #703

Uh oh!

Conversation

bharanidharan14 commented Oct 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Oct 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

phanikumv left a comment

Choose a reason for hiding this comment

Uh oh!

pankajastro left a comment

Choose a reason for hiding this comment

Uh oh!

phanikumv left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bharanidharan14 commented Nov 30, 2022

Uh oh!

bharanidharan14 commented Nov 30, 2022

Uh oh!

Uh oh!

Uh oh!

bharanidharan14 commented Dec 1, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

phanikumv left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Allow Async Google Operators to accept `impersonation_chain` and `delegate_to` parameters for authentication #703

Allow Async Google Operators to accept `impersonation_chain` and `delegate_to` parameters for authentication #703

bharanidharan14 commented Oct 6, 2022 •

edited

Loading

codecov bot commented Oct 7, 2022 •

edited

Loading

phanikumv left a comment •

edited

Loading

phanikumv left a comment •

edited

Loading