Skip to content

Add Microsoft ADF DAG in master for integration test #260

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Apr 22, 2022
Merged

Add Microsoft ADF DAG in master for integration test #260

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
dd5f3c1
Add folder for azure
bharanidharan14 Apr 19, 2022
9abb679
Implement Azure Data Factory Pipeline run status sensor
bharanidharan14 Apr 20, 2022
4eeb962
Add Test case, code coverage
bharanidharan14 Apr 20, 2022
5c8e86d
Merge branch 'main' into azure_datafactory_pipeline_run_status_sensor
bharanidharan14 Apr 20, 2022
a600cf2
Fix poll interval spell
bharanidharan14 Apr 20, 2022
88626b5
Update Hook class with params, Update example dag with script
bharanidharan14 Apr 21, 2022
96a12bc
Add Azure Data factory pipeline DAG in integration master DAG
bharanidharan14 Apr 22, 2022
0470895
Merge branch 'main' into azure_datafactory_pipeline_run_status_sensor
bharanidharan14 Apr 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .circleci/integration-tests/master_dag.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,12 @@ def prepare_dag_dependency(task_info, execution_time):
dag_run_ids.extend(ids)
chain(*livy_trigger_tasks)

# microsoft Azure Data factory pipeline DAG
adf_pipeline_task_info = [{"adf_pipeline_dag": "example_adf_run_pipeline"}]
adf_pipeline_trigger_tasks, ids = prepare_dag_dependency(adf_pipeline_task_info, "{{ ds }}")
dag_run_ids.extend(ids)
chain(*adf_pipeline_trigger_tasks)

report = PythonOperator(
task_id="get_report",
python_callable=get_report,
Expand All @@ -181,6 +187,7 @@ def prepare_dag_dependency(task_info, execution_time):
http_trigger_tasks[0],
snowflake_trigger_tasks[0],
livy_trigger_tasks[0],
adf_pipeline_trigger_tasks[0],
]

last_task = [
Expand All @@ -193,6 +200,7 @@ def prepare_dag_dependency(task_info, execution_time):
http_trigger_tasks[-1],
snowflake_trigger_tasks[-1],
livy_trigger_tasks[-1],
adf_pipeline_trigger_tasks[-1],
]

last_task >> end
Expand Down
4 changes: 2 additions & 2 deletions astronomer/providers/microsoft/azure/example_dags/example_adf_run_pipeline.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@
default_args = {
"execution_timeout": timedelta(minutes=30),
"azure_data_factory_conn_id": "azure_data_factory_default",
"factory_name": "ADFProvidersTeamDataFactory", # This can also be specified in the ADF connection.
"resource_group_name": "team_provider_resource_group_test", # This can also be specified in the ADF connection.
"factory_name": "ADFProvidersTeamDataFactoryTest", # This can also be specified in the ADF connection.
"resource_group_name": "team_provider_resource_group_test_1", # This can also be specified in the ADF connection.
}

CLIENT_ID = os.getenv("CLIENT_ID", "")
Expand Down