Skip to content

Commit e636293

Browse files
authored
Reorder sequence for execution of tasks (#405)
I am unable to reproduce issue #366 locally after trying multiple times. Looking at the code, I am understanding that the `run_pipeline_no_wait` and `run_pipeline_wait` try to execute the same pipeline with same name and resources. The `run_pipeline_no_wait` tasks starts the pipeline and does not wait for the pipeline to finish before marking the task as success. My hunch here is that since the pipeline launched by `run_pipeline_no_wait` is still in progress, another immediate attempt by `run_pipeline_wait` task to start the same pipeline is the reason for the failure of the task observed in issue #366. Hence, I am reordering the task sequence so that `run_pipeline_wait` task runs first; the task waits for the pipeline to finish before marking the task as success and then we launch the `run_pipeline_no_wait` task as the downstream task. I would like to push this commit & merge to main so that we can observe in our Astro Cloud deployment's master DAG on whether this solves the problem. If the problem still persists, we will then need to spend additional efforts in investigating the issue.
1 parent 6e553a2 commit e636293

File tree

1 file changed

+6
-6
lines changed

1 file changed

+6
-6
lines changed

astronomer/providers/microsoft/azure/example_dags/example_adf_run_pipeline.py

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -185,17 +185,17 @@ def delete_azure_data_factory_storage_pipeline() -> None:
185185
# [END howto_create_resource_group]
186186

187187
# [START howto_operator_adf_run_pipeline]
188-
run_pipeline_no_wait = AzureDataFactoryRunPipelineOperatorAsync(
189-
task_id="run_pipeline_no_wait",
188+
run_pipeline_wait = AzureDataFactoryRunPipelineOperatorAsync(
189+
task_id="run_pipeline_wait",
190190
pipeline_name=PIPELINE_NAME,
191-
wait_for_termination=False,
192191
)
193192
# [END howto_operator_adf_run_pipeline]
194193

195194
# [START howto_operator_adf_run_pipeline]
196-
run_pipeline_wait = AzureDataFactoryRunPipelineOperatorAsync(
197-
task_id="run_pipeline_wait",
195+
run_pipeline_no_wait = AzureDataFactoryRunPipelineOperatorAsync(
196+
task_id="run_pipeline_no_wait",
198197
pipeline_name=PIPELINE_NAME,
198+
wait_for_termination=False,
199199
)
200200
# [END howto_operator_adf_run_pipeline]
201201

@@ -214,8 +214,8 @@ def delete_azure_data_factory_storage_pipeline() -> None:
214214

215215
(
216216
create_azure_data_factory_storage_pipeline
217-
>> run_pipeline_no_wait
218217
>> run_pipeline_wait
218+
>> run_pipeline_no_wait
219219
>> pipeline_run_sensor_async
220220
>> remove_azure_data_factory_storage_pipeline
221221
)

0 commit comments

Comments
 (0)