Fix failing Databricks tests

[pankajkoti]

Following tests for Databricks are failing and observed in a non-related PR: #290

1. test_databricks_submit_run_operator_async
2. test_databricks_run_now_operator_async

The reason for failure is that the execute method called in these
tests is trying to push an XCOM against task instance 'ti' of
the context. Previously, context was an empty dictionary missing
the 'ti' key. We now are using the create_context utility to
generate a context object which includes the 'ti' key against which
the XCOM is getting pushed.

Closes: #291

[pankajkoti]

I observed Databricks tests failing as part of this PR: #290 and hence created a separate issue and PR to fix these here. Once this is merged to main, I will get these changes into my branch.

[codecov]

Codecov Report

Merging #292 (dc47db9) into main (8310221) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #292   +/-   ##
=======================================
  Coverage   96.61%   96.61%           
=======================================
  Files          53       53           
  Lines        2834     2834           
=======================================
  Hits         2738     2738           
  Misses         96       96           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8310221...dc47db9. Read the comment docs.

[pankajastro]

By default, the value of do_xcom_push param is false of DatabricksSubmitRunOperator and DatabricksRunNowOperator and also I can see in these particular tests we are not setting param do_xcom_push so it is weird that the test is failing.

When do_xcom_push is false it should not push value to the xcom.

[pankajkoti]

By default, the value of do_xcom_push param is false of DatabricksSubmitRunOperator and DatabricksRunNowOperator and also I can see in these particular tests we are not setting param do_xcom_push so it is weird that the test is failing.

Yes, weird why it's failing only in my PR: https://app.circleci.com/pipelines/github/astronomer/astronomer-providers/1445/workflows/6a04ddcb-3480-4f37-b321-0e735af0f7a4/jobs/4527

But, I believe, we need this fix anyway, right @pankajastro ?

[pankajastro]

By default, the value of do_xcom_push param is false of DatabricksSubmitRunOperator and DatabricksRunNowOperator and also I can see in these particular tests we are not setting param do_xcom_push so it is weird that the test is failing.

Yes, weird why it's failing only in my PR: https://app.circleci.com/pipelines/github/astronomer/astronomer-providers/1445/workflows/6a04ddcb-3480-4f37-b321-0e735af0f7a4/jobs/4527

But, I believe, we need this fix anyway, right @pankajastro ?

wait, In my local airflow code I can see the default value for do_xocm_push param is false but in airflow main, I see it is true https://github.com/apache/airflow/blob/main/airflow/providers/databricks/operators/databricks.py#L315 @kaxil @phanikumv

[phanikumv]

By default, the value of do_xcom_push param is false of DatabricksSubmitRunOperator and DatabricksRunNowOperator and also I can see in these particular tests we are not setting param do_xcom_push so it is weird that the test is failing.

Yes, weird why it's failing only in my PR: https://app.circleci.com/pipelines/github/astronomer/astronomer-providers/1445/workflows/6a04ddcb-3480-4f37-b321-0e735af0f7a4/jobs/4527
But, I believe, we need this fix anyway, right @pankajastro ?

wait, In my local airflow code I can see the default value for do_xocm_push param is false but in airflow main, I see it is true https://github.com/apache/airflow/blob/main/airflow/providers/databricks/operators/databricks.py#L315 @kaxil @phanikumv

Ideally the behaviour should be same as that of apache airflow

[kaxil]

Looks like it was changed recently - apache/airflow#22541 on the main branch of Airflow which is also released as part of databircks provider 2.6.0 - https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/index.html#id1

[pankajastro]

Looks like it was changed recently - apache/airflow#22541 on the main branch of Airflow which is also released as part of databircks provider 2.6.0 - https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/index.html#id1

Yes, we are having this issue because of airflow databricks operator behaviour change. But, because of this change, there will be a backward compatibility issue for others also.

[pankajkoti]

I think in general for Airflow, most places I see the value set to True. Irrespective of the value, I believe we should have the fix in this PR for the tests by creating the needed context. Please let me know if you have alternative thoughts.