Surfacing of DBT Cloud Logs to the logs of the DBTCloudRunJobOperator

[harveymarshall]

Description

Currently if a DBT Cloud Job triggered by the DBT Cloud Job Operator fails. The airflow task which uses the operator will fail and error but it will not get the logs from DBT Cloud so we are aware of why it failed.

Use case/motivation

Easier developer experience when debugging tasks. Helps remove extra steps when seeing why a DAG has failed.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[geraj1010]

Took a quick look and I think this is possible. Just need update DbtCloudHook to pull the logging that's available from the API response. Perhaps we would want to grab status_message from Retrieve Run response? (https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Retrieve%20Run)

I'd be happy to take this on if no one else already hasn't.

[potiuk]

I'd be happy to take this on if no one else already hasn't.

Feel free :D

[harveymarshall]

Any updates has this been taken up? Any updates would be appreciated :)

[geraj1010]

Any updates has this been taken up? Any updates would be appreciated :)

Greetings! I should have an update later this week :) Just coming off of a cold.

[geraj1010]

Took a deeper dive and it looks like the logs (for steps) can be found with this API URL: https://YOUR_ACCESS_URL/api/v2/accounts/{accountId}/runs/{runId}/?include_related=["run_steps"] (Source: https://docs.getdbt.com/docs/dbt-versions/2022-release-notes?utm_source=forethought-chat&utm_lp=forethought-chat#list-steps-api-endpoint-deprecation-warning)

However, you get all steps of a DAG back including preliminary steps like Clone git repository. I was thinking of surfacing only the steps with status: 20 (i.e. ERROR). Would that be helpful @harveymarshall ?