Skip to content

always push ECS task ARN to xcom in EcsRunTaskOperator #33703

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Aug 24, 2023

Conversation

vandonr-amz
Copy link
Contributor

that xcom value was removed in #29447 and it broke our system test that relied on that xcom value

While I agree that changing the logic around how to reattach is good, removing the xcom value is potentially a breaking change.
It used to be written only when the task didn't complete, but I think returning it all the time makes sense.

@o-nikolas
Copy link
Contributor

It used to be written only when the task didn't complete, but I think returning it all the time makes sense.

We'll now have to support this new/expanded behaviour (unless/until we deprecate it of course). But on the grand scheme of things I think this is fine 👍

@Taragolis
Copy link
Contributor

I have no objection for returning this value, just my 50 cents


This never work as it described

if self.reattach:
# Save the task ARN in XCom to be able to reattach it if needed
self.xcom_push(context, key=self.REATTACH_XCOM_KEY, value=self.arn)

Airflow should clear all previous XComs for the task instance before execute happen


Initial solution might work but it save value in the "fake task_id", which might be a reason for some issues

if self.reattach:
# Save the task ARN in XCom to be able to reattach it if needed
self._xcom_set(
context,
key=self.REATTACH_XCOM_KEY,
value=self.arn,
task_id=self.REATTACH_XCOM_TASK_ID_TEMPLATE.format(task_id=self.task_id),
)

@vincbeck
Copy link
Contributor

Unit tests are failing

Copy link
Contributor

@Taragolis Taragolis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, miss that this operator never return value as it might be expected. And for work EcsRunTaskOperator -> EcsSensor you should set reattach=True and wait_for_completion = False

So make sense now

@vandonr-amz
Copy link
Contributor Author

Oh, miss that this operator never return value as it might be expected.

yeah, the "return value" is the last line of the logs, only in some case... I don't think it's a good idea, but we have to live with it.

@eladkal eladkal merged commit 83efcaa into apache:main Aug 24, 2023
@vandonr-amz vandonr-amz deleted the vandonr/fix branch August 24, 2023 19:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants