Skip to content

Exit when CD task is done #1056

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Mar 21, 2025
Merged

Exit when CD task is done #1056

merged 9 commits into from
Mar 21, 2025

Conversation

lionello
Copy link
Member

@lionello lionello commented Mar 13, 2025

Description

Include wait for CD service to complete as well as the services health check. Resolves an issues where the CD service fails (i.e. setting up ALBs, S3, etc..) but services are still running.

Linked Issues

fixes #1058

Checklist

  • I have performed a self-review of my code
  • I have added appropriate tests
  • I have updated the Defang CLI docs and/or README to reflect my changes, if necessary

@lionello lionello requested a review from nullfunc March 13, 2025 19:07
@lionello lionello changed the title wip Exit when CD task is done Mar 16, 2025
@nullfunc nullfunc requested a review from jordanstephens March 18, 2025 05:14
@nullfunc
Copy link
Contributor

@lionello I've made updates here, please have a look at the changes.

Comment on lines 17 to 19
if DoDryRun {
return ErrDryRun
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we push this up into the caller? If we know it's a dry run, we probably don't even need to invoke WaitCdTaskState, (and print the debug message above) right?

@@ -189,20 +190,27 @@ func WaitAndTail(ctx context.Context, project *compose.Project, client client.Fa
ctx, cancelTail := context.WithCancelCause(ctx)
defer cancelTail(nil) // to cancel WaitServiceState and clean-up context

deploymentStatusCh := make(chan error, 2)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is last referenced about 50 lines later. Do you think we can factor out a function to set up the channel, the wait group, the listener goroutines, and maybe return a cleanup function that we can call at the end?

var cancel context.CancelCauseFunc
ctx, cancel = context.WithCancelCause(ctx)
go func() {
if err := ecs.WaitForTask(ctx, taskArn, 2*time.Second); err != nil {
if stopWhenCDTaskDone || errors.As(err, &ecs.TaskFailure{}) {
isTaskFailure := !errors.Is(err, io.EOF)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused to read that the name isTaskFailure now means notEOF. It seems like errors.As(err, &ecs.TaskFailure{}) is stronger assertion than !errors.Is(err, io.EOF). So, why was this change made?

@nullfunc nullfunc requested a review from jordanstephens March 21, 2025 07:01
@nullfunc nullfunc marked this pull request as ready for review March 21, 2025 16:29
@nullfunc nullfunc merged commit c22d63d into main Mar 21, 2025
12 checks passed
@nullfunc nullfunc deleted the cd-task-fail branch March 21, 2025 20:23
lionello added a commit that referenced this pull request Mar 23, 2025
lionello added a commit that referenced this pull request Mar 23, 2025
lionello added a commit that referenced this pull request Mar 23, 2025
lionello added a commit that referenced this pull request Apr 11, 2025
* Revert "Revert "Exit when CD task is done (#1056)" (#1073)"

This reverts commit 5d5e61a.

* review update

* review updates

* clean up workgroup go code

fix DO so the serverstream will handle cancel

* use testdata dockerignore from main

* clean and refactor

* review update

* remove warning, cancel of tail does not always mean deployment not completed

* review updates

* review updates

* Stream tests

* fix CD deployment failure case

---------

Co-authored-by: Eric Liu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CD Task and Service Status both need to be accounted for before a COMPLETED Deplpyment
3 participants