Skip to content

TPU Provisioner: Fix skipped scale ups & garbage collect zombie node pools #277

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 10 commits into from
Mar 19, 2024
Merged

TPU Provisioner: Fix skipped scale ups & garbage collect zombie node pools #277

merged 10 commits into from
Mar 19, 2024

Conversation

nstogner
Copy link
Collaborator

@nstogner nstogner commented Mar 4, 2024 •
edited
Loading

  • Avoid scale ups when node pool is in a deletion state.
  • Avoid scale ups when the Pod being reconciled is in a deletion state.
  • Use owner UID for node pool names to avoid scale up failure when the scheduler has placed another workload on the same existing node pool.
  • Run a loop that deletes node pools that have no Nodes, are in an errored state, and where the Pod that created the node pool no longer exists (the deletion reconciler would not see these b/c there are no Node objects).
  • Update events to be more descriptive for easier debugging.
  • Make Secure Boot configurable (defaults to same behavior).
  • Make deletion delay configurable to ensure make test passes consistently.

@nstogner nstogner changed the title TPU Provisioner: Fix skipped scale ups TPU Provisioner: Fix skipped scale ups & garbage collect zombie node pools Mar 8, 2024
@nstogner nstogner force-pushed the tpu-provisioner--fix-scale-up-bug branch from 1e51be3 to 1039c27 Compare March 12, 2024 01:26
danielvegamyhre
danielvegamyhre reviewed Mar 12, 2024
View reviewed changes
Copy link
Contributor

@danielvegamyhre danielvegamyhre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall, left a few minor comments and questions

@nstogner
Copy link
Collaborator Author

Updated to address comments. Also made deletion delay configurable to make sure make test passes consistently.

@nstogner nstogner force-pushed the tpu-provisioner--fix-scale-up-bug branch from b9fdef5 to 72c34c6 Compare March 19, 2024 19:47
@nstogner
Copy link
Collaborator Author

Rebased @danielvegamyhre

@danielvegamyhre
Copy link
Contributor

/gcbrun

@danielvegamyhre danielvegamyhre merged commit 4bbabed into GoogleCloudPlatform:main Mar 19, 2024
4 checks passed
leroyjb pushed a commit to leroyjb/ai-on-gke that referenced this pull request Jan 24, 2025
@nstogner
TPU Provisioner: Fix skipped scale ups & garbage collect zombie node …
c25b7f3 
…pools (GoogleCloudPlatform#277)

* Switch to using owner UID for node pool naming and retry on creation of node pool in stopping state

* Rename error

* Fix not-found handling on delete and fix event msg

* Add node pool garbage collector and improve events

* Update garbage collector logging and fix case where pod ref does not exist

* Add logs

* Make secure-boot configurable

* Fix typo in log

* Fix integration tests

* Address comments and make node pool deletion delay configurable to allow tests to pass
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@danielvegamyhre danielvegamyhre danielvegamyhre approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@nstogner @danielvegamyhre