Tidy app entrypoint #7668
Merged
Tidy app entrypoint #7668
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
keturn
reviewed
Feb 21, 2025
RyanJDick
force-pushed
the
ryan/tidy-entry
branch
from
79f7b8e to
9fa365f
Compare
7 tasks
RyanJDick
requested review from
blessedcoolant,
psychedelicious,
brandonrising and
hipsterusername
as code owners
psychedelicious
approved these changes
Feb 24, 2025
There was a problem hiding this comment.
I've improved custom node loading in #7698, moving that code to a function instead of running it implicitly as python loads modules.
Maybe should be re-organised per your other changes.
…esponsible for just initializing the FastAPI app. This also gives clearer control over the order of the initialization steps, which will be important as we add planned torch configurations that must be applied before torch is imported.
…to review comment.
RyanJDick
force-pushed
the
ryan/tidy-entry
branch
from
9ba2713 to
1e2c7c5
Compare
RyanJDick
added a commit
that referenced
this pull request
Feb 28, 2025
…7673) ## Summary This PR adds a `pytorch_cuda_alloc_conf` config flag to control the torch memory allocator behavior. - `pytorch_cuda_alloc_conf` defaults to `None`, preserving the current behavior. - The configuration options are explained here: https://pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf. Tuning this configuration can reduce peak reserved VRAM and improve performance. - Setting `pytorch_cuda_alloc_conf: "backend:cudaMallocAsync"` in `invokeai.yaml` is expected to work well on many systems. This is a good first step for those looking to tune this config. (We may make this the default in the future.) - The optimal configuration seems to be dependent on a number of factors such as device version, VRAM, CUDA kernel version, etc. For now, users will have to experiment with this config to see if it hurts or helps on their systems. In most cases, I expect it to help. ### Memory Tests ``` VAE decode memory usage comparison: - SDXL, fp16, 1024x1024: - `cudaMallocAsync`: allocated=2593 MB, reserved=3200 MB - `native`: allocated=2595 MB, reserved=4418 MB - SDXL, fp32, 1024x1024: - `cudaMallocAsync`: allocated=3982 MB, reserved=5536 MB - `native`: allocated=3982 MB, reserved=7276 MB - SDXL, fp32, 1536x1536: - `cudaMallocAsync`: allocated=8643 MB, reserved=12032 MB - `native`: allocated=8643 MB, reserved=15900 MB ``` ## Related Issues / Discussions N/A ## QA Instructions - [x] Performance tests with `pytorch_cuda_alloc_conf` unset. - [x] Performance tests with `pytorch_cuda_alloc_conf: "backend:cudaMallocAsync"`. ## Merge Plan - [x] Merge #7668 first and change target branch to `main` ## Checklist - [x] _The PR has a short but descriptive title, suitable for a changelog_ - [x] _Tests added / updated (if applicable)_ - [x] _Documentation added / updated (if applicable)_ - [ ] _Updated `What's New` copy (if doing a release after this PR)_
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Prior to this PR, most of the app setup was being done in
api_app.pyat import time. This PR cleans this up, by:api_app.pyfile - it just initializes theFastAPIappThe main motivation for this changes is to make it easier to support an upcoming torch configuration feature that requires more careful ordering of app initialization steps.
Related Issues / Discussions
N/A
QA Instructions
Merge Plan
Checklist
What's Newcopy (if doing a release after this PR)