Add pytorch_cuda_alloc_conf config to tune VRAM memory allocation

[RyanJDick]

Summary

This PR adds a pytorch_cuda_alloc_conf config flag to control the torch memory allocator behavior.

• pytorch_cuda_alloc_conf defaults to None, preserving the current behavior.
• The configuration options are explained here: https://pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf. Tuning this configuration can reduce peak reserved VRAM and improve performance.
• Setting pytorch_cuda_alloc_conf: "backend:cudaMallocAsync" in invokeai.yaml is expected to work well on many systems. This is a good first step for those looking to tune this config. (We may make this the default in the future.)
• The optimal configuration seems to be dependent on a number of factors such as device version, VRAM, CUDA kernel version, etc. For now, users will have to experiment with this config to see if it hurts or helps on their systems. In most cases, I expect it to help.

Memory Tests

VAE decode memory usage comparison:

- SDXL, fp16, 1024x1024:
  - `cudaMallocAsync`: allocated=2593 MB, reserved=3200 MB
  - `native`:          allocated=2595 MB, reserved=4418 MB

- SDXL, fp32, 1024x1024:
  - `cudaMallocAsync`: allocated=3982 MB, reserved=5536 MB
  - `native`:          allocated=3982 MB, reserved=7276 MB

- SDXL, fp32, 1536x1536:
  - `cudaMallocAsync`: allocated=8643 MB, reserved=12032 MB
  - `native`:          allocated=8643 MB, reserved=15900 MB

Related Issues / Discussions

N/A

QA Instructions

• Performance tests with pytorch_cuda_alloc_conf unset.
• Performance tests with pytorch_cuda_alloc_conf: "backend:cudaMallocAsync".

Merge Plan

• Merge Tidy app entrypoint #7668 first and change target branch to main

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable)
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

[hipsterusername]

As confirmation, i presume this does not play nicely on AMD?

[RyanJDick]

As confirmation, i presume this does not play nicely on AMD?

I haven't tested on AMD, but I would not expect the recommended config of backend:cudaMallocAsync to work on AMD. That being said, the native allocator configs documented here might work with AMD (don't have a way to test and couldn't find it documented clearly anywhere). We'd need someone to test whether they do and experiment to find a good recommendation.