[Perf] Speed up LoRA Batch Initialization

[lifuhuang]

Motivation

prepare_lora_batch is triggered once per forward pass and is one of the main sources of perf overheads from LoRA. Based on suggestion from @Fridge003 , there are some low-hanging fruits for perf optimization such as eliminating unnecessary cuda device syncs.

• Eliminate unnecessary H2D transfer in set_lora_info (Speed up set_lora_info by eliminating unnecessary H2D transfers #6960)
• Eliminate cuda stream syncs and redundant compute in LoRAManager ([Perf] Refactor LoRAManager to eliminate stream syncs and redundant computations  #6994)
• Experiment torch.compile / cuda grpah for prepare_lora_batch to reduce gaps between kernels (Idea from @hebiao064 , to be verified)

Related resources

No response