[Feature] Phi-4-MM support

[lifuhuang]

Update

Currently the basic text + vision support is already in main. However, there are many known issues across the board.

Known limitations: (See Execution Plan before for full list):

1. Audio capabilities: currently we do not support audio at all.
2. LoRA / Image quality: Phi4MM depends on LoRA for full image capability, but there is some compatibility issues with the native SGL LORA solution. We are working on solving it by refactoring / generalizing SGL LoRA capabilities. Fixed with Refactor LoRA handling to support adapter tensors in fused format #6585, Fix incorrect LoRA weight loading for fused gate_up_proj #6734, Support LoRA in TestOpenAIVisionServer and fix fused kv_proj loading bug. #6861)
3. Token: Phi4MM supports two types of image token conventions (<|image1|> and <|endoftext10|>), currently we only support the latter. If you use the default chat template, it will automatically pick up the supported one.

Motivation

Supporting the Phi4 Multimodal model (https://huggingface.co/microsoft/Phi-4-multimodal-instruct in SGL.

Execution Plan:

• Basic text + image support (@lifuhuang Support Phi-4 Multi-Modal (text + vision only) #6494)
• LoRA support (required for full image understanding capability): (@lifuhuang Refactor LoRA handling to support adapter tensors in fused format #6585, Fix incorrect LoRA weight loading for fused gate_up_proj #6734, Support LoRA in TestOpenAIVisionServer and fix fused kv_proj loading bug. #6861)
• perf optimization
• Precomputed feature support.
• Refactor SGL MM processor logic support for support the original token variable image token (e.g., <image_1>)
• SGLang LoRA compatibility with Radix Attention
• audio support

Related resources

No response