[Inference Performance] Enable vLLM support

[PatchouliTIS]

I have adapted FireredASR-LLM based on the community vLLM 0.10.1.

Testing on a local single-GPU H20 environment shows the QPM reaching approximately 1200 with a batch size of 16. I would appreciate your feedback on these results.

Additionally, is it possible to modify the FireredASR-LLM structure to better fit vLLM's loading workflow?

[kaituoxu]

Great work!

And how "modify the FireredASR-LLM structure to better fit vLLM's loading workflow"?

[PatchouliTIS]

Great work!

And how "modify the FireredASR-LLM structure to better fit vLLM's loading workflow"?

The weights loading procedure in vLLM is done by a common class AutoWeightsLoader , which only takes .pt or safetensors as its input for model weights.

In FireredASR-LLM hf repo it shows a complicate structure .pth.tar containing LoRA parts of Qwen2-7B, which is loaded separately from Qwen2-7B LLM itself. If possible, please provide the model weights in safetensor format, including the encoder component, projector component, and the Qwen2-7B model component with integrated LoRA weights.

[PatchouliTIS]

Hi there, I have noticed that the new model FireRedASR2 has been released and it has been supported in vLLM already. May I ask if the current FireRedASR will continue to receive updates? Or will future development focus on FireRedASR2, with the current version being archived? @kaituoxu

[kaituoxu]

will future development focus on FireRedASR2
thx :)