Unlocking Ultra-Fast Inference with Transformers and vLLM Integration
Transformers have revolutionized the field of machine learning, establishing themselves as the reference modeling library for various architectures. With support for over 450 machine learning frameworks, the Transformers library is designed to be self-contained and straightforward, making it a favorite among developers. The recent integration of Transformers as a modeling backend for vLLM has taken this a step further, enabling ultra-fast inference without the need for extensive coding adjustments.
TL;DR
Discover how the vLLM backend has achieved speeds comparable to custom implementations, allowing model authors to effortlessly leverage Transformers for super-fast inference. Simply upgrade with the command below:
bash
pip install –upgrade vllm –torch-backend auto
The Transformers Ecosystem
The Transformers library serves as a foundational tool for understanding machine learning architectures. It empowers developers to explore model implementations, allowing them to port functionalities to various frameworks like vLLM, SGLang, and llama.cpp. This commitment to accessibility has solidified Transformers’ role within the broader machine-learning ecosystem.
The integration of Transformers into vLLM not only simplifies the use of Transformers models but also enhances performance through optimized inference techniques. This includes features like continuous batching and custom attention kernels, streamlining processes for model authors.
More Read
Head-to-Head Performance Showcase
To illustrate the efficacy of the Transformers modeling backend within vLLM, we conducted comparative tests among three different Qwen3 models:
- 4B Dense Model on a single GPU.
- 32B Dense Model utilizing tensor parallelism.
- 235B-Parameter FP8 Mixture-of-Experts Model with both data and expert parallelism on the same 8×H100 node.
The results are compelling: the Transformers modeling backend demonstrates performance that meets or exceeds native throughput parameters across all models. This advancement reinforces the benefits of leveraging established frameworks for lightning-fast inference without modifying existing setups.
Running Models Made Simple
Deploying a model using the Transformers modeling backend now requires a straightforward command line adjustment:
bash
vllm serve Qwen/Qwen3-4B –model-impl transformers
vllm serve Qwen/Qwen3-32B –model-impl transformers –tensor-parallel-size 2
vllm serve Qwen/Qwen3-235B-A22B-FP8 –model-impl transformers –data-parallel-size 8 –enable-expert-parallel
While models utilizing linear attention may not yet be supported, their future inclusion is promising.
Understanding the Measurement Methodology
Performance comparisons are made under three distinct scenarios, ensuring a comprehensive evaluation of the models:
- Native Implementation (
--model-impl vllm) – Utilizing the hand-written model as the baseline for comparison. - Post-Integration (
--model-impl transformerswith PR) – Observing performance after the inclusion of enhancements. - Pre-Integration (
--model-impl transformerswithout PR) – Evaluating effective speeds prior to updates.
For reproducibility, a detailed runner is available as a script: benchmark.sh.
The Latest Innovations
The earlier focus of the Transformers modeling backend centered around optimizing attention mechanisms for inference. However, the newly enhanced pipeline addresses various deployment dimensions that contribute to peak performance, including parallelization across GPUs, compilation, and kernel optimization.
Previously, model authors often resorted to custom implementations within vLLM to achieve the highest performance. Now, integrating a model into the Transformers library automatically allows it to be deployed through vLLM with no additional customization, offering substantial time savings.
Leveraging Runtime Analysis
The updated Transformers modeling backend for vLLM utilizes torch.fx for static analysis of the model graph, enabling dynamic optimizations. The technology identifies known patterns and employs abstract syntax tree manipulation to enhance computational efficiency. This results in significant performance improvements, allowing models that once required custom tweaks to achieve native vLLM inference speeds using the base Transformers implementation effortlessly.
Here’s what this means for practical deployment:
- Fused operations that maximize speed via vLLM-optimized kernels, especially beneficial for models requiring expert parallelization.
- Operations like
MergedColumnParallelLinearandQKVParallelLinearcontribute to optimized tensor and pipeline parallel plans, enhancing overall throughput. - The modified models remain fully compatible with
torchcompilation, effectively balancing inference with training processes.
Exciting Future Developments
The community is eagerly anticipating a more in-depth blog post dedicated to exploring optimized inference methods. This forthcoming content will delve into the mechanics of how these model manipulations achieve such impressive performance metrics, further enriching the understanding of deep learning frameworks.
As the integration between Transformers and vLLM strengthens, the potential for ultra-fast inference across machine learning applications continues to expand, offering a promising landscape for developers and researchers alike.
Inspired by: Source
