Exciting New Feature in SGLang: Native Support for NVIDIA Model Optimizer Quantization
(Updated on Dec 2)
We’re thrilled to unveil a major enhancement in SGLang: it now includes native support for NVIDIA Model Optimizer quantization! This revolutionary integration simplifies the entire workflow of model optimization and deployment, allowing developers to transition from full-precision models to high-performance, quantized endpoints entirely within the SGLang ecosystem.
The Importance of Model Quantization in AI
Serving large language models (LLMs) effectively poses one of the most significant challenges in production environments. Model quantization plays a pivotal role in this context, serving as a critical technique for shrinking the memory footprint and boosting the inference speed of models. Before this feature, users navigated multi-step workflows that involved different tools for model optimization and deployment, creating unnecessary complexity.
With our latest updates (namely PRs #7149, #9991, and #10154), we’ve streamlined this process.
Enhanced Performance with Model Optimizer Integration
The optimizations achieved through the combination of NVIDIA’s Model Optimizer and SGLang can enhance GPU throughput by up to 2x when comparing NVFP4 and FP8 inference. This is a game-changer for developers looking to maximize efficiency while maintaining performance.
More Read
What’s New: Access to Direct ModelOpt APIs
SGLang now integrates NVIDIA’s Model Optimizer directly, allowing developers to execute powerful quantization APIs within their SGLang code effortlessly. This new capability introduces a straightforward three-step workflow for maximizing performance:
-
Quantize: Utilize the new SGLang-ModelOpt interface to implement advanced quantization techniques, enabling accelerated low-precision inference in configurations like NVFP4, MXFP4, FP8, etc.
-
Export: Save the optimized model artifacts, ensuring they are fully compatible with the SGLang runtime.
- Deploy: Load the quantized model directly into the SGLang runtime and serve it on NVIDIA platforms, reaping immediate benefits such as lower latency and reduced memory usage.
Performance Outcomes You Can Expect
The models optimized via this innovative API allow for substantial performance boosts. The optimizations are compatible with other components in the NVIDIA software-hardware stack and across various implementations of the latest Blackwell architecture, from the DGX Spark to the GB300 NVL72.
The graph above illustrates the NVIDIA B200 per GPU throughput against End-to-End Latency for DeepSeek-R1-0528 across several configurations using the Model Optimizer’s NVFP4 quantized model. This version, although promising, is not yet supported in the initial API release.
According to the latest findings from InferenceMAX, performance optimizations achieved through the combination of the Model Optimizer and SGLang can yield up to 2x better per GPU throughput when contrasted with the original FP8 baseline.
Getting Started: A Simple Example Script
SGLang simplifies the process with an example script that showcases the entire Model Optimizer quantization and export workflow. Prior to running the script, ensure you’ve installed nvidia-modelopt and accelerate packages in your SGLang environment.
python
import sglang as sgl
from sglang.srt.configs.device_config import DeviceConfig
from sglang.srt.configs.load_config import LoadConfig
from sglang.srt.configs.model_config import ModelConfig
from sglang.srt.model_loader.loader import get_model_loader
model_config = ModelConfig(
model_path="Qwen/Qwen3-8B",
quantization="modelopt_fp8", # or "modelopt_fp4"
trust_remote_code=True,
)
load_config = LoadConfig(
modelopt_export_path="./quantized_qwen3_8b_fp8",
modelopt_checkpoint_save_path="./checkpoint.pth", # optional, for fake quantized checkpoint
)
device_config = DeviceConfig(device="cuda")
model_loader = get_model_loader(load_config, model_config)
quantized_model = model_loader.load_model(
model_config=model_config,
device_config=device_config,
)
After quantizing and exporting your model, deploying it with SGLang is straightforward:
bash
python -m sglang.launch_server
–model-path ./quantized_qwen3_8b_fp8
–quantization modelopt
–port 30000 –host 0.0.0.0
Alternatively, you can utilize the Python API:
python
import sglang as sgl
from transformers import AutoTokenizer
def main():
llm = sgl.Engine(
model_path="./quantized_qwen3_8b_fp8",
quantization="modelopt"
)
tokenizer = AutoTokenizer.from_pretrained("./quantized_qwen3_8b_fp8")
messages = [
[{"role": "user", "content": "Hello, how are you?"}],
[{"role": "user", "content": "What is the capital of France?"}]
]
prompts = [
tokenizer.apply_chat_template(m, tokenize=False, add_generation_prompt=True)
for m in messages
]
sampling_params = {"temperature": 0.8, "top_p": 0.95, "max_new_tokens": 512}
outputs = llm.generate(prompts, sampling_params)
for i, output in enumerate(outputs):
print(f"Prompt: {prompts[i]}")
print(f"Output: {output[‘text’]}")
if name == "main":
main()
Join the Community
We invite you to explore this exciting new feature and witness the performance gains for yourself. Please visit our GitHub repository to access the latest version and try it out.
Don’t forget to join our dedicated Slack channel #modelopt for discussions around model optimization, quantization, and low-precision numerics! If you haven’t joined our workspace yet, you can do so here.
Acknowledgements
Special thanks to the NVIDIA team: Zhiyu Cheng, Jingyu Xin, Huizi Mao, Eduardo Alvarez, Pen Chung Li, and Omri Almog, along with the SGLang team and community members: Qiaolin Yu and Xinyuan Tong.
Inspired by: Source
