SGLang Joins the PyTorch Ecosystem: A Game-Changer for Serving Large Language Models

We are excited to announce that the SGLang project has officially been integrated into the PyTorch ecosystem! This integration signifies a commitment to aligning with PyTorch’s high standards and practices, ensuring that developers have access to a reliable and community-supported framework for the rapid and flexible serving of large language models (LLMs).

About SGLang

SGLang is designed as a fast-serving engine for large-scale language and vision language models. By co-designing the backend runtime and frontend language, it enhances model interactions, making them both faster and more controllable.

Core Features of SGLang

1. Fast Backend Runtime: SGLang incorporates cutting-edge features such as RadixAttention for prefix caching, zero-overhead CPU scheduling, and a variety of advanced techniques like token attention (paged attention), speculative decoding, tensor parallelism, chunked prefill, structured outputs, and quantization options (FP8/INT4/AWQ/GPTQ). These features combine to provide unparalleled serving efficiency.
2. Flexible Frontend Language: The intuitive interface allows developers to create sophisticated LLM applications effortlessly. Key capabilities include chained generation calls, advanced prompting, control flow, multi-modal inputs, parallelism, and external interactions.
3. Extensive Model Support: SGLang supports a diverse array of generative models such as Llama, Gemma, Mistral, Qwen, DeepSeek, and LLaVA. It also accommodates embedding models (like e5-mistral and gte) and reward models (like Skywork), with an easy extensibility framework for integrating new models.
4. Active Community: As an open-source project, SGLang benefits from a vibrant community of contributors and industry users, which helps drive continuous improvements and innovation.

SGLang has gained recognition for its speed and efficiency, often outperforming other leading frameworks regarding serving throughput and latency. For a deeper dive into its underlying technologies, you can explore the release blog posts for versions v0.2, v0.3, and v0.4.

Adoption in Industry

Leading tech companies and research institutions have embraced SGLang due to its robust performance. For instance, xAI utilizes SGLang to serve its flagship model, Grok 3, currently topping the Chatbot Arena leaderboard. Similarly, Microsoft Azure employs SGLang to serve DeepSeek R1 on AMD GPUs, also recognized as a leading open-source model.

Serving DeepSeek Models

Deploying a DeepSeek model using SGLang is straightforward. You can launch a Docker container with the following command:

More Read

Reflecting on the Past and Anticipating the Future

Judge Arena: Evaluating LLM Performance Through Benchmarking

Understanding Digital Object Identifiers (DOIs) for Datasets and Models: A Comprehensive Guide

Unlocking Agentic AI: Join the AWS & NVIDIA Hackathon to Shape the Future of Intelligent Agents

Explore the Measuring Intelligence Summit at the PyTorch Conference

# Pull the latest image
docker pull lmsysorg/sglang:latest

# Launch a server
docker run --gpus all --shm-size 32g -p 30000:30000 -v ~/.cache/huggingface:/root/.cache/huggingface --ipc=host --network=host --privileged lmsysorg/sglang:latest 
    python3 -m sglang.launch_server --model deepseek-ai/DeepSeek-V3 --tp 8 --trust-remote-code --port 30000

Once your server is running, you can query it using the OpenAI-compatible API:

import openai
client = openai.Client(base_url=f"http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

This command is optimized for 8xH200 configurations. For details on other hardware setups (MI300X, H100, A100, H20, L40S), refer to the DeepSeek documentation.

Optimizations for DeepSeek

SGLang integrates specific optimizations tailored for DeepSeek models, including MLA throughput optimizations, MLA-optimized kernels, data-parallel attention, multi-token prediction, and DeepGemm. These enhancements establish SGLang as the preferred choice for numerous companies, including AMD, NVIDIA, and various cloud service providers. The team is also committed to introducing further optimizations in line with their 2025 H1 roadmap.

Serving Llama Models

Launching a server for a Llama 3.1 text model can be done effortlessly with the following command:

python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct

For a multimodal Llama 3.2 model, the command adjusts slightly:

python3 -m sglang.launch_server --model-path meta-llama/Llama-3.2-11B-Vision-Instruct  --chat-template=llama_3_vision

Roadmap for Future Development

The SGLang team is dedicated to pushing the boundaries of system efficiency. The roadmap for the first half of 2025 focuses on:

• Throughput-oriented large-scale deployments akin to the DeepSeek inference system
• Long context optimizations
• Low-latency speculative decoding
• Integration of a reinforcement learning training framework
• Kernel optimizations

SGLang has already made significant strides in large-scale production, generating trillions of tokens daily. With an active community of over three hundred contributors on GitHub, it enjoys robust support from numerous institutions, including AMD, Atlas Cloud, Baseten, LinkedIn, and many more.

Get Involved!

We invite you to dive into the SGLang GitHub repository, connect with the community on Slack, and reach out to contact@sglang.ai for any inquiries or collaboration opportunities. By working together, we can democratize access to powerful AI models, making them available for everyone.

Inspired by: Source