Cloudflare Enhances AI Infrastructure for Large Language Models

Introduction to Cloudflare’s AI Infrastructure

Cloudflare has recently made headlines with its innovative approach to running large AI language models (LLMs) across its global network. As the demand for AI-driven solutions grows, the challenges associated with processing substantial volumes of text and handling expensive hardware become increasingly pronounced. Understanding this landscape, Cloudflare’s latest infrastructure innovations focus on optimizing efficiency and performance in LLM operations.

Optimized Processing with Disaggregated Prefill

One of the key enhancements from Cloudflare is the introduction of disaggregated prefill processing. This method breaks down the processing of LLM requests into two discrete stages, effectively utilizing separate machines for each. In the first stage, known as prefill, the system reads and prepares the input text. The second stage, called decode, is responsible for generating the output—a crucial distinction because these two processes have different resource needs.

According to Cloudflare representatives Michelle Chen, Kevin Flansburg, and Vlad Krasnov, the prefill stage is typically compute-bound, while the decode stage is memory-bound. This split allows for greater specialization and efficiency in resource usage, ensuring that the strengths of different hardware configurations are maximized.

An Insight into Prefill and Decode

In their technical breakdown, Cloudflare emphasizes:

“One hardware configuration that we use to improve performance and efficiency is disaggregated prefill… Prefill processes the input tokens and populates the KV cache, while decode generates output tokens.”

This strategic decision illustrates Cloudflare’s dedication to refining the mechanics of LLM processing, ultimately leading to faster and more reliable outputs.

More Read

Wild Refitting Techniques for Enhanced Black Box Prediction: A Comprehensive Study [2506.21460]

Optimizing Gradient-Based Dictionaries for Learning Dynamical Systems from Data: Insights from Paper 2411.04775

Exploring Semantic and Syntactic Relationships through Geometric Rotation Techniques

Evaluating Language Model Compliance with User Privacy Preferences

Challenges in Aligning Large Language Models with Asian Public Opinion

Introducing Infire: The Custom AI Inference Engine

To further enhance how LLMs operate, Cloudflare developed a custom AI inference engine known as Infire. Launched during Cloudflare Birthday Week 2025, this engine is designed to run large models across multiple GPUs efficiently. Infire accomplishes this by optimizing resource use, significantly reducing memory usage, and decreasing the startup time for models, which culminates in swifter response times for end-users.

The Complexity of Large Language Models

Operating large language models like Kimi K2.5—boasting over 1 trillion parameters and around 560GB—requires intricate hardware support. For instance, loading the model into memory alone demands a minimum of eight H100 GPUs. The need for additional memory during processing only compounds this requirement.

Cloudflare’s tech team details:

“For pipeline parallelism, Infire attempts to properly load balance all stages of the pipeline… On the other hand, for tensor parallelism, Infire optimizes for reducing cross-GPU communication.”

This double approach—leveraging both pipeline and tensor parallelism—strikes an optimal balance between throughput and latency, a critical factor in delivering real-time AI responses.

Efficient Resource Usage and Model Operation

In a bid to ensure efficiency, Cloudflare further optimized Infire to manage GPU memory use during internal processing. This advancement enables it to handle Llama 4 Scout on just two H200 GPUs or Kimi K2.5 on eight H100 GPUs while still reserving necessary memory for the KV cache.

The Unweight System for Improved Model Efficiency

Alongside Infire, Cloudflare introduced another innovative system: Unweight. This groundbreaking technology compresses the weights of large language models by approximately 15–22%. By reducing the data that GPUs need to load and move during inference, Unweight streamlines operations, ensuring models run at an ideal pace without sacrificing accuracy.

Industry Insights on AI Infrastructure Challenges

While Cloudflare pushes the envelope in AI infrastructure, it’s important to note that challenges persist across the industry. A recent report from Cockroach Labs underscores that many organizations struggle with inadequate infrastructure as they scale their AI systems for everyday use. The report states:

“Legacy infrastructure… simply wasn’t designed for this kind of pressure. To handle the pace and unpredictability of AI, companies need more than performance upgrades; they need a fundamental shift in how systems are architected.”

This acknowledgment from Cockroach Labs resonates with the ongoing developments at Cloudflare, reinforcing the need for adaptable solutions. As the AI landscape evolves, innovative infrastructure becomes paramount for companies aiming to stay ahead.

Conclusion

Cloudflare’s dedication to pioneering AI infrastructure through enhancements like disaggregated prefill and the Infire inference engine showcases its commitment to optimizing large language model operations. By addressing both hardware configurations and software efficiencies, Cloudflare is setting a new standard for LLM performance and reliability.

Inspired by: Source