View a PDF of the paper titled ParallelComp: Parallel Long-Context Compressor for Length Extrapolation, by Jing Xiong and 9 other authors
View PDF
Abstract: Extrapolating ultra-long contexts (text length >128K) remains a major challenge for large language models (LLMs), as most training-free extrapolation methods are not only severely limited by memory bottlenecks but also suffer from the attention sink, which restricts their scalability and effectiveness in practice. In this work, we propose ParallelComp, a parallel long-context compression method that effectively overcomes the memory bottleneck, enabling 8B-parameter LLMs to extrapolate from 8K to 128K tokens on a single A100 80GB GPU in a training-free setting. ParallelComp splits the input into chunks, dynamically evicting redundant chunks and irrelevant tokens, supported by a parallel KV cache eviction mechanism. Importantly, we present a systematic theoretical and empirical analysis of attention biases in parallel attention—including the attention sink, recency bias, and middle bias—and reveal that these biases exhibit distinctive patterns under ultra-long context settings. We further design a KV cache eviction technique to mitigate this phenomenon. Experimental results show that ParallelComp enables an 8B model (trained on 8K context) to achieve 91.17% of GPT-4’s performance under ultra-long contexts, outperforming closed-source models such as Claude-2 and Kimi-Chat. We achieve a 1.76x improvement in chunk throughput, thereby achieving a 23.50x acceleration in the prefill stage with negligible performance loss and pave the way for scalable and robust ultra-long contexts extrapolation in LLMs. We release the code at this https URL.
Submission History
From: Jing Xiong [view email]
[v1] Thu, 20 Feb 2025 07:10:43 UTC (2,133 KB)
[v2] Mon, 9 Jun 2025 09:48:43 UTC (2,003 KB)
Understanding the Challenge of Long Contexts in Large Language Models
The emergence of Large Language Models (LLMs) has reshaped how we interact with artificial intelligence, enabling revolutionary advancements in natural language processing. However, a significant hurdle remains: extrapolating ultra-long contexts—those exceeding 128,000 tokens—presents a formidable challenge. Traditional methods often succumb to memory constraints, limiting their potential.
The Essence of ParallelComp
Enter ParallelComp, a groundbreaking parallel long-context compression method designed to overcome these limitations. It not only enhances scalability but also sidesteps the memory bottlenecks that plague conventional extrapolation strategies. With ParallelComp, researchers can now extrapolate from an 8,000-token context to an astonishing 128,000 tokens on a single A100 80GB GPU, all without the need for extensive retraining.
The Innovation in Chunk Management
One of the key innovations of ParallelComp lies in its chunk management system. By dynamically splitting the input into manageable chunks, the method effectively evicts redundant data and irrelevant tokens. This process is underpinned by a sophisticated parallel KV cache eviction mechanism, which ensures that only the most pertinent information is retained, thereby streamlining the overall analysis and processing.
Unpacking Attention Biases
ParallelComp also tackles a previously underexplored issue: the attention biases inherent in long-context settings. These biases—such as recency bias and the attention sink—can skew the processing of information, leading to inefficiencies and decreased performance. Through systematic theoretical and empirical analysis, the researchers reveal clear patterns in these biases, paving the way for more effective mitigation strategies.
Empirical Success in Performance
The empirical findings of ParallelComp are nothing short of impressive. An 8B parameter model, initially trained on an 8K context, demonstrated an astonishing 91.17% performance of OpenAI’s GPT-4 when dealing with ultra-long contexts. Furthermore, ParallelComp outperforms other leading models, including the closed-source Claude-2 and Kimi-Chat. With a 1.76x improvement in chunk throughput, the method achieves an exceptional 23.50x acceleration in the prefill stage without compromising on performance.
More Read
Enhancing Scalability in LLMs
In a field where scalability is crucial, ParallelComp offers a pathway to more robust ultra-long context extrapolation. The ability to handle increased text lengths seamlessly opens doors for broader applications of LLMs, allowing developers and researchers to push the boundaries of what these models can achieve.
Code Accessibility for Further Research
To foster further innovation and exploration, the authors are committed to making their work accessible. The release of code associated with the ParallelComp methodology encourages other researchers to build upon and refine the technology. This open-access approach is pivotal for driving advancements in the field of artificial intelligence.
Conclusion
With tools like ParallelComp, the future of language models is brighter than ever. These innovations not only address existing challenges but also set the stage for future breakthroughs in natural language processing.
Inspired by: Source
