<p>
View a PDF of the paper titled <strong>StreamingThinker: Large Language Models Can Think While Reading</strong>, by Junlong Tong and 4 other authors.
</p>
<a href="URL_TO_PDF" target="_blank">View PDF</a>
<html>(experimental)</html>
<blockquote class="abstract mathjax">
<span class="descriptor">Abstract:</span> Large language models (LLMs) have demonstrated remarkable capabilities in chain of thought (CoT) reasoning. However, the current LLM reasoning paradigm initiates thinking only after the entire input is available, which introduces unnecessary latency and weakens attention to earlier information in dynamic scenarios. Inspired by human cognition of thinking while reading, we first design a <strong>streaming thinking</strong> paradigm for LLMs, where reasoning unfolds in the order of input and further adjusts its depth once reading is complete. We instantiate this paradigm with <strong>StreamingThinker</strong>, a framework that enables LLMs to think while reading through the integration of streaming CoT generation, streaming-constraint training, and streaming parallel inference. Specifically, StreamingThinker employs streaming reasoning units with quality control for CoT generation, enforces order-preserving reasoning through streaming attention masks and position encoding, and leverages parallel KV caches that decouple input encoding from reasoning generation, thereby ensuring alignment and enabling true concurrency. We evaluate StreamingThinker on the Qwen3 model family across math reasoning, logical reasoning, and context-based QA reasoning tasks. Experimental results show that StreamingThinker preserves performance comparable to batch thinking, while yielding an 80% reduction in token waiting before the onset of reasoning and a more than 60% reduction in time-level latency for producing the final answer, demonstrating the effectiveness of the streaming paradigm for LLM reasoning. Code is publicly available at this <a href="https://example.com/code" target="_blank">URL</a>.
</blockquote>Submission History
From: Junlong Tong [view email]
[v1] Mon, 20 Oct 2025 07:27:37 UTC (1,078 KB)
[v2] Tue, 9 Dec 2025 17:34:02 UTC (1,086 KB)
[v3] Thu, 19 Mar 2026 08:17:22 UTC (1,085 KB)
Understanding the StreamingThinker Paradigm
Large Language Models (LLMs) have revolutionized the field of artificial intelligence, showcasing their capacity for complex reasoning and decision-making. A noteworthy advancement in this arena is the introduction of the StreamingThinker paradigm, which allows these models to initiate reasoning as data streams in, rather than waiting for the complete input to be processed. This innovative approach mimics human cognitive processes, enhancing performance in dynamic contexts where timing and responsiveness are crucial.
The Need for Streaming Thinking
Traditionally, LLMs engage in reasoning only after receiving the entirety of the input. This batch processing model can introduce latency, significantly hindering efficiency, especially in applications requiring real-time interactions. The StreamingThinker addresses these limitations by enabling LLMs to think while reading. This advancement not only reduces the token waiting period—leading to faster response times—but also promotes improved attentiveness to input data as it is received.
Features and Functionality of StreamingThinker
StreamingThinker is a meticulously designed framework that incorporates several key elements to facilitate its groundbreaking streaming reasoning:
-
Streaming CoT Generation: This feature allows LLMs to generate chain-of-thought reasoning in a flowing manner. As the model reads each part of the input sequentially, it begins to reason, rather than delaying until the entire input is available.
-
Streaming-Constraint Training: This training method ensures that the model learns to process information in a streamlined fashion, preserving the order of reasoning and enabling the model to adapt its depth of thinking based on newly received data.
-
Streaming Parallel Inference: The framework incorporates parallel knowledge vector (KV) caches to decouple input encoding from reasoning tasks. This not only optimizes performance but also introduces true concurrency in processing—an essential requirement for applications that demand quick turnaround times.
Evaluation and Results
To assess the effectiveness of the StreamingThinker, extensive evaluations were carried out using the Qwen3 model family. The framework was put to the test across various reasoning tasks, including mathematical reasoning, logical reasoning, and context-based question answering. The results were striking:
-
Performance Retention: Despite the shift to streaming reasoning, the StreamingThinker maintained performance levels comparable to traditional batch processing methods.
-
Significant Latency Reduction: Notably, the architecture achieved an impressive 80% reduction in token waiting times before reasoning commenced, alongside a 60% reduction in latency for producing final answers. These metrics underscore the powerful efficiency enhancements offered by the StreamingThinker.
Real-World Applications and Implications
The implications of the StreamingThinker paradigm are vast. In fields such as customer service, real-time translation, and interactive AI applications, the ability to process information and respond with minimal latency can significantly improve user experience and operational efficiency. The insights gained from this model not only pave the way for future enhancements in language models but also stimulate further exploration of human-like reasoning processes in artificial intelligence.
More Read
As researchers and developers continue to refine LLMs, it’s clear that paradigms like StreamingThinker will play a critical role in the evolution of these technologies, unlocking new potentials and applications that leverage the incredible capabilities of machine learning and artificial intelligence.
For those interested in exploring the code behind StreamingThinker, it is publicly available for access and experimentation, encouraging collaboration and innovation within the AI community.
Inspired by: Source
