Transforming Root Cause Analysis with LLMs: The Pipeline Challenge

In the ever-evolving landscape of observability engineering, a paradigm shift is quietly taking place. Engineers increasingly believe that the reasoning capabilities of large language models (LLMs) are no longer the primary bottleneck in AI-assisted root cause analysis (RCA). Instead, the real challenge lies in the data pipeline—essentially, the framework that determines which data is presented to the model.

The Importance of Context Preparation

For tech teams integrating LLMs into their incident response strategies, a key takeaway emerges: investing effort into preparing contextual data might yield greater dividends than merely opting for larger or more sophisticated models. By refining the information fed into these models, teams can enhance the accuracy and reliability of RCA outcomes.

Two Distinct Approaches to RCA

When delving into AI-driven RCA, two primary methodologies come into play: agent-based designs and deterministic designs. Agent-based systems empower the model with investigative autonomy, allowing it to select relevant telemetry as it processes information. In contrast, deterministic systems pre-process signals and provide the model with a singular, curated context.

Coroot’s innovative work exemplifies the shifting trend towards determinant designs. Similar to Dynatrace’s Davis AI, which utilizes topology-based causal analysis, Coroot’s approach leverages real-time dependency maps to identify root causes. By constraining the model’s input to a defined context, practices like these facilitate clearer diagnostics where errors can be directly traced back to either model limitations or inadequate evidence supplied to it.

Separating Reasoning from Data Quality

Recent research conducted by Nikolay Sivko from Coroot aims to disentangle the two pivotal aspects of RCA involving LLMs: the reasoning that occurs with the data at hand and the mechanism that dictates what reaches the model. Sivko argues that the focus shouldn’t solely be on whether AI can effectively perform RCA but rather on assessing and optimizing these dual functions independently.

Testing the Framework

In his experiments, Sivko constructed scenarios involving a Chaos Mesh NetworkChaos setup, deliberately injecting artificial delays that caused erratic query behavior. This included misleading signals designed to test the robustness of the RCA. He evaluated eleven different LLMs using an extensive prompt, totaling around 9,800 tokens, to analyze the root cause, causal chains, and suggested fixes.

Notably, frontier models like Claude Opus 4.8, GPT-5.5, and Gemini 3.1 Pro successfully identified the underlying issues of the chaos experiment, emphasizing that clean context can significantly influence model effectiveness. While some larger, open-weight models performed satisfactorily, their limitations indicated a qualitative difference in RCA outcomes based on how data was curated.

Agent-Based vs. Deterministic Models

The debate surrounding agent-based versus deterministic approaches continues to be a hot topic. Agent-based models indeed offer a distinct advantage by allowing models to autonomously gather real-time data, which can be crucial for novel incidents outside previously established correlation sets. However, this flexibility often comes with operational intricacies that can complicate debugging in a production environment.

Insights from platforms like ZenML and Incident.io reveal that employing multi-agent LLM investigations can be remarkably challenging. Failed runs don’t yield clear stack traces but can lead to unpredictable interactions and coordination issues among agents. Such difficulties have prompted many practitioners to transition towards primarily deterministic workflows. This shift favors systems with tighter LLM integrations for enhanced reliability and reduced token costs, highlighting the value of repeatability and easier evaluation over the flexibility offered by agent-based strategies.

The Cost Perspective

From a cost-efficiency standpoint, Sivko notes that even short LLM calls are economical, running just a few cents as the correlation work is conducted prior. He posits that the reasoning aspect of AI-driven RCA is effectively resolved, pushing future engineering efforts toward refining the context preparation aspect. This reflects a broader trend towards context engineering—crafting the right, compact dataset to achieve reliable LLM-driven reasoning and observability.

The Growing Discipline of Context Engineering

As the industry increasingly aligns around the concept of context engineering, guidance from organizations like Anthropic and LangChain, along with observability vendors like Mezmo, emphasizes the necessity of curating high-signal, succinct contexts. This focus ensures that practitioners can unlock the full potential of LLMs in RCA tasks, transforming the paradigm of how root causes are identified and addressed.