Understanding Agentic AI: Insights from Aaron Erickson at QCon AI NYC 2025

At QCon AI NYC 2025, Aaron Erickson delivered an intriguing presentation that reframed the conversation around agentic AI. He emphasized that agentic AI is fundamentally an engineering challenge, diverting attention away from mere prompt crafting. The core of his argument focused on how reliability can be achieved by blending probabilistic elements with deterministic boundaries.

Agentic AI as an Operational Layer

Erickson’s perspective posited that agentic AI should function as an added layer over existing operational systems, rather than seeking to replace them. This layered approach allows the model to handle complex tasks such as interpreting queries, retrieving relevant evidence, classifying scenarios, and suggesting actions. Meanwhile, deterministic systems take on the responsibility of executing these actions, imposing constraints, and tracking performance metrics. This interplay creates a feedback loop that is essential for evaluating the overall system’s effectiveness.

Challenges of Query Generation

A critical issue Erickson highlighted concerns the common pitfalls in translating natural language to SQL queries. He pointed out that while initial demonstrations may succeed with simple questions and small schemas, performance drastically declines when dealing with complex schemas containing multiple joins and edge cases. To counteract this, he advised reducing degrees of freedom by flattening schemas and constraining query forms. He emphasized that expressiveness needs to be balanced with the costs associated with deeper evaluations and additional safeguards.

Classification vs. Code Generation: A Pragmatic Distinction

In discussing classification and code generation, Erickson made an important distinction: models are effective for selecting from a small set of predefined options but struggle when tasked with generating arbitrary programs over vast search spaces. This discrepancy can be seen as a design leverage point. By first classifying an intent, the system can then route queries to deterministic templates or constrained tool calls, improving reliability.

More Read

Exploring Attentional Image Classification: Are 256 Superpixels Worth 16×16 Pixels in Image Analysis? [2605.27144]

Google Unveils Gemini 3: Key Features and Insights on InfoQ

Mastering Search Techniques for the Traveling Salesperson Problem: A Comprehensive Guide

Enhanced Single Cell Representation Learning: A Variational Framework Approach

Enhancing Ontology Versioning Through Effective Ontology Matching Techniques

The ‘Paradox of Choice’ in Tool Selection

Erickson illustrated the reliability challenges surrounding tool selection by referencing a "cheesecake menu." When confronted with too many similar options, an agent may select suboptimal or unsafe tools, thereby compromising system efficiency. This highlights the engineering necessity for differentiated, well-described tool catalogs. Properly constraining tools enhances decision quality, enabling the agent to avoid becoming overwhelmed by choices.

The Importance of Role Specialization

Erickson elaborated on why role specialization is crucial in agentic systems. While a general-purpose agent might be sufficient for routing and summarization, the accuracy of such a system depends on specialized components designed for specific tasks. He advocated for a structured layering approach, where a managing layer oversees delegation, but does not contain the domain logic. This emphasis on specialization allows for more reliable interactions with underlying systems.

Taxonomy of Agent Behaviors

Building on the concept of specialization, Erickson presented a taxonomy of agent behaviors. An eye-catching example was his "Worker Agent" illustration. This agent was depicted as examining clusters and flagging those worth attention, demonstrating how agents can be employed across numerous records, conducting repetitive analyses while storing structured outputs for future reference.

Managing Complexity in Growing Systems

As systems evolve, complexity grows, which can challenge reliability. To combat this, Erickson proposed several additional roles. A tool selection agent can minimize ambiguity when multiple pathways exist to achieve a goal. An observer agent can monitor interactions and alert for unsafe communication patterns or policy violations. Furthermore, a director agent can manage task delegation and track progress towards defined outcomes. This layered approach mirrors classic testing recommendations, aiming to manage risk by embedding confidence into less expensive tests.

The Role of Deterministic Anchors

Erickson used an operational analogy to establish the necessity for deterministic anchors within agentic systems. He questioned whether routine operations need to be reinvented for every task and concluded that they do not; instead, operators should rely on deterministic runbooks. He suggested that agentic systems should emulate this behavior, incorporating repeatability into their processes and allowing agents to decide when established procedures are applicable.

Balancing Certainty and Discovery

Finally, Erickson touched on the critical balance between certainty and discovery in agentic systems. Discovery involves agents exploring possibilities and identifying anomalies, while certainty relates to deterministic tools that conduct bounded operations. The engineering boundary between these realms is where platform engineering thrives—emphasizing the importance of authentication, authorization, auditing, telemetry, and safe degradation.

For those eager to delve deeper into Erickson’s insights, the full video of his talk will be available starting January 15.

Inspired by: Source