Understanding Agentic AI: Insights from Aaron Erickson at QCon AI NYC 2025
At QCon AI NYC 2025, Aaron Erickson delivered an intriguing presentation that reframed the conversation around agentic AI. He emphasized that agentic AI is fundamentally an engineering challenge, diverting attention away from mere prompt crafting. The core of his argument focused on how reliability can be achieved by blending probabilistic elements with deterministic boundaries.
- Agentic AI as an Operational Layer
- Challenges of Query Generation
- Classification vs. Code Generation: A Pragmatic Distinction
- The ‘Paradox of Choice’ in Tool Selection
- The Importance of Role Specialization
- Taxonomy of Agent Behaviors
- Managing Complexity in Growing Systems
- The Role of Deterministic Anchors
- Balancing Certainty and Discovery
Agentic AI as an Operational Layer
Erickson’s perspective posited that agentic AI should function as an added layer over existing operational systems, rather than seeking to replace them. This layered approach allows the model to handle complex tasks such as interpreting queries, retrieving relevant evidence, classifying scenarios, and suggesting actions. Meanwhile, deterministic systems take on the responsibility of executing these actions, imposing constraints, and tracking performance metrics. This interplay creates a feedback loop that is essential for evaluating the overall system’s effectiveness.
Challenges of Query Generation
A critical issue Erickson highlighted concerns the common pitfalls in translating natural language to SQL queries. He pointed out that while initial demonstrations may succeed with simple questions and small schemas, performance drastically declines when dealing with complex schemas containing multiple joins and edge cases. To counteract this, he advised reducing degrees of freedom by flattening schemas and constraining query forms. He emphasized that expressiveness needs to be balanced with the costs associated with deeper evaluations and additional safeguards.
Classification vs. Code Generation: A Pragmatic Distinction
In discussing classification and code generation, Erickson made an important distinction: models are effective for selecting from a small set of predefined options but struggle when tasked with generating arbitrary programs over vast search spaces. This discrepancy can be seen as a design leverage point. By first classifying an intent, the system can then route queries to deterministic templates or constrained tool calls, improving reliability.
More Read
The ‘Paradox of Choice’ in Tool Selection
Erickson illustrated the reliability challenges surrounding tool selection by referencing a "cheesecake menu." When confronted with too many similar options, an agent may select suboptimal or unsafe tools, thereby compromising system efficiency. This highlights the engineering necessity for differentiated, well-described tool catalogs. Properly constraining tools enhances decision quality, enabling the agent to avoid becoming overwhelmed by choices.
The Importance of Role Specialization
Erickson elaborated on why role specialization is crucial in agentic systems. While a general-purpose agent might be sufficient for routing and summarization, the accuracy of such a system depends on specialized components designed for specific tasks. He advocated for a structured layering approach, where a managing layer oversees delegation, but does not contain the domain logic. This emphasis on specialization allows for more reliable interactions with underlying systems.
Taxonomy of Agent Behaviors
Building on the concept of specialization, Erickson presented a taxonomy of agent behaviors. An eye-catching example was his "Worker Agent" illustration. This agent was depicted as examining clusters and flagging those worth attention, demonstrating how agents can be employed across numerous records, conducting repetitive analyses while storing structured outputs for future reference.
Managing Complexity in Growing Systems
As systems evolve, complexity grows, which can challenge reliability. To combat this, Erickson proposed several additional roles. A tool selection agent can minimize ambiguity when multiple pathways exist to achieve a goal. An observer agent can monitor interactions and alert for unsafe communication patterns or policy violations. Furthermore, a director agent can manage task delegation and track progress towards defined outcomes. This layered approach mirrors classic testing recommendations, aiming to manage risk by embedding confidence into less expensive tests.
The Role of Deterministic Anchors
Erickson used an operational analogy to establish the necessity for deterministic anchors within agentic systems. He questioned whether routine operations need to be reinvented for every task and concluded that they do not; instead, operators should rely on deterministic runbooks. He suggested that agentic systems should emulate this behavior, incorporating repeatability into their processes and allowing agents to decide when established procedures are applicable.
Balancing Certainty and Discovery
Finally, Erickson touched on the critical balance between certainty and discovery in agentic systems. Discovery involves agents exploring possibilities and identifying anomalies, while certainty relates to deterministic tools that conduct bounded operations. The engineering boundary between these realms is where platform engineering thrives—emphasizing the importance of authentication, authorization, auditing, telemetry, and safe degradation.
For those eager to delve deeper into Erickson’s insights, the full video of his talk will be available starting January 15.
Inspired by: Source
