Submitted on 8 Oct 2025 (v1), last revised 13 Jan 2026 (this version, v4)
In the ever-evolving landscape of artificial intelligence, the integration of natural language processing with database management systems has paved the way for transformative tools in various business applications. A significant development in this domain is the creation of a benchmark titled **Agent Bain vs. Agent McKinsey: A New Text-to-SQL Benchmark for the Business Domain**, authored by Yue Li and six co-authors. This benchmark, known as **CORGI**, aims to refine the way AI understands and processes complex business queries.
Abstract: Text-to-SQL benchmarks have traditionally only tested simple data access as a translation task of natural language to SQL queries. But in reality, users tend to ask diverse questions that require more complex responses, including data-driven predictions or recommendations. Using the business domain as a motivating example, we introduce CORGI, a new benchmark that expands text-to-SQL to reflect practical database queries encountered by end users. CORGI is composed of synthetic databases inspired by enterprises such as DoorDash, Airbnb, and Lululemon. It provides questions across four increasingly complicated categories of business queries: descriptive, explanatory, predictive, and recommendational. This challenge calls for causal reasoning, temporal forecasting, and strategic recommendation, reflecting multi-level and multi-step agentic intelligence. We find that LLM performance degrades on higher-level questions as question complexity increases. CORGI also introduces and encourages the text-to-SQL community to consider new automatic methods for evaluating open-ended, qualitative responses in data access tasks. Our experiments show that LLMs exhibit an average 33.12% lower success execution rate (SER) on CORGI compared to existing benchmarks such as BIRD, highlighting the substantially higher complexity of real-world business needs. We release the CORGI dataset, an evaluation framework, and a submission website to support future research.
Understanding CORGI
The CORGI benchmark introduces a fresh perspective on text-to-SQL tasks by encompassing a wider array of queries that users in business settings typically require. Unlike traditional methods, which primarily focus on straightforward data retrieval, CORGI delves into more intricate questions that demand higher-order cognitive skills, such as predictive analysis and strategic recommendations.
The Structure of CORGI
CORGI is structured around synthetic databases modeled after well-known companies like DoorDash, Airbnb, and Lululemon, ensuring it reflects practical scenarios faced by businesses today. This benchmark categorizes questions into four distinct levels of complexity: descriptive, explanatory, predictive, and recommendational. Each category progressively challenges the language models (LLMs) as they are designed to require increased reasoning and analytical capabilities.
Performance Insights
Through rigorous testing, the authors discovered that as question complexity increases, LLM performance significantly declines. Their research indicated an average 33.12% lower success execution rate (SER) for LLMs on the CORGI benchmark as opposed to established benchmarks like BIRD. Such findings highlight the necessity of developing more sophisticated AI models capable of addressing real-world business inquiries effectively.
Importance of Causal Reasoning and Temporal Forecasting
One of the standout features of the CORGI benchmark is its focus on causal reasoning and temporal forecasting. Many traditional text-to-SQL benchmarks do not require models to understand the relationships between different pieces of data over time. In contrast, CORGI encourages LLMs to perform analyses that mimic human thought processes, making it more applicable to real-world scenarios where context and timing significantly affect decision-making.
Encouraging Qualitative Evaluations
In introducing CORGI, the authors also advocate for new methodologies in evaluating AI performance in text-to-SQL tasks. Existing metrics often focus primarily on quantitative success rates, leaving out qualitative assessments which are essential for understanding the practical applicability of AI responses. CORGI promotes the exploration of open-ended evaluation frameworks, enabling researchers and developers to measure not just the correctness of an answer but its relevance and utility in a business context.
Release of CORGI Dataset and Support for Future Research
The release of the CORGI dataset, along with its accompanying evaluation framework and submission website, marks a significant asset for the academic community and industry practitioners alike. By offering resources that facilitate future research and exploration, CORGI stands to advance the capabilities of natural language processing in the business domain, driving innovations that meet the complexities of modern business needs.
Submission History
For those interested in tracking the progress of this benchmark, it’s worth noting the submission history:
- [v1] Submitted on Wed, 8 Oct 2025
- [v2] Revised on Thu, 9 Oct 2025
- [v3] Revised on Sun, 11 Jan 2026
- [v4] Last revised on Tue, 13 Jan 2026
Inspired by: Source
