Experiments and Results: Evaluating Agentic RAG on FramesQA
In the rapidly evolving world of AI, understanding how various systems perform across different tasks is crucial. One such system, the agentic Retrieval-Augmented Generation (RAG), has shown promising results in evaluating multi-hop questions. In this article, we delve into the details of our experiments with agentic RAG on the FramesQA dataset, which is grounded in the FRAMES paper.
Understanding the Multi-Hop Question
A multi-hop question poses a unique challenge, requiring the model to dive into multiple pieces of information to arrive at a comprehensive answer. For example, consider this intriguing question:
“Of the top two most watched television season finales (as of June 2024), which finale ran the longest in length and by how much?”
To tackle this question, our RAG system must undertake several steps. First, it identifies the top two most watched finales, namely MASH and Cheers*. Next, it tracks down their respective running times and computes the difference in length.
Challenges with Traditional RAG Models
In conventional RAG settings, both the Vanilla RAG and agentic RAG operating without adequate context may struggle with such tasks. A typical response could resemble:
More Read
“Despite multiple scans, I found no explicit runtimes for MASH or Cheers. The documents provide viewership data, but not the duration in minutes or hours.”*
This kind of output fails to address the original question, emphasizing the critical need for a more sophisticated approach.
The Solution: Agentic RAG
What sets agentic RAG apart is its capability to handle complex queries effectively. Instead of providing vague or unrelated information, it utilizes advanced components like the Query Rewriter and the Sufficient Context Agent. In our example, the system first searches for the TV shows and then specifically looks for their runtimes. This meticulous approach allows it to provide a robust answer:
“The MASH finale ran for 150 minutes, making it the longest of the top two. It was 52 minutes longer than the Cheers finale, which ran for approximately 98 minutes.”*
Experimental Setup
We put agentic RAG to the test with an extensive experiment on FramesQA. This dataset consists of 824 queries and a corpus of 2,676 PDF documents. We contrasted the performance of the agentic RAG with Google’s Vanilla RAG, which is equipped with an advanced retrieval engine, large language model (LLM) parser, and re-ranker.
Single-Corpus vs. Cross-Corpus Settings
In our evaluations, we analyzed agentic RAG in two contexts: the single-corpus setting, where retrieval was limited to FramesQA documents, and the cross-corpus setting, which included three additional distracting datasets. The latter simulates real-world applications where organizations may possess multiple databases managed by different teams.
Evaluating Performance
To gauge the effectiveness of our systems, we computed accuracy using an LLM-as-a-judge, comparing the responses generated by the systems against the ground truth answers in the dataset.
Accuracy in Cross-Corpus Settings
In an impressive display of capability, agentic RAG managed to demonstrate nearly the same accuracy in cross-corpus settings as in the single-corpus context. Even with a Planner Agent that had to discern the correct corpus from four possibilities, the system successfully answered 90.1% of the questions accurately. This is a remarkable feat, showcasing agentic RAG’s ability to navigate multiple, unrelated data sources seamlessly.
Latency Concerns
An important aspect of any system is its response time, particularly when handling large datasets. To our relief, the latency for both the single- and cross-corpus versions of the agentic RAG remained nearly identical, deviating by just about 3% on average. This consistency reflects the robustness of our model, indicating that the enhanced search capabilities do not compromise speed.
Implications for Retrieval Scenarios
The results of these experiments unveil exciting possibilities for the application of agentic RAG. Its proficiency in reasoning over multiple, unrelated sources of data can open doors to highly flexible retrieval scenarios, benefiting industries that rely on information from diverse datasets.
By highlighting the remarkable performance of agentic RAG in complex querying situations, these experiments contribute valuable insights into the advancement of AI models in real-world applications. As the technology continues to evolve, the potential for applications in various fields remains vast and promising.
Inspired by: Source
