View a PDF of the paper titled In-N-Out: A Parameter-Level API Graph Dataset for Tool Agents, authored by Seungkyu Lee and two other contributors.
Abstract: Tool agents — LLM-based systems that interact with external APIs — provide a way to execute real-world tasks. However, as tasks become increasingly complex, these agents struggle to identify and call the correct APIs in the proper order. To tackle this problem, we investigate converting API documentation into a structured API graph that captures API dependencies and leveraging it for multi-tool queries that require compositional API calls. To support this, we introduce In-N-Out, the first expert-annotated dataset of API graphs built from two real-world API benchmarks and their documentation. Using In-N-Out significantly improves performance on both tool retrieval and multi-tool query generation, nearly doubling that of LLMs using documentation alone. Moreover, graphs generated by models fine-tuned on In-N-Out close 90% of this gap, showing that our dataset helps models learn to comprehend API documentation and parameter relationships. Our findings highlight the promise of using explicit API graphs for tool agents and the utility of In-N-Out as a valuable resource. We will release the dataset and code publicly.
Submission History
From: Seungkyu Lee [view email]
[v1] Mon, 1 Sep 2025 15:42:21 UTC (1,610 KB)
[v2] Wed, 19 Nov 2025 02:12:14 UTC (1,612 KB)
Understanding In-N-Out: A New Era in Tool Agents and API Interaction
In the rapidly evolving landscape of artificial intelligence and machine learning, tool agents have emerged as crucial players, employing LLM (Large Language Model) technologies to interact seamlessly with external APIs. However, as those tasks grow in complexity, many of these agents face significant challenges when it comes to accurately identifying the correct APIs and executing them in the appropriate sequence.
The Challenge of Compositional API Calls
As API ecosystems expand, the need for sophisticated, efficient interactions becomes more pressing. Traditional methods of managing API documentation often fall short, failing to portray the intricate dependencies between various APIs. This is where the beauty of compositional API calls comes into play. It’s about more than just accessing a single API; it involves a series of interdependent interactions that must be orchestrated skillfully.
Introducing In-N-Out: The Innovative API Graph Dataset
In response to these challenges, the authors introduce a game-changing dataset called In-N-Out. This dataset is the first of its kind, featuring expert-annotated API graphs derived from two real-world API benchmarks along with their accompanying documentation. By structuring API relationships within a graph framework, In-N-Out aids tool agents in making quick, informed decisions about API usage.
Significant Improvements in Performance
The implications of using In-N-Out are substantial. Initial tests suggest that leveraging this API graph dataset can nearly double the performance of tool agents compared to those relying solely on conventional documentation methods. Users can expect significantly improved tool retrieval and multi-tool query generation capabilities. This level of performance enhancement is critical, especially in industries where efficiency and accuracy are paramount.
Fine-Tuning With In-N-Out
One of the most intriguing findings from the authors’ research is the potential for fine-tuning LLMs using the In-N-Out dataset. Models trained with this dataset show the ability to close about 90% of the performance gap when compared to their counterparts using only standard documentation. This highlights the importance of structured data in bridging the comprehension gap that often exists between the language models and API parameter relationships.
More Read
The Value of Structured API Graphs
By employing explicit API graphs, developers and researchers can create tool agents that not only fulfill basic functions but also navigate the complexities of real-world tasks. The structured format inherent in In-N-Out provides insights into how APIs interrelate, allowing agents to harness the full power of available tools. This shift toward graphed data represents a significant leap in how we think about API interactions in AI systems.
Making In-N-Out Accessible to All
One of the key commitments from the study’s authors is the promise to release In-N-Out and its corresponding code publicly. This will allow researchers, developers, and industry professionals to implement the findings in their own projects, fostering innovation and improvement across sectors that rely on API interactions.
The landscape of tool agents is changing, and with advancements like In-N-Out, we are on the brink of new possibilities in how these systems function and interact with APIs. As we delve into these innovations, the focus remains on understanding and navigating the complexities of API interactions, paving the way for more advanced applications in the future.
Inspired by: Source
