At QCon AI NYC 2025, a fascinating presentation by Will Hang from OpenAI unveiled the innovative approach of Agent RFT—a reinforcement fine-tuning method specifically designed to enhance the performance of tool-using agents.
Hang laid out a tactical roadmap that emphasizes initial improvements in prompt and task optimization before diving into adjustments in model weights. This pragmatic path includes simplifying task requirements, implementing guardrails to mitigate tool misuse, enhancing tool descriptions, and refining tool outputs to empower agents in making more informed downstream decisions. Notably, he observed that while these modifications can yield significant benefits, they may reach a plateau for tasks demanding consistent multi-step reasoning during tool interactions.
In his discourse, Hang positioned fine-tuning strategies along a spectrum. He highlighted supervised fine-tuning as particularly effective for tasks with predictable input-output mappings, aiming to replicate a specific style or structure. On the other hand, preference optimization was discussed as a technique for steering outputs towards desired responses through paired comparisons. According to OpenAI’s Direct Preference Optimization guide, this method currently focuses on text inputs and outputs. Hang argued that reinforcement fine-tuning is a more suitable choice for scenarios requiring the discovery of strategies over extended sequences rather than merely replicating a singular completion pattern.
Beware of reward hacking! Resolve any edge cases in your grader. Continuous rewards work better than binary rewards. – Will Hang, OpenAI
Hang introduced Agent RFT as a reinforcement fine-tuning approach tailored for tool-using agents, where models explore diverse strategies during training rollouts and receive feedback from a defined grader. OpenAI’s documentation illustrates this process as one that involves sampling potential responses, evaluating them with a custom grader, and updating the model based on those evaluations. He stressed the importance of credit assignment along the entire trajectory, so earlier choices—like tool selection and the structure of tool calls—can be reinforced or discouraged based on the outcomes downstream. In this context, an agent is not limited to responding to user prompts but is capable of interacting proactively with external tools.
Examples of tools discussed included coding terminals for agents, internal business systems for customer support, and search or retrieval endpoints for documents. Hang emphasized that tool outputs are integrated back into the context window, meaning that tool calls, outputs, reasoning tokens, and final responses collectively form a single, comprehensive multi-step trajectory. He noted that graders have become essential artifacts within this workflow, employing various grading techniques like simple matchers, model-based judges, code-based graders, endpoint graders, and hybrid systems optimizing both accuracy and latency.
The presentation also addressed operational characteristics beyond simple answer accuracy. Hang demonstrated how Agent RFT can help minimize unnecessary tool calls, uphold tool-call budgets, and reduce lengthy trajectories that might lead to unpredictable latency and a diminished user experience. Training traces illustrated a decrease in both reasoning tokens and tool calls, aligning with the concept that agents can learn to achieve equivalent or superior outcomes with fewer steps.
Following Hang’s insights, Wenjie Zi expanded on real-world applications, sharing platform setup details and use cases, including a finance-oriented scenario. In this example, an agent must sift through a large document corpus to locate relevant information while operating under a constrained tool-call budget. Here, the agent employed a series of tools—searching, listing, and reading files—with a grader evaluating the final output. Zi emphasized the advantage of a model-based grader, even for numeric responses, as it helps minimize false negatives caused by minor formatting differences or variations in units.
Further illustrating the adaptability of Agent RFT, Zi covered broader applications in coding and other sectors, highlighting environments rich in tools, isolated execution contexts, and reward designs that strike a balance between correctness and efficiency. The reported benefits included enhanced planning abilities, shorter long trajectory tails, and in some instances, a pivot towards parallel tool calls, thereby accelerating responsiveness.
Developers eager to delve deeper into this cutting-edge approach can consult OpenAI’s Reinforcement Fine-Tuning and Model Optimization documentation. Future broadcasts on infoq.com will feature a video recording of Hang and Zi’s engaging presentation, making this invaluable content readily accessible.
Inspired by: Source
