Revolutionizing Code Review: HubSpot’s AI-Powered Sidekick

In the fast-paced world of software development, code review is a critical yet often cumbersome process. HubSpot engineers recently unveiled a groundbreaking solution: Sidekick, an AI-powered code review agent designed to streamline this vital task. By leveraging large language models (LLMs), Sidekick automates code analysis, directly posting actionable feedback on GitHub pull requests. The results are impressive, boasting a remarkable 90% reduction in time-to-first-feedback and empowering developers to identify issues much earlier.

The Importance of Code Review

Code review serves as a safeguard in software development, ensuring that changes meet quality standards and adhere to best practices. However, traditional review processes can encounter delays, particularly when reviewers are unavailable. HubSpot noticed that while AI coding assistants were rapidly accelerating code creation, the manual review process often lagged behind. With Sidekick, engineers can receive immediate feedback on their pull requests, enabling human reviewers to concentrate on more complex architectural considerations and design improvements. This shift not only boosts efficiency but also eliminates review bottlenecks.

Emily Adams, a HubSpot engineer, captured the essence of this transformation in a recent blog post:

“What we found might surprise you: our AI code reviewer catches real issues, understands HubSpot‑specific context, and maintains a high signal to noise ratio, often leaving no comments at all.”

The Evolution of Sidekick

The inaugural version of Sidekick operated within an internal platform called Crucible. In this initial setup, the LLM agents were housed in Kubernetes environments, interacting with repositories via a command line interface. This configuration allowed the agents to retrieve pull request changes and generate relevant review comments. However, it came with operational complexities—each review necessitated separate containerized workloads, raising latency and infrastructure challenges. Moreover, control over agent interactions with developer tools was limited.

Migration to Aviator Framework

To enhance functionality and streamline operations, HubSpot’s engineering team migrated Sidekick to a Java-based agent framework known as Aviator. This integration facilitates the running of review agents within existing services as opposed to isolated environments, significantly reducing overhead. Aviator is also adaptable, supporting multiple model providers like Anthropic, OpenAI, and Google, which allows for continuous experimentation and fallback options. The implementation of Remote Procedure Call (RPC) based tool abstractions enables agents to efficiently retrieve repository context, making their feedback increasingly relevant and accurate.

Ensuring Quality Feedback

One of the challenges encountered during Sidekick’s deployment was ensuring the quality of feedback. Initial iterations produced verbose or overly positive comments that cluttered discussions. To address this, the team developed a “judge agent,” which assesses comments before they are submitted to pull request discussions. This innovative evaluator pattern significantly reduces low-value comments and enhances the overall signal-to-noise ratio. So far, developers have reacted positively, with a consistent 80% thumbs-up rate indicating strong trust and adoption of the tool.

Feedback from Leadership

Brian L, the VP of Engineering at HubSpot, shared his insights on the adoption of Sidekick on LinkedIn, emphasizing the effectiveness of the judge agent in refining feedback quality:

“The most impactful change was adding a second agent to evaluate reviews before posting. The result: fewer, better, and more actionable comments. We knew we’d gotten it right when engineers started asking to see Sidekick’s feedback even before opening a PR.”

Future Developments

Looking ahead, HubSpot engineers aim to enhance the capabilities of Sidekick further. Plans include adding persistent memory for review agents, which would facilitate a better understanding of code changes across related repositories. This improvement could potentially deepen the context and relevance of the automated feedback, making Sidekick an indispensable tool in the software development lifecycle.