Benchmarking Claude Code: Exploring AI Efficiency Across Programming Languages
A recent benchmark by Ruby committer Yusuke Endoh has shed light on the efficiency of Claude Code (Opus 4.6) in generating working implementations across 13 programming languages. This exploratory study, aimed at understanding how dynamic and static languages perform under AI-driven coding challenges, provides valuable insights for developers and teams choosing the right tools for their projects.
Methodology of the Experiment
The benchmark’s primary task was to ask Claude Code to implement a simplified version of Git across various programming languages. This was divided into two phases:
- v1 involved implementing
init,add,commit, andlogfrom an empty directory. - v2 extended the project to include
status,diff,checkout, andreset.
Each programming language was executed 20 times, generating comparable performance data. To maintain consistency, Endoh opted for a custom hash algorithm over SHA-256, minimizing the impact of varying library dependencies across languages.
Result Highlights: Speed, Cost, and Variance
The results of this benchmark revealed compelling insights into the performance of languages classified as dynamic versus those categorized as statically typed.
Dynamic Languages on Top
The standout performers were dynamic languages: Ruby, Python, and JavaScript. Their average costs and times per run were as follows:
More Read
- Ruby: $0.36 per run, 73.1 seconds
- Python: $0.38 per run, 74.6 seconds
- JavaScript: $0.39 per run, 81.1 seconds
Each of these languages exhibited low variance and successfully passed all tests across 40 runs.
Statically Typed Languages Lagging Behind
In contrast, statically typed languages demonstrated slower and costlier performances:
- Go: Averaged $0.50 at 101.6 seconds
- Rust: Averaged $0.54, with a wide variance of 54.8 seconds and notable test failures
- C: Closed out as the most expensive mainstream language, averaging $0.74 while generating 517 lines of code.
Insights on Type Systems
The findings regarding type systems were particularly enlightening for teams assessing AI-enhanced coding workflows. Adding strict type checking, such as mypy for Python or Steep for Ruby, significantly decreased performance:
- Python’s strict checking made it 1.6 to 1.7 times slower.
- Ruby’s strict checking slowed it by 2.0 to 3.2 times compared to regular Ruby.
Interestingly, TypeScript proved more expensive than JavaScript, costing $0.62 per run compared to JavaScript’s $0.39, despite generating similar line counts. The overhead likely stems from the complexities of reasoning about type constraints within the generated code.
Addressing Limitations and Community Concerns
Endoh was transparent about potential biases in the study, given his affiliation as a Ruby committer. He recognized the obstructions of conducting a uniform experiment across 15 languages, particularly at larger scales. The generated code, averaging about 200 lines, was considered appropriate for prototyping but did not encompass the intricacies of larger codebases.
Community feedback on platforms like Lobsters raised valid concerns regarding the small-scale outputs. Commenters pointed out that very few useful prototypes would be that compact, and the benchmark might not account for ecosystem advantages where languages with robust libraries could reduce the need for generated code in practical applications.
Endoh tackled these points directly. He noted that while larger benchmarks would add value, distinguishing fairness across diverse languages presents significant challenges. He also defended the relevance of speed differences in practical scenarios, stating that a 30-second gap might impede a developer’s workflow, especially in iterative AI-assisted development.
Failures and Observations
Of the 600 runs conducted, only three resulted in failures—two in Rust and one in Haskell. In one case, the Rust agent claimed that the tests were incorrect, but it was later noted as an instance of hallucination, as the majority of Rust trials were successful.
Access to Data
The entire dataset, including execution logs, per-run results, and the generated source code, is accessible through the benchmark repository on GitHub. This resource provides transparency and allows for deeper exploration of the data by interested developers and researchers.
This benchmark not only highlights the efficiency disparities between dynamic and static programming languages but also invites further discussion and exploration in the evolving landscape of AI-assisted coding. With the data made publicly available, developers can better inform their choices about the programming languages best suited for their projects and workflows.
Inspired by: Source
