Introducing LiveCodeBench: A New Benchmark for LLM Code Generation

We are thrilled to unveil the LiveCodeBench leaderboard, a pioneering benchmark designed by researchers from UC Berkeley, MIT, and Cornell to measure the code generation capabilities of Large Language Models (LLMs). LiveCodeBench stands out by collecting coding problems over time from various coding contest platforms, providing a robust framework for evaluating the performance of AI programming agents.

What is LiveCodeBench?

LiveCodeBench is a comprehensive benchmarking tool that curates coding challenges from popular coding contest platforms such as LeetCode, AtCoder, and CodeForces. These platforms host regular contests, where thousands of participants tackle problems that assess their coding and problem-solving skills. Each problem is accompanied by a natural language description and example input-output pairs, ensuring clarity and correctness.

Holistic Evaluation of Coding Capabilities

The benchmark covers a range of coding scenarios that go beyond mere code generation. LiveCodeBench evaluates models on four distinct scenarios:

• Code Generation: Given a problem statement, the model must produce a correct solution. The evaluation focuses on the functional correctness of the generated code against a set of test cases.
• Self Repair: In this scenario, the model generates a candidate program and, upon receiving error feedback, must generate a fix. Evaluation is based on whether the revised code passes the same functional correctness tests.
• Code Execution: The model is tasked with predicting the output of a provided program snippet based on a test input. Correctness is determined by asserting that the model’s output matches the expected result.
• Test Output Prediction: Here, the model generates expected outputs based solely on the problem statement and a provided test case, evaluated through an exact match checker.

Evaluation Metrics: Pass@1

For each scenario, LiveCodeBench employs the Pass@1 metric, which measures the probability of generating a correct answer. This metric is calculated by dividing the number of correct answers by the total number of attempts, yielding a straightforward measure of a model’s performance.

Addressing Benchmark Contamination

One of the significant challenges in LLM evaluations is benchmark contamination. Reports have shown that models often overfit to existing benchmarks, undermining their ability to generalize. To combat this, LiveCodeBench annotates problems with release dates, allowing for the evaluation of models on unseen problems released after their training cutoff date. This “scrolling over time” feature enables researchers to assess a model’s true generalization capabilities.

Key Findings from LiveCodeBench

The initial findings from LiveCodeBench highlight several important trends in model performance:

• Model performances show correlation across different scenarios, but their relative rankings can vary significantly.
• GPT-4-Turbo consistently emerges as the top-performing model across most scenarios, particularly excelling in self-repair tasks.
• Claude-3-Opus surpasses GPT-4-Turbo in the test output prediction scenario, indicating stronger natural language reasoning abilities.
• Mistral-Large exhibits superior performance on natural language reasoning tasks, particularly in test output prediction and code execution.

How to Submit Your Models to LiveCodeBench

If you’re eager to evaluate your coding models using LiveCodeBench, follow these steps:

1. Environment Setup: Use conda to create a new environment and install LiveCodeBench.

git clone https://github.com/LiveCodeBench/LiveCodeBench.git 
cd LiveCodeBench
pip install poetry
poetry install

2. Model Evaluation: To evaluate new Hugging Face models, run the following command:

python -m lcb_runner.runner.main --model {model_name} --scenario {scenario_name}

3. Submission of Results: Once you generate your results, submit them by filling out the designated form.

How to Contribute to LiveCodeBench

LiveCodeBench is an evolving project, and we welcome collaborations and suggestions. The dataset and code are publicly available, so feel free to reach out by submitting an issue or sending an email. Your input can help shape the future of this benchmarking tool and advance the field of AI programming agents.