Unlocking the Power of StarCoder and StarCoderBase: The Next Generation of Code LLMs

In the rapidly evolving landscape of artificial intelligence, Large Language Models (LLMs) have emerged as powerful tools for code generation and comprehension. Among these, StarCoder and StarCoderBase stand out as remarkable Code LLMs, trained on a wealth of permissively licensed data sourced from GitHub. This extensive dataset encompasses over 80 programming languages, Git commits, GitHub issues, and Jupyter notebooks, providing a robust foundation for these models.

Much like the well-known LLaMA, StarCoder is built on a ~15 billion parameter model trained on an impressive 1 trillion tokens. Its base model, StarCoderBase, was further fine-tuned with an additional 35 billion tokens specifically in Python, resulting in a model that has demonstrated exceptional performance across various programming benchmarks.

StarCoder’s Performance: A Cut Above the Rest

When it comes to performance, StarCoderBase has outperformed existing open Code LLMs on numerous popular programming benchmarks. Surprisingly, it even matches or exceeds the capabilities of closed models, such as code-cushman-001 from OpenAI—the foundational model that powered early iterations of GitHub Copilot.

One of the standout features of StarCoder models is their ability to process a context length of over 8,000 tokens, enabling them to handle more input than any other open LLM available. This capability opens up a multitude of applications, including:

• Code autocompletion
• Code modification based on instructions
• Explaining code snippets in natural language
• Acting as a technical assistant through dialogue prompts

Evaluation: Setting New Benchmarks

The evaluation process for StarCoder involved rigorous comparisons with several similar models across various benchmarks. One particularly notable benchmark is HumanEval, which tests a model’s ability to complete functions based on their signatures and docstrings. StarCoder and StarCoderBase have shown remarkable results, outperforming even the largest models like PaLM, LaMDA, and LLaMA, despite being smaller in size.

Specifically, in the HumanEval benchmark, StarCoder achieved a score of 40.8% when prompted effectively, setting a new state-of-the-art result for open models. This was a significant improvement from its base model, which scored 33.6%. The innovative prompting technique allowed the models to generate actual solutions rather than placeholder code, showcasing their true potential.

Multilingual Capabilities

An intriguing aspect of StarCoder is its multilingual capability. Evaluated on the MultiPL-E benchmark, which extends HumanEval to various languages, StarCoder demonstrated a performance level that often matched or surpassed code-cushman-001. Additionally, in a data science benchmark known as DS-1000, it significantly outperformed all other open-access models, showcasing its versatility and potential in diverse programming environments.

StarCoder as a Technical Assistant

Beyond code generation, we explored StarCoder’s potential as a technical assistant. Inspired by Anthropic’s HHH prompt, we developed a dedicated Tech Assistant Prompt. To our surprise, the model effectively acted as a tech assistant, providing insightful responses to programming-related inquiries with just a simple prompt.


Training Data and Ethical Considerations

The training of StarCoder was conducted on a carefully curated subset of The Stack 1.2. This dataset consists exclusively of permissively licensed code, ensuring ethical use in AI development. An opt-out process is available, allowing code contributors to remove their data from the dataset if they choose. Collaborating with Toloka, we have implemented strategies to eliminate Personal Identifiable Information from the training data, such as names, passwords, and email addresses.

About BigCode

BigCode is a collaborative effort led by Hugging Face and ServiceNow, focused on the responsible development of large language models for code. This initiative aims to advance the field while ensuring ethical practices in AI research and application.

Resources and Additional Releases

In conjunction with the model’s release, several valuable resources and demos are made available, including:

• The model weights, including intermediate checkpoints under the OpenRAIL license
• All code for data preprocessing and training, licensed under Apache 2.0
• A comprehensive evaluation harness for code models
• A new PII dataset for training and evaluating PII removal
• The fully preprocessed dataset utilized for training
• A code attribution tool for tracing generated code back to the dataset

Connecting with BigCode

Models

• Paper: A technical report about StarCoder.
• GitHub: All you need to know about using or fine-tuning StarCoder.
• StarCoder: StarCoderBase further trained on Python.
• StarCoderBase: Trained on 80+ languages from The Stack.
• StarEncoder: Encoder model trained on The Stack.
• StarPii: StarEncoder-based PII detector.

Tools & Demos

Explore various tools and demos available for developers and researchers.