Understanding Hugging Face’s Open LLM Leaderboard and Vectara’s HHEM
Hugging Face’s Open LLM Leaderboard has become an invaluable resource for the open-source community, meticulously tracking the performance of various open-source Large Language Models (LLMs). Originally created by Ed Beeching and Lewis Tunstall, and now maintained by Nathan Habib and Clémentine Fourrier, this leaderboard compares LLMs based on their performance across a range of tasks, including TruthfulQA and HellaSwag. For practitioners in the field, this tool offers a clear perspective on which models are excelling and which may require further refinement.
The Significance of the Leaderboard
The ability to track the best-performing open-source models is crucial for developers and researchers striving to enhance the capabilities of LLMs. By providing a structured comparison, the leaderboard encourages transparency and fosters innovation, pushing the boundaries of what these models can accomplish.
In late 2023, Vectara introduced an exciting new addition to this ecosystem: the Hughes Hallucination Evaluation Model (HHEM). This open-source model specifically measures the propensity of LLMs to hallucinate—essentially, to generate nonsensical or unfaithful content based on the input provided. This evaluation is essential, as hallucinations can significantly undermine the reliability of AI-generated content.
What is the Hughes Hallucination Evaluation Model (HHEM)?
The HHEM serves as a benchmark for assessing the frequency of hallucinations in document summaries generated by popular LLMs like GPT-4, Google’s Gemini, and Meta’s Llama 2. By providing a standardized evaluation metric, Vectara aims to democratize the assessment of LLM performance, raising awareness of the discrepancies between models in terms of their likelihood to hallucinate.
This initiative is particularly timely, as the landscape of LLMs continues to grow rapidly, with both open-source solutions like Llama 2 and commercial models like OpenAI’s GPT-4 evolving swiftly. By establishing the HHEM, Vectara not only offers a tool for evaluation but also promotes a culture of accountability and improvement within the LLM community.
More Read
Honoring a Legacy
It’s heartwarming to note that the HHEM is named in memory of Simon Hughes, a valued peer who passed away in November 2023. The naming serves as a tribute to his lasting impact on the field, reinforcing the importance of community and collaboration in the tech industry.
Setting Up the HHEM Leaderboard
To create the HHEM leaderboard, Vectara utilized the newly released Hugging Face leaderboard templates. This open-source solution simplifies the process of managing and updating the leaderboard, making it more accessible for developers looking to evaluate their models. Here’s how Vectara set up the HHEM leaderboard:
-
Repository Cloning: The team began by cloning the space repository to their organization and creating two associated datasets—“requests” for new model evaluations and “results” for storing evaluation outcomes.
- Data Population: After populating the results dataset with initial findings, they updated the “About” and “Citations” sections to provide context and credibility.
For a straightforward leaderboard, this setup suffices. However, Vectara’s evaluation process is more nuanced, necessitating further customization.
Customization for Complex Evaluations
The team made several adjustments to the HF leaderboard template to fit the specific needs of the HHEM:
-
Model Operations: In the
leaderboard/src/backend/model_operations.pyfile, they implemented two primary classes:SummaryGenerator, which creates summaries and calculates metrics like Answer Rate and Average Summary Length, andEvaluationModel, which loads the HHEM to assess these summaries, yielding metrics such as Factual Consistency Rate and Hallucination Rate. -
Evaluation Integration: The
leaderboard/src/backend/evaluate_model.pyfile houses theEvaluatorclass, which utilizes both theSummaryGeneratorandEvaluationModelto compute results in JSON format. -
Evaluation Execution: In
leaderboard/src/backend/run_eval_suite.py, therun_evaluationfunction leverages theEvaluatorto obtain and upload results to the results dataset, ensuring they appear on the leaderboard. - Request Management: Finally,
leaderboard/main_backend.pymanages pending evaluation requests and executes auto-evaluations using the aforementioned classes.
With careful modifications, Vectara successfully established a comprehensive evaluation pipeline, ready for deployment as a Hugging Face Space.
The Impact of HHEM on the LLM Community
The HHEM is a groundbreaking tool designed to gauge the hallucination rates of popular LLMs. Leveraging Hugging Face’s leaderboard template not only provided a robust framework for managing model submissions but also facilitated the regular updating of evaluation results.
The open-source nature of both the HHEM and the leaderboard template encourages collaboration and innovation across the community, inviting contributions from other developers looking to publish their own LLM leaderboards. This collaborative spirit is essential for the ongoing evolution of LLM technology, as it drives continuous improvement and fosters a shared understanding of model performance.
If you’re interested in contributing to the HHEM or have suggestions for new models to evaluate, the team at Vectara welcomes your input. Additionally, for any inquiries regarding the Hugging Face LLM front-end or Vectara’s initiatives, feel free to reach out through the relevant forums.
By integrating these innovative models and fostering a spirit of collaboration, the LLM community can continue to advance and refine the capabilities of AI-driven technologies.
Inspired by: Source
