Understanding the Impact of Curation in Models: A Deep Dive into LLM Experiments
In the rapidly evolving landscape of AI, understanding the nuances of model performance is essential, particularly when working with large language models (LLMs). This article explores our experiments to untangle the benefits of a detailed curation process, emphasizing how it affects different model sizes and task complexities.
The Experiment Setup
To gain insights into the efficacy of our curation process, we embarked on a systematic comparison involving two LLMs: Gemini Nano-1, which boasts 1.8 billion parameters, and Nano-2 with a larger capacity of 3.25 billion parameters. These models were fine-tuned for two distinct tasks, each representing different levels of complexity. The tasks were evaluated based on input from crowdsourced labels, creating a robust framework for analysis.
The datasets for these tasks comprised approximately 100,000 annotations each, yet they displayed a significant class imbalance—around 95% of the labels were classified as benign. This imbalance posed an intriguing challenge, as we sought to refine our models to not only recognize benign instances but also to properly identify and respond to more complex inputs.
Baseline Conditions and Curation Process
In our approach, we established baseline conditions whereby each LLM was fine-tuned against its uncurated datasets. The first task involved a lower complexity level, while the second was categorized as higher complexity based on expert alignment metrics. Using our curated process, we implemented multiple fine-tuning rounds, adjusting our models based on real-time evaluations derived from selectively chosen examples.
Each iteration involved careful selection of curated examples, greatly influencing how we trained and evaluated our models. We reached a plateau in performance before aligning the models to expert levels, indicating a need for further refinement. For the simpler task, we concluded the experiment after six iterations, which included roughly 400 fine-tuning samples and 250 evaluation instances. In contrast, the higher complexity task warranted a slightly different approach, culminating in just five iterations with 250 fine-tuning and 150 evaluation samples.
More Read
Insights from Model Performance
As we progressed through our iterations, we observed that the task complexity significantly influenced the models’ convergence rates. The more complex tasks benefitted from a richer diversity of examples, which likely contributed to the extended time needed to achieve optimal performance. Notably, by the end of our experiments, both datasets reflected a balanced class representation of around 40% positive examples, a significant shift from the original imbalance.
To gauge the effectiveness of our curation process, we employed Cohen’s Kappa statistic as a measure of agreement between expert annotations and model predictions. On the lower complexity task, experts achieved an impressive average pairwise Kappa score of .81, while for the more complex task, this score was .78. These values represent a performance ceiling, providing a benchmark against which we can measure improvements in model outputs.
Evaluating Crowdsourced Data Quality
A vital aspect of our experiments hinged on understanding the quality of the crowdsourced data compared to our curated set. By calculating the Kappa alignment between the crowdsourced annotations and expert evaluations, we observed a Kappa score of .59 for the lower complexity task and .41 for the higher complexity task. These figures underscore the discrepancies that can arise from relying solely on crowdsourced inputs, highlighting the necessity of curation to elevate the quality of model training.
Final Thoughts on Model Fine-Tuning
Throughout the course of our experiments, it became evident that the interplay between model size, task complexity, and curation significantly influences performance outcomes. The iterative refinement through the curation process not only improved our models’ alignment with expert evaluations but also provided deeper insights into the behavior of LLMs in diverse settings. As we continue to refine our approach, the lessons gleaned from these experiments will undoubtedly shape future endeavors in the realm of AI development.
Inspired by: Source
