Introducing the New Palmyra-mini Models: Redefining Efficient AI Solutions

The team at WRITER is excited to unveil three new open models in the Palmyra-mini family. These models, designed for outstanding performance, are lightweight and highly efficient, with parameters ranging from 1.5 billion to 1.7 billion. This strategic design makes them perfect for a myriad of applications that require efficient inference without compromising quality.

The Power of Chain of Thought

What sets the new Palmyra-mini models apart is their training approach, known as the Chain of Thought (CoT). This methodology enhances reasoning capabilities, allowing the models to tackle more complex tasks with clarity and precision. The team is keen to see how the community utilizes these innovative models in real-world applications.

Explore the Benchmark Highlights

The Palmyra-mini models come with a host of impressive benchmarks that showcase their capabilities:

• palmyra-mini: This non-reasoning enhanced base model achieves a score of 52.6% on the Big Bench Hard (get-answer)(exact_match). Its versatility makes it an excellent all-rounder for varied generative tasks.
• palmyra-mini-thinking-a: Ideal for intricate logical challenges, this variant boasts a score of 82.87% on GSM8K (strict match). Trained using the Chain of Thought approach, it excels in reasoning tasks, making it a favorite among complex problem solvers.
• palmyra-mini-thinking-b: This model pushes the boundaries of problem-solving with a remarkable 92.5% score on AMC23. When faced with challenging tasks, it’s the go-to choice, having the highest average benchmark scores across various tests like AIME24, GPQA, and MMLU_PRO.

Understanding the Benchmark Methodology

To ensure transparency and accuracy, we provide pass@1(avg-of-1) and pass@1(avg-of-64) results. The benchmarking methodology employed was crafted to validate the models accurately:

Scores for pass@1(avg-of-1) were collected using the lm_eval framework, while those for AIME24 through HMMT25 came from the lighteval framework. For pass@1(avg-of-64), data was gathered through the nemoskills framework.

Technical Insights

Since all base models operate on the Qwen architecture, they can be run on popular inference frameworks like vLLM, SGLang, TRTLLM, and TGI. For the palmyra-thinking-b model, the base was derived from NVIDIA’s OpenReasoning-Nemotron-1.5B.

After implementing reinforcement learning fine-tuning, we observed an increase in single-shot accuracy (pass@1) while noting a drop in sampling diversity. This trade-off showcases a delicate balance between accuracy and diversity, inviting further discussions and research on mode collapse and the performance of smaller models.

Looking Ahead

With the introduction of the Palmyra-mini models, WRITER aims to explore new frontiers in small parameter models. We eagerly anticipate innovation in inference efficiency, allowing developers to achieve top-notch performance without sacrificing quality. Join us in this exciting journey as the AI community continues to evolve!