As of February 9, 2022, GPT-NeoX-20B checkpoints are available for download from The Eye under Apache 2.0. More in-depth information on GPT-NeoX-20B can be found in the associated technical report on arXiv.
Looking for a demo? Try GPT-NeoX-20B via CoreWeave and Anlatan’s inference service, GooseAI!
After a year-long odyssey through months of chip shortage-induced shipping delays, technical trials and tribulations, and aggressively boring debugging, we are excited to announce EleutherAI’s latest open-source language model: GPT-NeoX-20B. This groundbreaking model boasts 20 billion parameters and has been trained using the innovative GPT-NeoX framework on GPUs provided by our partners at CoreWeave. This collaboration has been instrumental in making such an ambitious project a reality.
GPT-NeoX-20B stands as the largest publicly accessible pretrained general-purpose autoregressive language model to date. It is designed to excel across a myriad of tasks, from natural language understanding to generation, making it a versatile tool for researchers and developers alike. With its vast capabilities, we anticipate that GPT-NeoX-20B will significantly contribute to the landscape of AI research.
The accessibility of models of this size is pivotal in promoting responsible research and the safe application of AI systems. We invite researchers and developers interested in exploring the potentials of GPT-NeoX-20B to reach out to us, as collaboration and knowledge-sharing are fundamental to driving advancements in AI.
In appreciation of our generous compute donors, we decided to delay the public release of the model by an additional seven days. Consequently, on February 9, 2022, the complete model weights will be available for free download under the permissive Apache 2.0 license from The Eye, allowing anyone to experiment with and build upon this powerful resource.
For discussions and community engagement around GPT-NeoX-20B, a dedicated #20b channel has been established on our Discord server. We encourage users to join in and share insights, experiences, and challenges encountered while working with this model. It’s important to note that much like our other models, GPT-NeoX and GPT-NeoX-20B are primarily research artifacts. We do not recommend deploying them in a production environment without thorough consideration and testing, as there are still bugs and inefficiencies to be addressed. We appreciate your understanding as we continue to refine our work!
| Task | Category | Babbage | Curie | GPT-J-6B | FairSeq-13B | GPT-NeoX-20B | DaVinci |
|---|---|---|---|---|---|---|---|
| LAMBADA | Sentence Completion | 62.49% | 69.51% | 68.29% | 70.95% | 72.00% | 75.16% |
| ANLI R1 | Natural Language Inference | 32.40% | 32.80% | 32.40% | 34.00% | 34.00% | 36.30% |
| ANLI R2 | Natural Language Inference | 30.90% | 33.50% | 34.00% | 33.00% | 34.40% | 37.00% |
| ANLI R3 | Natural Language Inference | 33.75% | 35.50% | 35.50% | 34.75% | 35.40% | 36.83% |
| WSC | Coreference Resolution | 54.54% | 49.54% | 49.54% | 55.44% | 50.00% | 59.18% |
| WinoGrande | Coreference Resolution | 59.51% | 64.56% | 64.01% | 67.40% | 66.10% | 69.93% |
| HellaSwag | Sentence Completion | 40.38% | 54.81% | 36.53% | 57.69% | 53.50% | 63.46% |
| Average | 44.85% | 48.60% | 45.75% | 50.43% | 49.34% | 53.98% |
| Subject Group | Babbage | Curie | GPT-J-6B | FairSeq-13B | GPT-NeoX-20B | DaVinci |
|---|---|---|---|---|---|---|
| Humanities | 27.01% | 26.48% | 28.07% | 27.27% | 28.70% | 32.30% |
| Social Science | 27.94% | 29.24% | 28.73% | 27.94% | 30.80% | 35.87% |
