Unlocking the Power of AWS Inferentia2 for Machine Learning
AWS Inferentia2 is the newest addition to Amazon Web Services’ (AWS) suite of machine learning (ML) chips, specifically designed to enhance AI workloads. Available through the Amazon EC2 Inf2 instances, Inferentia2 promises exceptional performance and cost-effectiveness for production workloads.
In collaboration with AWS’s product and engineering teams, Hugging Face has worked diligently over the past year to integrate the performance and cost-efficiency of AWS Trainium and Inferentia chips into the Hugging Face ecosystem. The result is the open-source library optimum-neuron, which simplifies the process of training and deploying Hugging Face models on these powerful accelerators. This initiative makes it easier than ever for developers and data scientists to leverage cutting-edge AI technology.
Enabling Over 100,000 Models on AWS Inferentia2 with Amazon SageMaker
A significant development in the deployment of Large Language Models (LLMs) occurred a few months ago with the introduction of AWS SageMaker’s new Inferentia/Trainium option. This allows users to deploy models such as Meta’s Llama 3 on Inferentia2 instances, making it possible to serve inference at scale. SageMaker offers a fully managed environment that facilitates model building, fine-tuning, and governance, ensuring users can maximize their AI capabilities without getting bogged down in infrastructure management.
The scope of this deployment capability has now expanded to over 100,000 public models available on Hugging Face. This includes newly supported architectures like ALBERT, BERT, RoBERTa, and DistilBERT, among others, as well as various machine learning tasks such as text classification, text generation, and question answering. This extensive support allows developers to choose from a wide range of models tailored to their specific needs.
Hugging Face Inference Endpoints Introduces Support for AWS Inferentia2
For those looking for a seamless deployment experience, Hugging Face Inference Endpoints provide an excellent solution. With the introduction of Inferentia2 instances, deploying models from the Hugging Face Hub has never been easier. Users can deploy their chosen model in just a few clicks by selecting the model, choosing the Inf2 instance option in the AWS instance configuration, and launching the deployment.
More Read
For supported models, such as Llama 3, two instance types are available:
- Inf2-small: Featuring 2 cores and 32 GB of memory, priced at $0.75 per hour, this option is perfect for deploying Llama 3 with 8 billion parameters.
- Inf2-xlarge: Offering 24 cores and 384 GB of memory at $12 per hour, this instance is ideal for larger models like Llama 3 with 70 billion parameters.
Hugging Face Inference Endpoints utilize a billing model that charges by the second based on capacity used, making it a flexible and cost-effective solution for developers. The autoscaling feature allows costs to scale up with demand and down to zero during idle periods, providing both efficiency and savings.
Leveraging Text Generation Inference for Optimal Performance
When deploying Llama 3 on AWS Inferentia, Hugging Face Inference Endpoints utilize Text Generation Inference (TGI) for Neuron. TGI is a specialized solution crafted for deploying and serving LLMs at scale, offering features like continuous batching and streaming for optimal performance. Additionally, models deployed with TGI are compatible with the OpenAI SDK Messages API, allowing existing Gen AI applications to integrate seamlessly without requiring code modifications.
Users can interact with their deployed endpoints through a user-friendly widget in the UI or via the OpenAI SDK, making it easy to send requests and receive responses from their models.
Future Developments and Enhancements
Looking ahead, Hugging Face is committed to expanding the range of models that can be deployed on AWS Inferentia2 through Inference Endpoints. Upcoming plans include adding support for Diffusion and Embedding models, which will enable users to generate images and build semantic search and recommendation systems. By leveraging the speed and efficiency of AWS Inferentia2, Hugging Face aims to enhance the deployment experience further.
Moreover, ongoing efforts to improve performance for Text Generation Inference on Neuronx will ensure that LLM deployments on AWS Inferentia 2 become faster and more efficient. As updates continue to roll out, users can expect even more capabilities designed to optimize their machine learning workflows.
With these developments, AWS Inferentia2 and Hugging Face are set to redefine the landscape of machine learning, making it more accessible and efficient for users worldwide.
Inspired by: Source
