Integrating Stable-Baselines3 with Hugging Face Hub: A New Era for Reinforcement Learning

At Hugging Face, we’re committed to enhancing the ecosystem for Deep Reinforcement Learning (RL) researchers and enthusiasts. Our latest integration of Stable-Baselines3 (SB3) with the Hugging Face Hub is a significant step forward in this direction. This collaboration provides an easy platform for users to host, share, and utilize RL models, making it easier than ever to train and test agents across various environments, including Gym, Atari, and MuJoCo.

What is Stable-Baselines3?

Stable-Baselines3 is a highly regarded RL library built on PyTorch. It allows researchers and developers to easily train and test RL models using a streamlined approach. With access to a variety of environments, users can experiment with different algorithms and improve their models with minimal overhead. This integration with the Hugging Face Hub opens the door for users to not only save their models but also leverage the work of the community to enhance their projects.

Getting Started: Installation

To start using Stable-Baselines3 with the Hugging Face Hub, you’ll need to install two essential libraries. Open your terminal and run the following commands:

pip install huggingface_hub
pip install huggingface_sb3

These libraries will allow you to seamlessly interact with the Hugging Face Hub and utilize SB3 functionalities.

Finding and Utilizing Models

One of the most exciting aspects of this integration is the ability to access a plethora of pre-trained models. Currently, you can find saved models for agents that have mastered games like Space Invaders, Breakout, and LunarLander. To locate these models, simply visit the Hugging Face Hub and search for the desired model.

More Read

Enhancing Large Language Model Systems Using User Logs: Insights from Paper [2602.06470]

How Selection Format Influences LLM Performance: Insights from Study 2503.06926

Cactus v1: Seamless Cross-Platform LLM Inference for Mobile Devices with Instant Performance and Complete Privacy

Unlocking Scientific Formula Discovery with Multimodal Large Language Models

Cloudflare and ETH Zurich Share Strategies for AI-Enhanced Cache Optimization

Once you find a model that fits your needs, copying the repository ID is straightforward. Here’s a visual guide to help you:

Downloading Models from the Hub

Loading a saved model from the Hugging Face Hub into Stable-Baselines3 is a breeze. You only need two pieces of information: the repository ID of your chosen model and the name of the saved model zip file.

For example, if you’re interested in the CartPole-v1 model, you can use the following code snippet:

import gym
from huggingface_sb3 import load_from_hub
from stable_baselines3 import PPO
from stable_baselines3.common.evaluation import evaluate_policy

checkpoint = load_from_hub(
    repo_id="sb3/demo-hf-CartPole-v1",
    filename="ppo-CartPole-v1.zip",
)
model = PPO.load(checkpoint)

eval_env = gym.make("CartPole-v1")
mean_reward, std_reward = evaluate_policy(
    model, eval_env, render=True, n_eval_episodes=5, deterministic=True, warn=False
)
print(f"mean_reward={mean_reward:.2f} +/- {std_reward}")

This code imports the necessary libraries, loads your model, and evaluates its performance in the CartPole-v1 environment.

Sharing Your Models on the Hub

If you’ve created a model that you’re proud of, sharing it with the community is simple. First, ensure you are logged into Hugging Face. You can log in using the following commands in your Colab or Jupyter Notebook:

from huggingface_hub import notebook_login
notebook_login()

Or, if you prefer the command line:

huggingface-cli login

Next, train your model. Here’s an example of training a PPO agent for CartPole-v1 and pushing it to the Hugging Face Hub:

from huggingface_sb3 import push_to_hub
from stable_baselines3 import PPO

model = PPO("MlpPolicy", "CartPole-v1", verbose=1)
model.learn(total_timesteps=10_000)
model.save("ppo-CartPole-v1")

push_to_hub(
    repo_id="ThomasSimonini/demo-hf-CartPole-v1",
    filename="ppo-CartPole-v1.zip",
    commit_message="Added Cartpole-v1 model trained with PPO",
)

This code snippet demonstrates how to train a model and upload it to your Hugging Face repository, making your work available to the broader community.

Future Developments

Looking ahead, we have exciting plans to expand this integration further. Some upcoming features include:

• Integration with RL-baselines3-zoo: A collection of pre-trained RL agents using Stable-Baselines3.
• Uploading RL-trained agent models: A comprehensive library of pre-trained models will be made available on the Hub.
• Integration with other RL libraries: We plan to incorporate additional libraries to enhance the ecosystem.
• Implementation of Decision Transformers: This cutting-edge development aims to push the boundaries of RL.

To stay updated on these developments and interact with the community, consider joining our Discord server.

Additional Learning Opportunities

For those eager to dive deeper into Deep Reinforcement Learning, we’ve prepared a tutorial that covers:

• Training a Deep Reinforcement Learning agent to land a spacecraft on the Moon.
• Uploading your trained model to the Hub.
• Downloading and utilizing a model that plays Space Invaders.

These resources are designed to help you make the most of the tools and libraries available.

How to Integrate Your Library with the Hub

If you’re interested in integrating your own library with the Hugging Face Hub, the process is facilitated by the huggingface_hub library. This library provides all the necessary widgets and APIs for integration. We offer a detailed guide to help you get started.

This new integration between Stable-Baselines3 and Hugging Face Hub marks a significant advancement for the Deep Reinforcement Learning community. With the ability to share, access, and utilize powerful models, researchers and developers can push the boundaries of what’s possible in RL. We look forward to seeing the innovative projects that emerge from this collaboration!

Inspired by: Source