Verified project record

OpenRLHF/OpenRLHF

openrlhf is an open-source framework for high-performance, scalable reinforcement learning from human feedback (RLHF) and agentic RL training. It uses a unified agent-based design paradigm built on Ray and vLLM distributed architecture to train large language and vision-language models.

Project overview

As the first production-ready open-source RLHF framework built on the Ray and vLLM distributed architecture, it provides an efficient, unified agent-based paradigm for large-scale model training.

Project type: AI Agent · Model Development · Model Runtime · Infrastructure
Deployment: Refer to project documentation
License: Apache-2.0

Best for

AI engineers who need to scale RLHF training for large language and vision-language models using distributed multi-GPU environments.
Researchers seeking a framework with multiple advanced RL algorithms like PPO, REINFORCE++, GRPO, and RLOO.

Key capabilities

Implements multiple advanced RL algorithms including PPO, REINFORCE++, REINFORCE++-baseline, GRPO, and RLOO, decoupled from agent execution modes.
Implements a unified agent-based paradigm with single-turn and multi-turn execution modes using a token-in-token-out pipeline, supporting custom reward functions and environments.
Supports training Vision-Language Models (VLMs) with image inputs in both single-turn and multi-turn RL training.

Limitations and risks

Requires a GPU for training and inference.
Requires multi-GPU configurations for distributed training and vLLM inference.
Requires pretrained language or vision-language models to operate.

Getting started

Setup is hard and requires distributed multi-GPU environment setup utilizing Ray, vLLM, and DeepSpeed.
To begin, launch a Docker container with NVIDIA runtime, install openrlhf via pip, start the Ray master node, prepare your dataset, and run the training CLI.

Evidence and sources

README: OpenRLHF is **the first** high-performance, production-ready open-source RLHF framework that combines **Ray + vLLM distributed architecture** with a **unified agent-based design p…
GitHub project description: An Easy-to-use, Scalable and High-performance Agentic RL Framework based on Ray (PPO & DAPO & REINFORCE++ & VLM & TIS & vLLM & Ray & Async RL)
README: OpenRLHF is **the first RLHF framework** built on Ray + vLLM distributed architecture, orchestrating multiple components across GPUs efficiently:
README: | **SFT** | [train_sft.sh](./examples/scripts/train_sft.sh) | Supervised fine-tuning with packing |
README: OpenRLHF implements **PPO, REINFORCE++, REINFORCE++-baseline, GRPO, RLOO** with advanced optimization tricks inspired by practical guides and community best practices.

Reviewed at: 2026-07-18T03:22:43Z