Verified project record

lucidrains/PaLM-rlhf-pytorch

palm-rlhf-pytorch is an open-source library that implements Reinforcement Learning with Human Feedback on top of the PaLM architecture, providing the components needed to train models with human feedback. It includes the PaLM architecture, a reward model, and an RLHF trainer designed to replicate systems like ChatGPT.

Project overview

It offers an open-source blueprint for reproducing ChatGPT-style training by combining the PaLM architecture with reward model training and a reinforcement learning trainer.

Project type: Model Development · Model Runtime · Infrastructure
Deployment: Refer to project documentation
License: MIT

Best for

AI engineers and researchers who want an open-source implementation of RLHF on top of the PaLM architecture to train models with human feedback.

Key capabilities

Implements the base PaLM autoregressive transformer for training and sequence generation.
Trains a reward model using curated human feedback, offering options for LoRA finetuning.
Reinforcement Learning trainer that utilizes a pretrained PaLM and reward model to generate optimized samples.
Incorporates Flash Attention to optimize the attention mechanism.
Simple web interface with django + htmx for collecting human feedback.
Integration of Hugging Face accelerate and wandb for logging and instrumentation.

Limitations and risks

No trained model is included; the project provides only the architecture and training pipeline.
Millions of dollars of compute and data are required to reach the correct point in high-dimensional parameter space.

Getting started

Run pip install palm-rlhf-pytorch, then instantiate and train PaLM. Meaningful use requires substantial coding and compute beyond the basic pip installation.

Evidence and sources

GitHub project description: Implementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architecture. Basically ChatGPT but with PaLM
README: Implementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architecture.
README: If you are interested in replicating something like ChatGPT out in the open
README: There is no trained model. This is just the ship and overall map. We still need millions of dollars of compute + data to sail to the correct point in high dimensional parameter sp…
README: palm = PaLM( num_tokens = 20000, dim = 512, depth = 12, flash_attn = True # https://arxiv.org/abs/2205.14135 ).cuda() seq = torch.randint(0, 20000, (1, 2048)).cuda() loss = palm(s…

Reviewed at: 2026-07-18T05:47:17Z