Verified project record

karpathy/nanochat

nanochat is a minimal, hackable codebase that trains a GPT-2 capability LLM end to end for under ~$100 and lets you chat with the trained model over a CLI. It covers all major LLM stages, including tokenization, pretraining, finetuning, evaluation, and inference.

Project overview

It condenses the full ChatGPT-style training pipeline—normally expensive and dependent on complex framework code—into a small harness whose single-file speedrun script trains a GPT-2 grade model on an 8XH100 node in about 1.5-2 hours.

Project type: Model Development · Model Runtime
Use cases: Chat Assistants
Deployment: Refer to project documentation
License: MIT

Best for

Researchers and AI engineers who want a minimal, hackable baseline codebase for studying end-to-end LLM training.
Users who want to train a GPT-2 capability LLM for under ~$100 and interact with it over a CLI.

Key capabilities

Covers all major LLM stages including tokenization, pretraining, finetuning, evaluation, and inference.
A single-file pipeline (runs/speedrun.sh) trains a GPT-2 grade model on an 8XH100 node in about 1.5-2 hours.
Talk to the trained model via python -m scripts.chat_cli.

Limitations and risks

GPUs with less than 80GB require tuning hyperparameters or you will run out of VRAM.
Running on CPU/MPS shrinks the model drastically and you will not get strong results.
The speedrun is designed for an 8XH100 GPU node; a single GPU is possible but roughly 8x slower, and GPUs under 80GB need hyperparameter tuning.

Getting started

Setup requires uv, a GPU node for the full pipeline, and CLI usage. First success path: uv sync --extra gpu, source .venv/bin/activate, bash runs/speedrun.sh, then python -m scripts.chat_cli.

Evidence and sources

README: nanochat is the simplest experimental harness for training LLMs. It is designed to run on a single GPU node, the code is minimal/hackable, and it covers all major LLM stages inclu…
README: The entire pipeline to do so is contained in the single file runs/speedrun.sh, which is designed to be run on an 8XH100 GPU node.
README: Once it's done, you can talk to your model over the CLI. Make sure again that your local uv virtual environment is active (run `source .venv/bin/activate`), and chat: ```bash pyth…
README: change dataset to NVIDIA ClimbMix
README: nanochat uses [uv](https://docs.astral.sh/uv/) for dependency management. To install: ```bash uv sync --extra gpu # Use for CUDA (A100/H100/etc.) ```

Source updated at: 2026-11-02T18:43:26Z

Open original GitHub project