karpathy/autoresearch

Project overview

This project provides an autonomous research loop where an AI agent modifies training code, runs short experiments, evaluates results, and iterates automatically to improve a small LLM. It requires a single NVIDIA GPU and uses a fixed 5-minute training budget per iteration.

Project type

AI Agent · Model Development · Infrastructure

Deployment

Refer to project documentation

License

License pending

Best for

• AI engineers and researchers seeking to automate repetitive training code modifications and evaluations on a single GPU.

Key capabilities

• Allows an AI agent to autonomously edit training code, run 5-minute training experiments, evaluate validation metrics, and keep or discard changes iteratively.
• Ships a simplified single-GPU implementation of nanochat (GPT model, Muon + AdamW optimizer) used as the training setup.
• Provides a baseline Markdown file that serves as a lightweight 'skill' providing context and instructions to the AI agent.
• Includes a prepare.py script to download training data, train a BPE tokenizer, and provide runtime utilities.
• Standardizes experiments with a 5-minute wall-clock training budget and vocab-size-independent validation bits per byte (val_bpb) metric for comparison.

Limitations and risks

• Currently requires a single NVIDIA GPU; no built-in support for CPU, MPS, or AMD platforms.
• The fixed time budget makes results comparable on the same compute platform but not comparable to runs on other compute platforms.
• The AI agent only modifies a single train.py file to keep the scope manageable and diffs reviewable.

Getting started

• Install the uv project manager, sync dependencies, run the data preparation script to download data and train a tokenizer, and manually test a single training experiment before starting the autonomous loop.

Evidence and sources

• README: give an AI agent a small but real LLM training setup and let it experiment autonomously overnight. It modifies the code, trains for 5 minutes, checks if the result improved, keeps…
• README: train.py — the single file the agent edits. Contains the full GPT model, optimizer (Muon + AdamW), and training loop.
• README: Requirements: A single NVIDIA GPU (tested on H100)
• README: The `program.md` file is essentially a super lightweight "skill".
• README: prepare.py — fixed constants, one-time data prep (downloads training data, trains a BPE tokenizer), and runtime utilities (dataloader, evaluation).

Reviewed at: 2026-07-22T00:02:52Z