Verified project record

karpathy/autoresearch

An autonomous research loop in which an AI agent edits single-file Python training code, runs short 5-minute experiments on a single NVIDIA GPU, and iteratively keeps or discards changes based on a standardized validation metric.

Project overview

Demonstrates a focused, automated overnight experimentation loop for small LLM training using a fixed time budget and a standardized metric to keep iterations comparable.

Project type: AI Agent · Model Development · Infrastructure
Deployment: Refer to project documentation
License: License pending

Best for

AI engineers and researchers running automated, single-file training code iterations on a single NVIDIA GPU.
Users needing an autonomous loop that evaluates changes using a fixed 5-minute time budget and validation bits per byte metric on the same compute platform.

Key capabilities

An AI agent autonomously edits training code, runs 5-minute training experiments, evaluates validation metrics, and keeps or discards changes iteratively.
Ships a simplified single-GPU implementation of nanochat (GPT model, Muon + AdamW optimizer) as the training setup.
Provides a baseline Markdown file that serves as a lightweight skill, giving context and instructions to the AI agent.
Includes a prepare.py script to download training data, train a BPE tokenizer, and provide runtime utilities.
Standardizes experiments with a 5-minute wall-clock training budget and a vocab-size-independent validation bits per byte (val_bpb) metric for comparison.

Limitations and risks

Currently requires a single NVIDIA GPU. There is no built-in support for CPU, MPS, or AMD platforms.
The fixed 5-minute time budget allows results to be comparable on the same compute platform, but results are not comparable to runs on other hardware configurations.
The AI agent only modifies a single train.py file to keep changes manageable and diffs reviewable.

Getting started

Install the uv project manager, sync dependencies, and run the one-time data preparation script to download data and train the BPE tokenizer. Expect medium setup difficulty due to the NVIDIA GPU requirement and multiple installation steps.
Manually run a single training experiment to verify the environment and test the training loop before starting the autonomous agent.

Alternatives and comparisons

An all-in-one framework for full-parameter and lightweight fine-tuning, evaluation, and deployment of LLMs and multimodal LLMs, including multi-GPU cluster support via Megatron parallelism.
A Python framework for training multi-step LLM agents using GRPO reinforcement learning, supporting deployment from local GPUs to managed serverless backends.
A hands-on LLM programming tutorial series covering model fine-tuning, deployment, and prompt-based reasoning in educational notebooks.

Project comparisons

Evidence and sources

README: give an AI agent a small but real LLM training setup and let it experiment autonomously overnight. It modifies the code, trains for 5 minutes, checks if the result improved, keeps…
README: train.py — the single file the agent edits. Contains the full GPT model, optimizer (Muon + AdamW), and training loop.
README: Requirements: A single NVIDIA GPU (tested on H100)
README: The `program.md` file is essentially a super lightweight "skill".
README: prepare.py — fixed constants, one-time data prep (downloads training data, trains a BPE tokenizer), and runtime utilities (dataloader, evaluation).

Reviewed at: 2026-07-22T00:02:52Z

Open original GitHub project