An open-source educational project that enables individuals to train and understand a 64M-parameter LLM from scratch in approximately 2 hours using native PyTorch. It covers the complete model training lifecycle and provides open-source data, model architectures, and inference tools.
Project overview
The project implements all core LLM training algorithms from scratch without high-level abstractions, providing developers a direct way to understand and run the complete training pipeline.
Project type
Infrastructure
Deployment
Refer to project documentation
License
Apache-2.0
Best for
Developers, educators, and researchers who want to inspect and run unabstracted PyTorch code to learn how LLMs are trained from scratch.
Key capabilities
Covers the complete training process including Pretrain, SFT, LoRA, RLHF-DPO, RLAIF, Tool Use, Agentic RL, and model distillation, implemented from scratch in PyTorch.
Provides complete LLM structure code for Dense and MoE models, aligned with the Qwen3/Qwen3-MoE ecosystem, including RoPE and YaRN scaling.
Provides a minimal server compatible with OpenAI API protocols, supporting reasoning_content, tool_calls, and open_thinking for integration with third-party UIs.
Provides a minimal Streamlit-based web chat UI supporting thought display, tool selection, and multi-turn tool calls.
Provides full-stage open-source data, covering collection, distillation, cleaning, and deduplication of high-quality datasets.
Limitations and risks
The trained models have limited factual knowledge and generalization capabilities due to their small scale.
Getting started
Setup requires GPU familiarity, command-line execution, and manual data downloading. The first success path involves cloning the repository, installing dependencies, downloading desired model weights, and running CLI inference using a command such as python eval_llm.py --load_from ./minimind-3.
Alternatives and comparisons
A hands-on educational repository and book companion focused on developing a small reasoning LLM and understanding reasoning techniques from scratch.
A Chinese translation of the LLMs-from-scratch project providing detailed notes and annotated notebooks to help Chinese learners build LLMs step-by-step.
A structured educational repository offering learning paths and practical guides covering the complete AI infrastructure stack from GPU architecture to LLM training and deployment.