FareedKhan-dev/train-llm-from-scratch

Project overview

The project provides hand-written implementations of core transformer components and training loops, offering a transparent view of the mechanics behind pretraining, supervised fine-tuning, reward modeling, and alignment.

Project type

Model Development · Infrastructure · Learning Resources

Use cases

Learning & Education

Deployment

Refer to project documentation

License

MIT

Best for

• Researchers, developers, and educators who require an educational walkthrough of the complete LLM training lifecycle using hand-written PyTorch code.
• Users who want to run the tutorial pipeline on accessible compute such as a free Colab or Kaggle T4 GPU for the 13 million parameter model.

Key capabilities

• Builds a transformer model from small pieces (MLP, attention, blocks) in plain PyTorch without using trl, peft, or transformers.
• Implements the pretraining loop to train a base model by predicting the next token, including DistributedDataParallel and gradient accumulation.
• Implements post-training techniques to turn a base model into an assistant, including SFT, reward model training, and alignment methods.
• Provides scripts to generate text from a trained model checkpoint by predicting the next token.
• Ships a Streamlit control panel to talk to the trained model.
• Provides scripts for downloading, tokenizing, and preparing data streams for pretraining, SFT, preferences, and RL prompts.
• Includes steps and scripts to evaluate the trained models.

Limitations and risks

• A free Colab or Kaggle T4 GPU is sufficient for the 13 million parameter model documented in the tutorial, but it will not fit a billion parameter model.
• The pipeline integrates with Weights & Biases (wandb) for logging, which requires an external service during operation.

Getting started

• Setup requires cloning the repository, installing Python packages in editable mode, and understanding config systems alongside GPU requirements. The path to first success involves cloning, installing, and running the training script.

Alternatives and comparisons

• Provides an autonomous research loop where an AI agent modifies training code, runs short experiments, evaluates results, and iterates automatically to improve a small LLM.

Project comparisons

• FareedKhan-dev/train-llm-from-scratch vs Autonomous Research Loop

Evidence and sources

• GitHub project description: A straightforward method for training your LLM, from downloading data to generating text.
• README: I implemented a transformer model from scratch using PyTorch, based on the paper Attention is All You Need. You can use my scripts to train your own billion or million parameter L…
• README: If you are a developer, the commands and file paths are all here. You can copy, run, and read the referenced source files directly.
• README: The whole journey is one idea repeated: turn text into numbers, predict the next token, then keep changing the data and the loss until the model does what we want.
• README: Here is the path we will walk, end to end: raw text -> tokens -> a Transformer -> next-token loss -> a base model base model -> SFT -> Reward Model -> {PPO, DPO} -> GRPO -> evalua…

Reviewed at: 2026-07-23T17:42:35Z

Open original GitHub project