rasbt/LLMs-from-scratch

Project overview

A step-by-step educational repository that teaches developers how to code a GPT-like large language model from the ground up using plain PyTorch. The materials cover the full pipeline from text data preparation to attention mechanisms, pretraining, and finetuning.

Project type

Model Runtime

Use cases

Learning & Education

Deployment

Refer to project documentation

License

License pending

Best for

• Developers, educators, and researchers seeking to understand how large language models work internally through guided, step-by-step PyTorch implementation.

Key capabilities

• Provides step-by-step Jupyter notebooks and Python code for implementing a GPT model from scratch, covering text data processing, attention mechanisms, pretraining, and finetuning.

Limitations and risks

• The project cannot accept contributions that extend or modify the contents of the main chapter code to avoid deviations from the physical book.

Getting started

• Clone the repository, set up the local Python environment following the setup instructions, and run the chapter Jupyter notebooks. A GPU is optional and the project runs on a conventional laptop.

Evidence and sources

• README: This repository contains the code for developing, pretraining, and finetuning a GPT-like LLM and is the official code repository for the book [Build a Large Language Model (From S…
• README: [](https://github.com/rasbt/LLMs-from-scratch/actions/workflows…
• README: [](https://github.com/rasbt/LLMs-from-scratch/actions/w…
• README: [](https://github.com/rasbt/LLMs-from-scratch/actions/workflows…
• README: This book uses PyTorch to implement the code from scratch without using any external LLM libraries.

Reviewed at: 2026-07-17T04:22:14Z