minbpe provides short, thoroughly commented educational Python implementations of Byte Pair Encoding (BPE) tokenizers used in LLM tokenization. It includes a GPT-4 equivalent tokenizer that can be trained from scratch.
Project overview
Unlike tiktoken, minbpe lets users train their own tokenizer vocabulary and merges while also exactly reproducing GPT-4 tokenization, filling a gap for developers and educators who need clean, hackable BPE code.
Project type
Model Development · Prompt Engineering
Deployment
Refer to project documentation
License
MIT
Best for
Developers and educators who want to understand, inspect, and modify a minimal BPE tokenizer implementation in Python.
Researchers who need to train a custom tokenizer vocabulary from scratch on their own text.
Key capabilities
Runs the simplest implementation of the BPE algorithm directly on text, supporting training, encoding, and decoding.
Splits input text by a regex pattern before tokenization to avoid merges across category boundaries and handles special tokens.
A light wrapper around the RegexTokenizer that exactly reproduces the tokenization of GPT-4 in the tiktoken library.
Allows users to train their own tokenizer vocabulary and merges on a given text.
A more optimized Python version that could run over large files and big vocabs.
An even more optimized C or Rust version of the algorithm.
A LlamaTokenizer similar to GPT4Tokenizer, attempting a sentencepiece equivalent.
Limitations and risks
The current Python implementation is not optimized for large files and big vocabularies; users needing that scale should monitor the planned optimized Python version and the planned C or Rust version.
Getting started
Import Tokenizer from minbpe, initialize a tokenizer class such as BasicTokenizer, then call train to build vocabulary or encode/decode to process text. GPU is not required and all processing is local.
Evidence and sources
GitHub project description: Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
README: 1) train the tokenizer vocabulary and merges on a given text, 2) encode from text to tokens, 3) decode from tokens to text.
README: All of the files above are very short and thoroughly commented, and also contain a usage example on the bottom of the file.
README: We use the pytest library for tests. All of them are located in the `tests/` directory.
README: Implements the `BasicTokenizer`, the simplest implementation of the BPE algorithm that runs directly on text.
AI Search
Find projects, verify facts, compare options, or turn a complex need into an actionable plan
Try a searchA click only fills the search box; you stay in control