BitNet is a Microsoft open-source framework of optimized kernels for fast, lossless inference of 1.58-bit (ternary) LLMs on CPU and GPU. It builds on the llama.cpp framework so these models can run on local devices with speedups and energy reductions.
Project overview
It targets a concrete cost in local LLM use—full-precision weights driving slow, energy-hungry inference—with a kernel suite that runs ternary models on ordinary CPUs as well as GPUs, plus a documented setup-and-quantization pipeline from Hugging Face checkpoints.
Project type
Model Runtime
Use cases
Chat Assistants
Deployment
Refer to project documentation
License
MIT
Best for
Developers and AI engineers who want to run 1.58-bit (ternary) LLMs locally on CPU or GPU with fast, lossless inference.
Users comfortable building from source and using a command-line workflow, including cloning the repo and setting up a model.
Key capabilities
A suite of optimized kernels supporting fast, lossless inference of 1.58-bit models on CPU and GPU.
Run inference on a quantized GGUF model with a prompt, optionally enabling conversational chat mode for instruct models.
setup_env.py downloads a model from a Hugging Face repo, sets up the environment and quantizes to i2_s or tl1; safetensors checkpoints can be converted to gguf.
Limitations and risks
For inference of general low-bit LLMs beyond ternary models, the project recommends using T-MAC.
The build can fail with errors building llama.cpp due to issues with std::chrono in log.cpp; check for this before adopting.
Getting started
Clone the repository with git clone --recursive https://github.com/microsoft/BitNet.git, then run python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s to set up and quantize the model, and run python run_inference.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -p "You are a helpful assistant" -cnv. Setup requires installing Python, cmake and a clang toolchain, then building from source.
Evidence and sources
GitHub project description: Official inference framework for 1-bit LLMs
README: 10/17/2024: 📣 bitnet.cpp 1.0 released.
README: bitnet.cpp is the official inference framework for 1-bit LLMs (e.g., BitNet b1.58). It offers a suite of optimized kernels that support **fast** and **lossless** inference of 1.58…
README: git clone --recursive https://github.com/microsoft/BitNet.git cd BitNet