bitsandbytes is a Python library that reduces memory consumption for PyTorch model inference and training through 8-bit and 4-bit quantization techniques. It targets users with limited hardware resources who need to run or train large language models that would otherwise exceed available memory.
Project overview
LLM.int8() quantization enables inference at half the memory cost without performance degradation, and QLoRA 4-bit quantization opens up large language model training on hardware that could not previously support it.
Project type
Infrastructure
Deployment
Refer to project documentation
License
MIT
Best for
AI engineers and researchers who need to run inference or training on large language models with limited GPU memory.
Developers working within the PyTorch ecosystem who want to reduce model memory consumption through quantization.
Key capabilities
Uses block-wise quantization to maintain 32-bit performance at a small fraction of the memory cost.
Enables large language model inference with only half the required memory and without any performance degradation using vector-wise quantization and 16-bit matrix multiplication for outliers.
Enables large language model training by quantizing a model to 4-bits and inserting a small set of trainable low-rank adaptation (LoRA) weights to allow training with several memory-saving techniques.
Provides quantization primitives for 8-bit and 4-bit operations through bitsandbytes.nn.Linear8bitLt and bitsandbytes.nn.Linear4bit modules.
Limitations and risks
8-bit Optimizers are currently in development for Metal (Apple MPS). Users on Apple Mac platforms should verify optimizer support before relying on this feature.
Getting started
Install the package through a Python package manager (pip). Coding is required to use the library.
Evidence and sources
README: bitsandbytes enables accessible large language models via k-bit quantization for PyTorch. We provide three main features for dramatically reducing memory consumption for inference…
README: The continued maintenance and development of bitsandbytes is made possible thanks to the generous support of our sponsors.
Release: 0.49.2
README: 🤗 Transformers
README: bitsandbytes has the following minimum requirements for all platforms: * Python 3.10+ * PyTorch 2.4+
AI Search
Find projects, verify facts, compare options, or turn a complex need into an actionable plan
ProjectsSkillsFactsCompareRankingsSolutions
Try a searchA click only fills the search box; you stay in control