Verified project record

lyogavin/airllm

AirLLM enables 70B-parameter Hugging Face language models to run on a single 4GB GPU by loading only one model layer at a time. This memory reduction is achieved without quantization, distillation, or pruning, but inference speed is bottlenecked by disk loading.

Project overview

The project lowers the VRAM barrier for running 70B models to approximately 4GB by keeping only a single layer on the GPU at any point during inference.

Project type: Model Runtime · Infrastructure · Model Development
Use cases: Chat Assistants
Deployment: Refer to project documentation
License: Apache-2.0

Best for

AI engineers, researchers, and developers needing to execute large models locally on hardware limited to approximately 4GB of GPU VRAM.

Key capabilities

Loads and computes only one layer of the model on the GPU at a time, making VRAM requirements depend on the model's layer size rather than total size.
Automatically detects model type and initializes it using AutoModel without needing to specify the model class manually.
Provides block-wise quantization-based model compression (4bit/8bit) to speed up inference by up to 3x with almost ignorable accuracy loss.
Overlaps model loading and compute operations via prefetching for a 10% speed improvement.
Allows running inference on CPU and supports non-sharded models.

Limitations and risks

Inference speed is primarily bottlenecked by disk loading because only one layer is kept on the GPU at a time.
The process of splitting the model into layers is very disk-consuming.
Only Apple silicon hardware is supported for MacOS deployments.
Disk exhaustion during model splitting can result in MetadataIncompleteBuffer errors.

Getting started

Install the library via pip (pip install airllm), initialize a model using AutoModel.from_pretrained('model_id'), and call model.generate() to produce text. Setup requires Python coding and sufficient disk space for model layer splitting.

Alternatives and comparisons

Enables LLM inference with minimal setup and high performance across a wide range of hardware via integer quantization.
Optimizes CPU and GPU usage for MoE models using heterogeneous computing to reduce hardware requirements.
Provides a lightweight deep learning inference framework with an LLM runtime to deploy models locally on constrained devices.

Project comparisons

Evidence and sources

README: First, install the airllm pip package. ```bash pip install airllm ```
README: model = AutoModel.from_pretrained("Qwen/Qwen3-32B")
GitHub project description: AirLLM 70B inference with single 4GB GPU
README: AirLLM: scaling large language models on low-end commodity computers
README: The trick: AirLLM only ever keeps **one layer on the GPU at a time**, so the VRAM you need depends on the model's layer size — not its total size.

Source updated at: 2026-07-16T18:17:49Z

Open original GitHub project