lyogavin/airllm

Project overview

AirLLM enables inference of large language models (up to 70B parameters) on GPUs with as little as 4GB VRAM. It achieves this by loading only one model layer onto the GPU at a time without requiring quantization, which trades memory savings for significantly slower speeds due to disk loading.

Project type

Model Runtime · Infrastructure

Use cases

Chat Assistants

Deployment

Refer to project documentation

License

Apache-2.0

Best for

• AI engineers and researchers who need to run or test very large Hugging Face open LLMs (e.g., Llama 3.x, Qwen, DeepSeek-V3) on hardware limited to around 4GB of GPU VRAM.

Key capabilities

• Loads and computes only one layer of the model on the GPU at a time, making VRAM requirements depend on the model's layer size rather than total size.
• Automatically detects model type and initializes it using AutoModel without needing to specify the model class manually.
• Provides block-wise quantization-based model compression (4bit/8bit) to speed up inference by up to 3x with almost ignorable accuracy loss.
• Overlaps model loading and compute operations via prefetching for a 10% speed improvement.
• Allows running inference on CPU and supports non-sharded models.

Limitations and risks

• Inference speed is bottlenecked primarily by disk loading because only one layer is kept on the GPU at a time.
• The process of splitting the model into layers is very disk-consuming.
• Only Apple silicon is supported for MacOS deployments.
• Disk exhaustion may occur during model splitting, which can result in a MetadataIncompleteBuffer error.

Getting started

• Install via 'pip install airllm', initialize the model using AutoModel.from_pretrained('model_id'), and generate text using model.generate(). Requires Python coding and sufficient disk space for layer splitting.

Evidence and sources

• README: First, install the airllm pip package. ```bash pip install airllm ```
• README: model = AutoModel.from_pretrained("Qwen/Qwen3-32B")
• GitHub project description: AirLLM 70B inference with single 4GB GPU
• README: AirLLM: scaling large language models on low-end commodity computers
• README: The trick: AirLLM only ever keeps **one layer on the GPU at a time**, so the VRAM you need depends on the model's layer size — not its total size.

Reviewed at: 2026-07-17T11:22:58Z