PowerInfer is a local LLM-serving application that uses activation sparsity to optimize hybrid CPU/GPU utilization on a single consumer-grade GPU. It is designed to address GPU-memory and CPU-GPU transfer constraints that can slow local LLM inference.
Project overview
PowerInfer reports up to 11x speedup over llama.cpp on a single consumer-grade GPU while reducing GPU memory demands through hybrid CPU/GPU inference.
Project type
Model Runtime · Infrastructure
Use cases
Chat Assistants
Deployment
Refer to project documentation
License
MIT
Best for
Developers, AI engineers, and researchers who can compile from source and run local LLM inference from the command line.
Users with compatible ReLU-sparse GGUF model weights who want hybrid CPU/GPU inference on a single consumer-grade GPU.
Key capabilities
Keeps hot neurons on the GPU and computes cold neurons on the CPU by exploiting activation locality.
Provides backward compatibility with llama.cpp model weights and examples including server and batched generation.
Provides optimized quantization support for INT4 (Q4_0) models.
Lets users limit GPU VRAM usage during inference.
A Metal backend for sparse inference on macOS is planned.
Support for running inference across multiple GPUs is planned.
Integration with text-generation-webui is planned.
Supports AMD devices with ROCm.
Limitations and risks
Only models with ReLU, ReGLU, or Squared ReLU activation functions are supported.
Performance improvement is not significant on CPU-only macOS systems with Apple M chips.
Some fine-tuned ReLU models, such as the 70B model, may have noticeable performance downgrades because of insufficient retraining tokens.
Plan for the possibility of CUDA_ERROR_OUT_OF_MEMORY during use.
Getting started
Install CMake and Python dependencies, clone the repository and install requirements, build with CMake for GPU or CPU, then run ./build/bin/main with a model path and prompt.
Use ReLU-sparse model weights in GGUF format.
Evidence and sources
GitHub project description: High-speed Large Language Model Serving for Local Deployment
README: [2024/5/17] We now provide support for AMD devices with ROCm.
README: PowerInfer exploits such an insight to design a GPU-CPU hybrid inference engine: hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons are…
README: Using `CMake`(3.17+): * If you have an NVIDIA GPU: ```bash cmake -S . -B build -DLLAMA_CUBLAS=ON cmake --build build --config Release ```
README: Local Deployment Ease: Designed and deeply optimized for local deployment on consumer-grade hardware, enabling low-latency LLM inference and serving on a single GPU.