ggml-org/llama.cpp

Project overview

llama.cpp is a dependency-free C/C++ library for running large language model (LLM) inference locally or in the cloud. It provides tools for model quantization, grammar-constrained generation, and API serving, requiring models to be in the GGUF format.

Project type

Model Runtime · Infrastructure

Use cases

Chat Assistants

Deployment

Refer to project documentation

License

MIT

Best for

• Developers and AI engineers who need a self-hosted, dependency-free library for running LLM inference with optional GPU acceleration.
• Users looking to compress models using integer quantization to run within limited hardware memory constraints.

Key capabilities

• Provides a CLI tool (llama-cli) for accessing most of the library's functionality, including text completion and conversation mode.
• Includes utilities to convert and compress models using 1.5-bit to 8-bit integer quantization, lowering memory requirements and accelerating generation.
• Contains a tool (llama-perplexity) to measure perplexity and other quality metrics of a model over a provided text corpus.
• Features a benchmarking program (llama-bench) to test inference performance across various parameters.
• Processes images alongside text when using compatible multimodal models.
• Constrains model output using custom grammars (GBNF) to enforce specific formats such as JSON.
• Serves embedding and reranking models via specific API server endpoints.

Limitations and risks

• The library requires input models to be stored in the GGUF file format, which may necessitate conversion for models originally stored in other formats.

Getting started

• Installation is rated as easy, supporting pre-built binaries and package managers. Users can achieve first success by downloading binaries, acquiring a GGUF model, and running llama-cli.

Evidence and sources

• GitHub project description: LLM inference in C/C++
• Release: b10025
• README: The main goal of `llama.cpp` is to enable LLM inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.
• README: Multimodal support arrived in `llama-server`: #12898 | documentation
• README: Run with Docker - see our Docker documentation

Reviewed at: 2026-07-17T03:59:51Z