Verified project record

getumbrel/llama-gpt

A self-hosted, offline chatbot powered by Llama 2 that operates fully locally to preserve data privacy. It provides a web interface and an OpenAI-compatible API, requiring substantial memory and disk space depending on the selected model size.

Project overview

Project type: Model Runtime
Use cases: Chat Assistants
Deployment: Refer to project documentation
License: MIT

Best for

General users and developers who need a local-only, offline chatbot environment with no paid service requirements.
Systems equipped with optional Nvidia GPUs or Apple Silicon M1/M2 Macs for hardware acceleration.

Key capabilities

A self-hosted, offline chatbot interface that allows users to interact with LLMs privately without data leaving their device.
Provides a drop-in replacement for the OpenAI API at port 3001, enabling integration with existing OpenAI-compatible tools.
Can run with CUDA support for Nvidia GPUs to accelerate model inference.
Supports Apple Silicon Metal framework for M1/M2 Macs.
Ability to load custom models.
Allows users to switch between models in the UI.

Limitations and risks

The system cannot currently load custom models, though this is planned for the future.
Users cannot dynamically switch between models in the UI yet; this is planned for the future.
Model download sizes are large, requiring up to 38.87GB of data for the 70B model.
Memory requirements range from 6.29GB for the 7B model up to 41.37GB for the 70B model.

Getting started

Setup involves medium difficulty, requiring users to clone the repository, run shell scripts for Docker deployment, and download multi-GB models on the first run using the command: ./run.sh --model 7b.

Evidence and sources

GitHub project description: A self-hosted, offline, ChatGPT-like chatbot. Powered by Llama 2. 100% private, with no data leaving your device.
GitHub repository metadata: {"primary_language":"TypeScript"}
README: A self-hosted, offline, ChatGPT-like chatbot, powered by Llama 2. 100% private, with no data leaving your device.
README: Running LlamaGPT on an [umbrelOS](https://umbrel.com) home server is one click. Simply install it from the [Umbrel App Store](https://apps.umbrel.com/app/llama-gpt).
README: Simply install it from the [Umbrel App Store](https://apps.umbrel.com/app/llama-gpt).

Reviewed at: 2026-07-18T02:28:17Z