oobabooga/textgen

Project overview

It provides local execution with zero telemetry and serves as a drop-in local replacement for OpenAI and Anthropic APIs.

Project type

Model Runtime

Deployment

Refer to project documentation

License

AGPL-3.0

Best for

• Developers and general users who want to run large language models on their own hardware without relying on external APIs or cloud services.
• Users who require local execution with zero telemetry during operation.

Key capabilities

• Run text generation models locally with instruct and chat modes, supporting branching and notebook interfaces.
• Attach images to messages for visual understanding during chat.
• Models can call custom functions during chat, including web search, page fetching, and math, defined in single .py files. Supports MCP servers.
• Serves Chat, Completions, and Messages endpoints locally as a drop-in replacement for OpenAI/Anthropic APIs.
• Fine-tune LoRAs on multi-turn chat or raw text datasets.
• A dedicated tab for diffusers models like Z-Image-Turbo with quantization and gallery features.
• Supports built-in and community extensions including TTS, voice input, and translation.

Limitations and risks

• Some model types require the full installation instead of the portable build.
• Full installation requires approximately 10GB of disk space and downloads PyTorch.

Getting started

• Download, unzip, and double-click the textgen executable to open the application window.

Evidence and sources

Open original GitHub project