BerriAI/litellm

Project overview

It standardizes communication across 100+ LLM providers into a single OpenAI-compatible format while adding production features like load balancing, fallback logic, and an admin dashboard.

Project type

MCP · Model Runtime · Evaluation & Observability · Infrastructure

Deployment

Python environment

License

License pending

Best for

• Enterprise teams, developers, and AI engineers who need to standardize API calls, manage costs, and apply consistent routing rules across multiple external LLM providers.
• Teams requiring an admin dashboard and virtual keys to manage multi-tenant LLM usage.

Key capabilities

• Provides a single interface to call 100+ LLM providers using the OpenAI format via either a Python SDK or a Proxy Server.
• Offers centralized API gateway authentication, virtual keys for access control, multi-tenant cost tracking, and an admin dashboard UI.
• Production-ready gateway features providing load balancing, auto routing, and retry/fallback logic across multiple deployments.
• Provides production-ready guardrails, caching, logging, and observability callbacks for LLM calls.
• Allows invoking A2A (Agent-to-Agent) agents via the Python SDK or the AI Gateway proxy server.
• Allows connecting Model Context Protocol (MCP) servers to any LLM, exposing MCP tools in OpenAI format via the SDK or Gateway.

Limitations and risks

• GCP Cloud Run cannot pull directly from ghcr.io, requiring a one-time Artifact Registry remote repository setup.
• The gateway does not directly host or serve open-source models itself, but integrates with endpoints that do, such as HuggingFace, vLLM, or SageMaker.

Getting started

• Install easily via uv using 'uv add litellm' or 'uv tool install'.
• After installation via uv, you can run the proxy server or import the library to start making LLM calls.

Alternatives and comparisons

• Provides OpenAI-compatible routing, load balancing, virtual keys, and failover without persisting prompt or response content.
• Packages predictive or generative models into production APIs with built-in chat UI and MCP server capabilities.
• Aggregates access to various LLM and media generation providers via OpenAI-compatible REST APIs and clients.

Project comparisons

• BerriAI/litellm vs Self-hosted privacy-first LLM proxy
• BerriAI/litellm vs Model serving and API packaging
• BerriAI/litellm vs Multi-provider LLM aggregation

Evidence and sources

• GitHub project description: Python SDK, Proxy Server (AI Gateway) to call 100+ LLM APIs in OpenAI (or native) format, with cost tracking, guardrails, loadbalancing and logging. [Bedrock, Azure, OpenAI, Verte…
• README: Open Source AI Gateway for 100+ LLMs. Self-hosted. Enterprise-ready. Call any LLM in OpenAI format.
• README: LiteLLM is an open source AI Gateway that gives you a single, unified interface to call 100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure, and more — using the OpenAI…
• README: Managing LLM calls across providers gets complicated fast — different SDKs, auth patterns, request formats, and error types for every model. LiteLLM removes that friction:
• README: **Production-ready gateway** — virtual keys, spend tracking, guardrails, load balancing, and an admin dashboard out of the box - **8ms P95 latency** at 1k RPS ([benchmarks](https:…

Reviewed at: 2026-07-17T06:30:35Z

Open original GitHub project