Verified project record
BerriAI/litellm
LiteLLM provides a unified gateway and Python SDK for calling 100+ LLM providers using the OpenAI format, addressing the operational complexity of managing disparate SDKs and authentication patterns. It includes production-ready features such as load balancing, virtual keys, cost tracking, guardrails, and observability callbacks, requiring configuration of external or self-hosted LLM endpoints rather than hosting models itself.
Project overview
LiteLLM provides a unified gateway and Python SDK for calling 100+ LLM providers using the OpenAI format, addressing the operational complexity of managing disparate SDKs and authentication patterns. It includes production-ready features such as load balancing, virtual keys, cost tracking, guardrails, and observability callbacks, requiring configuration of external or self-hosted LLM endpoints rather than hosting models itself.
- Project type
- MCP · Model Runtime · Infrastructure
- Deployment
- Python environment
- License
- License pending
Best for
- Enterprise teams, developers, and AI engineers who need to unify access to 100+ LLM providers behind a single OpenAI-compatible interface.
- Teams requiring production-ready gateway features such as load balancing, retry/fallback logic, and multi-tenant cost tracking.
Key capabilities
- Provides a single, unified interface to call 100+ LLM providers (OpenAI, Anthropic, Gemini, Bedrock, Azure, etc.) using the OpenAI format via either a Python SDK or a Proxy Server.
- Offers centralized API gateway authentication, virtual keys for secure access control, multi-tenant cost tracking, and an admin dashboard UI for monitoring and management.
- Production-ready gateway features providing load balancing, auto routing, and retry/fallback logic across multiple deployments.
- Provides production-ready guardrails, caching, logging, and observability callbacks (Lunary, MLflow, Langfuse) for LLM calls.
- Allows invoking A2A (Agent-to-Agent) agents via the Python SDK or the AI Gateway proxy server.
- Allows connecting Model Context Protocol (MCP) servers to any LLM through the Python SDK or the AI Gateway, exposing MCP tools in OpenAI format.
Limitations and risks
- GCP Cloud Run cannot pull directly from ghcr.io, requiring a one-time Artifact Registry remote repository setup.
Getting started
- The setup difficulty is rated as easy. Users can install via one-command installs like 'uv add litellm' or 'uv tool install'. The first success path involves installing via uv and then running the proxy server or importing the library.
Evidence and sources
- GitHub project description: Python SDK, Proxy Server (AI Gateway) to call 100+ LLM APIs in OpenAI (or native) format, with cost tracking, guardrails, loadbalancing and logging. [Bedrock, Azure, OpenAI, Verte…
- README: Open Source AI Gateway for 100+ LLMs. Self-hosted. Enterprise-ready. Call any LLM in OpenAI format.
- README: LiteLLM is an open source AI Gateway that gives you a single, unified interface to call 100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure, and more — using the OpenAI…
- README: Managing LLM calls across providers gets complicated fast — different SDKs, auth patterns, request formats, and error types for every model. LiteLLM removes that friction:
- README: **Production-ready gateway** — virtual keys, spend tracking, guardrails, load balancing, and an admin dashboard out of the box - **8ms P95 latency** at 1k RPS ([benchmarks](https:…
AI 搜索
把需求说清楚，让项目选择更有依据
告诉我们你要解决什么、运行在哪里、哪些条件不能妥协。雷达会从已核验项目中给出主推荐、备选和采用前检查。