LiteLLM has two related but distinct forms. The Python SDK translates calls to OpenAI, Anthropic, Vertex AI, Bedrock, Azure, Ollama, vLLM and many other providers into a more consistent interface and response shape. LiteLLM Proxy is a self-hosted AI Gateway exposing OpenAI-compatible endpoints with central routing and governance. The SDK reduces provider-specific application code; the Proxy is for platform teams issuing virtual keys, budgets, rate limits, fallbacks, logs and guardrails. LiteLLM supplies neither model compute nor perfect feature equivalence. Its 2026 documentation also limits support to the four latest stable minor lines, so production users need pinned versions, upgrade tests and current model metadata.
Quick Verdict
- Best for: developers needing multi-provider adaptation and platform teams operating an internal model gateway.
- Worth using? Start with the SDK for one application. Add Proxy when multiple teams need shared credentials, policy and accounting, not merely because a unified endpoint sounds convenient.
- Related alternatives: OpenRouter, Ollama, vLLM and Open WebUI.
Best For
The SDK fits Python applications comparing or switching model providers. Proxy fits platform engineering teams with many applications and vendors that need centralized credentials, quotas, model allowlists, logging destinations and incident policies. A small project using one model may be simpler with the vendor SDK. Ollama and vLLM can supply local inference; LiteLLM provides adaptation and governance rather than inference itself.
Key Features
- Python SDK: A common
completion()interface plus embeddings, image and other endpoints, with normalized responses and exceptions. Provider-specific capabilities are not always losslessly portable. - AI Gateway/Proxy: OpenAI-compatible entry point, model aliases, virtual keys, user/team/project budgets, RPM/TPM limits and an administration UI.
- Routing and resilience: Load balancing, retries and fallbacks across deployments. Retryable failures must be classified carefully to avoid duplicate billing or repeated tool effects.
- Cost and observability: Attribute usage by key, team, user, tag or model and export logs/metrics through callbacks, Prometheus and external platforms. Model cost metadata can lag upstream billing.
- Guardrails and gateway extensions: Custom guardrails, PII handling, MCP and A2A gateway features are available. Some built-in moderation, scoped controls and enterprise administration require a commercial license.
Use Cases
- Map an application model alias to different provider or regional deployments without changing every client.
- Issue restricted virtual keys with allowed models, budget, RPM/TPM and expiry instead of distributing upstream secrets.
- Give interactive, batch and high-value tasks distinct routing, timeout, retry and fallback policies.
- Export request metrics and cost while deciding per team whether prompts and responses may be logged.
- Govern MCP and agent access through one endpoint while retaining authorization and human confirmation for tools.
Reliability requires more than enabling fallback. Probe every provider, track success rate, 429/5xx errors, time to first token, P50/P95 latency, route distribution, retry amplification and cost variance. Set an overall deadline and retry budget, and use circuit breakers to contain outages. Once a stream or tool call begins, failover can duplicate output or side effects; applications need idempotency keys and approval boundaries.
Pricing
LiteLLM OSS can be self-hosted, but total cost includes upstream models, Proxy compute, PostgreSQL, optional cache/queue systems, log storage and operations. Enterprise is quoted by usage or scale. Official documentation says Admin UI SSO is free for up to five users; larger SSO use and capabilities such as audit logs, fine-grained governance, multi-region deployment and professional support require a license.
| Option | Cost | Suitable for |
|---|---|---|
| Python SDK | Free open-source software; upstream usage separate | Application-level provider adaptation |
| OSS Proxy | Free open-source software; infrastructure and providers separate | Self-hosted gateway and baseline governance |
| Enterprise | Seven-day trial path; contact sales for production pricing | SSO/SCIM, audit, granular RBAC, multi-region and support |
Check current boundaries in the Enterprise documentation. Releases move quickly; pin images and dependencies, retain rollback capability and test before a minor line leaves the four-line support window.
Pros
- Clear progression from an embedded SDK to an organization-wide gateway.
- Broad provider support and OpenAI-client compatibility reduce repeated integration work.
- OSS includes virtual keys, budgets, routing, logging and a basic guardrail framework.
- Self-hosting gives control over secrets, logging location and network boundaries.
- Routing, fallback and aliases can reduce dependence on one deployment.
Cons
- OpenAI compatibility does not make tools, streaming, structured output and new parameters equivalent across providers.
- A central Proxy needs multiple instances, database recovery, rate limiting and capacity exercises.
- Retries and fallbacks can increase spend or repeat requests with side effects.
- Some enterprise guardrails, SSO, auditing and multi-region controls require a paid license.
- Fast-changing model prices, context limits and capability metadata can diverge from upstream reality.
Alternatives
| Tool | Better for | Advantage | Tradeoff |
|---|---|---|---|
| OpenRouter | Developers wanting a managed unified API | Broad model access without running a gateway | Traffic and governance use a third-party service |
| Ollama | Local model users | Straightforward local runtime | Not equivalent organization-wide multi-provider governance |
| vLLM | High-throughput self-hosted inference | Strong serving performance | Not a full budget and virtual-key gateway |
| Open WebUI | Self-hosted chat users | Complete user-facing model interface | Not primarily low-level API governance |
FAQ
What is the difference between LiteLLM SDK and Proxy?
The SDK runs inside a Python application to adapt calls. Proxy is a separate service providing central routing, keys and governance to many clients.
Does LiteLLM host models?
No. It forwards requests to configured cloud or self-hosted inference endpoints. Upstream cost and availability remain separate.
Can an application migrate with zero code changes?
OpenAI-compatible clients may only need a base URL and key change, but model parameters, tools, error behavior and output quality still require testing.
Can fallback cause duplicate charges?
Yes. An upstream may process a request before the gateway observes a timeout. A retry can bill again and tool calls can repeat side effects.
Does a production Proxy need PostgreSQL?
Full deployments that persist virtual keys, budgets and management state generally use a database. Confirm current dependencies in official deployment documentation rather than promoting a single-process demo.
How should cost records be verified?
Reconcile LiteLLM records with provider invoices by model, region and time, restrict unknown models conservatively and keep the official model cost map current.
Bottom Line
LiteLLM is an adaptation layer plus governance gateway, not a model marketplace or inference engine. The SDK lowers multi-provider integration work, and Proxy centralizes budgets, credentials, routing and logs, while creating a central infrastructure responsibility. This page was verified on 2026-07-15 against the official quickstart, Enterprise documentation and GitHub repository. Unsupported fixed performance and customer-adoption claims were removed.