What Is LiteLLM? Unified LLM API Library

If you've shipped an AI feature into production, you've hit the multi-provider problem. OpenAI's SDK expects a different call shape than Anthropic's, which is different from Google's, which is different from Cohere's or AWS Bedrock's. Each has its own error codes, retry rules, and streaming format, so switching models — or just adding a backup — means rewriting integration code by hand.

LiteLLM is an open-source project built by BerriAI that solves this by giving every provider the same face: one OpenAI-compatible API in front of more than 100 LLM providers, plus a proxy server that handles the production plumbing (auth, routing, budgets, logging) that hand-rolled glue code never quite gets right. This piece explains what LiteLLM actually does, how its two operating modes differ, where it beats and loses to alternatives like OpenRouter and Portkey, and what teams get wrong when they adopt it.

What LiteLLM actually does

LiteLLM is not a new model, and it doesn't host any inference itself. It's a translation and orchestration layer that sits between your application code and whichever LLM providers you've chosen. Think of it as a universal power adapter for LLM APIs: the socket on your wall (your app's code) never changes, but the plug on the other end (the provider) can be swapped without touching the wiring in between.

The project ships in two modes, and understanding the difference matters because they solve different problems.

1. Python SDK (library mode)

In library mode, LiteLLM is a drop-in replacement for openai.chat.completions.create(). You import one function, completion(), and call any provider through it with the same argument shape:

from litellm import completion

# OpenAI
r = completion(model="gpt-4o", messages=[{"role": "user", "content": "Hi"}])

# Anthropic
r = completion(model="claude-sonnet-4-5", messages=[{"role": "user", "content": "Hi"}])

# Google Gemini
r = completion(model="gemini-2.5-pro", messages=[{"role": "user", "content": "Hi"}])

# Local model via Ollama
r = completion(model="ollama/llama3.3", messages=[{"role": "user", "content": "Hi"}])

The input shape is identical across all four calls. The output shape is identical too, normalized into OpenAI's response format regardless of which provider actually generated the text. LiteLLM handles the translation of parameters (temperature, max tokens, stop sequences), the different authentication schemes, and the mapping of provider-specific error codes into a consistent exception hierarchy under the hood.

This mode is meant for use inside your own Python codebase — no separate service to deploy, no infrastructure to run. You install a package and swap a model string.

2. Proxy server (gateway mode)

Gateway mode is a different animal. Here, you deploy LiteLLM as a standalone service — typically a Docker container — that sits in front of every provider your organization uses. Your applications talk to this one service using the standard OpenAI API format, and the proxy does the rest:

• Authentication — issues virtual API keys scoped to individual teams, users, or applications, separate from your actual provider keys
• Routing — picks which underlying model or provider handles a given request, with automatic fallback to alternates if the primary fails
• Rate limiting — enforced per virtual key and per model
• Cost tracking — logs token usage and dollar cost for every single call, broken down by key, team, and model
• Budget enforcement — hard caps on spend per team or per key, with the ability to block requests once a budget is exhausted
• Retries and timeouts — configurable per model, with exponential backoff
• Response caching — a Redis-backed cache that can return identical prompts from cache instead of re-calling the provider
• Observability hooks — native integrations with Langfuse, Helicone, Datadog, and OpenTelemetry for tracing every request
• Guardrails — pluggable content-safety and PII-redaction layers, including third-party integrations with Lakera, Aporia, Microsoft Presidio, AWS Bedrock Guardrails, and Azure AI Content Safety, according to LiteLLM's own proxy documentation

LiteLLM reports on its official site that its gateway has been pulled more than 240 million times from Docker Hub and has routed over one billion LLM requests in production, with proxy overhead measured at roughly 0.66 milliseconds at the 99th percentile — a fraction of the network round-trip time to any LLM provider ([litellm.ai](https://www.litellm.ai/)).

The mental model here: gateway mode is a reverse proxy for LLMs, the same architectural role Nginx or Envoy play for HTTP traffic, except the backends are OpenAI, Anthropic, Gemini, and dozens of others instead of your own microservices.

Why teams adopt it

The underlying motivation is vendor lock-in defense. If your codebase calls the OpenAI SDK directly in fifty different places, switching providers — or even switching models within OpenAI's own lineup after a deprecation — becomes a multi-week refactor touching every call site. With LiteLLM sitting in between, the same switch is a one-line config change or an environment variable update.

That abstraction unlocks several concrete workflows that are painful to build from scratch:

• A/B testing models. Route 10% of production traffic to Claude and 90% to GPT, then compare quality, latency, and cost metrics side by side without maintaining two separate code paths.
• Cost-tiered routing. Send simple, high-volume requests to a cheaper, faster model like Claude Haiku or GPT-4o mini, and reserve a larger model like Claude Sonnet or GPT-4o for requests that need more reasoning.
• Automatic failover. If a provider has an outage — and every major LLM provider has had one — the router retries against a configured fallback model instead of surfacing a 500 error to the end user.
• Regional compliance routing. Route EU customer traffic to Bedrock or Azure's EU regions and US traffic to a US-based endpoint, satisfying data-residency requirements without maintaining separate application deployments.
• Centralized billing. One dashboard and one set of logs across every provider, instead of reconciling five separate billing portals at the end of the month.
• Per-team budgets. Give the marketing team a $500 monthly cap and engineering a $5,000 cap, with the proxy blocking further calls once a team hits its ceiling — a feature that's genuinely difficult to replicate by hand across multiple provider dashboards.

Netflix's engineering team has publicly credited LiteLLM with speeding up model rollout: "LiteLLM has enabled my team to deliver the latest models to users within a day of release... saving months of work," according to a testimonial published on LiteLLM's official site. Other organizations listed as users on the same page include NVIDIA, Zapier, IBM, Stripe, and Twilio, though the specifics of each deployment aren't public.

How the routing mechanism actually works

Underneath the simple completion() call sits a router that has to make a real decision every time a request comes in: which deployment handles it, and what happens if that deployment fails. According to LiteLLM's routing documentation, the router supports several selection strategies, and picking the right one changes how the system behaves under load.

What routing strategies does LiteLLM support?

LiteLLM's router can select a deployment using simple round-robin shuffling, least-busy selection (favoring the deployment with the fewest in-flight requests), usage-based routing that accounts for each deployment's requests-per-minute and tokens-per-minute limits, latency-based routing that favors the fastest-responding deployment, and cost-based routing that favors the cheapest available option, per the official docs.

How does failover work when a provider is down?

When you configure multiple deployments behind the same logical model name — say, three different Azure OpenAI regions behind "gpt-4o" — the router tries the first, and if it errors out after its configured retry count, it moves to the next. LiteLLM also supports fallback groups, where an entirely different model (Claude instead of GPT-4o) steps in if every deployment of the primary model has been exhausted, per LiteLLM's routing documentation.

This two-tier structure — retry within a model, then fall back across models — is the part hand-rolled integration code almost never gets right, because it requires normalizing error semantics across providers that don't agree on what a "rate limit" error even looks like in their raw API responses.

Does the proxy add noticeable latency?

In library mode, overhead is negligible because the SDK is just formatting a request before it goes to the provider. In proxy mode, requests make an extra network hop through the LiteLLM service before reaching the provider, but LiteLLM's own published benchmark puts p99 overhead at roughly 0.66 milliseconds, which is small next to typical LLM response times measured in hundreds of milliseconds to several seconds (litellm.ai).

Supported providers and framework compatibility

LiteLLM's provider list is the project's central selling point. According to the LiteLLM homepage, the platform now covers more than 140 providers and over 1,800 individual models through a single interface. The major categories:

Category	Providers covered
Major closed-source APIs	OpenAI, Azure OpenAI, Anthropic (direct, via Bedrock, and via Vertex AI)
Google	Gemini via AI Studio and Vertex AI
Cloud platforms	AWS Bedrock (any hosted model), Azure AI Foundry, Google Vertex AI
Open-weight hosts	Together AI, Fireworks, Groq, Anyscale, Replicate, Hugging Face Inference Endpoints
Local inference	Ollama, vLLM, LM Studio
Chinese ecosystem	DeepSeek, Qwen (Alibaba), Moonshot, Zhipu AI
Other gateways / search-augmented	Perplexity, OpenRouter, OpenPipe
Enterprise	Cohere, Mistral, AI21

Table: LiteLLM's major supported provider categories, based on the project's official provider documentation.

Embeddings, image generation, audio transcription (Whisper-style), text-to-speech, and reranking endpoints are all exposed through the same OpenAI-compatible URL structure, so a team that has already standardized on OpenAI's embeddings API can point that same code at LiteLLM and route it to Cohere, Voyage AI, or any other embeddings provider without rewriting the call.

Does LiteLLM work with LangChain, LlamaIndex, and agent frameworks?

Yes. Because LiteLLM's proxy speaks OpenAI's API format natively, any tool built on the OpenAI SDK works simply by pointing its base URL at the LiteLLM proxy:

from openai import OpenAI

client = OpenAI(
    api_key="sk-your-litellm-key",
    base_url="http://localhost:4000",  # your LiteLLM proxy
)
# Now every call routes through LiteLLM to whichever provider you've configured

This one change means LangChain, LlamaIndex, CrewAI, AutoGen, and most other agent frameworks work transparently with LiteLLM as the backend, since they were built to talk to an OpenAI-shaped endpoint in the first place. LiteLLM also ships a dedicated ChatLiteLLM integration inside LangChain for teams that prefer library mode over deploying a proxy.

LiteLLM versus the other gateways

LiteLLM is not the only project trying to solve provider fragmentation. The field has settled into a few distinct approaches, and the right pick depends less on features and more on who's operating the infrastructure and who's paying for the markup.

Gateway	Type	Pricing model	Self-host?	Best for
LiteLLM	Open-source library + proxy	Free software; you pay providers directly	Yes	Teams with infra capacity who want no markup and full data control
OpenRouter	Hosted service	Free routing, small per-token markup	No	Fast prototyping, indie builders, no infra to manage
Portkey	Hosted SaaS + open-source SDK	Free tier, paid plans for scale	Partial	Enterprise teams wanting strong observability out of the box
Helicone	Observability platform + proxy	Free tier, paid plans	Yes (open-source)	Teams whose priority is logging and analytics, gateway is secondary
OneAPI	Open-source proxy	Free	Yes	Chinese-language ecosystem and regional model coverage
AWS Bedrock	Managed cloud service	Per-token, AWS-set pricing	No	Teams already committed to AWS infrastructure
Azure AI Foundry	Managed cloud service	Per-token, Azure-set pricing	No	Teams already committed to the Microsoft stack

Table: How LiteLLM compares to the main alternative LLM gateways, by deployment model and pricing structure.

Where LiteLLM wins

LiteLLM's biggest structural advantage is that it's software you run, not a service you pay a markup to use. Every request goes directly from your LiteLLM instance to the provider using your own API keys, so there's no third party sitting in the request path collecting a per-token fee. For a team already running Kubernetes or comfortable with Docker, that difference compounds fast at high request volume, and it also means sensitive prompts never transit a third-party company's infrastructure — a real consideration for regulated industries.

Where LiteLLM loses

The self-hosting requirement is also the catch. Someone has to deploy the proxy, keep Redis and Postgres healthy, monitor the service, and handle upgrades. A two-person startup building a prototype over a weekend gets to a working multi-provider setup faster with OpenRouter, which needs nothing more than a single API key and a credit card. LiteLLM's advantage only pays off once a team has enough request volume, or serious enough compliance requirements, to justify running infrastructure.

How does LiteLLM compare specifically to OpenRouter?

OpenRouter is the easier on-ramp: sign up, get one API key, call more than a hundred models, and OpenRouter handles routing and billing for a small markup on top of the underlying provider's price. LiteLLM requires you to host the proxy yourself and bring your own provider keys, but it charges nothing on top and keeps request data inside your own infrastructure rather than passing through OpenRouter's servers.

Production deployment patterns

Teams that run LiteLLM's proxy in production converge on a similar set of practices, largely because the failure modes — runaway costs, silent outages, budget overruns from a misbehaving agent — are the same regardless of company size.

• Deploy behind Kubernetes or a platform like Fly.io, with Redis attached for caching and distributed rate-limit state, since a single-instance deployment can't share rate-limit counters across replicas.
• Use Postgres as the persistence layer for virtual keys, spend records, and team configuration — LiteLLM's proxy supports it natively and it's the difference between losing all budget history on a restart versus not.
• Wire up Langfuse or Helicone for request-level tracing from day one. Debugging why a specific user got a bad response is close to impossible after the fact without a trace, since the raw provider logs may already have rotated out.
• Set per-model and per-team budget alerts, not just hard caps. A hard cap that fires silently just means an outage; an alert at 80% of budget gives someone time to intervene before an agent loop burns through a month's spend in an afternoon.
• Configure fallback chains, not single models. A common pattern is claude-sonnet-4-5 → gpt-4o → gemini-2.5-pro, so a single provider's regional outage doesn't take down the whole application.
• Point embedding calls through the same proxy as chat completions. For RAG pipelines specifically, this means the same cost tracking and failover logic that protects your chat traffic also protects the retrieval step, which is easy to forget when embeddings are added as an afterthought.

What people get wrong about LiteLLM

The most common mistake is treating LiteLLM as a performance or quality layer, which it isn't. LiteLLM doesn't make any individual model faster or smarter. It's routing and observability infrastructure — the model you route to still has whatever transformer architecture and training data it was built with. A team disappointed that "LiteLLM didn't improve our output quality" has usually misunderstood the tool's job.

The second mistake is assuming self-hosting the proxy is free once the software itself is free. Redis, Postgres, and the compute to run the proxy container all cost money, and at high request volume that infrastructure bill is real. It's usually still cheaper than a per-token markup from a hosted gateway, but "usually" isn't automatic, and teams should measure before assuming.

The third mistake is skipping guardrails because the default LiteLLM setup doesn't force you to configure them. PII redaction, prompt-injection detection, and content moderation are all available as pluggable integrations, but they're opt-in. A proxy that dutifully logs every prompt to your observability stack without ever redacting a customer's phone number or address is a compliance problem waiting to surface, not a hypothetical one.

Finally, some teams over-rotate on cost-based routing without checking quality. Automatically sending every request to whichever model is currently cheapest sounds efficient until a batch of user-facing responses degrade because the router quietly downgraded to a weaker model mid-conversation. Cost-based routing should generally be scoped to requests where quality is genuinely interchangeable across models, not applied blanket-wide.

When to pick LiteLLM versus calling the OpenAI SDK directly

Calling the provider SDK directly still makes sense in a narrower set of cases than it might seem:

• You're committed to a single provider (OpenAI or Azure only) with no near-term plan to add a second.
• The project is small enough that a provider outage is an acceptable, low-stakes inconvenience rather than a business risk.
• You don't need centralized cost tracking because one person or one small team owns the entire API budget.

Reach for LiteLLM once any of the following becomes true:

• You're calling two or more providers, even if only for redundancy rather than active multi-model use.
• You need cost visibility broken down by team, key, or model rather than one lump sum on a provider invoice.
• You're building for an organization, where per-team budgets, audit logs, and role-based access aren't nice-to-haves but requirements.
• You want to future-proof against provider pricing and model changes, since a new flagship model release shouldn't require an application redeploy to adopt.

Frequently Asked Questions

Q: Is LiteLLM free? A: Yes. The core SDK and proxy are open source under the MIT license, and you pay only the underlying LLM providers directly with no markup added by LiteLLM. BerriAI also sells a paid Enterprise tier that adds SSO, role-based access control, and an SLA for teams that need commercial support, per LiteLLM's official site.

Q: Does LiteLLM add noticeable latency? A: Library mode adds essentially no overhead since it's just formatting the request before sending it to the provider. Proxy mode adds one extra network hop, but LiteLLM's own published figures put p99 overhead at roughly 0.66 milliseconds, small relative to typical LLM response times.

Q: Can LiteLLM proxy embedding models too? A: Yes. The proxy exposes /v1/embeddings, image generation, audio transcription and text-to-speech, and reranking endpoints in the same OpenAI-compatible format used for chat completions.

Q: How does LiteLLM compare to OpenRouter? A: OpenRouter is a hosted service that's faster to start with but charges a small per-token markup and routes traffic through its own infrastructure. LiteLLM is self-hosted, adds no markup, and keeps request data inside your own infrastructure, at the cost of having to run and maintain the proxy yourself.

Q: Can I use LiteLLM with LangChain or LlamaIndex? A: Yes. LangChain ships a dedicated ChatLiteLLM integration, and any framework built on the OpenAI SDK — including LlamaIndex, CrewAI, and AutoGen — works by simply pointing its base URL at your LiteLLM proxy.

Q: Does LiteLLM support streaming responses? A: Yes. Every major provider's streaming format is normalized through LiteLLM's unified API, and the proxy passes streamed tokens through to the client in real time rather than buffering the full response first.

Q: What routing strategies can I configure? A: According to LiteLLM's documentation, the router supports simple round-robin shuffling, least-busy selection, usage-based routing that tracks each deployment's rate limits, latency-based routing, and cost-based routing, configurable per model group.

Q: Does LiteLLM support guardrails and PII redaction? A: Yes, as opt-in integrations. The proxy supports Lakera, Aporia, Microsoft Presidio, AWS Bedrock Guardrails, and Azure AI Content Safety, alongside a built-in content filter and OpenAI's own moderation endpoint, configured per request or set to run on all traffic by default.

Q: Who actually uses LiteLLM in production? A: LiteLLM's official site lists Netflix, NVIDIA, Zapier, Cloudera, Twilio, IBM, Stripe, and SAP among its users, with Netflix specifically crediting the tool for letting its team deploy new models to users within a day of release.

Q: Is LiteLLM the same thing as an LLM itself? A: No. LiteLLM doesn't train or host any language model. It's routing and infrastructure software that sits between your application and whichever models you choose to call, similar in role to how a payment gateway sits between a checkout page and the actual banks processing a transaction.

The takeaway

LiteLLM solves an infrastructure problem, not a modeling problem: it doesn't make any LLM smarter, but it removes the integration tax of supporting more than one. The real decision isn't "LiteLLM or nothing" — it's whether your team has the operational capacity to self-host a proxy in exchange for zero markup and full data control, versus paying a hosted gateway like OpenRouter a small fee to skip that operational overhead entirely. Teams running enough volume, or facing real compliance requirements around where prompts and completions travel, tend to land on LiteLLM. Teams still validating an idea usually don't need it yet.

---

Read next:

• How Large Language Models Work
• What Is RAG? Retrieval-Augmented Generation Explained
• Complete AI Guide 2026