LiteLLM

LiteLLM is an open-source Python library and proxy server that provides a unified API to interact with over 100 large language model (LLM) providers. It abstracts different providers' APIs into a consistent OpenAI API-like format, allowing developers to switch models or providers with minimal code changes and manage multiple deployments with retries, fallbacks, and cost tracking.

The core features of LiteLLM include a unified API to call over 100 LLMs using OpenAI-compatible input/output formats, support for text completion, embeddings, and image generation endpoints, and retry and fallback logic across multiple AI providers such as Azure and OpenAI. It also offers spend tracking, budgeting per project, and rate limiting, along with a central proxy server (LLM Gateway) for load balancing and centralized access management. Furthermore, LiteLLM integrates with observability and logging tools like Lunary, MLFlow, LangFuse, and Helicone, and supports streaming responses and batch requests.

To start using LiteLLM, you need to install the Python SDK from the GitHub repository. Afterward, use the completion function to make chat or text completion requests through various models by specifying the model name with the provider prefix, for example, openai/gpt-3.5-turbo or vercel_ai_gateway/openai/gpt-4o. You can also run LiteLLM as a local proxy server or integrate it with cloud platforms to enable centralized management and enterprise features.

LiteLLM integrates with a wide range of providers including OpenAI, Azure OpenAI, Anthropic, Google Vertex AI, AWS Bedrock, Ollama, HuggingFace, and others, all normalized to OpenAI API formats for chat completions, embeddings, and other endpoint types.

Yes, LiteLLM can serve as the LLM provider for frameworks like LangChain and LlamaIndex to leverage its unified API and routing capabilities.

LiteLLM handles security and API keys by securely storing provider API keys (master keys) encrypted with a user-provided salt key, LITELLM_SALT_KEY. Data in transit is encrypted using TLS/SSL in both the open-source and enterprise versions. The proxy architecture is also designed with security best practices to safely manage multiple provider keys.

LiteLLM can be deployed as a self-hosted proxy server using Docker, Helm, or Terraform, integrating with PostgreSQL as the backend database for cost tracking and configuration. It supports multi-cloud and multi-model routing scenarios for enterprise-scale usage.

The LiteLLM open-source project is free to use. It is designed as an abstraction layer, so you pay for the underlying API usage with individual LLM providers. Enterprise versions may offer additional features and support.

Yes, LiteLLM supports streaming output from models to handle token-by-token generation efficiently.

This depends on the underlying LLM provider's constraints rather than LiteLLM itself, as it acts as a proxy routing requests.