Introduction to AI Gateway

Unified API for all LLMs

LangDB provides OpenAI compatible APIs to connect with multiple Large Language Models (LLMs) with a single line of code change.

Features:

  • Cost Management: Track and control LLM usage to optimize spending.

  • Dynamic Routing: Automatically direct requests to the most appropriate LLM based on requirements.

  • Scalability: Simplifies scaling across projects and environments without added complexity.

Built for developers, the AI Gateway focuses on providing a practical and streamlined experience for integrating LLMs into your workflows.

Just change base_url and extra_headers to get full tracing and observability.

You can use any of the supported models.

from openai import OpenAI
client = OpenAI(
    base_url="https://api.us-east-1.langdb.ai"  # LangDB API base URL,
    api_key=api_key,  # Replace with your LangDB token
)
# Make the API call to LangDB's Completions API
response = client.chat.completions.create(
    model="gpt-4o",  # Use the model
    messages=[{"role": "developer", "content": "You are a helpful assistant."},
              {"role": "user", "content": "Hello!"}],  
    extra_headers={"x-project-id": "xxxxx"} # LangDB Project ID
)

Objectives

  • Provide access to all major LLMs
    Ensure seamless integration with leading large language models to maximize flexibility and power.

  • No framework code required
    Enable plug-and-play functionality using any framework like Langchain, Vercel AI SDK, CrewAI, etc., for easy adoption.

  • Plug & Play Tracing & Cost Optimization
    Simplify implementation of tracing and cost optimization features, ensuring streamlined operations.

  • Automatic routing based on cost, quality, and other variables
    Dynamically route requests to the most suitable LLM based on predefined parameters.

  • Benchmark and provide insights
    Deliver insights into the best-performing models for specific tasks, such as coding or reasoning, to enhance decision-making.

Roadmap

  • Open Source AI Gateway (In Progress)
    Open-source AI Gateway for community collaboration and transparency. Built in Rust.

  • Prompt Caching & Optimization (In Progress)
    Introduce caching mechanisms to optimize prompt usage and reduce redundant costs.

  • GuardRails (In Progress)
    Implement safeguards to enhance reliability and accuracy in AI outputs.

  • Leaderboard of models per category
    Create a comparative leaderboard to highlight model performance across categories.

  • Ready-to-use evaluations for non-data scientists
    Provide accessible evaluation tools for users without a data science background.

  • Readily fine-tunable data based on usage
    Offer pre-configured datasets tailored for fine-tuning, enabling customized improvements with ease.

Updated on
Supported Models