What is an AI Gateway?

An AI gateway is a middleware that acts as a unified access point to multiple LLMs, optimizing, securing, and managing AI traffic. It simplifies integration with different AI providers while enabling cost control, observability, and performance benchmarking. With an AI gateway, businesses can seamlessly switch between models, monitor usage, and optimize costs.

LangDB provides OpenAI compatible APIs to connect with multiple Large Language Models (LLMs) by just changing two lines of code.

Govern, Secure, and Optimize all of your AI Traffic with Cost Control, Optimisation and Full Observability.

What AI Gateway Offers Out of the Box

LangDB provides OpenAI-compatible APIs, enabling developers to connect with multiple LLMs by changing just two lines of code. With LangDB, you can:

• Provide access to all major LLMs Ensure seamless integration with leading large language models to maximize flexibility and power.

• No framework code required Enable plug-and-play functionality using any framework like Langchain, Vercel AI SDK, CrewAI, etc., for easy adoption.

• Plug & Play Tracing & Cost Optimization Simplify implementation of tracing and cost optimization features, ensuring streamlined operations.

• Automatic routing based on cost, quality, and other variables Dynamically route requests to the most suitable LLM based on predefined parameters.

• Benchmark and provide insights Deliver insights into the best-performing models for specific tasks, such as coding or reasoning, to enhance decision-making.

Quick Start with LangDB

LangDB offers both managed and self hosted versions for organisations to manage AI traffic . Choose between the Hosted Gateway for ease of use or the Open-Source Gateway for full control.

Roadmap

• Prompt Caching & Optimization (In Progress) Introduce caching mechanisms to optimize prompt usage and reduce redundant costs.

• GuardRails (In Progress) Implement safeguards to enhance reliability and accuracy in AI outputs.

• Leaderboard of models per category Create a comparative leaderboard to highlight model performance across categories.

• Ready-to-use evaluations for non-data scientists Provide accessible evaluation tools for users without a data science background.

• Readily fine-tunable data based on usage Offer pre-configured datasets tailored for fine-tuning, enabling customized improvements with ease.

Instantly connect to managed MCP servers — skip the setup and start using fully managed MCPs with built-in authentication, seamless scalability, and full tracing. This guide gives you a quick walkthrough of how to get started with MCPs.

Quick Example

Steps:

1. Select Slack and Gmail from MCP Severs in the Virtual MCP Section.

2. Generate a Virtual MCP URL automatically.

3. Install the MCP into Cursor with a single command.

Example install command:

npx @langdb/mcp setup slack_gmail_virtual https://api.langdb.ai/mcp/xxxxx --client cursor

What Happens Under the Hood?

• Authentication is handled (via OAuth or API Key)

• Full tracing and observability are available (inputs, outputs, errors, latencies)

• MCP tools are treated just like normal function calls inside LangDB

Next Steps:

LangDB API provides robust support for HTTP headers, enabling developers to manage API requests efficiently with enhanced tracing, observability, and organization.

These headers play a crucial role in structuring interactions with multiple LLMs by providing tracing, request tracking, and session continuity, making it easier to monitor, and analyze API usage

Thread ID (x-thread-id)

Usage: Groups multiple related requests under the same conversation

• Useful for tracking interactions over a single user session.

• Helps maintain context across multiple messages.

Run ID (x-run-id)

Usage: Tracks a unique workflow execution in LangDB, such as a model call or tool invocation.

• Enables precise tracking and debugging.

• Each Run is independent for better observability.

Label (x-label)

Usage: Adds a custom tag or label to a LLM Model Call for easier categorization.

• Helps with tracing multiple agents.

Project ID (x-project-id)

Usage: Identifies the project under which the request is being made.

• Helps in cost tracking, monitoring, and organizing API calls within a specific project.

• Can be set in headers or directly in the API base URL https://api.us-east-1.langdb.ai/${langdbProjectId}/v1

LangDB AI Gateway supports every LLM parameter like temperature, max_tokens, stop sequences, logit_bias, and more.

API Usage:

from openai import OpenAI

response = client.chat.completions.create(
    model="gpt-4o", # Change Model
    messages=[
        {"role": "user", "content": "What are the earnings of Apple in 2022?"},
    ],
    temperature=0.7,               # temperature parameter
    max_tokens=150,                # max_tokens parameter
    stream=True                   # stream parameter
)

const response = await client.chat.completions.create({
  model: 'gpt-4o-mini',          
  messages,                     
  temperature: 0.7,              // temperature parameter
  max_tokens: 150,               // max_tokens parameter
  logit_bias: { '50256': -100 }, // logit_bias parameter
  stream: true,                  // stream parameter
});

UI

You can also use the UI to test various parameters and getting code snippet

Playground

Samples

Explore ready-made code snippets complete with preconfigured parameters—copy, paste, and customize to fit your needs.

LangDB AI enables user tracking to collect analytics and monitor usage patterns efficiently. By associating metadata with requests, developers can analyze interactions, optimize performance, and enhance user experience.

Example: Chatbot Analytics with User Tracking

For a chatbot service handling multiple users, tracking enables:

• Recognizing returning users: Maintain conversation continuity.

• Tracking usage trends: Identify common queries to improve responses.

• User segmentation: Categorize users using tags (e.g., "websearch", "support").

• Analytics: Identify heavy users and allocate resources efficiently.

curl 'https://api.us-east-1.langdb.ai/v1/chat/completions' \
-H 'authorization: Bearer LangDBApiKey' \
-H 'Content-Type: application/json' \
-d '{
  "model": "openai/gpt-4o-mini",
  "stream": true,
  "messages": [
    {
      "role": "user",
      "content": "Def bubbleSort()"
    }
  ],
  "extra": {
    "user": {
      "id": "7",
      "name": "mrunmay",
      "tags": ["coding", "software"]
    }  
  }
}'

User Tracking Fields

• extra.user.id: Unique user identifier.

• extra.user.name: User alias.

• extra.user.tags: Custom tags to classify users (e.g., "coding", "software").

Fetching User Analytics & Usage Data

Once users are tracked, analytics and usage APIs can be used to retrieve insights based on id, name, or tags.

Example:

curl -L \
  --request POST \
  --url 'https://api.us-east-1.langdb.ai/analytics/summary' \
  --header 'Authorization: Bearer langDBAPIKey' \
  --header 'X-Project-Id: langDBProjectID' \
  --header 'Content-Type: application/json' \
  --data '{
    "user_id": "7",
    "user_name": "mrunmay",
    "user_tags": ["software", "code"]   
  }'

Example response:

{
  "summary": [
    {
      "total_cost": 0.00030366,
      "total_requests": 1,
      "total_duration": 6240.888,
      "avg_duration": 6240.9,
      "duration": 6240.9,
      "duration_p99": 6240.9,
      "duration_p95": 6240.9,
      "duration_p90": 6240.9,
      "duration_p50": 6240.9,
      "total_input_tokens": 1139,
      "total_output_tokens": 137,
      "avg_ttft": 6240.9,
      "ttft": 6240.9,
      "ttft_p99": 6240.9,
      "ttft_p95": 6240.9,
      "ttft_p90": 6240.9,
      "ttft_p50": 6240.9,
      "tps": 204.46,
      "tps_p99": 204.46,
      "tps_p95": 204.46,
      "tps_p90": 204.46,
      "tps_p50": 204.46,
      "tpot": 0.05,
      "tpot_p99": 0.05,
      "tpot_p95": 0.05,
      "tpot_p90": 0.05,
      "tpot_p50": 0.05,
      "error_rate": 0.0,
      "error_request_count": 0
    }
  ],
  "start_time_us": 1737547895565066,
  "end_time_us": 1740139895565066
}

A Thread is simply a grouping of Message History that maintains context in a conversation or workflow. Threads are useful for keeping track of past messages and ensuring continuity across multiple exchanges.

Core Features:

• Contextual Continuity: Ensures all related Runs are grouped for better observability.

• Multi-Turn Support: Simplifies managing interactions that require maintaining state across multiple Runs.

Example:

A user interacting with a chatbot over multiple turns (e.g., asking follow-up questions) generates several messages, but all are grouped under a single Thread to maintain continuity.

Headers for Thread:

• x-thread-id: Links all Runs in the same context or conversations.

Hosted

The Hosted AI Gateway allows you to connect with multiple Large Language Models (LLMs) instantly, without any setup.

A Trace represents the complete lifecycle of a workflow, spanning all components and systems involved.

Core Features:

• End-to-End Visibility: Tracks model calls, tools across the entire workflow.

• Multi Agent Ready: Perfect for workflows that involve multiple services, APIs, or tools.

• Error Diagnosis: Quickly identify bottlenecks, failures, or inefficiencies in complex workflows.

Parent-Trace:

For workflows with nested operations (e.g., a workflow that triggers multiple sub-workflows), LangDB introduces the concept of a Parent-Trace, which links the parent workflow to its dependent sub-workflows. This hierarchical structure ensures you can analyze workflows at both macro and micro levels.

Headers for Trace:

• trace-id: Tracks the parent workflow.

• parent-trace-id: Links sub-workflows to the main workflow for hierarchical tracing.

This allows developers to track interactions between agents seamlessly, ensuring clear visibility into workflows and dependencies.

What is a Multi-Agent System?

A multi-agent system consists of independent agents collaborating to solve complex tasks. Agents handle various roles such as user interaction, data processing, and workflow orchestration. LangDB streamlines tracking these interactions for better efficiency and transparency.

Why Track Workflows?

Tracking ensures:

• Clear Execution Flow: Understand how agents interact.

• Performance Optimization: Identify bottlenecks.

• Reliability & Accountability: Improve transparency.

LangDB supports two main concepts.

• Example

Using the same Run ID and Thread ID across multiple agents ensures seamless tracking, maintaining context across interactions and providing a complete view of the workflow

from openai import OpenAI
from uuid import uuid4
client = OpenAI(
    base_url="https://api.us-east-1.langdb.ai/{langdb_project_id}/v1"  # LangDB API base URL,
    api_key=api_key,  # Replace with your LangDB token
)
response1 = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "developer", "content": "You are a helpful assistant."},
              {"role": "user", "content": "Hello!"}],  
    extra_headers={"x-thread-id": thread_id, "x-run-id": run_id}
)

# Agent 2 processes the response
response2 = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "developer", "content": "Processing user input."},
              {"role": "user", "content": response1.choices[0].message["content"]}],  
    extra_headers={"x-thread-id": thread_id, "x-run-id": run_id}
)

A Virtual MCP Server lets you create a customized set of MCP tools by combining functions from multiple MCP servers — all with scoped access, unified auth, and full observability.

Why Use a Virtual MCP?

• Selective Tools: Pick only the tools you need from existing MCP servers (e.g. Airtable's list_records, GitHub's create_issue, etc.)

• Clean Auth Handling: Add your API keys only if needed. Otherwise, LangDB handles OAuth for you.

• Full Tracing: Every call is traced on the LangDB — with logs, latencies, input/output, and error metrics.

• Easy Integration: Works out of the box with Cursor, Claude, Windsurf, and more.

• Version Lock-in: Virtual MCPs are pinned to a specific server version to avoid breaking changes.

• Poisoning Safety: Prevents injection or override by malicious tool definitions from source MCPs.

How to Set It Up

1. Go to your Virtual MCP server on LangDB Project.

2. Select the tools you want to include.

3. (Optional) Add API keys or use LangDB-managed auth.

4. Click Generate secure MCP URL.

Install in Cursor / Windsurf / Claude

Once you have the MCP URL:

npx @langdb/mcp setup figma https://api.staging.langdb.ai/mcp/xxxxx --client cursor

You're now ready to use your selected tools directly inside the editor.

Try it in the playground

A Message is the basic unit of communication in LangDB workflows. Messages define the interaction between the user, the system, and the model. Every workflow is built around exchanging and processing messages.

Core Features:

• Structured Interactions: Messages define roles (user, system, assistant) to organize interactions clearly.

• Multi-Role Flexibility: Different roles (e.g., system for instructions, user for queries) enable complex workflows.

• Dynamic Responses: Messages form the backbone of LangDB’s chat-based interactions.

Example:

A simple interaction to generate a poem might look like this:

[
  { "role": "system", "content": "You are a helful assistant" },
  { "role": "user", "content": "Write me a poem about celluloids." }
]

Label in LangDB defines an LLM instance with a unique identifier for categorization and tracking.

Core Features

• Model Categorization: Assign labels to LLM instances.

• Observability: Track models by label.

Headers for Label:

• x-label: Defines a label for an LLM instance.

{
    "x-label" : "research-agent"
}

A Run represents a single workflow or operation executed within LangDB. This could be a model invocation, a tool call, or any other discrete task. Each Run is independent and can be tracked separately, making it easier to analyze and debug individual workflows.

Example of a Run:

Core Features:

• Granular Tracking: Analyze and optimize the performance and cost of individual Runs.

• Independent Execution: Each Run has a distinct lifecycle, enabling precise observability.

Example:

Generating a summary of a document, analyzing a dataset, or fetching information from an external API – each is a Run.

Headers for Run:

• x-run-id: Identifies a specific Run for tracking and debugging purposes.

LangDB Gateway provides detailed tracing to monitor, debug, and optimize LLM workflows.

Below is an example of a trace visualization from the dashboard, showcasing a detailed breakdown of the request stages:

In this example trace you’ll find:

• Overview Metrics

  • Cost: Total spend for this request (e.g. $0.034).

  • Tokens: Input (5,774) vs. output (1,395).

  • Duration: Total end-to-end latency (29.52 s).

• Timeline Breakdown A parallel-track timeline showing each step—from moderation and relevance scoring to model inference and final reply.

• Model Invocations** Every call to gpt-4o-mini, gpt-4o, etc., is plotted with precise start times and durations.

• Agent Hand-offs Transitions between your agents (e.g. search → booking → reply) are highlighted with custom labels like transfer_to_reply_agent.

• Tool Integrations External tools (e.g. booking_tool, travel_tool, python_repl_tool) appear inline with their execution times—so you can spot slow or failed runs immediately.

• Guardrails Rules like Min Word Count and Travel Relevance enforce domain-specific constraints and appear in the trace.

With this level of visibility you can quickly pinpoint bottlenecks, understand cost drivers, and ensure your multi-agent pipelines run smoothly.

Virtual Models in LangDB let you save and reuse AI configurations, ensuring streamlined workflows and consistent behavior across applications. They allow you to define system and user messages upfront, customize model parameters like temperature, max tokens, and penalties, and integrate LangDB’s built-in tools for enhanced response capabilities.

Once saved, these configurations can be quickly accessed and reused across multiple applications.

Features

• Predefined Conversations: Set system and user messages to guide AI behavior.

• Custom Parameters: Fine-tune settings like temperature, max tokens, top-p, logit bias, and penalties.

• Tools Integration: Augment AI responses with LangDB’s built-in tools.

• Reusable Configurations: Save models under a unique name for easy retrieval and use.

• Guardrails: Attach guardrails directly to a virtual model.

How to Create a Virtual Model

Monitoring complements tracing by providing aggregate insights into the usage of LLM workflows.

Limits

LangDB enforces limits to ensure fair usage and cost management while allowing users to configure these limits as needed. Limits are categorized into:

1. Daily Limits: Maximum usage per day, e.g., $10 in the Starter Tier.

2. Monthly Limits: Total usage allowed in a month, e.g., $100.

Total Limits: Cumulative limit over the project’s duration, e.g., $500.

Best Practices

• Monitor usage regularly to avoid overages.

• Plan limits based on project needs and anticipated workloads.

• Upgrade tiers if usage consistently approaches limits.

Setting limits not only helps you stay within budget but also provides the flexibility to scale your usage as needed, ensuring your projects run smoothly and efficiently.

Usage APIs

Retrieves the total usage statistics for your project for a timeframe.

curl --location 'https://api.us-east-1.langdb.ai/usage/total' \
--header 'x-project-id: langdbProjectID' \ 
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer langDBAPIKey' \ 
--data '{"start_time_us": 1693062345678, 
         "end_time_us": 1695092345678}'

Example Response:

{
  "total": {
    "total_input_tokens": 4181386,
    "total_output_tokens": 206547,
    "total_cost": 11.890438685999994
  },
  "period_start": 1737504000000000,
  "period_end": 1740131013885000
}

Fetches timeseries usage statistics per model, allowing users to analyze the distribution of LLM usage.

curl --location 'https://api.us-east-1.langdb.ai/usage/models' \
--header 'x-project-id: langdbProjectID' \  
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer langDBAPIKey' \ 
--data '{"start_time_us": 1693062345678, "end_time_us": 1695092345678, 
             "min_unit": "hour"} '

Example Response:

{
  "models": [
    {
      "hour": "2025-02-14 08:00:00",
      "provider": "openai",
      "model_name": "gpt-4o-mini",
      "total_input_tokens": 13408,
      "total_output_tokens": 2169,
      "total_cost": 0.0039751199999999995
    },
    {
      "hour": "2025-02-13 08:00:00",
      "provider": "openai",
      "model_name": "gpt-4o-mini",
      "total_input_tokens": 55612,
      "total_output_tokens": 786,
      "total_cost": 0.01057608
    }
  ],
  "period_start": 1737504000000000,
  "period_end": 1740130915098000
}

Filtering By Users

As discussed in User Tracking, we can use filters to retrieve insights based on id, name, or tags.

Available Filters:

• user_id: Filter data for a specific user by their unique ID.

• user_name: Retrieve usage based on the user’s name.

• user_tags: Filter by tags associated with a user (e.g., "websearch", "support").

curl -L \
  --request POST \
  --url 'https://api.us-east-1.langdb.ai/usage/models' \
  --header 'Authorization: Bearer langDBAPIKey' \
  --header 'X-Project-Id: langDBProjectID' \
  --header 'Content-Type: application/json' \
  --data '{
    "user_id": "123",
    "user_name": "mrunmay",
    "user_tags": ["websearch", "testings"]
  }'

Example response:

{
  "models": [
    {
      "day": "2025-02-21 10:00:00",
      "provider": "openai",
      "model_name": "gpt-4o-mini",
      "total_input_tokens": 1112,
      "total_output_tokens": 130,
      "total_cost": 0.00029376
    },
    {
      "day": "2025-02-21 14:00:00",
      "provider": "openai",
      "model_name": "gpt-4o-mini",
      "total_input_tokens": 3317,
      "total_output_tokens": 328,
      "total_cost": 0.00083322
    }
  ],
  "period_start": 1737556513673410,
  "period_end": 1740148513673410
}

LangDB AI Gateway optimizes LLM selection based on cost, speed, and availability, ensuring efficient request handling. This guide covers the various dynamic routing strategies available in the system, including fallback, script-based, optimized, percentage-based, and latency-based routing.

This ensures efficient request handling and optimal model selection tailored to specific application needs.

Understanding Targets

Before diving into routing strategies, it's essential to understand targets in LangDB AI Gateway. A target refers to a specific model or endpoint to which requests can be directed. Each target represents a potential processing unit within the routing logic, enabling optimal performance and reliability.

[
    { "model": "openai/gpt-4o", "temperature": 0.7, "max_tokens": 300, "top_p": 0.95 },
    { "model": "deepseek/deepseek-chat", "temperature": 0.8, "max_tokens": 400, "frequency_penalty": 0.5 }
]

Target Parameters

Each target in the routing configuration can have custom parameters that define its behavior. These parameters allow fine-tuning of model outputs to align with specific requirements.

Common parameters include:

• model: The model identifier (e.g., openai/gpt-4o, deepseek/deepseek-chat).

• temperature: Controls the randomness of responses (higher values make responses more creative).

• max_tokens: Limits the number of tokens in the response.

• top_p: Determines the probability mass for nucleus sampling.

• frequency_penalty: Reduces repetition by penalizing frequent tokens.

• presence_penalty: Encourages diversity by discouraging token reuse.

Customizing Model Parameters

You can customize parameters for each target model to fine-tune the behavior and output of the models. Parameters such as temperature, max_tokens, and frequency_penalty can be adjusted to meet specific requirements.

Example of customizing model parameters:

{
    "router": {
        "type": "fallback",
        "targets": [
            { "model": "openai/gpt-4o-mini", "temperature": 0.9, "max_tokens": 500, "top_p": 0.9 },
            { "model": "deepseek/deepseek-chat", "frequency_penalty": 1, "presence_penalty": 0.6 }
        ]
    }
}

Routing Strategies

LangDB AI Gateway supports multiple routing strategies that can be combined and customized to meet your specific needs:

Fallback Routing

Fallback routing allows sequential attempts to different model targets in case of failure or unavailability. It ensures robustness by cascading through a list of models based on predefined logic.

{
    "model": "router/dynamic",
    "messages": [
        { "role": "system", "content": "You are a helpful assistant." },
        { "role": "user", "content": "What is the formula of a square plot?" }
    ],
    "router": {
        "router": "router",
        "type": "fallback", // Type: fallback/script/optimized/percentage/latency
        "targets": [
            { "model": "openai/gpt-4o-mini", "temperature": 0.9, "max_tokens": 500, "top_p": 0.9 },
            { "model": "deepseek/deepseek-chat", "frequency_penalty": 1, "presence_penalty": 0.6 }
        ]
    },
    "stream": false
}

Optimized Routing

Optimized routing automatically selects the best model based on real-time performance metrics such as latency, response time, and cost-efficiency.

{
    "model": "router/dynamic",
    "router": {
        "name": "fastest",
        "type": "optimized",
        "metric": "ttft",
        "targets": [
            { "model": "gpt-3.5-turbo", "temperature": 0.8, "max_tokens": 400, "frequency_penalty": 0.5 },
            { "model": "gpt-4o-mini", "temperature": 0.9, "max_tokens": 500, "top_p": 0.9 }
        ]
    }
}

Here, the request is routed to the model with the lowest Time-to-First-Token (TTFT) among gpt-3.5-turbo and gpt-4o-mini.

Metrics:

• Requests – Total number of requests sent to the model.

• InputTokens – Number of tokens provided as input to the model.

• OutputTokens – Number of tokens generated by the model in response.

• TotalTokens – Combined count of input and output tokens.

• RequestsDuration – Total duration taken to process requests.

• Ttft (Time-to-First-Token) (Default) – Time taken by the model to generate its first token after receiving a request.

• LlmUsage – The total computational cost of using the model, often used for cost-based routing.

Percentage-Based Routing

Percentage-based routing distributes requests between models according to predefined weightings, allowing load balancing, A/B testing, or controlled experimentation with different configurations. Each model can have distinct parameters while sharing the request load.

{ 
  "model": "router/dynamic",
  "router": {
    "name": "dynamic",
    "type": "percentage",
    "targets": [
      { "model": "openai/gpt-4o-mini", "temperature": 0.9, "max_tokens": 500, "top_p": 0.9 },
      { "model": "openai/gpt-4o-mini", "temperature": 0.8, "max_tokens": 400, "frequency_penalty": 1 }
    ],
    "targets_percentages": [ 70, 30 ]
  }
}

Latency-Based Routing

Latency-based routing selects the model with the lowest response time, ensuring minimal delay for real-time applications like chatbots and interactive AI systems.

{
  "model": "router/dynamic",
  "router": {
    "name": "fastest_latency",
    "type": "latency",
    "targets": [
      { "model": "openai/gpt-4o-mini", "temperature": 0.9, "max_tokens": 500, "top_p": 0.9 },
      { "model": "deepseek/deepseek-chat", "frequency_penalty": 1, "presence_penalty": 0.6 },
      { "model": "gemini/gemini-2.0-flash-exp", "temperature": 0.8, "max_tokens": 400, "frequency_penalty": 0.5 }
    ]
  }
}

Nested Routing

LangDB AI allows nesting of routing strategies, enabling combinations like fallback within script-based selection. This flexibility helps refine model selection based on dynamic business needs.

{
    "model": "router/dynamic",
    "messages": [
        { "role": "system", "content": "You are a helpful assistant." },
        { "role": "user", "content": "What is the formula of a square plot?" }
    ],
    "router": {
        "type": "fallback",
        "targets": [
            {
                "model": "router/dynamic",
                "router": {
                    "name": "cheapest_script_execution",
                    "type": "script",
                    "script": "const route = ({ models }) => models \
                        .filter(m => m.inference_provider.provider === 'bedrock' && m.type === 'completions') \
                        .sort((a, b) => a.price.per_input_token - b.price.per_input_token)[0]?.model;"
                }
            },
            {
                "model": "router/dynamic",
                "router": {
                    "name": "fastest",
                    "type": "optimized",
                    "metric": "ttft",
                    "targets": [
                        { "model": "gpt-3.5-turbo", "temperature": 0.8, "max_tokens": 400, "frequency_penalty": 0.5 },
                        { "model": "gpt-4o-mini", "temperature": 0.9, "max_tokens": 500, "top_p": 0.9 }
                    ]
                }
            },
            { "model": "deepseek/deepseek-chat", "temperature": 0.7, "max_tokens": 300, "frequency_penalty": 1 }
        ]
    },
    "stream": false
}

LangDB AI Gateway provides an intuitive user interface (UI) to configure and manage routing strategies. Through the UI, users can set up model routing dynamically, ensuring requests are directed efficiently based on cost, speed, and performance.

Accessing the Router Configuration

You can monitor API usage with key insights.

After integrating LangDB into your project, the Analytics Dashboard becomes your central hub for understanding usage.

Metrics

LangDB’s Analytics Dashboard is segmented into several key panels:

Cost:

• Tracks your total cost consumption across all integrated models.

• Enables you to compare costs by provider/model/tags, helping you identify the most cost-effective options for your use cases.

Time:

• Displays the average duration of requests in milliseconds.

• Useful for benchmarking response times and optimizing performance for latency-sensitive applications.

Number of Requests:

• Shows the total number of API calls made.

• Helps you analyze usage patterns and allocate resources effectively.

Average Time to First Token (TTFT)

• Indicates the average time taken to receive the first token from the API response.

• This metric is critical for understanding initial latency.

Tokens Per Second (TPS)

• Measures the throughput of token generation.

• High TPS is indicative of efficient processing.

Time Per Output Token (TPOT)

• Tracks the average time spent per output token.

• Helps in identifying and troubleshooting bottlenecks in model output.

Error Rate

• Displays the percentage of failed requests over total requests.

• Helps monitor system stability and reliability.

Error Request Count

• Tracks the total number of failed API requests.

• Useful for debugging and troubleshooting failures effectively.

Analytics APIs

Provides a detailed timeseries view of API usage metrics. Users can filter data by time range and group it by provider, model, or tags to analyze trends over different periods.

Example response:

Provides aggregated usage metrics, allowing users to get a high-level overview of API consumption and error rates.

Example response:

Filtering By Users

As discussed in User Tracking, we can use filters to retrieve insights based on id, name, or tags.

Available Filters:

• user_id: Filter data for a specific user by their unique ID.

• user_name: Retrieve usage based on the user’s name.

• user_tags: Filter by tags associated with a user (e.g., "websearch", "support").

Example response:

Model Context Protocol (MCP) is an open standard that enables AI models to seamlessly communicate with external systems. It allows models to dynamically process contextual data, ensuring efficient, adaptive, and scalable interactions. MCP simplifies request orchestration across distributed AI systems, enhancing interoperability and context-awareness.

With native tool integrations, MCP connects AI models to APIs, databases, local files, automation tools, and remote services through a standardized protocol. Developers can effortlessly integrate MCP with IDEs, business workflows, and cloud platforms, while retaining the flexibility to switch between LLM providers. This enables the creation of intelligent, multi-modal workflows where AI securely interacts with real-world data and tools.

Using Virtual MCPs

Using API

Here's an example of how you can use a Virtual MCP Server in your project:

Using MCP Clients

You can instantly connect LangDB’s Virtual MCP servers to editors like Cursor, Claude, or Windsurf.

Run this in your terminal to set up MCP in Cursor:

You can now call tools directly in your editor, with full tracing on LangDB.

Connecting to External MCP Servers

If you already have an MCP server hosted externally — like Smithery’s Exa MCP — you can plug it straight into LangDB with zero extra setup.

Just pass your external MCP server URL in extra_body when you make a chat completion request. For example Smithery:

LangDB enables cost tracking, project budgeting, and cost groups to help manage AI usage efficiently.

Cost Groups (Business Tier & Above)

• Available in Business & Enterprise tiers under User Management.

• Organize users into cost groups to track and allocate spending.

• Cost groups help in budgeting but are independent of user roles.

Project-Level Spending Limits

• Set daily, monthly, and total spending limits per project.

• Enforce per-user limits to prevent excessive usage.

• Available in Project Settings → Cost Control.

Cost Group-Based Role Management

• Admins and Billing users can define spending limits for cost groups.

• Set daily, monthly, and total budgets per group.

• Useful for controlling team-based expenses independently of project limits.

LangDB allow developers to enforce specific constraints and checks on their LLM calls, ensuring safety, compliance, and quality control.

Guardrails currently support request validation and logging, ensuring structured oversight of LLM interactions.

These guardrails include:

• Content Moderation: Detects and filters harmful or inappropriate content (e.g., toxicity detection, sentiment analysis).

• Security Checks: Identifies and mitigates security risks (e.g., PII detection, prompt injection detection).

• Compliance Enforcement: Ensures adherence to company policies and factual accuracy (e.g., policy adherence, factual accuracy).

• Response Validation: Validates response format and structure (e.g., word count, JSON schema, regex patterns).

Guardrails can be configured via the UI or API, providing flexibility for different use cases.

Guardrail Behaviour

When a guardrail blocks an input or output, the system returns a structured error response. Below are some example responses for different scenarios:

Example 1: Input Rejected by Guard

{
  "id": "",
  "object": "chat.completion",
  "created": 0,
  "model": "",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Input rejected by guard",
        "tool_calls": null,
        "refusal": null,
        "tool_call_id": null
      },
      "finish_reason": "rejected"
    }
  ],
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0,
    "cost": 0.0
  }
}

Example 2: Output Rejected by Guard

{
  "id": "5ef4d8b1-f700-46ca-8439-b537f58f7dc6",
  "object": "chat.completion",
  "created": 1741865840,
  "model": "gpt-4o-mini",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Output rejected by guard",
        "tool_calls": null,
        "refusal": null,
        "tool_call_id": null
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 21,
    "completion_tokens": 40,
    "total_tokens": 61,
    "cost": 0.000032579999999999996
  }
}

Limitations

It is important to note that guardrails cannot be applied to streaming outputs.

Guardrail Templates

LangDB provides prebuilt templates to enforce various constraints on LLM responses. These templates cover areas such as content moderation, security, compliance, and validation.

The following table provides quick access to each guardrail template:

Toxicity Detection (content-toxicity)

Detects and filters out toxic, harmful, or inappropriate content.

JSON Schema Validator (validation-json-schema)

Validates responses against a user-defined JSON schema.

Competitor Mention Check (content-competitor-mentions)

Detects mentions of competitor names or products in LLM responses.

PII Detection (security-pii-detection)

Detects personally identifiable information (PII) in responses.

Prompt Injection Detection (security-prompt-injection)

Identifies prompt injection attacks attempting to manipulate the AI.

Company Policy Compliance (compliance-company-policy)

Ensures that responses align with predefined company policies.

Regex Pattern Validator (validation-regex-pattern)

Validates responses against specific regex patterns.

Word Count Validator (validation-word-count)

Ensures responses meet specified word count requirements.

Sentiment Analysis (content-sentiment-analysis)

Evaluates the sentiment of responses to ensure appropriate tone.

Language Validator (content-language-validation)

Checks if responses are in allowed languages.

Topic Adherence (content-topic-adherence)

Ensures responses stay on specified topics.

Factual Accuracy (content-factual-accuracy)

Validates that responses contain factually accurate information.

LangDB provides role-based access control to manage users efficiently within an organization. There are three primary roles: Admin, Developer, and Billing.

Each role has specific permissions and responsibilities, ensuring a structured and secure environment for managing teams.

Admin

Admins have the highest level of control within LangDB. They can:

• Invite and manage users

• Assign and modify roles for team members

• Manage cost groups and usage tracking

• Access billing details and payment settings

• Configure organizational settings

Best for: Organization owners, team leads, or IT administrators managing team access and billing.

Developer

Developers focus on working with APIs and integrating LLMs. They have the following permissions:

• Access and use LangDB APIs

• Deploy and test applications using LangDB’s AI Gateway

• View and monitor API usage and performance

Best for: Software developers, data scientists, and AI engineers working on LLM integrations.

Billing

Billing users have access to financial and cost-related features. Their permissions include:

• Managing top-ups and subscriptions

• Monitoring usage costs and optimizing expenses

Best for: Finance teams, accounting personnel, and cost management administrators.

Role Management

Admins can assign roles to users when inviting them to the organization. Role changes can also be made later through the user management panel.

Key Points:

• Users can have multiple roles (e.g., both Developer and Billing).-

• Only Admins can assign or update roles.

• Billing users cannot modify API access but can track and manage costs.

• Role Management is only available in Professional, Business, and Enterprise tiers.

LangDB simplifies working with multiple Large Language Models (LLMs) through a single API. It excels at analytics, usage monitoring, and evaluation, giving developers insights into model performance, usage stats, and costs. This guide covers installation, setup, and key functionalities.

Installation

To install the LangDB Python client, run:

Initialize LangDB Client

Initialize the client with your API key and project ID:

Making a Chat Completion Request

You can generate a response using the completion method:

Retrieve Messages from a Thread

You can fetch messages from a specific thread using its thread_id:

Get Thread Usage

Retrieve cost and token usage details for a thread:

Get Analytics

You can retrieve analytics for specific model tags:

Alternatively, you can convert analytics data into a Pandas DataFrame for easier analysis:

Evaluate Multiple Threads

To generate an evaluation DataFrame containing message and cost information for multiple threads:

List Available Models

To list all models supported by LangDB:

Usage in Evaluation

LangDB provides built-in evaluation capabilities, allowing developers to assess model performance, response accuracy, and cost efficiency. By analyzing messages, token usage, and analytics data, teams can fine-tune their models for better results.

You can use LangDB as a drop-in replacement for OpenAI APIs, making it easy to integrate into existing workflows and libraries such as OpenAI Client SDK.