Example: Building an Enterprise Routing Configuration
End-to-end example of a multi-layer enterprise routing setup with tiering, cost and fallbacks.
This example demonstrates a multi-layered routing strategy for a SaaS company that balances performance for premium users, cost for standard users, and flexibility for internal development.
Goals:
Provide the fastest possible responses for "premium" customers on support-related queries.
Minimize costs for "standard" tier users.
Allow the internal "development" team to test a new, experimental model without affecting customers.
Complete Chat Completion Request with Enterprise Routing:
Configuration Breakdown:
Request Structure:
Uses "model": "router/dynamic" to enable dynamic routing
User information is passed via the extra.user object
Router configuration is specified in the router field with "type": "conditional"
Route 1: premium_support_fast_track
Conditions: Applies when extra.user.tier equals "premium" AND extra.user.request.topic equals "support"
Targets: Routes to high-performance models (anthropic/claude-4-opus, openai/gpt-o3, gemini/gemini-2.5-pro, xai/grok-4) with filtering for fast response times (ttft < 1000ms) and sorts by minimum time-to-first-token
Route 2: standard_user_cost_optimized
Conditions: Catches all requests from "standard" tier users via extra.user.tier
Targets: Uses cost-effective models (mistral/mistral-large-latest, anthropic/claude-4-sonnet) and sorts by minimum price
Route 3: internal_dev_testing
Conditions: Applies to users in the "development" cost group via metadata.group_name (automatically set by LangDB based on the LangDB user's cost group assignment)
Targets: Routes directly to google/gemini-2.5-pro for isolated testing
Route 4: fallback_route
Conditions: Empty conditions array ("all": []) catches all remaining requests
Targets: Routes to openai/gpt-4o-mini as a reliable fallback option
Key Features Demonstrated:
Conditional Logic: Uses all operators and comparison operators ($eq, $lt)
Target Selection: Shows both single targets and pools with filtering/sorting
Request Context: Leverages both extra user data and metadata for routing decisions
Cost Group Integration: metadata.group_name is automatically populated by LangDB based on the LangDB user's cost group assignment
Retry Configuration: Includes max_retries: 2 for resilience
Additional Example Scenarios
Scenario 1: Standard User Cost-Optimized Request
This request will match standard_user_cost_optimized route and use cost-effective models.
Scenario 2: Internal Development Testing
This request will match internal_dev_testing route because the LangDB user belongs to the "development" cost group, automatically setting metadata.group_name = "development".
Scenario 3: Fallback Route
This request doesn't match any specific conditions and will use the fallback_route with GPT-4o-mini.
{
"model": "router/dynamic",
"messages": [
{
"role": "system",
"content": "You are an expert customer support agent for TechCorp SaaS platform. You have access to user account information and can help resolve billing, technical, and feature-related issues. Always provide helpful, accurate responses and escalate complex issues when needed."
},
{
"role": "user",
"content": "Hi, I'm experiencing critical downtime with our enterprise API integration. Our production system has been failing to authenticate for the past 30 minutes and we're losing revenue. This is urgent - can you help me troubleshoot the OAuth token refresh process immediately?"
}
],
"stream": true,
"extra": {
"user": {
"tier": "premium",
"request": {
"topic": "support"
}
}
},
"router": {
"type": "conditional",
"routes": [
{
"name": "premium_support_fast_track",
"conditions": {
"all": [
{
"extra.user.tier": {
"$eq": "premium"
}
},
{
"extra.user.request.topic": {
"$eq": "support"
}
}
]
},
"targets": {
"$any": [
"anthropic/claude-4-opus",
"openai/gpt-o3",
"gemini/gemini-2.5-pro",
"xai/grok-4"
],
"filter": {
"ttft": {
"$lt": 1000
}
},
"sort_by": "ttft",
"sort_order": "min"
}
},
{
"name": "standard_user_cost_optimized",
"conditions": {
"extra.user.tier": {
"$eq": "standard"
}
},
"targets": {
"$any": [
"mistral/mistral-large-latest",
"anthropic/claude-4-sonnet"
],
"sort_by": "price",
"sort_order": "min"
}
},
{
"name": "internal_dev_testing",
"conditions": {
"metadata.group_name": {
"$eq": "development"
}
},
"targets": "google/gemini-2.5-pro"
},
{
"name": "fallback_route",
"conditions": {
"all": []
},
"targets": "openai/gpt-4o-mini"
}
],
"max_retries": 2
}
}
{
"model": "router/dynamic",
"messages": [
{
"role": "system",
"content": "You are a helpful assistant for analyzing business data and generating reports."
},
{
"role": "user",
"content": "Can you help me create a monthly sales report template with charts for our Q4 review? I need to include revenue trends, customer acquisition metrics, and regional performance comparisons."
}
],
"extra": {
"user": {
"tier": "standard"
}
},
"router": { /* same router config */ }
}
{
"model": "router/dynamic",
"messages": [
{
"role": "system",
"content": "You are a code review assistant that helps identify potential bugs and suggests improvements."
},
{
"role": "user",
"content": "Please review this TypeScript function for potential issues:\n\n```typescript\nfunction processUserData(users: any[]) {\n return users.map(u => {\n return {\n id: u.id,\n name: u.firstName + ' ' + u.lastName,\n email: u.email?.toLowerCase()\n }\n })\n}\n```"
}
]
// Note: metadata.group_name = "development" is automatically set by LangDB
// based on the user's cost group assignment - no need to specify manually
}
