Example: Building an Enterprise Routing Configuration

End-to-end example of a multi-layer enterprise routing setup with tiering, cost and fallbacks.

This example demonstrates a multi-layered routing strategy for a SaaS company that balances performance for premium users, cost for standard users, and flexibility for internal development.

Goals:

1. Provide the fastest possible responses for "premium" customers on support-related queries.

2. Minimize costs for "standard" tier users.

3. Allow the internal "development" team to test a new, experimental model without affecting customers.

Complete Chat Completion Request with Enterprise Routing:

{
  "model": "router/dynamic",
  "messages": [
    {
      "role": "system",
      "content": "You are an expert customer support agent for TechCorp SaaS platform. You have access to user account information and can help resolve billing, technical, and feature-related issues. Always provide helpful, accurate responses and escalate complex issues when needed."
    },
    {
      "role": "user",
      "content": "Hi, I'm experiencing critical downtime with our enterprise API integration. Our production system has been failing to authenticate for the past 30 minutes and we're losing revenue. This is urgent - can you help me troubleshoot the OAuth token refresh process immediately?"
    }
  ],
  "stream": true,
  "extra": {
    "user": {
      "tier": "premium",
      "request": {
        "topic": "support"
      }
    }
  },
  "router": {
    "type": "conditional",
    "routes": [
      {
        "name": "premium_support_fast_track",
        "conditions": {
          "all": [
            {
              "extra.user.tier": {
                "$eq": "premium"
              }
            },
            {
              "extra.user.request.topic": {
                "$eq": "support"
              }
            }
          ]
        },
        "targets": {
          "$any": [
            "anthropic/claude-4-opus",
            "openai/gpt-o3",
            "gemini/gemini-2.5-pro",
            "xai/grok-4"
          ],
          "filter": {
            "ttft": {
              "$lt": 1000
            }
          },
          "sort_by": "ttft",
          "sort_order": "min"
        }
      },
      {
        "name": "standard_user_cost_optimized",
        "conditions": {
          "extra.user.tier": {
            "$eq": "standard"
          }
        },
        "targets": {
          "$any": [
            "mistral/mistral-large-latest",
            "anthropic/claude-4-sonnet"
          ],
          "sort_by": "price",
          "sort_order": "min"
        }
      },
      {
        "name": "internal_dev_testing",
        "conditions": {
          "metadata.group_name": {
            "$eq": "development"
          }
        },
        "targets": "google/gemini-2.5-pro"
      },
      {
        "name": "fallback_route",
        "conditions": {
          "all": []
        },
        "targets": "openai/gpt-4o-mini"
      }
    ],
    "max_retries": 2
  }
}

Configuration Breakdown:

• Request Structure:

  • Uses "model": "router/dynamic" to enable dynamic routing

  • User information is passed via the extra.user object

  • Router configuration is specified in the router field with "type": "conditional"

• Route 1: premium_support_fast_track

  • Conditions: Applies when extra.user.tier equals "premium" AND extra.user.request.topic equals "support"

  • Targets: Routes to high-performance models (anthropic/claude-4-opus, openai/gpt-o3, gemini/gemini-2.5-pro, xai/grok-4) with filtering for fast response times (ttft < 1000ms) and sorts by minimum time-to-first-token

• Route 2: standard_user_cost_optimized

  • Conditions: Catches all requests from "standard" tier users via extra.user.tier

  • Targets: Uses cost-effective models (mistral/mistral-large-latest, anthropic/claude-4-sonnet) and sorts by minimum price

• Route 3: internal_dev_testing

  • Conditions: Applies to users in the "development" cost group via metadata.group_name (automatically set by LangDB based on the LangDB user's cost group assignment)

  • Targets: Routes directly to google/gemini-2.5-pro for isolated testing

• Route 4: fallback_route

  • Conditions: Empty conditions array ("all": []) catches all remaining requests

  • Targets: Routes to openai/gpt-4o-mini as a reliable fallback option

Key Features Demonstrated:

• Conditional Logic: Uses all operators and comparison operators ($eq, $lt)

• Target Selection: Shows both single targets and pools with filtering/sorting

• Request Context: Leverages both extra user data and metadata for routing decisions

• Cost Group Integration: metadata.group_name is automatically populated by LangDB based on the LangDB user's cost group assignment

• Retry Configuration: Includes max_retries: 2 for resilience

---

Additional Example Scenarios

Scenario 1: Standard User Cost-Optimized Request

{
  "model": "router/dynamic",
  "messages": [
    {
      "role": "system", 
      "content": "You are a helpful assistant for analyzing business data and generating reports."
    },
    {
      "role": "user",
      "content": "Can you help me create a monthly sales report template with charts for our Q4 review? I need to include revenue trends, customer acquisition metrics, and regional performance comparisons."
    }
  ],
  "extra": {
    "user": {
      "tier": "standard"
    }
  },
  "router": { /* same router config */ }
}

This request will match standard_user_cost_optimized route and use cost-effective models.

Scenario 2: Internal Development Testing

{
  "model": "router/dynamic", 
  "messages": [
    {
      "role": "system",
      "content": "You are a code review assistant that helps identify potential bugs and suggests improvements."
    },
    {
      "role": "user", 
      "content": "Please review this TypeScript function for potential issues:\n\n```typescript\nfunction processUserData(users: any[]) {\n  return users.map(u => {\n    return {\n      id: u.id,\n      name: u.firstName + ' ' + u.lastName,\n      email: u.email?.toLowerCase()\n    }\n  })\n}\n```"
    }
  ]
  // Note: metadata.group_name = "development" is automatically set by LangDB 
  // based on the user's cost group assignment - no need to specify manually
}

This request will match internal_dev_testing route because the LangDB user belongs to the "development" cost group, automatically setting metadata.group_name = "development".

Scenario 3: Fallback Route

{
  "model": "router/dynamic",
  "messages": [
    {
      "role": "user",
      "content": "What's the weather like today?"
    }
  ],
  "extra": {
    "user": {
      "tier": "basic"
    }
  },
  "router": { /* same router config */ }
}

This request doesn't match any specific conditions and will use the fallback_route with GPT-4o-mini.