Example: Routing with Interceptors and Compliance

This example showcases a sophisticated routing configuration that uses pre-request interceptors to enforce usage quotas and guardrails, while handling region-specific compliance and prioritizing performance for premium users.

Goals:

1. Enforce a daily rate limit on all users to prevent abuse.

2. Check all requests for policy violations using a semantic guardrail.

3. Provide high-performance models for premium users in the EU, but only if they are reliable.

4. Ensure GDPR compliance by using a specialized model for requests with that requirement.

5. Provide a clear error message to users who have exceeded their quota.

Routing Configuration (router.json):

{
  "pre_request": [
    { "name": "rate_limiter", "type": "interceptor", "limit": 1000, "period": "day", "target": "user_id" },
    { "name": "semantic_guardrail", "type": "guardrail" }
  ],
  "routes": [
    {
      "name": "premium_eu_high_performance",
      "conditions": {
        "all": [
          { "metadata.user.tier": { "eq": "premium" } },
          { "metadata.region": { "eq": "EU" } },
          { "rate_limiter.result": { "eq": true } },
          { "provider.health.error_rate": { "lt": 0.02 } }
        ]
      },
      "targets": {
        "$any": ["anthropic/claude-4-opus", "openai/gpt-o3"],
        "sort": { "ttft": "MIN" }
      }
    },
    {
      "name": "gdpr_compliance_fallback",
      "conditions": {
        "metadata.compliance_tags": { "$in": ["GDPR"] }
      },
      "targets": [ { "model": "eu-specialist/gdpr-compliant-model" } ]
    },
    {
      "name": "rate_limit_exceeded_block",
      "conditions": {
        "rate_limiter.result": { "eq": false }
      },
      "message_mapper": {
        "modifier": "block",
        "content": "You have exceeded your daily quota. Please try again tomorrow."
      }
    }
  ]
}

Configuration Breakdown:

• Interceptors: Before any routing rules are evaluated, two interceptors run:

  • rate_limiter: Checks if the user has exceeded 1,000 requests today. Its result (true or false) is added to the request context.

  • semantic_guardrail: Scans the request for any content that violates policies. (This example assumes it passes). Note that semantic_guardrail is a custom guardrail you would need to create.

• Rule 1: premium_eu_high_performance

  • Conditions: This rule requires four conditions to be met: the user is "premium", is in the "EU", has not been rate-limited, and the target model provider has a low error_rate.

  • Targets: If all conditions pass, it routes to the fastest available high-performance model.

• Rule 2: gdpr_compliance_fallback

  • Conditions: This rule acts as a high-priority catch-all for any request that requires GDPR compliance, regardless of user tier.

  • Targets: It forces the request to a specific, compliant model, ensuring regulatory needs are met.

• Rule 3: rate_limit_exceeded_block

  • Conditions: This rule checks for the false result from the rate_limiter.

  • Targets: Instead of routing to a model, it uses message_mapper to block the request and return a custom error message directly to the user.