Routing Engine (Enterprise Only)

Use JSON rules or an embedded script to manage AI models, cut costs, and boost performance. A guide for enterprise-level routing.

LangDB Routing enables organisations to control how user requests are handled by AI models, optimizing for cost, performance, compliance, and user experience. By defining routing rules in JSON or through an embedded script, businesses can automate decision-making, ensure reliability, and maximize value from their AI investments.

What is Routing in LangDB?

Routing in LangDB is the process of directing each user request to the most appropriate AI model or service, based on business logic, user profile, request content, or real-time system metrics. This enables:

• Cost savings by using cheaper models for non-critical tasks

• Performance optimization by routing to the fastest or most reliable models

• Compliance by enforcing data residency or content policies

• Personalization by serving different user segments with tailored models

Example Use Cases

Enterprise Use Case	Business Goal	Key Variables & Metrics	Routing Logic Summary
SLA-Driven Tiering	Guarantee premium performance for high-value customers.	user_tier, ttft_ms	Route user_tier: "premium" to models with the lowest ttft_ms.
Geographic Compliance	Ensure data sovereignty and meet regulatory requirements (e.g., GDPR).	region, compliance_tags	If region: "EU", route to models with compliance_tags: "GDPR".
Intelligent Cost Management	Reduce operational expenses for internal or low-priority tasks.	user_group, price	If user_group: "internal", sort available models by price: MIN.
Model A/B Testing	Evaluate new AI models on a subset of live traffic before full rollout.	user_id (hashed), percentage split	Route 10% of traffic to a new model and 90% to the current default.
Content-Aware Routing	Improve accuracy by using specialized models for specific topics.	semantic_guardrail.result.topic	If topic: "finance", route to a finance-tuned model.
Brand Safety Enforcement	Prevent brand damage by blocking or redirecting inappropriate content.	toxicity_guardrail.result	If toxicity_score > 0.8, block the request or route to a safe-reply model.

Checkout the full examples below:

Example: Building an Enterprise Routing ConfigurationExample: Routing with Interceptors and Compliance

Routing Rule Anatomy

JSON Field	Meaning	Example Value/Usage	Business Impact
routes	List of routing rules	[ ... ]	Controls all routing logic
name	Name of the rule	"vip_fast_lane"	For audit, clarity, and reporting
conditions	When to apply this rule	{ "user_tier": { "eq": "VIP" } }	Target specific users, topics, etc.
targets	Which model(s) to use if rule matches	{ "$any": [ ... ] }	Pool of models for flexibility
$any	Pool of models to choose from	["openai/gpt-4.5", ... ]	Enables failover and optimization
sort	How to pick the best model	{ "price": "MIN" }	Optimize for cost, speed, or reliability
pre_request	Checks/enrichments before routing	[ ... ]	Add business logic, compliance, etc.
post_response	Actions after model response	[ ... ]	Moderate, fallback, or redact responses
message_mapper	Modify request/response	{ ... }	Customizes user experience

Additional Features

Script-Based Routing for Advanced Users

For advanced users, LangDB supports script-based routing using inline WASM (WebAssembly) scripts. These scripts can access all the same variables and metrics as JSON rules, enabling highly flexible, programmable routing logic for complex enterprise needs.

Router Metrics & Observability

All router metrics (including routing decisions, latencies, error rates, and more) are available via OpenTelemetry and can be exported to your observability or monitoring stack for real-time analytics and alerting.

Best Practices

• Start simple: Begin with clear, high-value rules (e.g., VIP fast lane, cost control)

• Use metrics: Leverage available metrics to optimize for business goals

• Test and iterate: Monitor outcomes and refine rules for better results

• Document rules: Use the name field and comments for clarity

• Plan for compliance: Use region and content checks for regulatory needs

• Monitor costs: Use token-based optimization to control spend