Classify each request as simple, medium, or complex and route it to the cheapest model that can handle it.
Most production traffic is not uniformly hard. A greeting, a one-line factual lookup, and a request to design a distributed rate limiter all arrive on the same endpoint — and if every one of them is served by your most capable model, you pay top-tier prices for the easy majority.Auto Routing classifies each request into a complexity tier — simple, medium, or complex — and routes it to the target you configured for that tier. Your application keeps calling one virtual model name; the AI Gateway decides which model actually serves each request.Unlike the other routing strategies, the routing decision depends on what is in the request, not on weights, priorities, or measured latency.
Complexity-based routing is generally available — it is offered to every tenant in the console with no flag to request or enable. Note that this makes the routing type available to select; it does not change the default, which is still weight-based routing when you create a virtual model.
Auto Routing is available on virtual models only. It is not a valid rule type in the tenant-level Routing Config YAML (gateway-load-balancing-config).
The virtual model must also meet these conditions, or it is rejected when you save it:
Model types must be limited to chat, completion, and responses. Embedding, image, audio, rerank, and moderation are not supported.
Each target serves exactly one tier, and the same model cannot appear under two tiers.
The classifier model (LLM strategy only) must be a catalog chat model, not a virtual model.
If any target defines metadata_match, non-matching targets are dropped before anything else happens. Classification and escalation only ever consider the remaining targets.
2
Check the conversation pin
For a multi-turn request, the AI Gateway looks up the tier and target that previously served the conversation. Conversation pinning is automatic and requires no virtual-model configuration.
3
Classify the request
The heuristic classifier (default) or the LLM classifier assigns the current turn to simple, medium, or complex. If the conversation was previously pinned higher, it stays at that higher tier. A conversation already pinned to complex skips classification.
4
Build the target chain
The targets for the selected tier come first. This is either the current classification or a higher tier retained by the conversation pin. The escalation chain follows — higher tiers, then lower tiers.
5
Send the request, falling forward on failure
The first target is attempted with its own retry configuration. If it fails with a fallback status code, the AI Gateway moves to the next target in the chain.
6
Update the conversation pin
Once a target returns a successful response, the AI Gateway records the tier and target for the next turn. The pin can move to a higher tier, but it never moves down during an active conversation.
Multi-step summaries, drafting, standard code, light reasoning
A balanced mid-tier model
complex
Deep reasoning, long context, hard problem-solving, agentic chains
Your most capable model
These descriptions are the same ones shown next to each tier in the dashboard.
You do not have to configure all three tiers. Configuring only simple and complex, for example, is valid — requests classified as medium escalate to complex. See Escalation and fallback.
Two strategies are available. The heuristic classifier is the default and costs nothing; the LLM classifier is more accurate on ambiguous prompts but adds a model call to each classified turn. A turn whose conversation is already pinned to complex skips classification because it cannot move any higher.
Heuristic (default)
LLM classifier
Added latency
None — runs in-process
One chat completion before the request is forwarded
Both strategies look at the same two pieces of text, extracted from the request body:
The last user message — the current turn.
The last system or developer message — for context. This includes the top-level system field on Anthropic /messages requests and the top-level instructions field on /responses requests.
Earlier turns, assistant replies, and tool messages are ignored by the classifier. Consistency across a multi-turn conversation comes from conversation pinning instead.
Classification scans at most 8,000 characters of user text and 2,000 characters of system text. A prompt longer than that is still recognised as a long prompt.
The heuristic classifier reads the prompt text, looks for a fixed set of signals, and picks a tier. It needs no configuration — this is what you get if you do not set classification_strategy at all.
routing_config: type: complexity-based-routing classification_strategy: type: heuristic # this is the default — the block can be omitted
What the heuristic looks for
These signals push a request toward a higher tier, listed from most to least influential:
Signal
What it matches
Code
Programming keywords in the user or system text — function, class, api, docker, sql, language names, and similar.
Explicit reasoning requests
Phrases in the user text that ask the model to reason — step by step, think through, pros and cons, compare and contrast, explain your reasoning, and similar.
Technical vocabulary
Systems and ML terms — architecture, distributed, neural network, concurrency, throughput, and similar.
Prompt length
Long user prompts.
Multi-step structure
Cues such as first … then, step 2, numbered or lettered lists.
Many questions at once
More than three question marks in one request.
These push a request toward a lower tier:
Signal
What it matches
Lookup and small-talk phrasing
what is, who is, how many, define, hello, thanks, and similar.
Short prompts
Very short user prompts.
Reasoning override. Two or more distinct reasoning phrases in the user text always route to complex, whatever the other signals say.Empty prompts. A request with no classifiable text is treated as simple.The signals and keyword lists are fixed and cannot be customised. If they don’t match how difficulty shows up in your traffic, use the LLM classifier instead.
Worked examples
Prompt
Signals matched
Tier
Hi, thanks!
short prompt, small talk
simple
What is the capital of France?
short prompt, lookup phrasing
simple
Write a Python function that reverses a linked list.
code
medium
Design a distributed rate limiter. Walk me through the trade-offs step by step and explain your reasoning.
two reasoning phrases → override
complex
Our gRPC service has high tail latency under concurrency. Analyze this and break down the likely causes.
The AI Gateway classifies the request by calling a fast chat model you nominate. Use this when the heuristic’s keyword-based view is too coarse for your traffic — for example domain-specific prompts where difficulty is not signalled by vocabulary.
routing_config: type: complexity-based-routing classification_strategy: type: llm-classifier model: openai-main/gpt-4o-mini # must be a catalog chat model timeout_ms: 2000 # default: 2000 fallback_strategy: # required type: heuristic
fallback_strategy is required whenever classification_strategy.type is llm-classifier. The console pre-fills it with the heuristic fallback, so this only affects configs written by hand or through the API — omitting the key there is rejected with must have required property 'fallback_strategy'.{"type": "heuristic"} reproduces the behaviour that used to apply implicitly, so add it to existing configs to keep them working unchanged. classification_strategy itself remains optional and still defaults to the heuristic strategy.
Each classifier call is a billable gateway request. It appears in your request logs, and its tokens and cost are attributed to the same tenant and subject as the request that triggered it. Use a small, fast model and keep timeout_ms tight.
How the classifier call works
The classifier is sent the request’s system context and user text, along with a short instruction block describing the three tiers.
Its reply is constrained to a JSON schema ({"tier": "simple" | "medium" | "complex"}) and capped at 256 output tokens. Models that ignore the schema and answer in prose still work — the AI Gateway extracts the first tier word it finds.
The call is abandoned after timeout_ms (default 2000), and the fallback strategy applies.
If the classifier times out, errors, returns a non-2xx status, or returns nothing usable, the configured fallback strategy decides the tier. The classifier failure never fails the request.
classification_strategy: type: llm-classifier model: openai-main/gpt-4o-mini fallback_strategy: type: heuristic # classify with the built-in heuristic instead
classification_strategy: type: llm-classifier model: openai-main/gpt-4o-mini fallback_strategy: type: static default_tier: medium # always use this tier; default: medium
Use heuristic to keep a content-aware decision during an outage, or static to send everything to one known tier. default_tier: complex preserves answer quality at higher spend; default_tier: simple does the opposite.
Auto Routing does not just pick one target. It builds a fully ordered chain, so a request still gets served when the tier it classified into has no usable target.For a request classified into tier T, the order is:
Targets in tier T, in the order you declared them.
Targets in higher tiers, working upward one tier at a time.
Targets in lower tiers, working downward one tier at a time.
So a request classified as simple is attempted on simple, then medium, then complex. A request classified as complex is attempted on complex, then medium, then simple. A request fails only after every target has been tried.Each target in the chain uses its own retry_config, fallback_status_codes, and fallback_candidate settings — these behave exactly as they do for the other routing strategies.
Escalation also covers tiers you did not configure. With only simple and complex targets, a request classified as medium is escalated straight to complex and the response header reports the escalation. This is how you run a two-tier setup.
Unhealthy-target cooldown does not apply here. Under the other strategies a failing target is demoted to the end of the list; under Auto Routing, target order is always tier order. Per-target rate limits and budgets are still enforced, and a rate-limited or over-budget target is skipped.
Auto Routing automatically keeps a multi-turn conversation on a consistent tier and target. You do not configure sticky_routing, add a conversation header, or choose a TTL on the virtual model.The pin behaves like a ratchet:
The first turn is classified normally. After a target responds successfully, its tier and target are recorded.
Each later user turn is classified again.
If the new turn is harder, the conversation moves up to the higher tier and the pin is updated after a successful response.
If the new turn is easier, the existing higher tier wins. A short follow-up such as “thanks” therefore cannot move an active conversation back to a cheaper model.
Once a conversation reaches complex, later turns skip classification because no higher tier exists.
This balances cost and consistency: conversations can gain capability when the task becomes harder, but do not lose capability midway through the interaction.Nothing is required from your application: keep sending the conversation history as the API you call already expects. A pin lasts through 10 minutes of inactivity and is refreshed on every turn, so an active conversation stays on its tier.
Single-turn requests — legacy /completions, and /responses with a string input — have no conversation to follow, so they are classified individually every time.
A target that failed is never recorded as the conversation’s next target.
We benchmarked Auto Routing against sending every request to a single top-tier model (Claude Opus 5), over the same prompts. Cost is computed from the tokens each request actually used at list prices, and quality retained is the router’s pass rate divided by the baseline’s. Every answer is graded deterministically; generated code is run against unit tests, and math and multiple-choice are matched to answer keys.
Realistic production traffic (chat, developer, agent)
80%
cost-only (no ground truth)
Against a prior-generation top model the graded savings are about 50%; the exact figure depends on which model you route away from. Auto Routing was also faster on average, since most requests skip the top-tier reasoning model.A few honest notes: savings depend on your traffic mix, and the free heuristic classifier gives up some accuracy on short-but-hard prompts (the LLM classifier recovers most of it at lower savings). Figures measured August 2026 on a Claude Haiku / Sonnet / Opus tier ladder.To measure your own savings, see Observability — the gateway logs the resolved tier for every request.
Auto Routing is configured on a virtual model, alongside the other routing strategies. See Create a virtual model for the full walkthrough.
1
Create the virtual model and choose Complexity
Go to AI Gateway → Models → Virtual Model and add or edit a model. Give it a Name, and under Model Types select only from Chat, Completion, and Responses.Under Balance them across following targets based on, choose Complexity.
Creating a virtual model with Complexity selected as the routing type
2
Pick a classification strategy
Heuristic Classification is selected by default and needs nothing else.Choose LLM Classification instead to classify with a model, then set Classifier Model, Timeout (ms), and Fallback Strategy. Switching Fallback Strategy to Static Fallback adds a Default Tier field.
LLM Classification with a small, fast model as the classifier
3
Set a target for each tier
The form has one Target slot under each of Simple, Medium, and Complex. Pick a model for each tier you want to serve and leave the rest empty — an empty tier is skipped and its traffic escalates.Each target carries its own Fallback status codes, plus optional Retries, Override Params, Override Headers, and Metadata Filters — all behaving as they do for the other routing strategies.
A target for each tier, from lowest to highest cost
Current dashboard limitation. The virtual model form supports one target per tier. To configure multiple targets in the same tier, apply the virtual model manifest with tfy apply or the API instead.
routing_config: type: complexity-based-routing # How requests are classified. Optional; defaults to { type: heuristic }. classification_strategy: type: heuristic | llm-classifier # llm-classifier only: model: string # catalog chat model FQN (not a virtual model) timeout_ms: integer # default: 2000 fallback_strategy: # required when type is llm-classifier type: heuristic | static default_tier: simple | medium | complex # static only; required when type is static load_balance_targets: - target: string # catalog model FQN, e.g. openai-main/gpt-4o tier: simple | medium | complex # required; each target serves exactly one tier retry_config: attempts: integer # retries on the SAME target; default: 0 delay: integer # ms between retries; default: 100 on_status_codes: string[] # default: ["429","500","502","503"] fallback_status_codes: string[] # codes that move to the NEXT target in the chain # default: ["401","403","404","408","429","500","502","503"] fallback_candidate: boolean # eligible to receive escalated traffic; default: true metadata_match: # all pairs must match resolved request metadata key: value headers_override: set: header-name: header-value remove: - header-name override_params: temperature: number max_tokens: integer prompt_version_fqn: string
The common starting point: cheap model for the easy majority, mid-tier for everyday work, top model for the hard tail. No classifier cost, no extra latency.
LLM classifier with a conservative static fallback
Use a small model to classify, and route everything to the complex tier if that classifier is ever unavailable — quality never regresses, spend rises only during the outage.
Per-tier resilience with retries and a second complex target
Multiple targets in the same tier are attempted in declaration order before escalating. This shape requires the YAML manifest or API — the dashboard form allows one target per tier.
Every response carries an x-tfy-applied-rules header. For Auto Routing it includes a complexity block with the tier that was served and how the tier was decided, plus the full ordered chain in routing_model_order:
cause tells you which mechanism produced the tier:
cause
Meaning
heuristic
The built-in heuristic classifier decided the tier — either as the configured strategy, or as the fallback after an LLM classifier failure.
llm_classifier
The LLM classifier returned the tier.
session_pin
An existing conversation pin kept the request on a higher tier than the current turn required. Classification is skipped only when the pin is already complex.
default
The LLM classifier failed and a static fallback strategy applied its default_tier.
reason on each entry in routing_model_order tells you why that target is in that position:
reason
Meaning
complexity-tier
Target belongs to the tier the request classified into.
complexity-escalation
Target belongs to a different tier and is in the chain as failover.
complexity-session-pin
Target was placed first because a sticky session pin selected it.
The actual model that served the request is also returned in the x-tfy-resolved-modelresponse header.
Every routing decision increments a counter, so you can see your tier mix and how often escalation happens.
Metric
Type
Labels
ai_gateway_complexity_routing_decisions_total
Counter
tenant_name, model_name, decided_tier, resolved_tier, cause
decided_tier — the tier assigned to the current turn before target fallback. For a complex pin that skips classification, this remains complex.
resolved_tier — the tier of the first target actually attempted.
When the two differ, check cause. A session_pin means the conversation deliberately stayed on a higher tier than the current turn’s classification. Another cause means the classified tier had no eligible target and routing escalated to another tier. A persistent non-pin gap is a signal that a tier is misconfigured or missing. See Prometheus and Grafana integration for scraping setup.
Each LLM classifier call produces a ComplexityBasedRouting: LLMClassifier span, with the classifier’s own chat-completion span nested beneath it — so classifier latency is visible separately from the served model’s latency.
The request span carries tfy.complexity.tier and tfy.complexity.cause.
The gateway logs each decision with the tier, the cause, the signals that matched, and the resolved model — useful for checking why a given request landed on the tier it did.
These configurations are rejected when you save the virtual model, with a 400:
Error
Fix
cannot use Auto Routing with model_types [embedding]. Auto Routing supports only [chat, completion, responses]
Remove the unsupported model types, or serve them from a separate virtual model. Mixed lists are rejected too — declaring both chat and embedding fails.
Duplicate target across tiers
Assign the model to a single tier.
classifier model "…" could not be resolved in this tenant
Use a model FQN that exists in your model catalog.
classifier model "…" cannot be a virtual model
Point the classifier at a catalog model.
classifier model "…" must support [chat]
Pick a model whose model types include chat.
The three classifier checks also run at request time, so a classifier model that is later deleted or changed surfaces the same error on the request rather than misrouting silently.
That depends on your traffic mix, so measure it. Break ai_gateway_complexity_routing_decisions_total down by resolved_tier to see what share of traffic landed on each tier, and compare per-model cost in analytics. Your saving is that share multiplied by the price difference between tiers.
Can I use Auto Routing in the tenant-level Routing Config YAML?
No. It is a virtual-model-only routing type. Create a virtual model and point your clients at it.
Can I customise how the heuristic classifies requests?
No, its signals are fixed. If the heuristic misclassifies your traffic, switch to the LLM classifier and nominate your own judging model.
Does the LLM classifier call show up in my usage and cost?
Yes. It appears in request logs, and its tokens and cost are attributed to the same tenant and subject as the request that triggered it. Use a small, cheap model and keep timeout_ms tight. A conversation already pinned to complex does not make another classifier call.
What happens if the classifier is down?
The request is never failed by a classifier problem. On timeout, error, non-2xx, or an unusable reply, the configured fallback_strategy decides the tier — the built-in heuristic by default, or a fixed default_tier if you chose static.
What if the tier a request classified into has no target?
It escalates to the next higher tier, and if there is none, down to lower tiers. The response header reports the tier actually used, and the metric records decided_tier and resolved_tier separately so you can spot it.
Can a conversation change models mid-conversation?
Yes, but only when it needs more capability or the previous target fails. Conversation pinning prevents later turns from moving to a lower tier. A harder follow-up can move the conversation to a higher tier, and a successful fallback can update the target used by later turns.
Do unhealthy-target cooldowns apply?
No. Targets are ordered by tier and escalation, never reordered by health. Per-target retries, fallback status codes, rate limits, and budgets all still apply.
Does it work with streaming?
Yes. Classification finishes before the upstream request is made, so streaming responses are unaffected.
Does it work with the Anthropic Messages API and the Responses API?
Yes. A chat virtual model serves both /chat/completions and /messages, and a responses virtual model serves /responses. In both cases the top-level system prompt (system and instructions respectively) is included as classification context, and multi-turn conversations are pinned the same way.
