Smart Router

The Smart Router picks a model for each request. You set the policy once, in Your workspace, and your code sends auto or a Router address instead of a model name.

resp = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Summarise this contract clause."}],
)
print(resp.model)  # the model that actually served

Routing works on /v1/chat/completions, /v1/messages and /v1/responses.

Three ways to address a model

model="auto"                          # your workspace's default Router
model="@acme/eu-only"                 # a Router your workspace created
model="anthropic/claude-sonnet-4-5"   # a pinned model — no routing

auto runs your workspace's default Router. Change the default in Your workspace → Connect → Routers and every auto call follows, with no deploy.

A Router is addressed as @workspace-slug/router-slug. It resolves only inside your own workspace:

A Router that is not your default never falls back to anything else. If its pool is empty you get an error.

A pinned model skips routing. Your workspace policy and sovereignty floor still apply. A key restricted to a Router cannot pin a model; it may only send auto or one of its Routers.

How a decision is made

1. Build the candidate pool

Start from the models your workspace may use, then remove any model that:

• is below your sovereignty floor (see Sovereignty)
• is below the quality floor, if one is set (the strictest of workspace, key and Router wins)
• has no active Provider Endpoint
• cannot serve a chat request
• fails the Router's own filters: allowed or blocked models, allowed creators, required capabilities, blocked provider countries, allowed provider jurisdictions, its own sovereignty floor (it can only tighten the workspace floor), and BYOK-only mode
• lacks a capability the request needs (below)
• has a context window too small for the request's estimated input, tool definitions included

2. Score what survives

Default weights are quality 40 / cost 30 / latency 30. Each Router carries its own weights. A workspace set to prefer sovereign models adds a bonus to sovereign open-source models.

2a. What a Router pays for

A Router can set two prices instead of three weights: what a good answer is worth and what one second of waiting costs, in your workspace currency (Custom in the Router's Objective). The router then picks the model with the best value: the chance of a good answer times its worth, minus the expected cost, minus the waiting. Presets (Balanced, Lowest cost, Lowest latency, Highest quality) are fixed pairs of these two prices. Routers that set weights keep working.

2c. Or order the pool instead of scoring it

A Router can skip the score. How this Router picks on the Router's page offers Smart Router (the scoring above: it reads how demanding the request is, then weighs quality, cost and speed for it), Cheapest first, Fastest first or Highest throughput. With one of the last three the Router takes the same pool its rules allow and tries the models and their Provider Endpoints in that one order — the lowest expected cost for this request on the price you pay, the lowest measured time to first token, or the highest measured output speed. If the first fails, the next in the same order is tried. The objective, its prices and the reasoning choice are not used; a request keeps the reasoning level it sends, else the provider's default. An endpoint with no measurement goes last.

Cheapest first with no intelligence floor picks the weakest model your rules allow, so set a floor or allow only the models you trust. The reason header says which order picked: X-Stav-Route-Reason: Cheapest eligible: mistralai/mistral-small on ionos, EUR 0.0002 expected; Router order cheapest first ..., and the request log shows every runner-up's value on the same key. A request cannot override a Router's order (provider.sort is for a request that names a model). This choice is being tested with a few workspaces first; the Router page shows it when yours is one of them.

2b. Pick the reasoning level

If you sent no reasoning level, the router chooses one for the model it picks, from the levels that model's Provider Endpoint accepts — never above the model's default unless the Router allows it, and never a level whose thinking would not fit under your max_tokens when a lower one fits. Your Router's Reasoning setting (Auto / Never / At most a level) bounds it. See Reasoning.

3. Serve and keep the runners-up

The top model serves. The next three are kept as the failover chain.

Shortcuts

• Scoring cache. Rankings are cached per policy and request shape. A hit replays the earlier ranking and its failover chain. X-Stav-Route-Reason starts with Auto-routed (cached).
• Continuity lock. In an agent loop, a request that returns tool results for calls the previous model made — or carries signed thinking — stays on that model if it is still eligible. Switching mid-loop would break the loop. X-Stav-Route-Lock names the trigger (tool_loop, signed_thinking) or says none.

Capability filtering

A hard filter, applied before scoring:

The same flags are in stav_capabilities on GET /v1/models. If no model qualifies, the request fails naming the capability.

The classifier

Stav runs a small classifier on /v1/chat/completions and /v1/messages. It runs inside Stav's gateway, not at an Inference Provider. Every way of addressing a model is classified. The difference is when, and what the result is for:

/v1/responses, /v1/embeddings, /v1/audio/*, /v1/decisions and /v1/rerank run no classifier. A routed /v1/responses request is routed without it.

What it reads. The latest user message and the last assistant reply, cut to the first 128 tokens before routing and the first 256 in the background. Before routing it also uses a result for the system prompt and tool descriptions, classified once per application and cached, and a pattern scan for credentials reads the whole message text. These are today's settings; Stav can change them.

It fails open. Classifying before routing has a time budget, 1,200 ms today. On a timeout or an error, routing continues without it. The request is then classified in the background like a named model, and that result is stored only. A turn held by the continuity lock also skips the classifier before routing and is classified in the background. A background run that fails leaves a row with empty scores.

What is stored. One row per request: complexity, domain, sensitivity and language, the hash and length of the system prompt, the role and size of each message, and the names of the tools offered. The row holds no prompt text. If you open a prompt capture window limited to flagged requests, the sensitivity result helps decide which requests are kept.

Turning it off. A workspace can stop the classifier being used for routing, but it cannot stop classification:

• Set Pre-route classification to Off in Your workspace → Govern → Key policies. An owner or admin can change it. PATCH /api/api-keys/{id} with classifier_routing_enabled sets it for one API key, and the key's setting wins over the workspace's.
• Off applies to auto and Routers. They route without the classifier.
• Off does not stop the background run. Every request on /v1/chat/completions and /v1/messages is still classified and stored, whatever the setting.

What the router does with it

Its main output is a complexity band, which becomes the expected output length. That number drives the cost and speed estimates — output tokens usually dominate the bill, so a one-line answer and a full document can have different cheapest models. Without the classifier, expected output falls back to a multiple of the input size. max_tokens always caps the estimate.

Reasoning scales the estimate up: low ×2, medium ×4, high ×6, more for xhigh and max.

The result also helps decide the kind of work a request is (code, writing, …), which sets the benchmarks the quality score reads. Sensitivity and language are recorded on the request log. They do not change the score.

When no policy applies

If auto or your default Router yields no eligible model, Stav re-scores across every active chat model. The sovereignty floor, quality floor and capability filters still apply. The reason header reads:

Auto-routed (fallback - no workspace model policy, scored across N active chat models): <slug>

A non-default Router never does this. If even the fallback finds nothing, the request fails with 404 no_eligible_models.

Failover

On /v1/chat/completions, if the chosen model fails, Stav tries the next candidate from the same ranked list. The list was built from your policy, so failover never leaves it. Pinned models have no chain; their errors reach you directly.

Failover triggers on:

• upstream 5xx
• upstream 429 (the provider client already retried and lost)
• Stav's account being refused by the provider
• connect, read, write and pool timeouts
• an open circuit breaker for the provider

Other 4xx do not fail over — a different model would fail the same way.

Streaming: failover happens only before the first chunk is sent. One case is handled for you: a reasoning model that returns an empty stream is replaced silently, because nothing has reached you yet.

What you can see

Response headers:

X-Stav-Request-Id:        chatcmpl-8f3a…
X-Stav-Model:             mistralai/mistral-small-4-119b-2603
X-Stav-Provider:          mistral
X-Stav-Sovereignty-Level: L3
X-Stav-Route-Reason:      Picked mistralai/mistral-small-4-119b-2603 for code work with the best score, 78.4; reasoning level none chosen by the router.
X-Stav-Reasoning-Chosen-By: router
X-Stav-Route-Time-Ms:     41.3
X-Stav-Route-Lock:        none
X-Stav-Router:            @acme/eu-only

response.model is the model that served, not the string you sent. Log that one.

The full decision — candidates, which filter removed what, scores, classification, estimated tokens — is on the request in Your workspace → Monitor → Request log. The Routers page also shows what routed traffic would have cost on a single frontier model, at catalogue price.

Why this pick, in the request log

Open a request under Your workspace → Monitor → Request log and expand Why it went here. It shows the kind of work the router read from the request (code, agent loop, extraction, …), the needs it matched (tool calling, JSON output, image input, …), the model and endpoint that served, the reasoning level and who chose it, whether the pick was an explored one, and the runners-up with the quality, expected cost and expected time they were scored on. The same data is GET /api/inference/logs/{request_id}/routing.

Router examples

• Compliance lane. A sovereignty floor and a list of cleared provider jurisdictions. Regulated code calls @acme/regulated.
• Cost lane. Cost weight high, a quality floor, five qualified models. Batch jobs call @acme/bulk.
• Latency lane. Latency weight high. Autocomplete calls @acme/interactive.

The point is that your code says @acme/bulk, so changing models is a change in Your workspace, not a release.

When to pin instead

• Reproducing an evaluation.
• A contract names the model.
• You depend on one model's exact output format.

Next steps

• Routing hints — per-request session identity and provider filters
• Sovereignty
• Reasoning
• Response headers