Hard-coding a model name is a decision you make once and then live with for a year. Prices move, new models ship, a provider has a bad afternoon. The Smart Router turns that decision into a policy: you describe what you care about, and Stav picks per request.
resp = client.chat.completions.create(
model="auto",
messages=[{"role": "user", "content": "Summarise this contract clause."}],
)
print(resp.model) # the model that actually served
Three ways to address it
model="auto" # your team's default routing policy
model="@acme/eu-only" # a named router your team authored
model="claude-sonnet-4-5-20250929" # pin — no routing
auto resolves through your team's default router, so changing the default in the portal changes the behaviour of every auto call without a deploy.
Named routers are @team-slug/router-slug. They resolve strictly within your own team — a router belonging to another team returns 404 router_not_found, never 403, because confirming that someone else's router exists would itself be a leak. An unknown or archived router also returns 404. A named router never silently falls back to auto; if you asked for that policy, you get that policy or an error.
What a routing decision actually does
For every request, in this order:
1. Build the candidate pool. Start from the models your team is permitted to use, then remove anything that cannot serve this request:
- below your sovereignty floor (see Sovereignty)
- below your quality floor, if one is set
- no healthy endpoint right now
- outside a named router's allowlists (models, creators, provider jurisdictions)
- missing a capability this request needs — see below
2. Score what survives. Three objectives, weighted by your team's configuration:
| Objective | What it measures |
|---|---|
| Quality | A normalised 0–100 index per model, tilted by how well the model fits the request's detected domain |
| Cost | The expected cost of this request — estimated input and output tokens against each candidate's real per-endpoint price, taking each model's cheapest endpoint |
| Speed | Expected end-to-end time — measured time-to-first-token plus estimated output length divided by measured throughput. A model with no trusted measurement yet gets a conservative constant instead |
Default weights are quality 40 / cost 30 / latency 30, set per team in Team → Routing (the portal asks them to sum to 100).
Note what "cost" and "speed" mean here: both are request-shaped. A 20-token answer and a 2,000-token answer have different cheapest models and different fastest models, and the router knows which one it is looking at before it decides.
3. Serve, and keep the runners-up. The top-scoring model serves the request; ranks two through four are retained as the failover chain.
Two things that skip the scoring step
Cached decisions. Scoring is cached per policy, eligible pool and predicted output size. A cache hit replays the earlier ranking, including its failover chain.
Semantic cache hits. When a prompt closely matches a recent one, the earlier decision is replayed directly. X-Stav-Route-Reason says so — and note that a semantic-cache replay carries no failover chain, so that request has no in-request retry.
No routing policy at all. If your team has no Smart Router configuration, or its policy yields an empty pool, plain auto and the default router fall back to the first active chat model that satisfies your sovereignty floor, tier and capability requirements — unscored. X-Stav-Route-Reason says Auto-routed (fallback — no team Smart Router config). A non-default named router never takes this path: an empty pool there is no_eligible_models, which is the point of naming it.
Capability filtering
If your request needs something, the router only considers models that have it. This is a hard filter, applied before scoring — not a preference.
| You send | Required capability |
|---|---|
a tools array | tool_use |
| an image content block | vision |
response_format with json_schema or grammar | structured_output |
reasoning_effort / reasoning / |
You can see the same flags per model in stav_capabilities on GET /v1/models. If the filter empties the pool you get a clean error naming the capability, rather than a model that quietly ignores your tools.
The classifier
Before routing, Stav runs a small multilingual classifier over the prompt. It runs in a few hundred milliseconds on truncated input, and it fails open — if it times out or errors, routing proceeds exactly as it would have without it.
What it produces and what uses it:
| Signal | Used for |
|---|---|
| Complexity → expected output length | The cost estimate, the speed estimate, and the routing cache key |
| Domain | Tilting the quality score toward models that are strong in that domain |
| Sensitivity | The sensitive-domain gate (see Sovereignty) |
| Language | Recorded for analytics |
The expected-output-length prior is the piece that does the most work. "Cheapest model" is meaningless without knowing roughly how long the answer will be — output tokens usually dominate the bill, and the ratio between a one-line answer and a full document is large enough to change which model wins.
If you send reasoning_effort, the expected output is scaled up accordingly, because a high-effort reasoning request will spend tokens you are not going to see.
Failover
If the selected model fails, Stav retries the next candidate from the same scored list. This applies to routed requests that have a candidate list — pinned models and semantic-cache replays have none, and their failures reach you directly. Because that list was already filtered by your policy, failover can never leave your allowed pool — it will not "helpfully" fall back to a foreign-jurisdiction model to keep a request alive.
Failover triggers on:
- provider
5xx - provider
429(the provider client has already retried in-band and lost, so only a different provider can help) - connect, read, write and pool timeouts
- an open circuit breaker on the provider
It does not trigger on other 4xx — those are your request's problem, and retrying elsewhere would just produce the same error more slowly.
Streaming caveat: failover only happens before the first chunk reaches you. Once bytes are on the wire the response is committed. One special case is handled for you — a reasoning model that burns its whole budget on invisible thinking tokens and returns an empty completion is failed over silently, because nothing user-visible has been sent yet.
What you can observe
Every response carries the decision:
X-Stav-Model: mistralai/mistral-nemo-instruct-2407
X-Stav-Provider: mistral
X-Stav-Model-Type: routed_commercial
X-Stav-Sovereignty-Level: L1
X-Stav-Route-Reason: Auto-routed: best score 78.4 based on team weights
X-Stav-Route-Time-Ms: 4021.3
X-Stav-Router: @acme/eu-only
X-Stav-Request-Id: chatcmpl-8f3a…
The model field in the response body is also the model that actually served — not the string you sent. If you are logging request.model, log response.model instead.
The full decision — every candidate considered, its score, the classification, the estimated tokens, the alternatives — is written to the routing log and surfaced in Monitor → Routing in the Customer Portal. That is where you go when the question is "why did it pick that".
The routers page also shows saved versus a frontier model: what your routed traffic would have cost had every eligible request gone to a single frontier model instead. It counts successful, non-BYOK requests with a known price, and it is computed on model cost before the platform fee — there is a toggle for that.
Named routers in practice
A named router is a saved policy with a stable address. Typical shapes:
Compliance lane. A sovereignty floor plus an allowlist of the providers you have cleared. Your regulated workload calls @acme/regulated and cannot drift.
Cost lane. Cost weight at 70, a quality floor set just below your acceptance threshold, allowlist of five models you have qualified. Bulk classification and enrichment call @acme/bulk.
Latency lane. Latency weight at 70, precision tier off. Interactive autocomplete calls @acme/interactive.
The value is not the weights — it is that your application code contains @acme/bulk instead of a model name, so re-qualifying models is a portal change rather than a release.
When not to route
- Reproducing an evaluation. Pin the model, or your baseline moves under you.
- A contract names a model. Pin it, and let your team's blocked jurisdictions enforce the rest.
- You depend on one model's exact output formatting. Pin it, and revisit when you have a schema instead.
Everything else is better routed.
Per-request provider preferences
provider narrows which providers may serve an explicitly pinned model:
client.chat.completions.create(
model="qwen/qwen3-6-27b",
messages=[…],
extra_body={"provider": {"only": ["heimdall"]}},
)
Supported today: only, ignore, max_price, quantizations. These apply to explicit model selection; team policy and named routers are the mechanism for shaping auto.
Next steps
- Sovereignty — floors, levels, the sensitive-domain gate
- Reasoning — how effort changes routing
- Response headers