Tools & structured output

Two features carry most of the weight in production LLM systems: getting the model to call your functions, and getting the model to return output your parser can trust. Stav normalises both across four different provider contracts and makes them routable — if your request needs tools or a schema, the router only considers models that actually support them.

Function calling

Send the standard OpenAI tools array. It works on every dialect: OpenAI, Anthropic, Google and Mistral models all receive the equivalent native shape.

import json

tools = [{
    "type": "function",
    "function": {
        "name": "get_invoice",
        "description": "Look up an invoice by number.",
        "parameters": {
            "type": "object",
            "properties": {"invoice_no": {"type": "string"}},
            "required": ["invoice_no"],
        },
    },
}]

resp = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "What's the total on INV-4471?"}],
    tools=tools,
)

call = resp.choices[0].message.tool_calls[0]
print(call.function.name, call.function.arguments)

Your application executes the tool. Stav is an inference gateway — it routes the call and returns the model's decision; it does not run your functions, fetch URLs, or execute code on your behalf. The loop is the standard one: append the assistant message, append a tool message with the result, call again.

messages.append(resp.choices[0].message)
messages.append({
    "role": "tool",
    "tool_call_id": call.id,
    "content": json.dumps(lookup_invoice(...)),
})

final = client.chat.completions.create(model="auto", messages=messages, tools=tools)

tool_choice is honoured on all four dialects when sent alongside a non-empty tools array — on Anthropic it rides with the tools, and sent alone it has no effect. Tool-call deltas are emitted on the streaming surface, so streaming agent loops work normally.

parallel_tool_calls is OpenAI-only. Sending it with a model on another dialect returns a 400 naming the parameter — Stav refuses rather than pretending. If you want it, either pin an OpenAI model or leave it off and handle the sequential case.

Routing with tools

Sending tools narrows the candidate pool to models with the tool_use capability, before scoring. This is the practical reason to route agent traffic rather than pin: your agent keeps working when a model is deprecated, and it never lands on one that would drop the tools.

client.chat.completions.create(model="auto", messages=…, tools=tools)
# → only tool-capable models considered; failover chain is also tool-capable

Structured output

Four response_format variants:

{"type": "text"}                                    # default
{"type": "json_object"}                             # valid JSON, no schema
{"type": "json_schema", "json_schema": {...}}       # schema-conformant
{"type": "grammar",     "grammar": {...}}           # Stav extension

Strict JSON Schema

resp = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": contract_text}],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "clause_extraction",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "governing_law"

strict must be true. OpenAI allows strict: false, where the schema is a hint. Stav rejects it:

{
  "error": {
    "message": "response_format.json_schema.strict must be true. Stav does not support non-strict JSON Schema mode — the schema guarantee is binary.",
    "type": "invalid_request_error"
  }
}

The reasoning is that a schema which is sometimes enforced is worse than no schema, because it removes the pressure to validate while providing none of the guarantee. If you want a hint, use json_object and validate yourself.

Grammar-constrained decoding

A Stav extension, useful when the output is not JSON at all:

response_format={
    "type": "grammar",
    "grammar": {"syntax": "regex", "pattern": "^(APPROVE|REJECT|ESCALATE)$"},
}

response_format={
    "type": "grammar",
    "grammar": {"syntax": "ebnf", "pattern": "root ::= \"SELECT \" columns \" FROM \" table ..."},
}

syntax is regex or ebnf. This is the tool for classification labels, enum outputs, and constrained query generation, where a JSON wrapper is pure overhead.

Routing with schemas

json_schema and grammar require the structured_output capability, and filter the pool the same way tools do. Plain json_object does not filter — it is a prompt-level convention rather than a decoding guarantee, and most models honour it.

Combining the two

The pattern that does most of the work in real systems: tools to fetch, schema to return.

resp = client.chat.completions.create(
    model="auto",
    messages=messages,
    tools=tools,
    response_format={"type": "json_schema", "json_schema": schema},
    reasoning_effort="medium",
)

That request filters the pool to models that support tools and structured output and extended thinking, then picks the best one on your team's weights. Any model it selects can serve all three, and so can every failover candidate.

Prompt caching

cache_control markers are an inline annotation on messages, tools and the system prompt rather than a request parameter, and they are handled per provider:

• Anthropic — honoured natively, on both /v1/messages and /v1/chat/completions
• Google — stripped, because Gemini's OpenAI-compatible surface rejects the unknown key
• OpenAI, xAI, Mistral — ignored; these providers cache implicitly

Cache reads show up as usage.prompt_tokens_details.cached_tokens, which is a breakdown of prompt_tokens, not an addition to it.

For long-lived agent system prompts against a Claude model, this is the single largest cost lever available — a large system prompt plus tool schemas that would otherwise be re-billed on every turn becomes a cache read.

What Stav does not do

To be explicit, because these are commonly assumed:

• No server-side tool execution. No hosted web search, code execution, file search, URL fetch or OCR. tools entries are your functions, executed by you.
• No document store or file upload endpoint. Retrieval is your pipeline; Stav provides /v1/embeddings and /v1/rerank as building blocks.
• No computer-use tooling. Anthropic's computer-use tools work on /v1/messages with a claude-* model because that path proxies verbatim, but they are not a Stav feature.

Vision input is supported — send image content blocks as you would to OpenAI, and the router filters to vision-capable models.

Next steps

• Request parameters
• Smart Router
• Reasoning