stuntman / providers / ai models

AI models

The model APIs your app calls, answered without a model: rules shape the reply, failures come on cue, spend is counted.

Stands in for

  • OpenAI (chat completions, Responses, embeddings, models)
  • Azure OpenAI (deployments and v1)
  • Anthropic Messages (count_tokens, streaming)
  • Google Gemini (generateContent, embedContent, countTokens)
  • OpenRouter
  • DeepSeek

Everything AI models gives you

  1. 01

    Six APIs, your SDK

    Change only the SDK's base URL. Any key is accepted and never sent anywhere.

  2. 02

    Rules shape answers

    First match wins on provider, API, model prefix, the last user message, the system prompt or a tool offered.

  3. 03

    Tool calls on cue

    The model calls your tool first with the arguments you set, then answers the text once the result comes back.

  4. 04

    JSON and long answers

    Answer a JSON object, pad to about N tokens, or stop by length as max_tokens would.

  5. 05

    Streamed or not

    Each provider's own response and error shapes, streamed or in one piece.

  6. 06

    Every call kept

    Prompt, answer, model, tokens in and out, cached tokens, cost and why it got that answer.

  7. 07

    Spend you can read

    ai_spend per provider; prices per million tokens are editable with set_ai_price.

  8. 08

    Stable embeddings

    Embeddings are deterministic: the same text always gives the same vector.

The screens

Each number is a control or a reading on the real screen. Everything you can click there, your agent can do with a tool.

Every model call, with why

Every model call, with why: the AI models screen in Stuntman
1Six providers

OpenAI, Azure OpenAI, Anthropic, Gemini, OpenRouter and DeepSeek, each with its calls.

1 of 5
2Point the SDK here

Any key is accepted; it is never sent anywhere.

2 of 5
3Why this answer

The rule that matched, the default answer, or a failure someone forced.

3 of 5
4The prompt

System, user and the tools the app offered, as the app sent them.

4 of 5
5Tokens and cost

Tokens in and out, cached tokens and the cost from your prices.

5 of 5
  1. Six providers

    OpenAI, Azure OpenAI, Anthropic, Gemini, OpenRouter and DeepSeek, each with its calls.

  2. Point the SDK here

    Any key is accepted; it is never sent anywhere.

  3. Why this answer

    The rule that matched, the default answer, or a failure someone forced.

  4. The prompt

    System, user and the tools the app offered, as the app sent them.

  5. Tokens and cost

    Tokens in and out, cached tokens and the cost from your prices.

Rules and failures

Rules and failures: the AI models screen in Stuntman
1A rule with a tool call

When the message contains refund: call lookup_order, then answer the text.

1 of 5
2JSON on a system prompt

A system prompt that asks to classify gets a JSON object.

2 of 5
3Add a rule

Name, condition, answer text, and an optional tool with its arguments.

3 of 5
4Failures on the next calls

429 with Retry-After, 500, 529, filter, length, bad_json, slow and cut.

4 of 5
5How many

Bound the failure to the next N calls; kind=none clears it.

5 of 5
  1. A rule with a tool call

    When the message contains refund: call lookup_order, then answer the text.

  2. JSON on a system prompt

    A system prompt that asks to classify gets a JSON object.

  3. Add a rule

    Name, condition, answer text, and an optional tool with its arguments.

  4. Failures on the next calls

    429 with Retry-After, 500, 529, filter, length, bad_json, slow and cut.

  5. How many

    Bound the failure to the next N calls; kind=none clears it.

Break it on purpose

The failures you need to test, exactly when you ask: from the screen or by a tool call, each one shown on the timeline with who caused it.

  • Rate limit

    429 with Retry-After, in the provider's error shape

    force_ai_failure kind=429
  • Server error

    500 from the provider

    force_ai_failure kind=500
  • Overloaded

    529 overloaded

    force_ai_failure kind=529
  • Content filtered

    The answer is blocked by the content filter

    force_ai_failure kind=filter
  • Cut by max_tokens

    The answer stops with a length finish reason

    force_ai_failure kind=length
  • Broken tool arguments

    A tool call whose JSON arguments do not parse

    force_ai_failure kind=bad_json
  • Slow model

    40 ms per token

    force_ai_failure kind=slow
  • Stream cut

    The stream stops halfway

    force_ai_failure kind=cut

Why not the real sandbox

Your AI agent drives it

Claude Code, Codex or any MCP client connects to /mcp (or calls /api over plain HTTP), sets the stand-in up, breaks it on cue and reads what your app did.

you

Make our support bot retry when Anthropic is overloaded and still call lookup_order for refunds.

set_ai_rule{"name": "refund", "contains": "refund", "text": "The refund takes 5 days.", "tool_call": {"name": "lookup_order", "arguments": {"order_id": "A-1042"}}}
force_ai_failure{"provider": "anthropic", "kind": "529", "count": 1}
wait_for_ai_call{"provider": "anthropic", "contains": "refund"}
→ ok 529 once, then 200: rule refund, tool call lookup_order

MCP tools

  • list_ai_callsCalls with provider, model, tokens, cost and why
  • get_ai_callOne call with the whole request and response
  • wait_for_ai_callWaits until the app calls a model
  • list_ai_rulesRules, the default answer and prices
  • set_ai_ruleAdds or changes a rule
  • delete_ai_ruleRemoves a rule
  • force_ai_failureThe next calls fail on purpose
  • ai_spendCalls, tokens and cost per provider
  • set_ai_pricePrice per million tokens, and the default answer

The same over HTTP

  • GET/api/ai/calls
  • GET/api/ai/calls/wait
  • PUT/api/ai/rules
  • POST/api/ai/force
  • GET/api/ai/spend