Test LLM fallbacks against the product promise
A model fallback is ready only when it preserves the application behavior contract under the failures that trigger it. Test equivalent inputs through both adapters, prevent duplicate side effects, and fail explicitly when the fallback cannot satisfy required capabilities or quality.
Inside this article 5 sections +
A fallback provider can keep requests moving during an outage and still break the product. It may use a different refusal format, omit a required field, stream content in another order, or suggest a tool action that the application interprets differently. Transport availability is only one part of continuity.
Define the feature's promise before comparing providers. For a support-response assistant, the promise might be a draft grounded in supplied account facts, no automatic customer message, and an explicit review state when evidence is insufficient. The fallback must meet that contract even if its wording and internal reasoning differ.
Normalize outcomes without hiding their meaning
Put provider-specific requests and responses behind adapters that return a small application-owned result type. Include completed, refused, incomplete, unavailable, and invalid-output outcomes. Preserve the provider request identifier and usage evidence for diagnostics, but keep those details out of ordinary product copy.
A typed envelope prevents an empty answer from being mistaken for success. It should also distinguish a provider's refusal from a network failure. Retrying a refusal through another model may violate the feature's intended policy; the fallback router needs a deliberate decision for that category.
{
"outcome": "incomplete",
"answer": null,
"tool_intents": [],
"failure": {
"stage": "generation",
"category": "output_limit",
"effect_state": "none_committed"
},
"adapter_version": "draft-adapter-v3"
}
This invented envelope omits provider payloads and sensitive request content. A completed result still goes through business validation. Do not force every provider event into completed merely to make the adapters look interchangeable.
Maintain a capability declaration for each configured route: structured output, required modalities, maximum supported input under your chosen settings, tool-call semantics, and applicable deployment constraints. If a request needs a capability the fallback lacks, stop with a useful unavailable state or use another approved path. Silent truncation changes the task.
Inject failures at different points in the request
| Injected condition | Risk | Required assertion |
|---|---|---|
| Connection fails before dispatch | Request never reached the first provider | Fallback receives the same authorized input |
| Stream ends after partial text | User may already have read a draft | No seamless splice of two incompatible answers |
| Tool call sent; acknowledgement lost | External action may have occurred | Reconcile before another action attempt |
| Provider refuses | Fallback could bypass intended policy | Apply the explicit refusal-routing policy |
| Fallback returns malformed fields | Transport success masks product failure | Reject or review; do not accept as completed |
Use a controlled adapter stub to reproduce network errors, rate-limit responses, malformed events, slow tokens, and abrupt disconnects. This exercises application behavior without relying on a real outage. Keep a small live compatibility suite too, because provider response formats and capabilities can change.
Streaming deserves its own policy. Once text reaches the user, restarting with a different provider can produce contradictory continuations. Buffer until validation when feasible, or visibly replace the incomplete draft with a newly generated answer. Do not concatenate the first provider's opening with the fallback's conclusion and call it one verified response.
Keep tool execution outside provider ambiguity
Represent model tool calls as proposed intents that the application validates and executes. Give the business operation a stable idempotency key independent of provider-generated tool identifiers. A fallback may assign a new tool-call ID to the same requested action; that must not create a second refund, message, or record.
If the first route requested an action and the result is unknown, resolve that uncertainty before granting the fallback another attempt. For read-only tools, duplicate calls may waste money or violate a budget. For writes, they can change the business outcome. The same retry rule should not govern both.
Account for SDK retries and router retries together. Two retries inside each adapter plus another fallback sequence can multiply calls beyond the intended limit. Record an overall deadline and attempt budget at the application boundary. Pass the remaining allowance into adapters rather than letting each start a fresh timer.
Evaluate the traffic that actually falls back
The fallback workload is not necessarily a random sample. Long requests, unusual languages, large attachments, or complex outputs may fail more often. Test those slices directly. A good average score on short English prompts does not establish that the fallback can handle the requests most likely to reach it.
Run paired task evaluations with the same evidence snapshot and acceptance rules. Compare factual errors, missing conditions, schema failures, unsupported claims, and reviewer corrections. Treat latency and cost as separate measures rather than allowing a faster answer to compensate numerically for an incorrect policy statement.
Test capacity under a simulated primary outage. The fallback may have a lower quota or share an upstream dependency. Limit admission and provide a queue or manual path instead of flooding the second provider until both routes fail. A circuit breaker should recover cautiously so a brief primary recovery does not create rapid oscillation.
Make degraded behavior explicit and reversible
Record which requests used the fallback, why they switched, which configuration completed them, and whether users saw partial output. Alert on prolonged fallback operation and quality regressions. Preserve a switch that disables a faulty fallback while leaving a useful unavailable or review experience.
Use model upgrade evaluation for the comparison discipline and model routing economics for path-level accounting. Our AI integration work helps make provider substitution a tested product behavior.