zero gravitySOLUTIONS
← All insightsAI & AUTOMATION / RESILIENCE GUIDE

Test LLM fallbacks against the product promise

QUICK ANSWER

A model fallback is ready only when it preserves the application behavior contract under the failures that trigger it. Test equivalent inputs through both adapters, prevent duplicate side effects, and fail explicitly when the fallback cannot satisfy required capabilities or quality.

Inside this article 5 sections

A fallback provider can keep requests moving during an outage and still break the product. It may use a different refusal format, omit a required field, stream content in another order, or suggest a tool action that the application interprets differently. Transport availability is only one part of continuity.

Define the feature's promise before comparing providers. For a support-response assistant, the promise might be a draft grounded in supplied account facts, no automatic customer message, and an explicit review state when evidence is insufficient. The fallback must meet that contract even if its wording and internal reasoning differ.

Normalize outcomes without hiding their meaning

Put provider-specific requests and responses behind adapters that return a small application-owned result type. Include completed, refused, incomplete, unavailable, and invalid-output outcomes. Preserve the provider request identifier and usage evidence for diagnostics, but keep those details out of ordinary product copy.

A typed envelope prevents an empty answer from being mistaken for success. It should also distinguish a provider's refusal from a network failure. Retrying a refusal through another model may violate the feature's intended policy; the fallback router needs a deliberate decision for that category.

{
  "outcome": "incomplete",
  "answer": null,
  "tool_intents": [],
  "failure": {
    "stage": "generation",
    "category": "output_limit",
    "effect_state": "none_committed"
  },
  "adapter_version": "draft-adapter-v3"
}

This invented envelope omits provider payloads and sensitive request content. A completed result still goes through business validation. Do not force every provider event into completed merely to make the adapters look interchangeable.

Maintain a capability declaration for each configured route: structured output, required modalities, maximum supported input under your chosen settings, tool-call semantics, and applicable deployment constraints. If a request needs a capability the fallback lacks, stop with a useful unavailable state or use another approved path. Silent truncation changes the task.

Inject failures at different points in the request

FALLBACK DRILL / FAILURE LOCATION MATTERS
Different failures require different continuity decisions
Injected conditionRiskRequired assertion
Connection fails before dispatchRequest never reached the first providerFallback receives the same authorized input
Stream ends after partial textUser may already have read a draftNo seamless splice of two incompatible answers
Tool call sent; acknowledgement lostExternal action may have occurredReconcile before another action attempt
Provider refusesFallback could bypass intended policyApply the explicit refusal-routing policy
Fallback returns malformed fieldsTransport success masks product failureReject or review; do not accept as completed
A timeout before any output and a timeout after a possible side effect are different states. The test harness must preserve that distinction.

Use a controlled adapter stub to reproduce network errors, rate-limit responses, malformed events, slow tokens, and abrupt disconnects. This exercises application behavior without relying on a real outage. Keep a small live compatibility suite too, because provider response formats and capabilities can change.

Streaming deserves its own policy. Once text reaches the user, restarting with a different provider can produce contradictory continuations. Buffer until validation when feasible, or visibly replace the incomplete draft with a newly generated answer. Do not concatenate the first provider's opening with the fallback's conclusion and call it one verified response.

Keep tool execution outside provider ambiguity

Represent model tool calls as proposed intents that the application validates and executes. Give the business operation a stable idempotency key independent of provider-generated tool identifiers. A fallback may assign a new tool-call ID to the same requested action; that must not create a second refund, message, or record.

If the first route requested an action and the result is unknown, resolve that uncertainty before granting the fallback another attempt. For read-only tools, duplicate calls may waste money or violate a budget. For writes, they can change the business outcome. The same retry rule should not govern both.

Account for SDK retries and router retries together. Two retries inside each adapter plus another fallback sequence can multiply calls beyond the intended limit. Record an overall deadline and attempt budget at the application boundary. Pass the remaining allowance into adapters rather than letting each start a fresh timer.

Evaluate the traffic that actually falls back

The fallback workload is not necessarily a random sample. Long requests, unusual languages, large attachments, or complex outputs may fail more often. Test those slices directly. A good average score on short English prompts does not establish that the fallback can handle the requests most likely to reach it.

Run paired task evaluations with the same evidence snapshot and acceptance rules. Compare factual errors, missing conditions, schema failures, unsupported claims, and reviewer corrections. Treat latency and cost as separate measures rather than allowing a faster answer to compensate numerically for an incorrect policy statement.

Test capacity under a simulated primary outage. The fallback may have a lower quota or share an upstream dependency. Limit admission and provide a queue or manual path instead of flooding the second provider until both routes fail. A circuit breaker should recover cautiously so a brief primary recovery does not create rapid oscillation.

Make degraded behavior explicit and reversible

Record which requests used the fallback, why they switched, which configuration completed them, and whether users saw partial output. Alert on prolonged fallback operation and quality regressions. Preserve a switch that disables a faulty fallback while leaving a useful unavailable or review experience.

Use model upgrade evaluation for the comparison discipline and model routing economics for path-level accounting. Our AI integration work helps make provider substitution a tested product behavior.

FROM INSIGHT TO ACTIONAI-native implementation

Keep fallback behavior inside a clear product contract.

We help implement provider adapters, fault-injection tests, and release criteria so an outage does not silently change what your AI feature promises.

Explore AI implementationDiscuss your AI workflow

Start with your goals.
We’ll work through the next step together.