zero gravitySOLUTIONS
← All insightsAI & AUTOMATION / FAILURE DIAGNOSTICS

Record AI tool failures without losing the outcome

QUICK ANSWER

A useful tool-failure record says what operation was attempted, where it failed, whether an effect may have occurred, and what recovery is allowed. Preserve correlation identifiers and redacted arguments, then separate retryable transport failures from invalid requests, authorization failures, and uncertain writes.

Inside this article 5 sections

A tool error saying “request failed” leaves an operator with the most important question unanswered: did anything happen? A read-only lookup may be safe to repeat. A customer-message send with a lost acknowledgement may already have succeeded. The same exception label cannot determine both recovery paths.

Record failure as a structured event tied to a business operation. The agent needs a concise actionable result, while operators need a protected diagnostic record. Those two views should share identifiers and meaning without sharing every raw argument or response body.

Identify the stage and the known effect state

Distinguish argument validation, authorization, dispatch, remote execution, response decoding, and business validation. A missing required field before dispatch establishes that the remote service was not called. A timeout after dispatch usually establishes less. The application should not infer non-execution from the absence of a response.

Use an effect-state field such as not_dispatched, no_effect_confirmed, effect_confirmed, or unknown. Keep it separate from error category and retryability. A rate-limit response may confirm no operation was accepted for one API, while another service's asynchronous admission semantics require different handling.

Let the tool adapter interpret provider-specific status codes and documented guarantees. Do not ask the language model to decide whether an arbitrary error string authorizes a retry. The model can suggest a next step within the tool contract; the application enforces whether that step is permitted.

Keep identifiers and remove unnecessary content

{
  "event_id": "tool-failure-demo-9",
  "task_id": "support-demo-44",
  "tool_call_id": "call-3",
  "operation_key": "support-demo-44:send-resolution:v2",
  "tool": "send_customer_message",
  "tool_version": "2",
  "attempt": 1,
  "stage": "awaiting_acknowledgement",
  "category": "timeout",
  "effect_state": "unknown",
  "arguments": {"recipient_ref": "customer-ref", "body": "[redacted]"},
  "provider_request_id": "remote-reference",
  "recovery": "reconcile_before_retry"
}

This synthetic example preserves the operation key and uncertainty while excluding the message body. In production, even customer references and provider identifiers may require restricted log access. Use a stable internal reference where operators do not need the external identifier directly.

Capture elapsed duration, configured timeout, response status, adapter version, and trace correlation. Provider request IDs are valuable when investigating remote failures; some APIs expose them in both headers and error bodies. Retain the value without assuming it is the same as the application's tool-call ID.

Validate redaction before serialization, not only in the dashboard. Raw payloads can leak through exception messages, nested objects, query strings, or automatically captured HTTP bodies. Apply an allowlist of diagnostic fields and test it with synthetic secrets. Do not hash low-entropy secrets and assume the result is anonymous.

Choose recovery from the contract

FAILURE CLASS / PERMITTED NEXT STEP
Illustrative recovery policy for application-owned tools
FailureKnown stateRecovery
Invalid enum before dispatchNo remote callReturn the allowed values; permit a bounded corrected attempt
Authorization deniedAction prohibitedStop; do not broaden credentials or switch tools to bypass it
Transient read failureNo business writeRetry within deadline and call budget
Write acknowledgement lostEffect unknownQuery by operation key or reconcile with an owner
Response parses but violates business rulesRemote outcome known; output unusablePreserve evidence and route the contract mismatch for review
Error severity and recovery are different dimensions. A small timeout can require careful reconciliation when it follows a consequential write.

Count corrected attempts and network retries against the appropriate task budget. Otherwise an agent can repeatedly submit malformed arguments while appearing to stay within the external-call allowance. Keep SDK retries visible so the application's record reflects actual attempts rather than only the outer call.

Use backoff and jitter for transient failures, but do not retry permanent validation or permission errors unchanged. Return a machine-readable reason and a short explanation that helps the agent choose an allowed correction. Avoid including stack traces or raw downstream instructions in the model-visible error.

Treat tool responses as data

A remote error body can contain arbitrary text, including material copied from a customer field or an upstream page. Do not promote that text into system instructions. The adapter should extract the expected fields and present a bounded diagnostic message.

For a failed lookup, a useful model-visible result might say that the account identifier was not found and list the permitted next action. It should not suggest searching every tenant or using an administrator endpoint. Recovery guidance must preserve the same access boundary as the original tool.

If the tool returns partial data, label the scope clearly: which page, time range, or subset succeeded. An empty list after a failed query is not evidence that no records exist. Use separate complete and partial outcomes so the agent cannot turn missing data into a factual conclusion.

Test records under realistic interruption

Inject a timeout before dispatch, a timeout after a remote write, a malformed success response, a permission denial, and a redaction-sensitive exception. Check both the model-visible result and the operator trace. They should agree on the effect state while exposing different levels of detail.

Verify that replaying a failure event does not repeat the business action. Reconciliation should use the operation ledger and remote identifier, not the presence of a log entry as a command. Keep diagnostic retention separate from the workflow's durable recovery state.

Track unresolved unknown-effect calls, repeated invalid arguments, and adapter contract failures. These measures identify work that needs engineering attention before it becomes an endless retry loop.

The tool-budget guide bounds attempts, and job reconciliation handles uncertain outcomes. Our AI integration services connect diagnostics to those recovery paths.

FROM INSIGHT TO ACTIONAI-native implementation

Make failed tool calls repairable.

We help define tool contracts, redacted traces, and recovery policies that distinguish a failed request from an action whose outcome is still unknown.

Explore AI implementationDiscuss your AI workflow

Start with your goals.
We’ll work through the next step together.