zero gravitySOLUTIONS
← All insightsBACKEND & RELIABILITY / OPERATOR RUNBOOK

Recovering jobs with uncertain side effects

QUICK ANSWER

Reconcile the business operation before restarting a job that may already have caused an external effect. Separate confirmed success, confirmed non-execution, and unresolved uncertainty. Replay only when the evidence and the provider contract make the next action safe.

Inside this article 5 sections

A job marked failed does not prove that its external action failed. The worker may have submitted a shipment request, lost the response, and stopped before saving the provider reference. Replaying that job immediately can create another shipment while making the queue dashboard look healthier.

Recovery begins with the business operation: which shipment was intended, what evidence exists, and what actions remain permissible? Queue state is one piece of that evidence. It describes execution bookkeeping, not the complete history of a remote system.

Stop competing recovery attempts

Assign one recovery owner and identify the logical operation before changing anything. Capture the tenant, order revision, stable operation key, queue job identifiers, attempt times, and last saved provider response. Avoid copying addresses or full customer payloads into incident notes when identifiers and sanitized evidence are sufficient.

Prevent automated and manual recovery from racing. Depending on the system, this may mean placing the operation in a reconciliation state, excluding it from automatic dispatch, or acquiring a guarded recovery claim. Pausing a queue does not necessarily stop work that is already active. Inspect running attempts and account for delayed responses before assuming the operation is idle.

Keep the original record intact. Deleting and re-creating a job can remove the evidence needed to distinguish an old attempt from a new request. If a replacement job is necessary, link it to the original operation and record why it was created. Recovery should add an explanatory event to the history.

Classify the provider outcome

Reconciliation decision for an illustrative shipment request
EvidenceBusiness interpretationPermitted next action
Provider confirms matching shipmentThe intended effect existsAttach its reference and finalize local state
Authoritative lookup confirms no executionA new attempt may be neededReplay under the original identity and current policy
Lookup is delayed, incomplete, or unavailableOutcome remains unknownHold, query again, or request an explicit business decision
Provider finds a conflicting shipmentIntent and external state disagreeInvestigate before either completion or replay
A transport timeout belongs in the unknown row. It is not evidence that the provider performed no action.

Prefer the provider's operation identifier when one was saved. Otherwise, use a supported lookup by the original idempotency key or business reference. Verify that the match includes the correct account, order revision, and meaningful request fields. A shipment for the same customer on the same day is not sufficient evidence that this operation completed.

Understand the lookup's visibility guarantees. Some systems expose newly accepted operations only after a delay. An empty search result during that interval does not establish non-execution. Record the provider's documented delay or consistency behavior in the runbook and use a bounded observation period before escalating.

This Python decision function accepts normalized evidence collected by a provider adapter. “Absent” is stronger than a lookup timeout: it means the provider’s supported lookup conclusively found no matching operation. The fixture deliberately refuses to turn uncertainty into another write.

def recovery_action(provider_state, identity_matches, replay_allowed):
    if not identity_matches:
        return "review_identity"
    if provider_state == "succeeded":
        return "reconcile_success"
    if provider_state == "rejected":
        return "record_rejection"
    if provider_state == "absent" and replay_allowed:
        return "retry_original_intent"
    return "hold_for_evidence"

assert recovery_action("unknown", True, True) == "hold_for_evidence"
assert recovery_action("succeeded", True, False) == "reconcile_success"
assert recovery_action("absent", True, True) == "retry_original_intent"
assert recovery_action("succeeded", False, True) == "review_identity"

Persist the evidence and take the chosen action under the operation’s concurrency guard. The boolean for replay permission represents checks on intent validity, provider contract, and retention; it is not something the model or operator should set merely to clear a backlog. Re-read current state before committing a repair.

Check whether the original key still protects a retry

Provider idempotency can make replay safer, but its scope and retention are part of the contract. Check whether the key belongs to this account, endpoint, and operation type; whether its remembered result still exists; and whether repeating the request requires identical parameters. An expired key may be treated as a new request even if its text is unchanged.

Do not generate a fresh key simply because the original call timed out. Conversely, do not reuse an old key for a genuinely different shipment revision. Keep the business operation identifier stable across attempts, and create a new business operation only when the intended effect has changed and the previous one has been resolved.

If the provider offers neither reliable status lookup nor usable deduplication, the system cannot manufacture certainty locally. The recovery owner must choose between possible duplication, possible omission, and waiting for more evidence. Present that choice with the affected orders and consequences. A human decision is useful only when the uncertainty is explicit.

Repair the local record before resuming automation

For confirmed success, save the provider reference and reconcile the application state through a guarded transition. Re-read the current operation first: another webhook may have finalized it during investigation. A conditional update should make repeated reconciliation harmless and prevent stale evidence from overwriting a later state.

For confirmed non-execution, verify that the action is still wanted. The customer may have canceled the order while the job was stuck. Revalidate current policy, retain the original intent history, and schedule one controlled attempt. Replaying yesterday's payload without checking today's business state can be incorrect even when duplication is impossible.

For unknown outcomes, make the hold visible with an owner, next check time, and escalation condition. Do not continually retry just to refresh an error timestamp. Report unresolved business operations separately from failed worker attempts so that operational reporting reflects the work that still needs a decision.

Rehearse the uncertainty before an incident

In a controlled environment, interrupt a worker after the provider accepts and before the application records the response. Have an operator follow the runbook without asking the original developer what happened. Verify that the evidence is enough to reach the right branch, that concurrent webhook updates are safe, and that every action leaves an audit record.

The reliable BullMQ design guide explains the preventative operation model, while messaging choices clarify delivery responsibilities. Background processing implementation should include this recovery path as part of the feature, because an unexplained retry button transfers application ambiguity directly to the operator.

FROM INSIGHT TO ACTIONCustom software development

Turn job recovery into a controlled operation.

We can implement provider reconciliation, safe replay eligibility, and an operator view tied to durable business state.

Explore software developmentDiscuss your build

Start with your goals.
We’ll work through the next step together.