Recovering jobs with uncertain side effects
Reconcile the business operation before restarting a job that may already have caused an external effect. Separate confirmed success, confirmed non-execution, and unresolved uncertainty. Replay only when the evidence and the provider contract make the next action safe.
Inside this article 5 sections +
A job marked failed does not prove that its external action failed. The worker may have submitted a shipment request, lost the response, and stopped before saving the provider reference. Replaying that job immediately can create another shipment while making the queue dashboard look healthier.
Recovery begins with the business operation: which shipment was intended, what evidence exists, and what actions remain permissible? Queue state is one piece of that evidence. It describes execution bookkeeping, not the complete history of a remote system.
Stop competing recovery attempts
Assign one recovery owner and identify the logical operation before changing anything. Capture the tenant, order revision, stable operation key, queue job identifiers, attempt times, and last saved provider response. Avoid copying addresses or full customer payloads into incident notes when identifiers and sanitized evidence are sufficient.
Prevent automated and manual recovery from racing. Depending on the system, this may mean placing the operation in a reconciliation state, excluding it from automatic dispatch, or acquiring a guarded recovery claim. Pausing a queue does not necessarily stop work that is already active. Inspect running attempts and account for delayed responses before assuming the operation is idle.
Keep the original record intact. Deleting and re-creating a job can remove the evidence needed to distinguish an old attempt from a new request. If a replacement job is necessary, link it to the original operation and record why it was created. Recovery should add an explanatory event to the history.
Classify the provider outcome
| Evidence | Business interpretation | Permitted next action |
|---|---|---|
| Provider confirms matching shipment | The intended effect exists | Attach its reference and finalize local state |
| Authoritative lookup confirms no execution | A new attempt may be needed | Replay under the original identity and current policy |
| Lookup is delayed, incomplete, or unavailable | Outcome remains unknown | Hold, query again, or request an explicit business decision |
| Provider finds a conflicting shipment | Intent and external state disagree | Investigate before either completion or replay |
Prefer the provider's operation identifier when one was saved. Otherwise, use a supported lookup by the original idempotency key or business reference. Verify that the match includes the correct account, order revision, and meaningful request fields. A shipment for the same customer on the same day is not sufficient evidence that this operation completed.
Understand the lookup's visibility guarantees. Some systems expose newly accepted operations only after a delay. An empty search result during that interval does not establish non-execution. Record the provider's documented delay or consistency behavior in the runbook and use a bounded observation period before escalating.
This Python decision function accepts normalized evidence collected by a provider adapter. “Absent” is stronger than a lookup timeout: it means the provider’s supported lookup conclusively found no matching operation. The fixture deliberately refuses to turn uncertainty into another write.
def recovery_action(provider_state, identity_matches, replay_allowed):
if not identity_matches:
return "review_identity"
if provider_state == "succeeded":
return "reconcile_success"
if provider_state == "rejected":
return "record_rejection"
if provider_state == "absent" and replay_allowed:
return "retry_original_intent"
return "hold_for_evidence"
assert recovery_action("unknown", True, True) == "hold_for_evidence"
assert recovery_action("succeeded", True, False) == "reconcile_success"
assert recovery_action("absent", True, True) == "retry_original_intent"
assert recovery_action("succeeded", False, True) == "review_identity"
Persist the evidence and take the chosen action under the operation’s concurrency guard. The boolean for replay permission represents checks on intent validity, provider contract, and retention; it is not something the model or operator should set merely to clear a backlog. Re-read current state before committing a repair.
Check whether the original key still protects a retry
Provider idempotency can make replay safer, but its scope and retention are part of the contract. Check whether the key belongs to this account, endpoint, and operation type; whether its remembered result still exists; and whether repeating the request requires identical parameters. An expired key may be treated as a new request even if its text is unchanged.
Do not generate a fresh key simply because the original call timed out. Conversely, do not reuse an old key for a genuinely different shipment revision. Keep the business operation identifier stable across attempts, and create a new business operation only when the intended effect has changed and the previous one has been resolved.
If the provider offers neither reliable status lookup nor usable deduplication, the system cannot manufacture certainty locally. The recovery owner must choose between possible duplication, possible omission, and waiting for more evidence. Present that choice with the affected orders and consequences. A human decision is useful only when the uncertainty is explicit.
Repair the local record before resuming automation
For confirmed success, save the provider reference and reconcile the application state through a guarded transition. Re-read the current operation first: another webhook may have finalized it during investigation. A conditional update should make repeated reconciliation harmless and prevent stale evidence from overwriting a later state.
For confirmed non-execution, verify that the action is still wanted. The customer may have canceled the order while the job was stuck. Revalidate current policy, retain the original intent history, and schedule one controlled attempt. Replaying yesterday's payload without checking today's business state can be incorrect even when duplication is impossible.
For unknown outcomes, make the hold visible with an owner, next check time, and escalation condition. Do not continually retry just to refresh an error timestamp. Report unresolved business operations separately from failed worker attempts so that operational reporting reflects the work that still needs a decision.
Rehearse the uncertainty before an incident
In a controlled environment, interrupt a worker after the provider accepts and before the application records the response. Have an operator follow the runbook without asking the original developer what happened. Verify that the evidence is enough to reach the right branch, that concurrent webhook updates are safe, and that every action leaves an audit record.
The reliable BullMQ design guide explains the preventative operation model, while messaging choices clarify delivery responsibilities. Background processing implementation should include this recovery path as part of the feature, because an unexplained retry button transfers application ambiguity directly to the operator.