zero gravitySOLUTIONS
← All insightsAI & AUTOMATION / FAILURE DESIGN

Cancel long-running AI work without losing track of effects

QUICK ANSWER

Cancellation should stop new work, request interruption of running work, and reconcile effects already created. Persist cancellation intent separately from completion, fence late results, and show whether cleanup or external reconciliation is still pending.

Inside this article 5 sections

A stop button is a promise about the future, not a time machine. An AI workflow may already have uploaded a draft, started a paid analysis, or sent a request to another service. Closing the browser connection cannot undo those effects, and marking the job cancelled does not establish that every worker stopped.

For a concrete design, imagine a document-analysis workflow that extracts files, asks a model to produce findings, stores a report, and optionally notifies a reviewer. Cancellation should prevent publication and notification where still possible while recording anything that already occurred. The user needs a truthful final state, even when a remote operation cannot be interrupted.

Define cancellation as a durable request

Persist a cancellation request with the workflow identifier, requester, timestamp, and current execution generation. Authorize it against the workflow's owner and tenant. Return that the request was accepted, then expose progress through the normal status endpoint. Do not claim cancellation is complete merely because a message was placed on a queue.

Separate business intent from worker status. A workflow can have cancellation requested while a model request is still running. It can later reach cancelled with cleanup complete, or cancelled with reconciliation pending. A terminal business state may coexist with a maintenance task that removes temporary artifacts, provided the interface and operations model make that distinction clear.

Choose the race policy with normal completion. If the report was already committed and published before cancellation was accepted, report that completion won the race. If cancellation was recorded first, prevent a late result from being published. The decision needs an atomic boundary, not whichever status update reaches the browser last.

CANCELLATION STATES / DOCUMENT ANALYSIS
RunningNew stages may start under the current execution generation.
Cancel requestedStop scheduling; notify active workers; reject publication.
DrainingWait for interruption or reconcile outstanding external calls.
Cleanup pendingRemove temporary artifacts and resolve uncertain effects.
CancelledRecord the final effect inventory and any retained artifacts.
These are application states. They distinguish accepting the user's intent from proving that running work and its effects have been handled.

Make every long stage cooperate

Check cancellation before starting expensive work, between bounded batches, and before committing a result. A file extractor can check after each document; a paginated retrieval task can check before fetching the next page. Choose checkpoint frequency from the work's cost and acceptable stopping delay, rather than checking only once at startup.

Use the runtime's cancellation mechanism where supported, but understand its scope. Temporal's ordinary Activities need heartbeat behavior to receive cancellation; local activities behave differently. An aborted HTTP request may stop the client from waiting without guaranteeing that the remote service stopped processing. Document which adapters offer a verified remote cancellation operation.

Never perform cleanup solely in the original worker's memory. A crashed process will not execute its finalizer. Store the cleanup obligation durably so another worker can continue it. Short local cleanup may live in a finally block, but the authoritative inventory should survive a restart.

Record effects before they become ambiguous

Maintain an effect record for every externally visible operation: intended action, stable operation key, remote identifier when available, current knowledge of its outcome, and compensation policy. For the report workflow, separate temporary upload, final report publication, and reviewer notification. They have different reversibility and visibility.

An illustrative effect record can be small. The application should use an enum for status and store content in a protected artifact store rather than putting the report text into this ledger.

{
  "workflow_id": "analysis-demo-17",
  "generation": 4,
  "effect": "publish_report",
  "operation_key": "analysis-demo-17:report:v4",
  "status": "not_started",
  "remote_id": null,
  "on_cancel": "prevent_if_not_committed"
}

Compensation is a new business operation, not automatic reversal. Deleting a temporary object is often straightforward. Retracting a message cannot guarantee that nobody read it. If a committed effect cannot be reversed, preserve it in the final cancellation explanation and stop subsequent effects. Avoid running compensation without the permissions and preconditions that action requires.

Fence late results and duplicate cancellation

Assign each execution a generation or fencing token. When a worker publishes a result, condition the write on both the expected generation and an active status. A result from an old generation may be retained for debugging under the retention policy, but it must not become the current report.

For a local database, a conditional update can decide whether publication is still allowed. For an external service, the local check and remote write are not one atomic transaction. Use a durable outbox or effect worker, recheck at admission, and reconcile uncertain outcomes using the remote operation key. Be explicit about the small interval in which an accepted external action may still complete.

Repeated cancellation requests should return the same logical intent. They must not create repeated cleanup jobs or additional compensating actions. Deduplicate by workflow generation, and let cleanup operations succeed when the target has already been removed. Retrying cleanup should reduce uncertainty, not create a second workflow.

Test interruption at the boundaries

Run a synthetic workflow and pause it immediately before model dispatch, after dispatch but before response, before publication, and after publication but before notification. At each point, request cancellation and restart a worker. Assert which effects are allowed, which are prevented, and which are reported as uncertain.

Then deliver the original model response after cancellation and redeliver the cancellation message twice. The report must not reappear and cleanup must not multiply. Measure time to stop scheduling separately from time to resolve all effects; a single cancellation-duration metric conceals the harder remote cases.

The agent budget guide covers bounded execution, while background-job reconciliation explains recovery from uncertain work. Our AI integration services connect those controls to the user-facing cancellation promise.

FROM INSIGHT TO ACTIONAI-native implementation

Make cancellation a dependable workflow outcome.

We help implement durable cancellation, late-result checks, and effect reconciliation so users and operators can see what stopped and what remains.

Explore AI implementationDiscuss your AI workflow

Start with your goals.
We’ll work through the next step together.