Deployments that protect in-flight background work.
Stop acquiring work, drain or checkpoint active jobs, and acknowledge only durable completion. Treat graceful shutdown as a bounded optimization while keeping abrupt termination recoverable through operation identity and durable progress.
Inside this article 6 sections +
A graceful worker shutdown stops acquiring new jobs, gives current work a bounded opportunity to finish or checkpoint, and acknowledges only effects that are durably complete. The queue must still support recovery if the process is killed before that sequence finishes.
On ECS and Fargate, a deployment can terminate a container while it is processing an SQS message. Extending the container stop timeout helps only when the worker handles the stop signal and the remaining work fits the available interval.
Separate process lifetime from job lifetime
Consider an illustrative media-indexing job that downloads metadata, writes several index pages, and publishes a completion event. The complete job may take longer than a container's shutdown grace period. The design therefore needs durable checkpoints or a safe restart path even when ordinary deployments usually drain cleanly.
For Linux containers, ECS sends the configured stop signal, which defaults to SIGTERM, and later sends SIGKILL if the process has not exited within its stop timeout. Fargate task definitions support a stopTimeout up to 120 seconds, with a default of 30 seconds when unspecified. Treat those as deployment constraints, not guaranteed job-completion windows.
Verify that the application actually receives the signal. An entrypoint shell that does not forward it can leave the worker unaware of shutdown. Use an appropriate process entrypoint and test the deployed container image, not only a handler invoked in a unit test.
- ImmediatelyStop admission
Cancel polling and reject newly delivered work from the closing loop.
- During drainFinish or checkpoint
Maintain ownership while current work makes durable progress.
- Before deadlineSettle completed work
Delete only messages whose business result is committed.
- Final reserveRelease resources
Stop heartbeats, flush bounded telemetry, and exit.
Close the race between polling and shutdown
Stopping the polling loop is more subtle than setting a boolean. A long-poll receive request may already be in flight when SIGTERM arrives. Its response can deliver messages after the worker has entered drain mode. The receive completion path must check the shutdown state before launching new handlers.
Cancel the outstanding receive where the client supports cancellation. If messages have already arrived, avoid beginning expensive work. Decide whether to shorten their visibility immediately or let it expire, accounting for the queue's retry and redrive policy. Record the disposition so operators can explain why an apparently received job never started.
Track active jobs explicitly. The shutdown controller should wait on the known set, not an unbounded stream of newly discovered work. This makes it possible to calculate remaining drain time and determine which jobs need a checkpoint or cancellation request.
This JavaScript fixture isolates the receive-completion race with injected adapters. It starts a receive, requests draining, and only then resolves the receive. The expected result is that the new message is returned through a release path rather than started as fresh work.
import assert from 'node:assert/strict';
let draining = false;
let finishReceive;
const received = new Promise(resolve => { finishReceive = resolve; });
const started = [], released = [];
async function pollOnce(receive, start, release) {
const messages = await receive();
for (const message of messages) {
if (draining) await release(message);
else await start(message);
}
}
const poll = pollOnce(() => received,
async m => { started.push(m.id); },
async m => { released.push(m.id); });
draining = true;
finishReceive([{ id: 'job-8' }]);
await poll;
assert.deepEqual(started, []);
assert.deepEqual(released, ['job-8']);
For SQS, the release adapter must use the receipt handle and the chosen visibility policy; it must never delete an unprocessed message. This fixture does not drain handlers that started earlier. Track those separately, keep their visibility ownership valid while they run, and stop within the container’s remaining shutdown budget.
Keep message visibility separate from completion
SQS visibility hides a received message for a period; it is not a transaction lock or a guarantee that duplicates cannot occur. A long-running handler may extend visibility while it owns the work. That extension should stop when the handler is no longer executing, otherwise recovery may be delayed unnecessarily.
Do not delete a message merely because shutdown has begun. Delete only after the business effect and its completion record are durable. If the process commits its result and dies before deletion, redelivery must discover the completed operation and avoid repeating the effect.
Likewise, reducing visibility to zero while a handler is still writing can invite overlapping processing. Coordinate cancellation and ownership release. If a dependency ignores cancellation, use an operation-level idempotency or fencing design rather than assuming the old worker has stopped because a local promise was cancelled.
Checkpoint business progress at restartable boundaries
For the media-indexing example, store the immutable source version, completed page range, and operation state outside the container. A replacement worker can resume from the next page or safely repeat the last checkpointed unit. Temporary files and in-memory counters are not durable checkpoints.
A checkpoint must describe a consistent boundary. Writing “page 20 complete” before page 20's output is durable can skip work after restart. Writing the output first and failing before the checkpoint may repeat it, so the output key or write operation should tolerate repetition.
Separate final publication from partial progress. Customers should not see a completed index while some pages remain from an older source version. Publish a manifest or final pointer only after all required units are verified. The resumable import guide applies the same principle to larger ingestion workflows.
Budget sidecars and observability during termination
If telemetry or proxy sidecars participate in the task, check container dependency and stop behavior. The application may still need a sidecar while finishing a request or exporting final spans. Do not assume that every container remains available for the same interval.
Flush telemetry with a bounded timeout. A collector outage should not consume the entire shutdown window and prevent a business checkpoint. Persist essential operation evidence in the application store, while logs and traces provide diagnostic detail.
Observe shutdown reason, active-job count, checkpoint success, completed acknowledgments, and forced termination. Distinguish planned deployment drains from crashes and task health failures. A rising number of unfinished jobs during ordinary deployments may reveal a job-size or checkpointing problem even when retries eventually succeed.
Test four interruption points
Stop a test worker before it starts a message, during a checkpointable unit, after committing the final effect, and immediately before SQS deletion. Restart another worker and verify that every accepted operation reaches one valid result without missing work or duplicate publication.
Also force termination without allowing the graceful handler to run. This is the test that distinguishes recovery design from polite shutdown behavior. Exercise a job longer than the configured grace period and confirm that the operator can trace its continuation.
The reliable background jobs guide explains durable completion and retries across queue implementations. Architecture design can align ECS task lifecycle, SQS visibility, and application checkpoints so deployments remain routine while work is in flight.