zero gravitySOLUTIONS
← All insightsBACKEND & RELIABILITY / OUTAGE SIMULATION

Preventing queue retry storms

QUICK ANSWER

Prevent retry storms by limiting total retry work, spreading eligible attempts, and controlling admission while a dependency recovers. Jitter reduces synchronization, but it does not replace an attempt budget, a deadline, or a shared capacity limit.

Inside this article 4 sections

A dependency outage turns ordinary work into retry work. If every worker retries on the same schedule, the dependency receives a synchronized burst precisely when it has the least spare capacity. When the service starts recovering, the accumulated backlog can keep it overloaded even though the original fault has ended.

The remedy has three parts: limit how much extra work retries may create, spread that work over time, and control how quickly recovery admits it. Exponential backoff addresses timing. It does not, by itself, establish a capacity budget or decide which jobs are still worth completing.

Count attempts across all retrying layers

Imagine an export worker that allows four total job attempts. Its HTTP client independently allows three total requests per call. One logical export can therefore produce up to twelve provider requests if each job attempt reaches that client and exhausts its own policy. This is illustrative multiplication, not a measured workload, but it reveals why individually modest settings can combine badly.

Inventory retries in the queue, SDK, HTTP client, gateway, and calling service. Choose an owner for the overall retry policy and disable or constrain redundant layers. Preserve a total elapsed-time budget as well as an attempt budget. A job that is no longer useful after an hour should not spend the next day dutifully retrying.

ILLUSTRATIVE RETRY AMPLIFICATION

4 job attempts × 3 client attempts = 12 possible calls

Logical work
One export requested by one customer.
Physical work
Every initial call and retry consumes provider capacity.
Control point
Apply one overall budget and count all attempts against the shared dependency limit.
Hidden retry layers multiply potential load. Counting only queue attempts understates the dependency's exposure.

Make retry eligibility a business decision

Separate temporary unavailability from permanent rejection and uncertain completion. A malformed document request will not become valid after a delay. An authorization failure may need configuration repair. A timeout after an external side effect needs reconciliation before another attempt. Treating all exceptions identically wastes capacity and can repeat business effects.

Record an error category alongside a sanitized diagnostic detail. Keep categories stable enough to drive policy and dashboards. A provider's exact message text is useful evidence, but a fragile string match should not be the only thing preventing a dangerous retry. When classification is uncertain, choose a visible holding path rather than unlimited automatic attempts.

Honor provider retry guidance where the protocol supplies it, while preserving the operation's deadline. A retry-after indication does not make an obsolete job useful again. For scheduled customer messages, late delivery may be worse than dropping the expired intent with a clear reason. Define that rule with the product owner before an outage forces an improvised decision.

Combine jitter with shared admission

Backoff increases the delay between attempts, and jitter varies eligible delays so workers do not all wake together. BullMQ provides these mechanisms, but configuration details should match the installed version. Verify the actual generated delay range rather than assuming every system uses the same definition of full or partial jitter.

Even perfectly spread retries can exceed the provider's capacity when the backlog is large. Put first attempts and retries behind the same dependency budget, with an explicit allocation if one class deserves priority. Otherwise, new traffic can starve recovery or recovery can starve current customers. A rate limit and a concurrency limit may both be required because they control different units.

During broad failure, stop spending attempts at the normal rate. A dependency-level circuit or pause policy can hold eligible work while limited probes check recovery. Coordinate that behavior across replicas; a hundred independent “single probes” are still a hundred requests. Keep the pause observable so that queue growth is understood rather than mistaken for an idle worker defect.

This Python fixture demonstrates full jitter with a remaining-deadline check. It is an application policy example, separate from BullMQ’s built-in jitter options. The caller supplies monotonic elapsed milliseconds and an injected random sample so tests can reproduce boundary cases.

def next_delay_ms(attempt, remaining_ms, draw):
    if attempt < 1 or not 0 <= draw < 1:
        raise ValueError("invalid attempt or random sample")
    attempt_budget_ms = 1200
    if remaining_ms <= attempt_budget_ms:
        return None
    ceiling = min(8000, 500 * 2 ** min(attempt - 1, 4))
    delay = int(ceiling * draw)
    return delay if delay + attempt_budget_ms <= remaining_ms else None

assert next_delay_ms(1, 5000, 0.5) == 250
assert next_delay_ms(8, 1500, 0.5) is None
assert next_delay_ms(1, 1000, 0.0) is None

The 1,200 ms attempt allowance and 8,000 ms delay ceiling are synthetic settings. Replace them with the actual operation budget and bound the attempt count separately. A permitted delay still needs shared admission control: thousands of independently valid retries can overload the same dependency.

Simulate the recovery, not only the outage

Build a controlled provider stub with three phases: healthy responses, sustained unavailability, then gradual recovery with a deliberately limited acceptance rate. Feed a representative mix of new jobs and existing retry candidates. Record offered calls, accepted calls, useful completions, retry counts, oldest job age, and jobs that expire or require review.

Compare synchronized fixed delays with a bounded jittered policy under the same arrival stream. The purpose is to inspect request distribution and business completion, not to publish a universal performance ranking. Keep all random seeds and test inputs so the experiment can be repeated, and vary seeds before assuming one attractive run represents typical behavior.

At the recovery transition, release only a controlled amount of work. Increase admission if error rate and latency remain acceptable; reduce it again if the provider struggles. Verify that an operator can pause the recovery without losing operation identities, and that a worker restart does not reset the business deadline or total attempt budget.

Use the results to set alert thresholds and an outage runbook. The concurrency tuning guide establishes ordinary capacity, while reliable job design establishes duplicate safety. Queue implementation should make retry load visible as a first-class part of that capacity plan.

FROM INSIGHT TO ACTIONCustom software development

Build recovery that does not repeat the outage.

We can implement bounded retry policies and run controlled dependency-failure drills before production traffic depends on them.

Explore software developmentDiscuss your build

Start with your goals.
We’ll work through the next step together.