Reliable BullMQ jobs: retries without duplicate side effects.
Make the business effect safe to repeat, rather than relying only on queue deduplication. Persist intent, use stable operation identities, reconcile uncertain external effects, and give exhausted jobs a visible recovery path.
Inside this article 5 sections +
A worker sends an invoice email. The provider accepts it. Before the worker records success, the process stops. The queue eventually runs the job again. From the queue’s perspective, recovery is working; from the customer’s perspective, the same invoice may arrive twice.
This hypothetical failure is a useful way to design background work because it separates two questions: did the worker finish, and did the business effect already happen? A reliable BullMQ implementation needs answers to both. Queue-level state alone cannot describe every external action.
Inspect the gap after the side effect
Draw the operation as a sequence of durable facts. The invoice exists. A request to notify the customer exists. The provider accepted the notification. The application recorded that acceptance. A crash can occur between any two facts. The design should explain what the next worker can learn at each point, rather than simply declaring the whole function retryable.
- Intent savedNotify invoice INV-1042
- Provider acceptsExternal effect may exist
- Worker stops hereLocal success is missing
- Job returnsReconcile before repeating
Make the operation identity represent the business intent, such as notifying a particular recipient about a particular invoice revision. It should remain stable across worker attempts. A random identifier generated inside each attempt cannot help recognize that the same intent is being executed again. Include the tenant in the identity when identifiers are only unique within a customer account.
Persist intent before publishing work
If an API saves an invoice and separately adds a queue job, either operation can succeed alone. A transactional outbox lets the API save the invoice and notification intent in one database transaction. A dispatcher then publishes pending intent to the queue. Publishing can happen more than once, so this design still requires duplicate-safe processing; it does not remove the need for idempotency.
Keep the business operation record separate from the queue’s retention policy. Completed jobs may be removed to control queue storage. The application may still need to remember that a notification was delivered. A stable queue job ID can reduce duplicate enqueues while its record exists, but it should not be the only record of a customer-visible effect.
Use database constraints and atomic state transitions to prevent two workers from independently claiming the same logical operation. A lease can help recover abandoned work, but lease expiry introduces another race: an old worker may still be running when a new one starts. External idempotency or reconciliation remains necessary even with careful local coordination.
This Node.js publisher fragment assumes queue is an existing BullMQ queue and operation came from the committed outbox. The operation ID is globally unique; queue identity is derived from it rather than generated for each dispatch.
import { createHash } from 'node:crypto';
const stableId = createHash('sha256')
.update(operation.id)
.digest('hex');
await queue.add('send-invoice-notice', {
operationId: operation.id
}, {
jobId: `invoice-notice-${stableId}`,
attempts: 4,
backoff: { type: 'exponential', delay: 1500, jitter: 0.5 }
});
The prefix avoids an all-numeric ID and the digest avoids colon separators. Repeated dispatches use the same queue ID while the earlier job exists. Once that job is removed, the ID can be admitted again, so the worker still checks the durable business operation before any provider call. Four attempts means four total processing attempts; the elapsed-time budget and permanent-error handling belong in the worker policy.
Resolve uncertainty at the provider boundary
When the external provider supports an idempotency key, derive it from the stable operation identity and reuse it according to that provider’s documented scope and retention period. Save the provider’s operation identifier as soon as it is available. A retry can then inspect the existing result instead of assuming the first call failed because the response was lost.
Some providers cannot deduplicate an action or expose its status reliably. In that case, decide which failure the business can tolerate: a possible duplicate, a possible omission, or a manual reconciliation delay. Do not label the workflow “exactly once” while leaving that choice implicit. For a customer notification, an uncertain delivery might be routed to an operator before another send is attempted.
Classify failures before adding more retries
A temporary network interruption is different from a malformed recipient address. Retry the former within a bounded policy; surface the latter for correction. Exponential backoff with jitter can spread retries after a shared outage, but the worker also needs a maximum attempt count or elapsed-time budget. Otherwise, repeated failure becomes an invisible operating expense.
Set concurrency with the downstream system in mind. More workers can empty a queue faster while overwhelming a provider or exhausting database connections. Monitor the age of the oldest waiting job and the completion rate alongside queue length. A short queue containing a single stuck, high-value operation can matter more than a long queue of harmless batch work.
Use a visible recovery state for exhausted or uncertain jobs. Include the operation ID, last known provider state, sanitized error, attempt history, and a documented next action. Operators need to know whether retrying will repeat an external effect. A button labeled “retry all” is not a substitute for that information.
Prove the recovery path with interruption tests
In a test environment, stop the worker after the provider accepts but before local completion is saved. Deliver the same intent twice. Expire a lease while a worker is delayed. Simulate a provider outage and a permanent validation failure. Verify business outcomes, not just the final queue status: one intended notification, a clear unresolved state where certainty is impossible, and no orphaned invoice intent.
These drills are particularly useful when building NestJS services and background processing, because the happy path often looks correct early. The Kafka-versus-job-queue guide explains how the same reasoning changes when retained events and independent consumers enter the design.
After making retries safe, choose a worker capacity that downstream systems can sustain. Continue with Choosing a BullMQ concurrency limit.