zero gravitySOLUTIONS
← All insightsBACKEND & RELIABILITY / RETENTION WORKSHEET AND DRILL

Redis memory pressure in job queues

QUICK ANSWER

Protect a Redis-backed job system by budgeting retained work, keeping payloads small, and separating queue storage from disposable cache eviction. BullMQ requires a no-eviction policy for its queue keys; reaching the memory limit should produce controlled admission failure, not arbitrary key loss. Retention and durable business recovery must be designed together.

Inside this article 5 sections

Redis memory pressure in a job system is not the same problem as a full cache. Cache entries may be disposable. Queue keys describe pending work, locks, schedules, and execution state, so arbitrary eviction can remove information the worker needs to operate correctly.

For BullMQ, configure the documented no-eviction policy and plan what happens when writes can no longer be admitted. This preserves the distinction between a rejected enqueue and a silently damaged queue. It also makes capacity planning and producer error handling essential parts of the application.

Count every retained class of work

Inventory waiting, active, delayed, completed, failed, and recurring-job data. Include job logs, stack traces, return values, event streams, and auxiliary keys. A queue with few waiting jobs can still consume substantial memory because completed results or failed payloads remain retained.

Measure representative key groups using the tools available for the deployed Redis version and hosting service. Do not scan a large production database carelessly during an incident. Establish a bounded diagnostic method in advance and record per-queue trends so an operator can distinguish growing backlog from growing history.

Separate queue Redis from a general-purpose cache where practical. A shared instance creates conflicting policies: the cache wants eviction, while BullMQ requires its queue structures to remain intact. A shared memory budget can also let an unrelated cache surge prevent business work from being enqueued.

Queue memory ownership and retention decisions
Data classWhy it existsSafe policy question
Waiting and delayed jobsAccepted future workCan intake be bounded or paused before capacity is exhausted?
Completed jobsDiagnostics and recent execution historyWhat age and count are operationally useful?
Failed jobsInvestigation and replay evidenceWhere does unresolved business intent survive retention?
Logs and return valuesSupporting diagnosticsCan large content move to durable object storage?
Different data classes have different deletion rules. Clearing a queue indiscriminately is not a retention policy.

Build a payload estimate, then measure overhead

Suppose an illustrative workload completes 40,000 jobs daily and retains seven days of completed jobs. At 2 KiB of payload per job, the retained payload alone is 560,000 KiB, or about 546.9 MiB. That excludes queue indexes, metadata, logs, allocator overhead, replication buffers, and unfinished work.

Illustrative completed-payload estimate:
40,000 jobs/day × 7 days = 280,000 retained jobs
280,000 × 2 KiB = 560,000 KiB
560,000 / 1,024 = 546.875 MiB

Add measured metadata and infrastructure headroom.
Do not treat payload bytes as total Redis memory.

Repeat the estimate for failed and delayed work using realistic retention periods and payload distributions. Failure payloads may be larger because they contain diagnostic context. A dependency outage can simultaneously increase waiting jobs, retries, failure records, and logs, so steady-state averages are not sufficient.

Store large immutable inputs and outputs in durable object storage and put references plus integrity metadata in jobs. The worker must still verify tenant access and input existence. Moving content out of Redis introduces an object-retention contract: deleting the object before the job finishes makes a small queue payload useless.

Keep queue cleanup separate from business memory

Configure completed and failed retention by both age and count where supported. A count cap bounds bursts, while age retains a useful investigation window. Understand that BullMQ auto-removal is triggered lazily by job activity; it is not a guarantee that every expired job disappears at the exact retention timestamp.

Removing a completed job also removes queue-level knowledge of its job identifier. A later enqueue with that identifier may be accepted again. Therefore, do not use retained queue records as the only protection against repeating a business effect. Keep durable operation identity and outcome in application storage where the required lifetime can be enforced.

Failed-job cleanup needs an owner. Before removing evidence, ensure unresolved operations are represented in a recovery ledger or another durable worklist. A retention policy that deletes failures faster than the team investigates them silently abandons accepted work. Store concise diagnostics rather than indefinitely retaining full sensitive payloads in every attempt record.

Handle rejected admission explicitly

When memory pressure prevents an enqueue, the API must not claim that background work was scheduled unless durable intent exists elsewhere. A transactional outbox can preserve the intent for later dispatch, provided its own backlog is bounded and visible. Without that design, return a clear failure that the client can retry safely under the original operation identity.

Workers also need to tolerate Redis command failures without assuming a business effect failed. A provider action may succeed before completion state can be written. Preserve reconciliation evidence and avoid creating replacement jobs with new identities merely to escape the error. Memory pressure is an infrastructure fault with business-state consequences.

Leave headroom for the hosting model's replication, persistence, and failover behavior. Redis dataset size and process memory are not interchangeable. Review the managed service's metrics and limits, and test maintenance or failover under representative occupancy rather than sizing only from a quiet dataset snapshot.

Alert on the rate of memory growth as well as current utilization. A large import or provider outage can consume the remaining budget faster than a static threshold gives operators time to respond. Pair the alert with an intake control and an estimated remaining window based on recent measured growth, clearly treating that estimate as conditional.

Practice a controlled capacity incident

Use an isolated test Redis instance with a deliberately constrained memory budget. Populate representative jobs, retained results, and failures. Increase intake until admission fails, then verify that producers report the correct outcome, outbox intent remains recoverable, and workers do not lose track of external effects.

Apply the approved retention policy or capacity increase, then drain at a bounded rate. Confirm that job structures remain valid and reconcile accepted intents against completed operations. The concurrency guide controls processing pressure, and failed-work operations assigns recovery decisions. Background processing implementation should treat memory limits as a normal operational boundary with an explainable failure path.

FROM INSIGHT TO ACTIONCustom software development

Make queue memory limits operationally safe.

We can implement bounded retention, admission handling, and a recovery drill that protects the integrity of your background work.

Explore software developmentDiscuss your build

Start with your goals.
We’ll work through the next step together.