zero gravitySOLUTIONS
← All insightsBACKEND & RELIABILITY / TRADEOFF ANALYSIS

Kafka vs. job queues: decide what must survive.

QUICK ANSWER

Choose a job queue when a worker must complete a task. Consider Kafka when several independent consumers need a retained event history and their own progress. In either design, external side effects need explicit duplicate handling and recovery.

Inside this article 5 sections

A product team says it needs a message system. One engineer wants Kafka because several services will consume events. Another wants BullMQ because the immediate requirement is generating reports. Both may be describing a real need, but they are talking about different units of work.

Ask what should remain after processing. If a worker generates a report, the useful outcome is the report and its delivery status. If an account changes plan, the event may remain useful to analytics, billing, and a future audit view independently. That distinction often explains more than a comparison of headline throughput.

Two contracts hidden behind the word “message”

A job asks a worker to do something: generate a file, refresh a search document, or send a notification. Its lifecycle is organized around completion, failure, retries, and ownership of the task. A retained event records something that happened. Multiple consumers can use that history for different purposes, with separate progress and recovery decisions.

Kafka organizes records in partitioned logs and lets consumer groups track their own progress. Ordering is tied to partitions, not a universal order across the entire system. BullMQ is commonly used to coordinate background jobs; SQS offers managed queueing with its own delivery and ordering options. These products have overlapping capabilities, so use the contract to narrow the choice rather than treating the categories as absolute limits.

ORIGINAL COMPARISON / WHAT IS THE DURABLE ASSET?
TASK OWNERSHIP

Generate report

WaitingWorkerResult

The work is complete when the intended result exists.

SHARED HISTORY

Plan changed

Event 01Event 02Event 03
Billing positionAnalytics position

Each consumer decides how far it has processed.

Conceptual responsibilities, not benchmark results or a complete feature matrix.

Make replay a business decision

Suppose a new analytics model needs six months of account-change history. A retained event stream can be useful if those records were preserved with the fields the model needs. Replaying them rebuilds a projection without asking the transactional application to reconstruct every historical state.

Now consider the same replay reaching a customer messaging consumer. Re-sending six months of “your plan changed” emails would be a failure. Give consumers explicit replay behavior. A projection may rebuild into a fresh table. A notification service may recognize previously handled business events and suppress historical sends. Retention creates an opportunity to replay; it does not decide whether replay is appropriate.

Schema evolution matters as much as storage duration. If an old event contains only a plan name, a future calculation cannot recover a price that was never recorded. Decide which facts belong in the event and which references can safely be resolved later. Keep the original occurrence time distinct from the time a consumer processes the event.

Compare the operating questions

Questions to take into an architecture review
RequirementJob-oriented designRetained-event design
Primary outcomeA named task completes.Consumers interpret a durable fact.
Recovery focusRetry or reconcile the incomplete operation.Resume or replay from a known position.
Adding a consumerDefine routing and a separate task lifecycle.Define a consumer’s history and progress.
Operational questionsWho owns stuck work and downstream capacity?Who owns partitions, lag, retention, and schema compatibility?
External effectsRequire duplicate-safe handling.Require duplicate-safe handling.

That final row is deliberate. Kafka transactions can provide specific guarantees when processing and writing Kafka records under the right configuration. They do not automatically make an unrelated payment API or email provider part of the same transaction. Standard SQS queues can deliver a message more than once. Whatever transport you select, trace the actual business effect through failures.

A small hybrid can have a clear purpose

In an illustrative SaaS design, account changes enter a retained stream because billing analysis and product analytics need independent history. One consumer decides that a customer report should be generated and creates a job for a worker. The stream preserves the fact; the queue manages the task. Each boundary has a distinct responsibility.

This adds infrastructure and failure paths. It is justified only if the application needs both contracts. A small team producing a few scheduled reports may be better served by a job queue and a well-designed database record. A team already operating Kafka may avoid introducing another system when its existing platform can meet the requirement safely. Operational familiarity is a real input, not an embarrassment to hide from the design review.

Leave a decision someone can revisit

Write down the selected transport, the business unit it carries, the ordering requirement, retention, replay behavior, and ownership of failed processing. Include the condition that would trigger reconsideration: a new independent consumer, a larger history requirement, or an operational burden the team can no longer support. This makes the architecture adaptable without turning every new feature into another platform debate.

For implementation details, see reliable BullMQ jobs. For event consumers with different business responsibilities, see PostHog, Customer.io, and Redshift. Our architecture consulting can help compare these options against your workload and team.

A transport choice also needs an operating plan for exhausted work. Explore Dead-letter queues need an owner.

FROM INSIGHT TO ACTIONSoftware architecture consulting

Choose the architecture your workload needs.

Work through messaging, data ownership, and operational tradeoffs with us. Leave with a clear design and an implementation path matched to your team.

Explore architecture consultingDiscuss your architecture

Start with your goals.
We’ll work through the next step together.