zero gravitySOLUTIONS
← All insightsBACKEND & RELIABILITY / PARTITION DESIGN EXPERIMENT

Kafka keys for account-level ordering

QUICK ANSWER

Use an account key when events for that account must share a Kafka partition, but recognize that one hot account then shares a serial ordering boundary. Broker order does not guarantee ordered external effects: producers, consumer concurrency, retries, and partition changes must preserve the same business contract.

Inside this article 5 sections

An account identifier is a good Kafka key when the business needs ordered events within an account. It also makes that account a unit of contention. Adding partitions can distribute different accounts, but it cannot divide one account's strictly ordered work without changing the ordering contract.

Consider a credit-management service emitting account opened, credit limit changed, and account suspended events. A downstream fulfillment system must not apply a stale limit after a suspension. The partition key is part of this design, but it is only one of several places where order can be lost.

Define which order the consumer actually needs

Kafka records have an order within a partition. That order reflects publication into the log, not necessarily the original order of database transactions across several producers. If two services publish changes concurrently, the broker cannot infer which business revision should come first merely because both records have the same key.

Include a stable event identifier, account identifier, and business sequence or revision where the domain can define one. The sequence should come from the authority that commits the account change. Consumers can then distinguish a duplicate revision, an expected successor, and a gap requiring investigation. Event timestamps are useful context but can be equal, delayed, or affected by clocks.

Illustrative event contract, not a broker configuration:
{
  "event_id": "evt-482",
  "account_id": "acct-27",
  "account_revision": 18,
  "event_type": "account.suspended",
  "schema_version": 1
}
Kafka record key: acct-27

State whether every account revision produces an event in this stream. If revisions can skip because unrelated updates are omitted, a consumer cannot interpret every numerical gap as loss. Use a stream-specific sequence or define a reconciliation query that distinguishes expected omissions from missing required events.

Model skew before increasing parallelism

Illustrative key choices for account operations
KeyOrdering scopeConsequence
Account IDAll records for one accountSimple account sequence; a hot account remains concentrated
Account + order IDOne order within an accountMore distribution; account-wide changes need coordination
Random event IDNo useful account groupingEven distribution cannot preserve account order
Dedicated hot-account routeExplicit isolated account streamLimits collateral impact but adds routing and migration work
Changing the key changes the business concurrency model. Distribution is not a free improvement when operations depend on account-wide state.

Create a synthetic workload with many ordinary accounts and one account producing a large share of records. Keep the exact distribution in the fixture rather than assuming uniform keys. Measure per-partition lag and per-account completion age. Overall consumer throughput can look healthy while the hot account and unrelated accounts sharing its partition fall behind.

If events for separate orders are genuinely independent, a narrower key can help. But an account suspension may affect every order, so the design now needs a way to coordinate that account-wide state with order-local streams. A materialized account revision check or another explicit protocol may be appropriate. Randomly salting the account key without that protocol merely distributes the race.

Preserve completion order inside the consumer

A consumer can fetch records in partition order and then destroy that order by dispatching them to an unconstrained thread pool. Revision eighteen might complete before revision seventeen, and an offset commit can acknowledge work that has not safely finished. Keep ordered processing within the required scope or maintain a contiguous completed-offset frontier with carefully bounded concurrency.

For the credit example, conditional state updates provide another defense. A duplicate event should not repeat the business action, and a stale revision should not overwrite a newer projection. An unexpected future revision can be held for reconciliation. These rules belong in durable consumer state if they must survive restarts and partition reassignment.

External effects require additional care. Updating a local projection in sequence does not guarantee that remote fulfillment requests finish in sequence. If suspension must precede any new fulfillment, the consumer needs an explicit check or serialized effect boundary. A timed-out remote call remains uncertain even if its Kafka record has a well-defined offset.

Exercise retries and reassignment deliberately

Verify producer settings against the deployed Kafka client version. Idempotent production and its compatible retry settings help preserve broker-side behavior, but they do not deduplicate arbitrary external effects or repair incorrect business sequencing from multiple writers. Keep producer identity, event identity, and account revision conceptually separate.

Stop a consumer after it commits a business change but before committing its offset. On restart, the replay should be harmless. Stop another consumer while an external call is unresolved, then trigger partition reassignment. Confirm that the new owner does not assume the old call disappeared. Leases and ownership transitions require a recovery rule for work already outside the process.

A poison record creates an ordering choice. Skipping it into a dead-letter path may allow later revisions to execute without their prerequisite. Pause the affected scope, reconcile the missing transition, or define a domain-specific skip rule. A generic “continue on error” policy cannot determine whether a suspended account is safe to fulfill.

Expose the oldest uncompleted offset alongside the highest fetched offset. A consumer that fetches rapidly while one earlier operation remains unresolved can accumulate a large hidden working set. Bound that in-flight distance and pause acquisition when necessary, so preserving a contiguous commit frontier does not require retaining unlimited records.

Treat partition expansion as a routing migration

With partitioners that map a key using the partition count, increasing that count can move future records for an existing account while older records remain in the previous partition. Two consumers may then process that account concurrently. Validate the actual partitioner and design a cutover, drain, routing version, or consumer sequence protocol before expansion.

Test the proposed change with retained records and active publication, not only an empty topic. The messaging decision guide establishes Kafka's role, and dead-letter operations develops the failure decisions. Event-processing implementation should make the partition contract visible in schemas, consumers, and operational runbooks.

FROM INSIGHT TO ACTIONCustom software development

Make Kafka ordering match your account model.

We can design partition contracts, consumer sequencing, and migration tests that expose hot-key and replay behavior.

Explore software developmentDiscuss your build

Start with your goals.
We’ll work through the next step together.