Kafka keys for account-level ordering
Use an account key when events for that account must share a Kafka partition, but recognize that one hot account then shares a serial ordering boundary. Broker order does not guarantee ordered external effects: producers, consumer concurrency, retries, and partition changes must preserve the same business contract.
Inside this article 5 sections +
An account identifier is a good Kafka key when the business needs ordered events within an account. It also makes that account a unit of contention. Adding partitions can distribute different accounts, but it cannot divide one account's strictly ordered work without changing the ordering contract.
Consider a credit-management service emitting account opened, credit limit changed, and account suspended events. A downstream fulfillment system must not apply a stale limit after a suspension. The partition key is part of this design, but it is only one of several places where order can be lost.
Define which order the consumer actually needs
Kafka records have an order within a partition. That order reflects publication into the log, not necessarily the original order of database transactions across several producers. If two services publish changes concurrently, the broker cannot infer which business revision should come first merely because both records have the same key.
Include a stable event identifier, account identifier, and business sequence or revision where the domain can define one. The sequence should come from the authority that commits the account change. Consumers can then distinguish a duplicate revision, an expected successor, and a gap requiring investigation. Event timestamps are useful context but can be equal, delayed, or affected by clocks.
Illustrative event contract, not a broker configuration:
{
"event_id": "evt-482",
"account_id": "acct-27",
"account_revision": 18,
"event_type": "account.suspended",
"schema_version": 1
}
Kafka record key: acct-27
State whether every account revision produces an event in this stream. If revisions can skip because unrelated updates are omitted, a consumer cannot interpret every numerical gap as loss. Use a stream-specific sequence or define a reconciliation query that distinguishes expected omissions from missing required events.
Model skew before increasing parallelism
| Key | Ordering scope | Consequence |
|---|---|---|
| Account ID | All records for one account | Simple account sequence; a hot account remains concentrated |
| Account + order ID | One order within an account | More distribution; account-wide changes need coordination |
| Random event ID | No useful account grouping | Even distribution cannot preserve account order |
| Dedicated hot-account route | Explicit isolated account stream | Limits collateral impact but adds routing and migration work |
Create a synthetic workload with many ordinary accounts and one account producing a large share of records. Keep the exact distribution in the fixture rather than assuming uniform keys. Measure per-partition lag and per-account completion age. Overall consumer throughput can look healthy while the hot account and unrelated accounts sharing its partition fall behind.
If events for separate orders are genuinely independent, a narrower key can help. But an account suspension may affect every order, so the design now needs a way to coordinate that account-wide state with order-local streams. A materialized account revision check or another explicit protocol may be appropriate. Randomly salting the account key without that protocol merely distributes the race.
Preserve completion order inside the consumer
A consumer can fetch records in partition order and then destroy that order by dispatching them to an unconstrained thread pool. Revision eighteen might complete before revision seventeen, and an offset commit can acknowledge work that has not safely finished. Keep ordered processing within the required scope or maintain a contiguous completed-offset frontier with carefully bounded concurrency.
For the credit example, conditional state updates provide another defense. A duplicate event should not repeat the business action, and a stale revision should not overwrite a newer projection. An unexpected future revision can be held for reconciliation. These rules belong in durable consumer state if they must survive restarts and partition reassignment.
External effects require additional care. Updating a local projection in sequence does not guarantee that remote fulfillment requests finish in sequence. If suspension must precede any new fulfillment, the consumer needs an explicit check or serialized effect boundary. A timed-out remote call remains uncertain even if its Kafka record has a well-defined offset.
Exercise retries and reassignment deliberately
Verify producer settings against the deployed Kafka client version. Idempotent production and its compatible retry settings help preserve broker-side behavior, but they do not deduplicate arbitrary external effects or repair incorrect business sequencing from multiple writers. Keep producer identity, event identity, and account revision conceptually separate.
Stop a consumer after it commits a business change but before committing its offset. On restart, the replay should be harmless. Stop another consumer while an external call is unresolved, then trigger partition reassignment. Confirm that the new owner does not assume the old call disappeared. Leases and ownership transitions require a recovery rule for work already outside the process.
A poison record creates an ordering choice. Skipping it into a dead-letter path may allow later revisions to execute without their prerequisite. Pause the affected scope, reconcile the missing transition, or define a domain-specific skip rule. A generic “continue on error” policy cannot determine whether a suspended account is safe to fulfill.
Expose the oldest uncompleted offset alongside the highest fetched offset. A consumer that fetches rapidly while one earlier operation remains unresolved can accumulate a large hidden working set. Bound that in-flight distance and pause acquisition when necessary, so preserving a contiguous commit frontier does not require retaining unlimited records.
Treat partition expansion as a routing migration
With partitioners that map a key using the partition count, increasing that count can move future records for an existing account while older records remain in the previous partition. Two consumers may then process that account concurrently. Validate the actual partitioner and design a cutover, drain, routing version, or consumer sequence protocol before expansion.
Test the proposed change with retained records and active publication, not only an empty topic. The messaging decision guide establishes Kafka's role, and dead-letter operations develops the failure decisions. Event-processing implementation should make the partition contract visible in schemas, consumers, and operational runbooks.