Testing every outbox crash boundary
An outbox test should prove that committed business changes retain publishable intent and that repeated publication does not repeat the business effect. Inject crashes at database commit, broker acknowledgement, dispatch marking, and consumer commit boundaries. Treat duplicates as an expected recovery case, while testing ordering and retention separately.
Inside this article 5 sections +
The most revealing outbox failure happens after the broker accepts a message but before the dispatcher records that acceptance. On restart, the dispatcher sees an unsent row and publishes again. That behavior can be correct: the outbox preserves delivery intent, while the consumer must recognize repeated delivery.
A test suite that asserts every outbox row becomes sent misses this boundary. The useful assertions concern business state, durable intent, broker-visible events, and final consumer effects. Build the suite around those four views, then interrupt execution where they can diverge.
Write the invariants before the failure hooks
For an illustrative order-confirmation operation, every committed confirmation must have a corresponding durable outbox event. A rolled-back confirmation must not produce a committed event intent. After infrastructure recovers and eligible work is processed, every required event must be delivered or remain visibly blocked with an owner.
On the consumer side, repeated delivery of the same event must not repeat the intended business effect. This is distinct from requiring exactly one broker record, which the ordinary outbox relay does not promise. Give each event a stable identifier created with the business transaction and retain that identifier across dispatcher attempts.
Illustrative minimal outbox fields:
event_id stable unique event identity
tenant_id business ownership scope
aggregate_id order or account whose sequence matters
aggregate_version committed business revision
payload_version event contract version
payload immutable event data
created_at durable intent time
published_at nullable dispatch acknowledgement marker
Optional claim fields require an explicit lease/recovery policy.
Do not reconstruct the payload from mutable current state each time the relay retries unless that is the declared event contract. Otherwise, the same event identity can carry different data across attempts. Persist the immutable event body or the exact immutable revision needed to reproduce it.
Interrupt at durable boundaries
| Interruption point | Expected durable evidence | Recovery assertion |
|---|---|---|
| Before business transaction commit | No committed confirmation or event | Retry creates one logical confirmation |
| After commit, before relay reads | Confirmation and pending event exist | Relay eventually publishes the event |
| After claim, before publish | Event remains recoverable under claim policy | Another attempt can recover abandoned ownership |
| After broker accepts, before sent marker | Broker may contain event; row still pending | Duplicate publication causes one consumer effect |
| After consumer commit, before acknowledgement | Effect and deduplication record exist | Redelivery is harmless |
| During cleanup | Required recovery evidence remains | No eligible intent disappears prematurely |
Implement deterministic test hooks immediately before and after the relevant operations. A hook should pause execution so the harness can terminate the process or sever a connection at the intended point. Random process killing is useful later, but it can repeatedly miss the narrow acknowledgement gap and leave a false sense of coverage.
Use real database transactions and a broker with the relevant delivery behavior in the integration environment. In-memory mocks can verify branching but cannot reproduce durability, redelivery, connection loss, or transaction rollback semantics. Keep fixture accounts and event identities isolated so each test can inspect all resulting records.
Test concurrency and expired ownership
A dispatcher may claim rows with database locks, leases, or another coordination mechanism. With PostgreSQL, FOR UPDATE SKIP LOCKED can let competing workers select different available rows, but skipped rows are deliberately absent from that worker's view. It is not a general guarantee of aggregate sequence.
Hold one claim while another worker runs. Then expire or abandon the first claim according to the implementation. Verify that recovery occurs without losing the event. Also let the original worker continue after its lease expires: a new worker may already own the row, so stale updates must not overwrite a newer claim's state.
Use a claim generation or token for conditional acknowledgement where appropriate. That protects local ownership transitions, but it cannot retract a message an old worker already published. Consumer idempotency remains necessary even if the dispatcher's lease protocol is carefully fenced.
Assert the business effect and deduplication together
For a database-owned consumer effect, record event consumption and apply the mutation in the same transaction. A crash before commit should leave neither; a crash after commit should leave both. If deduplication commits first and the effect fails afterward, redelivery can be skipped permanently. If the effect commits first without deduplication, it can be repeated.
External effects need their own operation identity and reconciliation tests. A consumed Kafka or SQS message does not make a remote invoice send atomic with local storage. Include a provider stub that accepts an action and then loses the response so the test can verify the consumer's uncertain-outcome branch.
Compare final business state with an uninterrupted reference run. Count logical confirmations, invoices, or notifications, not just queue acknowledgements. Preserve the delivered event identifiers so unexpected duplicates can be explained without treating every repeated delivery as a correctness failure.
Verify the meaning of the broker acknowledgement used by the relay. A successful local send call may mean only that a client buffer accepted the record. The sent marker should follow the configured acknowledgement that establishes the required broker acceptance, and the test harness should distinguish a buffered record from one observable after producer termination.
Add sequence and retention tests separately
If order matters, create two revisions for the same aggregate and delay the first relay attempt. Verify the second cannot produce an invalid final consumer state. A sorted query alone may not be enough once several dispatchers publish concurrently. The design may require per-aggregate serialization or consumer revision checks.
Run cleanup while an event is claimed, retryable, or awaiting reconciliation. The retention policy must not delete the only evidence needed for recovery. Test restoration from the supported database backup process as well: restored outbox rows can be republished, so the consumer's duplicate horizon must match the recovery assumptions.
The reliable jobs guide explains the operation model, and API idempotency establishes repeated-intent handling. Outbox implementation should finish with a reproducible failure matrix and durable evidence for each invariant.