Turning incident patterns into architecture priorities.
Group incidents by the mechanisms that amplify customer impact. Choose interventions supported by evidence, define a reproducible failure exercise, and separate immediate repairs from longer-term architecture investments.
Inside this article 6 sections +
Use incident patterns to prioritize architecture changes by grouping the mechanisms that turned faults into customer impact. Compare repeated propagation, detection, and recovery problems across incidents. Then choose an intervention that changes those mechanisms and define how its effect will be tested.
An incident-driven architecture review is different from collecting every postmortem action into a larger backlog. Several incidents may share a dependency but need different remedies; incidents in different services may share the same missing boundary.
Normalize the evidence before ranking it
Begin with a bounded review period and an incident set that includes relevant near misses. Record the customer workflow, trigger, propagation path, detection delay, recovery dependencies, and unresolved uncertainty. Keep observed facts separate from hypotheses formed during the response.
Do not compare incident severity labels without checking how teams assign them. One team may classify a single-tenant outage as high severity while another reserves that label for global impact. Reconstruct the actual consequence: affected accounts, unavailable actions, delayed work, or incorrect results. Use counts only when the evidence supports them.
A missing postmortem is also evidence about the review process, but it is not permission to invent a root cause. Create an “unknown mechanism” group and identify what logs, traces, or operational records would resolve it. Architecture priorities should expose uncertainty instead of hiding it behind a confident ranking.
| Incident | Immediate trigger | Amplifying mechanism | Candidate intervention |
|---|---|---|---|
| A · exports stall | Partner latency increases | Unbounded concurrent calls exhaust workers | Dependency admission control |
| B · imports stall | Retry burst after network recovery | Same shared worker pool becomes saturated | Workload isolation and retry budget |
| C · reminders repeat | Worker stops after sending | Effect has no durable deduplication record | Message-intent ledger |
| D · reports remain stale | Credentials rotate | Only process uptime is monitored | Freshness signal and credential-refresh test |
Separate trigger, propagation, and recovery
The trigger explains what started the incident: a deployment, a dependency slowdown, or an expired credential. Propagation explains why the disturbance spread: shared pools, synchronous fan-out, uncontrolled retries, or a missing isolation boundary. Recovery explains why normal service took time to return: manual reconciliation, inaccessible credentials, or a large undrained backlog.
A fix aimed only at the trigger can leave the system vulnerable to the next variation. Replacing a failed provider may not solve the unbounded call pattern that would overload its replacement. Conversely, a broad architectural redesign is unnecessary when the evidence supports a local configuration defect with a clear regression test.
Write a causal claim for each candidate investment. For the synthetic export and import cases, the claim might be that bounding dependency concurrency and separating interactive work prevents one delayed workload from occupying all workers. That claim is specific enough to test and narrow enough to challenge.
Prioritize by consequence and confidence
Consider impact, recurrence, detectability, recovery burden, and confidence in the mechanism. Avoid multiplying arbitrary scores into a supposedly precise risk number. A simple ordered discussion is often more useful: which failure can create irreversible harm, which repeatedly interrupts core workflows, and which has a clear intervention supported by evidence?
Mark the evidence level. A repeated mechanism demonstrated in traces supports a stronger commitment than a plausible explanation from an incomplete timeline. For the uncertain case, the first investment may be an instrumented reproduction or a small telemetry change rather than a service extraction.
Estimate the ongoing operational cost of the remedy. A separate queue and worker pool can improve isolation, but it also adds dashboards, deployment configuration, and an owner. A proposal that reduces one incident class while creating an unowned subsystem is not complete architecture work.
Turn one cluster into a testable change
Choose a representative failure and define a before-and-after exercise. Inject partner latency into a controlled environment while both export and import workloads run. Observe completion by workload, concurrency at the dependency, rejection or deferral behavior, and recovery after latency returns to normal.
Set acceptance criteria before implementing the fix. The protected workflow should continue meeting its agreed objective under the chosen disturbance. Deferred work should remain durable and eventually recover without duplicate effects. The test should include an overload beyond the admission limit so the system's refusal behavior is visible.
Record the workload and environment limitations with the result. A small synthetic fixture can establish that the boundary behaves correctly, but it does not establish production capacity. Link it to the capacity planning experiment when load assumptions matter.
Keep immediate actions and architecture work connected
An incident may require an urgent repair, a near-term guardrail, and a longer-term ownership change. Keep those connected through the causal claim. For example, lowering worker concurrency may stabilize the service today, while separate resource pools and account fairness provide the durable boundary.
Assign a responsible owner, completion evidence, and a review date to each intervention. “Improve observability” is not a finishable task. “Detect stale report data before the freshness deadline and test missing telemetry” describes a result an engineer can demonstrate.
Review whether earlier fixes changed later incidents. A recurring pattern may mean the remedy was incomplete, the test was unrepresentative, or the change was never fully deployed. Do not treat repeated incidents as evidence of individual carelessness when the system still permits the same failure mechanism.
Produce a short investment decision
The review output should name the leading incident clusters, the evidence supporting each, the selected intervention, and what remains uncertain. Include decisions to leave a system unchanged where a redesign would not address the observed mechanism. This keeps the review useful for product planning rather than becoming an inventory of architectural preferences.
The architecture review before a rewrite provides the broader assessment frame. Technical consulting can turn incident evidence into a prioritized set of experiments and changes, with explicit reasons for what the team should tackle first.