zero gravitySOLUTIONS
← All insightsARCHITECTURE / ARCHITECTURE DECISION GUIDE

Turning incident patterns into architecture priorities.

QUICK ANSWER

Group incidents by the mechanisms that amplify customer impact. Choose interventions supported by evidence, define a reproducible failure exercise, and separate immediate repairs from longer-term architecture investments.

Inside this article 6 sections

Use incident patterns to prioritize architecture changes by grouping the mechanisms that turned faults into customer impact. Compare repeated propagation, detection, and recovery problems across incidents. Then choose an intervention that changes those mechanisms and define how its effect will be tested.

An incident-driven architecture review is different from collecting every postmortem action into a larger backlog. Several incidents may share a dependency but need different remedies; incidents in different services may share the same missing boundary.

Normalize the evidence before ranking it

Begin with a bounded review period and an incident set that includes relevant near misses. Record the customer workflow, trigger, propagation path, detection delay, recovery dependencies, and unresolved uncertainty. Keep observed facts separate from hypotheses formed during the response.

Do not compare incident severity labels without checking how teams assign them. One team may classify a single-tenant outage as high severity while another reserves that label for global impact. Reconstruct the actual consequence: affected accounts, unavailable actions, delayed work, or incorrect results. Use counts only when the evidence supports them.

A missing postmortem is also evidence about the review process, but it is not permission to invent a root cause. Create an “unknown mechanism” group and identify what logs, traces, or operational records would resolve it. Architecture priorities should expose uncertainty instead of hiding it behind a confident ranking.

Original incident clustering example · synthetic cases
IncidentImmediate triggerAmplifying mechanismCandidate intervention
A · exports stallPartner latency increasesUnbounded concurrent calls exhaust workersDependency admission control
B · imports stallRetry burst after network recoverySame shared worker pool becomes saturatedWorkload isolation and retry budget
C · reminders repeatWorker stops after sendingEffect has no durable deduplication recordMessage-intent ledger
D · reports remain staleCredentials rotateOnly process uptime is monitoredFreshness signal and credential-refresh test
Cases A and B share a capacity boundary problem despite different triggers. Cases C and D need different changes even if the same worker service runs them.

Separate trigger, propagation, and recovery

The trigger explains what started the incident: a deployment, a dependency slowdown, or an expired credential. Propagation explains why the disturbance spread: shared pools, synchronous fan-out, uncontrolled retries, or a missing isolation boundary. Recovery explains why normal service took time to return: manual reconciliation, inaccessible credentials, or a large undrained backlog.

A fix aimed only at the trigger can leave the system vulnerable to the next variation. Replacing a failed provider may not solve the unbounded call pattern that would overload its replacement. Conversely, a broad architectural redesign is unnecessary when the evidence supports a local configuration defect with a clear regression test.

Write a causal claim for each candidate investment. For the synthetic export and import cases, the claim might be that bounding dependency concurrency and separating interactive work prevents one delayed workload from occupying all workers. That claim is specific enough to test and narrow enough to challenge.

Prioritize by consequence and confidence

Consider impact, recurrence, detectability, recovery burden, and confidence in the mechanism. Avoid multiplying arbitrary scores into a supposedly precise risk number. A simple ordered discussion is often more useful: which failure can create irreversible harm, which repeatedly interrupts core workflows, and which has a clear intervention supported by evidence?

Mark the evidence level. A repeated mechanism demonstrated in traces supports a stronger commitment than a plausible explanation from an incomplete timeline. For the uncertain case, the first investment may be an instrumented reproduction or a small telemetry change rather than a service extraction.

Estimate the ongoing operational cost of the remedy. A separate queue and worker pool can improve isolation, but it also adds dashboards, deployment configuration, and an owner. A proposal that reduces one incident class while creating an unowned subsystem is not complete architecture work.

Turn one cluster into a testable change

Choose a representative failure and define a before-and-after exercise. Inject partner latency into a controlled environment while both export and import workloads run. Observe completion by workload, concurrency at the dependency, rejection or deferral behavior, and recovery after latency returns to normal.

Set acceptance criteria before implementing the fix. The protected workflow should continue meeting its agreed objective under the chosen disturbance. Deferred work should remain durable and eventually recover without duplicate effects. The test should include an overload beyond the admission limit so the system's refusal behavior is visible.

Record the workload and environment limitations with the result. A small synthetic fixture can establish that the boundary behaves correctly, but it does not establish production capacity. Link it to the capacity planning experiment when load assumptions matter.

Keep immediate actions and architecture work connected

An incident may require an urgent repair, a near-term guardrail, and a longer-term ownership change. Keep those connected through the causal claim. For example, lowering worker concurrency may stabilize the service today, while separate resource pools and account fairness provide the durable boundary.

Assign a responsible owner, completion evidence, and a review date to each intervention. “Improve observability” is not a finishable task. “Detect stale report data before the freshness deadline and test missing telemetry” describes a result an engineer can demonstrate.

Review whether earlier fixes changed later incidents. A recurring pattern may mean the remedy was incomplete, the test was unrepresentative, or the change was never fully deployed. Do not treat repeated incidents as evidence of individual carelessness when the system still permits the same failure mechanism.

Produce a short investment decision

The review output should name the leading incident clusters, the evidence supporting each, the selected intervention, and what remains uncertain. Include decisions to leave a system unchanged where a redesign would not address the observed mechanism. This keeps the review useful for product planning rather than becoming an inventory of architectural preferences.

The architecture review before a rewrite provides the broader assessment frame. Technical consulting can turn incident evidence into a prioritized set of experiments and changes, with explicit reasons for what the team should tackle first.

FROM INSIGHT TO ACTIONSoftware development consulting

Turn incident evidence into focused architecture investment.

We can cluster failure patterns, challenge uncertain explanations, and define experiments that show whether a proposed change addresses the problem.

Explore technical consultingDiscuss your challenge

Start with your goals.
We’ll work through the next step together.