Capacity planning around the slowest dependency.
Plan capacity around the constrained dependency and useful completed work. Translate demand into occupancy, account for retries and workload mix, then test latency, rejection, backlog, and recovery before choosing scaling or admission controls.
Inside this article 5 sections +
Plan software capacity around the dependency that limits useful completed work. More application instances will not increase throughput when database connections, a partner API, or a serialized operation already controls the rate. It can instead increase contention and make recovery slower.
Begin with a workload definition, trace how each request consumes constrained resources, and test the system near its observed limits. Treat simple calculations as hypotheses for the load test, not as a production capacity guarantee.
Translate demand into resource occupancy
Imagine an illustrative service that processes document requests. Each request occupies one external-service slot for an average of 0.4 seconds, and the service has 40 usable concurrent slots. Ignoring all other overhead, the arithmetic ceiling is 40 ÷ 0.4 = 100 requests per second.
That ceiling assumes stable service time, no retries, and continuous useful utilization. It does not include tail latency or safety margin. At 60 requests per second and the same average slot occupancy, expected in-flight work is 60 × 0.4 = 24 slots under steady-state assumptions. A spike in dependency latency can consume the remaining slots without any increase in incoming traffic.
40 ÷ 0.4 = 100 requests/second arithmetic ceiling.
60 × 0.4 = 24 occupied slots on average.
60 × 0.8 = 48 slots demanded; more than the 40 available.
Trace how one customer action multiplies work
A request may perform several queries, call multiple providers, and schedule background work. Count visits to each dependency, not only HTTP requests entering the application. If one document request makes three sequential partner calls, its contribution to partner demand is three calls unless caching or batching changes that relationship.
Retries increase the multiplier during failure. If the browser, API client, worker, and service library each retry independently, the dependency can receive much more work than the original user volume suggests. Define retry ownership, a total deadline, and a bounded attempt policy.
Separate workload classes. A metadata lookup and a large export should not share one average service time. Use representative proportions, payload sizes, and account concurrency. Keep long-running work from exhausting the resources needed for interactive requests.
Design a controlled load test around completed work
Measure offered load, admitted load, completed useful operations, rejected requests, latency distributions, and resource occupancy. Recording only successful response latency can make overload look harmless because timed-out or rejected requests disappear from the chart. Include them in the result.
Use a load generator capable of maintaining the intended arrival pattern. A fixed number of clients that waits for each response may send less traffic as the service slows, masking the queue growth expected from real independent arrivals. Report how the generator behaves so readers can interpret the experiment.
Increase demand in controlled stages with explicit stop conditions. Include a period at the target workload, a burst, a dependency slowdown, and recovery after the burst ends. Check that the test environment's database size, cache behavior, and provider configuration are representative enough for the decision being made.
Protect the constrained resource before scaling callers
Bound concurrency near the dependency, set deadlines that prevent abandoned work from running indefinitely, and reject or defer excess work deliberately. A queue can absorb a temporary burst, but an unbounded queue converts a capacity problem into growing latency and storage consumption.
For asynchronous work, estimate whether the backlog can drain while normal traffic continues. If arrivals average 60 jobs per second and workers complete 75, the spare drain rate is 15 jobs per second. An illustrative backlog of 9,000 jobs would take 600 seconds to clear under those stable assumptions. If completion falls to 60, the backlog does not shrink at all.
Define fairness between customers so one large account cannot consume every slot. Choose whether to reserve capacity, apply per-account admission limits, or isolate heavy jobs. The right policy follows the product promise and the cost of unused reserved capacity.
Repeat the decisive experiment after changing connection pools, retry policy, or a major dependency. These settings can move the bottleneck even when application code remains unchanged. Keep the workload fixture and configuration with the results so the comparison remains meaningful.
Turn observations into a capacity envelope
Report the tested workload mix, resource limits, dependency behavior, and the load at which the service stops meeting its objective. Keep a margin appropriate to burstiness and failure behavior. A single maximum requests-per-second number hides too many conditions to guide an operating decision.
The reactive Java experiment guide distinguishes concurrency from useful throughput, while Kafka versus job queues clarifies buffering choices. Architecture design can connect the measured envelope to admission control, scaling rules, and a workload growth plan.