zero gravitySOLUTIONS
← All insightsCLOUD & OPERATIONS / OPERATING GUIDE

Choosing timeouts as a system rather than per service.

QUICK ANSWER

Start with the customer deadline and reserve time for every required stage. Bound attempts and retries inside that envelope, and keep asynchronous job deadlines, message visibility, and process termination as separate timers.

Inside this article 5 sections

Choose distributed-system timeouts from one end-to-end deadline, then allocate the remaining time to dependencies, retries, and response cleanup. Independent per-service timeout values can let work continue after the caller has given up or leave no time for a meaningful fallback.

The design needs two related limits: how long the customer operation may remain unresolved, and how long each attempt may occupy scarce resources. A timeout bounds waiting; it does not prove that a remote write failed or was cancelled.

Start with the customer deadline

Consider an illustrative price-confirmation endpoint with a two-second application deadline. It authenticates the caller, reads account rules, requests a partner quote, and saves the selected result. The application must reserve time to return a response even when the partner is slow.

Allocate a planning budget of 150 milliseconds for authentication, 250 for account data, 1,200 for the partner-call envelope, 150 for the final save, and 250 for assembly and uncertainty. These allocations total 2,000 milliseconds. They are illustrative design inputs, not measured latency or a recommendation for every API.

The partner envelope includes connection establishment, request execution, retry delay, and any second attempt. Two attempts each allowed 1,200 milliseconds do not fit a 1,200-millisecond envelope. Keep the distinction visible in code and configuration.

Original timeout tree · illustrative milliseconds
Application deadline: 2,000 ms
Before partner400 ms planned

Authentication 150 + account data 250.

Partner envelope1,200 ms total

Attempt one 500 + retry wait 100 + attempt two 500 + envelope reserve 100.

After partner400 ms planned

Save 150 + response and uncertainty reserve 250.

The second attempt is conditional on remaining time and retry safety. The diagram shows a budget, not a promise that every stage always consumes its allocation.

Pass remaining time through the call chain

Use a monotonic clock to measure elapsed time inside a process. For remote calls, propagate a deadline or a remaining-duration value through a defined protocol, clamp untrusted inputs, and account for the protocol's clock assumptions. A timestamp received from a client should not grant unlimited execution time.

Before each attempt, calculate what remains after reserving time for required later work. The following pure Python function illustrates the local decision. It uses millisecond inputs and does not perform network calls or cancellation.

def attempt_budget_ms(deadline_ms, now_ms, reserve_ms,
                      attempt_cap_ms, minimum_ms):
    remaining = deadline_ms - now_ms - reserve_ms
    if remaining < minimum_ms:
        return None
    return min(attempt_cap_ms, remaining)

# Deadline at 2,000 ms; 900 elapsed; reserve 400 for later work.
assert attempt_budget_ms(2000, 900, 400, 500, 100) == 500
assert attempt_budget_ms(2000, 1550, 400, 500, 100) is None

The minimum represents whether an attempt has a plausible chance to finish usefully. Do not start a new expensive operation simply because one millisecond remains. For a retry, subtract the planned backoff before deciding whether another attempt is eligible.

Choose one layer to own retries

Inspect retries in the browser, application, HTTP client, AWS SDK, and worker. If three layers each make up to three attempts, a single logical call can produce as many as 27 attempts at the deepest dependency under the relevant failure pattern. That amplification consumes the very capacity needed for recovery.

Assign retry responsibility where enough context exists to judge safety and remaining budget. Disable or constrain overlapping policies. Include SDK connection and read behavior in the experiment, because a wrapper timeout may not cover every stage in the same way across libraries.

Use backoff and jitter for retryable failures, but do not confuse delay with safety. A repeated non-idempotent write can duplicate an action after a timeout. Use an operation key or query the original result where supported. The retry storm guide covers fleet-level amplification beyond one request.

Separate request deadlines from job lifetimes

An asynchronous operation should have a short acceptance deadline and a separate completion objective. Once a job is durably accepted, the API can return its identifier. The worker then uses a job deadline and per-dependency budgets appropriate to that work, rather than inheriting the nearly expired HTTP timeout.

On AWS, API Gateway or a load balancer has its own connection and integration behavior. A Lambda function has an execution timeout, while an ECS worker has process and shutdown limits. SQS visibility controls when a message may become available again; it is not the job's business deadline and does not cancel the running worker.

Document which timer owns each transition. If visibility expires while a worker continues, another delivery may overlap it. If the customer deadline expires while a partner later commits, the application needs a reconciliation path. Timeouts should produce understandable states rather than a generic “failed” flag for every uncertainty.

Include cold-path behavior in the test. Connection reuse can hide DNS lookup, TLS negotiation, and pool creation costs that appear during deployment or scale-out. A timeout selected only from warm requests may reject healthy new instances just as capacity is being added.

When calls run in parallel, the wall-clock path follows the slowest required branch rather than the sum of every branch. Capacity consumption still includes all branches, including abandoned work. Record which branches are mandatory and cancel optional work when its result can no longer affect the response.

Test the boundaries of the budget

Inject delays in connection establishment, response headers, response body, and the final database write. Verify that the intended timer covers each stage and that resources are released. Test a dependency that ignores cancellation to see whether the application still limits concurrent abandoned work.

Include a slow first attempt followed by a healthy second attempt. The retry should run only if it fits the remaining envelope and the operation is safe to repeat. Also test a late success after the caller receives a timeout; the result lookup should resolve the operation without asking the customer to guess whether to submit again.

Record observed timings and remaining budgets in traces, avoiding high-cardinality labels on aggregate metrics. The dependency capacity worksheet connects long occupancy to saturation. Architecture design can align caller expectations, retry ownership, and resource limits into one coherent deadline policy.

FROM INSIGHT TO ACTIONSoftware architecture consulting

Give every timeout a clear system-level purpose.

We can design deadline propagation, bounded retries, and interruption tests that connect customer expectations to dependency capacity.

Explore architecture consultingDiscuss your architecture

Start with your goals.
We’ll work through the next step together.