Reliable systems are not systems where nothing goes wrong. They are systems where failure has a shape that people and software can understand.
Every dependency needs a deadline
An outbound request without a timeout is an unfinished decision. The caller cannot know whether to wait, retry, or offer a fallback.
const response = await fetch(endpoint, {
signal: AbortSignal.timeout(3_000),
});The right timeout depends on the operation, but an explicit timeout is always more useful than an accidental one imposed by a load balancer or browser.
Retries are not free
Retries help with brief network blips, but they can amplify an outage when every caller retries at the same time.
| Strategy | Good for | Main risk |
|---|---|---|
| Fixed delay | Simple internal jobs | Synchronized retry storms |
| Exponential backoff | Transient service failures | Slow recovery if uncapped |
| Jittered backoff | Many concurrent clients | More complex debugging |
| No retry | Non-idempotent operations | More visible transient errors |
For a safe retry, ask whether the operation is idempotent and whether the server can recognize a duplicate request.
Partial success is still a state
A dashboard that loaded eight of ten widgets should not pretend it loaded ten. Show the successful data, identify the unavailable pieces, and provide a specific recovery action:
Refresh billing dataReconnect to searchView cached results from 10:41
This is better than replacing the entire screen with a generic error.
A failure review checklist
- Is the timeout visible in configuration?
- Are retries bounded?
- Do logs distinguish timeout from cancellation?
- Can the user keep useful partial results?
- Is there a runbook for the alert?
Failure handling is product design with a stopwatch attached.