How to Design a Retry Ladder That Respects Customer Endpoints

The retry loop is where webhook delivery either works or does not. Everything upstream can be perfect: durable event log, HMAC signing, per-customer isolation, and if the retry ladder is wrong, the whole system amplifies outages instead of absorbing them. This is how to design a ladder that behaves.

What is a retry ladder and why does it fail?

A retry ladder is the sequence of delays between attempts when a webhook delivery fails. Simple in concept: fail, wait a bit, retry, wait longer, retry again, until you succeed or give up.

Simple ladders fail in three predictable ways.

• Retry storms. All the events that failed at the same second retry at the same next second, hammering the recovering endpoint.
• Endpoint exhaustion. A slow endpoint at three attempts per second gets thirty attempts per second during a retry surge because your queue is not throttled per endpoint.
• Ignored Retry-After. The endpoint tells you "come back in 60 seconds" and you come back in 10 because your ladder is fixed.

Each of these turns a small outage into a bigger one.

What is the right base backoff curve?

Exponential, capped, jittered. The specific curve matters less than committing to one and applying it consistently.

A workable curve for eight attempts over roughly thirteen hours:

The reason it starts at 30 seconds instead of 1 second: an endpoint that just returned 503 is not likely to be ready one second later, but it is highly likely to be flooded by retries if you send them immediately. Giving the endpoint 30 seconds to recover is polite and materially improves success rate on attempt two.

How does jitter actually help?

Consider 1,000 events that all failed on their first attempt because your consumer's database went down for 45 seconds. Without jitter, all 1,000 retry attempts fire at exactly attempt-time-plus-30-seconds. That is a 1,000 request-per-second spike into a database that just came back up. You crash it again.

With 20% jitter, those 1,000 retries spread across a 12 second window (24 to 36 seconds). The spike becomes 83 requests per second, which most databases handle without noticing.

Jitter is not decoration. It is the mechanism that keeps a small outage from becoming a two-outage event.

When should you honor Retry-After?

Always, with a cap. Retry-After is HTTP's built-in way for a receiver to say "I know when I will be ready." Ignoring it is rude and costs you success rate.

• Retry-After under 4 hours. Honor exactly. Skip whatever your ladder would have scheduled and use the endpoint's suggested delay.
• Retry-After between 4 and 24 hours. Cap at 4 hours. The endpoint is either misconfigured or intentionally hostile.
• Retry-After above 24 hours. Ignore, fall back to your ladder. Nobody legitimately means "come back tomorrow."
• Retry-After as a date (HTTP-date format). Compute the delta, then apply the same caps.

Track how often you honor Retry-After. Endpoints that regularly return it in the 60 to 300 second range are giving you a signal you can build on: they are backpressure-aware, they scale, and their operators know what they are doing.

How do you implement per-endpoint concurrency isolation?

Each endpoint gets its own token bucket or semaphore. The bucket size is the concurrency limit for that endpoint, tunable per customer.

• Default cap. 100 concurrent in-flight requests per endpoint.
• Slow endpoint override. If p99 latency to an endpoint exceeds 3 seconds, halve the cap automatically.
• Enterprise override. Customers on higher tiers can request up to 500 concurrent.
• Never zero. Even a degraded endpoint gets at least 1 concurrent token, so probes can go through.

The isolation happens before the retry ladder consults its schedule. If the endpoint is at its concurrency cap, the retry queues in-endpoint rather than firing. This is what prevents one slow customer from stalling the whole system.

What triggers the circuit breaker?

Two triggers, whichever hits first.

• Rate trigger. Five consecutive failed attempts to the same endpoint within a 60 second window.
• Volume trigger. Twenty failed attempts within a five minute window.

When tripped, the endpoint enters "open" state. In open state:

1. No new deliveries are attempted to that endpoint.
2. Retries in the ladder pause instead of firing.
3. A single probe is scheduled for 5 minutes later.

If the probe succeeds, the endpoint enters "half-open" state, where a small trickle of deliveries goes through. If those succeed, the endpoint returns to "closed" (normal). If any fail, it re-enters open with an extended backoff.

The customer-facing dashboard must show the endpoint's circuit state. A "your endpoint is currently in circuit-open due to repeated failures" banner does more for a customer's debugging than any log.

How do you drain the dead-letter queue?

Never automatically. Always manually or scheduled.

Events that exhaust the retry ladder end in dead-letter. They stay there. The customer can replay them from their dashboard with one click, or you can schedule a bulk replay after a known outage recovers. The retention window is the same as your normal event retention (typically 30 days on paid plans).

Two rules for dead-letter management:

• Never merge into normal retry flow automatically. Dead-lettered events represent a decision to give up. Auto-retrying them undoes that decision.
• Preserve original event id. When replayed, the new attempt uses the original id so consumer deduplication continues to work.

What about idempotency on the receiver side?

Design retries around the assumption that consumers dedupe on event id. Every event you send carries a unique id in the payload or header. When the same id arrives twice at a receiver, they drop the second.

Most modern consumers implement this. For consumers who do not, the retry ladder amplifies bugs on their side. That is unavoidable and the fix is to their receiver, not your ladder. The best you can do is make your event id prominent in the payload and document deduplication in your integration guide.

What actually matters

The mistake to avoid is treating the retry ladder as a set of magic numbers you copy from a blog post. The specific numbers matter less than the shape: exponential with jitter, capped total window, honored Retry-After, per-endpoint isolation, circuit breaker on sustained failure. Get the shape right, tune the numbers with data from your own consumer fleet, and the retry engine becomes invisible. The single biggest improvement most teams can make in one afternoon is adding jitter to a ladder that does not have it. That one change reduces retry-driven amplification of outages more than any other tweak.