How to Build a Webhook Delivery System That Survives Real Customers
Webhooks look simple. Client calls your API, you post JSON to their URL, done. Then the endpoint goes down at 3am, retries pile up, the customer opens a ticket, and someone on your team spends the afternoon grepping logs. Multiply by a hundred customers and webhooks becomes the second-most expensive undocumented product line at your company.
A webhook delivery system that survives real customers has eight components. Skipping any one of them shows up as tickets, on-call pages, or lost revenue within six months.
What is a webhook delivery system and why is it hard?
A webhook delivery system is the infrastructure that fans an event out from your application to every customer endpoint subscribed to it, retries when the endpoint fails, signs each request so the receiver can verify authenticity, and keeps a searchable record of every attempt so you can prove what happened.
The hard part is not the send. It is everything around the send. Customer endpoints go down. Slow endpoints back up your queue. Signature verification breaks on rotation. Payloads change shape between versions and consumers on the old shape silently drop events. A working system has to solve all of that without paging your on-call every time a customer's server sneezes.
What are the eight components you actually need?
The minimum viable set is eight distinct pieces. Not seven.
| Component | What it does | What breaks without it |
|---|---|---|
| Durable event log | Persists every event before delivery | Events lost on worker restart |
| Per-endpoint queue | Isolates delivery per subscriber | One slow customer stalls all delivery |
| Retry ladder | Exponential backoff with jitter | Thundering-herd retries on outage |
| HMAC signing | Signs each request, rotates secrets | Customers cannot verify, or rotation breaks integrations |
| Rate limiting | Caps per-endpoint request rate | You DDoS your own customers |
| Circuit breaker | Stops sending to dead endpoints | Wasted retries and support noise |
| Attempt log with payload search | Records request, response, timing | Support cannot answer "did you send it" |
| Customer debug dashboard | Lets subscribers see their own attempts | Every debug question routes through your team |
The component most teams skip is the customer-facing dashboard. It is also the one that eats the most support hours.
How should you design the retry ladder?
Eight attempts, exponential backoff with jitter, capped at roughly thirteen hours total. The specific curve matters less than committing to one.
A workable ladder: 30 seconds, 2 minutes, 10 minutes, 30 minutes, 1 hour, 2 hours, 4 hours, 5 hours. Add random jitter of plus or minus 20% on each interval so retries from many events do not stack on the same wall-clock second.
Two rules that people learn the hard way:
- Never retry on 4xx. A 400 or 404 is not going to become a 200 later. Retrying them wastes your capacity and pollutes the endpoint's dashboard with garbage.
- Always retry on 5xx and network errors. Even 501 Not Implemented, because that is often a misconfigured proxy, not a permanent state.
Cap the total window at 24 hours. Past that, the consumer needs replay, not retry.
How do you sign webhooks without breaking rotation?
HMAC-SHA256 with a shared secret per endpoint, sent in a header. The header should include the timestamp of the send, so consumers can reject replays outside a small window.
Rotation is where most implementations fail. The correct pattern is a rolling window with two active secrets.
- Generate a new secret and expose it via API and UI.
- Sign every outbound request with both the old and new secret, sent as two headers.
- Give consumers 30 days to switch verification to the new secret.
- After 30 days, drop the old secret and stop sending its header.
That is the only rotation pattern that never breaks a customer during a rotation. Single-secret cutover always breaks somebody.
What does per-endpoint rate limiting protect against?
Two things: your customers, and yourself.
Your customers ship with a rate limit on their webhook receiver. If you fan out an event to a webhook that gets a burst of 500 events in one second, you rate-limit them, they 429, you retry, you 429 again. Everyone loses. A per-endpoint cap that respects a Retry-After header prevents this.
Yourself, because a slow customer whose endpoint takes 30 seconds to respond will consume every worker in your pool if you let them. Isolating each endpoint into its own queue with its own concurrency cap keeps one bad citizen from stalling every other delivery.
The right default is 100 requests per second per endpoint, tunable per customer. Almost no customer needs more than that. Anyone who says they do usually needs paginated backfill instead.
How do you make webhook debugging not eat your DX team?
Give customers their own dashboard. This is the single highest-leverage decision.
The dashboard has to show, per endpoint:
- Every attempt in the last 30 days, with request body, response body, and status code.
- Response time p50 and p99.
- A replay button that reissues the delivery with the current signing secret.
- A health score with a "why is this endpoint degraded" explanation.
- A filter for event type, status code, and time range.
Without this, every "did you send it" question routes into a support ticket, which routes to engineering, which greps logs, which takes 40 minutes of expensive time. With this, the customer answers their own question in 20 seconds. Companies that ship customer-facing dashboards see webhook-related tickets fall by 60 to 80% within a quarter.
What about receiving webhooks?
If your product integrates with Stripe, GitHub, Slack, or any other API-first vendor, you are also on the ingest side. The engineering is symmetric but the surface area is different.
- Stable per-source URL. Never let the receiver URL change across deploys.
- Signature verification. Verify every inbound webhook against the sender's published rotation.
- Deduplication by event id. Vendors retry too. Persist the event id on first receipt and drop duplicates for at least 24 hours.
- Buffer for replay. If your handler fails, hold the raw payload so you can replay after fixing the bug. A bad deploy that eats a payment notification is a resume-generating event.
Ingest is not a different system from delivery. It is the same retry engine pointed inward.
What actually matters
The mistake to avoid is treating webhook delivery as a feature you ship once. It is a system you operate forever, and the operational cost dwarfs the initial engineering cost. The teams that get this right make two decisions early: they treat the customer-facing debug dashboard as a required launch component, not a nice-to-have, and they build the attempt log with payload search on day one, not after the first ticket. Everything else, the retries, the signing, the rate limits, is table stakes that any competent team can ship in a sprint. Observability is the part that turns a working webhook system into one your customers stop opening tickets about.
Frequently asked questions
How long does it take to build webhook delivery in-house?
The first working version takes about two engineering weeks. The version that survives real customers, complete with retries, signing rotation, rate limits, and payload search, takes six to twelve engineering weeks spread across a year. Most teams stop shipping features on it once the first version works and quietly absorb the ongoing cost as support tickets instead.
What is at-least-once versus exactly-once delivery?
At-least-once guarantees every event reaches its endpoint but permits duplicates during retries. Exactly-once is a fiction for webhooks over HTTP because you cannot atomically commit the send and the ack across two systems. Design for at-least-once, ship an idempotency key in every payload, and let consumers dedupe on their side. Vendors who promise exactly-once are usually promising at-least-once with a debounce window.
Should webhooks be signed with HMAC or JWT?
HMAC with a rotating shared secret is the standard for outbound webhooks because it is fast to verify, portable across languages, and trivial to rotate. JWT adds signature claims that most webhook consumers do not need and complicates verification. Reserve JWT for cases where you need embedded claims a downstream service must trust without a callback.
How many retries should a webhook attempt?
Eight to ten attempts across roughly thirteen hours is the practical range. Fewer than six loses events to normal endpoint flakiness. More than twelve trains consumers to depend on your queue as their durable store, which they will regret. Use exponential backoff with jitter and cap the total retry window at 24 hours.
Do you need a separate ingest pipeline for incoming webhooks?
Yes, if you receive webhooks from third parties like Stripe, GitHub, or your own vendors. The ingest side needs signature verification, deduplication by event id, and buffering for replay, so a bad deploy on your side does not swallow a payment notification. Sending and receiving share the same delivery engine but the ingest surface is a separate URL per source.
Send webhooks. Prove they arrived.
Yaranex handles retries, signing, rate limits, and payload search behind one API, so your customers stop opening tickets and your on-call sleeps.
Request early access