Building a Reliable Webhook Delivery System

Webhooks look simple on the surface — send an HTTP POST when something happens — but building a delivery system customers can actually rely on requires handling a surprising number of failure modes.

Retries With Exponential Backoff

Customer endpoints go down, get redeployed, or hit rate limits. A single failed delivery attempt shouldn't mean lost data. We retry failed deliveries with exponential backoff — 1 minute, 5 minutes, 30 minutes, then hourly — for up to 24 hours before marking a delivery as permanently failed.

Idempotency Keys

If a delivery attempt times out on our side but actually succeeded on the customer's server, a naive retry creates a duplicate event. Every webhook payload includes a unique event ID that customers can use to deduplicate, and we document this pattern clearly so integrators build it in from day one.

Signature Verification

Every webhook is signed with an HMAC signature derived from a per-customer secret, allowing receivers to verify the payload genuinely originated from us and wasn't tampered with in transit. This is non-negotiable for any webhook system handling customer data.

Delivery Logs and Replay

Customers can see a log of every delivery attempt — timestamp, response code, response body, and latency — directly in their dashboard, along with a one-click "replay" button for any failed event. This turns webhook debugging from an opaque black box into something customers can self-serve.

Circuit Breaking for Dead Endpoints

If an endpoint has failed the last 50 consecutive deliveries, we pause further attempts and notify the customer rather than continuing to hammer a clearly broken URL. This protects both our infrastructure and the customer's, and it's caught more than one forgotten decommissioned staging server.

Ordering Guarantees (and Their Limits)

We deliver webhooks in best-effort order but don't guarantee strict ordering, since retries can cause a delayed event to arrive after a newer one. Every payload includes both an event timestamp and a monotonically increasing sequence number, so integrators who genuinely need ordering can buffer and reorder on their end rather than assuming our delivery order matches the true event order.

Load Testing the Delivery Pipeline

Before launch, we simulated a burst scenario — ten thousand responses arriving within a single minute, as happens during a product launch campaign — to confirm the queue-based delivery architecture could absorb the spike without falling behind. This testing surfaced a connection pool exhaustion bug under sustained high concurrency that would have been invisible under normal testing volumes.

A webhook system is a promise to deliver an event exactly once, eventually. Meeting that promise requires treating every possible failure as the expected case, not the exception.