Webhook Delivery Throttling and Retries

Outbound webhooks invert every assumption of inbound rate limiting: you are the client, the limit belongs to someone else’s server, and the consequence of getting it wrong is that your delivery workers spend the afternoon retrying into a subscriber that cannot keep up. The cost function here — one unit per delivery attempt, paced per destination — is the fourth case in protocol-specific throttling, and the one where restraint matters most.

The problem in concrete numbers

A payments platform emits an event per transaction. One subscriber processes webhooks synchronously against a database and answers in 900 ms; another answers in 15 ms. During a Black Friday spike you generate 4,000 events per minute for the slow subscriber. With a shared worker pool of 50 and no per-endpoint pacing, 50 workers end up blocked on that one endpoint, every other subscriber’s deliveries queue behind them, and a partial outage for one customer becomes a delivery outage for all of them.

Add retries and it compounds: each of those 4,000 events fails after a 30-second timeout, is retried three times, and produces 16,000 delivery attempts against a subscriber that was already struggling.

Shared worker pool versus per-endpoint partitions With one shared queue a slow subscriber occupies every worker and blocks all other deliveries; with a partition and concurrency cap per endpoint, the slow subscriber's backlog grows in isolation while other subscribers are unaffected. One slow subscriber, two queue designs shared pool 50 workers, all blocked on endpoint A endpoints B..Z: starved one customer's outage becomes everyone's per-endpoint partitions endpoint A: 2 workers, backlog grows endpoints B..Z: unaffected blast radius: one subscriber isolation first, throughput second

Delivery policy reference

Parameter Typical value Why
per_endpoint_rps 5–20 Pacing that a modest subscriber can absorb; raise per subscriber on request
per_endpoint_concurrency 2–5 Bounds worker occupancy for one slow endpoint
request_timeout 10 s (connect 3 s) Long enough for a synchronous handler, short enough to free the worker
max_attempts 8–12 With exponential backoff, spans hours without infinite retries
backoff min(30s × 2^n, 6h) + jitter Fast early retries for blips, slow later ones for outages
retry_on timeouts, connection errors, 408, 429, 5xx 4xx other than 408/429 will never succeed on retry
honour_retry_after always The subscriber told you the wait; ignoring it invites a block
disable_after 24 h of consecutive failures Stops burning capacity on dead endpoints; notify the subscriber
queue_ttl 24–72 h Events older than this are usually useless to deliver

The retry_on line deserves emphasis: retrying a 400 or 422 is pure waste — the payload will still be malformed on the fourth attempt. Retrying a 401 is worse, because repeated failed authentication attempts get you rate limited or blocked by the subscriber’s own defences.

Step-by-step implementation

python
# delivery.py — per-endpoint pacing, honest retries, idempotent payloads.
import asyncio, hashlib, hmac, json, math, random, time
import httpx

MAX_ATTEMPTS = 10
BASE_BACKOFF = 30           # seconds
MAX_BACKOFF = 6 * 3600      # 6 hours

class EndpointLimiter:
    """One token bucket + concurrency semaphore per destination."""
    def __init__(self, rps: float, concurrency: int):
        self.rps, self.tokens, self.updated = rps, rps, time.monotonic()
        self.sem = asyncio.Semaphore(concurrency)
        self.paused_until = 0.0

    async def acquire(self):
        while True:
            now = time.monotonic()
            if now < self.paused_until:            # subscriber said "wait"
                await asyncio.sleep(self.paused_until - now)
                continue
            self.tokens = min(self.rps, self.tokens + (now - self.updated) * self.rps)
            self.updated = now
            if self.tokens >= 1:
                self.tokens -= 1
                return
            await asyncio.sleep((1 - self.tokens) / self.rps)

    def pause(self, seconds: float):
        self.paused_until = max(self.paused_until, time.monotonic() + seconds)

def sign(secret: bytes, ts: int, body: bytes) -> str:
    mac = hmac.new(secret, f"{ts}.".encode() + body, hashlib.sha256)
    return f"t={ts},v1={mac.hexdigest()}"

async def deliver(client: httpx.AsyncClient, ep, event, limiter: EndpointLimiter) -> bool:
    """Returns True when the event is done (delivered or permanently failed)."""
    body = json.dumps(event["payload"], separators=(",", ":")).encode()
    ts = int(time.time())
    headers = {
        "Content-Type": "application/json",
        "Webhook-Id": event["id"],                 # stable across retries: idempotency key
        "Webhook-Timestamp": str(ts),
        "Webhook-Signature": sign(ep.secret, ts, body),
        "User-Agent": "acme-webhooks/1.0",
    }

    await limiter.acquire()
    async with limiter.sem:
        try:
            r = await client.post(ep.url, content=body, headers=headers,
                                  timeout=httpx.Timeout(10.0, connect=3.0))
        except (httpx.TimeoutException, httpx.TransportError):
            return schedule_retry(event, reason="transport")

    if 200 <= r.status_code < 300:
        return True                                # delivered

    if r.status_code == 429:
        # The subscriber's own limiter spoke: pause the whole partition, not one job.
        wait = float(r.headers.get("Retry-After") or 60)
        limiter.pause(wait)
        return schedule_retry(event, reason="429", delay=wait)

    if r.status_code in (408,) or r.status_code >= 500:
        return schedule_retry(event, reason=f"http_{r.status_code}")

    # Any other 4xx will not succeed on retry — record and stop.
    record_permanent_failure(event, r.status_code, r.text[:500])
    return True

def schedule_retry(event, reason, delay=None) -> bool:
    n = event["attempts"] = event.get("attempts", 0) + 1
    if n >= MAX_ATTEMPTS:
        record_exhausted(event, reason)
        return True
    if delay is None:
        # Exponential with full jitter: spreads a subscriber's recovery wave.
        window = min(MAX_BACKOFF, BASE_BACKOFF * (2 ** (n - 1)))
        delay = random.uniform(0, window)
    enqueue_after(event, delay)
    return False
Retry schedule with capped exponential backoff and jitter Attempts start seconds apart to ride out brief blips, spread to minutes and then hours as failures continue, and stop at the tenth attempt after roughly a day; jitter spreads each subscriber's retries so recovery is not synchronised. Ten attempts across about a day seconds apart: blips minutes apart: deploys hours apart: outages give up, notify full jitter on every delay, so a recovering subscriber is not hit by a synchronised wave What each subscriber response means for the sender A 2xx completes the delivery, a 429 pauses the whole endpoint partition for the advertised duration, a 5xx or timeout schedules a jittered retry, and any other 4xx is recorded as a permanent failure without retrying. Four responses, four different actions 2xx delivered drop from the queue 429 pause the partition for Retry-After seconds 5xx / timeout jittered backoff retry count the attempt other 4xx permanent failure never retry retrying a malformed payload wastes capacity and looks like an attack from the far side

Gotchas and edge cases

  • Retrying without an idempotency key creates duplicates. A delivery that succeeded but whose response was lost will be retried; without a stable Webhook-Id, the subscriber processes the event twice. Send the same ID on every attempt and document that subscribers must deduplicate on it.
  • Two workers delivering the same event. Without a visibility timeout or lock, a slow delivery can be picked up again and sent concurrently. Single-flight per event.
  • Ignoring Retry-After gets you blocked. Subscribers with their own rate limiting escalate: first 429, then connection refusal, then a firewall rule. Honour the number they send.
  • Pausing one job instead of the partition. If a 429 only delays the failed event, the next 200 queued events hit the same wall immediately. Pause the whole endpoint.
  • Unbounded retention. Events retried for a week against a dead endpoint occupy storage and worker cycles forever. Cap attempts and expire the queue.
  • Signature timestamp drift. Subscribers validating a timestamp window will reject deliveries that sat in a queue for hours. Re-sign with a fresh timestamp on each attempt rather than reusing the original signature.
  • Fan-out to many endpoints for one event. One event with 40 subscribers is 40 deliveries, each with its own pacing and retry state. Model delivery as (event, endpoint), never as event.

Verification and testing

bash
# Stand up a deliberately hostile receiver and watch the sender behave.
python3 - <<'PY'
import http.server, time, random
class Hostile(http.server.BaseHTTPRequestHandler):
    seen = {}
    def do_POST(self):
        wid = self.headers.get("Webhook-Id")
        Hostile.seen[wid] = Hostile.seen.get(wid, 0) + 1
        n = Hostile.seen[wid]
        if n == 1: self.send_response(500)               # transient failure
        elif n == 2:                                     # tell them to slow down
            self.send_response(429); self.send_header("Retry-After", "5")
        else: self.send_response(200)                    # finally succeed
        self.end_headers()
        print(f"{wid} attempt {n} -> {self.responses if False else ''}")
    def log_message(self, *a): pass
http.server.HTTPServer(("", 8099), Hostile).serve_forever()
PY

# Expected sender behaviour, observable in your delivery log:
#   attempt 1 -> 500, retry scheduled with jitter within ~30 s
#   attempt 2 -> 429, PARTITION paused 5 s, retry after that
#   attempt 3 -> 200, delivery marked complete
#   Webhook-Id identical on all three attempts

Assert two invariants in an automated test: the Webhook-Id is stable across attempts, and a 429 from the receiver stops all deliveries to that endpoint for the advertised duration, not just the one that failed.

Frequently Asked Questions

How many times should a failed webhook be retried?

Eight to twelve attempts with capped exponential backoff, spanning roughly a day. That rides out brief errors, a bad deploy, and a short outage, without spending resources indefinitely on an endpoint that is gone. Pair it with automatic disabling and a notification after a sustained failure window.

What should happen when a subscriber returns 429?

Pause the entire delivery partition for that endpoint for the duration in Retry-After, then resume at the paced rate. Retrying only the failed event leaves the rest of the backlog to hit the same limit immediately, which reads as an attack from the subscriber's side.

Do webhooks need idempotency keys if delivery is at-least-once?

Precisely because delivery is at-least-once. Send a stable event identifier on every attempt so a subscriber can detect a duplicate caused by a lost response, and document the header so they know what to deduplicate on. Without it, a network timeout silently doubles a payment notification.

Should the delivery rate be configurable per subscriber?

Yes. Default to a conservative pace, and let subscribers raise it once they have demonstrated they can absorb more — ideally as a self-service setting with the delivery success rate visible next to it. A single global rate either starves fast subscribers or overwhelms slow ones.