Webhook reliability: patterns from processing a million payment events

When a payment gateway confirms a charge, that confirmation arrives as a webhook. If you drop it, you have a paid customer with no ticket — the worst failure mode in ticketing. After processing over a million payment events, these are the patterns we consider mandatory.

1. Verify, then trust

Every inbound webhook is signed — in Nexpay's case, HMAC-SHA512 over the raw body. Verify with a constant-time comparison before parsing anything. Log rejected signatures: a burst of them is either a misconfigured secret or someone probing your endpoint.

2. Idempotency is the whole game

Gateways deliver at-least-once, which means you will receive duplicates. Key every state transition on the payment ID: 'mark order paid' must be a no-op the second time. We issue tickets inside the same transaction that flips the order state, so a redelivered webhook can never double-issue.

3. Log first, process second

• ▸Persist the raw payload and signature before any business logic runs — if processing crashes, you can replay.
• ▸Keep the webhook handler thin: verify, log, enqueue. Heavy work (emails, PDFs, CRM sync) belongs in the job queue.
• ▸Alert on the gap metric: payments in PENDING older than your gateway's p99 confirmation time mean webhooks are being lost upstream.

4. Reconcile anyway

Webhooks are the fast path, not the source of truth. A periodic reconciliation job pulls payment history from the gateway and diffs it against local state. In a healthy month it finds nothing; the month your webhook endpoint silently 500s behind a proxy change, it saves the business.