Skip to content
All writing

My Stripe webhook handler looked safe. Replaying 14 real events broke it in 24 of 50 runs.

I chaos-tested a careful-looking NestJS and Postgres webhook handler with real Stripe test-mode events. It had two bugs: a double-payment race and a refund that could be undone.

I've worked with several kinds of webhooks, both inbound and outbound: receiving events from third-party platforms, and sending our own to other services. The delivery rules are the same in both directions, and payments are where breaking them costs the most, so that's where I tested.

Every webhook provider says the same thing in its docs: delivery is at least once, and order is not guaranteed. Stripe retries failed deliveries for up to three days. Paystack retries every few minutes, then hourly. Your endpoint will see the same event twice, events out of order, and retries of requests it already processed.

Most of us read that, nod, add an "already processed?" check, and move on. Then we test the handler by triggering one event and watching it work.

I wanted to know what actually happens when delivery gets messy. So I built a small tool, Hookstorm, and pointed it at a handler written the way I'd normally write one. It found two real bugs.

The handler

The app is a small ticket shop: NestJS, Postgres, Stripe in test mode. Checkout creates an order and a PaymentIntent. The webhook handler does four things when payment_intent.succeeded arrives: mark the order paid, take the tickets out of stock, write a ledger entry, and email a receipt. charge.refunded reverses it.

Here's the payment handler, trimmed slightly. I'd have approved this in code review:

async markPaid(intent: Stripe.PaymentIntent) {
  const order = await this.db.one<Order>('SELECT * FROM orders WHERE id = $1', [intent.metadata.order_id]);
  if (!order) return;
  if (order.status === 'paid') return; // already processed

  await this.db.transaction(async (tx) => {
    await tx.query(`UPDATE orders SET status = 'paid' WHERE id = $1`, [order.id]);
    await tx.query('UPDATE products SET stock = stock - $1 WHERE id = $2', [order.quantity, order.product_id]);
    await tx.query(`INSERT INTO ledger_entries (order_id, type, amount) VALUES ($1, 'payment', $2)`, [order.id, intent.amount_received]);
  });

  await this.email.sendReceipt(order);
}

It checks for duplicates, uses a transaction, and decrements stock atomically. The signature is verified with Stripe's own SDK. It looks fine.

What Hookstorm does

Hookstorm plays the webhook provider. You give it a set of events, and it delivers them to your endpoint with the things providers really do:

  • duplicates, sequential and concurrent;
  • reordering and delays;
  • dropped requests, where the provider retries;
  • dropped responses, where your app processed the event but the provider never saw the 200, so it retries;
  • timeouts.

It uses each provider's retry rules, compressed from hours into milliseconds, and re-signs every attempt so your real signature check still runs.

Then it checks business rules you write in a TypeScript config, against two things:

  • the side effects your app performed, recorded by a local "sink" you point outgoing calls at;
  • your app's state, read from a test-only endpoint.

Every run is driven by a seed, so any failure comes with a command that replays exactly the same delivery plan.

It replays recorded events rather than sitting in front of the real provider. Waiting on Stripe's real retry schedule would take days and couldn't be reproduced.

Recording real Stripe events

I didn't want hand-written fixtures, so I recorded real test-mode events. Hookstorm has a recorder that sits between stripe listen and the app, saving each event byte for byte while passing it through:

npx hookstorm record --provider stripe --port 4500 --forward http://127.0.0.1:3001/webhooks/stripe -o hookstorm/events.jsonl
stripe listen --api-key "$STRIPE_SECRET_KEY" --forward-to localhost:4500 \
  --events payment_intent.created,payment_intent.succeeded,charge.succeeded,charge.updated,charge.refunded,refund.created

Three checkouts and one refund produced 14 events across 6 types.

My first recording captured nothing, with no error anywhere. stripe listen was listening on the account the CLI was signed in to, while my app's secret key belonged to a different sandbox. The payments went to one account and the listener watched the other. Passing --api-key with the app's own key fixed it, and Hookstorm's recorder now prints a hint when nothing arrives.

The rules

These are the things that must stay true however the events arrive (abridged; the full config is in the repo):

invariants: [
  atMostOnce('email.receipts', (c) => c.body.orderId, 'each order gets at most one receipt'),
  exactlyOnce('email.receipts', (c) => c.body.orderId, {
    forEachEvent: 'payment_intent.succeeded', eventKey: orderIdOf, description: 'every paid order gets a receipt',
  }),
  custom('at most one payment ledger entry per order', ({ state }) => /* count ledger rows per order */),
  finalState('a refunded order stays refunded', (state, obs) => /* refunded orders are still refunded */),
  custom('ticket stock matches paid, unrefunded orders', ({ state, events, isAcked }) => /* recompute */),
  eventsTakeEffect({ forEachEvent: ['payment_intent.succeeded', 'charge.refunded'], eventKey: orderIdOf,
    record: (state, id) => state.orders.find((o) => o.id === id) }),
],

The last rule guards the test itself; more on that below.

What broke

With the realistic profile over 50 seeds, 24 of 50 runs broke 5 of the 6 rules. With the hostile profile, 29 of 30 did.

Bug 1: two deliveries at once, two payments

26ms  deliver evt_2#0 (payment_intent.succeeded) attempt 1 → acknowledged 200
27ms  deliver evt_2#1 (payment_intent.succeeded) attempt 1 → acknowledged 200
→ order a09b…0e3b has 2 payment entries

Two copies of the same event arrive a millisecond apart. Both read the order while it's still pending, both pass the status === 'paid' check, and both run the transaction. The result is two ledger entries, tickets taken out of stock twice, and two receipts.

The check is outside the transaction and nothing locks the row, so it doesn't protect anything when deliveries overlap. It only works when they don't.

This bug is timing-dependent, which makes it nasty: rerunning the same seed doesn't always fail. Hookstorm's --repeat-seed replays one delivery plan many times. The first time I ran it, seed 4 failed in 19 of 20 runs, and the first run passed (a later run failed all 20). If I had rerun that seed once after "fixing" something, I could have believed it was fixed.

Bug 2: a retry after a refund pays the order again

196ms  payment_intent.succeeded attempt 1 → response dropped (app processed it, provider saw a failure)
199ms  charge.refunded attempt 1 → acknowledged 200
257ms  payment_intent.succeeded attempt 2 → acknowledged 200        ← processed again
→ order ended as "paid" after its refund was acknowledged

(This timeline is from my first run, with hand-written events. With the real recording, the same rule failed in 2 of 50 realistic runs and 15 of 30 hostile runs.)

The app processed the payment, but the 200 got lost, so the provider retried. Meanwhile the refund arrived and set the order to refunded. When the retry came in, the guard asked "is it paid?". It wasn't (it was refunded), so the payment ran again. The order went from refunded back to paid, with a second ledger entry, stock taken out again, and a second receipt.

Testing one event at a time never produces this sequence. In production it needs a lost response and a fast refund, which is rare but certain to happen eventually.

The fix

Four changes, all standard patterns, none specific to Stripe:

1. Apply each event at most once, in the same transaction as its effects.

private async applyOnce(event: Stripe.Event, apply: (tx: PoolClient) => Promise<void>) {
  await this.db.transaction(async (tx) => {
    const claimed = await tx.query(
      'INSERT INTO processed_events (event_id, type) VALUES ($1, $2) ON CONFLICT DO NOTHING RETURNING event_id',
      [event.id, event.type],
    );
    if (claimed.rowCount === 0) return; // already applied
    await apply(tx);
  });
}

A concurrent duplicate blocks on the primary key until the first transaction commits, then inserts nothing and skips. If the first one rolled back, the duplicate applies it instead, so nothing is lost.

2. One payment and one refund per order, enforced by the database. A unique index on ledger_entries (order_id, type) plus SELECT … FOR UPDATE on the order. This holds even if the provider sends two different events for the same payment.

3. Status only moves forward. A payment that arrives after its refund still records the money, but it doesn't change the status back:

UPDATE orders SET status = CASE WHEN status = 'refunded' THEN status ELSE 'paid' END WHERE id = $1

4. Receipts go through an outbox. The payment transaction writes one outbox row per order, and a worker sends it after the commit. The webhook no longer waits on the email API, and a failing email can't cause a 500 that loses the receipt.

Before and after

Same 14 real events, same seeds, same six rules:

RunBeforeAfter
realistic, 50 seeds24/50 broke 5 of 6 rulesall held
hostile, 30 seeds29/30 broke 5 of 6 rulesall held
seed 4, replayed 20×failed 20/20all held
hostile, 200 more seedsn/aall held (9,215 delivery attempts)

The race fix makes the race impossible, not just less likely. That's why it holds on every repeat, and why "passes 200 hostile seeds" means something.

The trap: a test that tests nothing

To replay recorded events, the orders they refer to have to exist, in their state before the webhooks. The obvious reset route just empties the tables. Do that, and every replayed event points at an order that doesn't exist. The handler does what most handlers do: logs "unknown order", returns 200, and does nothing.

Now look at the rules. "At most one payment per order", "at most one receipt" and "a refunded order stays refunded" all pass: nothing was double-paid because nothing was paid at all. In my config, two rules would have noticed, because they expect something to happen: "every paid order gets a receipt", and the last rule, which checks that each payment and refund actually changed its order. Plenty of test setups only check that bad things don't happen, and for them the whole run is green while testing nothing.

I avoided it by exporting the orders after recording and re-inserting them in reset. If you write tests that replay events, any kind, check this. Make sure a run where your handler ignored everything would fail: add at least one rule that expects each event to change something. (Hookstorm now warns when a run's events had no visible effect.)

Takeaways

  • "Already processed?" checks outside a transaction don't stop concurrent duplicates. Claim the event id in the same transaction as the work (INSERT … ON CONFLICT DO NOTHING), or lock the row.
  • Idempotency by status is not enough when events are reordered. Make status changes forward-only, and enforce "once per order" with a unique constraint.
  • Don't do slow or failure-prone work before the 200. Use an outbox.
  • A race that passes on rerun is still a race. Measure it with repeated runs, and fix it so it can't happen.
  • Make sure your tests can fail. A replay test whose events point at nothing passes forever.

Try it

npm i -D hookstorm-cli
npx hookstorm init --provider stripe --target http://localhost:3000/webhooks/stripe

There are presets for Stripe, Paystack and GitHub, plus a custom provider for anything else (Flutterwave, Monnify, in-house). The guide walks through recording, seeding and writing rules.

If you point Hookstorm at your own handler, I'd like to hear what breaks, and what doesn't. Open an issue on the repo or message me. A seed that breaks your handler is a better story than mine.

Have a role or a project in mind? Let’s talk.