If a Discord bot takes your money, you should get to see how its billing breaks. This is the postmortem of the July incident where every Stripe webhook handler in Loracle started throwing on its first line, our own endpoint told Stripe everything was fine anyway, and the only evidence in the entire system was a single telemetry event whose error string was one word: get.

Nothing about this failure is exotic. It is three ordinary decisions, each defensible on the day it was made, that interlocked into silent data loss. If you run a service that charges people, there is a decent chance you have all three in production right now. Here is how to check.

The one-word error

The first symptom: a plan switch to Pro through the Stripe customer portal. Stripe charged the card. The app never found out, so it kept serving the old tier the account had just paid to leave.

The webhook handlers were the suspects, and every one of them read payload fields the same way:

customer_id = payload.get("customer")

Through stripe-python 14.x that worked, because the SDK’s StripeObject subclassed dict. In 15.x it no longer does. Calling .get on the new object falls through to __getattr__, which raises AttributeError("get"). So every subscription handler, and checkout.session.completed with them, raised on their first field read. The exception message was the string get. That was the whole clue.

How the upgrade got in

We never chose to upgrade to 15.x. The backend requirements said, in full:

stripe

An unpinned dependency means every image rebuild quietly re-rolls the dice on your SDK major. One routine deploy, built on a day when 15.x happened to be the latest, swapped the library out from under handlers that had encoded a 14.x implementation detail as if it were a guarantee. Nothing in our code changed. Nothing in our tests failed, because the tests fed the handlers plain dicts, which still have .get.

The invariant “Stripe payloads are dict-like” lived in the SDK’s implementation, and we had been treating it as if it lived in the API contract. The compiler can’t warn you about that category of assumption. A pin can.

The part that turned a bug into data loss

A broken handler should be a delay, you’d hope, since Stripe retries failed webhooks on a backoff ladder for days. Delivery fails, Stripe retries, you ship a fix, the retries land, the data heals itself. That is the design, and it is a good one.

Our endpoint had this in it:

except Exception:
    # return 200 to prevent Stripe from retrying
    return {"status": "ok"}

That comment is the most expensive line in this story. Someone (fine: me, in a hurry, tired of a noisy retry queue during development) decided retries were an annoyance and disabled the one mechanism built to survive exactly this incident. Every exploding handler answered 200. Stripe marked each delivery successful and discarded it. Two paid plan changes crossed the wire during the window, got a thumbs-up at the door, and were lost for good; the only recovery path left was digging the events out of the Stripe dashboard and replaying them by hand.

Swallowing an exception and reporting success is always a gamble that the exception didn’t matter. At a billing boundary, it always matters.

What we actually shipped

Four changes, the same night, each aimed at the class of failure rather than the instance.

One accessor, stated rule. Handlers no longer touch payloads directly. stripe_field(obj, name, default) is the only way a field gets read. It handles dict-shaped and attribute-only objects alike, and it distinguishes a field that is absent from a field that is explicitly null, which Stripe payloads genuinely use to mean different things. Thirty-three call sites, one place to be wrong.

This one earned its keep within the hour. The first commit fixed the .get calls in the billing client, and the failure didn’t stop: it moved, from the processing stage to the tier_sync stage, because the API layer held a second, independent copy of the same dict-or-attribute branching. A fix that relocates a failure is the signature of a duplicated invariant. Stage labels on the failure telemetry are what made the movement visible; without them we would have read the second crash as “the fix didn’t work” and thrashed. The second commit consolidated both copies onto the accessor.

Pin your majors. stripe>=12.0.0,<16.0.0 in both requirements files, with the reason written next to the pin. A major upgrade is now a deliberate act by a person reading a changelog, on a branch, with the test suite between them and production.

Tests that drive both shapes. The suite now runs every affected handler against a plain dict and against an attribute-only stand-in object. The stand-in asserts that it really does raise AttributeError("get"), so the test can’t silently become a no-op if someone “simplifies” it later. We verified the suite fails when a single raw .get is restored.

Let Stripe retry. Unexpected exceptions now raise 500. Stripe retries on its backoff ladder, which means a future handler bug is a delay followed by self-healing instead of permanent loss. Signature verification failures still get their own explicit rejection; “I can’t verify you” and “I broke” are different answers and now return different codes.

The grep you should run today

Three checks, ten minutes, in order of likely damage:

  1. Search your webhook endpoints for return 200 or status.HTTP_200_OK inside an except block. Each hit is a place where you have told your payment processor to never retry, no matter what just went wrong.
  2. Open your requirements or lockfile and look at the line for your payment SDK. If there is no upper bound on the major version, your billing code’s dependencies are decided by your rebuild schedule.
  3. Check what your webhook tests feed the handlers. If the fixtures are plain dicts and production receives SDK objects, the tests are exercising a payload shape production will never see.

None of these require a refactor. All three are cheaper than explaining to a customer why the plan they paid for didn’t show up.

Why a D&D bot is telling you this

Loracle’s pitch is that your table’s memory is safe with us: the recording, the recap, the campaign wiki. Billing runs on the same promise with different data. A company that would quietly drop your plan change is making a statement about how it treats state it is trusted with, and you are entitled to judge the whole product by it.

So the standing rule that came out of this incident applies everywhere in the system now: a boundary that swallows an exception and reports success is a bug, even when the comment explains why it seemed convenient. Especially then.