In July we found out that Loracle’s welcome email had never once been capable of sending in production. Every attempt, for the feature’s whole life, died on its first line with No module named 'resend'. CI was green the entire time. The tests passed the entire time. The code was fine the entire time. The bug lived somewhere most teams never look: in the gap between two requirements files.

If you deploy Python with Docker, there is a decent chance you have this exact gap open right now. This is the postmortem, plus the five-minute check that tells you whether you’re carrying the same bug.

Two manifests, one of which ships

The repo had a requirements.txt at the root. Dev machines installed it. CI installed it. It listed resend, the email SDK, because the machine that added the email feature installed the package and dutifully recorded it where everything else was recorded.

Production never read that file. The Dockerfile that builds the prod image installs src/web/backend/requirements.txt, a separate, smaller manifest for the backend service, and that file had never heard of resend. So dev worked, CI worked, and the production image was built, every time, without the one library the email path imports.

This is the whole bug. There was no version conflict, no broken wheel, no platform quirk. One list of packages fed the environments that run tests, a different list fed the environment that runs users, and a package landed on only one of them.

The galling part: this failure was predicted. A red-team review five days earlier had flagged the split-manifest setup with the note “the Python pins never reach prod” and filed a follow-up. The follow-up was still sitting in the queue when the real bug arrived wearing its exact description. When a review predicts a failure class, that prediction has a shelf life. We got to the follow-up after the incident, which is the expensive order.

Why no test could have caught it

Here’s the trap. The obvious regression test, “import every module and make sure nothing’s missing,” passes. CI installs the root requirements file, resend is in it, the import succeeds. The test environment can’t see the problem because the test environment is built from the healthy manifest. Any check that runs code is checking the wrong world.

Silence made it worse. The email path caught the ImportError, logged it where nobody was looking, and moved on. No user-visible error, no alert, no failed deploy. A latent failure in a feature nobody had flipped on yet makes no sound at all.

What finally surfaced it was procedure, and honestly, a boring one: the launch checklist said “run the welcome-email test against production and read the logs,” so that’s what happened. The log line was right there. The whole reason a prod smoke test exists is that some bugs are only visible from inside production; this one was invisible from literally everywhere else.

The fix is static analysis, on purpose

Patching the manifest takes one line. The class of bug (“two dependency manifests, only one of which ships”) survives the patch and waits for the next package. So the fix that mattered is a test, and the design constraint on it came directly from the paragraph above: since nothing that runs code in CI can detect the problem, the test must never import anything.

It reads the production requirements file as text, walks every source file in the backend with an AST parser, collects each third-party import, and asserts that every one of them appears in the manifest production actually installs. No environment, no imports, no luck involved. A small hand-maintained table maps import names to package names where they differ (jwt to pyjwt, dateutil to python-dateutil); keeping that table embarrassingly small is part of the discipline.

The payoff was immediate, and bigger than the incident. The very first run of the guard found four more undeclared packages in the backend:

  • reportlab, a lazy import inside the PDF export path. Lazy imports dodge every startup check by definition, so PDF export would have crashed in production the first time a real user clicked the button. Same bug as resend, one unlucky click away from being the next incident.
  • aiohttp, pydantic, and starlette, each imported directly by our code but present in the image only because some other dependency happened to drag them in. That arrangement is a bet that fastapi’s internals never reshuffle. Transitive availability is luck wearing a contract’s clothes.

One incident, five bugs of the same shape. That ratio is the argument for fixing classes over instances: the instance costs you a line, the class keeps paying you back.

The debugging detour worth stealing

One more lesson from that night, from before the root cause surfaced. The first prod test attempt reported reason=too_old age_days=183.85, claiming the test account was six months old. It was a day old. The age came from the database, which meant an earlier UPDATE meant to backdate the account had matched zero rows, because its WHERE clause used an email address that didn’t match the prod account. In SQL, an UPDATE that matches nothing reports success.

The only reason any of this was diagnosable is that the email scheduler logged why it skipped each send, with the inputs that drove the decision (reason=too_old, the age, the timestamp it computed the age from). Without that one log line the incident report would have read “no email arrived” and the investigation would have had nothing to grab. When you write a skip path, log the reason and the numbers behind it. Skip reasons are the cheapest observability you will ever add, and in this story they carried the entire diagnosis.

The five-minute check

Three questions, straight from this incident, answerable with grep:

  1. How many dependency manifests do you have? find . -name "requirements*.txt" -not -path "*/node_modules/*" (or the poetry/uv equivalent). More than one is a gap waiting for a package.
  2. Which one does production install? Open your Dockerfile and read the COPY and pip install lines. If that file and the one CI installs can drift apart, they eventually will.
  3. Does your code import anything your prod manifest never names? That’s the guard test above, and it’s maybe forty lines of ast walking. Steal the idea. Run it in CI, where it will do what our whole test suite couldn’t: fail on a machine that’s green.

A table trusts a bot with the only recording of their game night. That trust is built out of unglamorous things like reading prod logs on purpose and writing tests that refuse to import anything. This is what that looks like from the inside.