13 Ways a Recording Bot Can Lose Your Session
We paid an adversarial audit to attack our own recording pipeline the day we started buying traffic. It found thirteen ways to lose your table's only copy of the night. Here's every one.
Your table’s session has exactly one copy. Six people spent four hours making something that will never happen again, and the only record of it lives inside a bot that promised to be listening. If you’re a GM deciding whether to trust a recording bot with that, you should want to know how these things fail. So here is how ours fails, all thirteen ways, because the day we started paying for traffic we commissioned an adversarial audit of our own recording pipeline with one mandate: “from /record to /end we minimize the data we lose, always, and always package it up correctly.”
The auditor traced every failure mode to the exact file and line. Four of the thirteen were bad enough that we shipped structural fixes the same night. This post is the honest version of that audit, written for the person whose session is on the line.
Where your audio actually lives
To understand the failure modes you need the shape of the machine. When Loracle records, your audio exists in three places at any moment:
- RAM: up to 30 seconds of raw audio per speaker, the in-flight buffer.
- Object storage: every 30 seconds, each speaker’s buffer is encoded and uploaded as a chunk. This happens continuously all session.
- The database: a row that knows the recording exists, which machine owns it, and a heartbeat updated every 15 seconds.
The design goal is that at most 30 seconds of audio is ever in a place that can die. Everything else should already be somewhere durable. Every failure mode below is a story about one of those three places, or about the handoffs between them.
The thirteen
1. The voice connection drops and quietly resumes. Discord’s voice websocket hiccups, the library resumes it, and recording continues. Audio during the gap was never transmitted, so it can’t be saved. The timeline stays honest (silence is marked as silence, so speakers don’t drift out of sync), but those seconds are gone.
2. Discord migrates your voice region mid-session. Same shape: a brief gap while the connection re-handshakes on the new server, then business as usual.
3. End-to-end encryption fails mid-session. Discord’s E2EE voice protocol can desync. We tolerate a burst of undecryptable packets, then force a recovery re-handshake, and health counters measure exactly how much was lost (typically one to two seconds per event). This one took most of July to chip down.
4. The bot process crashes. The supervisor restarts it, the new process sees a recording row with a stale heartbeat, atomically claims it, rejoins your voice channel, and resumes from the next chunk index. You lose the in-RAM buffer plus the restart gap, bounded at roughly 30 seconds. When the session ends, the composer merges chunks from both the dead process and the new one, so the final file spans the crash.
5. We deploy new code while you’re recording. The old process flushes its buffers, waits for uploads, stamps the row as a handoff, and exits without closing the voice connection, deliberately leaving it dangling so the new process can adopt it within seconds. You should never notice a deploy.
6. A server admin kicks the bot from voice. Your audio is already in storage, so it’s safe, and the bot apologizes in the channel and marks the session interrupted so you can finish it with /end.
7. Everyone leaves the voice channel and forgets the bot. Nothing is lost, because there’s nothing to record. The failure here is duration accounting: a forgotten overnight session burns wall-clock time against your allowance for hours of silence. This one is on our list rather than fixed, and I’d rather tell you that than pretend.
8. /end races the bot’s own disconnect. This was the nastiest finding. When you type /end, two independent pieces of code both react to the same disconnect signal: the finalizer that composes your audio, and the watchdog that flips sessions to “interrupted” when the bot drops unexpectedly. With the wrong timing, the watchdog won the race, and your session composed perfectly and then sat there marked interrupted, recap never delivered. Audio safe, product silently absent. A previous fix had closed one entrance to this room; the audit found three more. The structural fix gives finalization a single owner: the finalizer removes the connection from the shared state before disconnecting, so the watchdog’s check fails closed. And the write that used to fail silently now logs loudly, because after this fix a silent failure genuinely indicates a bug.
9. A chunk upload fails. Storage blip, network blip, whatever. Before the audit, this was the one true hole: the code caught the error, logged it, and moved on, and those 30 seconds were gone forever while the recording looked perfectly healthy. The audit ranked it the worst finding, the only mode that destroys audio after we captured it. The fix is an invariant now stated in the code: a flushed buffer is never dropped until some durable copy exists. On upload failure, chunk bytes spill to local disk and a retry queue drains them, with drains at flush, at finalize, and at the next process startup. A storage outage now delays your chunks instead of deleting them.
10. Transcription fails after upload. Your audio is durable at this point, so this mode was always safe. The session is marked with an error you can retry, and a background sweep repairs anything stranded.
11. Someone starts a second recording while one is running. The old session gets finalized, the new one starts, and there’s an ugly one-second window where their bookkeeping can collide. Confusing UX, no lost audio.
12. Your session runs long and the composer runs out of memory. The old composer decoded the entire session into RAM to stitch it. A three-hour talkative table could hit two gigabytes at exactly the moment of finalization, crash, and then crash again on retry. The most engaged tables, the ones with the longest sessions, were the most likely to hit it. The fix: composition is now a streaming concatenation (ffmpeg stitching compressed frames without decoding them), so memory use is constant whether the session ran one hour or six.
13. One server’s kick nuked every server’s recording. Found in passing, the kind of bug that only bites at scale: the disconnect handler queried for all live recordings on the machine instead of the affected server’s. One table’s kick would have flipped every concurrent table’s session to interrupted and posted apologies into all of their channels. Scoped to the affected server now.
There was a fourteenth finding, sort of: the health check that our host uses to auto-restart sick machines couldn’t see the bot at all, only the web API. A hung bot would sit mute in your channel forever while every dashboard glowed green. The health endpoint now probes the bot’s own event loop, so a hang triggers an automatic restart instead of a silent wedge. The dangerous failures are the ones that look like health.
What I’d ask any recording bot
Strip away our specifics and the audit reduces to four questions worth asking of anything you trust with your table’s only copy:
- How often does audio reach durable storage? If the answer is “at the end of the session,” a crash at hour three costs you three hours. Ours is every 30 seconds.
- What happens when an upload fails? “We retry” should come with a where-do-the-bytes-wait answer. Ours wait on disk.
- What happens when the process dies mid-session? Resume-from-where-it-stopped and merge-across-the-crash, or start over?
- Can the longest sessions actually finalize? Memory that grows with session length means your marathon sessions, the ones you most want recorded, are the ones most likely to fail at the finish line.
The uncomfortable truth of the audit is that our capture spine was already strong and the finish line was where sessions went to die: composed audio stranded by a race condition, finalization crashing on the sessions that mattered most. Recording is a promise about the whole pipeline, from the moment someone says something great at 9:40 p.m. to the moment the recap posts in your channel the next morning. Thirteen failure modes, four structural fixes, one standing invariant. We’ll publish the next audit too.