The Corpse Space: How We Chased a Crackle Through Three Different Bugs

There's a particular kind of dread that comes from hearing "there's no audio" about a feature you just spent weeks building. This week, Tula Spaces gave us that moment three times — and each time, the real bug was hiding one layer deeper than we thought.

Here's the full story, because we think the debugging path is as useful as the fixes.

The setup: getting iOS onto real MoQ

Spaces is Tula's live audio feature, built on a self-hosted MoQ (Media over QUIC) relay stack. Android had already proven the transport worked — the real milestone this week was getting an iPhone onto the native lane, using react-native-moq's Rust QUIC core instead of a WebView shim. That was the whole reason we chose that path in the first place: it was the only way to make iOS work at all.

We built a fresh Release build, stood up a real escalated Space on production with a test beep, and joined from an iPhone 16 Pro.

The transport worked perfectly. The audio did not.

Bug 1: iOS can decode Opus — it just can't decode long packets

The logs showed MoQKit.SessionError Code=4 — "Audio decode error" firing roughly ten times a second, on every single frame. The stream itself was healthy — packets were arriving continuously — but the decoder rejected every one of them.

The easy conclusion would have been "iOS can't decode Opus," and switching to AAC would have made the symptom disappear. But that conclusion would have been wrong, and it would have cost us the 2× bandwidth savings Opus gives us over AAC — savings that matter a lot when you're trying to keep relay costs sane at scale.

So we went looking for the real mechanism instead. Reading through moqkit's source confirmed it has a full Opus decode path via Apple's AVAudioConverter — this isn't file-playback support (the kind you get for free in the Files app or Safari), it's genuine low-level streaming decode. We replicated the failure on a Mac against packets encoded exactly like our production bridge, and found the actual rule: Apple's Opus decoder refuses any packet longer than 60ms, and our bridge was publishing 120ms frames — a choice made specifically to keep relay group density low.

The fix was one environment variable: TULA_BRIDGE_AUDIO_FRAME_MS=20. Zero app changes, zero codec switch. The iPhone went from ten decode errors a second to zero, playing the exact same production Opus stream. We only gave up the packet-rate efficiency of larger frames — not the format itself, and not the 2× egress win. That efficiency gap is now a tracked upstream item, not a workaround we're stuck with.

Bug 2: the crackle that wasn't ours to fix on the client

Fixing the decode error should have been the finish line. Instead, we started hearing something new: audio that would crackle, drop to silence for a moment, and then resume.

The instinct was to suspect the client — a starved jitter buffer, a blocked real-time audio thread, a session interruption. We built a stats pipeline to check: ring buffer depth, stall counts, dropped-frame counters, latency, all logged live off the device. Every single one of those numbers came back healthy for the entire session. No accumulating stalls, no buffer drift, stable latency throughout.

Clean client stats with an audibly broken stream meant the bug had to be somewhere the client-side instrumentation couldn't see — so we captured the stream directly off the relay, no phone involved. The recording confirmed roughly 1.7 micro-silences per second, and the relay-side capture showed the published stream itself was missing about 1.5 packets a second. This wasn't a client bug at all. It hit Android too.

The root cause was a subtle interaction between three pieces of infrastructure: the bridge's ffmpeg flag frag_every_frame was creating one fMP4 fragment per 20ms audio frame, the MoQ importer was mapping each fragment to its own MoQ group, and the relay was quietly skipping single-frame groups under the slightest scheduling pressure. Every skipped group was a clean 20ms hole that the client dutifully filled with silence — because as far as the client was concerned, nothing had gone wrong on its end.

The fix was to fragment by duration instead of by frame: -frag_duration, exposed as TULA_BRIDGE_AUDIO_FRAG_MS. We tested 100ms and 200ms — both eliminated the holes entirely, but 200ms introduced an occasional pop of its own, because the lumpier packet arrivals let the client's 250ms buffer scrape bottom. 100ms is the value we shipped.

The scare that turned out to be two different problems

Between these fixes, we had a moment that looked like the whole feature had regressed overnight: a Space that had audio the night before had none the next morning. It turned out to be neither an audio bug nor an Android-specific issue — it was two genuine gaps in our operational plumbing.

Bridge restarts were silently killing live audio. moq-bridge keeps its session state only in memory, and nothing was re-arming it after a restart. The nightly unattended-upgrades run had bounced the service overnight, and the escalated test Space simply went silent with no error anywhere — a production Space would have had the exact same fate at the same time every night.

Dead Spaces stayed joinable indefinitely. A Space whose host disconnects without properly ending it just sits marked live forever. We joined a Space from a drill the night before whose host was long gone, and got silence that had nothing to do with codecs, frame sizes, or relay groups — there was simply no one broadcasting.

Both are now fixed, with defense in depth rather than a single point of prevention:

• Box 2 now exempts moq-bridge and moq-relay from automatic restarts, removing the nightly trigger entirely.
• A new job, spaces.moq_bridge_reconcile, runs every 60 seconds and compares which Spaces are marked escalated in the database against which sessions the bridge actually reports as live — re-arming anything missing, and refusing to act at all if the bridge's status endpoint is unreachable, so it can never mass-start duplicate sessions.
• The existing stale-Space reaper now runs every 60 seconds instead of every 15 minutes, and force-ends any live Space with no connected host or co-host for a configurable grace period. A corpse Space can now live for at most a few minutes, not overnight.

What we're taking away from this

Every one of these bugs looked, at first glance, like a worse version of itself. A decode error looked like a hard iOS limitation. A crackle looked like a client buffer problem. A silent Space looked like the codec fix regressing. In every case, the fix was to keep pushing past the first plausible explanation and check the layer underneath — the actual decoder constraint instead of "Opus doesn't work," the actual relay behavior instead of "the client is buffering wrong," the actual session lifecycle instead of "the audio pipeline broke again."

Spaces audio is in solid shape now: full native decode on iOS with no bandwidth compromise, a clean relay path with no dropped groups, and infrastructure that heals itself within a minute if the bridge process disappears out from under it. All of it shipped through the normal rails — regression tests proven red against the reintroduced bugs first, a full test suite green against real Postgres, and migrations applied through an audited deploy gate.

If you're building anything on live audio over QUIC, we hope this saves you a step or two.