Debugging Live Audio at Scale

That number was scary enough to derail a roadmap conversation — and, as it turned out, almost entirely wrong. This is the story of how we got from that panic number down to a real, measured, production-ready plan: five boxes, not a hundred, with every claim backed by a test rather than a guess.

If you're building anything real-time at scale — live audio, video, multiplayer, anything running over QUIC — the failure modes we hit are probably waiting for you too.

The panic number was built on an assumption, not a measurement

The original math went something like: X% of a CPU core per listener, therefore Y cores needed, therefore Z boxes. It felt rigorous because it had numbers in it. But every number in that chain was inherited from an untuned, default configuration nobody had actually interrogated.

The first rule we relearned the hard way: a scary capacity number is a hypothesis, not a fact, until you've tried to break it on purpose.

Myth #1: "It's too much crypto for one core"

Our relay runs one process per box, encrypting media for every connected listener over QUIC. At ~0.4% of a CPU core per listener at a modest bitrate, the instinctive explanation was cryptographic overhead — AES doesn't usually struggle at these data rates, so something had to be off.

It wasn't crypto. It was packet count. Our encoder was emitting one CMAF fragment every 21 milliseconds — about 47 fragments a second, each one a separate send over the wire, per listener. That's a syscall and framing tax, not a cipher tax. Crypto didn't even show up in a profiler trace once we went looking.

The instinct to reach for "batch the fragments" didn't work, though — that's the first place the story took a turn.

Myth #2: Batching the container doesn't touch the wire

We tried extending the CMAF fragment duration from 21ms to 200ms, expecting a straightforward 5-10x drop in packet rate. Nothing changed. The importer in our media stack demuxes the container down to individual codec frames before it ever reaches the wire — and our codec (AAC) has a fixed 21.3ms frame duration, full stop. Container-level batching was invisible to the actual bottleneck. The real lever was one level deeper: the codec itself.

The real fix: change the codec, not the container

Our relay's native codec is actually Opus, not AAC — AAC was a leftover from an earlier protocol era. Opus natively supports 120ms frames. One encoder flag later (libopus, 48k mono, frame_duration 120, voip), and:

• Packet rate fell 5.4×
• CPU per listener dropped 2.2× (down to 0.145% from 0.32%)
• Bandwidth per listener halved
• Measured latency cost: effectively nothing — invisible under normal player buffering

We didn't ship this on the strength of a benchmark alone. Before it went anywhere near production, we ran a real phone through the new codec path and looked for the actual Android decoder instantiating with the right sample rate, sustained audio output with zero decode errors, for 30+ seconds. Only once we had that concrete evidence — not an assumption that "the stack natively speaks Opus, so it'll probably work" — did the change go live.

The bug that was quietly eating 80% of packets

With the codec fixed, we pushed further — real external listeners, real trans-continental network paths, watching for where things actually broke. Somewhere between 5,500 and 7,000 simulated listeners, things got ugly: per-listener bitrate sagged, listeners started silently dying, and packet drops accelerated in a way that looked like a wall, not a slope.

The instinct was to blame the round-trip time — real listeners on the other side of a continent, with real jitter and packet loss, cost more CPU per connection than a co-located test rig ever would. That's a real effect, and it does cost something. But it wasn't the actual story.

The actual story was an 80× undersized default. Our QUIC library binds exactly one UDP socket, and that socket's receive buffer was sitting at the Linux kernel default — 208KB. Nobody had ever explicitly configured it. Every dropped packet traced back to that one buffer overflowing under load, invisible right up until QUIC's own loss-recovery machinery couldn't mask it anymore.

The fix was one sysctl line:

net.core.rmem_default = 16777216
net.core.wmem_default = 16777216

Before: 1.27 million dropped packets at 4,000 listeners. After: 25,800 — a 50× reduction, from the same box, same load, same codec. The "collapse" we'd been chasing simply moved from ~6,800 listeners to a clean ~8,000, with the actual production ceiling only appearing once the socket itself, not the CPU, ran out of room to keep up.

One socket, one process — and what that means for scaling

QUIC libraries like ours bind one UDP socket per process. That socket has a hard drain-rate ceiling somewhere around 6,500-8,000 listeners, no matter how many idle CPU cores are sitting behind it. This has a very concrete consequence: buying a bigger box with more cores does nothing by itself. A 32-core box running one relay process hits almost exactly the same wall as a 12-core box running one process.

The actual scaling lever is running multiple, independently-socketed relay processes on the same box, cluster-connected so they all serve the same live stream and a router can hand new listeners to whichever process has headroom. We tested this directly rather than assuming it would work:

• Two cluster-connected processes on the same 12-core box that had capped at ~8,600 with one process now ran cleanly to 10,000, collapsing only around 11,000 — and the failure mode itself changed character, from a socket-drain wall to genuine CPU-plus-memory exhaustion.
• Cluster overhead measured at essentially zero — running two coordinated processes cost no more CPU than the same total load split across two isolated ones.

That gave us a real formula, not a guess, for sizing any box:

capacity per box ≈ min(
  cores × ~900 listeners/core            (CPU ceiling, real-world RTT)
  processes × ~6,500 listeners           (per-socket drain ceiling, comfortable)
  RAM ÷ 1.3MB/listener ÷ 2               (memory ceiling, with disconnect headroom)
)

Whichever term is smallest wins — and matching the process count and RAM to the box's core count, rather than defaulting to "2 processes" out of habit, turned out to matter a lot once we started looking at much bigger machines.

The bug we almost shipped: what happens when everyone leaves at once

Every test so far had assumed listeners arrive and depart gracefully. Real networks don't work that way — a mobile carrier hiccup, an app crash, a Wi-Fi handoff can drop thousands of listeners at the exact same instant. We simulated it directly, freezing hundreds of listener processes mid-session to create a genuine network blackhole rather than a clean disconnect.

At healthy CPU load, our relay's own housekeeping — a periodic reaper clearing out dead sessions — won the race comfortably: a brief memory bump, then a clean recovery. But under CPU saturation, that reaper starves, memory balloons far faster than it can be reclaimed, and the Linux kernel eventually kills the process outright, picking whatever it judges the worst offender on the box — which can just as easily be an innocent neighbor process as the actual culprit.

There's no code-level fix available upstream today for this — no memory budget flag, no backpressure on session cleanup. So we built the next best thing: every relay process now runs under a memory-ceilinged systemd unit with automatic restart. When a process hits its ceiling, it's killed in isolation — its siblings on the same box are untouched — and it's back up and serving within seconds. What used to be an unpredictable, box-wide outage is now a bounded, self-healing blip affecting one process at a time.

Choosing infrastructure: the fine print is where the real cost lives

Once the relay itself was solid, the remaining question was where to run it — and this is where marketing language earned the most scrutiny. "Unmetered," "zero-cost egress," and "free" bandwidth claims turned out to mean genuinely different things provider to provider:

• One budget dedicated-server provider's generous-sounding bandwidth allowance turned out to be a hard daily cap — cross it, and you're throttled to a fraction of your contracted speed for the rest of the day, right when a real traffic spike would need it least.
• A major cloud platform's "free" CDN tier evaporates within days under real video traffic volume, and its metered egress beyond that tier is exactly the cost problem you were trying to avoid.
• A cloud provider's "zero-cost egress" turned out to be a fair-share quota that scales with the size of the box — genuinely fine for most workloads, but one we calculated could be exhausted in under 48 hours of a large box running near full capacity, with a real throttle waiting on the other side.
• Only one provider we evaluated had bandwidth pricing with no cap or quota we could find anywhere in its terms — the closest thing to what "unmetered" actually implies.

None of these are objectively bad providers — they're just priced and capped for different assumptions about your traffic. The lesson that stuck: read the actual terms for the specific resource shape you're buying, every time, and do the arithmetic against your real bandwidth needs before trusting the headline.

Where we landed

• Audio (Spaces) stays on our own relay fleet — cheap enough per listener, and the only way to keep the sub-second latency that makes a live conversation feel live.
• Video (Tikos Live) fans out through CDN-backed HLS instead — a few seconds of latency is invisible for a broadcast audience, and CDN delivery costs the same whether ten people are watching or a hundred thousand.
• Steady, predictable load runs on flat-rate dedicated hardware, priced and sized around actual RAM-per-listener needs rather than raw core count.
• Burst, event-shaped load — a viral moment, a national broadcast-scale Space — spins up on hourly-billed cloud instances with confirmed unmetered egress, torn back down to zero the moment the event ends.

The final capacity plan, extrapolated from real measurements rather than a guess: 100,000 concurrent listeners across roughly 9-13 boxes, not the 100 we feared at the outset. The gap between those two numbers wasn't found by throwing more hardware at the problem — it was found by refusing to trust any single claim, ours included, until it survived an actual test.

---

We're building Tula, a super app for Africa combining social video, live audio, encrypted messaging, and payments. This kind of infrastructure work happens in public because we think the reasoning matters as much as the result.