We load-tested our Go backend to 2,000 concurrent WebSockets. The app wasn't what broke

Engineering at Tula — 31st July 2026

It handles chat (end-to-end encrypted, so the server only ever relays ciphertext), 1-1 voice and video call signaling, payments, and live audio rooms. Boring is a feature — until you have to answer the question every small team eventually faces: how far does this actually go before we need to scale?

We had run load tests before, but they all ended the same way: the load generator died before the server did. A 1-vCPU orchestration box OOM-crashed at 200 virtual users. A laptop on consumer WiFi drowned in its own bufferbloat. Our honest conclusion for months was "the ceiling is somewhere above 200 concurrent calls, and we can't see it."

This month we finally got a proper load box — 12 dedicated EPYC cores on a wired datacenter network — and went looking for the wall. We found it. It wasn't where we expected.

The setup

Three ingredients, all of which we'd recommend:

Synthetic users seeded straight into the database. Our seeder writes users, devices, and sessions directly and mints real JWTs with the server's signing key — no OTP flows to script, so seeding 5,000 users with 2,500 direct conversations takes 46 seconds. Load-test users live in a dedicated phone-number range so cleanup is one SQL predicate.

k6 driving the real protocol. Each virtual user is one call pair: two WebSocket connections driving the full lifecycle — invite → ringing → accept → ICE relay → hang up, fifteen calls per session, plus interleaved REST. For chat we wrote a second scenario: signal-shaped ciphertext blobs, send → server fan-out → peer delivery → delivery ack, at one message per second per pair.

A dumb 2-second sampler on the server. A shell loop logging CPU busy, CPU steal, memory, Postgres connection count, and the latency of a real authenticated API call via loopback. Client-side metrics from a load generator 170ms away tell you what users feel; the on-box sampler tells you what the server is actually doing. You need both, and they disagree in instructive ways.

Finding #1: the front door was the ceiling

At around 1,300 concurrent WebSockets, connections started getting refused. The Go server's error rate was zero. CPU had headroom. Postgres didn't blink.

It was nginx — specifically, two Ubuntu defaults nobody ever thinks about:

worker_connections 768;        # nginx default
Max open files     1024        # per-worker soft limit

A proxied WebSocket costs nginx two file descriptors (one to the client, one to the upstream). Four workers × ~half of 1024 FDs each ≈ a hard wall around 1,500–2,000 sockets — reached while the application behind it was at 60% CPU with zero errors. The error log had 2,854 lines of worker_connections are not enough to confirm it.

The fix is two lines and a graceful reload:

worker_rlimit_nofile 65536;
worker_connections 8192;

After the change: 2,000 concurrent WebSockets, zero connection failures, protocol checks 100%.

The lesson generalizes: your capacity ceiling is the minimum across every layer, and the layers you didn't write are the ones you forget to audit. We had spent our tuning attention on the Go server and Postgres. The distro-default reverse proxy in front of them was capped an order of magnitude lower than either.

Finding #2: on a shared VPS, "CPU usage" is a lie — watch steal

Every load stage told the same story: CPU busy would climb to about 60–65% and plateau — while latency kept degrading. The missing cycles were CPU steal: the hypervisor giving our vCPUs' time to somebody else's VM.

Averaged over our test stages, steal consumed 14–20% of all the CPU time we tried to use. In the worst 2-second samples it took half to two-thirds of it — moments where the box wanted CPU and got busy 27% + steal 63%. Those bursts are invisible in average CPU graphs, but they map one-to-one onto our p95/p99 latency tails: the medians stayed fine while the tails blew out 5–10×.

Two operational consequences:

1. Alert on busy + steal, not busy. Our scale-out trigger is now (busy + steal) > 70% sustained — by the time plain busy looks scary on a shared VPS, your users have been living in the tails for a while.
2. Steal rises with your own load. The hypervisor throttles you hardest exactly when you need the headroom. Sizing decisions made from idle-time steal numbers will be wrong.

The numbers

Call signaling, stepped holds (server-side, 4-core VPS; client latencies include ~170ms WAN RTT from the EU load box):

Concurrent call pairs	CPU busy+steal (p95)	API probe p50 / p95	Errors
600 (1,200 sockets)	80%	33ms / 485ms	0
800 (1,600 sockets)	85%	51ms / 330ms	0
1,000 (2,000 sockets)	83%	70ms / 625ms	0 non-200; 6.5% of invites missed a 15s deadline

Chat throughput (one E2EE message per pair per second, peer-acknowledged):

Sustained rate	CPU busy+steal (p95)	Delivery p50 / p95	Delivered
250 msg/s	41%	177ms / 211ms	100%
500 msg/s	62%	183ms / 289ms	99.99%
1,000 msg/s	86%	508ms / 2.2s	99.61%

At 250–500 messages per second, delivery latency is essentially network RTT — the server adds 10–100ms. Sustained 500 msg/s is roughly what a heavy-messaging user base of 15–25k daily actives generates. On a 4-core shared VPS, with the database on the same box. Boring architecture goes further than people think.

Postgres deserves a special mention: through every stage, at every knee, the connection pool sat pinned at its cap doing its job — zero errors, zero measurable degradation attributable to the database. The conventional wisdom that "the DB is always the bottleneck" did not survive contact with measurement.

Finding #3: at high concurrency, your harness lies to you

Two artifacts nearly sent us down wrong roads, and both are worth knowing about:

The redial race. Our k6 scenario hangs up and redials the same conversation 250ms later — fifteen times per session. Under load, with 170ms of WAN RTT, the next invite regularly arrived before the server finished tearing down the previous call, and got correctly rejected as "busy." At peak, 93% of all "failed calls" were this — the harness arguing with its own teardown. The server's busy-state invariant held at 100% throughout. If we'd taken k6's raw success rate at face value, we'd have concluded the server collapses at 600 pairs. It doesn't; the harness pattern does. Separate "the server refused correctly" from "the server failed" before believing any load-test failure rate.

The teardown wedge. k6's experimental WebSocket module wedges on teardown with thousands of open sockets — and if you kill it, the end-of-run summary you were waiting for dies with it. We lost a full run's client metrics learning this. The fix: always stream metrics continuously (--out csv=...) so the data survives regardless of how the process ends.

What changed because of this test

Half a day of testing produced four durable outcomes:

1. A config fix in production (the nginx limits), removing a silent 10× front-door cap — now baked into our provisioning script so a rebuild can't regress it.
2. A real scale trigger: a paging alert on sustained busy+steal >70%, wired into the same Prometheus → Alertmanager → Slack rail as everything else. No more guessing when to buy the bigger box.
3. Hard capacity numbers where we previously had "somewhere above 200": comfortable at 600 concurrent call pairs and 500 chat messages/second; knees at 800–1,000 pairs and just under 1,000 msg/s.
4. A reusable harness — the chat scenario, the seeder, and the sampler are all in the repo, so the next test is an afternoon, not a project.

The meta-lesson: the ceiling you assume is almost never the ceiling you have. Ours was two default config lines in a reverse proxy, hiding behind a hypervisor that skims a fifth of our CPU. You don't find these in code review. You find them by aiming real load at production-shaped infrastructure and watching every layer at once.

---

Tula is building payments, encrypted messaging, and live audio for African markets — on infrastructure priced for them. If this kind of engineering interests you, talk to us.