How much traffic can a 1 vCPU box take?

The question

Sumimi promises to stay inside 256 MB on a 512 MB box and 512 MB on a 1 GB box. A promise like that is only useful if you can turn it into a number of visitors. So: how many requests a second can a one-core VPS take before the promise breaks?

The answer has to be a measurement, because the memory a request costs is not a property of the request. It is a property of the churn — how fast the allocator is asked for memory, and how much of what it asked for it kept.

The instrument

The load generator and the probe live in a separate repository, sumimi-gotest, written in Go with the standard library only, so it runs on a machine that has nothing on it but the server. It is a second implementation of the API on purpose: a probe that shares code with the server cannot disagree with it, and a probe that cannot disagree cannot find anything.

Three rules made the numbers comparable:

• The server reports its own peak. The exact high-water mark comes from /proc/self/status (VmHWM) — the process's own number, not the parent's guess about it. The report says which machine was measured, so a number without a machine is not a number.
• A fresh server for every row. The promise is about the peak, so every point is the peak of that point.
• Two runs each. The number is not one lucky run.

The rule, and then the curve

The first machine gave a clean line: the peak ≈ 68 MB + 7.8 MB per 1 000 requests a second. The engine was only 2–12 MB of it. The rest was the collection path and the allocator.

One machine is an observation, not a law — two points cannot prove a rule, and the two might be a coincidence. So we measured it on three machines with three different CPUs, and the rule held on all of them. At the same rate, they cost the same memory. A stronger CPU did not mean less memory; it meant the same memory reached a higher rate.

Then we went looking for where the memory actually went, and found seven things. The biggest was not where anyone would have guessed:

• The queue moved every hit in it. Growing a VecDeque copies the whole queue, so the longer the queue was, the more each request allocated — and the queue's length is exactly what the traffic rate decides. A push into a queue of 1 000 rows cost 10 331 bytes; into the new one, 1 054.
• There was no batch. The writer only waited when the queue was empty, so one request was usually one DuckDB transaction. Now it waits for a full batch: 1 000 requests are 3 transactions, not 1 000 — and that is where the 17% of extra throughput came from, because the ceiling had been the CPU all along.
• The payload was a tree. The body was parsed into a serde_json::Value tree, copied into a map, and every field copied out of it again: 697 bytes of tree per request, now 49.
• The channel lookup lowercased the referrer once per domain — over a hundred times over the same string. 1 731 bytes per request, now nothing.
• The allocator kept the pages it took instead of giving them back.

The result is the table a deployment is sized from — measured on a 2 vCPU Linux box, the exact high-water mark, a fresh server per row:

One line of the environment file

glibc keeps the pages of a freed large block instead of returning them to the operating system: it expects to be asked for the same size again, which is usually true and is not true here. The peak was therefore a high-water mark that never came back down, even after the burst that caused it was over.

MALLOC_MMAP_THRESHOLD_=65536 tells glibc to map anything over 64 KB instead of taking it from the heap, so the pages go back to the OS as soon as the block is freed. Measured on the same server, at the same rate, changing nothing else:

The cost is a syscall per large block: the pages come back, but the CPU pays for them. Measured on the 2 vCPU box, that is about 5% of the throughput — 19 724 → 18 665 requests a second. It is a knob, not a free win. The Docker image sets it; a bare-metal deployment should set it in the environment file.

On macOS it does nothing at all — that is Apple's allocator, not glibc's — and the server is fine there, it just does not need the help. Which is also why the table only believes Linux: the macOS peak is the kernel's ru_maxrss, which counts the shared pages, and the same binary at 5 000 requests a second measured 104 MB in one round and 133 MB in the next. Linux repeats to ±2 MB.

Does it depend on the machine?

The same sweep, the same release, the same tool, back to back — on a 2 vCPU AMD EPYC box and on a 10-core Apple M5. Neither column has the allocator's line, so the two are the same variant and can be compared with each other:

The memory a request costs is the same on both — 138 MB against 139 MB at 5 000 requests a second, 145 against 147 at 10 000. What differs is how much traffic the box can take. So sizing a box is two questions: how much traffic do you get, and can this machine take it?

Above the small box's ceiling the two columns stop being comparable: it never served 20 000 requests a second, so it did less work, and its peak is lower because it finished less — not because it was more frugal.

The network is not free

Every number above was loopback traffic. The controlled measurement — the same container, the same server, the same rate, the server restarted every time, the generator inside the container and then out on the laptop over the public internet (39 ms of round trip):

The network costs about 20% of the memory. The requests arrive in batches — they wait for each other on the wire — so the server parses a batch at a time and the peak is higher. The tail follows: at 5 000/s over the internet 0.7% of the requests timed out (529 of 75 000), against zero over loopback.

This is why the numbers to run are 15 000/s on the minimum tier and 30 000/s on the recommended one: the memory figures were taken over loopback, so they carry no margin, and the network takes about a fifth of it.

The gate: refusing is a feature

The server admits a fixed rate and turns the rest away with a 503 and a Retry-After. Measured in a container, with the released binary and the minimum tier's 128 MB engine — this one is from v0.4.0, before the seven fixes, so its peaks are the old ones; the point of the table is the refusals:

Offering 30 000/s to a server limited to 20 000/s produces 180 313 refusals — and the memory does not grow. The box is protected. The same thing happens past a machine's ceiling: the EPYC served 19 700/s and refused the rest, and its memory stopped growing rather than climbing.

Two traps

Both of these produced numbers that were wrong, and both had to be found before the rule could be believed.

The container that thought it had 7.7 GB

The server sizes itself from the memory it can see. In a container it read /proc/meminfo — which is the host's memory, not the container's. A container limited to 512 MB believed it had 7.7 GB, sized the engine at 256 MB, set the gate at 40 000/s, and got OOM-killed. Fixed in v0.4.1: the cgroup limit is read too, and the smaller of the two wins.

The peak that was not the server's

ru_maxrss on Linux is not the child's peak: Linux does not reset it on exec, so a child's high-water mark still carries the parent's resident set from the moment of the fork. The load generator was a big process by the third measurement, so its memory was reported as the server's. The same rate measured 418 MB and then 831 MB — while the server's own VmHWM sat at 130–140 MB the whole time.

The rule that came out of it: the server reports its own peak, in /api/heartbeat, and that is the number to believe.

So: how much traffic?

Here is the whole thing, with the caveats attached:

• A 512 MB / 1 vCPU box holds 20 000 requests a second on memory alone — measured, 182 MB of the 256 MB it is promised, 74 MB to spare. Over a real network, run it at 15 000/s.
• A 1 GB / 1 vCPU box has twice the budget, so the ceiling doubles to 40 000/s and the safe number over a real network is 30 000/s.
• The CPU is the limit, not the memory. A 2 vCPU EPYC box served about 19 700 requests a second and refused everything above it; the 10-core M5 served more than 40 000. One core lands in between — and the gate has to be set below the machine's ceiling, not above it.
• The disk fills first. 15 000 page views a second is 1.3 million a day, which is about 50 GB a year of rows. Keep the history you want, let the retention window and sumimi prune do the rest, and keep a backup.

The full method, the four limits and the tables are in the memory and capacity reference; the design behind them is in MEMORY.md in the repository, and the reports themselves are published in benchmarks/ — including the ones that did not come out the way we expected.

This is the third post of three. The first is why this exists; the second is why DuckDB.