Sumimi 数咪咪

7 min read

Leaving Umami: one small VPS, two problems

This is the story behind Sumimi. It starts with a box I already pay for, a disk that kept filling up, and a dashboard that kept stuttering.

The box

I have a small VPS: 1 GB of RAM, one core, a 20 GB disk. For a long time it ran Umami for the traffic statistics of several small sites of mine — a personal blog, a static site, a couple of side projects. It is not a big machine and it was never meant to be one. It is the box I already pay for, and the sites are small.

Umami is a good piece of software. I want to be clear about that, because what follows is not an argument against it. It is an argument about what a different storage engine does to the same workload.

Problem one: the disk

The disk kept filling up. Not slowly, and not at a rate I could plan around: every few months I would notice the box was unhappy, look at what had grown, and prune by hand. Umami stores every page view in PostgreSQL — one row per view, in a table with about thirty columns — and on top of that it maintains thirty indexes. I measured that later: on a real dataset, 68% of the space was indexes.

And here is the thing that made it impossible to plan around: analytics data is append-only. Almost everything that arrives is new. Almost nothing is ever updated or deleted. PostgreSQL does not hand space back to the filesystem — you need VACUUM FULL — and Umami has no automatic retention window. So the file only ever grows, and the only question was when I would notice.

Problem two: the stutter

The second problem was the dashboard. Every time I opened a panel there was a stutter — a beat of delay before the numbers appeared. After six months of history, the stutter had become a wait: the page was aggregating half a year of rows on every click, over a network connection to a database that was itself busy.

The two problems looked unrelated. They were the same problem seen from two sides: a storage model that is a bad fit for data that only ever grows.

The realization

A row store keeps a whole record together because records get read whole. Analytics records do not get read whole — a panel reads one column across millions of rows and ignores the rest. And append-only data is exactly the data where the repetition is high: the same referrers, the same browsers, the same URLs, the same titles. Columnar storage reads the column, and lets the engine find the repetition inside it.

So the question stopped being "which analytics tool should I use" and became "which storage engine". If the same 20 GB disk could hold seven years of history instead of fifteen months, then the pruning chore would disappear — and something else would appear in its place: a history long enough to be worth reading.

What the number actually is

Fifteen months became seven years on my own box. The measured factor is larger than that — 50–85× less space, depending on how repetitive your URLs and titles are — and my own sites sit at the conservative end of that range. The point is not the multiplier; the point is that the disk stopped being the thing I think about. The next post has the measurement.

The decision

DuckDB: a columnar engine that is compiled into the binary. That last part is what made it fit the box. There is no database server to install, secure, back up, upgrade or run out of memory — the whole engine is a file on disk, and the file is the database.

Everything else followed from that choice:

  • One binary, one file. The release archive holds an executable and a directory. There is no queue to operate, no cache to tune, no object storage, no container orchestration. Copy one file to the box, run one systemd unit, and it is done.
  • The panels became fast. The same queries that took a quarter of a second now take milliseconds, because the engine reads the column it needs and nothing else. The stutter was a storage problem, and it is gone for the same reason the disk problem is.
  • The history became worth keeping. With a retention window and a rolling deletion policy (sumimi prune), old data leaves on its own and the file never grows without bound.
  • Backups became trivial. It is one file. sumimi backup copies it; a nightly cron job and an off-box copy is the whole disaster-recovery story.

What it turned into

A tool that stops being a chore and starts being a status monitor. The last step was the one I did not expect: once the history is cheap enough to keep for years, the analytics stop being a report you look at when something feels wrong and start being a signal you can compare against. Did that deploy actually help? Is the traffic shape healthy? Did that outage show up? A dashboard with six months of history answers those; a dashboard with a fortnight of history does not.

The project is called Sumimi — count, plus a cat's name — and it is Apache-2.0 licensed, and it speaks the Umami v2 HTTP API, so the dashboard and the tracker snippets I already had kept working throughout.

Next: why DuckDB — the storage measurement, and why the answer is not "the compression". Or the documentation, if you would rather just install it.

Sumimi / 数咪咪 is an independent open-source project. It is an open-source alternative to Umami and implements the Umami v2 HTTP API; no Umami assets are used. Licensed under the Apache-2.0 licence. Third-party names are referenced nominatively.