# Running out of Disk Space in Production

DevFeed: [Running out of Disk Space in Production](<https://devfeed.tech/articles/running-out-of-disk-space-in-production-27922.md>)

Original publisher: [Read original article](<http://alt-romes.github.io/posts/2026-04-01-running-out-of-disk-space-on-launch.html>)

Published: 2026-04-01T00:00:00Z

Content type: article

Language: en

Sources: [Romes' Musings](<https://devfeed.tech/sources/romes-musings.md>)

Topics: [Nix](<https://devfeed.tech/topics/nix.md>), [Server](<https://devfeed.tech/topics/server.md>), [clickhouse](<https://devfeed.tech/topics/clickhouse.md>), [Grafana](<https://devfeed.tech/topics/grafana.md>), [nginx](<https://devfeed.tech/topics/nginx.md>), [Haskell](<https://devfeed.tech/topics/haskell.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [disk-space](<https://devfeed.tech/tags/disk-space.md>), [logs](<https://devfeed.tech/tags/logs.md>), [production](<https://devfeed.tech/tags/production.md>), [reverse-proxy](<https://devfeed.tech/tags/reverse-proxy.md>), [storage](<https://devfeed.tech/tags/storage.md>)

## AI overview

An incident report about a small NixOS server running out of disk space after a surge of customers downloaded large Kanjideck files. The author investigates full-disk errors, identifies Plausible Analytics and the Nix store as major consumers, and tries clearing logs, the Nix store, and ClickHouse tables; the supplied excerpt ends while space is still being consumed.

## Source excerpt

Last night I put up a simple server which allowed customers to download the digital Kanjideck files. This server is hosted on a small Hetzner machine running NixOS, at 4GB of RAM and 40GB of disk space. One of these downloadable files weights 2.2GB. The matter at hand boils down to a simple Haskell program which serves static files (with some extra steps regarding authorization) plus an nginx reverse proxy which proxies requests to a certain "virtual host" to the Haskell program. Fig 1. Simplified server architecture 1 First, Panic Not even minutes after I announced that the files were finally available, hundreds of customers visited my server all at once. As the logs started flying off of my screen with all the accesses, I started noticing a particularly interesting message, repeated over and over again: Mar 31 20:43:03 mogbit kanjideck-fulfillment[2528300]: user error (Unexpected reply to: MAIL "<...> at kanjideck.com", Expected reply code: 250, Got this instead: 452 "4.3.1 Insufficient system storage\r\n") Oh no. No one's able to access their files and I'm already receiving emails about it. I messaged the users explaining the server was having some issues that I was resolving. Grafana shows 40GB/40GB disk space used up, so does df -h have 100% usage of /dev/sda. I have to clear up space fast. I'm afraid at this point that I'm not even receiving the user complaints anymore since my mail could be getting dropped by lack of space. I rushed to run du -sh on everything I could, as that's as good as I could manage. The two larger culprits were /var/lib's Plausible Analytics, with a 8.5GB (clickhouse) database, and the /nix/store with the full server configuration, installation, and executables, at 15GB. (In hindsight, I should have stopped right here to think carefully about what could possibly be occupying the remaining 20GB. I assumed "the rest of the files", but looking back the "rest of the files" could hardly total 20GB.) Delete everything that I can. First off, t