A production disk-full crisis, and what I changed after
An EC2 box quietly filled its disk until PM2 couldn't write logs anymore. Here's how I diagnosed it, recovered without extended downtime, and what I put in place so it wouldn't happen again.
The first sign was small: PM2 stopped writing new log lines. Then requests started timing out. The root cause was almost embarrassingly simple once I found it — the EC2 instance's 8GB root volume had filled up, partly from accumulated PM2 logs and partly from desktop packages that had no business being on a headless server in the first place.
Recovering without a full outage
The immediate fix was freeing space: removing unused desktop packages bought enough headroom to get services responding again. But that's a patch, not a fix — an 8GB volume with a production Node.js app, its logs, and MongoDB-adjacent tooling was always going to fill up again. I expanded the EBS volume from 8GB to 20GB, which gave real headroom instead of a temporary reprieve.
Making sure it couldn't happen quietly again
The part that actually matters long-term was configuring PM2's log rotation properly, so logs get truncated and archived instead of growing without bound. Disk space is one of those things that fails silently right up until it fails completely — there's no gradual slowdown to warn you, just a cliff. Log rotation turns an unbounded growth problem into a bounded, predictable one.
- Headless production servers don't need desktop packages — audit what's actually installed
- Disk usage is a silent-failure risk; log rotation should be configured on day one, not after an incident
- A capacity fix (more disk) and a root-cause fix (bounded log growth) are both necessary — one without the other just delays the next incident
It's a small, almost boring incident in hindsight. But it's the kind of thing that's easy to overlook when you're focused on shipping features, and it's exactly the kind of gap that a basic infra checklist would have caught before it became a fire drill.