I run vtimers.com, a small set of free timers. No login. Nothing worth stealing in a database. Just pages people leave open for pomodoro, tabata, emom and the rest.
On August 16 the box felt wrong. Load was high, the app hung, and it looked like crypto miners. The site was not fully down, which made it worse. The proxy still answered and bounced http to https like normal. The cert was still valid. But the app never returned a body. It sat and timed out. I tried to ssh in and my key was rejected. I tried the other logins I had. Nothing.
When your key stops working and the machine still answers on the web, you stop thinking about debugging. You start thinking about containment.
The good news came fast. The repo was clean. All the posts lived in git, so there was nothing to ransom and nothing to lose by wiping. I had just finished a security and speed pass with real headers and zero audit warnings. The code was the least of my problems. The running machine was the problem.
I believe they came in through a bug in the older Next.js release the live image was on. I was months behind on that image and the app port was exposed more widely than it needed to be. I never got shell access back to prove the exact chain, because the keys were already replaced. I did not want to keep a compromised box alive just to satisfy my curiosity.
So I treated the box as lost.
I built a fresh deploy package from the clean repo. The app now listens only on loopback, so nothing talks to it except the local proxy. The container runs with least privilege and a real health check. The proxy sends strict headers. Then I wrote two scripts I should have written on day one. One hardens a fresh machine with a tight firewall, key-only login and automatic security updates. The other pulls the repo, rebuilds, and verifies the routes.
I also added smoke tests. Nine checks, zero dependencies. Boot the production server, hit every route, confirm the markers and headers. They caught a real slug bug on the first run.
The part I keep thinking about is how normal the failure looked from the outside. DNS was fine. TLS was fine. The proxy was fine. Only the app was dead and my keys were dead with it. If I had relied on uptime alone, I might have waited another day.
What I do different now is boring on purpose. Bind to loopback. Least privilege by default. Key-only login. Auto updates. Keep every post in git so a wipe costs me an hour, not a year. Keep the repo bootable so a fresh machine plus two scripts equals a site.
A couple months later I rebuilt clean on a fresh box and put vtimers.com back up from git. Same repo, two scripts, and it just worked. The old machine stayed off.
Miners taught me more in one afternoon than a year of quiet uptime did. If you cannot rebuild it from git in one go, you do not really own it.