I have a confession that sounds like a brag, which is exactly the problem. My home rack runs a high-availability Kubernetes control plane — three etcd members behind a floating VIP that fails over on its own — and it is guarding a personal blog nobody depends on. Eight inches away, in the same rack, a database sat for months on the password changeme, and a stack of backups I had never once restored. I spent my evenings on the impressive thing and skipped the boring thing that would actually save me. So did GitLab, at a scale that cost them 300 gigabytes. This is a post about telling the difference between building a curriculum and building a hiding place.
I Run a Three-Member etcd Cluster for Services Nobody Depends On
Start with the specifics, because they're the indictment. My home rack runs a high-availability Kubernetes control plane: three etcd members spread across two Proxmox nodes, behind a floating VIP that fails over via etcd leader election the moment a machine drops. It is genuinely nice engineering. Quorum holds, certs rotate, the API endpoint survives a node reboot without anyone noticing.
Here is what all that protects: a personal blog and a handful of hobby apps that could vanish for an afternoon and inconvenience exactly one person — me. I built three-member consensus so that nobody's nothing stays up.
Eight inches away, in the same rack, on the same weekend I was tuning etcd fsync behavior, sat a database still carrying the default changeme password its compose file shipped with — and another service still on the stock secrets I'd flagged to rotate and kept not touching. And a stack of Longhorn backups shipping to object storage every night at 3 AM — backups I configured, monitored, felt good about, and never once restored to verify. Tested failover for a workload nobody depends on. Untested backup for the data I'd actually cry over.
That's the whole post, so I'll say it plainly: homelab complexity is usually avoidance wearing ambition's clothes. You reach for the sophisticated build — HA etcd, a service mesh, a five-tool observability stack — because it's more fun and more flattering than the boring thing that actually saves you when a disk dies. The three-member cluster isn't the achievement. It's the tell — the impressive project you did instead of the ten-minute one you didn't.
The sophistication isn't the sin. The sophistication standing in for the restore script you still haven't written is the sin.
I'm going to hold my own rack against far more expensive versions of the same mistake — engineers better than me who lost 300 gigabytes of live production data with five backup systems in place, none of them working; a YouTube channel that lost its array twice. By the end you'll be able to do two things. Tell a curriculum apart from a hiding place — because sometimes the elaborate build genuinely is the point, and I'll defend exactly when. And name the one unglamorous thing worth doing tonight, before you add another node to the cluster.
GitLab Had Five Backups. It Had Zero Restores.
Let me be precise about what this section is not. It is not a story about incompetent people. The engineers at GitLab in 2017 were very good — good enough to run one of the largest Rails applications on the planet, good enough to have built five separate backup and replication mechanisms, good enough to live-stream their own disaster recovery on YouTube while it was still on fire. Hold that in your head, because it's the whole point. This is not what happens to sloppy operators. It's what happens to skilled ones.
On the night of January 31, 2017, GitLab.com was getting hammered. A spam spike drove database load through the roof; a cleanup job on a flagged (troll) account swept up legitimate data with it; between the two, PostgreSQL replication fell behind and broke. An engineer — the post-mortem calls him team-member-1 — set out to fix it the normal way: re-sync the secondary from scratch with pg_basebackup. It kept refusing, because the secondary's data directory wasn't empty. It was late. He was tired. So he cleared the directory by hand.
In GitLab's own words:
Trying to restore the replication process, an engineer proceeds to wipe the PostgreSQL database directory, errantly thinking they were doing so on the secondary. Unfortunately this process was executed on the primary instead.
He hit Ctrl-C "a second or two after noticing their mistake." By then around 300 GB of live production data was gone. By The Register's account, just 4.5 GB remained by the time the command was cancelled. One keystroke, wrong host, wrong hour — the kind of mistake every one of us is exactly one bad night away from making. That part isn't interesting. What's interesting is what happened next, when they went looking for a backup.
Five places to look. Watch them fall.
Net 1 pg_dump → S3 Silently aborting for MONTHS. The workers ran
Postgres 9.2 binaries against a 9.6 database;
a version mismatch makes pg_dump error out
instead of dumping. "No SQL dumps were made."
Net 2 S3 bucket Empty. Even the failing dumps had nowhere to land.
"Our backups to S3 apparently don't work either:
the bucket is empty."
Net 3 Azure snapshots Enabled on the NFS tier. Never enabled on the
database tier — "we assumed that our other backup
procedures were sufficient enough."
Net 4 LVM snapshots Not a DR system. Taken by hand, roughly once a day,
for staging.
Net 5 Replication Conflated with backup. It was for failover, not DR —
and it was the thing that was already broken.Read that column again. Not one of those is exotic. A version-pinned dump nobody validated. A bucket nobody checked was populated. Snapshots nobody switched on for the tier that mattered. A snapshot nobody scheduled as DR. Replication mistaken for backup — the single most common category error in this entire discipline. Five sophisticated-sounding nets, and every one was a boring fundamental that had been configured and never verified.
And the alarm that should have caught it? The cron jobs did email when the backups failed. Those emails went straight into the void — the receiving mail server rejected them, because DMARC was never set up for the sending domain. The monitoring meant to watch the backups had itself gone unwatched for months. The smoke detector was wired to a phone number that had been disconnected.
Which brings us to the line. Sometime that night, an engineer typed this into the public incident doc, and it has been quoted in every serious conversation about backups since:
So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place.
Five. None.
They copied it over a slow link, restored it, and came back after about eighteen hours — having permanently lost every database write between 17:20 and 00:00 UTC: roughly 5,000 projects, 5,000 comments, and 700 new user accounts, gone. The Git repositories survived, because they live on disk, not in Postgres. Small mercy.
Here is what I want you to take from the flagship exhibit before we get to my own confession. Skill does not inoculate you. These engineers had more backup infrastructure than most companies ever build, and it bought them exactly nothing — because sophistication is not safety, and having five nets is not the same as having tested one. A single boring restore — run once, end to end, "delete the database and bring it back from the copy and see if the site works" — would have exposed all five failures in an afternoon, months before that night. Nobody ever ran it. It wasn't fun, it wasn't flattering, and it wasn't on fire yet.
Five untested backups are worth less than one restore you've actually performed. GitLab paid eighteen hours and 5,000 projects to learn that the count of your safety nets is not the number that matters. The number that matters is how many times you've pulled the ripcord to see if the chute opens.
The sophistication is the tell, not the achievement
Put the two exhibits side by side and stare at what connects them. GitLab, January 2017: five separate backup and replication mechanisms — pg_dump to S3, the S3 bucket itself, Azure disk snapshots, LVM snapshots, streaming replication — every one silently non-functional or never set up in the first place.
And my rack: a three-member etcd quorum stood up across three VMs on two Proxmox hosts, fronted by a floating VIP so the API server survives a node dying — guarding a cluster whose headline tenant is a personal blog. Meanwhile, in the same rack, a database shipped with its password still literally set to changeme, and backups nobody had ever restored to confirm they'd come back. Different scale, identical shape. Elaborate machinery standing at attention in front of a fundamental nobody checked was breathing.
Here's what I want to name precisely, because it's the move everyone dodges. This is not incompetence. GitLab's engineers were first-rate — you don't run one of the largest Rails deployments on earth by accident, and the operator who ran that rm -rf against the wrong host was skilled, working a real incident late at night. And it is not laziness. Building a three-node etcd cluster with a floating control-plane endpoint is genuine work; nobody backs into HA by cutting corners. Reaching for the easy read — they were sloppy, I was cheap — lets you off the hook, because you're neither sloppy nor cheap, so obviously this could never be you.
It's the third thing, the one without a comfortable name: misdirected effort. You have exactly one scarcest resource, and it isn't compute or budget — it's attention. The finite number of evenings you'll actually spend on this. And you spent them on the impressive thing instead of the resilient one.
Redundancy you can demo. Recoverability you can only prove by breaking something on purpose — and nobody wants to break something on purpose at 9pm on a Tuesday.
The reason is human and a little unflattering. A three-node quorum, a service mesh, a five-tool observability stack — Prometheus and Grafana and Loki and Alloy and a SMART-health dashboard — these are fun to build and flattering to show off. There's a topology diagram at the end of them. You can screenshot the VIP failing over and post it. A cron job that restores into a scratch namespace at 3am and exits 0 without waking anyone produces nothing to screenshot. It's the highest-leverage thing in the whole rack, and it's aesthetically worthless. So it doesn't get built, the etcd cluster does, and both operators — GitLab's and mine — land in the same place: maximally sophisticated, minimally survivable.
Which flips the diagnostic. We're trained to read architectural sophistication as competence — look how much this person knows. But sophistication is orthogonal to survival, and once that lands, an impressive homelab stops being reassuring and starts being a question. The more baroque the architecture in front of an untested fundamental, the louder the tell. Every hour of ambition poured into the mesh is an hour not poured into the restore; the elaborateness isn't evidence the fundamentals are handled — it's a map of where the attention went instead. GitLab had five backup systems precisely because backups felt important to them. What they never had was one test.
So retrain the instinct. When someone walks you through their gorgeous topology diagram — or when you catch yourself about to draw your own — don't admire the topology. Ask what it's standing in front of. Ask when the backup was last restored, not whether it exists. The sophistication is not the achievement. It's the thing waving its arms so you don't look at the corner of the rack where the changeme password lives.
I'll concede the one exception the argument has to survive, because it's real and I'm standing in it: HA-for-a-blog is entirely defensible when the cluster is the curriculum — when the whole point is learning etcd quorums and VIP failover and you know that's the trade you're making. I ran three etcd members partly to learn how three etcd members behave, and that's honest. But it only stays honest while you say it out loud. The moment the quorum stops being the thing you're learning and becomes the thing you're hiding behind — the impressive build you point at so you never have to admit the restore script still isn't written — the curriculum defense evaporates, and you're just GitLab with a smaller rack.
A RAID Rebuild Is Not a Restore
The database was a specific fault: replication conflated with backup. If that were the whole story, you could file it under "Postgres is hard" and move on. But run the exact same move through the storage layer and it comes out identical — because the mistake was never about databases. It's about mistaking redundancy for recoverability, and RAID is where that confusion is most seductive. RAID is real, it works, and it protects you against precisely one thing.
One thing. RAID protects against a disk dying. That's the entire warranty, and it's a genuinely useful one. Everything else the array does, it does faithfully and instantly — including all the things you desperately do not want it to. Delete a directory and RAID replicates the deletion across every spindle before your finger leaves Enter. Run the wrong migration and RAID commits the corruption to parity in real time. Let ransomware start encrypting and the array keeps every encrypted block consistent across all its disks, because from RAID's point of view the data simply changed, and preserving the new state is the whole job. As the canonical reference puts it: RAID "replicates everything, instantly, even the stuff you don't want it to."
The reason is mechanical. RAID has no time dimension. It works at the block layer and holds exactly one copy of the present moment, redundantly stored. There is no yesterday in an array. A backup is defined by the one property RAID structurally cannot have: distance from the live data — a prior point in time, on separate media, ideally offsite and immutable. That makes them orthogonal problems, not points on a spectrum. Redundancy keeps you available when a component dies. A backup makes you recoverable after a data-loss event redundancy can't prevent. You need both, and having one tells you nothing about the other.
Redundancy maintains availability during a component failure. Recoverability enables a rollback in time after the event redundancy was never designed to catch. A rebuild restores parity. It does not restore your data.
Which is the whole confession in the section title. Swap a dead drive, let the array rebuild, and it reconstructs redundancy — it does not undo the file you deleted last week, reverse the ransomware, roll back the failed migration, or recover a clean copy when the live filesystem already holds bad data. Rebuilding parity feels like recovery. It recovers nothing you actually lost. That feeling is the trap.
Linus Tech Tips is the homelab-scale sermon on this, delivered twice, because once wasn't enough. January 2016 — the video is literally titled "All of our data is GONE!" — their "Whonnock" server started crashing under load, and mid-way through re-making a stale backup, the RAID card died. The redundancy topology was a marvel of ambition: cheap controllers that could only address eight drives each, so three separate hardware RAID-5 arrays striped together in software. Elaborate, buildable, and completely useless the moment the problem wasn't a single disk. There was no current backup — with one, they'd have swapped the board and card and restored. Instead the only copy of the data lived on the array that had just failed, and they were saved by paying a data-recovery firm to boot a vendor Ubuntu image and pull the bits off by hand. The lesson got a slogan: RAID is redundancy, not a backup.
Then they did it again. The 2022 Petabyte Project — two 45Drives chassis, roughly 60 drives each, ZFS pools built from RAIDZ2 vdevs that each tolerate two disk failures. Serious, expensive, correctly-specced redundancy. But ZFS scrubs were never enabled, so latent bit-rot accumulated silently for about two years. When a drive finally failed and a resilver kicked off, the heavy resilver I/O plus the pre-existing errors on the sibling disks pushed more drives over the edge in the same vdev — cascading past the two-disk tolerance and taking the pool down. The redundancy was flawless on paper. The boring maintenance that keeps redundancy honest — periodic scrubs, watching the degraded state, an actual backup of anything that mattered — was the part that got skipped. A widely-cited post-mortem pinned ~99% of the blame on that missing maintenance, and noted that at 60 drives you should expect about a failure a month — exactly how an unmaintained array degrades to death in two to four years.
That asymmetry is the whole disease. Redundancy is a thing you can point at, so it's the thing that gets the attention. A tested restore is a script you run into the void — no dashboard, no bragging rights — so it's the thing that waits. Both LTT disasters, and GitLab's five dead safety nets before them, are the same sentence: someone spent their scarcest resource on the impressive half of the problem and left the resilient half for a later that arrived as an outage. A RAID rebuild is not a restore. Neither is a three-member etcd quorum. The array coming back green means your redundancy worked. Whether you can get your data back is a completely different question — and the array has never once been asked it.
Meanwhile, the Front Door Still Says "changeme"
I run a single sign-on layer. Authentik, with forward-auth outposts in front of the apps that can't speak OIDC on their own, an LDAP outpost so even my hypervisor logins flow through it. Behind that sits a HashiCorp Vault cluster that unseals itself so I never have to think about it. It is, genuinely, a beautiful identity stack. I am proud of it.
And in the same rack, for far longer than I'll admit, a database was still running on the password changeme — the one its compose file ships with. Because the compose file came that way, and standing up the container was the fun part, and rotating a password nobody was going to see felt like it could wait. The auth stack was the cathedral. The default credential was the unlocked side door I walked past every day on my way in to admire the cathedral.
This is the security-shaped version of the whole argument, and it might be the most embarrassing one, because the attacker economics are not subtle.
Consider what actually owns things at scale. In 2016 the Mirai botnet — the one that hit Brian Krebs's site with a then-record ~620 Gbps, knocked OVH offline with over a terabit of traffic, and took out Dyn's DNS badly enough to drag Twitter, Netflix, and Reddit down with it — did all of that with a scanning dictionary of roughly sixty username/password pairs. Sixty. admin/admin, root/root, root/12345, the default the camera shipped with that nobody changed. It didn't crack anything. It found no zero-day. It knocked on the front door of millions of IoT devices, typed the password printed in the manual, and a meaningful slice of the internet fell over.
The exposure half is just as boring and just as well-measured. Point Shodan at TCP port 3389 — plain Remote Desktop, on the open internet — and you'll find something like 4.8 million endpoints answering, up from roughly 3.5 million before the 2020 shift to remote work bolted another million-plus reachable machines onto the public net. Those boxes are not idle. At the April–May 2020 peak, ESET counted over 100,000 unique machines hit by RDP brute-force attacks every single day, and RDP attack volume grew 768% from the first quarter of 2020 to the last — 29 billion attempts across the year. Not against exotic targets. Against anything with 3389 hanging open, because the scan is free and some fraction is still running Administrator with a guessable password.
Here is the part that should sting: the fix for all of it is unglamorous and costs nothing.
"Don't expose it" has a well-worn answer: put your private things on a private network. Tailscale pulls your trusted devices onto the LAN with zero inbound ports open — outbound-only, happy behind CGNAT, no 3389 for Shodan to find because there is no open port at all. That last part is specifically Tailscale's trick: its coordination and DERP relay layer makes both ends dial out, so no port is ever opened inbound. WireGuard is the raw primitive underneath it, and the same private-network idea, but on its own the server side normally needs one inbound UDP port reachable — or a VPS bounce host when you're behind CGNAT — so "outbound-only, no open port" isn't automatic with plain WireGuard the way it is with Tailscale. Cloudflare Tunnel is a different tool for a different job, and the discipline is knowing that: the tunnel publishes a web app to the public internet on purpose. You reach for it only for the handful of things you actually mean to put in front of strangers — never as a shortcut to reach your own admin panels. The failure isn't the technology. It's using the publish-it-to-the-world tool to expose the Proxmox login you meant to keep private, because sticking it on a public hostname was the quick win.
What I built: SSO + forward-auth + Vault HA on a Raft quorum
What owns a box: an open port + the password from the manual
Where I spent: the cathedral
Where it counts: the side door labeled "changeme"Sophisticated authentication isn't wrong. It's just the wrong thing to have finished first. Every hour I spent making the login flow elegant was an hour of attention — the one resource that doesn't scale — routed away from the two questions that actually decide whether I get owned: is it exposed, and is the password still the default. Mirai never needed my threat model. It needed my camera to still say admin. The impressive build was, once again, standing exactly where the boring one should have been.
When HA-for-a-Blog Is Actually Honest
I've spent this whole piece indicting my own rack. Let me be fair to it — because there's exactly one version of this story where the three etcd members and the floating VIP aren't a confession. They're the point.
If the cluster is the curriculum, HA-for-a-blog is airtight. When the whole reason the box exists is to learn Talos — to watch a 3-member etcd quorum shrug off a node dropping, to make the VIP actually float to a new control-plane member mid-election and prove to yourself you know how — then the complexity isn't overhead. It's the deliverable. Nobody depends on the blog, and that's fine: the blog was never the workload. Your own understanding was, and the etcd cluster is the lab bench you built to produce it. XDA and VirtualizationHowto — the two loudest voices telling you your home server doesn't need Kubernetes — both stop and carve out this exact exception: run it if you're deliberately building the skill. "I run Talos so I could learn Talos" is an honest sentence. Every hour I spent bringing that third etcd member online taught me something the CKA charges money to certify. I'd defend that spend to anyone.
So this isn't a rule against sophistication. Sophistication you can name — "I built HA etcd to learn etcd, and I know I'm trading resilience I don't need for knowledge I want" — is a trade made with open eyes. That's ambition doing its job.
Here's the bright line: it stops being defensible the instant the sophistication stands in for the boring thing you still haven't done. The moment "I run three etcd members" becomes the answer you reach for so you don't have to face that a database is still on the changeme password the compose file shipped with, and your snapshots have never once been restored — the cluster stopped being a curriculum and became a hiding place. Same YAML. Same VIP. Completely different sin. GitLab had five separate backup and replication mechanisms on the books — and the night it mattered, every one turned out to be broken, empty, or never switched on. What they never did was the one dull restore drill that would have surfaced that months earlier. The elaborate machinery was the tell.
Curriculum you can name and defend out loud, to another engineer, without flinching: "this taught me quorum, and I knew the trade." A hiding place you can only rationalize: "well, HA is just good practice" — said fast, to get past the restore script you keep not writing. The test isn't whether the build is complex. GitLab's engineers were excellent; complexity was never their problem — the boring, untested restore path was. The test is whether, when you name what the build was for, the answer is a lesson you sought or an excuse you're reciting. One of those sentences you finish looking someone in the eye. The other you finish looking at the floor. You already know which of your services is which.
Go run a restore tonight
Everything above collapses into one instruction, and it fits on a sticky note: don't test your backup. Test your restore. A backup that exists is a belief. A restore you've walked is a fact. GitLab held five beliefs — pg_dump to S3, uploads to S3, Azure snapshots, LVM snapshots, streaming replication — and on the night of January 31, 2017, an engineer who typed rm -rf against the primary instead of the secondary converted all five to facts in about the time it takes to read this sentence. "Out of 5 backup/replication techniques deployed none are working reliably or set up in the first place." The dumps had been failing silently on a Postgres version mismatch. The S3 bucket was empty. Snapshots were never enabled on the database tier. The alerts that would have caught any of it died at the mail server, because DMARC wasn't set. Every one was a boring, well-documented fundamental nobody had ever exercised end to end.
So exercise one tonight. Take the single most unglamorous fundamental underneath your most impressive build — the one you'd be quietest about at a meetup — and walk it the whole way. Pull the backup. Stand it up somewhere empty: a scratch VM, a throwaway namespace, nothing that can see production. Confirm the actual rows come back, not that the tarball has a plausible size. And time it — because the minutes from "disk is dead" to "data is queryable" are your real recovery objective, and right now you don't know that number. You're guessing, and a guess is just a belief with better posture.
# not: does a backup exist? → yes, look, a file
# but: does a RESTORE work? → fresh box, empty, prove the data is there
pull the latest backup to a machine that has never held this data
stand it up in isolation — new host, new volume, nothing production can touch
verify SELECT count(*) — the rows, the users, the last write, by eye
time the whole thing, start to finish. that number is your RTO. write it down.While you're in the rack with your sleeves rolled up, change the changeme. You know the one. Mine sat in a live stack the entire time I was standing up a three-member etcd quorum and a floating VIP so a blog nobody depends on could survive a node reboot. The changeme and the un-drilled backup lived in the same rack as the HA I was so proud of. That's the tell: the sophistication wasn't the achievement. It was where my attention went instead of the fundamentals.
This is what 3-2-1 has always actually meant. Three copies, two media, one offsite — and the part everyone recites and nobody performs: a restore path you have personally walked, end to end, recently enough to trust. Peter Krogh's rule was never a storage-counting exercise. Three copies are worthless if you've never proven you can climb back out of one. GitLab survived only because an engineer happened to take a manual LVM snapshot on staging about six hours before the outage, while load-testing. Read that back. The thing that saved a company was a coincidence — not a design, a lucky side effect of somebody poking at something unrelated. If your disaster recovery plan is "I'll probably have grabbed the right copy by accident," you don't have a plan. You have a folk tale.
So here's the reframe, and it's meant to sting, because it stung me. The topology diagram was never the achievement. The three etcd members, the service mesh, the five-tool observability stack — that photographs well and demos beautifully and means nothing at 2 a.m. when a disk dies. The achievement is the ugly little script that runs green while you sleep, restores a real database to a real empty box, checks the rows are there, and emails you the elapsed time. It has no dashboard. Nobody will ever be impressed by it. It is the whole game.
Stop building things that look resilient. Go be recoverable.