Of course, I use AI for development today. In my opinion, anyone who still isn’t using it in 2026 is completely crazy. I hardly code manually anymore because AI has sped up my development by an estimated 80% — and that’s a figure you simply can’t ignore. So I definitely recommend using AI.
But something else has really amused me lately. For a long time, I kept coming across those infamous nightmare stories on Reddit — stories where AI deleted someone’s entire production database in a matter of seconds. I always took them as a warning, but at the same time, I told myself it couldn’t happen to me. After all, I exclusively use flagship models, and I keep everything properly separated. And nothing happened. Months, more than a year, nothing.
Until now.
Serious message for you — I’m stopping the tests
I was recently reprogramming my blog codedtrip.com — I’m preparing a members’ section, comments, a few new features, and a slightly better experience for readers. AI was quietly doing something in the background when, out of nowhere — and I mean completely out of nowhere — a message popped up that left me staring open-mouthed for a moment. It amused me and scared me a little at the same time:
“Serious message for you — I’m stopping the tests. The dev DB was emptied during testing.”
So what actually happened? During the first attempts to run the tests, an unfortunate combination of a misconfigured setup and an automated mechanism that resets the database before testing occurred. Instead of resetting the test database, it ran against the dev database — and within a single second, the entire database was empty.
All application data disappeared. Only the system tables remained, and it’s hard to reconstruct anything meaningful from those.
Fortunately, it was only the dev database, so to be honest, it didn’t bother me much. I restored it, reran the migrations, seeded the data, and carried on. No problem, no panic. An interesting experience to add to the collection.
I blacked out the DB table names, etc., for security reasons.
But production databases are no joke
However, I can very vividly imagine developers who are vibe coding, have no backups, are connected directly to the production database, and get hit by a disaster like this for real. If they don’t have backups, they’re simply screwed.
And does it happen? There’s no doubt about it. Probably the best-known case in recent history happened at GitLab in January 2017. For those unfamiliar with the incident: one of the engineers was trying to resolve replication lag between the primary and replica databases and, as part of the fix, wanted to clear the data directory on the replica. But he was logged into the primary. By the time he managed to stop it, only around 4.5 GB remained out of 300 GB of live data. GitLab lost six hours of data — issues, merge requests, comments, users. And what was even worse: not one of the five deployed backup systems was working properly. The only usable backup was one that the same engineer had made manually by pure chance six hours earlier while debugging something else. Ironically, his own paranoia saved him from his own mistake. Recovery took more than eighteen hours, and GitLab livestreamed it on YouTube — to this day, it remains one of the most instructive postmortem stories in the industry.
And unfortunately, it has happened to me too
Something similar happened to me around two years ago, although fortunately it had nothing to do with AI.
I was making a quick change in a database client for the Editee project. I always have three databases open beneath one another — production, staging, and dev. Back then, the setup was simpler than it is today; now our databases are split up in various ways by service. Because of an issue with the payment integration, I needed to quickly clear one table, so I issued a command to delete it.
But in my haste, I didn’t notice that I had clicked one tab over in the client — onto the production database.
So I deleted the table in production.
Fortunately, we have very robust backups, so we didn’t lose any data, but I had to restore the production database — and because we have a truly large volume of data, the outage lasted almost an hour. It isn’t a one-click process. But it was a very interesting and necessary experience that reinforced one rule even more strongly for me: back up, back up, back up. And then back up once more.
Water in the data center, or why I’m paranoid about backups
But this isn’t what caused my backup paranoia. It started much earlier — and much more painfully.
I think it was in 2012, so nearly 14+ years ago now. At the time, we had servers in Casablanca, a data center in Prague, and in its Big Blue One cloud hosting service, which constantly boasted about having fully mirrored redundancy — replicated one-to-one to a second data center in Ostrava, so if one data center went down, the other would immediately take over. It sounded wonderful.
Except there was a gym above the data center (yes, a gym). And a water pipe burst in that gym. Water seeped through the ceiling into the data center, flooded the servers — and they simply stopped working. Recovery took them almost 14 days, and in the end, they didn’t recover our data at all. We lost everything that had been stored on those servers.
Fortunately, I had made a large manual backup two days before the incident (back then, I made them regularly, more out of intuition than as part of any formal or consistent process). I spent the entire night restoring everything and migrating it elsewhere — I think that was when we moved to DigitalOcean, where nothing similar has happened since and which has worked great.
It was a powerful lesson for me. Never fully trust a provider’s promise of replicated redundancy until you have verified it yourself.
When an entire data center burns down — the 2021 OVH case
If you think water in a data center is extreme, probably the biggest warning in recent years came on March 10, 2021. That was when the SBG2 data center in Strasbourg, operated by the French company OVHcloud, one of Europe’s largest cloud providers, completely burned down. The 500 m² SBG2 building was completely destroyed, and the neighboring SBG1 building was seriously damaged. Two recently repaired backup UPS units that overheated were identified as the probable cause.
But what makes the story even more interesting is that a whole range of customers lost their data permanently — including those who had paid extra for backups. A French court later ruled in several cases that OVHcloud had stored customer backups in the same data center as the live data, so when production burned down, the “backup” burned with it. For one customer, the outcome was even more absurd — OVH claimed it had restored the data from the customer’s backup server, but allegedly turned the server on without warning, and an automated script deleted the customer’s data as “old.” The customer then received an empty server back.
The popular game Rust lost its entire European game server infrastructure at the time, with no possibility of recovery. In total, more than 130 companies were reportedly affected by the fire and subsequently joined together in a class-action lawsuit.
The lesson? A backup stored in the same data center as production isn’t a backup. It’s a copy.
How I handle backups
Because of all these experiences, I’ve developed backup paranoia that borders on obsession. At Editee, we have:
-
Hourly backups in six different locations across multiple continents.
-
Large full backups once a week with long retention periods.
-
Second-by-second point-in-time backups for production databases, with retention of up to one week.
-
I also back up GitHub — yes, even that.
-
And manual checkpoints before every major change.
So our backups are truly large in volume and cost some money, but when something goes wrong (and sooner or later, something always does), it’s an investment that pays for itself with a single recovery.
In conclusion
AI is an incredible tool, and it has sped up my development so much that I can no longer imagine working without it. But at the same time, it’s another player at the keyboard who can make a mistake. Just as you can make one yourself through a database client, just as a cloud provider can make one, and just as one can be caused by a burst water pipe above a data center or an overheated UPS unit.
For me personally, this merely confirmed the rule I’ve followed for more than ten years:
Back up everything. Then back it up once more. And keep a copy of your backups somewhere completely different.
Fortunately, only my dev database was emptied. But it could have ended much worse — and that’s exactly why I have 20 layers of overlapping backups in every possible location. Because you never know when you’ll need them. And believe me, that day will come; it isn’t a question of if, only when.
