I Dropped the Production Database on a Friday Night
45 comments
·June 18, 2025munchler
orochimaaru
That's the first thing I took away. The author ignores every sane software engineering practice, is saved by pure luck and then dives into what commands not to use in supabase. Why do this? Why not spend a week or two before you launch to setup a decent ci/cd pipeline? That's the real lesson here.
000ooo000
I cut my dev teeth in a financial institution so I'll concede I'm biased away from risk, but devving directly on the prod DB, not having a local enviroment to test changes against, and worse: literally no backups.. it screams wreckless, stupid, cheap, arrogant, and immature (in the tech sense). Nothing I'd like my name against publicly.
physix
A colleague upgraded the production database for a securities financing settlement system on a Friday evening by accident 20 years ago.
We were devs with root access to production and no network segregation. He wanted to upgrade his dev environment, but chose the wrong resource file.
He was lucky it was a Friday, because it took us the whole weekend working round the clock to get the system and the data to a consistent state by start of trading.
We called him The Dark Destroyer thereafter.
So I would add network segregation to the mix of good ideas for production ops.
null
grepfru_it
I’m building my toy project and I have an LTO drive taking backups every night. Here I am complaining that having 2Tb of backups is too much.
lol good luck op
lovehashbrowns
Right?! This whole post is kinda absurd. It has the feel of a kid putting a fork into an outlet, getting the shock of a lifetime and then going “and thanks to this, everyone in my household now knows not to put a fork into an outlet.” You didn’t have to go through all this to figure out that you need backups. The fluff is the cherry on top
jacobsenscott
Yeah. Imagine everything else that's completely wrong in that app.
physix
I dunno. The effort needed to ensure you have backups is tiny compared to the work done to create the product. And to pull a backup before deleting stuff in production only needs a smidgen of experience.
They were extremely lucky. Imagine what the boss would have said if they hadn't managed to recover the data.
geuis
Owww. The first or second paragraph of this made me cringe
"I had just finished what I thought was a clean migration: moving our entire database from our old setup to PostgreSQL with Supabase" ... on a Friday.
Never do prod deploys on a Friday unless you have at least 2 people available through the weekend to resolve issues.
The rest of this post isn't much better.
And come one. Don't do major changes to a prod db when critical team members have signed off for a weekend or holiday.
I'm actually quite happy OP posted their experiences. But it really needs to be a learning experience. We've all done something like this and I bet a lot of us old timers have posted similar stories.
grepfru_it
No. Never release/upgrade on a Friday. Had too many late night weekends when I should be happily drinking beer. Never release at eod Friday. Never.
oulu2006
This is such a poorly written post, and im sure there are on-going disasters waiting to happen -- I've built 3 startups and sold 2 of them and never ever developed on production. ?? What level of crazy is this?
codesnik
supabase kiinda pushes you in that direction though.
Arainach
>Here's the technical takeaway: Never use CASCADE deletes on critical foreign keys.
The technical takeaway, as others have said, is to do prod deployment during business hours when there are people around to monitor and to help recover if anything goes wrong, and where it will be working hours for quite a while in the future. Fridays are not that.
danesparza
Also: don't brag about doing the opposite of what this guy says.
Arnt
I hope the poster will learn about transactions at some point. Postgres even lets you alter the schema within a transaction.
What I learned, once upon a time, is that with a database, you shouldn't delete data you want to keep. If you want to keep something, you use SQL's fine UPDATE to update it, you don't delete it. Databases work best if you tell them to do what you want them to do, as a single transaction.
XorNot
I mean
UPDATE users SET name='test'
is still effectively a delete...
heroprotagonist
I'm sorry, but there's "move fast and break things" and then there's a group of junior devs not even bothering to google a checklist of development or moving to production best practices.
Your Joe AI customers should be worried. Anyone actually using the RankBid you did a Show HackerNews on 8 months ago should be worried (particularly by the "Secure by design: We partner with Stripe to ensure your data is secure." line.
If you don't want to get toasted by some future failure where you won't be accidentally saved by a vendor, then maybe start learning more on the technical side instead of researching and writing blogspam like "I Read 10 Business Books So You Don't Have To".
This might sound harsh, but it's intended as sound advice that clearly nobody else is giving you.
b0a04gl
this is exactly how you earn your prod stripes. dropped the db on day 3? good. now you’re officially a backend engineer.
no backups? perfect. now you'll never forget to set one up again. friday night? even better. you got the full rite of passage.
people act like this's rare. it’s not. half of us have nuked prod, the other half are lying or haven't been given prod access yet.
you’re fine. just make the checklist longer next time. and maybe alias `drop` to `echo "no"` for a while
bloudermilk
I dropped the production database at the first startup I worked at, three days after we went live. We were scrappy™ and didn’t have backups yet, so we lost all the data permanently. I learned that day that running automated tests on a production database isn’t a good idea!
wombatpm
Here is another one: Don't trust ops when they say they have backups. I asked and was told there are weekly full backups, with daily incrementals. The time came when I needed a production DB restored due to an upgrade bug in our application. That was bad - thank $DEIITY we have backups.
OPS: Huh, it appears we can't find your incremental.
ME: Well just restore the weekly, its only Tuesday.
Two Days later.
OPS:About that backup. Turns out it's a backup of the servers, not the database. We'll have to restore to new VM's in order to get at the data.
ME: How did this happen?
OPS: Well the backups work for MSSQL Server.
ME: This is PostgreSQl.
OPS: Yeah, apparently we started setting that up but never finished.
ME: You realize we have about 20 applications using that database?
OPS: Now we do.
Lesson: Until you personally have seen a successful restore from backup, you do not have backups. You have hopes and prayers that you have backups. I am forever in the Trust but Verify camp.
citizenpaul
If your company is big enough to have dedicated ops then it should be running regular tests on backups. A disaster recovery process if you will.
At some point though its not your problem when the company is big enough. Are you gonna do everyone's job? You tell em what you need in writing and if they drop the ball its their head.
null
ghushn3
> I learned that day that running automated tests on a production database isn’t a good idea!
There's novel lessons to be learned in tech all the time.
This is not one of them.
doubled112
Learn lessons from other people. You can't learn them all yourself.
noman-land
I got deep pangs of pain and anguish for you and everyone involved. These lessons hurt so much to learn the hard way.
lawgimenez
Did I read that correctly? They’re on Supabase’ free plan in production?
We’re just getting started and we’re even in Supabase’ paid plan.
heyarviind2
Your website title is "Profitable Programming" with a blog post "How I Dropped the Production Database on a Friday Night"
Thats not very profitable
booleandilemma
Who is this guy? He seems like a poser. I wouldn't be surprised if these articles are AI-generated.
grepfru_it
To be fair this was the norm 10 years ago. Just seems like he is stuck in the past. Really no excuse to provision an ec2 volume and dump all backups there. I’m not even in prod yet and have full backups to LTO to be ready for launch next month
danesparza
This was never the norm for successful companies. This is only the norm for cowboys who have more pizza than good sense.
Developing directly on the production database with no known backups. Saved from total disaster by pure luck. Then a bunch of happy talk about it being a "small price to pay for the lessons we gained" and how such failures "unleash true creativity". It's amazing what people will self-disclose on the internet.