Webflow Down for >31 Hours

More likely that their core database hit some scaling limit and fell over. Their status page talks constantly about them working with their "upstream database provider" (presumably AWS) to find a fix.

My guess. They use AWS hosted Postgresql and autovacuuming fell permanently behind without them noticing, and can't keep up with organic growth, and they can't scale vertically because they already maxed that out before. So they have to do crash migrations of data off their core DB which is why it's taking so long.

If so it is probably a good time to apply for an SRE position there unless they really do not get it!

And engineers never are the cause of mistakes? There can't possibly be any data to back up that major outages are more often caused by leadership. I've been in SIEs simply because someone pushed a network outage to a switch network. Statements like these only go to show how much we have to learn, humble ourselves, and stop blaming others all the time.

Prediction: Someone confidently broke something, then confidently 'fixed' it, with the consequence of breaking more things instead. And now they have either been pulled off of the cleanup work or they wish they had been.

I doubt it’s infra failure but software failure. Their bad design has caught up and they can’t toss more hardware for some reason. Most companies have this https://xkcd.com/2347/ in their stack and it’s fallen over.

> 99.99%+ uptime is the standard we need to meet, and lately, we haven’t.

Four nines is not what I would be citing at this point. (That's less than an hour per year, so they burned that for next three decades)

Maybe aim for 99% first.

Otherwise a pretty honest and solid response, kudos for that!

Interesting the phrase "I'm sorry" was in there. Almost feels like someone in the Big Chair taking a bit of responsibility. Cheers to that.

> Change controls are tighter, and we’re investing in long-term performance improvements, especially in the CMS.

This reads as if overall performance was an afterthought and this doesn’t seem practical; it should be a business metric, it is important to the users after all.

Then again, it’s easy to comment like this in hindsight. We’ll see what happens long term.

Decent update. Guess people are really waiting for a fix tho!

Hopefully they are rotating teams not people staying awake for a dangerous amount of time.

Because imagine your local biz can either pay a designer 1k a year or DIY and pay godaddy 200 bucks. Or 30 bucks for Wordpress and 20 hours of fiddling and asking their cousin for help.

Its not great by our standards but I bet many of us drink the house wine not something more sophisticated, right :)

Why? Genuinely asking. Did you mean because there are free alternatives to self-host? I don't think that it would be so easy for someone in the market for a WYSIWYG blog builder to set everything up themselves.

We moved away from webflow because it was slow (got the nickname web-slow internally).

Plus, despite marketing begging for the WYSIWYG interface they actually weren't creative enough to generate new content at a pace that required it.

We massively increased conversion rates by going full native and having 1 Engineer churn out parts kits/kitbash LPs from said kits.

Scale for reference: ~$10M/month

The problem is they get good in that specific disaster. They can only plug a hole in the dike after the hole exists, then they look at the hole and make a plug the exact shape of that hole. The next hole starts the process over for it specifically. Each time. There's no generic plug that can be used each time. So sure, the get very good at making specific plugs. They never get to the point of making a better dike that doesn't spring so many leaks.