Build vs. Buy: What This Week's Outages Should Teach You

You can't build your own Cloudflare in any meaningful sense. You can choose not to have the functionality Cloudflare provides because you prioritize the risk of a Cloudflare outage as more important than the benefits Cloudflare gives you, but that probability tree is probably going to land in Cloudflare's favor for 99.99% of businesses.

If you can build a system with redundancy to continue working even if Cloudflare is unavailable then you should, but most years that's going to be a waste of time.

I think you'd be better off spending the time building good relationships with your customers and users so that in the event of an outage that's beyond your control they trust your business and continue to be happy customers when you're back up and running.

> Here’s the thing, if your core business function depends on some capability, you should own it if at all possible.

If I'm building something that allows my customers to do X, then yes I will own the software that allows my customers to do X. Makes sense.

> They’ll craft artisanal monitoring solutions while their actual business logic—the thing customers pay for—runs on someone else’s computer.

So instead I should build an artisanal hosting solution on my own hardware that I purchase and maintain? I could drop proxmox on them and go from there, or K8s, or even just bare metal and systemd scripts.

But my business isn't about any of those things, its about X. How does owning and running my own hardware get me closer to delivering on X?

Instead we need a startup that builds over every cloud provider. Think of a web server for example. AWS has EC2, GCP has its own equivalent and Azure has its own and so on. What if we had a startup that virtualizes a layer on top of these such that we AWS has an outage, you lose 1/3rd of your operating capacity, when Azure has an outage you lose 1/3rd of your operating capacity. In order for you startup s virtual webserver to go down, all of AWS, GCP and Azure wil have to go down simultaneously. Basically build on top of everyone s cloud service into one single unified virtual layer that offers end products to consumers. A 6GB RAM server that the end consumer purchases has 2GB of RAM running on AWS, 2GB on Azure and 2GB on GCP. I am sure we can also strategize something along the same lines for a database server with the added question of the database sharding strategy at play

Redundancy is a proven way to build resilience into your infrastructure. Ownership does not mean you have to build it. OP is correct that you need to understand it all, but that understanding also allows for solid DR plans that use multiple providers for a resilient infrastructure.

There's no easy answer, but you should definitely model what happens when X goes down if you depend on X.

It may even be a rational decision to take the downtime if the cost of avoiding it exceeds the expected cost of an eventual downtime, but that's a business decision that requires some serious thought.

Recoverable master and short dns ttl

What this outage teaches you is that when a third party vendor fails and the internet breaks you can point the finger at them with no issues.

If your shit breaks and everyone else's shit is still working that's a problem.

Yeah but my DevOps only know Aws or Cloudflare UIs and refuse to consider any other platforms. The leadership sees multiple bills as bad. Back to square one? No one will learn anything because people enjoy the pseudo holiday for problems they set themselves up to do nothing about.

Meh. This opinion highlights the fact that availability is the least understood pillar in security. The Right Way to Think About It is having good security analysis and doing proper Risk Management. This means it is their job to do business impact analysis, 3rd party assessments, and run tabletop exercises on all your critical systems to tell you what is rock solid and what is a house of cards.

[dead]