Geico repatriates work from the cloud, continues ambitious infra overhaul

This article read more like advertisement for VP spearheading all of this.

Thank you for posting this - reading this set off a lot of alarm bells, and there's a loud, growing "on prem" marketing movement that is likely to trumpet this as the downfall of "cloud" that I wasn't particularly looking forward to arguing with.

> Kubernetes on top of OpenStack on top of Kubernetes (yes, that's what they are doing).

OpenStack's services are running in Kube? And Kube itself is ran as an OpenStack thing? Why? Why not use the same tooling used to deploy that initial Kube to deploy as many as needed? Still a massive maintenance burden, but you don't need to add OpenStack into the mix.

First you charge them to put a star on their belly, and then you can charge them to take the star off their belly!

All the whey down

Dios mio mayne

The comment about "running legacy applications in the cloud was not any cheaper" stood out to me. Just moving the same legacy design into the cloud is not the optimal way to gain cost and availability improvements.

If you have ever seen a data center from Azure, GCP or AWS, you will realize how difficult it will be for any company to compete in the long run. Those companies develop new generations of data center infrastructure with power efficiency improvements every single year. They negotiate network and power contracts at a scale that exceeds any typical Fortune 500 company. I'm skeptical that running your own data center will end up a cost saver in the long run.

> Any bets on what's going to happen next?

Someone in the c-suite gets a massive bonus before moving to a new company.

according to the blog they started the cloud migration in 2013, there have been lot of improvements/changes to on-prem since then.

If your data center isn't large enough to need at least 5 people full time admins then you should just go cloud. With a part time person you will see downtime when a machine fails. With 1 person that person will sometimes be on vacation when a zero day takes you down. With 2 people 1 will be on vacation when the second gets sick. You end up needing at least 5 people before you have enough people that you have redundancy for humans issues and the ability to train people in whatever is the latest needed.

Of course even in the cloud you still need to apply security patches to everything. However it still saves a lot of issues and thus money in all but the largest setups.

It's not really about cloud vs. on-prem, it's the fact that people cut corners and lack knowledge on-prem, and don't have the budgets to do anything about it.

What you're referring to is mostly about elasticity, and it's true that if you don't need it, it doesn't make sense to pay for it.

But that doesn't mean that on-prem (which almost always turns into a virtual machine shitshow with crappy network design -- which will continue as long as nobody implements things like strong IAM and Security Groups in their on-prem setups) is 'the same' as cloud but just in a physical location you control.

The inverse is also true. If you just run some VMs 'in the cloud', you're doing it wrong. Playing datacenter is just as bad as not moving away from classic virtual machines, cloud or no cloud.

Find me a list of customers on cloud who got hacked, vs folks on-prem. I've got 3k+ customers, I know which one I see 99.99% of the time...

Disclaimer: this is anecdotal so n=1. All opinions are my own. No value judgment one way or another is expressed or implied.

Professional developers these days are primarily concerned with 1) getting their service running 2) as quickly as possible 3) someplace where they have instant access and control of it. Clicking around a cloud console accomplishes all three of these and allows you to write "Delivered the ____ service in 3 months that generates $XX M/year" on a performance review in short order. Having to build, rack, and configure a physical server or deal with "IT" (which has somehow become something separate from software engineering) does not. Because the developers are the ones delivering value they get to decide how it's done. AWS gets it done. A server in a datacenter in Texas that requires an SSH keypair to reach doesn't.

Your average SDE L4 does know or care about init systems or SANs or colos or 802.1q or any of the myriad of things required to run on-prem infra. They write software. Software makes money and so the business makes money - wash, rinse, repeat. Why would you have people on the front lines of your revenue stream worrying about these things when you can have a hyperscaler with a control plane do it for a nominal fee?

Because you're not Amazon you also probably don't have tech as your core competency and don't have the budget to hire people skilled enough to operate an your on-prem setup as well as they operate their cloud.

Because you're not a startup you there is a very good chance that you have a very process-driven (cover your ass), slow-moving culture - this very often translates to an IT department where getting even basic things done (like reserving extra compute or changing a network setting or starting to use a third party software) takes months of waiting or pleading. Maybe you have never encountered this kind of pathological IT department, but they're very common, and it's a major reason executives bought into cloud to begin with. Of course, many companies like Geico seem to have merely replicated their IT pathologies in the cloud, but at least in the cloud you have fewer sources of problems in areas like physical space management, buying/integrating hardware to grow or change your footprint and dealing with all the SKUs and supply chain problems therein, or negotiating on-prem licences.

There are many more moving pieces when operating on-prem: more operations staff across more kinds of roles (yes, you still have eg devops people when using the cloud, but you don't need as many building operations staff (where managing a datacenter is its own speciality), people managing hardware/software vendors and related supply chain issues, people skilled in physical networking, people to plug things in/out and physically operate the machines), managing and acquiring the physical space where your on-prem setup is, buying/accounting for all the different kinds of hardware you need, licensing/using more software with more difficult integration to achieve equivalent functionality to eg EC2, licensing all your 3P software to run on-prem... even if nominally less expensive than the cloud in some cases, there are many more places where things can go wrong. That's not as easy to account for in a direct TCO comparison because it manifests as slowing things down - which does introduce very substantial costs - and distracting management away from other opportunities to grow revenue or improve costs.

Also, cloud downtime is really overstated as a problem in 2024. It makes the news because it has a high blast radius and involves high profile companies, not because it's more common than on-prem. With the exception of AWS us-east1 issues (which can break many AWS products at once across the world), most cloud reliability issues these days are isolated to only a few products and only a few regions. I think a lot of small on-prem companies don't realize that they are not actually more reliable, but just operate at a smaller scale where the probability of downtime causes "lucky streaks" to be more common (ie if you play roulette for three rounds, you're much more likely to have an abnormally high win rate than someone who plays it for three hundred rounds, even though you both have the same odds). Most companies don't have as mature security/risk operations as cloud providers and so face an existential risk/the possibility of huge (months) of downtime in the event of a fire/natural disaster at their dc, cryptolocker attack, janitor unplugging the server that says "do not unplug" - this isn't something people have to worry about with cloud providers to nearly the same extent.

> expecting a significant ramp up of compute demand

Lots of data processing workloads don't need to be run constantly, but do need to be run in a shorter amount of time. Cloud is pretty good for that sort of thing.