Kubernetes Is Your Private Cloud
37 comments
·November 12, 2025nostrebored
bakies
You have to pick your battles. Most of this stuff isn't necessary to babysit until you're scaling your app tremendously. And by the time you're doing that I'm sure you've got the people to do these things.
pyrale
All things people used to own 10 years ago. It’s not like the people doing that stuff have vanished.
Cloud’s big promise was speed to market and price, and let’s be honest, price is no longer there compared to a decent operation.
The one thing where clouds remain kings is speed for small teams. Any large enough company should probably Ask themselves whether running their own operation using ias would be a better choice.
bushbaba
My company is on prem, spending north of 1 billion per year. Cloud is actually cheaper when considering total cost of ownership. Thats salaries, opex, capex costs. Worse, our speed to delivery is generally worse.
Because on prem is inelastic, we are at sub 10% peak utilization of compute resources. If we added in the likely higher cloud utilization rate we are talking of 30%+ savings from on prem.
ecshafer
Peak Utilization is a tough one for on prem and is a decent argument for cloud. I was at a company that was also at <10% peak utilization most of the time. It was finance, so it was mostly doing nothing, except for the couple days a year where we shot up 10000x, so we had to build for that case. So yeah losing the data centers, and cloud was a cost savings.
bakies
> we are at sub 10% peak utilization of compute resources
so... you bought way too much hardware?
almosthere
It's crazy to me how many people deploy unmaintainable spaghetti mess in all other environments I've been in. "koober" environments are the most organized.
mikeocool
And particularly the upgrades every 3 months. Not just your nodes and masters, but every operator you use, and your manifests each time they deprecate a manifest beta version.
LeSaucy
Ive found nomad to be a much simpler replacement for smaller scale deployments.
dilyevsky
It’s a well known thing that if you run on ec2 they handle all those things for you (especially the security part)
Glyptodon
IMO an article like this shouldn't just make the claim - it should show how to do it at the home lab level.
barbazoo
> This autonomy is a superpower for small teams. We detailed the financial side of this journey in How moving from AWS to Bare-Metal saved us $230,000 /yr. The cultural unlock has been even bigger.
This doesn't seem to be aimed at homelab but small teams.
cbsmith
Also, isn't this the promise that k8s had from the beginning... that it would be the one cloud abstraction to rule them all?
bakies
Pretty much just install talos and you're done. Deploy the services you need after that.
pavel_lishin
Then install the rest of the owl.
bakies
I mean yeah, unless you want a raven, or a hawk. Kubernetes is bare minimum included out of the box. It's very easy to add more services though.
zer00eyz
> it should show how to do it at the home lab level
I dont need to autoscale my home lab...
I want a better UI/DX/Interface than Kubernetes...
I need to be able to do things "by hand" as well as "automated" at home...
There is a reason that I use Proxmox at home. Because it is a joy to work with for the simple needs of my home lab.
TacticalCoder
[dead]
throwawaypath
Managed Ceph in the past. I cannot comprehend someone putting up with the headache that is Ceph in their home lab. To each their own!
dilyevsky
For small setups it’s honestly fine with rook. For large ones yeah better dust off your ceph phd
thyristan
I've used Ceph together with Proxmox VE excessively. No problems whatsoever.
And in related news, Proxmox VE is often probably a more sensible thing to use for a private cloud environment, because it is far more flexible and easier to use than Kubernetes.
bakies
as much as i'm glazing k8s in this thread I haven't managed to get ceph working. I wish it too since I dont want to use minio anymore.
Longhorn just kinda worked out of the box though with a couple kernel/system settings. No s3 api though.
But this isn't k8s fault out all.
esafak
As long as you have someone to babysit your cluster.
nimbius
Kubernetes is powerful, yes. it is also a feckless rats nest of bolt-ons and ride-alongs. its sharepoint levels of byzantine tuning so complex that, like sharepoint, it comes with its own bespoke administrators that often have little or no knowledge of basic networking or operating systems --only kubernetes--.
- Upgrading a kubernetes cluster may as well be an olympic sport. its so draconian most best practice documentation insists you build a second cluster for AB deployment.
- load balancers come in half a dozen flavours, with the default options bolted at the hip to the cloud cartel. MetalLB is an option, but your admin doesnt understand subnets let alone BGP.
- It is infested with the cult of immutability. pod not working? destroy it. network traffic acting up? destroy the node. container not working? time to destroy it. cluster down? rebuilt the entire thing. At no point does the "devops practitioner" stop to consider why or how a thing of kubernetes has betrayed them. it is assumed you have a football field of fresh bare metal to reinitialize everything onto at a moments notice, failure modes be damned.
what your company likely needs is some implementation of libvirtd or proxmox. run your workloads on rootless podman or (god forbid) deploy to a single VM.
dilyevsky
> MetalLB is an option, but your admin doesnt understand subnets let alone BGP
Maybe get someone competent then? Why are you tasking running onprem setup someone who doesn’t understand basic networking?
bakies
I dont have any of this experience. I only have to change the version number and the upgrades roll themselves out.
MetalLB is good yes, and admins should have IP knowledge. I ask this in interview questions.
Yes, sheep not pets is the term here. Self healing is wonderful. There's plenty to dig into if you run into the same problem repeatedly. Being able to yank a node out that's misbehaving is very nice from a maintenance pov.
Talos on bare metal to get kubernetes features is pretty good. That's what my homelab is. I hated managing VMs before that.
otabdeveloper4
Nix manages to be immutable without restarting everything from scratch.
The complaint isn't immutability, the complaint is that k8s does immutability is a broken, way too granular fashion.
bakies
I'm not really clear on the complaint. Is it immutability or not? I'm not saying delete the cluster and start over, I'm saying i can yank a node or destroy a container without (much of) a consequence. Talos is immutable similarly to nix afaik
themgt
It is infested with the cult of immutability
Immutability is like violence: if it doesn't solve your problem, you aren't using enough of it.
ForHackernews
Can't wait for k8s hype to go the way of microservices.
zug_zug
I don't think kubernetes is inherently bad... it's just a tool that engineers are about 10x as likely to use as a footgun than as a nailgun.
rdtsc
Now you have two problems: kubernetes and your private cloud. The second being that you decided you needed "cloud" to start with.
throwaway838112
You do not need kubernetes
“Everything You Expect from a Cloud, Running on Your Terms“*
Except you own ops, management, extension, interoperability, access, security, scalability, redundancy… words cannot express how ridiculous all of the koober propaganda is