Replacing Kubernetes with systemd (2024)

I can't speak on it's capabilities, but I feel like I have to ask: for what conceivable reason would you even want that extra error potential with migrations etc?

It means you're forced to make everything always compatible between versions etc.

For a deployment that isn't even making money and is running on a single node droplet with basically no performance... Why?

GP talks about personal websites on 1vCPU, there's no point in zero downtime then. Apples to oranges.

> It burns 450Watts on average

To put that into perspective, that's more than my entire household including my server that has an old GPU in it

Water heating is electric yet we still don't use 450W×year≈4MWh of electricity. In winter we just about reach that as a daily average (as a household) because we need resistive heating to supplement the gas system. Constantly 450W is a huge amount of energy for flipping some toggles at home with voice control and streaming video files

I run a similar number of services on a very different setup. Administratively, it’s not idempotent but Proxmox is a delight to work with. I have 4 nodes, with a 14900K CPU with 24 cores being the workhorse. It runs a Windows server with RDP terminal (so multiple users can get access windows through RDP and literally any device), Jellyfin, several Linux VMs, a pi-hole cluster (3 replicas), just to name a few services. I have vGPU passthrough working (granted, this bit is a little clunky).

It is not as fancy/reliable/reproducible as k3s, but with a bunch of manual backups and a ZFS (or BTRFS) storage cluster (managed by a virtualized TrueNAS instance), you can get away with it. Anytime a disk fails, just replace and resilver it and you’re good. You could configure certain VMs for HA (high availability) where they will be replicated to other nodes that can take over in the event of a failure.

Also I’ve got tailscale and pi-hole running as LXC containers. Tailscale makes the entire setup accessible remotely.

It’s a different paradigm that also just works once it’s setup properly.

I have a question if you don't mind answering. If I understand correctly, Metallb on Layer 2 essentially fills the same role as something like Keepalived would, however without VRRP.

So, can you use it to give your whole cluster _one_ external IP that makes it accessible from the outside, regardless of whether any node is down?

Imo this part is what can be confusing to beginners in self hosted setups. It would be easy and convenient if they could just point DNS records of their domain to a single IP for the cluster and do all the rest from within K3s.

450W is ~£100 monthly. It's a luxury budget to host hobby stuff in a cloud.

How do you deal with persistent volumes for configuration, state, etc? That’s the bit that has kept me away from k3s (I’m running Proxmox and LXC for low overhead but easy state management and backups).

Do you have documentation somewhere, where you can share ?

You should look into fluxcd this stuff makes a lot of stuff even simpler.

"Basement Cloud" sounds like either a dank cannabis strain, or an alternative British rock emo grunge post-hardcore song. As in "My basement cloud runs k420s, dude."

https://www.youtube.com/watch?v=K-HzQEgj-nU

There are 3 components to "the control plane" and realistically only one of them is what you meant by idle. The Node-local kubelet (that reports in the state of affairs and asks if there is any work) is a constantly active thing, as one would expect from such a polling setup. The etcd, or it's replacement, is constantly(?) firing off watch notifications or reconciliation notifications based on the inputs from the aforementioned kubelet updates. Only the actual kube-apiserver is conceptually idle as I'm not aware of any compute that it, itself, does only in response to requests made of it

Put another way, in my experience running clusters, in $(ps auwx) or its $(top) friend always show etcd or sqlite generating all of the "WHAT are you doing?!" and those also represent the actual risk to running kubernetes since the apiserver is mostly stateless[1]

1: but holy cow watch out for mTLS because cert expiry will ruin your day across all of the components

I deployed CNPG (https://cloudnative-pg.io/ ) on my basement k3s cluster, and was very impressed with how easy I could host a PG instance for a service outside the cluster, as well as good practices to host DB clusters inside the cluster.

Oh, and it handles replication, failover, backups, and a litany of other useful features to make running a stateful database, like postgres, work reliably in a cluster.

It’s Kubernetes, out of the box.

A bit of a nitpick. You get monthly credit for 4c/24gb on ARM, no matter the region. So even if you chose your home region poorly, you can run those instances in any region and only be on the hook for the disk cost. I found this all out the hard way, so I'm paying $2/month to oracle for my disks.

I don't know the details but I know I made this mistake and I still have my Free Tier instances hosted in a different region then my home. It's charged me a month of $1 already so I'm pretty sure it's working.

That limitation (spinning up an instance) only exists if you don't put a payment card in. If you put a payment card in, it goes away immediately. You don't have to actually pay anything, you can provision the always free resources, but obviously in this regard you have to ensure that you don't accidentally provision something with cost. I used terraform to make my little kube cluster on there and have not had a cost event at all in over 1.5 years. I think at one point I accidentally provisioned a volume or something and it cost me like one cent.

> no commercial usage

I think that's if you are literally on their free tier, vs. having a billable account which doesn't accumulate enough charges to be billed.

Similar to the sibling comment - you add a credit card and set yourself up to be billed (which removes you from the "free tier"), but you are still granted the resources monthly for free. If you exceed your allocation, they bill the difference.

They will not even bother sending you an email explaining why, and you will not be able to ask it, because the system will just say your password is incorrect when you try to login or reset it.

If you are on free tier, they have nothing to lose, only you, so be particular mindful of making a calendar note for changing your CC before expiration and things like that.

It’s worth paying for another company just for the peace of mind of knowing they will try to persuade you to pay before deleting your data.

Are all of those stories related to people who use it without putting any payment card in? I’ve been happily siphoning Larry Ellisons jet fuel pennies for a good year and a half now and have none of these issues because I put a payment card in

IME, the vast majority of those horror stories end up being from people who stay in the "trial" tier and don't sign up for pay-as-you-go (one extra, easy step) and Oracle's ToS make it clear that trial accounts an resources can and do get terminated at any time. And at least some of those people admitted, with some prodding, that they were also trying to do torrents or VPNs to get around geographical restrictions.

But yes, you should always have good backups and a plan B with any hosting/cloud provider you choose.

Can confirm (old comment of mine saying the same https://news.ycombinator.com/item?id=43215430)

Traefik has some nice labeling for docker that allows you to colocate your reverse proxy config with your container definition. It's slightly more convenient than NGINX for that usecase with compose. It effectively saves you a dedicated vietualhost conf by setting some labels.

One can read more here: https://doc.traefik.io/traefik/routing/providers/docker/

This obviously has some limits and becomes significantly less useful when one requires more complex proxy rules.

Basically what c0balt said.

It's zero config and super easy to set everything up. Just run the traefik image, and add docker labels to your other containers. Traefik inspects the labels and configures reverse proxy for each. It even handles generating TLS certs for you using letsencrypt or zerossl.

I created a script that reads compose annotations and creates config for cloudflare tunnel and zero trust apps. Allows me to reach my services on any device without VPN and without exposing them on the internet.

There's very little advantage IMO. I've used both. I always end up back at Nginx. Traefik was just another configuration layer that got in the way of things.

That's pretty cheap. I have 4 vCPUs, 8GB RAM, 80GB disk, and 20TB traffic for €6. NetCup looks like it has 6VCPU, 8GB RAM, 256 GB, and what looks like maybe unlimited traffic for €5.26. That's really good. And it's in the US, where I am, so SSH would be less painful. I'll have to think about possibly switching. Thanks for the heads up.

Sure, if you still want it: https://hetzner.cloud/?ref=WwByfoEfJJdv

I guess it gives you 20 euros in credit, too. That's nice.

The only problem is Docker Swarm is essentially abandonware after Docker was acquired by Mirantis in 2019. Core features still work but there is a ton of open issues and PRs which are ignored. It's fine if it works but no one cares if you found a bug or have ideas on how to improve something, even worse if you want to contribute.

I got one missing from your list - systemd would kill screen and tmux sessions by design: https://superuser.com/questions/1372963/how-do-i-keep-system...

From what I remember that's still the default in the project, but people stopped complaining because the individual distros started overriding the relevant settings.

Thanks for adding the perspective. I was much more of a casual user at the time so didn't see as much of this side. Just knew Arch always being Arch lol

> systemd project tackles a lot of important problems

Fundamentally, this was it. SysV startup scripts had reached a local maximum decades earlier, and there was serious "overhang". When I said "superior", I really meant that it was superior to SysV, not that it was the best system that could have been imagined.

And I think the frustration was that, because it did solve so many problems, so many groups (like GNOME) were willing to switch over to it in spite of its warts; and this made it impossible for anyone who was seriously affected by its warts to avoid it. "If you don't like it, don't use it" not being an option was what drove so much of the vitriol, it seems to me.

As I said in that comment from 2019, if the maintainers had had Linus Torvald's commitment to backwards compatibility, I don't think there would have been any significant backlash.

Why did GNOME and basically all the large distros all jump with both feet in on using systemd? Because it was better. It was simply significantly better than all the alternatives. For the vast majority it was a no-brainer upgrade. The holdouts were the one's who had simple needs and were already happy with what they had. The rest of the world jumped on systemd. Because it was better.

Don't restart it. Let it crash, but take note of the situation, whatever may help investigation, maybe send out a page, flush pending writes to disk, reboot gracefully, etc. Kernel panic should be the last resort, not the default.

And I want 192.168.1.1 as the IP of my workstation on corporate LAN. Both requirements are completely arbitrary.

I guess if you really need that information, you could wait4 and dump pid/rusage to syslog. Nothing more to see here; these are zombies, orphans, by definition these processes have been disowned and there's nobody alive to tell the tale.

> have also actually sucked for 50 years

I agree with you, that's exactly right.

Not sure I'm on the same page with you on the cron. I have a similar experience but I'd rather say that cron was something that never gave me headaches. Unlike obviously systemd.

> what are the people upthread gloating about?

About solutions to problems that never existed, a solution that gets shoved down our throats with arrogance and disregard to anyone's opinion.

> everything else will die now.

Nah, cron will be just fine. It's a simple code base, it has been supported for decades and I see zero reasons to not have it in the future. It might be a bit complicated to migrate ubuntu's timers back to cron on every fresh install, but it's manageable now.

Fully agree with you on that. Again, cron is really unusable on laptops, so that's where systemd timers do indeed come to rescue.

Server side? I want none of that.

> I know systemd can be taught about this but I haven't bothered.

I think After=<your .mount> will work. If you believe it can be taught (and it can) why do you blame your lack knowledge on the tool is not a strong argument against the quality of the tool.

> Because grepping through simple rotated log files is a billion times faster than journalctl.

`journalctl -D <directory of the journal files> | grep ...` will give you what you want. Systemd is incredibly configurable and that makes its documentation daunting but damn it does everything you want it to do. I used it in embedded systems and it is just amazing. In old times lots of custom programs and management daemons needed to be written. Now it is just a bunch of conf files and it all magically works.

The most fair criticism is it does not follow the 'everything is a file philosophy' of Unix, and this makes discoverability and traditional workflows awkward. Even so it is a tool: if it does what you want, but you don't want to spend time understanding it, it is hardly the fault of the tool. I strongly recommend learning it, there will be many Ah-ha moments.

> I did manage to disable journalctl at least

My goodness. Absolutely fuck journald - a solution in search of a problem. I have created a bunch of different scripts to init my instances [1] on all projects. I do it differently from time to time, but one thing they all have in common is that journald gets removed and disabled.

[1] https://github.com/egorFiNE/desystemd

  > Because grepping through simple rotated log files is a billion times faster than journalctl

  $ time journalctl | grep "sshd" | wc -l
  12622
  journalctl  76.04s user 0.71s system 99% cpu 1:17.09 total
  grep --color=always --no-messages --binary-files=without-match "sshd"  1.28s   user 1.69s system 3% cpu 1:17.08 total
  wc -l  0.00s user 0.00s system 0% cpu 1:17.08 total

  $ time journalctl > /tmp/all.log && time wc -l /tmp/all.log
  journalctl > /tmp/all.log  76.05s user 1.22s system 99% cpu 1:17.56 total
  16106878 /tmp/all.log
  wc -l /tmp/all.log  0.03s user 0.20s system 98% cpu 0.236 total

  # THE SOLUTION
  $  time journalctl --grep=sshd | wc -l
  5790
  journalctl --grep=sshd  28.97s user 0.26s system 99% cpu 29.344 total
  wc -l  0.00s user 0.00s system 0% cpu 29.344 total
  

  > Sometimes docker starts before the filesystem is mounted.

Alternatively, you can make the drive start earlier. But truthfully, I have no reason to have docker start this early.

Here's a link to the target order diagram[0] and Arch wiki[1]. Thing that gets messy is that everyone kinda lazily uses multi-user.target

[0] https://www.freedesktop.org/software/systemd/man/latest/boot...

[1] https://wiki.archlinux.org/title/Systemd#Targets

All of your comments mention something existing for a long time, but X11 and ALSA and IPv4 and many more technologies have been used by people for many decades, and yet they still suck and have a replacement available.

It is. Laptops are totally unusable without systemd, and most of these features are needed there indeed.

My rant is: why the f are they shoved down my throat on the server side then?

I mean 5 decades ago people were using terminals, not terminal emulators. They weren't using the internet[0]. 5 decades ago Linux didn't exist, kinda making any argument moot.

[0] idk why people think Arpanet is the internet. For clarification, I'm not my das

I found Dan's recommendation to use rootful with `userns=auto`:

> User= causes lots of issues with running podman and rootless support is fairly easy. I also recomend that people look at using rootful with --userns=auto, which will run your containers each in a unique user namespace. ― https://github.com/containers/podman/issues/12778#issuecomme...

Look into podlet, it's a tool made to convert compose files, kube manfiests, running containers and maybe other stuff, into quadlets.

I'm using this tonspeedup my quadlet configs whenever I want to deploy a new service that invariably has a compose file.

I suspect there are few capabilities compose possesses that quadlets lack. Certainly, there are many capabilities that quadlets possess that compose lacks because you're really making systemd services, which exposes a host of possibilities.

Services are conceptually similar to pods in podman. Volumes and mounts are the same. Secrets or mounts can do configs, and I think podman handles secrets much better than docker. I searched for and found examples for getting traefik to work using quadlets. There are a few networking wrinkles that require a bit of learning, but you can mostly stick to the old paradigm of creating and attaching networks if that's your preference, and quadlets can handle all of that.

Quadlets use ini syntax (like systemd unit files) instead of YAML, and there is currently a lack of tooling for text highlighting. As you alluded, quadlets require one file per systemd service, which means you can't combine conceptually similar containers, networks, volumes, and other entities in a single file. However, podman searches through the quadlet directories recursively, which means you can store related services together in a directory or even nest them. This was a big adjustment, but I think I've come to prefer organizing my containers using the file system rather than with YAML.

You can if you convert your docker-compose.yaml into Kubernetes YAML and deploy that as a quadlet with a .kube extension.

Thanks! I'm more than happy to catch up to discuss the challenges. Feel free to reach out to me on twitter @psviderski or email 'me at psviderski.name'

Containers help in two ways. First in deployment, if you really have a complex system (and "modern" development practices seem to encourage complexity).

But containers really shine during development if you have more than a few developers working on the same projects. The ability to have a standard dev container for coding and testing saves so much time. And once you have that, deploying with containers is almost free.

The incomplete resource controls compared with other units is also annoying. Probably the biggest reason for me that I haven't used nspawn much.