Skip to content(if available)orjump to list(if available)

These days, systemd can be a cause of restrictions on daemons

dextercd

You can use systemd-run with --shell (or a subset of options enabled by --shell) and -p to specify service properties to run commands interactively in a similar environment as your service.

This can help troubleshoot issues and makes experimenting with systemd options faster.

I think there's been some talk about adding a built-in way for systemd-run to copy settings out of a .service file, but it doesn't exist yet.

I've written Perl/Python scripts to do this for me. They're not really aimed at working with arbitrary services, but it should be possible to adapt to different scenarios.

https://gist.github.com/dextercd/59a7e5e25b125d3506c78caa3dd...

There are some gotchas I ran into. For example, with RuntimeDirectory: systemd deletes the directory once the process exits, even if there's still another process running with the same RuntimeDirectory value set.

matheusmoreira

I use systemd-run very often to impose CPU usage limits on software. Awesome feature.

zdw

I feel like Docker and other containerization tools are becoming even less relevant given that systemd can twiddle the same isolation bits so there's no real difference in terms of security that using a container tool grants.

Seeing that podman can run containers as systemd services (see https://codesmash.dev/why-i-ditched-docker-for-podman-and-yo... ), it seems like using containers other than as a distribution mechanism has few advantages, and many disadvantages in terms of dependency updates requiring container rebuilds.

baby_souffle

> I feel like Docker and other containerization tools are becoming even less relevant given that systemd can twiddle the same isolation bits so there's no real difference in terms of security that using a container tool grants.

I see it as _exactly_ the opposite. Podman gives me more or less the same security controls as systemd and the package/delivery problem is solved.

Call me when `systemctl pull ...` fetches the binary and everything else needed to run it _and_ puts the .service file in the right spot.

nickysielicki

with podman-systemd/"Quadlet" we're basically there:

https://docs.podman.io/en/latest/markdown/podman-systemd.uni...

ndriscoll

nixos kind of does that except better. Usually just set services.foo.enabled to true along with any other config you want. It's also super easy to wrap services in a container if you want, and doing so is kept conceptually separate from dependency management. If you want to make your own systemd service, then referencing a package in `ExecStart` or whatever will make it automatically get pulled in as a dependency.

9dev

That, and dependency management, no? I’m not going back to installing libwhathaveyou-dev-0.28c1 ever again.

arianvanp

Literally exists.

importctl pull-tar https://example.com/image.tar.gz && portablectl attach image

vlovich123

Did you call him?

JoBrad

> Call me when `systemctl pull ...` fetches the binary and everything else needed to run it _and_ puts the .service file in the right spot.

That would be pretty awesome, actually.

o11c

I can already hear the systemd-haters complaining about The One True Unix Way™ is to have tools that only do one thing even if that leaves holes in their functionality.

That seems like a `machinectl` task though.

jeroenhd

Isn't this literally what podman-systemd does? You don't exactly run a command to pull a container, but just like systemd you place a config file in the right directory, tell podman-systemd to reconfigure itself, and run the service the standard systemd way.

speed_spread

That would mean systemd entering package management territory. Now THAT would not be well received.

miladyincontrol

Container rebuilds are disadvantages? Using mkosi and systemd-nspawn for containers it doesnt really feel that way, still a lot easier to build some distroless app container than to finangle a service to have zero access to other binaries, libraries, or other data entirely.

I dont get the distribution "advantage" building em with mkosi but I'd argue it a weakness as far too many are running containers with who-knows-what inside them.

oncallthrow

> I feel like Docker and other containerization tools are becoming even less relevant

Do you work in the software industry?

miladyincontrol

Systemd haters really are often a masterclass in finding problems with flexible, sanely configurable systems.

nickysielicki

The fact that systemd continues to get hate, ~15 years after mass adoption, is a cultural phenomenon worth understanding. Benno Rice of freebsd gave a super interesting talk about this: The Tragedy of systemd: https://www.youtube.com/watch?v=o_AIw9bGogo

jeroenhd

I can only imagine how long the Wayland haters will be writing blogs once LTS distro start shipping Wayland-first desktops. Looking at the whole upstart/systemd drama, I'm guessing we'll hit the 2k38 bug before they'll find something new to write about.

jlarocco

It's gas lighting to equate the two at this point.

Systemd is strictly better than what came before it, while Wayland still has missing functionality and breaks a lot of use cases.

vidarh

As a systemd user but Wayland "hater", to me the big difference is that you can adopt systemd without losing functionality - e.g. you can configure systemd to run sysV init style init scripts if you insist and no functionality is lost. The "complaints" in the linked article, are minor and about options that can just be turned off and that are offering useful additional capabilities without taking away the old.

Whereas with Wayland the effort to transition is significant and most compositors still have limitations X doesn't (and yes, I realise some of those means X is more vulnerable) - especially for people with non-standard setups. Like me.

I use my own wm. I could start with ~40-50 lines of code and expand to add functionality. That made it viable. I was productively using my own wm within a few days, including to develop my wm.

With Wayland, even if I start with an existing compositor, the barrier is far larger, and everything changes. I'm not going to do that. Instead I'll stick with X. The day an app I actually depend on stops supporting X, I'll just wrap those apps in a Wayland compositor running under X.

And so I won't be writing blog posts about how much I hate Wayland, and hence the quotes around "hater" above. But maybe I will one day write some about how to avoid running Wayland system-wide.

If Wayland gave me something I cared about, I'd take the pain and switch. It doesn't. Systemd did, so even if I hadn't liked it better than SysVinit, I'd still have just accepted the switch.

If I one day give up Xorg, my expectation is that it'll be a passive-aggressive move to a custom franken-server that is enough-of-X to run modern X apps coupled to enough-of-Wayland to run the few non-X-apps I might care about directly (I suspect the first/only ones that will matter to me that might eventually drop X will be browsers), just because I'd get some of the more ardent Wayland proponents worked up.

ziml77

I haven't seen this before. It's very interesting so far!

akkartik

When you see a large number of masters spanning diverse skill levels across a population, maybe it's an easy skill to acquire.

oncallthrow

I genuinely believe that systemd might have the highest “haters” to “benefit-to-humanity” ratio, out of any software project in history.

nine_k

PulseAudio also drew a lot of disapproval, until Pipewire appeared and finally did the same thing (and more) well.

Maybe systemd (service management, logind, the DNS resolver, the loggig system, etc) will eventually be re-implemented in a way that does not have the irritating properties of the original systemd.

/* I'd say that systemd feels like typical corporate software. It has a ton of features to check all the requisite boxes, it's not very ergonomic, it does things the way authors wanted and could sell to the corporate customers (who are not the end users), not the way end users prefer. It also used to be bug-ridden for quite some time, after having been pushed upon users. It comes from Red Hat (which is now a part of IBM), so you could say: here's why! But, say, podman also comes from Red Hat, and does not feel like such corporate software; to the contrary, end users enjoy it. */

Imustaskforhelp

Hey, now I am interested in more of such softwares overall.

Like imagine a list where we can create a form where people can give them and give reasonings or just something.

What if I can create a github repo and issues regarding this so that we can discuss about them and I can create a website later if it interests but its a really nice thought experiment.

Are we talking more about uh every software including proprietory too?

Are we talking about lets say websites too or services (what if we extend it to services like websites or even products outside of software niche into things beyond too, that's interesting too)

Another interesting point that comes to my mind might be that cryptocoins might be the lowest inverse to this software project in the sense that I believe that there was very little net positive done to all humanity in general, sure the privacy aspects are nice but still, its not worth having people invest their life savings into it thinking that its going to 100x y'know, I have created a whole article about it being frustated by this idea people think regarding crypto as an investment when it could very well be a crypto"currency" but that's a yap for another day.

I really nerded over this and I think I loved it, we need a really good discussion about it :>

correct_horse

I think I agree. I’m curious what software would be in places 2-10. If we’re talking about HN, maybe excel/google sheets? Maybe C++? Recent versions of macOS always seem to get hate, but I think macOS is in a different category.

oncallthrow

I think excel/google sheets are generally well regarded in online circles. I also don’t see that much C++ hate, at least not the same kind of viceral hate systemd receives.

nine_k

What makes me scratch my hand is why the failed access violations are not easy to show and log. A correctly configured service should not attempt to access things is is not intended to access. If it has to check if it has access and act conditionally, this also should be made explicit, either in the service code, or in its configuration.

There should be an strace-like tool that would collect a log of such "access denied" erros for troubleshooting. Even better, each service should run in its own process group, and tracing could be switched on / off for a particular process group.

amelius

I'm ok with it as long as it doesn't cause __any__ confusion whatsoever.

serbuvlad

> One of the traditional rites of passage for Linux system administrators is having a daemon not work in the normal system configuration (eg, when you boot the system) but work when you manually run it as root.

I've don't remember the last time I run a daemon by hand (that I wasn't developed it myself). I always just run the systemd unit via systemctl and debug that.

> A standard thing I do when troubleshooting a chain of programs executing programs executing programs is to shim in diagnostics that dump information to /tmp.

This seems like a very esoteric case in the days of structured logging and log levels.

> A mailer usually can't really tell the difference between 'no one has .forward files' and 'I'm mysteriously not able to see people's home directories to find .forward files in them'

Obviously a daemon that should access files in people's home directories shouldn't have ProtectHome=true. It's the responsibility of the daemon developer or the package maintainer to set appropriate flags based on what the daemon does. Someone had to explicitly write "ProtectHome=true". It's not the default, and it doesn't just appear in the service file.

When in doubt don't set security options at all, instead of shipping a broken daemon that you don't understand why it doesn't work.

Note: please base your daemon on D-Bus or a socket in /run and not on reading arbitrary files from my home directory.

I also don't understand the larger perspective? Should we not make our daemon run in more secure environments?

Zardoz84

The private /tmp strike us, when update to Debían 12 servers and find that a batch process cannot access the same temporal files that our web application. Luckily, it's very easy to fix, adding an extra systems file to disable that feature on the Tomcat service.

amaccuish

Old Man Yells at Cloud.

ranger207

Nah, this blog is neutral about it at worst. He publishes daily, and it's usually tips or notes on how things work, especially if it's different from the "traditional" way of doing things. One of my favorite blogs for letting me know how things work nowadays

greatgib

Systemd, as usual randomly and suddenly breaking things that worked for decade and for people that asked nothing. Because they know better what you need...

nickysielicki

And what's your preferred alternative to what's described in the article? Packaging every single service in its own 500mb ubuntu chroot and using docker? Running a local dhcp server and a bridge interface so that you can selectively expose ports?

Here's an alternative title for this post: these days, two lines in a systemd service file can easily constrain arbitrary applications to just the files and resources they need, and only those.

probably_wrong

My grumpy preferred alternative would be "you're supposed to be an init service. That's not your job".

Un1corn

> systemd is a suite of basic building blocks for a Linux system.

You can always use a simpler init system if you want

nickysielicki

I linked it elsewhere in this thread, but you should really watch this talk, particularly 12:45 through 16:20: https://www.youtube.com/watch?v=o_AIw9bGogo

tl;dr: systemd isn't meant to be an init system, it's meant to manage services, and the alternative world where you don't have a unified system for managing services and events actually sucks.

silverquiet

Doesn't SELinux do that (and more)?

amluto

The problem is the “more”. SELinux is extremely flexible and does what the configuration tells it to do. And it does not compose well. Want to point whateverd at /var/lib/whatever? Probably works if the distro packages are correct. Want to make /var/lib/whatever be a symlink? Probably does not do what you expect. Want to run a different daemon that accesses /var/lib/whatever or mount it into a container? Good luck. Want to run a second copy of the distro’s whateverd and point it at a different directory? The result depends on how the policy works.

And worst: want to understand what the actual security properties of your policy are? The answer is buried very, very deep.

null

[deleted]

wolrah

The example given is a distro changing their bundled systemd unit files to use new features, yet you choose to blame systemd?

You do realize distros can also change SysV shell scripts in ways that break your use case as well, right?

oncallthrow

Did you read the article? The author is complaining that aystemd introduced _optional_ security mechanisms for units. If you don’t like these mechanisms, don’t use them in your units.

Systemd didn’t “break” anything at all here. This author’s arcane debugging workflow doesn’t work for certain units who have opted into the new security mechanisms. But that is hardly systemd’s fault.

cwillu

“Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".” --https://news.ycombinator.com/newsguidelines.html