The order of files in /etc/ssh/sshd_config.d/ matters
115 comments
·April 3, 2025egberts1
dfc
The way you wrote the rule makes no sense to me. Maybe it's too early in the day for me?
"In placing configuration files higher than user-defined configuration but Only with SSH client, can want..."
atoav
With multiple config files overriding each other in an predictable order you can effectively allow users to change some settings while ignoring (overriding) whatever they set on others.
egberts1
I find this SSHD snippet to be extremely useful in enterprise network, notably with OpenLDAP.
Also the most dangerous but flexible way to authenticate a user.
https://jpmens.net/2019/03/02/sshd-and-authorizedkeyscommand...
efrecon
This is really good! Thanks!
egberts1
For those that are exploring software-based public certificate and OpenSSH, Ive broken down the settings for most PKI handlers.
https://egbert.net/blog/articles/openssh-file-authorized_key...
memco
Thanks for sharing this! I think I may now have what I need to set up a system with multi-user shared keys that only work for a given set of users.
egberts
I do enjoy dual-PK-certificate authentication in my homelab: one by equipment, and one by user/group.
Only misgiving is that the key management issues have worsen only for the key administrator(s). But it is a viable and sustainable AA model because there is the most important security component: instant denial of a user and/or a equupment.
ThePowerOfFuet
This comment seems to have a lot to say but it was word salad to me, quite confusing and hard to read :(
0xbadcafebee
This is perhaps old sysadmin knowledge, but different tools have very different heuristics about how they parse configuration, and you have to check every time and not assume. Among the consequences to not checking are gaping security holes.
INTPenis
You mean there are consequences to making assumptions? ;) (also old sysadmin)
wruza
That's why I erase sshd_config and put what I really meant there. You may say "but isn't it better to patch it properly?". It is not. Yet another vps hoster –> yet another /etc/ssh directory template that may have all sorts of access issues in it. Better to replace it and make it do exactly what you have planned.
timewizard
I've never liked the directory.d/* infrastructure. In so many cases, even with a properly configured sshd_config, the resulting configuration file is not so large that it benefits from being split up.
You have to deal with ordering issues, symlink management in some cases, and unless the "namespace" of sorting number prefixes is strictly defined, it's never something that's convenient or durable to "patch" new files into. The proliferation of 99_* files shows the anti-utility this actually provides.
I much prefer configuration files with a basic "include" or "include directory" configuration item. Then I can scope and scale the configuration in ways that are useful to me and not some fragile distribution oriented mechanism. Aside from that with xz I don't think I want my configurations "patchable" in this way.
rlpb
The .d directories are important on Debian and Ubuntu where packaging needs to provide different snippets based on the set of installed packages, the VM environment, other configuration inputs like through cloud-init and so forth, and update them during upgrades, but also (as per policy) preserve user customisations on anything in /etc.
Since pretty much every file has different syntax, this is virtually impossible to do any other way.
hnlmorg
Config directories are there to solve change management problems like idempotency.
If you have one big file then different tools, or even the same tool but different points of that tools life cycle, can result in old config not correctly removed, new config applied multiple times, or even a corrupt file entirely.
This isnt an issue if you’re running a personal system which you hand edit those config files. But when you have fleets of servers, it becomes a big problem very quickly.
With config directories, you then only need to track the lifecycle of files themselves rather than the content of those files. Which solves all of the above problems.
wruza
I never managed a fleet. I mean I occasionally manage up to 30 instances, does that count?
Either way, my notion about doing it properly is to have a set of scripts (ansible/terraform?) that rebuild the configuration from templates and rewrite, restart everything. Afaiu, there's no "let's turn that off by rm-ing and later turn it on again by cat<<EOF-ing", cause there's no state database that could track it, unless you rely on [ -e $path ], which feels not too straightforward for e.g. state monitoring.
(I do the same basically, but without ansible. Instead I write a builder script and then paste its output into the root shell. Poor man's ansible, but I'm fine.)
So as I understand it, these dirs are only really useful for manual management, not for fleets where you just re-apply your "provisioning", or what's the proper term, onto your instances, without temporary modifications. When you have a fleet, any state that is not in the "sources" becomes a big problem very quickly. And if you have "sources", there's no problem of turning section output on and off. For example when I need a change, I rebuild my scripts and re-paste them into corresponding instances. This ensures that if I lose any server to crash, hw failure, etc, all I have is to rent another one and right-click a script into its terminal.
So I believe that gp has a point, or at least that I don't get the rationale that replies to gp suggest itt. Feels not that important for automatic management.
loodish
The .d directories make management via tools such as ansible much much easier.
You don't have weird file patching going on with the potential to mess things up in super creative ways if someone has applied a hand edit.
With .d directories you have a file, you drop in that file, you manage that file, if that file changes then you change it back.
ecef9-8c0f-4374
I love that you can use validate: sshd -T -f %s To check if changes would break things.
vbezhenar
That's one of my issues with most Linux distros.
1. They add huge configuration files where 99% are commented out.
2. Sometimes they invent whole new systems of configuration management. For example debian with apache httpd does that.
I don't need all of that. I just need simple 5-line configuration file.
My wish: ship absolutely minimal (yet secure) configuration. Do not comment out anything. Ask user to read manuals instead. Ship your configuration management systems as a separate packages for those who need it. Keep it simple by default.
sneak
The conf.d isn’t because the config file is large. It’s because it’s easier to disable or enable something with an “echo blah > conf.d/10-addin.conf” or an “rm conf.d/50-blah.conf” than it is to do sed -i or grep blah || echo blah >>
claudex
Also, it allows different packages to handle the configuration and add their specific parameters.
shawnz
Exactly: if your templating logic accidentally produces a syntax error, now you can't log in to SSH. There's much less chance of that scenario with include directories. This applies for infrastructure as code scenarios, changes made by third party packages, updates of ssh, manual one-off changes, etc.
badgersnake
Totally, don’t use .d for ssh. The configuration is not that complicated. If it is, you’re doing it wrong.
eadmund
Yeah, first-wins is definitely surprising. Off the top of my head, it feels like one would have to go out of one’s way to write a parser that does that (by storing an extra bit of state for each configuration item, and then checking it before setting the configuration item and toggling the state, rather than just applying the configuration item each time it is encountered).
Is there a good reason for this design? I can’t think of one, again off the top of my head, but of course I could be missing something.
Joker_vD
The SSHD internally has a config structure. Initially, it's all initialized with -1 for flags/numbers/timeouts, and with NULL for strings. When an option is parsed, there is a check whether the config structure's corresponding field is -1/NULL (depending on the type) before the parsed value is saved.
Another program with first-wins I've seen used dict/map for its config so the check was even simpler: "if optname not in config: config[optname] = parsed_value".
o11c
It probably makes a bit more sense when you think about the fact that SSH frequently does a "try to match this host against a list of configured host-patterns" operation. In that case, "first match" is the obvious thing to do.
Thorrez
iptables uses first-wins.
https://serverfault.com/questions/367085/iptables-first-matc...
null
ajross
It's actually the simplest scheme. Reparse from the top whenever you need to query a setting. When you see one, exit. No need to even bother to store an intermediate representation. No idea if this matches the actual ssh implementation, but that's the way many historical parsers worked. The idea of cooking your text file on disk (into precious RAM!) is fairly modern.
Joker_vD
Nope, the actual ssh implementation parses all the config files once, at the startup, using buffered file I/O and getline(). That means that on systems with modern libc, the whole config file (if it's small enough, less than 4 KiB IIRC?) gets read into the RAM and then getline() results are served from that buffer.
The scheme you propose is insane and if it was ever used (can you actually back that up? The disk I/O would kill your performance for anything remotely loaded), it was rightfully abandoned for much faster and simpler scheme.
ajross
> getline() results are served from that buffer.
So... it doesn't parse them once! It just does its own[1] buffering layer and implements... exactly the algorithm I described? Not seeing where you're getting the "Nope" here, except to be focusing on the one historical note about RAM that I put in parentheses.
[1] Somewhat needless given the OS has already done this. It only saves the syscall overhead.
Dylan16807
Loading up your parsing code and reopening the file every time a setting is queried sounds to me like it would increase the average memory use of most programs.
naniwaduni
You don't care about average memory use, you care about peak memory use.
ajross
The ssh config format has almost no context, and the code is static and always "loaded up". I can all but guarantee this isn't correct. Modern hackers tend to wildly overestimate the complexity of ancient tasks like parsing.
noufalibrahim
This is weird. I've been hitting funny problems while trying to get ssh to authenticate (using passwords) via. a keycloak instance. I've been trying to do using PAM script but have been pretty unsuccessful till now. Apparently, they don't play nice together.
drpixie
There is a nice sshd option (-T) that tells you what it's really doing. Just run
sudo sshd -T | grep password
mmsc
Except that doesn't tell you what it's doing, that tells you what it _might_ do, if you (re)start the server.
sshd -T reads the configuration file and prints information. It doesn't print what the server's currently-running configuration is: https://joshua.hu/sshd-backdoor-and-configuration-parsing
aflukasz
Yes. Run this as a validation step during base os image creation, if such image is intended to start system with sshd. That way you can verify that distro you use did not pull the carpet from under your feet by changing something with base sshd config that you implicitly rely on.
eapriv
Of course the order matters, that’s why the file names have numbers in them.
extraduder_ire
I think their surprise comes from earlier config wins conflicts, rather than the other way around. That's not reflected in the title.
lubutu
I initially read "the order of files in /etc/ssh/sshd_config.d/" to mean the order of files in the underlying directory inode, i.e. as returned by `ls -f` — and thought, "oh god"... But the lexicographical order, that's not too surprising.
1oooqooq
yeah, this is confdir 101...
but i guess learning is better late than never type of thing.
also what confuses people more on this is that openssh is properly designed, so configs are first seen wins. exactly so that file 0_ wins from 99_... but most people see badly designed software where 99_ overrides 0_. openssh way is exactly so it works best with confir or ssh/options where it matches by hosts and you can place the more important stuff first without fear defaults will override it.
null
samlinnfer
I've made a big stink about this last time: https://news.ycombinator.com/item?id=42133181
They've updated the documentation on /etc/ssh/sshd_config https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/20...
jmclnx
Curious, since when is directory "/etc/ssh/sshd_config.d/" a thing ?
I checked on my OpenBSD (7.6) System and Slackware (15.0) and that directory does not exist. I checked the man page for sshd and there is no mention of that dir.
Is this a new thing/patch Linux people came up with ?
nubinetwork
Gentoo started doing this last year and I absolutely hate it.
null
dfc
It might not be an OpenBSD thing. It may be a Debian/ubuntu-ism.
SoftTalker
OpenBSD is the author of OpenSSH and yes they support the Include directive in sshd_config but they do not use it in a default install.
jmclnx
Just read the manual, nice to see they are not patching ssh for this.
czernobog
This is interesting, usually it's the latter because the config is ran line by line
Also, if it's not too much trouble, would someone help me understand why such files are required to start with numbers? In this case it's 10-no-password.conf.
I have noticed similar structure for apt and many more packages
glitchcrab
A lot of software which reads drop-in files will load them in numerical (or indeed alphabetical) order. Obviously this is important if the order your config files are loaded in matters, but otherwise it's just become a convention so people do it even if the load order doesn't actually matter.
47282847
Typically, config files are merged into one by loading them like ./conf.d/*, the order being determined alphabetically from their file names. You do not need to use numbers but they help to see that order.
therein
The only time I hear or see anything about cloudinit, it is always a problem. Nobody ever said "we don't need worry about that, cloudinit takes care of it".
What good does cloudinit do really?
aflukasz
In this particular case cloudinit presence in the story is incidental, delivery mechanism of said config file could have been different.
It's useful for initializing state that could not have been initialized before booting in the target environment. Canonical example, I guess, being ssh server and client keys management, but the list of modules it implements is long.
nightfly
Provides a moderately-configured starting point for new cloud VM deployments without requiring custom images
naniwaduni
> Nobody ever said "we don't need worry about that, cloudinit takes care of it".
Well, why would it come up? You don't need to worry about things you don't need to worry about.
barotalomey
Hence the tradition of numeric file naming in *.d directories.
Purpose of first-define is the rule: In placing configuration files higher than user-defined configuration but Only with SSH client, can want user to have control from their config files: Remove from config files Place a couple under Match/MatchGroup using deny/accept.
SSHD (server/non-client) still support admin-defined by having system-wide settings done firstly. For those who have multi-file SSHD configurations, breakdown of the many config file locations and scopes here as it covers default user, system-wide, specific user:
https://egbert.net/blog/articles/ssh-openssh-options-ways.ht...
Also I broken out each and every SSHD and SSH options along with their ordering by execution by using file name and numbering as well as its various state machine, dispatch, CLI equivalence, network context, and function nesting, all in:
https://github.com/egberts/easy-admin/tree/main/490-net-ssh
https://github.com/egberts/easy-admin/blob/main/490-net-ssh/...
Disclaimer: I do regular code reviews of OpenSSH and my employer authorizes me to release them (per se contract and NDA)
Also this showed how to properly mix and match authentication types using OR and AND logic(s) in
https://serverfault.com/a/996992
It is my dump mess so wade 'em and enjoy.