Do Users Verify SSH Keys? (2011) [pdf]
45 comments
·October 23, 2025zie
We transfer ACH files(i.e. paychecks) via SSH(SFTP) to several banks. You better believe I check keys. One of the banks forces key rotation every 2-ish years. I absolutely verify it every rotation and delete the old keys.
Occasionally it fails, almost always it's something unexpected happening, but occasionally we catch their errors(verified by connecting from various endpoints/DNS queries/etc). We used to call them all the time whenever that happened. Now we just auto-retry on failure in an hour and that fixes the issue all of the time(so far). We only re-try once and then fail with a ticket. Most of us like our paychecks, so we are pretty good about getting that ticket resolved quickly.
beala
Terminal.shop lets you order coffee over ssh, which is kind of novel and fun. I did it, and the coffee was good! This post reminded me that they've gotten enough questions about security that they've added this to their FAQ:
> is ordering via ssh secure?# you bet it is. arguably more secure than your browser. ssh incorporates encryption and authentication via a process called public key cryptography. if that doesn’t sound secure we don’t know what does. [1]
I think this is wrong though for exactly the reasons described in this post. TLS verifies that the URL matches the cert through the chain of trust, whereas SSH leaves this up to the user to do out-of-band, which of course no one does.
But then the author of this article goes on to say (emphasis mine):
> This result represents good news for both the SSL/TLS PKI camps and the SSH non-PKI camps, since SSH advocates can rejoice over the fact that the expensive PKI-based approach is no better than the SSH one, while PKI advocates can rest assured that their solution is no less secure than the SSH one.
Which feels like it comes out of left field. Certainly the chain of trust adds some security, even if it's imperfect. I know many people just click through the warning, but I certainly don't.
tw04
>TLS verifies that the URL matches cert through the chain of trust,
I think you need to point out that TLS utilizes the browsers cert store for that chain of trust. If a bad actor acquires an entity that has a trusted cert, or your cert store is compromised, that embedded cert store is almost entirely useless which has happened on more than one occassion (Chinese government and Symantec most recently).
https://expeditedsecurity.com/blog/control-the-ssl-cas-your-...
This is typically caught pretty quickly but there's almost nothing a user can do to defend against a chain of trust attack. With SSH, while nobody does it, at least you have the ability to protect yourself.
zie
in SSH, it's a two-way handshake, the client ordering the coffee also gets a cert to prove their identity.
In browser land, the client browser doesn't get a cert to prove their identity, it's one-way only.
Certainly TLS supports client certs, browsers(at least some) technically even implement a version, but the UX is SOOOO horrible that nobody uses it. Some people have tried, the only people that have ever seen any success with client side authentication certificates over a web browser are webauthn/passkeys and the US Military(their ID cards have a cert in them).
webauthn/passkeys are not fully baked yet, so time will tell if they will actually be a success, but so far their usage is growing.
kbolino
I think webauthn/passkeys will be more successful (frankly I think they already have been) because they're not part of TLS. The problem with client certs, and other TLS client auth like TLS-SRP, is that it inherently operates at a different layer than the site itself. This cross-cutting through layers greatly complicates getting the UX right, not just on the browser side (1) but also on the server side (2). Whereas, webauthn is entirely in the application layer, though of course there's also some supporting browser machinery.
(1) = Most browsers defer to the operating system for TLS support, meaning there's not just a layer boundary but a (major) organizational one. A lot of the relevant standards are also stuck in the 1990s and/or focused on narrow uses like the aforementioned U.S. military and so they ossified.
(2) = The granularity of TLS configuration in web servers varies widely among server software and TLS libraries. Requesting client credentials only when needed meant tight, brittle coupling between backend applications and their load balancer configuration, which was also tricky to secure properly.
zie
So true, two-way certs with TLS have crappy implementations everywhere, not just in the browser.
I have 2 problems with webauthn/passkeys:
* You MUST run Javascript, meaning you are executing random code in the browser, which is arguably unsafe. You can do things to make it safer, most of these things nobody does(never run 3rd party code, Subresource Integrity, etc).
* The implementations throughout the stack are not robust. Troubleshooting webauthn/passkey issues is an exercise in wasted time. About the only useful troubleshooting step you can do is delete the user passkey(s) and have them try again, and hope whatever broke doesn't break again.
kqr
I feel like this is unnecessarily reductive. The initial handshake is always fraught with security problems. I struggle to see a scenario in which a bad actor is able to give me the address of a bad machine, yet not be able to trick me into their host key being the correct one.
I would definitely however spend effort into verifying a host key that changes unexpectedly.
chasil
My corporate SFTP server that became mandated to use several years ago presented multiple keys, apparently because it was on DNS round robin.
My attempts to convince them to use the same key came to naught, so instead I use one of the IP addresses.
I could alternately erase the known hosts entry on each transfer. That would probably have been preferable.
I also got a shell on it when I attempted ssh, so you can guess the care that is taken with it.
kragen
They might be able to spoof your DNS, for example if you're using their Wi-Fi, so you get the wrong IP address for the right hostname, but not your mail server's SSL. You could pass the host key fingerprint across an existing secure connection, such as in email or with ssh to a host that you already have the fingerprint of.
whatevaa
5 dollar wrench xckd applies here.
integralid
It does not. DNS attacks happen. It won't be used by APT on you or me, but may be used as an escalation mechanism in a company, for example. It's also something a hacked router could do, but I never heard of it happening, to be fair.
I was actually personally a victim of such (unsuccessful) attack in the tor network. SSH login to my hidden service complained about a wrong fingerprint. The same happened when I tried again. After tor restart, problem disappeared. I assume this was an attempt at SSH mitm by one of exit nodes?
kragen
No, it really doesn't, as in most cases where people invoke it.
jrochkind1
I'll be honest, I have never spent effort on a host key that had changed unexpectedly, and at least a few have.
kragen
I've often called people on the phone and stuff. It depends somewhat on what's at stake. Authenticating users with SSH passwords puts much more at stake than using public keys, since an attacker who can get you to send your unencrypted password to a malicious server once can steal your account; deploying PAKE algorithms (successors to SRP, see https://eprint.iacr.org/2021/1492.pdf) could mitigate that, but I don't think any shipped SSH version has ever supported a PAKE algorithm.
zenmac
Server should publish their key fingerprint to at least the authorized personal of the group. So people know if the server they are connecting to is actually that server.
amelius
We need a new protocol, where installing the OS of a new machine automatically installs a trusted key from an inserted USB drive, so that the machine automatically becomes part of the "enclave".
Or something like that.
MaxMatti
The paper does mention that you can have your ssh keys signed by a ca, so in a company the it staff could configure everybodys os to only trust ssh keys signed by the organization.
otabdeveloper4
> you can have your ssh keys signed by a ca
Good idea. That way when your CA private key leaks (the key which we never ever rotate, of course) the bad guys can compromise the whole fleet and not just one server. Bonus points if the same CA is also used for authenticating users.
organsnyder
This is common in corporate environments.
thesuitonym
You are under no obligation to use the generated, self-signed key. Most people do because it's "good enough".
dspillett
> I struggle to see a scenario in which a bad actor is able to give me the address of a bad machine, yet not be able to trick me into their host key being the correct one.
If you aren't bothering to verify then they do not need to trick you at all.
In DayJob we have a lot of clients send feeds and collect automated exports via SFTP, and a few to whom we operate the other way (us pulling data via SFTP or pushing it to their endpoint). HTTPS based APIs are very common and becoming more so, but SFTP is still big in this area (we offer some HTTPS APIs, few use them instead of SFTP).
One possible exploit route, for a malicious actor playing a long and targetted game, that could affect us:
1. Attacker somehow poisons our DNS, or that of a specific prospective client of ours, sending traffic for sftp.ourname.tld to their server, and has access to our mail.
2. Initially they just forward traffic so key verification works for existing users. They monitor for some time to record host addresses that already access the server, so when they start intervening they can keep just forwarding connections from those addresses, so those users see no warnings (and are unaffected by the hack).
3. When they do start to intercept connections from hosts not already on the list make above instead of forwarding everything, existing users are unaffected¹ but new users coming in from entirely different addresses now go to their server and if they are not verifying the key will happily send information through it², authenticating with the initial user+pass we sent or PKI using the public key they sent, with the malicious server connecting through to ours to complete the transfers.
4. Now wait and collect data as no one realises there is a MitM, and later use any PII or other valuable information for ransom/extortion purposes.
Of course there are ways to mitigate this attack route. For one: source address whitelisting, supported by OpenSSH's key based auth as the acceptable source list can be included with the public key so only specific sources can use that key for auth. But they client would have to make effort to do this, and if they aren't going to make the effort to verify the host key then they aren't going to make other efforts either.
We do have some clients who verify the host properly and/or give us source addresses to limit connections to when they provide a public key, we work with financial institutions who are appropriately paranoid about their data and the data of their customers, some even use PGP for data in transit (and in case it is ever stored where it shouldn't be) for an extra level of paranoia. But most do none of this. Most utterly ignore our strong suggestion that they use keys, or change passwords in case of email breach, instead using the password we mail them before first connection for eternity.
--------
[1] none of our clients are likely to be sending files from dynamic source addresses, at most the source might move around a v4/24 or v6/64, currently I don't think all of them connect from a single IPv4 address, I've had one recently let us know (months in advance) that their source address will be changing.
[2] it can connect to us and send the data
otabdeveloper4
> ...a host key that changes unexpectedly.
Literally happens every single damn day and literally nobody on the face of this earth ever gives a shit.
Host keys are the stupidest idea in the history of computer so-called "security".
BenjiWiebe
Why are yours changing every day? If they always did that, then yes it would be a stupid idea. But they don't change on their own, or for no reason, so it isn't a stupid idea.
Mine change maybe once every couple of years, if I do a full reinstall without copying over the old host key. And then I know exactly why it changed.
chuckadams
No, and expecting users to actually do so is a sign that something is very wrong about the process. TOFU turns out to be good enough for most purposes anyway, but if a key changes (perhaps the server was reimaged) then verifying it is about as friendly as a tax audit. Or using GPG.
SoftTalker
Yeah that is my experience. Users don't understand public key cryptography. You ask them for their public key and they send you the private one. They use the same key everywhere. They don't understand the difference between a host key and a login key. Ask them to do anything with their authorized_keys file and your next ticket will be "I'm locked out of my system."
They do understand passwords, and most can manage an SMS code as a second factor. That's about the limit of what you can count on.
chuckadams
I've been doing this for 30 years and sometimes I give the wrong key file on the command line by forgetting to add '.pub' to the end. Far as I remember, I've always caught it before I managed to send it somewhere public, and thankfully most of my keys nowadays have a passphrase that gets remembered in my OS's keychain. But the UX is really that bad.
teeray
I kinda like the approach Github does: they just publish their fingerprints here: https://docs.github.com/en/authentication/keeping-your-accou...
This is served over TLS, so it's no worse than TLS. You can also benefit from the paved road that LetsEncrypt has provided. It might not be as smooth as SSH CAs once they're set up, but setting those up and the Day 2 operations involved isn't nearly as straightforward.
null
erikerikson
Fails to mention that you can paste in the expected key. Of course if there is a compromise of the source the key is copied from that no help but that's a higher bar. Still easy and doesn't rely on human frailty.
jon-wood
Or you can use SSH certificates, where you work on the basis that if the host key is signed by the correct CA then it's legit. No more tofu required beyond need to trust whatever source you got your CA's public key from.
radial_symmetry
I appreciate the to-the-point abstract
tobinfricke
Also in compliance with Betteridge's law of headlines: "Any headline that ends in a question mark can be answered by the word no."
dcminter
I guess the interesting question to me is: how often does this matter? How many successful mitm attacks on ssh connections are there and in what sort of circumstances do they occur?
It seems like it ought to matter, but if roughly nobody verifies and yet the sky has not fallen - does it?
marcosdumay
It's only for the first connection, and it's very rare that targets are valuable on the first connection.
On the other hand, we know of at least two suppliers of software that run with elevated access everywhere (including the dev side of every advanced military) that have been breached by unknown parties for years. The most likely explanation, by far, is that the sky only didn't fall yet because nobody wants it to. And that leaves us vulnerable to somebody suddenly wanting it.
1oooqooq
nobody robbed my house in years. i still lock the door.
it's so banal to check host keys.
kemotep
Thanks for sharing this! Yesterday I was just wondering about ssh key verification techniques for third party services.
SSH keys are amazing, portable and in some ways easier to use than Passkeys. But for them to successfully replace passwords and account configuration, which works decently well for a service like pico.sh, the user experience needs to be improved significantly. Not impossible but what does become a continuous and ongoing problem is verification.
hk1337
Coupled with that you have to have a matching private key that you created and using ssh-config, is this even necessary?
Until SSH servers implement PKIX based Host Key verification, it's always going to be fraught with issues like this. Users will just keep blindly accepting host keys because they "Don't got time for that."