Skip to content(if available)orjump to list(if available)

Open-sourcing OpenPubkey SSH (OPKSSH): integrating single sign-on with SSH

traceroute66

Whilst this is clearly an interesting development, I still prefer SSH CA backed by hardware on both issuer and client side (e.g. Yubikey).

This is for three reasons:

First, the SSH-CA+Hardware method does not require call-out to third-party code from SSHD, and thus minimises attack surface and attack vectors.

Second, the SSH-CA+Hardware method completely prevents key exfiltration or re-use attacks. Yes, I understand that the SSH keys issued by OPKSSH (or similar tools) are short-lived. But they are still sitting there in your the .ssh directory on your local host, and hence open to exfiltration or re-use. Yes it may be a short-timeframe but much damage can easily be done in a short timeframe, for example exfiltrate key, login, install a backdoor and continue your work via the backdoor.

Finaly, the SSH-CA+Hardware method has fewer moving parts. You don't even need software tools like step-ca. You can do everthing you need with the basic ssh-keygen command. Which means from a sysadmin perspective, perhaps especially sysadmin "emergency break-glass" perspective, you do not need to rely on any third-party services as gatekeeper to your systems.

e40

Can you recommend a setup guide?

traceroute66

> Can you recommend a setup guide?

Depends how far up the chain you want to go (e.g. use step-ca or not), but at the most primitive level, you are looking at something along the following lines of the below (the below is based off my rough notes, I might have missed something).

Note that I have ignored any Yubikey setup considerations here like setting PIN, touch-requirement etc. etc.

I have also assumed plain Yubikey, not the YubiHSM. The YubiHSM comes with SSH certificate signing functionality "out of the box".

Client Yubikey:

   - Use Yubikey ykman[1] to generate a PIV key according to your tastes
   - Grab the key in ssh format with `ssh-keygen -D $path_to/libykcs11 -e > $client_key.pub`
Issuer Yubikey:

   - Use Yubikey ykman[1] to generate a PIV key according to your tastes
   - Grab the key in ssh format with `ssh-keygen -D $path_to/libykcs11 -e > $issuer_key.pub` (save this for the next step and also put it into your sshd CA config)
   - Sign with the issuer Yubikey with `ssh-keygen -s $issuer_key.pub -D $path_to/libykcs11 -I $whatever_identity -n $principal_list -V +$validity_period $client_key.pub`
(libykcs11 is the yubikey library,it ships with yubikey-piv-tool[2])

[1] https://docs.yubico.com/software/yubikey/tools/ykman/PIV_Com... [2] https://developers.yubico.com/yubico-piv-tool/Releases/

throw0101c

Another option:

SSH certificates have been around for a while now, so you can create an in-house SSH CA, so that they are short-lived (compared to on-laptop keys) and you have to authenticate to get a fresh one.

To automate getting SSH certs there are a number of options, including the step-ca project, which can talk to OAUTH/OIDC systems (Google, Okta, Microsoft Entra ID, Keycloak):

* https://smallstep.com/docs/step-ca/provisioners/#oauthoidc-s...

as well as cloud providers:

* https://smallstep.com/docs/step-ca/provisioners/#cloud-provi...

There are commercial offerings as well:

* https://www.google.com/search?q=centrally+managed+ssh+certif...

EthanHeilman

Step-ca is really cool and has a lot of templating and policy stuff opkssh doesn't currently have. However step-ca does require two trusted parties: your IDP and the SSH CA.

The advantage of opkssh is that there is only one trusted party, your IDP.

While not available in opkssh yet, OpenPubkey even has a way of removing the trust assumption in your IDP.

I wonder if step-ca would ever consider using opkssh or the OpenPubkey protocol

Jnr

Step CA has some open source parts but it is built in a way where it is inconvenient to use without their admin infra. I also had to do some modifications to their server to make it work properly. If OPKSSH doesn't need all that crap then I am all for it. I'll certainly give it a shot.

blueflow

Theoretically, is the distinction between IDP and CA necessary? I kinda would expect the IDP to certify my pubkey.

EthanHeilman

Currently IDPs don't care about user public keys. OpenPubkey manages to slip the user's public key into an issued ID Token without the IDP having to know about it.

Ideally IDPs are CAs for identity and ID Tokens have a public key field.

There are neat projects and standards to do this like OIDC-squared [0] and OIDC4VC [1] but it is unclear if IDPs will implement them if they are standardized. We do have DPoP now [2] but it isn't available for any of the usecases that are important to me. OpenPubkey is largely an productive expression of my frustration with public keys in tokens being a promised feature that never arrives.

[0]: OIDC-squared https://jonasprimbs.github.io/oidc-squared [1]: OIDC4VC https://identity.foundation/jwt-vc-presentation-profile/ [2]: RFC-9449 OAuth 2.0 Demonstrating Proof of Possession (DPoP) - https://datatracker.ietf.org/doc/html/rfc9449

zokier

I don't love this.

> Unfortunately, while ID Tokens do include identity claims like name, organization, and email address, they do not include the user’s public key. This prevents them from being used to directly secure protocols like SSH

This seems like dubious statement. SSH authentication does not need to be key based.

I understand the practicality of their approach, but I would have preferred this to be proper first-class authentication method instead of smuggling it through publickey auth method. SSH protocol is explicitly designed to support many different auth methods, so this does feel like a missed opportunity. I don't know openssh internals, but could this have been implemented through gssapi? That's the traditional route for ssh sso. If not gssapi, then something similar to it.

https://datatracker.ietf.org/doc/html/rfc4462

EthanHeilman

> This seems like dubious statement. SSH authentication does not need to be key based.

Let's say you just use an ID Token as a bearer token to authenticate to SSH. The SSH server now has the secret you used to authenticate with. Doesn't this introduce replay attacks where the SSH server can replay your ID Token to log into other SSH servers?

Whereas if your ID Token functions like a "certificate" issued by your IDP binding your identity to a public key, it is no longer a secret. You can just use your public key to prove you are you. No secrets leave your computer.

My motto: always use public key rather than a bearer secret if possible.

> I understand the practicality of their approach, but I would have preferred this to be proper first-class authentication method instead of smuggling it through publickey auth method

Me too. I have a PR open to SSH3 (not connected with OpenSSH) so it can be support OpenPubkey as a built-in authentication mechanism.

https://github.com/francoismichel/ssh3/pull/146

jrozner

I think it's interesting they're choosing to use certificates this way. If they're already using certs, why not just leverage sshca auth? Also, at the end of the day, it's still effectively a bearer token. I founded a company called Based Security last year in this space. We're looking for design partners currently. We host a CA for you (or you can host yourself if you want) and use ssh certificates and bind the user identity (oidc to the IdP) to a physical device (yubikey, secure enclave, tpm, etc.) This ensures that the user is both in possession of the physical device and that the credential can't be stolen without stealing the device, unlike the bearer token examples here. Currently we're offering support for GitHub and GitLab authentication but it works out of the box with standard ssh tooling as well. It just currently requires manually handling user provisioning for standard ssh access.

EthanHeilman

> Why not just leverage sshca auth?

Because that has two trusted parties: the IDP and the SSH CA. OPKSSH has just one trusted party: the IDP.

> This ensures that the user is both in possession of the physical device and that the credential can't be stolen without stealing the device, unlike the bearer token examples here. Currently we're offering support for GitHub and GitLab authentication but it works out of the box with standard ssh tooling as well. It just currently requires manually handling user provisioning for standard ssh access.

That sounds valuable.

Have you looked in OpenPubkey, the cosigner protocol supports binding hardware tokens to ID Tokens? Although not as fancy as having the SSH key pair live in the hardware token but maybe we could figure out a way to get the best of both worlds.

slt2021

> The SSH server now has the secret you used to authenticate with.

secrets can be made unique per connection and single use

confiq

this ^

GSSAPI can be more secured than public/private key if configured right.

asmor

This is the purpose of the not so well known audience claim.

Though I'd still prefer to authenticate to something like Vault's SSH engine and get a very short lived SSH certificate instead. No new software to install on your servers, just the CA key.

c45y

CA key also allows those servers to avoid reaching out to some central location to validate which I've found to be a nice side bonus for disaster recovery type scenarios.

pcthrowaway

I'd assume the auth handshake would prevent this.

- client connects to SSH server at IP X.X.X.X or hostname SomeHost

- redirected to oAuth server

- Client signs in and receives token scoped to X.X.X.X or hostname SomeHost

- Client provides token to SSH server

rcarmo

Yeah, I don't like this approach either. There was a lot of plumbing added to sshd to support pluggable auth methods, and having used a few of them (including TOTP, for instance), I am not really a fan of "extending" publickey.

(Am also not really a fan of having to eventually use a browser for authenticating a terminal session, but that's another problem.)

johnisgood

> (Am also not really a fan of having to eventually use a browser for authenticating a terminal session, but that's another problem.)

That sounds awful, I hope this is not the direction we are heading towards.

cyberax

You can actually have a fully command-line driven Single Sign-On workflow, even anchored in hardware (TouchID, FIDO tokens, etc.)

It's not a common way to do it, but it's definitely a possibility.

godelski

It happens when there's a cloudflared instance. It is quite annoying

out-of-ideas

not just a browser - but coupled with the javascript-as-an-operatingsystem which first assumes you are a bot, but then you prove to it that you are not. lol

mathfailure

They reject the proofs now: they just show the spinner spinning indefinitely now. CloudFlare is broken and it has widespread so much that it looks like a cancer.

zokier

Googling around I see lot of work around this topic. One example is "Moonshot" project from Janet/Géant and the closely related abfab ietf wg

notTooFarGone

I do love this - everything that makes passwords less used makes the world more secure. Everything that is additionally user friendly has the potential to be the new let's encrypt.

bayindirh

I'd love to see the central repository to be breached and tons of computers get new users instantly.

I mean, the idea is nice. There's an alternative implementation being used already in some parts of the world, but their own OIDC provider of their choice.

Decentralization is the key here.

I can neither confirm nor deny the pun is intended.

notTooFarGone

If you lose your root CA certificate you sure are done for too.

Is it better than passwords? 100% - is it perfect? It does not have to be for a lot of use cases.

theamk

the central directory in this case is google.com (or github.com, or whatever your OIDC provider is)

If it gets breached, there will be significantly more problems than unauthorized ssh login.

(and this is a beauty of this compared to something like sshca: there is only one party that you need to trust, and you can choose a party that's unlikely to be breached)

gfody

ssh -k is too enterprise for ̶t̶e̶c̶h̶b̶r̶o̶ ̶s̶t̶a̶r̶t̶u̶p̶s̶ small companies that don't want to setup a kerberos realm

rcarmo

If you mention Kerberos to most "security" people these days they will think you're talking about Kubernetes.

dcow

really? that’s a shame

kerberos is old and clunky but conceptually it got so much right. I’m so sick of the modern idea that i should wake up and babysit my machine through N different oauth dances to log in to all the services i need on a daily basis. once I authenticate once I should be implicitly authenticated everywhere.

mschuster91

The problem is, even for large companies, Kerberos can be quite the pain. It's fine for fixed position desktop computers physically located on site or at a remote site with a hardware VPN tunnel - that was what it was built for.

But that is rarely the case any more. People use their own devices (BYOD) that aren't integrated into AD at all, they're using them outside of the office which means there is no VPN available at boot time to deal with token issuance, and the modern "zero trust" crap that uses weird packet filtering black magic instead of proper tun/tap virtual ethernet devices often doesn't play too nice with archaic authentication tools.

On top of that, implementing support for Kerberos in a Dockerized world is just asking for pain.

bradgessler

I started building an alternative to SSH at https://terminalwire.com that I think is more suitable for one-off commands run on a developer workstation against a SaaS. In more concrete terms, think of the stripe, heroku, and GitHub CLIs.

It’s similar to SSH in that it streams stdio from the server to a thin-client, but that’s where the similarities end. It has additional commands, like open a browser to a URL and set “cookies” on the client terminal from the sever.

When all of those commands are put together, you get something like https://github.com/terminalwire/demodx/blob/main/app/termina... that can open the users browser to an SSO, authorize the CLI, then set a token/nonce/whatever as a “cookie” in the users terminal so they can authenticate and run commands against the SaaS.

My intention isn’t to replace SSH—it’s still the best protocol for a lot of things, but I have found it cumbersome to use it to build CLIs for SaaS, which is why I built Terminalwire.

EthanHeilman

That's awesome. The web needs more terminal interfaces.

What did you use to build that slick explainer video?

teruakohatu

Interesting. Good luck with it.

tptacek

This is neat, and more people should be doing things like this. For what it's worth, we use (and like) Teleport, which does certificate-based SSH authentication; an SSO auth gets you a short-lived certificate. It also has the benefits of access control and (most importantly) audit logs; a generic reliable audit log for SSH sessions is a powerful tool to have for compliance stuff, since it transitively gives you an audit log for your CLI tools as well.

EthanHeilman

Is audit logs and access control the main features that would convince you to use Teleport vs something like opkssh? How important is the VPN functionality that lets you get packets to private IPs?

tptacek

I like that it's written in a memory-safe language too, but the killer feature is definitely the audit logs.

We keep all this stuff behind WireGuard, which is what I would recommend everybody do.

atonse

We use Tailscale for our WireGuard, which further integrates with SSO. So it’s a double whammy.

But Tailscale ssh has the identity stuff built in too.

antman

The open source version supports only GitHub SSO, is that your provider?

_hyn3

How does this compare to Userify's plain-jane SSH key technique?

That agent (Python, single-file https://github.com/userify/shim) sticks with decentralized regular keys and only centralizes the control plane, which seems to be more reliable in case your auth server goes offline - you can still login to your servers (obviously no new users or updates to existing keys). It just automates user and sudo configuration using things like adduser and /etc/sudoers.d. (It also actively kills user sessions and removes the user account when they're deleted, which is great for when you're walking someone out in case they have cron-jobs or a long-running tmux session with a revenge script.)

This project looks powerful but with a lot of heavy dependencies, which seem like an increased surface area (like Userify's Active Directory integration, but at least that's optional)

nullc

I believe the idea of this scheme is so that the NSA tailored access operations staff embedded in organizations such as google and cloudflare can authorize access without having to individually intercept each server (or jumphost) you own.

You benefit from more reliable shipping delivery times, no more mysterious city-of-industry->ftmeade->sanfrancisco detours or hardware that fails prematurely due to uncleaned flux or whiskers from implant installations.

EthanHeilman

If you really believe that then help me get the cosigner working with opkssh so even if Google is fully malicious they can't get ssh access.

EthanHeilman

Author of the blog post and main opkssh contributor here, happy to answer any questions.

ale42

The idea of using SSO for SSH sounds interesting for some applications, but does the login process really need to be browser-based? Can't we just have the login prompt in the terminal (without needing to run a headless browser behind-the-scenes of course)? I'm often working on headless machines, and other devices that definitely don't have a browser installed, and it would be pretty painful to use (not to mention that when working in a terminal, I find very annoying to have to switch to a browser, or to any other window by the way).

EthanHeilman

All the major OpenID Provider want you to use a browser so that users aren't exposing their raw credentials like passwords to an application. We did have an experimental version of this working in a terminal for integration tests a long time ago, but Google views that as malicious behavior and tries to prevent you from doing that. It turns into an arms race with the IdPs.

The good news is you only have to login through the browser once in the morning and then you can use the generated ssh key all day long.

fc417fc802

The browser is also an application. Of course I somehow doubt ELinks will be working with the Google auth page anytime soon.

It seems that the goal is to minimize the number of applications into which users view entering their credentials as "normal". The obvious missing piece then is some standardized FOSS project to handle CLI login.

Of course that is also unlikely to go over well since what the centralized providers actually want (which is fundamentally incompatible with user freedoms) is attestation. Arbitrarily blessing a few browser implementations that are extremely difficult to compile on your own is just a roundabout method to approximate that (IMO anyway).

Edit: It seems like a mistake to me to conflate anit-phishing efforts and IdP. Anti-phishing should be via hardware tokens or TOTP or whatever. IdP should be about large corporations managing user accounts or about individuals gaining convenience, including by serving as an adapter so that the latest popular standard can be used without each downstream service needing to adopt it.

kbolino

This is very interesting! It looks like there's a config file [1] to set up username <- email,issuer associations. It looks like multiple identities (from different IdPs even) can access the same username, which is useful. There's also a config file for allowed IdPs [2] specifying the expected client IDs (btw, the docs here say all fixed duration options are for "24 hours" even 48h and 1week). This does seem to impose the limitation of exactly one client ID per IdP, which could complicate rotating client IDs.

Walking this through, given that OpenID Connect is specifically mentioned vs. bare OAuth2, I assume the ID token signatures are themselves verified by looking up ${ISSUER_URI}/.well-known/openid-configuration and following the jwks_uri found there. Is the JWKS response cached? Can it be pre-seeded and/or replaced with an offline copy?

[1]: https://github.com/openpubkey/opkssh/blob/main/README.md#etc...

[2]: https://github.com/openpubkey/opkssh/blob/main/README.md#etc...

EthanHeilman

> This does seem to impose the limitation of exactly one client ID per IdP, which could complicate rotating client IDs.

Thanks for asking this. I don't see any reason why you use the same IdP with two different Client-IDs. I haven't tested this, but it is doesn't work currently, I'd like to add it as a feature.

Your description of the protocol is spot on. OpenPubkey currently only works with ID Tokens.

>Is the JWKS response cached? Can it be pre-seeded and/or replaced with an offline copy?

Currently the JWKS response is not cached, but caching support is a feature we want to add.

What is the interest in a pre-seeded copy? Availability concerns?

kbolino

I can think of two different uses for multiple client IDs with the same IdP: rotation as I mentioned already (e.g. if client secret leaks, although the secret is not used here), and for when there are multiple domains in use but the IdP clients are configured for internal use only. Both are pretty niche though. I think most IdPs will let you keep two client IDs active at the same time, so the rotation use case might already be covered.

As for pre-seeded/offline JWKS, yeah the biggest concern is around availability. The pre-seeded case would handle fresh VM setups before networking might be fully configured (though other auth methods as fallback would be good enough in most cases, I think). Completely offline JWKS would also be useful for machines with no outbound connectivity. Both use cases are again pretty niche though.

notTooFarGone

We are currently struggling with the exact ergonomics of user friendly and secure ssh and I just wanted to say you helped big time here!

Will test this for my current use-case and hopefully contribute in the future!

EthanHeilman

Excellent, feel free to email me at ethan.r.heilman[at]gmail[]com. Happy to in anyway

apitman

1. Can I use my own indiehosted OP?

2. Looks like this streamlines the server trusting the client. Does it do anything for the client trusting the server? I feel gross saying it, but I almost wonder if we should be moving towards some sort of TLS-based protocol for remote login rather than doubling down on SSH, due to the assurances provided by CAs.

deng

What would you say is the advantage of this approach over integrating OIDC into a separate service, like what Ubuntu is trying with authd?

(see https://ubuntu.com/blog/authd-oidc-authentication-for-ubuntu...)

kbolino

Some key differences I observe:

OPKSSH covers only logging in through SSH to an existing user account, while authd covers all forms of login (console, graphical, SSH) and user/group management. The latter makes it much more of a full AAA product rather than just a new way to login with SSH. This means it's a deeper investment, with implications for network file systems (as covered in the docs), while OPKSSH can be added on top of just about any existing infrastructure.

In terms of process, authd uses the Device Authorization Flow to handle logins, which is more vulnerable to phishing. It also requires both sides to have online access to the IdP, whereas the ID token-based approach of OPKSSH allows the authenticating side to have no (*) or limited outbound connectivity. Also, authd seems to support only Microsoft and Google as IdPs right now, whereas OPKSSH (since it builds on OpenPubkey) supports any OpenID Connect IdP.

* = In theory, at least; the current implementation doesn't fully deliver on this, though the one online resource it does need is fairly static and quite cacheable

EthanHeilman

Thanks for that link, I hadn't heard about this. It looks really cool and I am glad to see ubuntu doing this. I should send ubuntu an email.

Sadly I can not offer an opinion as I don't know how authd works. I intend to find.

samcat116

I can't tell the benefits of this vs running an SSH CA that supports OIDC. In that scenario, the server just needs to trust the CAs key, rather than running some sort of verifier.

EthanHeilman

The benefits of this is that you don't have the attack surface of an SSH CA. If you do this with an SSH CA that supports OIDC, if either the IDP or the SSH CA are compromised then security is lost.

With OpenPubkey and by extension opkssh, your IDP is functioning like the SSH CA by signing the public key that you make the SSH connection with. Thus, you have one fewer trusted party and you don't have maintain and secure an SSH CA.

Beyond this, rotating SSH CAs is hard because you need to put the public key of the SSH CA on each SSH server and SSH certs don't support intermediate certificates. Thus if you SSH CA is hacked, you need to update the CA public key on all your servers and hope you don't miss any. OpenID Connect IDPs rotate their public keys often and if they get hacked and they can immediately rotate their public keys without any update to relying servers.

c45y

You can have multiple trusted CAs which I've found to make rotation a non issue with tooling like Ansible etc.

New CA is minted, public key is added to the accepted list, client signing start using the new CA and you remove the old after a short while.

If missing servers is a common problem it sounds like there are some other fundamental problems outside just authenticated user sessions.

EthanHeilman

That's smart! You could probably automate that using a cron job pulling the latest CA public keys so servers automatically rotates CA public keys every few days.

> If missing servers is a common problem it sounds like there are some other fundamental problems outside just authenticated user sessions.

On one hand yes, on the other hand that is just the current reality in large enterprises. Consider this quote from Tatu Ylonen's (Inventor of SSH) recent paper [0]

“In many organizations – even very security-conscious organizations – there are many times more obsolete authorized keys than they have employees. Worse, authorized keys generally grant command-line shell access, which in itself is often considered privileged. We have found that in many organizations about 10% of the authorized keys grant root or administrator access. SSH keys never expire.”

If authorized keys get missed, servers are going to get missed.

opkssh was partially inspired by the challenges presented in this paper.

[0]: Challenges in Managing SSH Keys – and a Call for Solutions https://ylonen.org/papers/ssh-key-challenges.pdf

johnmaguire

Years ago, I tried building something like this using ProxyCommand to try to fetch the SSH certificate "just-in-time" without having to run a command first, but unfortunately the ordering of OpenSSH was such that ProxyCommand ran after checking the disk for SSH certs/keys. :(

EthanHeilman

I got this working at one point.

The trick is to use your SSH config to intercept SSH connections so the got to a local SSH server, this triggers ProxyCommand and let's you create the cert and then forward those packets into an outgoing SSH connection you don't intercept.

SSH --> Local SSH Server --> ProxyCommand (create cert) --> SSH --> Remote SSH Server

Foxboron

You could use `host match exec` instead of `ProxyCommand`. I believe it will run before you end up checking for files on disk.

password4321

Semi-related OpenSSH fork supporting auth with X.509 certificate CA: https://roumenpetrov.info/secsh/

And a walkthrough (2020): http://tech.ciges.net/blog/openssh-with-x509-certificates-ho...

dizhn

I like that they used (and abused) standard ssh server features to implement this.

Is anybody aware of something like this that can be automated for things like ansible or see a way to use this there?

bzmrgonz

has this project been audited tho? Because it seems to me we are shifting the authentication process to opkssh, so the question then becomes, how secure is the code-build?

Aeolun

I think it’s kinda funny that a standard to return a public key in a token, and a server side auth binary that uses that to log you into SSH, are presented here as something groundbreaking.

I’m not trying to downplay actually doing it, but it’s been possible since openid connect was invented.

EthanHeilman

> It’s been possible since openid connect was invented.

It has been possible since OpenID Connect was invented but figuring out how to get a public key into an ID Token without having to update IDPs or change the protocol in anyway was not known until we published OpenPubkey[0]. OpenID Connect was not designed to do this.

Figuring out how to smuggle this additional information into OpenSSH without requiring code changes or adding a SSH CA required a significant amount of work. I could be wrong but as far as I am aware the combined use of smuggling data in SSH public keys with AuthorizedKeyCommand to validate that data was not done until opkssh.

This was three years of careful work of reading through OpenID Connect specs, SSH RFCs, reading OpenSSH source code to get this to be fully compatible with existing IDPs and OpenSSH.

[0]: OpenPubkey: Augmenting OpenID Connect with User held Signing Keys (2023) https://eprint.iacr.org/2023/296

Aeolun

I didn’t mean to downplay the amount of work involved. It’s just that it feels to me like the ‘solution’ to problems like these seem very simple when that work has been put in.

It’s just that nobody really wants to (OpenID connect became a lot easier to understand when I read the spec, but I never got anywhere close to enjoying it), hence, we didn’t have this until now.

motoboi

Openid is just a bunch of http requests and browser redirects, if you think about it.