Skip to content(if available)orjump to list(if available)

Supply Chain Attacks on Linux Distributions – Fedora Pagure

blueflow

Because bash for some goddamn reason loads the bashrc for interactive shells AND when started by sshd, regardless of whether the shell is interactive or an tty is present. Bash (and only bash) literally has a special case for sshd to enable this kind of exploit.

As a result of this, git and rsync wont work at all if the bashrc on the remote machine writes any data to stdout. Like setting a window title.

To work around that, every bashrc on this earth needs a case switch to return early to avoid this specific bug.

yjftsjthsd-h

Wait, ssh doesn't let you specify the remote command without running it through the user's shell? That seems like a deficiency too IMHO.

blueflow

Command strings must be expanded to an argument vector by some kind of shell. SSH itself does not allow to execute a program by argument vector like execv.

null

[deleted]

pjc50

This is sort of a feature that allows for restricted shells such as menu systems.

yjftsjthsd-h

It seems like ForcedCommand should work regardless?

kpcyrd

It's a limitation in the ssh protocol. I wish they would fix it, but I'm not holding my breath. Trying to do anything about it would be a compatibility nightmare.

If you need to pass data through ssh you're better off doing it through stdin.

pabs3

Suggestion for adding it is here, basically they were skeptical there are use-cases for it:

https://bugzilla.mindrot.org/show_bug.cgi?id=2283

nurettin

It is simpler to export paths locally, so the remote doesn't have to know your file/folder structure.

yjftsjthsd-h

Sure, it's convenient to have PATH, but why not have an optional way to say "hey, ssh, run /usr/bin/rsync on the remote with the following arguments, directly and without a shell please"? Equivalent to a Dockerfile containing

  cmd foo
vs

  cmd ["/bin/foo"]
(IIRC, at least - been a minute since I needed to do this.)

lrvick

For those looking for alternatives to the status quo on Linux supply chain security, check out [Stageˣ].

It is 100% deterministic, hermetic, and reproducible. Also it is full source bootstrapped from 180 bytes of human-auditable machine code, all released artifacts are multi-party reproduced/reviewed/signed, and it is fully container-native all the way down "FROM scratch" making it easy to pin dependency hashes to reproduce your own projects with it.

I started it after years of unsuccessful pleading with existing distros to stop giving ultimate solitary trust to -any- maintainers or sysadmins involved in the project.

https://codeberg.org/stagex/stagex

green7ea

This project looks amazing — I didn't think bootstrapping like this was possible. Kudos on the project :-).

This might displace chainguard as my goto docker images :-).

noman-land

Pardon my naivete but I've heard Nix described in many of the same terms. What are the differences and similarities between Stageˣ and Nix/NixOS?

lrvick

I am obviously pretty biased having authored a failed RFC to Nix to mandate signing, and having founded StageX, but here goes.

Unlike StageX, Nix has a wikipedia-style low friction approach to packaging allowing a large community to maintain a huge number of packages which are signed by a central trusted party, while being -mostly- reproducible. It relies on a custom language and toolchain for packaging. Nix is designed for hobbyists seeking to reproduce their own workstations.

Unlike Nix, StageX is 100% reproducible and all changes are independently signed by authors and reviewers, then artifacts are reproduced and signed by multiple distributed maintainers. It is also full source bootstrapped, and is built on the broadly compatible OCI standard for packaging. StageX is designed for production threat models where only compiler and infrastructure toolchains are needed and trusting any single person is unacceptable.

One maintainer even uses Nix as a workstation to contribute to StageX. They have fundamentally different goals and use cases.

cadamsdotcom

Nice writeup!

Thinking generally it seems something like the xz/lzma vulnerability could be snuck in by 1-2 nefarious people colluding across packaging and package producing, especially if we are talking about nation-state actors who can afford to look legit for years before being trusted to work without oversight - then when no one is watching, sneak in a backdoor.

I feel we are in a very innocent age and will look wistfully back at the days we trusted our anonymous open source brethren.

On macOS I think about this every time I “brew install”, and every time oh-my-zsh auto-updates. Do Linux users think about this?

tremon

I think about this every time I install software. As a citizen of a small European nation, 100% of the software I use is under control of a foreign government, and I trust none of them. At least with open source software, there is a better chance of nefarious changes being detected by at least one of the parties building and packaging the software. With proprietary software, even that small level of assurance is not available.

akimbostrawman

There is a very big risk difference between a upstream package like XZ and running random application with brew or god forbid zsh auto update.

>Do Linux users think about this?

Yes that's why i avoid packages not in the official distro repositories and where possible further minimize risks by using additional security layer such as sandboxing (flatpak and firejail), mandatory access control (Apparmor or SELinux), virtualization (KVM/Qemu) and application firewalls (Opensnitch).

mmh0000

Think about it all the time. But, I find it overwhelming and shutdown.

I'll just enumerate my random thoughts

* If you're worried about nations, remember that they can, quite literally, send ninjas-in-attack-helicopters at you.

* Most nations already have "laws" that "require" you to provide passwords/data access on demand[4ab].

* If you use any "cloud", there's a high likelihood that they're already backdoored, either through legal means (i.e. through National Security Letters) or "not so legal, but who's going to stop them?" means such as just plain old hacking[1]

* All consumer CPUs already have builtin backdoors that can't really (but kind of, but who really know if it's effective) be disabled[2abc]

* Most printers print secret codes on printed documents that link back to the printer[3]

* I have no control over device firmware and some important drivers. I really don't know what my network card firmware is doing when I'm not using it and it has DMA to my system RAM. I "need" nvidia proprietary drivers to have a decent experience, no idea what they actually do.

* Nearly every piece of software includes some form of "Analytics" or "Telemetry", which often doesn't actually turn off when you click the stupid opt-out button.

[1] https://www.npr.org/sections/thetwo-way/2013/10/30/241855353...

[2a] https://en.wikipedia.org/wiki/Intel_Management_Engine

[2b] https://en.wikipedia.org/wiki/AMD_Platform_Security_Processo...

[2c] https://en.wikipedia.org/wiki/ARM_architecture_family#TrustZ...

[3] https://en.wikipedia.org/wiki/Printer_tracking_dots

[4a] https://en.wikipedia.org/wiki/Key_disclosure_law [4b] https://www.eff.org/wp/digital-privacy-us-border-2017

registeredcorn

> I feel we are in a very innocent age and will look wistfully back at the days we trusted our anonymous open source brethren.

There seems to be some kind of inflection point of things:

* Enough users

* Enough time

* Enough cost

* Enough financial reward

Eventually, as those things culminate together, there is a loss of the shared value and...innocence(?) of any scene.

I think back to when RMS was aghast at the idea of MIT adding usernames and passwords to their machines in 1977. [1] It's a battle that I still feel bad that he lost against; a kind of presumption of universal access is a thing that I am intrinsically drawn to. It breaks my heart to think that any sort of restriction exists to technological access, in spite of the obvious need for computer security. Think about it - in the modern day, it's not only unthinkable, it's largely impossible to access most computers without some form of locked down authentication - even for publicly accessible computers at libraries! Most Operating System setups are designed with a fundamental assumption that you will be using both, regardless of whether it's a computer shared by 50 people in an office, or a personal laptop in your home, locked room that no other person will ever touch or see.

I sit here and I think about how there was a period where it was simply understood that, you treat computers and that network with a kind of prestige and respect. That you conduct yourself and the things you do in a manner that is good, because you have a desire for that thing to continue being used, accessible, and enjoyable by other people who have that same inner passion - that hunger! That, there did not need to be a threat of some kind of legal consequence - no laws even existed in regards to it. You did so...merely because you grasped that it was right, and you desired to do what was right. You did things right because you had a passion, a care, and a shared common interest among all of those other people to use those systems in a way that was good for all.

Sure, perhaps there were a few jokes and gags. A few fights or arguments over this or that. Plenty of "frivolous" things that could be done like play games and whatnot, but there was a kind of obvious social understanding that you did not do what is bad because it was bad. You wouldn't do bad things for the same reason why you wouldn't stop in the middle of a sidewalk and start using the bathroom in public - because its unacceptable to do so; it was inconsiderate to do so.

I feel like there is a flower of naivety that has wilted with time. It's not quite dead, but so many of its petals have dried up, fallen off, and crushed beneath the boot of financial incentive.

I imagine it to be similar to how the invention of the car must have felt. So much early optimism and obvious benefits. Only to be used to escape police. To transport booze and drugs. To steal from owners so it can be sold for parts. To kidnap children in. To hit pedestrians with. To use in wartime. It's a grim bleakness in life, that all things that can be cherished and enjoyed by the kindhearted, so horribly abused by the malcontent.

[1] https://en.wikipedia.org/wiki/Richard_Stallman#Harvard_Unive...

superb_dev

> We can’t change git’s shell to /sbin/nologin or /bin/false, or users wouldn’t be able to connect over SSH.

Git actually has a solution for this! I don’t know if it would work with the custom python stuff going on, but you can set the login shell to `git-shell`

amelius

Yeah, I tried that, but it doesn't work well with git-lfs (large file storage). At least, it didn't last time I tried.

asdffdasy

So, it works perfectly to most sane use cases with git.

diggan

Are you saying storing pointers to large files isn't sane to do in Git? What are your suggested solution for dealing with large assets you want versioned and easily accessible?

If you're really dogmatic about it, I guess you would have no dependency lock files either, but commit all that code directly to git instead of having references. Some people do that, so wouldn't be a huge surprise.

amelius

Git isn't that opinionated.

jbverschoor

Or just use git over https. And heck, if that's such a big problem, switch to a vcs that you can properly manage

xenophonf

You shouldn't use Git over HTTPS. With SSH, you can use a hardware authenticator that requires both proof of ownership (i.e., the unlock PIN) and proof of possession (i.e., physical touch) out of the box. That's technically possible over HTTPS, of course, but I have yet to see a Git server that works that way.

3np

Both HTTPS and SSH are perfectly adequate transports for git, both can be used securely, both have their footguns. No need to be tribal about it. Use what's appropriate in your situation.

jbverschoor

Why not? Your password manager or passkeys do the same thing.

Heck, just true the public key as the authentication header.

Or use authentication certificates.

If ssh is such a big problem, use something else

arccy

it supports the mac osxkeychain for storing credentials

https://git-scm.com/docs/gitcredentials#_available_helpers

awesome_dude

FTR the "Proof of possession" relies on a (specific) random number being received from a device.

It won't be long before that hardware device that is supposedly being held by a person becomes a soft device, which will then be impersonated.

Joel_Mckay

Agreed, the ssh service on many git servers like gitea use their own user specific process instances to handle the connection.

Combined with port-knocking and fail2ban, the setup has proven rather reliable over the years. The Go language can make surprisingly resilient servers if you have the memory available.

The ssh key handling in gitea requires manual setup, and thus does not necessarily even have to use the same administrative user login key sets.

Best regards =3

ufo

I've got to say, Git resignifying -- and requiring --end-of-options instead is bonkers

musicale

The pathway of untrusted/malicious input -> trusted command line argument seems to be a common problem, and one that could possibly be mitigated by better type/taint checking.

It looks like there is some prior work in this area, but it hasn't resulted in commonly available implementations (even something basic like a type/taint checking version of exec() etc. on one side and getopt() etc. on the other.)

PhilipRoman

I could've sworn I remember something about bash and glibc cooperating to indicate which arguments come from expanded variables but I cannot find anything on the internet or in the sources. Either I'm going insane or it was an unmerged proposal.

BobbyTables2

Seems like it would be quite painful to do that in C without heavy refactoring.

Maybe a Rust alternative ?

pests

Sadly due to legacy.

—- disambiguates revisions and paths in some commands so another option was needed.

immibis

Just write all commands with structured I/O instead, like Powershell.

Now I want an operating system where everything is a YANG model...

INTPenis

>In addition, this is a self-service application, in the sense that anyone can create a Fedora contributor account and gain authenticated access to various services.

I legitimately wanted to get a package into Fedora a few years ago, a service that did not exist already, and I couldn't get past the fact that they require new contributor accounts to be sponsored by someone already a contributor. I was unable to secure sponsorship by anyone and just gave up.

pluto_modadic

this could be a useful mechanism actually... shame it didn't work out

ForOldHack

Then you have someone to blame for sponsoring an exploit vs some unsponsored person who identified the same exploit patched it, and no one bothered to check either way. I guess I got lucky never trusting Mint. I did trust Slackware, because I knew someone who trusted Pat V. I guess I also need to rub shoulders with Linux maintainers vs people like Marc Andressen. I did meet RMS, but he has no personality. Woz? Wolfram? Anna V? I met the SUSE people, and one of the RedHat maintainers, who gave me a tee-shirt, a poster, two bumper stickers, and the latest distribution of RedHat. It was easier a decade ago...

Despite the amount of brilliance here, again, we never have had a single meet and greet.

INTPenis

Agreed, I have been a RHEL ecosystem user for 11 years now. My experience in the Fedora community was actually comforting to me.

ForOldHack

Now that is cool.

guillem_lefait

NoLimitSecu, French cybersecurity podcast, released an episode yesterday with the authors: https://www.nolimitsecu.fr/compromission-de-distributions-li...

It was amazing to hear that they chose the weakest path, argument injection and were able to found a vector in two weeks twice (fedora + opensuse).

bbarnett

Does anyone else have the title overlay taking up 2/7ths of the top of the screen?

TZubiri

"Let's use red hat's products but let's not get them from red hat to save some bucks"

This is the risk you take