Popular GitHub Action tj-actions/changed-files is compromised
89 comments
·March 15, 2025mubou
ashishb
> Can you really say you trust all of these?
We need better capabilities. E.g. when I run `fd`, `rg` or similar such tool, why should it have Internet access?
IMHO, just eliminating Internet access for all tools (e.g. in a power mode), might fix this.
The second problem is that we have merged CI and CD. The production/release tokens should ideally not be on the same system as the ones doing regular CI. More users need access to CI (especially in the public case) than CD. For example, a similar one from a few months back https://blog.yossarian.net/2024/12/06/zizmor-ultralytics-inj...
hypeatei
OpenBSDs pledge[0] system call is aimed at helping with this. Although, it's more of a defense-in-depth measure on the maintainers part and not the user.
> The pledge() system call forces the current process into a restricted-service operating mode. A few subsets are available, roughly described as computation, memory management, read-write operations on file descriptors, opening of files, networking (and notably separate, DNS resolution). In general, these modes were selected by studying the operation of many programs using libc and other such interfaces, and setting promises or execpromises.
yencabulator
Pledge is for self-isolating, it helps with mistakes but not against intentional supply chain attacks.
redserk
I’ve been doing all of my dev work in a virtual machine as a way to clamp things down. I’ve even started using a browser in a VM as a primary browser.
Computers are fast enough where the overhead doesn’t feel like it’s there for what I do.
For development, I think Vagrant should make a comeback as one of the first things to setup in a repo/group of repos.
CamJN
You also need to block write access, so they can’t encrypt all your files with an embedded public key. And read access so they can’t use a timing side channel to read a sensitive file and pass that info to another process with internet privileges to report the secret info back to the bad guy. You get the picture, I’m sure.
ashishb
> You also need to block write access, so they can’t encrypt all your files with an embedded public key. And read access so they can’t use a timing side channel to read a sensitive file and pass that info to another process with internet privileges to report the secret info back to the bad guy. You get the picture, I’m sure.
Indeed.
One can think of a few broad capabilities that will drastically reduce the attack surface.
1. Read-only access vs read-write 2. Access to only current directory and its sub-directories 3. Configurable Internet access
Docker mostly gets it right. I wish there was an easy way to run commands under Docker.
E.g.
If I am running `fd`
1. Mount current read-only directory to Docker without Internet access (and without access to local network or other processes) 2. Run `fd` 3. Print the results 4. Destroy the container
ycombiredd
Yes, this...
I hope the irony is not completely lost on the fine folks at semgrep that the admittedly "overkill" suggested semgrep solution is exactly the type of pattern that leads to this sort of vulnerability: that of executing arbitrary code that is modifiable completely outside of one's own control.
usef-
Yes. Same with browser plugins. I've heard multiple free-plugin authors say they're receiving regular offers to purchase their projects. I'm sure some must take up the offer.
ronjouch
For an example of a scary list of such offers, see https://github.com/extesy/hoverzoom/discussions/670
remram
This is cool but useless because they redacted all the company names. The opposite of a name and shame, because no name and no shame.
Gigachad
I have long since stopped using any extension that doesn’t belong to an actual company (password managers for example). Even if they aren’t malware when you installed them, they will be after they get sold.
scrapcode
Are there examples of these types of actions in other circles outside of the .NET ecosystem? I knew about the FluentAssertions ordeal, but the Moq thing was news to me. I guess I've just missed it all.
do_not_redeem
node-ipc is a recent example from the Node ecosystem. The author released an update with some code that made a request to a geolocation webservice to decide whether to wipe the local filesystem.
sanex
Missed them too. Always was annoyed by FluentAssertions anyway, some contractor added it to a project that we took over couldn't see the value add.
puffybuf
Stealing crypto is so lucrative. So there is a huge 'market' for this stuff now that wasn't there before. Security is more important now than ever. I started sandboxing Emacs and python because I can't trust all the packages.
kubectl_h
npm supply chain attacks are the lone thing that keeps me up at night, so to speak. I shudder thinking about the attack surface.
I go out of my way to advocate for removing dependencies and pushing against small dependency introductions in a large ruby codebase. Some dependencies that suck and impose all sorts of costs, from funky ass idiosyncratic behavior or absurd file sizes (looking at you any google produced ruby library, especially the protocol buffer dependent libraries) are unavoidable, but I try to keep fellow engineers honest about introducing libraries that do things like determine the underlying os or whatever and push towards them just figuring that out themselves or, at the least, taking "inspiration" from the code in those libraries and reproducing behavior.
A nice side effect of AI agents and copilots is they can sometimes write "organic" code that does the same thing as third party libraries. Whether that's ethical, I don't know, but it works for me.
mh-
> eslint's dependency tree
And if you turn on devDependencies (top right), it goes from 85 to 1263.
Terr_
I'd also emphasize out that there's nothing safe about it being "only dev", given how many attacks use employee computers (non-prod) as a springboard elsewhere.
kilroy123
I agree completely.
If I see a useful extension, I want to use is on GitHub. I fork it. Sometimes I make a bookmarklet with the code instead.
I keep most extensions off until I need to use them. Then, I enable them, use them, and turn them off again. I try to even keep Mac apps to a minimum.
harrisi
It's always been shocking to me that the way people run CI/CD is just listing a random repository on GitHub. I know they're auditable and you pin versions, but it's crazy to me that the recommended way to ssh to a server is to just give a random package from a random GitHub user your ssh keys, for example.
This is especially problematic with the rise of LLMs, I think. It's the kind of common task which is annoying enough, unique enough, and important enough that I'm sure there are a ton of GitHub actions that are generated from "I need to build and deploy this project from GitHub actions to production". I know, and do, know to manually run important things in actions related to ssh, keys, etc., but not everyone does.
remram
People don't pin versions. Referencing a tag is not pinning a version, those can be updated, and they are even with the official actions from GitHub.
harrisi
Aren't GitHub action "packages" designate by a single major version? Something like checkout@v4, for example. I thought that that designated a single release as v4 which will not be updated?
I'm quite possibly wrong, since I try to avoid them as much as I can, but I mean.. wow I hope I'm not.
remram
No the "v4" tag gets updated from v4.1 to v4.2 etc as those minor versions are released. They are branches, functionally.
sestep
The crazier part is, people typically don't even pin versions! It's possible to list a commit hash, but usually people just use a tag or branch name, and those can easily be changed (and often are, e.g. `v3` being updated from `v3.5.1` to `v3.5.2`).
nextts
Fuck. Insecure defaults again. I argue that a version specifier should be only a hash. Nothing else is acceptable. Forget semantic versions. (Have some other method for determining upgrade compatibility you do out of band. You need to security audit every upgrade anyway). Process: old hash, new hash, diff code, security audit, compatibility audit (semver can be metadata), run tests, upgrade to new hash.
harrisi
You and someone else pointed this out. I only use GitHub-org actions, and I just thought that surely there would be a "one version to rule them all" type rule.. how else can you audit things?
I've never seen anything recommending specifying a specific commit hash or anything for GitHub actions. It's always just v1, v2, etc.
mcpherrinm
OpenSSF scorecard flags dependencies (including GitHub actions) which aren’t pinned by hash
https://github.com/ossf/scorecard/blob/main/docs/checks.md#p...
sieabahlpark
[dead]
dan_manges
GitHub Actions should use a lockfile for dependencies. Without it, compromised Actions propagate instantly. While it'd still be an issue even with locking, it would slow down the rollout and reduce the impact.
Semver notation rather than branches or tags is a great solution to this problem. Specify the version that want, let the package manager resolve it, and then periodically update all of your packages. It would also improve build stability.
nextts
Also don't het GH actions to do anything other than build and upload artifacts somewhere. Ideally a write only role. Network level security too no open internet.
Use a seperate system for deployments. That system must be hygienic.
This isn't foolproof but would make secrets dumping not too useful. Obviously an attack could still inject crap into your artefact. But you have more time and they need to target you. A general purpose exploit probably won't hurt as much.
cmckn
I always use commit hashes for action versions. Dependabot handles it, it’s a no brainer.
Terr_
> commit hashes
There is some latent concern that most git installations use SHA-1 hashes, as opposed to SHA-256.
Also the trick of creating a branch that happens to be named the same as a revision, which then takes precedence for certain commands.
password4321
creating a branch that happens to be named the same as a revision, which then takes precedence for certain commands
TIL; yikes! (and thanks)
mixologic
All the version tags got relabled to point to a compromised hash. Semver does nothing to help with this.
your build should always use hashes and not version tags of GHA's
mceachen
GitHub actions supports version numbers, version ranges, and even commit hashes.
frenchtoast8
The version numbers aren't immutable, so an attacker can just update the versions to point to the compromised code, which is what happened here. Commit hashes are a great idea, but you still need to be careful: lots of people use bots like Renovate to update your pinned hashes whenever a new version is published, which runs into the same problem.
werrett
Only commit hashes are safe. In this case the bad actor changed all of the version tags to point to their malicious commit. See https://github.com/tj-actions/changed-files/tags
All the tags point to commit `^0e58ed8` https://github.com/tj-actions/changed-files/commit/0e58ed867...
jasonthorsness
Since they edited old tags here … maybe GitHub should have some kind of security setting a repo owner can make that locks-down things like old tags so after a certain time they can't be changed.
CaliforniaKarl
In your GitHub Actions YAML, instead of referencing a specific tag, you can reference a specific commit. So, instead of …
uses: actions/checkout@v4
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683OptionOfT
That still doesn't help when the action is a docker action only marked with a tag.
So you need to check the action.yml itself to see if it has a sha256 pinned (in the case it uses Docker).
eddythompson80
You can always just fork it and reference your own fork.
postalrat
Or just write your own.
themgt
This is hilarious, the maven-lockfile project "Lockfiles for Maven. Pin your dependencies. Build with integrity" appears to have auto-merged a PR for the compromised action commit. So the real renovate bot immediately took the exfiltration commit from the fake renovate bot and started auto-merging it into other projects:
netvarun
@dang: The original URL (from Step Security, the company that discovered this flaw) is a better source for this:
https://www.stepsecurity.io/blog/harden-runner-detection-tj-...
moyer
Yeah maybe we can merge with https://news.ycombinator.com/item?id=43367987
nomilk
I'd used GitHub Actions for at least 6-12 months before even realising <thing>/<thing> was not something that got parsed by the action (like a namespace and method/function), but was simply a reference to a github user name and repo. That whole object should really have been called a 'repo', because that's what it is, and that would alert users to use extreme caution whenever using one that wasn't created by themselves.
londons_explore
So this dumps env to stdout using some obfustucated code? And then relies on the fact logs are viewable publicly so the attacker can go scrape your secrets.
If so, why did they use obfustucated code? Seems innocuous enough to load env into environment vars, and then later to dump all env vars as part of some debug routine. Eg. 'MYSQL env var not set, mysql integration will be unavailable. Current environment vars: ${dumpenv}'
mceachen
Presumably the cracker:
1. spoofed an account whose PRs were auto-merged (renovate[bot]) 2. found that `index.js` was marked as binary, and knew that GitHub is "helpful" (for the exploit), and hides diffs in the PR for that file by default 3. shoved the chunk of base64 wayyyy down the commit, so the maintainer had review fatigue by the time they scrolled. Having "memdump.py" in the commit in plaintext would certainly highlight the exploit more than the b64 string.
werrett
No idea. But they didn't do a great job -- they broke the action, which caused build failures that people were going to notice.
The malicious commit only landed at 09:57 PDT today (March 14) in one specific action (out of a number that is quite popular). Maybe they were planning on coming back and doing proper exfil?
oguz-ismail
It dumps entire process memory and requires sudo for that
random17
I wish Github required some sort of immutability for actions by default as most package managers do, either by requiring reusable actions to be specified via commit hash or by preventing the code for a published tag to be changed.
At the moment the convention is to only specify the tag, which is not only a security issue as we see here, but may also cause workflows to break if an action author updates the action.
wutwutwat
You can target `some/action@commithash` already, that's up to you. You're also free to fork or clone each action you use, vet the code, and consume your fork in your workflows. You can also disable the use of third party actions at an org level, or approve them on a case-by-case basis.
This all depends on your threat model and risk tolerance, it's not so much a GitHub problem. There will always be bad code that exists, especially on the largest open source code hosting platform. You defend against it because that's more realistic than trying to eradicate it.
jasonthorsness
I think this article from earlier today was the discoverer who opened the issue
btown
Does anyone know if https://github.com/tj-actions/verify-changed-files/ was compromised as well?
simonw
I've always felt uncomfortable adding other people's actions to my GitHub workflows, and this is exactly the kind of thing I was worried about.
I tend to stick to the official GitHub ones (actions/setup-python etc) plus the https://github.com/pypa/gh-action-pypi-publish one because I trust the maintainers to have good security habits.
In recent years, it's started to feel like you can't trust third-party dependencies and extensions at all anymore. I no longer install npm packages that have more than a few transitive dependencies, and I've started to refrain from installing vscode or chrome extensions altogether.
Time and time again, they either get hijacked and malicious code added, or the dev themselves suddenly decides to betray everyone's trust and inject malicious code (see: Moq), or they sell out to some company that changes the license to one where you have to pay hundreds of dollars to keep using it (e.g. the recent FluentAssertions debacle), or one of those happens to any of the packages' hundreds of dependencies.
Just take a look at eslint's dependency tree: https://npmgraph.js.org/?q=eslint
Can you really say you trust all of these?