Tj-actions/changed-files GitHub Action Compromised – used by over 23K repos
223 comments
·March 14, 2025rarkins
diggan
> 6. Some people had automerging of such updates enabled, but this is not Renovate's default behavior. Even without automerging, an action like this might be able to achieve its aim only with a PR, if it's run as part of PR builds
I'm not sure how this could exploited by just making a PR, unless you for some reason have secrets enabled for builds by unknown contributors, which obviously would be a mistake. Usually, only builds using secrets only run on certain branches which has a known contributor approving the code before it gets there.
> people mistakenly assume that git tags are immutable
If you're distributing a library on GitHub used by many other people/projects, then you really need to setup `protected branches` and `protected tags`, where you can prevent changes somewhat.
semiquaver
> I'm not sure how this could exploited by just making a PR, unless you for some reason have secrets enabled for builds by unknown contributors
mlor
Thanks for taking the time to comment. Not that it wasn't there before this, but this incident highlights a lot to take into consideration with respect to securing one's supply chain going forward.
mubou
In recent years, it's started to feel like you can't trust third-party dependencies and extensions at all anymore. I no longer install npm packages that have more than a few transitive dependencies, and I've started to refrain from installing vscode or chrome extensions altogether.
Time and time again, they either get hijacked and malicious code added, or the dev themselves suddenly decides to betray everyone's trust and inject malicious code (see: Moq), or they sell out to some company that changes the license to one where you have to pay hundreds of dollars to keep using it (e.g. the recent FluentAssertions debacle), or one of those happens to any of the packages' hundreds of dependencies.
Just take a look at eslint's dependency tree: https://npmgraph.js.org/?q=eslint
Can you really say you trust all of these?
ashishb
> Can you really say you trust all of these?
We need better capabilities. E.g. when I run `fd`, `rg` or similar such tool, why should it have Internet access?
IMHO, just eliminating Internet access for all tools (e.g. in a power mode), might fix this.
The second problem is that we have merged CI and CD. The production/release tokens should ideally not be on the same system as the ones doing regular CI. More users need access to CI (especially in the public case) than CD. For example, a similar one from a few months back https://blog.yossarian.net/2024/12/06/zizmor-ultralytics-inj...
nextaccountic
> We need better capabilities. E.g. when I run `fd`, `rg` or similar such tool, why should it have Internet access?
Yeah!! We really need to auto sandbox everything by default, like mobile OSes. Or the web.
People browse the web (well, except Richard Stallman) all the time, and run tons of wildly untrusted code, many of them malicious. And apart from zero days here and there, people don't pay much attention to it, and will happily enter any random website in the same machine they also store sensitive data.
At the same time, when I open a random project from Github on VSCode, it asks whether the project is "trusted". If not, it doesn't run the majority of features like LSP server. And why not? Because the OS doesn't sandbox stuff by default. It's maddening.
redserk
I’ve been doing all of my dev work in a virtual machine as a way to clamp things down. I’ve even started using a browser in a VM as a primary browser.
Computers are fast enough where the overhead doesn’t feel like it’s there for what I do.
For development, I think Vagrant should make a comeback as one of the first things to setup in a repo/group of repos.
bombcar
https://www.qubes-os.org/ is the extension of this.
hypeatei
OpenBSDs pledge[0] system call is aimed at helping with this. Although, it's more of a defense-in-depth measure on the maintainers part and not the user.
> The pledge() system call forces the current process into a restricted-service operating mode. A few subsets are available, roughly described as computation, memory management, read-write operations on file descriptors, opening of files, networking (and notably separate, DNS resolution). In general, these modes were selected by studying the operation of many programs using libc and other such interfaces, and setting promises or execpromises.
yencabulator
Pledge is for self-isolating, it helps with mistakes but not against intentional supply chain attacks.
mnahkies
I've found firejail to be a useful tool for this (https://github.com/netblue30/firejail), and additionally use opensnitch (https://github.com/evilsocket/opensnitch) to monitor for unexpected network requests.
For CI/CD using something like ArgoCD let's you avoid giving CI direct access to prod - it still needs write access to a git repo, and ideally some read access to Argo to check if deployment succeeded but it limits the surface area.
varunsharma07
Great points! Harden-Runner (https://github.com/step-security/harden-runner) is similar to Firejail and OpenSnitch but purpose-built for CI/CD context. Harden-Runner detected this compromise due to an anomalous outbound network request to gist.githubusercontent.com.
Interestingly, Firejail itself uses Harden-Runner in its GitHub Actions workflows! https://github.com/search?q=repo%3Anetblue30%2Ffirejail%20ha...
homebrewer
bubblewrap is a safer alternative to firejail because it does not use setuid to do its job, and it is used by flatpak (so hopefully has more eyes on it, but I have no idea).
https://wiki.archlinux.org/title/Bubblewrap
You do have to assemble isolation scripts by hand though, it's pretty low level. Here is a decent comment which closely aligns to what I'm using to isolate npm/pnpm/yarn/etc, I see no need to repeat it:
CamJN
You also need to block write access, so they can’t encrypt all your files with an embedded public key. And read access so they can’t use a timing side channel to read a sensitive file and pass that info to another process with internet privileges to report the secret info back to the bad guy. You get the picture, I’m sure.
pdimitar
I get the picture, yes, namely that probably 99% of project dependencies don't need I/O capabilities at all.
And even if they do, they should be controlled in a granular manner i.e. "package org.ourapp.net.aws can only do network and it can only ping *.aws.com".
Having finer-grained security model that is enforced at a kernel level (and is non-circumventable barring rootkits) is like 20 years overdue at this point.
Every single big org is dragging their feet.
ashishb
> You also need to block write access, so they can’t encrypt all your files with an embedded public key. And read access so they can’t use a timing side channel to read a sensitive file and pass that info to another process with internet privileges to report the secret info back to the bad guy. You get the picture, I’m sure.
Indeed.
One can think of a few broad capabilities that will drastically reduce the attack surface.
1. Read-only access vs read-write 2. Access to only current directory and its sub-directories 3. Configurable Internet access
Docker mostly gets it right. I wish there was an easy way to run commands under Docker.
E.g.
If I am running `fd`
1. Mount current read-only directory to Docker without Internet access (and without access to local network or other processes) 2. Run `fd` 3. Print the results 4. Destroy the container
h4ck_th3_pl4n3t
But that's what firejail and docker/podman are for. I never run any build pipeline on my host system, and neither should you. Build containers are pretty good for these kind of mitigations of security risks.
mschuster91
> We need better capabilities.
I'd love to say "just use Kubernetes and run Nexus as a service inside" but unfortunately Network Policies are seriously limited [1]...
[1] https://kubernetes.io/docs/concepts/services-networking/netw...
from-nibly
This is the death of fun. Like when you had to use SSL for buying things online.
Adding SSL was not bad, don't get me wrong. It's good that it's the default now. However. At one point it was sorta risky, and then it became required.
Like when your city becomes crime ridden enough that you have to lock your car when you go into the grocery store. Yeah you probably should have been locking it the whole time. what would it have really cost? But now you have to, because if you don't your car gets jacked. And that's not a great feeling.
asveikau
In the era of the key fob it's pretty automatic to lock the car every time. Some cars even literally do it for you. I hardly think of this, let alone get not great feelings about it.
aorloff
Just you wait. Here in America when your city becomes crime ridden enough you start leaving it unlocked again.
asveikau
Crime is lower than the 80s and 90s. It has been declining since 2023.
mubou
Used to live near San Francisco, and had a lot of coworkers say they intentionally leave their windows down when parking in SF so that burglars don't break the glass to steal something!
nozzlegear
On the other extreme, I can (and do) leave my keys inside my running car while I shop for groceries!
usef-
Yes. Same with browser plugins. I've heard multiple free-plugin authors say they're receiving regular offers to purchase their projects. I'm sure some must take up the offer.
ronjouch
For an example of a scary list of such offers, see https://github.com/extesy/hoverzoom/discussions/670
mubou
This is why I fork the extensions I use, with the exception of uBlock. Basically just copy the extension folder, if I can't find it on GitHub. That way I can audit the code and not have to worry about an auto-update sneaking in something nefarious. I've had two extensions in the past suddenly start asking for permissions they definitely did not need, and I suspect this is why.
Btw, here's a site where you can inspect an extension's source code before you install it: https://robwu.nl/crxviewer/
remram
This is cool but useless because they redacted all the company names. The opposite of a name and shame, because no name and no shame.
_boffin_
do you know of any other ones like this that post their offers?
Gigachad
I have long since stopped using any extension that doesn’t belong to an actual company (password managers for example). Even if they aren’t malware when you installed them, they will be after they get sold.
stockhorn
A bit off topic, but how is the bitwarden browser extension protected against supply-chain attacks (npm dependencies)?
fluidcruft
Actual companies also get sold and churned into shit. See LastPass for example.
from-nibly
I got an outreach for an extension I made as a joke. It had like maybe 5000 downloads ever.
touristtam
I have used https://github.com/lirantal/npq for a good while now, but I am yearning for that'd look deeper into the health of the package at hand.
icetank
49 modules with only one maintainer and over 600 modules with only one maintainer if devDependencies are included. This is only a matter of time until the next module becomes compromised.
ozim
You should never have trusted blindly in third-party dependencies in the first place.
Abnormal behavior was to trust by default.
mh-
> eslint's dependency tree
And if you turn on devDependencies (top right), it goes from 85 to 1263.
Terr_
I'd also emphasize out that there's nothing safe about it being "only dev", given how many attacks use employee computers (non-prod) as a springboard elsewhere.
puffybuf
Stealing crypto is so lucrative. So there is a huge 'market' for this stuff now that wasn't there before. Security is more important now than ever. I started sandboxing Emacs and python because I can't trust all the packages.
semi-extrinsic
What do you use for sandboxing?
ebfe1
Doing a bit of investigation with github_events in clickhouse, it is quite clear that the accounts used to perform the attack was "2ft2dKo28UazTZ", "mmvojwip" also seems suspicious:
https://play.clickhouse.com/play?user=play#c2VsZWN0ICogZnJvb...
Actions taken by the threat actor at the time can be seen here:
https://play.clickhouse.com/play?user=play#c2VsZWN0ICogZnJvb...
ebfe1
It seems i forgot to cater for the quota applied to free "play" user in ClickHouse in my previous query... In fact, the threat actor did a lot more... this should give a better list of actions that was performed - Clearly showed he was testing his payload:
https://play.clickhouse.com/play?user=play#c2VsZWN0ICogZnJvb...
ebfe1
Note that these account seems to be deleted now - 2ft2dKo28UazTZ clearly did more than just changed-files and also seem to target coinbase/agentkit as well (Actually .. they might be targeted by the threat actor)
harrisi
It's always been shocking to me that the way people run CI/CD is just listing a random repository on GitHub. I know they're auditable and you pin versions, but it's crazy to me that the recommended way to ssh to a server is to just give a random package from a random GitHub user your ssh keys, for example.
This is especially problematic with the rise of LLMs, I think. It's the kind of common task which is annoying enough, unique enough, and important enough that I'm sure there are a ton of GitHub actions that are generated from "I need to build and deploy this project from GitHub actions to production". I know, and do, know to manually run important things in actions related to ssh, keys, etc., but not everyone does.
remram
People don't pin versions. Referencing a tag is not pinning a version, those can be updated, and they are even with the official actions from GitHub.
jakub_g
I think a big part of the problem is the way one typically "installs" a GH action: by copy-pasting something from README of the action.
Let's have a look at a random official GH provided action:
https://github.com/actions/checkout
It lists the following snippet:
`uses: actions/checkout@v4`
Almost everyone will just copy paste this snippet and call it a day. Most people don't think twice that v4 is a movable target that can be compromised.
In case of npm/yarn deps, one would often do the same, and copy paste `yarn install foobar`, but then when installing, npm/yarn would create a lockfile and pin the version. Whereas there's no "installer" CLI for GH actions that would pin the version for you, you just copy-paste and git push.
To make things better, ideally, the owners of actions would update the workflows which release a new version of the GH action, to make it update README snippet with the sha256 of the most recent release, so that it looks like
`uses: actions/checkout@abcdef9876543210` # v4.5.6
Since GitHub doesn't promote good defaults, it's not surprising that third-party maintainers do the same.
harrisi
Aren't GitHub action "packages" designate by a single major version? Something like checkout@v4, for example. I thought that that designated a single release as v4 which will not be updated?
I'm quite possibly wrong, since I try to avoid them as much as I can, but I mean.. wow I hope I'm not.
remram
No the "v4" tag gets updated from v4.1 to v4.2 etc as those minor versions are released. They are branches, functionally.
sestep
The crazier part is, people typically don't even pin versions! It's possible to list a commit hash, but usually people just use a tag or branch name, and those can easily be changed (and often are, e.g. `v3` being updated from `v3.5.1` to `v3.5.2`).
nextts
Fuck. Insecure defaults again. I argue that a version specifier should be only a hash. Nothing else is acceptable. Forget semantic versions. (Have some other method for determining upgrade compatibility you do out of band. You need to security audit every upgrade anyway). Process: old hash, new hash, diff code, security audit, compatibility audit (semver can be metadata), run tests, upgrade to new hash.
harrisi
You and someone else pointed this out. I only use GitHub-org actions, and I just thought that surely there would be a "one version to rule them all" type rule.. how else can you audit things?
I've never seen anything recommending specifying a specific commit hash or anything for GitHub actions. It's always just v1, v2, etc.
evntdrvn
it is documented as recommended here fwiw: https://docs.github.com/en/actions/security-for-github-actio...
mcpherrinm
OpenSSF scorecard flags dependencies (including GitHub actions) which aren’t pinned by hash
https://github.com/ossf/scorecard/blob/main/docs/checks.md#p...
kalaksi
So much this. I recently looked into using GitHub Actions but ended up using GitLab instead since it had official tools and good docs for my needs. My needs are simple. Even just little scripting would be better than having to use and audit some 3rd party repo with a lot more code and deps.
And if you're new, and the repo aptly named, you may not realize that the action is just some random repo
sieabahlpark
[dead]
Sytten
I am surprised nobody here mentionned immutable github actions that are coming [1]. Been waiting for them since the issue was open in 2022. This would have significantly reduce impact and hopefully github will get it over the finish line.
I always fork my actions or at least use a commit hash.
ricardobeat
I thought actions were already immutable and published to a registry, not fetched directly from their repo. TIL.
Go also uses tags for module versioning, and while go.mod or package-lock.json stop this attack from reaching existing consumers, allowing remapping of all versions to the compromised one still expands the impact surface a lot. GitHub should offer a “immutable tags” setting for repos like these.
null
kurmiashish
Disclaimer: I am a co-founder of StepSecurity.
StepSecurity Harden-Runner detected this security incident by continuously monitoring outbound network calls from GitHub Actions workflows and generating a baseline of expected behaviors. When the compromised tj-actions/changed-files Action was executed, Harden-Runner flagged it due to an unexpected endpoint appearing in the network traffic—an anomaly that deviated from the established baseline. You can checkout the project here: https://github.com/step-security/harden-runner
cyrnel
The advertising in this article is making it actively difficult to figure out how to remediate this issue. The "recovery steps" section just says "start our 14 day free trial".
The security industry tolerates self-promotion only to the extent that the threat research benefits everyone.
kurmiashish
Thank you, cyrnel, for the feedback! We are trying our best to help serve the community. Now, we have separate recovery steps for general users and our enterprise customers.
shawabawa3
A simpler method to detect this would be to store GitHub action tag hashes and freeze an action if any tag is changed
themgt
This is hilarious, the maven-lockfile project "Lockfiles for Maven. Pin your dependencies. Build with integrity" appears to have auto-merged a PR for the compromised action commit. So the real renovate bot immediately took the exfiltration commit from the fake renovate bot and started auto-merging it into other projects:
sureIy
The fun part is that they used commits specifically for security, but then add an auto-updater. Might as well use tags.
mdaniel
heh, timing is everything https://github.com/chains-project/maven-lockfile/issues/1085...
> After some cleanup the changed-files (https://github.com/tj-actions/changed-files) action seems to be more work to remove. It would be awesome if it could be added to the allowlist
> Done. Allowed all versions of this action. Should I pin it to one version in the allowlist (won't be convenient if renovate updates this dependency)?
dan_manges
GitHub Actions should use a lockfile for dependencies. Without it, compromised Actions propagate instantly. While it'd still be an issue even with locking, it would slow down the rollout and reduce the impact.
Semver notation rather than branches or tags is a great solution to this problem. Specify the version that want, let the package manager resolve it, and then periodically update all of your packages. It would also improve build stability.
nextts
Also don't het GH actions to do anything other than build and upload artifacts somewhere. Ideally a write only role. Network level security too no open internet.
Use a seperate system for deployments. That system must be hygienic.
This isn't foolproof but would make secrets dumping not too useful. Obviously an attack could still inject crap into your artefact. But you have more time and they need to target you. A general purpose exploit probably won't hurt as much.
mixologic
All the version tags got relabled to point to a compromised hash. Semver does nothing to help with this.
your build should always use hashes and not version tags of GHA's
cmckn
I always use commit hashes for action versions. Dependabot handles it, it’s a no brainer.
Terr_
> commit hashes
There is some latent concern that most git installations use SHA-1 hashes, as opposed to SHA-256. [0]
Also the trick of creating a branch that happens to be named the same as a revision, which then takes precedence for certain commands.
password4321
creating a branch that happens to be named the same as a revision, which then takes precedence for certain commands
TIL; yikes! (and thanks)
mceachen
GitHub actions supports version numbers, version ranges, and even commit hashes.
werrett
Only commit hashes are safe. In this case the bad actor changed all of the version tags to point to their malicious commit. See https://github.com/tj-actions/changed-files/tags
All the tags point to commit `^0e58ed8` https://github.com/tj-actions/changed-files/commit/0e58ed867...
frenchtoast8
The version numbers aren't immutable, so an attacker can just update the versions to point to the compromised code, which is what happened here. Commit hashes are a great idea, but you still need to be careful: lots of people use bots like Renovate to update your pinned hashes whenever a new version is published, which runs into the same problem.
jasonthorsness
Since they edited old tags here … maybe GitHub should have some kind of security setting a repo owner can make that locks-down things like old tags so after a certain time they can't be changed.
CaliforniaKarl
In your GitHub Actions YAML, instead of referencing a specific tag, you can reference a specific commit. So, instead of …
uses: actions/checkout@v4
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683OptionOfT
That still doesn't help when the action is a docker action only marked with a tag.
So you need to check the action.yml itself to see if it has a sha256 pinned (in the case it uses Docker).
eddythompson80
You can always just fork it and reference your own fork.
postalrat
Or just write your own.
oefrha
> https://github.com/tj-actions/changed-files/pull/2460
This kind of auto dependency bump bots are more trouble than their worth. If your app works today, bumping random deps won’t make it work better in any meaningful sense in 95% of cases. With such a small upside, the downside of introducing larger attack surfaces, subtle breakages (despite semver), major breakages, and in the worst cases, compromises (whether it’s a compromised dep, or fake bot commits that people are trained to ignore) just completely outweighs the upside. You’re on the fast lane to compromises by using this kind of crap.
People should really learn from Go’s minimum version selection strategy.
lostmsu
Your app will have unpatched vulnerabilities.
oefrha
As long as you subscribe to security advisories, it’s a lot more likely that new vulnerabilities are introduced than old undiscovered vulnerabilities are accidentally patched. In fact barring rewrites (which usually won’t be picked up by semver-respecting auto bumps anyway) I can hardly think of an example of the latter.
alper
GitHub’s incident response to this took 17 hours give or take.
Actions is a paid service but Microsoft probably replaced all the security teams with AI.
jasonthorsness
I think this article from earlier today was the discoverer who opened the issue
londons_explore
So this dumps env to stdout using some obfustucated code? And then relies on the fact logs are viewable publicly so the attacker can go scrape your secrets.
If so, why did they use obfustucated code? Seems innocuous enough to load env into environment vars, and then later to dump all env vars as part of some debug routine. Eg. 'MYSQL env var not set, mysql integration will be unavailable. Current environment vars: ${dumpenv}'
mceachen
Presumably the cracker:
1. spoofed an account whose PRs were auto-merged (renovate[bot]) 2. found that `index.js` was marked as binary, and knew that GitHub is "helpful" (for the exploit), and hides diffs in the PR for that file by default 3. shoved the chunk of base64 wayyyy down the commit, so the maintainer had review fatigue by the time they scrolled. Having "memdump.py" in the commit in plaintext would certainly highlight the exploit more than the b64 string.
captn3m0
Sounds about right to me. We can use a few knowns about GitHub IAM to deduce a few things:
1. There are no deleted PRs or Issues on the repo (2461..2463 are all valid refs)
2. A legitimate `Renovate[Bot]` dep bump would have filed a PR. Last such PR was 5 days ago, and is presumably not the source for this. (I haven't gone through every dep change, but doesn't look like it).
3. That leaves us with the 0e58ed867 commit, which has to be a spoofed commit, since it doesn't belong to a branch and we don't have a corresponding PR(1). A complete takeover of the repo can result in a hanging commit (by deleting the renovate bump branch), but there must be a hanging PR-ref around. Since there isn't one:
4. All of the above points to a compromised account that has write access to the repo.
There is also the https://github.com/tj-actions-bot account, but unclear if it has write access.
Edit: gurchik's guess at https://github.com/tj-actions/changed-files/issues/2463#issu... seems more likely:
> 1. Fork the repository > > 2. Push compromised code to the fork > > 3. Update the tags in the parent repository to point to the SHA of the fork
rognjen
> Update the tags in the parent repository to point to the SHA of the fork
I don't think that's possible.
Forks are a GitHub UI construct.
There would be two .git dirs so for all intents and purposes they're two repos that don't know about each other.
Locally you can't refer to a commit that's in a different dir...
werrett
No idea. But they didn't do a great job -- they broke the action, which caused build failures that people were going to notice.
The malicious commit only landed at 09:57 PDT today (March 14) in one specific action (out of a number that is quite popular). Maybe they were planning on coming back and doing proper exfil?
null
Hi, Renovate author/maintainer here.
The affected repo has now been taken down, so I am writing this partly from memory, but I believe the scenario is:
1. An attacker had write access to the tj-actions/changed-files repo
2. The attacker chose to spoof a Renovate commit, in fact they spoofed the most recent commit in the same repo, which came from Renovate
3. Important: this spoofing of commits wasn't done to "trick" a maintainer into accepting any PR, instead it was just to obfuscate it a little. It was an orphan commit and not on top of main or any other branch
4. As you'd expect, the commit showed up as Unverified, although if we're being realistic, most people don't look at that or enforce signed commits only (the real bot signs its commits)
5. Kind of unrelated, but the "real" Renovate Bot - just like Dependabot presumably - then started proposing PRs to update the action, like it does any other outdated dependency
6. Some people had automerging of such updates enabled, but this is not Renovate's default behavior. Even without automerging, an action like this might be able to achieve its aim only with a PR, if it's run as part of PR builds
7. This incident has reminded that many people mistakenly assume that git tags are immutable, especially if they are in semver format. Although it's rare for such tags to be changed, they are not immutable by design