Tj-actions/changed-files GitHub Action Compromised – used by over 23K repos

I theorized about this vulnerability a while back when I noticed new commits didn't disable automerging. This is an insane default from GH.

EDIT: seems GitHub has finally noticed (or started to care); just went to test this and auto merge has been seemingly disabled sitewide. Even though the setting is enabled, no option to automerge PRs shows up.

Seems I was right to worry!

EDIT2: We just tested this on GitLab's CI since they also have an auto-merge function and it appears they've done things correctly. Auto-merge enablement is only valid for the commit for which it was enabled; new pushes disable auto-merge. Much more sensible and secure.

Tags can be signed, and the signature can be verified. It's about as easy as signing / verifying commits. One can even make signing tags as the default option when creating tags.

This won't help in this case though, because a legitimate bot was tricked into working with a rogue commit; a tricked bot could as well sign a tag with a legitimate key.

"Immutable tags" of course exist, they are commit hashes, but they are uninformative :(

How else should we do it?

  > I'm not sure how this could exploited by just making a PR, unless you for some reason have secrets enabled for builds by unknown contributors

Neither Branch Protection nor the newer Rulesets allow to protect secrets from someone with push acces to the repo. From what I understand, only environment secrets provide this feature (and have the drawback that you can't share them among multiple repos in the same org without copying them everywhere, although you can script the copying with the github api)

> We need better capabilities. E.g. when I run `fd`, `rg` or similar such tool, why should it have Internet access?

Yeah!! We really need to auto sandbox everything by default, like mobile OSes. Or the web.

People browse the web (well, except Richard Stallman) all the time, and run tons of wildly untrusted code, many of them malicious. And apart from zero days here and there, people don't pay much attention to it, and will happily enter any random website in the same machine they also store sensitive data.

At the same time, when I open a random project from Github on VSCode, it asks whether the project is "trusted". If not, it doesn't run the majority of features like LSP server. And why not? Because the OS doesn't sandbox stuff by default. It's maddening.

I’ve been doing all of my dev work in a virtual machine as a way to clamp things down. I’ve even started using a browser in a VM as a primary browser.

Computers are fast enough where the overhead doesn’t feel like it’s there for what I do.

For development, I think Vagrant should make a comeback as one of the first things to setup in a repo/group of repos.

OpenBSDs pledge[0] system call is aimed at helping with this. Although, it's more of a defense-in-depth measure on the maintainers part and not the user.

> The pledge() system call forces the current process into a restricted-service operating mode. A few subsets are available, roughly described as computation, memory management, read-write operations on file descriptors, opening of files, networking (and notably separate, DNS resolution). In general, these modes were selected by studying the operation of many programs using libc and other such interfaces, and setting promises or execpromises.

[0]: https://man.openbsd.org/pledge.2

I've found firejail to be a useful tool for this (https://github.com/netblue30/firejail), and additionally use opensnitch (https://github.com/evilsocket/opensnitch) to monitor for unexpected network requests.

For CI/CD using something like ArgoCD let's you avoid giving CI direct access to prod - it still needs write access to a git repo, and ideally some read access to Argo to check if deployment succeeded but it limits the surface area.

FreeBSD has Capsicum [0] for this. Once a process enters capability mode, it can't do anything except by using already opened file descriptors. It can't spawn subprocesses, connect to the network, load kernel modules or anything else.

To help with things that can't be done in the sandbox, e.g. DNS lookups and opening new files, it provides the libcasper library which implements them using helper processes.

Not all utilities are sandboxed, but some are and hopefully more will be.

Linux recently added Landlock [1] which seems sort of similar, although it has rulesets and doesn't seem to block everything by default, as far as I can tell from quickly skimming the docs.

[0] https://wiki.freebsd.org/Capsicum

[1] https://docs.kernel.org/userspace-api/landlock.html

You also need to block write access, so they can’t encrypt all your files with an embedded public key. And read access so they can’t use a timing side channel to read a sensitive file and pass that info to another process with internet privileges to report the secret info back to the bad guy. You get the picture, I’m sure.

But that's what firejail and docker/podman are for. I never run any build pipeline on my host system, and neither should you. Build containers are pretty good for these kind of mitigations of security risks.

> We need better capabilities.

I'd love to say "just use Kubernetes and run Nexus as a service inside" but unfortunately Network Policies are seriously limited [1]...

[1] https://kubernetes.io/docs/concepts/services-networking/netw...

Just you wait. Here in America when your city becomes crime ridden enough you start leaving it unlocked again.

In the era of the key fob it's pretty automatic to lock the car every time. Some cars even literally do it for you. I hardly think of this, let alone get not great feelings about it.

For an example of a scary list of such offers, see https://github.com/extesy/hoverzoom/discussions/670

I have long since stopped using any extension that doesn’t belong to an actual company (password managers for example). Even if they aren’t malware when you installed them, they will be after they get sold.

I got an outreach for an extension I made as a joke. It had like maybe 5000 downloads ever.

I'd also emphasize out that there's nothing safe about it being "only dev", given how many attacks use employee computers (non-prod) as a springboard elsewhere.

It was only recently removed in Java and there was a related concept (adopted from OSGi) designed to only export certain symbols -- not for security but for managing the surface area that a library vendor had to support

But I mentioned both of those things because [IMHO] they both fell prey to the same "humanity bug": specifying permissions for anything (source code, cloud security, databases, Kubernetes, ...) is a lot of trial and error, whereas {Effect: Allow, Action: ["*:*"]} always works and so they just drop a "TODO: tighten permissions" and go on to the next Jira

I had high hopes for the AWS feature "Make me an IAM Policy based on actual CloudTrail events" but it talks a bigger game than it walks

node-ipc is a recent example from the Node ecosystem. The author released an update with some code that made a request to a geolocation webservice to decide whether to wipe the local filesystem.

Missed them too. Always was annoyed by FluentAssertions anyway, some contractor added it to a project that we took over couldn't see the value add.

What do you use for sandboxing?

Thank you, cyrnel, for the feedback! We are trying our best to help serve the community. Now, we have separate recovery steps for general users and our enterprise customers.

I think a big part of the problem is the way one typically "installs" a GH action: by copy-pasting something from README of the action.

Let's have a look at a random official GH provided action:

https://github.com/actions/checkout

It lists the following snippet:

`uses: actions/checkout@v4`

Almost everyone will just copy paste this snippet and call it a day. Most people don't think twice that v4 is a movable target that can be compromised.

In case of npm/yarn deps, one would often do the same, and copy paste `yarn install foobar`, but then when installing, npm/yarn would create a lockfile and pin the version. Whereas there's no "installer" CLI for GH actions that would pin the version for you, you just copy-paste and git push.

To make things better, ideally, the owners of actions would update the workflows which release a new version of the GH action, to make it update README snippet with the sha256 of the most recent release, so that it looks like

`uses: actions/checkout@abcdef9876543210` # v4.5.6

Since GitHub doesn't promote good defaults, it's not surprising that third-party maintainers do the same.

Some people do actually pin versions, like me. For instance:

  - uses: Swatinem/rust-cache@f0deed1e0edfc6a9be95417288c0e1099b1eeec3 # v2.7.7

  - uses: subosito/flutter-action@f2c4f6686ca8e8d6e6d0f28410eeef506ed66aff # v2.18.0

Aren't GitHub action "packages" designate by a single major version? Something like checkout@v4, for example. I thought that that designated a single release as v4 which will not be updated?

I'm quite possibly wrong, since I try to avoid them as much as I can, but I mean.. wow I hope I'm not.

Fuck. Insecure defaults again. I argue that a version specifier should be only a hash. Nothing else is acceptable. Forget semantic versions. (Have some other method for determining upgrade compatibility you do out of band. You need to security audit every upgrade anyway). Process: old hash, new hash, diff code, security audit, compatibility audit (semver can be metadata), run tests, upgrade to new hash.

You and someone else pointed this out. I only use GitHub-org actions, and I just thought that surely there would be a "one version to rule them all" type rule.. how else can you audit things?

I've never seen anything recommending specifying a specific commit hash or anything for GitHub actions. It's always just v1, v2, etc.

Nice find. Its a bit strange that the PRs listed there, are not present at all in the coinbase repo. Seems like the attack was directed there, but I also did not hear anything from Coinbase on this.

eg. Target their NPM and PYPI tokens, so they can push compromised packages.

That is a really very likely scenario.

The attacker was trying to compromise agentkit and found changed-files used in the repo so looked around. Found that it was using a bot with a PAT to release.

Totally possible the bot account had a weak password, and the maintainer said it didn't have 2FA.

They got the release bot PAT so they tried possibly quite an obvious vector that. They didn't need anything sophisticated or to exfil the credentials because agentkit is public.

It just so happened that it was detected before agentkit updated dependencies.

It's possible that with if thye had checked the dependabot config they could've timed it a bit better so that it's picked up in agentkit before being detected.

edit: Although, I don't think PATs are visible after they're generated?

> commit hashes

There is some latent concern that most git installations use SHA-1 hashes, as opposed to SHA-256. [0]

Also the trick of creating a branch that happens to be named the same as a revision, which then takes precedence for certain commands.

[0] https://git-scm.com/docs/hash-function-transition

Only commit hashes are safe. In this case the bad actor changed all of the version tags to point to their malicious commit. See https://github.com/tj-actions/changed-files/tags

All the tags point to commit `^0e58ed8` https://github.com/tj-actions/changed-files/commit/0e58ed867...

The version numbers aren't immutable, so an attacker can just update the versions to point to the compromised code, which is what happened here. Commit hashes are a great idea, but you still need to be careful: lots of people use bots like Renovate to update your pinned hashes whenever a new version is published, which runs into the same problem.

In your GitHub Actions YAML, instead of referencing a specific tag, you can reference a specific commit. So, instead of …

    uses: actions/checkout@v4

    uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683

Or just write your own.

That's the sort of hook you should be able to write yourself pretty quickly. So I threw your comment into o3-mini-high and it gave me a decent-looking solution. Decent but wrong, since it thought "current git commit" referred to the project repo, rather than the referenced dependency.

Anyway here's the gist of a solution without any of the necessary checking that the files actually exist etc.

  #!/bin/sh
  for file in .github/workflows/*.yml; do
    grep -E "uses:[[:space:]]+[A-Za-z0-9._-]+/[A-Za-z0-9._-]+@v[0-9]+" "$file" | while read -r line; do
      repo=$(echo "$line" | sed -E 's/.*uses:[[:space:]]+([A-Za-z0-9._-]+\/[A-Za-z0-9._-]+)@v[0-9]+.*/\1/')
      tag=$(echo "$line" | sed -E 's/.*@((v[0-9]+)).*/\1/')
      commit_hash=$(git ls-remote "https://github.com/$repo.git" "refs/tags/$tag" | awk '{print $1}')
      [ -n "$commit_hash" ] && sed -i.bak -E "s|(uses:[[:space:]]+$repo@)$tag|\1$commit_hash|g" "$file" && git add "$file" && rm -f "$file.bak"
    done
  done
  exit 0

Check the timestamp on that commit push. It was from today, an hour or two before the repo was restored, not yesterday when the attack happened. The push actor != the committor or even the actual commit author, and there can be multiple push actors if the commit is pushed multiple times by different actors.

He probably just re-pushed the bad commit while trying to figure out how to fix this.

I find it very plausible that the bot token was compromised, not his user account token, as the attack was simply to push over the tags (which is something the automation bot would have access to do, as tag management is one of its functions)

As long as you subscribe to security advisories, it’s a lot more likely that new vulnerabilities are introduced than old undiscovered vulnerabilities are accidentally patched. In fact barring rewrites (which usually won’t be picked up by semver-respecting auto bumps anyway) I can hardly think of an example of the latter.