Shai-Hulud Returns: Over 300 NPM Packages Infected

I've worried about this for a while with Rust packages. The total size of a "big" Rust project's dependency graph is pretty similar to a lot of JS projects. E.g. Tauri, last I checked, introduces about 600 dependencies just on its own.

Like another commenter said, I do think it's partially just because dependency management is so easy in Rust compared to e.g. C or C++, but I also suspect that it has to do with the size of the standard library. Rust and JS are both famous for having minimal standard libraries, and what do you know, they tend to have crazy-deep dependency graphs. On the other hand, Python is famous for being "batteries included", and if you look at Python project dependency graphs, they're much less crazy than JS or Rust. E.g. even a higher-level framework like FastAPI, that itself depends on lower-level frameworks, has only a dozen or so dependencies. A Python app that I maintain for work, which has over 20 top-level dependencies, only expands to ~100 once those 20 are fully resolved. I really think a lot of it comes down to the standard library backstopping the most common things that everybody needs.

So maybe it would improve the situation to just expand the standard library a bit? Maybe this would be hiding the problem more than solving it, since all that code would still have to be maintained and would still be vulnerable to getting pwned, but other languages manage somehow.

I agree partly. I love cargo and can’t understand why certain things like package namespaces and proof of ownership isn’t added at a minimum. I was mega annoyed when I had to move all our Java packages from jcenter, which was a mega easy setup and forget affair, to maven central. There I suddenly needed to register a group name (namespace mostly reverse domain) and proof that with a DNS entry. Then all packages have to be signed etc. In the end it was for this time way ahead. I know that these measures won’t help for all cases. But the fact that at least on npm it was possible that someone else grabs a package ID after an author pulled its packages is kind of alarming. Dependency confusion attacks are still possible on cargo because the whole - vs _ as delimiter wasn’t settled in the beginning. But I don’t want to go away from package managers or easy to use/sharable packages either.

I'm a huge Go proponent but I don't know if I can see much about Go's module system which would really prevent supply-chain attacks in practice. The Go maintainers point [1] at the strong dependency pinning approach, the sumdb system and the module proxy as mitigations, and yes, those are good. However, I can't see what those features do to defend against an attack vector that we have certainly seen elsewhere: project gets compromised, releases a malicious version, and then everyone picks it up when they next run `go get -u ./...` without doing any further checking. Which I would say is the workflow for a good chunk of actual users.

The lack of package install hooks does feel somewhat effective, but what's really to stop an attacker putting their malicious code in `func init() {}`? Compromising a popular and important project in this way would likely be noticed pretty quickly. But compromising something widely-used but boring? I feel like attackers would get away with that for a period of time that could be weeks.

This isn't really a criticism of Go so much as an observation that depending on random strangers for code (and code updates) is fundamentally risky. Anyone got any good strategies for enforcing dependency cooldown?

[1] https://go.dev/blog/supply-chain

> It's not "node" or "Javascript" the problem, it's this convenient packaging model.

That and the package runtime runs with all the same privileges and capabilities as the thing you're building, which is pretty insane when you think about it. Why should npm know anything outside of the project root even exists, or be given the full set of environment variables without so much as a deny list, let alone an allow list? Of course if such restrictions are available, why limit them to npm?

The real problem is that the security model hasn't moved substantially since 1970. We already have all the tools to make things better, but they're still unportable and cumbersome to use, so hardly anything does.

Historically, arguments of "it's popular so that's why it's attacked" have not held up. Notable among them was addressing Windows desktop security vulnerabilities. As Linux and Mac machines became more popular, not to mention Android, the security vulnerabilities in those burgeoning platforms never manifested to the extent that they were in Windows. Nor does cargo or pip seem to be infected with these problems to the extent that npm is.

I think this is right about Rust and Cargo, but I would say that Rust has a major advantage in that it implements frozen + offline mode really well (which if you use, obviously significantly decreases the risks).

Any time I ever did the equivalent with NPM/node world it was basically unusable or completely impractical

Why the word "semblance" with regard to Go modules? Are you trying to say this system is lacking something?

There are ecosystems that have package managers but also well developed first party packages.

In .NET you can cover a lot of use cases simply using Microsoft libraries and even a lot of OSS not directly a part of Microsoft org maintained by Microsoft employees.

The concern is not 'could' happen, but _does_ happen. I know this could occur in many places. But where it seems highly prevalent is NPM.

And I am genuinely thinking to myself, is this making using npm a risk?

NPM has about 4 million packages, Maven Central has about 3 million packages.

If this were true, wouldn't there have been at least one Maven attack by now, considering the number of NPM attacks that we've seen?

Value is one thing but the average user (by virtue of being popular) will be just less clued in on any security practices that could mitigate the problem.

it's interesting that staying up to date with your dependencies is considered a vulnerability in Node

Not sure what the language has anything to do with it, we've built JavaScript applications within pulling in 100s of NPM packages before NPM was a thing, people and organizations can still do so today, without having to switch language, if they don't want to.

Does it require disciple and a project not run by developers who just learned program? You betcha.

Ah yes PHP, the language known for its strong security...

Microsoft owns npmjs.com. They could pay for AI analysis of published version deltas, looking for backdoors and malware.

.NET being so batteries-included is one of its best features. And when vulnerabilities do creep in, it's nice to know that Microsoft will fix it rather than hoping a random open source project will.

This a common refrain on HN, frequently used to dismiss what may be perfectly legitimate concerns.

It also ignores the central question of whether NPM is more vulnerable to these attacks than other package managers, and should therefore be considered an unreasonable security risk.

That is indeed what one should do IMO. We've known for a long time now in the ops world that keeping versions stable is a good way to reduce issues, and it seems to me that the same principle applies quite well to software dev. I've never found the "but then upgrading is more of a pain" argument to be persuasive, as it seems to be equally a pain to upgrade whether you do it once every six months or once every six years.

The arguments for doing frequent releases partially apply to upgrading dependencies. Upgrading gets harder the longer you put it off. It’s better to do it on a regular schedule, so there are fewer changes at once and it preserves knowledge about how to do it.

A cooldown is a good idea, though.

Because updates don't just include new features but also bug and security fixes. As always, it probably depends on the context how relevant this is to you. I agree that cooldown is a good idea though.

That worried me too, a sort of inverse tragedy of the commons. I'll use a weeklong cooldown, _someone else_ will find the issue...

Until no-one does, for a week. To stretch the original metaphor, instead of an overgrazed pasture, we grow a communally untended thicket which may or may not have snakes when we finally enter.

I don't think that this Kantian argument is relevant in tech. We've had LTS versions of software for decades and it's not like every single person in the industry is just waiting for code to hit LTS before trying it. There are a lot of people and (mostly smaller) companies who pride themselves on being close to the "bleeding edge", where they're participating more fully in discovering issues and steering the direction.

The assumption in the post is that scanners are effective at detecting attacks within the cooldown period, not that end-device exploitation is necessary for detection.

(This may end up not being true, in which case a lot of people are paying security vendors a lot of money to essentially regurgitate vulnerability feeds at them.)

The docs list this caveat:

> Note that previous stable versions will not be suggested. The package will be completely ignored if its latest published version is within the cooldown period.

Seems like a big drawback to this approach.

Your line of reasoning only makes sense if literally almost all developers in the world adopt cooldowns, and adopt the same cooldown.

That would be a level of mass participation yet unseen by mankind (in anything, much less something as subjective as software development). I think we're fine.

For zero/one days, the trick is that you'd pair dependency cooldowns with automatic scanning for vulnerable dependencies.

And in the cases where you have vulnerable dependencies, you'd force update them before the cooldown period had expired, while leaving everything else you can in place.

There are companies like Helix Guard scanning registries. They advertise static analysis / LLM analysis, but honeypot instances can also install packages & detect certain files like cloud configs being accessed

Very nice way of putting it, kudos!

The packages were published using a compromised key directly, not through our ci/cd. We rolled the key, and published a new clean version from our repo through our CI/CD: https://github.com/PostHog/posthog-js/actions/runs/196303581...

As a user of Posthog, this statement is absurd: > Or safer again not to use software this vulnerable.

Nearly all software you use is susceptible to vulnerabilities, whether it's malicious or enterprise taking away your rights. It's in bad taste to make a comment about "not using software this vulnerable" when the issue was widespread in the ecosystem and the vendor is already being transparent about it. The alternative is you shame them into not sharing this information, and we're all worse for it.

Popularity and vulnerability go hand in hand though. You could be pretty safe by only using packages with zero stars on GitHub, but would you be happy or productive?

No.1: Run untrusted code in a sandbox! https://github.com/sandbox-utils/sandbox-venv

Most of the best practices can be translated to python ecosystem. It’s not exact 1:1 mapping but change few key terms and tools, the underlying practices should be the same.

Or copy that repo’s markdown into an llm and ask it to map to the pip ecosystem

No. I am tired of seeing people dragged into poor quality environments for the wrong reasons. These people would do much better using other tools.

Your attempt to make it personal does not compute.

I don't get this explanation. How does it force you to run the infection code?

Yes, if you depend on an infected package, sure. But then I'd expect not just a list, but a graph outlining which package infected which other package. Overall I don't understand this at all.

Thanks. I saw that sentence but somehow didn't parse it. Need a coffee :/

See also Xe Iaso's posts about CVEs in the C ecosystem (https://xeiaso.net/shitposts/no-way-to-prevent-this/CVE-2025...)

You can host your own NPM reg, and examine every package, but your manager probably is NOT going to go for that.

I would say that npm likely has easier solutions here compared to Cargo.

Well before the npm attacks were a thing, we within the Rust project, have discussed a lot of using wasm sandboxing for build-time code execution (and also precompiled wasm for procedural macros, but that's its own thing.) However the way build scripts are used in the Rust ecosystem makes it quite difficult enforce sandbox while also enabling packages to build foreign code (C, C++ invoke make, cmake, etc.) The sandbox could still expose methods to e.g. "run the C compiler" to the build scripts, but once that's done they have an arbitrary access to a very non-trivial piece of code running in a privileged environment.

Whereas for Javascript rarely does a package invoke anything but other javascript code during the build time. Introduce a stringent sandbox for that code (kinda deno style perhaps?) and a large majority of the packages are suddenly safe by default.

This is a cultural problem created through a fundamental misunderstanding (and mis-application) of Unix philosophy. As far as I'm aware the Rust ecosystem doesn't have a problem appropriately sizing packages which in turn reduces the overall attack surface of dependencies.

An example: Java Maven artifacts typically name the exact version of their dependencies. They rarely write "1.2.3 or any newer version in the 1.2.x series", as is the de-facto standard in NPM dependencies. Therefore, it's up to each dependency-user to validate newer versions of dependencies before publishing a new version of their own package. Lots of manual attention needed, so a slower pace of releases. This is a good thing!

Another example: all Debian packages are published to unstable, but cannot enter testing for at least 2-10 days, and also have to meet a slew of conditions, including that they can be and are built for all supported architectures, and that they don't cause themselves or anything else to become uninstallable. This allows for the most egregious bugs to be spotted before anyone not directly developing Debian starts using it.

Build packages from source without any binaries (all the way down) and socially audit the source before building.

https://bootstrappable.org/ https://reproducible-builds.org/ https://github.com/crev-dev

Other languages seem to publish dependencies as self-contained packages whose installation does not require running arbitrary shell scripts.

This does not prevent said package from shipping with malware built in, but it does prevent arbitrary shell execution on install and therefore automated worm-like propagation.

I think some system would need to dynamically analyze the code (as it runs) and record what it does. Even then, that may not catch all malicious activity. It's sort of hard to define what malicious activity is. Any file read or network conn could, in theory, be malicious.

As a SW developer, you may be able to limit the damage from these attacks by using a MAC (like SELinux or Tomoyo) to ensure that your node app cannot read secrets that it is not intended to read, conns that it should not make, etc. and log attempts to do those things.

You could also reduce your use of external packages. Until slowly, over time you have very little external dependencies.

The same way it always has been done - vendor your deps.

Hire an antivirus company to provide a safe and verified feed of packages. Use ML and automatic scanners to send packages to manual review. While Halting problem prevents us from 100% reliably detecting malware, at least we can block everything suspicious.

Other than general security practices, here are few NPM ecosystem specific ones: https://github.com/bodadotsh/npm-security-best-practices