My failed attempt to shrink all NPM packages by 5%

After reading the article, this comment and the comment thread further down on pnpm[1], it feels to me like the NPM team are doing everyone a disservice by ignoring the inefficiencies in the packaging system. It may not be deliberate or malicious but they could easily have provided better solutions than the one proposed in the article which, in my opinion is a band-aid solution at best. The real fix would be to implement what you mention here: local registry and caching, and/or symlinking a la pnpm.

[1] https://news.ycombinator.com/item?id=42841658

In every org I've worked with, we had a local dependency mirror in the GitOps architecture.

Lots of places use a cache like Artifactory so they don't get slammed with costs, and are resilient to network outages and dependency builds vanishing.

I really don't want to go back to the old world where every part of your build is secretly stateful and fails in mysterious hard to reproduce ways.

You can and should have your own caching proxy for all your builds but local caches are evil.

Yeah, I would also note that in addition to speed/transfer-costs, having an organizational package proxy is useful for reproducibility and security.

"I don't find the cons all that compelling to be honest"

This is a solid example of how things change at scale. Concerns I wouldn't even think about for my personal website become things I need to think about for the download site being hit by 50,000 of my customers become big deals when operating at the scale of npm.

You'll find those arguments the pointless nitpicking of entrenched interests who just don't want to make any changes, until you experience your very own "oh man, I really thought this change was perfectly safe and now my entire customer base is trashed" moment, and then suddenly things like "hey, we need to consider how this affects old signatures and the speed of decompression and just generally whether this is worth the non-zero risks for what are in the end not really that substantial benefits".

I do not say this as the wise Zen guru sitting cross-legged and meditating from a position of being above it all; I say it looking at my own battle scars from the Perfectly Safe things I've pushed out to my customer base, only to discover some tiny little nit caused me trouble. Fortunately I haven't caused any true catastrophes, but that's as much luck as skill.

Attaining the proper balance between moving forward even though it incurs risk and just not changing things that are working is the hardest part of being a software maintainer, because both extremes are definitely bad. Everyone tends to start out in the former situation, but then when they are inevitably bitten it is important not to overcorrect into terrified fear of ever changing anything.

I feel massively increasing publish time is a valid reason not to push this though considering such small gains and who the gains apply to.

The pros aren't all that compelling either. The npm repo is the only group that this would really be remotely significant for, and there seemed to be no interest. So it doesn't take much of a con to nix a solution to a non-problem.

I felt the same. The proposal wasn't rejected! Also, performance gains go beyond user stories - e.g. they reduce infra costs and environmental impact - so I think the main concerns of the maintainers could have been addressed.

> I don't find the cons all that compelling to be honest

I found it reasonable.

The 5% improvement was balanced against the cons of increased cli complexity, lack of native JS zopfli implementation, and slower compression .. and 5% just wasn't worth it at the moment - and I agree.

>or at least I think they warrant further discussion

I think that was the final statement.

> I don't find the cons all that compelling to be honest, or at least I think they warrant further discussion

It needs a novel JS port of a C compresison library, which will be wired into a heavily-used and public-facing toolchain, and is something that will ruin a significant number of peoples' days if it breaks.

For me, that kind of ask needs a compelling use case from the start.

It's not even funny:

  $ ll /nix/store/*-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/*
  /nix/store/…-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/linux:
  .r-xr-xr-x 129k root  1 Jan  1970 xsel

  /nix/store/…-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/windows:
  .r-xr-xr-x 444k root  1 Jan  1970 clipboard_i686.exe
  .r-xr-xr-x 331k root  1 Jan  1970 clipboard_x86_64.exe

Yep, I wrote a script that starts at a root `node_modules` folder and iterates through to remove anything not required (dotfiles, Dockerfile, .md files, etc) - in one of our smaller apps this removes about 25Mb of fluff, some packages are up to 60-70mb of crap removed.

You might be interested in e18e if you would like to see that change: https://e18e.dev/

They’ve done a lot of great work already.

One of the things I like about node_modules is that it's not purely source code and it's not purely build artifacts.

You can read the code and you can usually read the actual README/docs/tests of the package instead of having to find it online. And you can usually edit library code for debugging purposes.

If node_modules is taking up a lot of space across a bunch of old projects, just write the `find` script that recursively deletes them all; You can always run `npm install` in the future when you need to work on that project again.

Totally agree with you. I wish npm did a better job of filtering the crap files out of packages.

At least, switch to pnpm minimize the bloat

That's on the package publishers, not NPM. They give you an `.npmignore` that's trivially filled out to ensure your package isn't full of garbage, so if someone doesn't bother using that: that's on them, not NPM.

(And it's also a little on the folks who install dependencies: if the cruft in a specific library bothers you, hit up the repo and file an issue (or even MR/PR) to get that .npmignore file filled out. I've helped folks reduce their packages by 50+MB in some cases, it's worth your own time as much as it is theirs)

I believe I knocked 10% off of our node_modules directory by filing .npmignore PRs or bug reports to tools we used.

Now if rxjs weren’t a dumpster fire…

I think "massive" is overstating it. I don't think deploying a new version of a package is something that happens many times a day, so it wouldn't be a constant pain point.

Also, since this is a case of having something compressed once and decompressed potentially thousands of times, it seems like the perfect tool for the job.

It's very likely zero or positive impact on the decompression side of things.

Starting with smaller data means everything ends up smaller. It's the same decompression algorithm in all cases, so it's not some special / unoptimized branch of code. It's yielding the same data in the end, so writes equal out plus or minus disk queue fullness and power cycles. It's _maybe_ better for RAM and CPU because more data fits in cache, so less memory is used and the compute is idle less often.

It's relatively easy to test decompression efficiency if you think CPU time is a good proxy for energy usage: go find something like React and test the decomp time of gzip -9 vs zopfli. Or even better, find something similar but much bigger so you can see the delta and it's not lost in rounding errors.

I can speak to this - there is no meaningful decompression effect across an insane set of tested data at Google and elsewhere. Zopfli was invented prior to brotli

Zopfli is easiest to think of as something that just tries harder than gzip to find matches and better encodings. Much harder.

decompression speed is linear either way.

It's easiest to think of decompression as a linear time vm executor[1], where the bytecoded instructions are basically

go back <distance> bytes, output the next <length> bytes you see, then output character <c>

(outputting literal data is the instruction <0,0,{character to output}>)

Assuming you did not output a file larger than the original uncompressed file (why would you bother?), you will, worst case, process N bytes during decompression, where N is the size of the original input file.

The practical decompression speed is driven by cache behavior, but it thrashes the cache no matter what.

In practice, reduction of size vs gzip occurs by either finding larger runs, or encodings that are smaller than the existing ones.

After all, if you want the compressed file to shrink, you need output less instructions somehow, or make more of the instructions identical (so they can be represented in less bits by later huffman coding).

In practice, this has almost exclusively positive effects on decompression speed - either the vm has less things to process (which is faster), or more of the things it does look the same (which has better cache behavior).

[1] this is one way archive formats will sometimes choose to deal with multiple compression method support - encode them all to the same kind of bytecode (usually some form of copy + literal instruction set), and then decoding is the same for all of them. ~all compression algorithms output some bytecode like the above on their own already, so it's not a lot of work. This doesn't help you support other archive formats, but if you want to have a bunch of per-file compression options that you pick from based on what works best, this enables you to still only have to have one decoder.

For formats like deflate, decompression time doesn't generally depend on compressed size. (zstd is similar, though memory use can depend on the compression level used).

This means an optimization like this is virtually guaranteed to be a net positive on the receiving end, since you always save a bit of time/energy when downloading a smaller compressed file.

Not necessarily - could retain backward compat by publishing both gzip and zstd variants and having downloaders with newer npm’s prefer to download zstd. Over time, you could require packages only upload zstd going forward and either generate zstd versions of the backlog of unmaintained packages or at least those that see some amount of traffic over some time period if you’re willing to drop very old packages. The ability to install arbitrary versions of packages probably means you’re probably better off reprocessing the backlog although that may cost more than doing nothing.

The package lock checksum is probably a more solvable issue with some coordination.

The benefit of doing this though is less immediate - it will take a few years to show payoff and these kinds of payoffs are not typically made by the kind of committee decisions process described (for better or worse).

Brotli and lzo1b have good compression ratios and pretty fast decompression speeds. Compression speed should not matter that much, since you only do it once.

https://quixdb.github.io/squash-benchmark/

There even more obscure options:

https://www.mattmahoney.net/dc/text.html

Thats a much higher hurdle to jump. I don’t blame the author for trying this first.

If accepted, it might have been a good stepping stone too. A chance to get to know everyone and their concerns and how they think.

So if you wanted to see how this works (proposal + in prod) and then come back later proposing something bigger by switching off zip that would make sense to me as a possible follow up.

>But oh boy was it slow to unpack those bzip2 packages! Since conda had good caching, if you build environments often at all you could be paying more in decompress time than you pay in compression time.

For Paper, I'm planning to cache both the wheel archives (so that they're available without recompressing on demand) and unpacked versions (installing into new environments will generally use hard links to the unpacked cache, where possible).

> If you were building a new system today you'd probably use zstd since it beats gzip on both speed and compression.

FWIW, in my testing LZMA is a big win (and I'm sure zstd would be as well, but LZMA has standard library support already). But there are serious roadblocks to adopting a change like that in the Python ecosystem. This sort of idea puts them several layers deep in meta-discussion - see for example https://discuss.python.org/t/pep-777-how-to-re-invent-the-wh... . In general, progress on Python packaging gets stuck in a double-bind: try to change too little and you won't get any buy-in that it's worthwhile, but try to change too much and everyone will freak out about backwards compatibility.

The main downside though, it's impressively slow.

Comparing to gzip isn't really worth it. Combine pigz (threaded) with zlib-ng (simd) and you get decent performance. pigz is used in `docker push`.

For example, gzipping llvm.tar (624MB) takes less than a second for me:

    $ time /home/harmen/spack/opt/spack/linux-ubuntu24.04-zen2/gcc-13.2.0/pigz-2.8-5ptdjrmudifhjvhb757ym2bzvgtcsoqc/bin/pigz -k hello.tar 
    
    real    0m0.779s
    user    0m11.126s
    sys     0m0.460s

It is almost 2700x slower than the state of the art for just 6.8% bytes saved.

> These days if you’re going to iterate on a solution you’d better make it multithreaded.

Repetition eliminating compression tends to be inherently sequential. You'd probably need to change the file format to support chunks (or multiple streams) to do so.

Because of LZ back references, you can't LZ compress different chunks separately on different cores and have only one compression stream.

Statistics acquisition (histograms) and entropy coding could be parallel I guess.

(Not a compression guru, so take above with a pinch of salt.)

+1 to brotli. Newly published packages could use brotli by default, so old ones stay compatible.

> This also took a lot of seeking time for disks because there were so many individual files.

The fact NPM keeps things in node_modules unzipped seems wild to me. Filesystems are not great at hundreds of thousands of little files. Some are bad, others are terrible.

Zip files are easier to store, take up less space, and CPUs are faster than disks so the decompression in memory is probably faster reading the unzipped files.

That was one of my favorite features of Yarn when I tried it - pnp mode. But since it’s not what NPM does it requires a shim that doesn’t work with all ps mage’s. Or at least didn’t a few years ago.