Rust’s dependencies are starting to worry me

> IMO any system where taking a dependency is "easy" and there is no penalty for size or cost is going to eventually lead to a dependency problem.

Go and C# (.NET) are counterexamples. They both have great ecosystems and just as simple and effective package management as Rust or JS (Node). But neither Go or C# have issues with dependency hell like Rust or even more JavaScript, because they have exceptional std libs and even large frameworks like ASP.NET or EF Core.

A great std lib is obviously the solution. Some Rust defenders are talking it down by giving Python as counter example. But again, Go and C# are proving them wrong. A great std lib is a solution, but one that comes with huge efforts that can only be made by large organisations like Google (Go) or Microsoft (C#).

> At each level a caller might need 5% of the functionality of any given dependency. The deeper the dependency tree gets the more waste piles on. Eventually you end up in a world where your simple binary is 500 MiB of code you never actually call, but all you did was take that one dependency to format a number.

I'm not convinced that happens that often.

As someone working on a Rust library with a fairly heavy dependency tree (Xilem), I've tried a few times to see if we could trim it by tweaking feature flags, and most of the times it turned out that they were downstream of things we needed: Vulkan support, PNG decoding, unicode shaping, etc.

When I did manage to find a superfluous dependency, it was often something small and inconsequential like once_cell. The one exception was serde_json, which we could remove after a small refactor (though we expect most of our users to depend on serde anyway).

We're looking to remove or at least decouple larger dependencies like winit and wgpu, but that requires some major architectural changes, it's not just "remove this runtime option and win 500MB".

Symbol culling and dead code removal is already a thing in modern compilers and linkers, and rust can do it too: https://github.com/johnthagen/min-sized-rust

Way back when, I used to vendor all the libraries for a project (Java/Cpp/Python) into a mono repo and integrate building everything into the projects build files so anyone could rebuild the entire app stack with whatever compiler flags they wanted.

It worked great, but it took diligence, it also forces you to interact with your deps in ways that adding a line to a deps file does not.

>At each level a caller might need 5% of the functionality of any given dependency. The deeper the dependency tree gets the more waste piles on. Eventually you end up in a world where your simple binary is 500 MiB of code you never actually call, but all you did was take that one dependency to format a number.

So, what's is the compiler doing that he doesnt remove unused code?

> Its super easy to say `import foolib`, then call `foolib.do_thing()` and just start running.

It's effectively an end-run around the linker.

It used to be that you'd create a library by having each function in its own compilation unit, you'd create a ".o" file, then you'd bunch them together in a ".a" archive. When someone else is compiling their code, and they need the do_thing() function, the linker sees it's unfulfiled, and plucks it out of the foolib.a archive. For namespacing you'd probably call the functions foolib_do_thing(), etc.

However, object-orientism with a "god object" is a disease. We go in through a top-level object like "foolib" that holds pointers to all its member functions like do_thing(), do_this(), do_that(), then the only reference the other person's code has is to "foolib"... and then "foolib" brings in everything else in the library.

It's not possible for the linker to know if, for example, foolib needed the reference to do_that() just to initialise its members, and then nobody else ever needed it, so it could be eliminated, or if either foolib or the user's code will somehow need it.

> Go and Rust, for example, encourage everything for a single package/mod to go in the same file.

I can say that, at least for Go, it has excellent dead code elimination. If you don't call it, it's removed. If you even have a const feature_flag = false and have an if feature_flag { foobar() } in the code, it will eliminate foobar().

> Go and Rust, for example, encourage everything for a single package/mod to go in the same file.

Clarification: Go allows for a very simple multi-file. It’s one feature I really like, because it allows splitting otherwise coherent module into logical parts.

> Eventually you end up in a world where your simple binary is 500 MiB of code you never actually call,

It’s getting hard to take these conversations seriously with all of the hyperbole about things that don’t happen. Nobody is producing Rust binaries that hit 500MB or even 50MB from adding a couple simple dependencies.

You’re also not ending up with mountains of code that never gets called in Rust.

Even if my Rust binaries end up being 10MB instead of 1MB, it doesn’t really matter these days. It’s either going on a server platform where that amount of data is trivial or it’s going into an embedded device where the few extra megabytes aren’t really a big deal relative to all the other content that ends up on devices these days.

For truly space constrained systems there’s no-std and entire, albeit small, separate universe of packages that operate in that space.

For all the doom-saying, in Rust I haven’t encountered this excessive bloat problem some people fret about, even in projects with liberal use of dependencies.

Every time I read these threads I feel like the conversations get hijacked by the people at the intersection of “not invented here” and nostalgia for the good old days. Comments like this that yearn for the days of buying paid libraries and then picking them apart anyway really reinforce that idea. There’s also a lot of the usual disdain for async and even Rust itself throughout this comment section. Meanwhile it feels like there’s an entire other world of Rust developers who have just moved on and get work done, not caring for endless discussions about function coloring or rewriting libraries themselves to shave a few hundred kB off of their binaries.

> Should a 'glob' library actually read the file system and give you filenames

The POSIX glob function after which these things are named traverses the filesystem and matches directory entries.

The pure matching function which matches a glob pattern against a filename-like string is fnmatch.

But yes, the equivalent of fnmatch should be a separate module and that could be a dependency of glob.

Nobody should be trying to implement glob from scratch using a fnmatch-like function and directory traversal. It is not so trivial.

glob performs a traversal that is guided by the pattern. It has to break the pattern into path components. It knows that "*/*/*" has three components and so the traversal will only go three levels deep. Also "dir/*" has a component which is a fixed match, and so it just has to open "dir" without scanning the current directory; if that fails, glob has failed.

If the double star ** is supported which matches multiple components, that's also best if it likewise integrated into glob.

If brace expansion is supported, that adds another difficulty because different branches of a brace can have different numbers of components, like {*/x,*/*/x,*/*/*/x}. To implement glob, it would greatly help us to have brace expansion as a separate function which expands the braces, producing multiple glob patterns, which we can then break into path components and traverse.

> I can't imagine it's any different in [R]ust land

Taste is important; programmers with good architectural taste tend to use languages that support them in their endeavour (like Rust or Zig) or at least get out of the way (C).

So I would argue the problems you list are statistically less often the case than in certain other languages (from COBOL to JavaScript).

> There's just too many devs and all of them, including myself, don't always make the best choices.

This point you raise is important: I think an uncoordinated crowd of developers will create a "pile of crates" ("bazaar" approach, in Eric Raymond's terminology), and a single language designer with experience will create a more uniform class library ("cathedral" approach).

Personally, I wish Rust had more of a "batteries included" standard library with systematically named and namespaced official crates (e.g. including all major data structures) - why not "stdlib::data_structures::automata::weighted_finite_state_transducer" instead of a confusing set of choices named "rustfst-ffi", "wfst", ... ?

Ideally, such a standard library should come with the language at release. But the good news is it could still be devised later, because the Rust language designers were smart enough to build versioning with full backwards compatibility (but not technical debt) into the language itself. My wish for Rust 2030 would be such a stdlib (it could even be implemented using the bazaar of present-day crates, as long as that is hidden from us).

We don't need to speak in hypotheticals, we can just look at the glob crate: https://crates.io/crates/glob

213M downloads, depends on zero external crates, one source file (a third of which is devoted to unit tests), and developed by the rust-lang organization itself (along with a lot of crates, which is something that people tend to miss in this discussion).

Historically, the borrow checker has been a good shield against developers that have no taste.

Not sure how long that’ll last.

It's worth pointing out that Node has a built in globbing function: https://nodejs.org/docs/latest-v24.x/api/fs.html#fspromisesg...

> Also there's arguably design. Should a 'glob' library actually read the file system and give you filenames or should it just tell you if a string matches a glob and leave the reset to you?

There's a function in Node's stdlib that does this as well (albeit it's marked as experimental): https://nodejs.org/docs/latest-v24.x/api/path.html#pathmatch...

What you’re describing regarding glob is not lack of taste, it’s an architectural “bug”.

Taste is what Steve Jobs was referring to when he said Microsoft had none. In software it’s defined by a humane, pleasant design that almost(?) anybody can appreciate.

Programming languages cannot be tasteful, because they require time and effort to learn and understand. Python has some degree elegance and Golang’s simplicity has a certain je ne sais quoi… but they’re not really fitting the definition.

Still, some technologies such as git, Linux or Rust stand out as particularly obscure even for the average developer, not just average human.

Bun has a glob too https://bun.sh/docs/api/glob

Yeah that's one huge advantage Rust has over NPM - Rust developers are a lot more skilled and crates are generally much higher quality.

You want a special purpose language.

In your particular example of image loading, you want WUFFS. https://github.com/google/wuffs

In WUFFS most programs are impossible. Their "Hello, world" doesn't print hello world because it literally can't do that. It doesn't even have a string type, and it has no idea how to do I/O so that's both elements of the task ruled out. It can however, Wrangle Untrusted File Formats Safely which is its sole purpose.

I believe there should be more special purpose languages like this, as opposed to the General Purpose languages most of us learn. If your work needs six, sixteen or sixty WUFFS libraries to load different image formats, that's all fine because categorically they don't do anything outside their box. Yet, they're extremely fast because since they can't do anything bad by definition they don't need those routine "Better not do anything bad" checks you'd write in a language like C or the compiler would add in a language like Rust, and because they vectorize very nicely.

Java and the .NET Framework had partial trust/capabilities mechanisms decades ago. No one really used them and they were deprecated/removed.

Maybe we need a stronger culture of Sans-IO dependencies in general. To the point of pointing out and criticising like it happens with bad practices and dark patterns. A new lib (which shouldn't be used its own file access code) is announced in HN, and the first comment: "why do you do your own IO?"

Edit - note it's just tongue in cheek. Obviously libraries being developed against the public approval wouldn't be much of a good metric. Although I do agree that a bit more common culture of the Sans-IO principles would be a good thing.

I don't think retrofitting existing languages/ecosystems is necessarily a lost cause. Static enforcement requires rewrites, but runtime enforcement gets you most of the benefit at a much lower cost.

As long as all library code is compiled/run from source, a compiler/runtime can replace system calls with wrappers that check caller-specific permissions, and it can refuse to compile or insert runtime panics if the language's escape hatches would be used. It can be as safe as the language is safe, so long as you're ok with panics when the rules are broken.

It'd take some work to document and distribute capability profiles for libraries that don't care to support it, but a similar effort was proven possible with TypeScript.

I love this idea and I hope I get to work on it someday. I've wanted this ever since I was a starry-eyed teenager on IRC listening to Darius Bacon explain his capability-based OS idea, aptly called "Vapor".

I think it could be possible in Rust with a linter, something like https://github.com/geiger-rs/cargo-geiger . The Rust compiler has some unsoundness issues such as https://github.com/rust-lang/rust/issues/84366 . Those would need fixing or linter coverage.

I've thought about this (albeit not for that long) and it seems like you'd need a non-trivial revamp of how we communicate with the operating system. For instance, allowing a library to "read from a stream" sounds safe until you realize they might be using the same syscalls as reading from a file!

I love this idea. There is some reminiscence of this in Rust, but it's opt in and based on convention, and only for `unsafe` code. Specifically, there's a trend of libraries using `#![deny(unsafe_code)]` (which will cause a compilation error if there is any `unsafe` code in the current crate), and then advertising this to their users. But there's no enforcement, and the library can still add `#[allow(unsafe_Code)]` to specific functions.

Perhaps a capability system could work like the current "feature" flags, but for the standard library, which would mean they could be computed transitively.

I don't think you need to get very complex to design a language that protects libraries from having implicit system access. If the only place that can import system APIs is in the entry program, then by design libraries need to use dependency injection to facilitate explicit passing of capabilities.

One can take just about any existing language and add this constraint, the problem however is it would break the existing ecosystem of libraries.

I'd argue that the severity varies between languages, despite the core problem being universal. Languages with comprehensive standard libraries have an advantage over those with minimal built-in functionality, where people rely on external dependencies even for the most basic things (e.g. see Java/.NET vs JS/Node). Lightweight is not always better.

> If it was pulled into Rust stdlib, that team would be stuck handling it, and making changes to any of that code becomes more difficult.

I think Rust really needs to do more of this. I work with both Go and Rust daily at work, Go has its library game down -- the standard library is fantastic. With Rust it's really painful to find the right library and keep up for a lot of simple things (web, tls, x509, base64 encoding, heck even generating random numbers.)

Gilad Bracha has a really interesting approach to sandboxing third party libraries: Remove imports, and do everything with dependency injection. That way if you never inject say the IO subsystem, the third party code won't be able to break out. And there's no overhead, since it's all based on capabilities.

Even cooler, if you want to only expose read operations, you can wrap the IO library in another library that only exposes certain commands (or custom filtering, etc).

EDIT: I should say this doesn't work with systems programming, since there's always unsafe or UB code.

Yes, but a lot of the complexity is unnecessary bloat. Almost every project I've ever seen or worked on was full of unnecessary complexity. People naturally tend to over-complicate things, all the programming books, including software design books focus on unimportant aspects and miss all the important ones. It's incredibly frustrating.

Yet, if someone were to write a book which explained things properly (probably a 3000 word article would suffice to turn anyone into a 10x dev), nobody would buy it. This industry is cooked.

Maybe we should have a way to run every single library we use in an isolated environment and have a structure like QubesOS. Your main code is dom0 and you can create bunch of TemplateVMs which are your libraries and then create AppVMs for using those libraries. Use network namespaces for communicating between these processes. For sensitive workloads (finance, healthcare, etc), it makes sense to deploy something like that

Regardless of language, really? I highly doubt that, you don't generally see such problems with C or even C++ because dependencies are more cumbersome to add, especially in a way that's cross-platform.

> We are doing more complex things

In my experience we have more complex methodologies to the same things, but the goals are not more complex.

This was linked from the top comment on the Rust subreddit: https://wiki.alopex.li/LetsBeRealAboutDependencies

I think it makes a good point that some of the difference here is just perception due to dependencies in C/C++ being less immediately visible since they're dynamically loaded. To some degree that is a plus though as you likely trust the maintainers of your OS distribution to provide stable, supported libraries.

As other commenters have said, perhaps this is an area where the Rust maintainers could provide some kind of extended standard library where they don't guarantee backwards compatibility forever, but do provide guarantees about ongoing fixes for security issues.

I take bit less unstable dependencies over the total mess of C++ dependencies with CMake, shared libraries, version conflicts etc any time. There's probably also a bit of an illusion about C++ transitive dependencies due to them usually being precompiled (because compiling them is such pain).

> the philosophy in Rust is to pull many small package

I'm not sure it's a philosophy, more a pragmatic consideration for compilation speeds. Anyone who's done a non-trivial amount of Rust knows that moment when the project gets too big and needs to split into separate crates. It's kinda sad that you can't organize code according to proper abstractions, many times I feel forced to refactor for compiler performance.

> Sure, but that means I won't write my project, because I won't write those things from scratch.

You need to think a bit harder about that, to help you decide whether your position is rational.

I have been wasting 6 hours yesterday on getting the bullet examples to compile outside of bullet itself with no success. It's more likely that a lot of software simply doesn't get written because C++ and CMake are a pain in the ass.

Also shout out to cargo-vet.

It lets you track what packages you "trust". Then you can choose to transitively trust the packages trusted by entities you trust.

This lets you have a policy like "importing a new 3rd party package requires a signoff from our dependency tzar. But, packages that Google claim to have carefully reviewed are fine".

You can also export varying definitions of "trust". E.g. Google exports statements like:

- "this package has unsafe code, one of our unsafe experts audited it and thinks it looks OK"

- "this package doesn't do any crypto"

- "this is a crypto library, one of our crypto experts audited it and thinks it looks ok"

https://github.com/google/rust-crate-audits/blob/main/auditi...

Basically it's a slightly more formal and detailed version of blessed.rs where you can easily identify all the "it's not stdlib, but, it's kinda stdlib" stuff and make it easily available to your team without going full YOLO mode.

It can also give you a "semi-YOLO" approach, it supports rationales like "this package is owned by a tokio maintainer, those folks know what they're doing, it's probably fine". I think this is a nice balance for personal projects.

Would love to see something like this for Python.

Did a review; this is solid!

And for what it's worth, people do audit tokio. I have audited tokio. Many times in fact. Sure, not everyone will, but someone will :)

If two different dependencies use a different version of some other dependency between them does cargo still include both versions by default?

This is something I've only ever seen cargo do.

Best way is to have CI/CD systems only connected to the official internal repos.

Devs can add whatever they feel like on their workstations but it will be a sad build server if they get pushed without permission.

There are some voices trying to address this security risk (e.g. the proponents of this new RFC: https://github.com/rust-lang/rfcs/pull/3810). However, for some reason (probably culture) there isn't much momentum yet to change the status quo.

The cool thing about rust is you can implement async yourself. You aren't tied to any specific implementation.

Java did this sometimes by essentially adding slightly tidied up versions of whatever was the de-facto standard to the standard library. Java 1.3 didn't have regexes but most people were using the same apache commons thing, so java 1.4 added regexes that looked exactly like that. Java's date handling was a pain so people mostly used joda-date; a later java version added something that mostly works like jodadate. Etc.

It is an easy way to get a somewhat OK standard library as the things you add became popular on their own merits at some point.

Once added, the lowest friction path is to just use the standard library; and as it is the standard library you have a slightly better hope someone will care to maintain it. You can still build a better one if needed for your use-case, but the batteries are included for basic usage

I have a lot of sympathy for this viewpoint, but I also ask that we try to remind ourselves. We are asking for professionalism from hobby projects.

If you want a mature protobuf implementation you should probably buy one. Expecting some guy/gal on the internet to maintain one for your for free seems ill advised.

I feel like the Go ecosystem almost serendipitously has this built in - modules marked v0.X.Y being immature and under development, and v1 or greater being mature, keeping changes mostly down to bug fixes. I think some folks may even follow this convention!

100%, I still miss feature flags in npm. Is there a package manager that can do this already? I'd love to expand our internal libs with framework-specific code

I just built it with "git clone --depth 1 ..." and the build from cargo build --release is 2.9GB (2.3GB in the target folder)?

I am counting 13 dependencies, the rest are internal ones. Are any of these superfluous or only needed for small edge cases? Serde seems exactly a case where you absolutely should use an external dependency.

Also, repository size seems an extremely irrelevant metric.

I agree that relying on unknown dependencies is a risk, but this misses the point IMO. Number of dependencies and disk space are kind of arbitrary.

> Meanwhile, the heaviest JavaScript parser implemented in JavaScript is more lightweight.

The lightest weight javascript program relies on V8 to run, which has multiple orders of magnitude more dependencies. Most of which you have never heard of.

At least cargo makes it easier to get a clearer picture of what the dependencies are for a program.

Go has a fatter standard library and a "fat" runtime with built-in green threads (an asynchronous runtime basically) and garbage collection, so you get more out of the box and thus end up using fewer dependencies.

I have yet to come across a go project that doesn't pull in tons of 3rd party code as well. It seems like maybe you're over-stating the "culture" a bit.