Wild – A fast linker for Linux

OTOH even lld, fast but fairly slower than mold, is already incredibly faster than MS's linker even without the incrmeentality. Like, I'm routinely linking hundreds of megabytes in less than a second anyways, not sure incrementality is that much worth it

Additionally the way precompiled headers are handled in Visual C++ and C++ Builder have always been much better than traditional UNIX compilers, and now we have modules as well.

It has to be a candidate for the longest biggest gap in build tooling ever.

[flagged]

Hmm, my naive summary of AGPL is "If you run AGPL code in your web backend you are obliged to offer the backend source to everyone using a web client". No wonder it's explicitly forbidden at Google.

What does that mean for a linker? If you ship a binary linked with an AGPL linker you need to offer the source of the linker? Or of the program being linked?

iirc the mold author wanted to make money off of it (and I dont blame him).

AGPL is avoided like the plague by big corps: same big corps are known for having money to pay for licenses and sometimes (yes, I look at you Amazon) being good at deriving value from FLOSS without giving back.

iirc AGPL was used so everyone can just use it, big biz is still compelled to buy a license. this has been done before and can be seen as one of the strategies to make money off FLOSS.

Corps want to be able to release and use tools that take away the freedoms that GPL-family licenses provide. Often this results in duplication of effort.

This is not theoretical; it happens quite frequently. For toolchains, in particular I'm aware of how Apple (not that they're unique in this) has "blah blah open source" downloads, but often they do not actually correspond with the binaries. And not just "not fully reproducible but close" but "entirely new and incompatible features".

The ARM64 saga is a notable example, which went on for at least six months (at least Sept 2013 to March 2014). XCode 5 shipped with a closed-source compiler only for all that time.

Corps don't want to have to release the source code for their internal forks. They could also potentially be sued for everything they link using it because the linked binaries could be "derivative works" according to a judge who doesn't know anything.

Mold will be faster than LLD even using LTO, but all of its benefits will be absolutely swamped by the LTO process, which is, more or less, recompiling the entire program from high-level LLVM-IR. That's extremely expensive and dwarfs any linking advantages.

So the benefit will be barely noticable. As another comment points out, LTO should only be used when you need a binary optimized to within an inch of its life, such as a release copy, or a copy for performance testing.

I think we're talking about non-release builds here. In those, you don't want to use LTO, you just want to get that binary as fast as possible.

Yeah, if you're development process requires LTO you may be holding it wrong....

Specifically, if LTO is so important that you need to be using it during development, you likely have a very exceptional case, or you have some big architectural issues that are causing much larger performance regressions then they should be.

Agreed. Both fast and small are desirable for sandboxed (least authority) isomorphic (client and server) microservices with WebAssembly & related tech.

You should be using LTO where incremental build times are a concern, i.e. for development builds.

And for realease builds link time is hardly a concern.

Yes if you are the only developper and never received nor accepted external contributions or if you managed to get permission from every single person who contributed or replaced their code with your own.

Yes. Generally you need permissions from contributors (either asking them directly or requiring a contribution agreement that assigns copyright for contributions to either the author or the org hosting the project), but you can relicense from any license to any other license.

That doesn't extinguish the prior versions under the prior license, but it does allow a project to change its license.

Great, I'll keep a look out but will hold off on using it for now.

Then it'd be the /usr/bin/ld as I believe my solution was for before they moved the linker it seems.

/usr/bin/ld will correctly invoke the right linker, it's a stub to look at your developer dir and reexec.

I totally see the point of this, but still, you have to admit this is pretty funny:

> Developers sometimes experience trouble debugging the quarter-million line amalgamation source file because some debuggers are only able to handle source code line numbers less than 32,768 [...] To circumvent this limitation, the amalgamation is also available in a split form, consisting of files "sqlite3-1.c", "sqlite3-2.c", and so forth, where each file is less than 32,768 lines in length

*concatenates

Apologies for the typo. And now it is too late to edit the post.

>Secondly, if you think any compiler is meaningfully doing anything optimal >>("whole program analysis") on a TU scale greater than say ~50kloc (ie ~10 files) >relative to compiling individually you're dreaming.

That's wrong. gcc generates summaries of function properties and propagate those up and down the call tree, which for LTO is then build in a distributed way. It does much more than mere inlining, but even advanced analysis like points to analysis.

https://gcc.gnu.org/onlinedocs/gccint/IPA.html https://gcc.gnu.org/onlinedocs/gccint/IPA-passes.html

It scales to millions of lines of code because it's partioned.

> if you think any compiler is meaningfully doing anything optimal ("whole program analysis") on a TU scale greater than say ~50kloc (ie ~10 files) relative to compiling individually you're dreaming.

You can build the Linux kernel with LTO: simply diff the LTO vs non-LTO outputs and it will be obvious you're wrong.

SQLite3 may be a counter-example:

https://sqlite.org/amalgamation.html

compiling crates in parallel is fast on a good machine. OTOH managing C++ dependencies without a standard build & packaging system is a nightmare

I don’t think I can use wasm with simd or syscalls, which is the bulk of my work.

How is this different than dynamically linking the business logic library?

Like ICF? Wait no, everyone supports that except GNU ld.

Can you name a few of these features, for those of us who don't know much about linking beyond the fact that it takes compiled object files and makes an executable (and maybe does LTO)?

Certainly no intention to be negative. Not having run the numbers, I don't know if the older ones got slower over time due to more features, or the new ones are squeezing out new performance gains. I guess it's also partly that the bigger codebases scaled up so much over this period, so that there are gains to be had that weren't interesting before.

Where is the effort going now? lld?

> It is possible to use gcc to invoke wild as the linker, but currently to do that, you need to create a directory containing the wild linker and rename the binary (or a symlink) to "ld", then pass `-B/path/to/directory/containing/wild` to gcc.

Instead of renaming and passing -B in, you can also modify the GCC «spec» file's «%linker» section to make it point to a linker of your choice, i.e.

  %linker:
  /scratch/bin/wild %{wild_options}

It is possible to either modify the default «spec» file («gcc -dumpspecs») or pass your own along via «-specs=my-specs-file». I have found custom «spec» files to be very useful in the past.

The «spec» file format is documented at https://gcc.gnu.org/onlinedocs/gcc/Spec-Files.html

Ah, good to know, thanks!

May be it's worth filing a feature request for gcc to have parity with clang for arbitrary linkers?

That's the reason to use llvm as part of Rust compiler toolchain, not to use gcc or clang as linker manager?

> Mold gets about a 10% speedup by using a faster allocator (mimalloc). I've tried using mimalloc with Wild and didn't get any measurable speedup

Perhaps it is worth repeating the experiment with heavy MLoC codebases. jmalloc or mimalloc.

Rust is a perfectly fine language, and there's no reason you should not be able to implement fast incremental linking using Rust, so - I wish you success in doing that.

... however...

> code patterns that are fine in Rust due to the borrow checker, would be footguns in languages like C or C++,

That "dig" is probably not true. Or rather, your very conflation of C and C++ suggests that you are talking about the kind of code which would not be used in modern C++ of the past decade-or-more. While one _can_ write footguns in C++ easily, one can also very easily choose not to do so - especially when writing a new project.

Yep in my case I have 11 * 450MB executables that take about 8 minutes to compile and link. But for small iterative programming cycles using the standard linker with g++, it takes about 30 seconds to link (If I remember correctly). I tried mold and shaved 25% of that time, which didn't seem worth the change overall; attempted wild a year ago but ran into issues, but will revisit at some point.

Exactly. But also even in build-from-scratch use-case when there's a multitude of binaries to be built - think 10s or 100s of (unit, integration, performance) test binaries or utilities that come along with the main release binary etc. Faster linkers giving even a modest 10% speedup per binary will quickly accumulate and will obviously scale much better.

True, I didn't think of that. However, the root cause here perhaps is fat binaries? My preferred development flow consists of many small self-contained dynamically linked libraries that executables link to. Then you only have to relink changed libraries and not executables that depend on them.

ELF(COFF) should now be only an assembler output format on modern large hardware architecture.

On modern large hardware architecture, for executable files/dynamic libraries, ELF(PE[+]) has overkill complexity.

I am personnally using a executable file format of my own I do wrap into an "ELF capsule" on linux kernel. With position independent code, you kind of only need memory mapped segments (which dynamic libraries are in this very format). I have two very simple partial linkers I wrote in plain and simple C, one for risc-v assembly, one for x86_64 assembly, which allow me to link into such executable file some simple ELF object files (from binutils GAS).

There is no more centralized "ELF loader".

Of course, there are tradeoffs, 1 billion times worth it in regards of the accute simplicity of the format.

(I even have a little vm which allows me to interpret simple risc-v binaries on x86_64).

Does it at least support -Ttext, -Tdata, etc.?