The next generation of Bazel builds

> other concepts that may not be so critical: `query` vs `aquery` vs `cquery`, action-graph vs configured-action-graph vs target-graph, providers vs outputs, macro vs rule-impl, etc

Almost all of the distinctions you mentioned are related to the way that Bazel has the concept of a "target", which lets the build graph work at a higher level than individual files.

Suppose you write the following in a BUILD file:

    cc_library(
        name = "foo",
        srcs = ["foo.c"],
        hdrs = ["foo.h"],
    )

    cc_library(
        name = "bar",
        srcs = ["bar.c"],
        hdrs = ["bar.h"],
        deps = [":foo"],
    )

It's nice to be able to query the build graph at this high level. It lets you see the relationship between components in the abstract, rather than a file-by-file level. That is what "bazel query" does.

But sometimes you might want to dig deeper into the specific commands (actions) that will be executed when you build a target. That is what "bazel aquery" is for.

Macros vs. rules is basically a question of whether the build logic runs before or after the target graph is built. A macro lets you declare a bit of logic where something that looks like a target will actually expand into multiple targets (or have the attributes munged a bit). It is expanded before the target graph is built, so you won't see it in the output of "bazel query."

If you took away the target graph, I think you'd take away a lot of what makes Bazel powerful. A key idea behind Bazel is to encapsulate build logic, so that you can use a rule like cc_library() without having to know how it's implemented or exactly what actions will run.

I don't say this to minimize any of the pain people experience when adopting Bazel. I'm actually curious to learn more about what the biggest pain points are that make it difficult to adopt.

I care a ton about fast and accurate build systems but one issue I think we haven't solved is that: people do not want to use other build tools for their language. "Why isn't it Cargo? Why not use NPM/Yarn? Why not use Pip? Why not CMake?" These questions are often rhetorical because they do not care about build systems. They don't care about the design. They don't care if your CI could be 5x faster. You will never make them care. It's good enough. You must have absolutely zero externalized cost (and therefore put in a lot of effort) to get over this hurdle. There's seemingly no way around it.

The reason a lot of people like Bazel is, I think, tools like Gazelle -- which reduce that whole problem back to "Run gazelle" and all the crap is taken care of for you. Dependencies, BUILD files, etc. People constantly talk about the "complexity" aspect, but very few people appreciate how complex Cargo, NPM, Yarn, Cabal, Dune, internally are. Because they just run "build", and it works. Bazel, Buck2, Mill, etc will all have this problem unless huge effort is put in.

TBH, this is one of the reasons why I think Nix has wildly succeeded in the past few years while more fine-grained and scalable systems have had adoption problems -- despite its numerous, numerous flaws. You get to Bring-Your-Own-Build-System, and Nix along with the blood of 10,000 upstream contributors keeps the juice flowing, and it's cached and hermetic so you see real savings. That greatly eases people into it. So they adopt it at all points on the curve (small, medium, huge projects), because it works with what they have at all those points. That makes them willing to get deeper and use the tool more.

Bazel runs into the problem that it expects to have the complete well-defined understanding of the inputs and outputs for your project. This might have made sense when Blaze was first designed and projects were done in languages with compilers that had rigid inputs and outputs. But now we're in a world where more and more systems are becoming layers of compilers, where each compiler layer wants to just have a bunch of dependencies thrown at it. In a frontend project, it wouldn't be weird for Tailwind CSS to be compiled and embedded in SCSS, where it's pulled into a JSX module via some import that magically provides type checking with the CSS under the hood. And so you either need to handwave over it and lose some of the benefits of incremental builds, or spend time getting it to work and making it continue to work as you add new layers.

So in my mind, Bazel is no longer worth it unless the savings are so great that you can afford to staff a build team to figure these things out. Most teams would benefit out of using simple command runners instead of fully-fledged build systems.

I'm glad to see someone else describe their experience this way too.

bazel has arrived at $WORK and it has been a non-trivial amount of work for even the passionate advocates of bazel. I know it was written by the Very Smart People at google. They are clearly smarter than me so I must be the dummy. Especially since I never passed their interview tests. :-)

Of course given all things google, by the time I'm fully onboard the train, the cool kids will be making a new train and then I'll have to hop onto that way to enjoy the rewards of the promised land that never quite seem to arrive.

Bazel is an investment to learn for sure but your effort estimates are way overblown. We have a Python, Go, and Typescript monorepo that I setup for our team and rarely have to touch anything. Engineers rarely think about Bazel as we use Gazelle to generate all our build files and have patterns for most everything we need to do.

Compared with build efforts using other tools and non-monorepo setups at other companies the effort here has felt much reduced.

I agree. The biggest issue with Buck 2 for me (apart from documentation) is the lack of something like bzlmod. There's actually a decent number of modules available for Bazel now:

https://registry.bazel.build/all-modules

But with Buck2 you're stuck with `http_archive` and vendoring.

I miss Meta's arc + buck2 + eden + "hg" + "phabricator" + testing infrastructure that does a lot of acceleration of builds, testing, and landing commits without rebasing.

The biggest problem with everything NotBazel in this space is IDE support. JetBrains have already moved their build for IntelliJ to it, and are aggressively adding native first party support.

yeah. author misses that buck2 is nix like. seems he is in mind of prev gen assembly systems.

Buck2 doesn't seem to support JS/TS, so it's not ready for our use case.

> The most frustrating part of Bazel is how shoddy the workmanship is.

Without commenting on the above statement:

> For example, Bazel throws away your analysis cache when you change flags that have nothing to do with what's being built or how, like flags that change what tests are run.

I don't think this is a good example. Bazel's analysis cache clearing is to preserve correctness in the context of a legitimately algorithmically difficult problem. The fact Bazel has this limitation is a testament toward its correctness strengths. I'm not aware of systems that have solved that algorithmic problem but curious if anyone knows any.

Also Bazel avoids that problem for most "flags that change which tests are run", since that subset of the problem is more solvable. --test_env was a notable exception, which was fixed (https://github.com/fmeum/bazel/commit/eb494194c1c466f7fd7355...) but not sure if it's in the latest Bazel release yet? But generally changing --test* has much smaller performance impact.

Everyone says they want a "tiny" "minimal" "lean" build-system, but there is lots of real complexity in these areas:

- Cross-compilation and target platform information

- Fetching dependencies

- Toolchains

I'm not sure a system that solves these would still be considered minimal by most, but those are table-stakes features in my view.

If you don't need these things, maybe stick with Make?

It's not Rust, it's not production ready, and I haven't actually used it (only read the README and blog posts), but I really enjoy the ideas behind https://github.com/256lights/zb - see https://www.zombiezen.com/blog/2024/09/zb-early-stage-build-... for a list of these ideas.

https://github.com/facebook/sapling/tree/main/eden

I'm still unsure if a complete, viable eden solution was released with all of its necessary components.

> Starlark is intended to be simple. There are no user-defined types, no inheritance, no reflection, no exceptions, no explicit memory management. Execution is finite. The language does not allow recursion or unbounded loops.

That has some pretty useful properties for a config language that I'm not sure TypeScript helps much on. That said, Lean also has these properties too as an alternate language with good IDE support, and it would be pretty fun to write proofs of your build scripts.

You'll be happy to know that the Starlark team has recently started working on adding static typing! See https://github.com/orgs/bazelbuild/projects/21 and https://bazel.build/rules/language#StarlarkTypes

> a lot of performance gotchas

Can you list some examples?

I worked on a project that used Bazel for a bit (but ended up not being a great fit).

I think the part I hated most about Bazel was the fact that there were in fact two dialects; one for BUILD files and one for .bzl files. The things you do in them are different but the documentation is always vague on what goes where.

I'm curious, how did you run into performance issues with starlark?

Performance of Java in general and Bazel in particular tends to be not CPU-limited but memory-limited. There was a time when MacBooks maxed out at 16 GB, and running Chrome + an IDE + large Bazel build on a machine with <16 GB memory was not a great experience.

Why do you think this author (a long-time senior Blaze maintainer) believes this? Do you think the Blaze team just not have the necessary expertise?

Proper, dependency-aware build caching makes or breaks a build system. Meta's buck2 builds tend to be crazy fast because of this and it can also do synthetic checkouts with arc/"hg" via eden.

You're not wrong about build systems being workflow engines, but in my experience the risk of over-generalizing is a worse user experience. One of the most valuable lessons I've learned in my career is that a clean UX often requires messy abstractions underneath.

These were not well-guarded secrets (and the ideas were pretty "obvious" if you had faced the problems before in dealing with Bazel and large codebases). In talking to people at the conference, several knew about these efforts (and some even knew the codenames), so I assume these had been mentioned here-and-there in discussion threads.

I can say with full certainty that thousands of engineers at Google were feeding their day to day LLM assisted coding queries in to the ChatGPT web ui for about 2 years before a custom Chrome extension was pushed to employee laptops to stop it.

So GPT has likely post-trained on a lot of Google code knowledge.

50 lines in a single .cc source file which is only compiled once to produce the final artifact - sure, easy to handle.

Now consider that you are editing 50 lines of source for a tool which will then need to be executed on some particular platform to generate some parts of your project.

Now consider that you are editing 50 lines defining the structure and dependency graph of your project itself.

That rule misses important cases:

• Adding a file. It hasn't been read before, so no tasks in your graph know about it. If you can intercept and record what file patterns a build tool is looking for it helps, but you can't easily know that because programs often do matching against directory contents themselves, not in a way you can intercept.

• File changes that yield no-op changes, e.g. editing a comment in a core utility shouldn't recompile the entire project. More subtly, editing method bodies in a Java program doesn't require the users to be recompiled, but editing class definitions or exposed method prototypes does.

• "Building" test cases.

• You don't want to repeat work that has been done before, so you want to cache it (e.g. switching branches back and forth shouldn't rebuild everything).

If your system is C based, tup [0] fulfills your request by watching the outputs of the specified commands. It isn't, however, appropriate for systems like java that create intermediate files that the developer didn't write [1].

[0] https://gittup.org/tup/ex_a_first_tupfile.html

[1] https://github.com/gittup/tup/issues/113

Back to bazel, I am of the impression that some of its complexity comes from a requirement to handle heterogeneous build systems. For example, some python development requires resolving both python dependencies and C. Being good at either is a bunch of work; but, handling both means rolling your own polyglot system or coordinating both as a second class citizen.

It is well possible by changing 50 lines of code in a 10GB project you have to rebuild the entire project, if everything (indirectly) depends on what you just changed.

I mean, easy things are easy to build?

It is not at all uncommon to have changes percolate out into larger impacts than you'd expect, though. Especially in projects that attempt whole program optimizations as part of the build.

Consider anything that basically builds a program that is used at build time. Which is not that uncommon when you consider that ML models have grown significantly. Change that tool, and suddenly you have to rebuild the entire project if you didn't split it out into a separate graph. (I say ML, but really any simple linter/whatever is the same here.)

> the parts that need rebuilding are the parts that read those files when they were last built...

...and the transitive closure of those parts, which is where things get complicated. It may be that the output didn't actually change, so you can prune the graph there with some smarts. It may be that the thing changed was a tool used in many other rules.

And you have to know the complete set of outputs and inputs of every action.

And on and on.

I'm super into the component model but honestly the tooling is still not ready. Basically only LLVM (and really just C) has good enough tooling. The rest of the wasm tool chains are still heavily focused on core modules and if they support components it's only `wasi:cli/run`

Also dying for an explanation here

Why?

I don’t understand

What? Isn't wasm just a bytecode? How could you write rules in wasm?

The author clarifies that he wrote the section about Buck2 to demonstrate the need to be Bazel compatible (as opposed to Buck2), because the friction to try it out in a real code base is essentially insurmountable.