The Ingredients of a Productive Monorepo

Do you have go-to multi repo automations and other tools for operating across multiple repos?

Note that the costs depend on the scale and size of the monorepo, and a polyrepo is not without costs either (which people often like to forget or ignore because they are less visible).

I spent some years at small to mid-sized companies (~30-100 devs) that would have profited from a monorepo. However, people were in the habit of splitting repositories every other month. Sometimes a single team would own a given repository, but more often than not several teams would contribute to each repository.

I have serious PTSD from that time. Every single pipeline in every repo worked differently, tools were different, setup, scripts and commands were different. In some repositories you could trust the CI pipeline, in others you absolutely couldn't. CI performance gains in one repo wouldn't translate to another. And of course you often still had some tech debt lying around from when repositories got split and people forgot to clean up thoroughly. Uggh.

Now, to be fair, people did put quite a bit of effort into their setups and pipelines and all that, and it wasn't that the outcome of that effort in and by itself was totally bad. Not at all. But everyone did things differently. And often teams just re-did the work other people had already done & solved – there was a ton of overhead.

Finally, the worst part were inter-repository dependencies. People argued coupling would be loose and all that, so we could easily split repos, but in reality there were so many implicit dependencies because ultimately all repositories made up one big application. Merge requests had to be coordinated. One repo imported files from another. Later a CI pipeline in one repo triggered a pipeline in another repo…

This brings me to another problem with polyrepos: They harm discoverability and cause certain dependencies not to be encoded in code. In a monorepo, in contrast, when people wonder where they can find X, or whether anyone uses or depends on Y, the answer is usually only a `ripgrep` or `git grep` away. In a polyrepo, however, they need to know where to look first.

Born from all these experiences, my mantra has been: Split repos only if you really have to and have a very good reason! (Entirely different applications or ecosystems; different parts of the code owned by different & independent teams; …)

That's confusing two different things, though.

A monorepo does make changes atomic in the code. There's no trap there.

You're talking about deployment, and yes when deployment is staggered, then obviously all atomic changes need to be backward-compatible, or else be very carefully orchestrated. But that doesn't have anything to do with monorepo vs polyrepo. That's just staggered deployment.

You have to deal with backwards compatibility in both cases. But at least with the monorepo you can see and track and merge all the changes related to a feature in one place, and be able to roll them back in one place.

That push to fragmentation is in large part because of hard lessons learned from the problems of a monolith.

The answer is IMO somewhere in between. Microservices can get too tiny and thus the system becomes impossible to understand. However a monolith is impossible to understand as well.

The real problem is you need good upfront architecture to figure out how the whole system fits together. However that is really hard to get right (and Agile discourages it - which is right for small projects where those architects add complex things to mitigate problems you will never have)

In my experience microservices are easier to manage and understand when organized in a monorepo.

Large software projects cycle back and forth between fragmentation and defragmentation. There is no right answer, only what's right for each project at the time.

Relevant xkcd: https://xkcd.com/2044/

Yes and No.

One top level "superrepo" is the best of both worlds. You don't break the limits of git, you can include third party repos, you can seal off access to some secret repos. All while still giving benefit of everything checked out in one place. Tooling for flattening the git log exists and is relatively cheap to build yourself, It's easy to design CI that runs both big and small jobs in the superrepo and the subcomponents.

Nested submodules = Game over. Don't ever do it. This is the worst of both worlds. Common components included twice or more need to be aligned to exact same revision, without help of semantic versioning. Components far down the dependency tree need to be uplifted recursively through a chain of submodules, resulting in 10 commits with no common CI for one tiny change.

Is a mono repo the answer, or is the real problem you just have a bad repo split.

I can't answer that question, and there are reasons to go monorepo anyway. However if your problem is a bad polyrepo split going to monorepo is the obvious answer, but it isn't the only answer. Monorepo and polyrepo each have very significant problems (see the article for monorepo problems) that are unique to that setup. You have to choose what set of problems to live with and mitigate them as best you can.

The general rule is that things should be versioned together that change together. Separate repositories should be thought of similarly to separately versioned libraries. Dependencies between repositories should have stable interfaces. Design decisions that are likely to change should be encapsulated within a module, so that these decisions are hidden from other modules (a seminal paper about that is [0]). These considerations should guide any split into separate repositories.

[0] https://wstomv.win.tue.nl/edu/2ip30/references/criteria_for_...

"11 repos with 4-5 merge requests" doesn't sound like Google-level, so I don't see why a monorepo wouldn't work without much work.

I think you might just have a badly architected backend. get rid of your microservices first and then we'll see how you're feeling

As an asside, I've found IntelliJ very helpful in this situation as it can load many repos into one project then doing commits / pushes / branches etc across various repos at the same time just seemed to work the way I wanted without much thinking about it.

use git subtree - first to concatenate the minor repos into one major repo, and then subtree split from that point forward to publish subtree folders back to the minor repos, if needed (e.g. open source projects to github). works for us with about 8 minor repos, eliminated submodule pain entirely. only the delivery lead has to even know the minor repos exist.

Yup, at work I have a few projects split across several repos in like four languages. A completely new feature implemented across the whole stack involves PRs in up to 8 different repos. Potentially more.

To be totally honest, yes this is an unbelievable pain in the ass, but I much prefer the strict isolation. Having worked with (much, much) smaller monorepos, I find the temptation to put code anywhere it fits too much, and things quickly get sloppy. With isolated repos, my brain much more clearly understands the boundaries and separation of concerns.

Then again, this results in a lot of code duplication that is not trivial to resolve. Submodules help to a degree, but when the codebase is this diverse, you're gonna have to copy some code somewhere.

I view it sort of like the split between inheritance and composition. You can either inherit code from the entire monorepo, or build projects from component submodules plugged together. I much prefer the latter solution, but clearly the former works for some people.

Do these 11 repos end up in separate binaries?

Because it sounds like you just need flag based feature releases.

I do think the 'run all tests on CI' part is not that fine, it bites a lot earlier than the others do. Git is totally fine for a few hundred engineers and 100ish services (assuming nobody does anything really bad to it, but then it fails for 10 engineers anyway), but running all tests rapidly becomes an issue even with tens of engineers.

That is mitigated a lot by a really good caching system (and even more by full remote build execution) but most times you basically end up needing a 'big iron' build system to get that, at which point it should be able to run the changed subset of tests accurately for you anyway.

That sounds worse than either option. At that point put it all in one repo with a directory for each language.

And then at some point your Rust people write a Python module in Rust via pyo3, and it has to be integrated into Python build system and Python type checkers, but also needs local rust crates as build dependencies and local python packages as runtime dependencies... hm.

Coupling and Cohesion likely has nothing to do with the language.

This will start to become a problem if the stacks need to communicate with each other using versioned protocols.

Every time I've been monorepos compares with polyrepos it's always "monorepo plus millions of dollars of custom tool engineering" vs "stock polyrepo"

Why can't we add millions of dollars of tool engineering on top of polyrepos to get some of the benefits of monorepos without a lot of the pain? E.g. it wouldn't be too hard to create "linked" PRs across repos for changes that span projects, with linked testing infrastructure

And I don't see how discovery changes significantly from browsing through loads of repositories instead of loads of directories in a repository

Monorepo monoliths make it hard to experiment. Getting something as benine as a later version of .NET becomes a mammoth task requiring the architecture team and everything stays old. Want to use a reasonable tool? No chance.

Yeah. My experience is that the teams managing shared repos tend to shift responsibility for integrating their changes onto their users. They then also more often make breaking changes because they’re insulated from the costs of those changes.

We have a person deditated to bringing in changes to our polyrepo. Nothing is considered done until it is in his mainline branches so there is incentive to get things integrated. Nothing goes in until it passes the full test suite, whith he verifies you ran before integrating and then runs again to be sure.

as someone who works on core parts that are lively to break everything I spend half of my time just integrating things and anouther quarter trying to figure out how to make my things either less core or not need changes so often.

I'd caution that a monorepo isn't a full fix to that. People often make multiple versions of libraries. You have separate 2.X and 3.X versions, with independent source code (or branches), and ask people to migrate to the new one.

There's not really a way around that when you need some behavioral change for the code using the library.

I think GP's claiming it's a feedback loop, not one-directional relationship. Communication structure of an organization ends up reflected in the structure of systems it designs, and at the same time, the structure of a system influences the communication structure of the organization building it.

This makes sense if you consider that:

1) Changes to system structure, especially changes to fundamentals when the system is already being built, are difficult, expensive and time consuming. This gives system designs inertia that grows over time.

2) Growing the teams working on a system means creating new organizational units; the more inertia system has, the more sense it makes for growth to happen along the lines suggested by system architecture, rather than forcing the system to change to accommodate some team organization ideals.

Monorepo/multirepo is a choice that's very difficult to change once work on building the system starts, and it's a choice you commit at the very beginning (and way before the choice starts to matter) - a perfect recipe for not a mere nudge, but a scaffolding the organization itself will grow around, without even realizing it.

Having done this a few times, I suspect the norm is that the decision is taken without a full understanding of the tradeoffs, both because the decision is taken before engineering has matured and can be hard to change later, and because the disadvantages are easy to downplay - "Sure, but our team will always stay on top of dependencies."

Typically someone has read a few blog posts like the ones linked to, and have some vague ideas of the positives but don't have a full understanding of how how the disadvantages will shape their workflow.

I've seen people with experience at hobby or small scale successfully campaigning for a switch at work and then hitting a wall - in both directions. Updating every call site for a breaking change doesn't sound that onerous, and at a small scale it isn't. Having each team update versioned depencies doesn't sound that hard, and at a small scale it isn't.

Just like with languages, don't listen to anyone who tells you this will solve all your problems. One of the options are merely the least bad for your situation.

required reading: https://en.wikipedia.org/wiki/Conway%27s_law

> Yeah - I worked on that build team probably at the same place you did!

I did not work at that place but the story sounds very familiar – I believe there might have been a blog post about that remote development environment here on HN some time ago?

I don't think you are wrong at all. As we are all founders with an OK salary and this is our area of expertise, so we're able to take advantage of our previous experiences and reap the benefits. We're probably uniquely positioned here.

I had massive issues at my previous employer with Bazel. They did not try to make Bazel work for non-SREs, which as you can imagine didn't work very well. So it's definitely not a silver bullet!

We should probably write a blog post about our setup!

Bazel is notoriously difficult to set up to begin with, but not so difficult to use after that. What was your bad experience?

> My experience with Bazel has been extremely bad

Would you mind elaborating and providing some examples of what was bad?

We have a monorepo built using bazel, and at first when new to bazel, I was pretty frustrated. But now I can't think of any issue I've had with it recently.

But we do have a relatively simple setup.

Not the OP but you can use tools like direnv + mise/asdf/nix so that every time a developer cd's into the monorepo, their shell environment loads a pinned, declaratively-configured set of dependencies and tools whose definitions are part of the monorepo.

The way I'd naively set up something like OP described would be to have direnv + nix flake deliver you a copy of bazelisk, and then have some custom shell scripts added to $PATH that alias `go = bazel run go`, `kubectl = bazel run kubectl` or whatever custom wrappers you want.

(Handwaving and I know the above isn't quite correct)

Hi! Previously mentioned ex-Google SRE! There are a few layers to it - to make it work "ok" you need to first have a tool runner wrapper rule that does something similar to:

```

ctx.actions.write(output="""

tool_path=$(realpath {tool_short_path})

cd ${{BUILD_WORKING_DIRECTORY}}

exec $tool_path

""".format(tool_short_path=tool.short_path)

```

The purpose of this rule is to ensure that the tool's CWD is actually where you are inside the repository and not within the runfiles folder that Bazel prepared for you.

The second step is to set up a symlink target, similar to this:

```

#! /usr/bin/env bash

tool_name=$(basename $0)

exec -a "$tool_name" bazel run --ui_event_filters=-info,-stdout,-stderr --noshow_progress //tools/bin:$tool_name -- "$@"

```

We need to filter out all UI events since for some tools we intercept (such as jq) it expects the stdout to be clean from other output when used programmatically.

We then create a symlink for each tool name (say kubectl) to this script from another folder, and then we use `direnv` to inject the folder of symlinks into the user's paths with an `.envrc` file in the repository root like this:

```

PATH=$PWD/tools/path:$PATH

```

We have had this in place for quite a while now - it does seem like this pattern has caught some more wind and buildbuddy.io has released a ruleset: https://github.com/buildbuddy-io/bazel_env.bzl paired with https://github.com/theoremlp/rules_multitool achieves the same thing that we have built internally, the main difference being that with the bazel run wrapper we have made, you always run the latest version, whereas with the bazel_env pattern you need to manually rerun their target to get the latest binaries. :)

Probably something similar to a py venv that you run to activate some aliases in your environment?

I think this take is quite shallow and lacks insight into how one would actually build a somewhat complex technical platform.

We are not using a microservice pattern at all. I am not sure where you get that from. If anything we have several "macro services".

Our final setup is quite complex as we are building a literal cloud provider, but in practice we have a Go API, a Docker registry, a Temporal Worker and a Kubernetes controller. Whats complicated is everything else around it. We run our platform on bare-metal and thus have auxiliary services like a full-blown Kubernetes cluster, Ory Hydra + Kratos, SpiceDB, Cilium, Temporal Cluster + Workers and some other small things. We need to be able to test this locally to feel safe to release to production. And in turn our production environment is almost identical to our local environments.

None of that would be possible unless we've done something similar to what we have built today. Most companies cannot run their entire stack on their laptop, more unlikely that they could run a full cloud provider.

Keeping code in Dropbox kinda sucks even with 1 or .5 developers though. That said, yeah, a regular old git or (I assume, never used it) svn seems fine.

Android is only recently switching to quarter releases instead of yearly. Most. Popular Linux distros only have major releases every 6 months. While chrome cuts a release branch every 4 weeks, it soaks it in a beta channel for another 4. Same goes for the rust compiler toolchain, albeit on a 6 week cadence.

It’s more common than you think if you expand your view of release a bit. On the one hand you very much still have shrink-wrap software (for example, all firmware) that ships on a very slow cadence.

On the other hand even the big tech companies will only expose code paths very slowly and very conservatively. Meta’s Threads.app for example combined both a constant churn of innovation on master with a very measured gating of new features shipping to the public.

The best teams do indeed, as you say, ship and test finished builds on a weekly or daily basis even if the stuff that gets under the customers’ / users’ / clients’ noses appears on a far less regular basis. After all, any kind of severe bug could necessitate a release at any moment.

not all the world is a web site or even internet connetted. not all the world has no safety concerns.

if you work in medical or aviation areas every release legally needs extensive - months - testing before you can release. If there are issuse found in that testing you start over. Not all tests can be automated.

i work in agraculture. the entire month of July there will be nobody in the world using a planter or any of the software on it. there is no point in a release then. the lack of users means we cannot use automated rollback if the change somehow fails for customers - we could but it would be months of changes rolled back whe Brasil starts planting season.

Every company that uses SAFe agile has quarterly, or bi-quarterly, releases [1].

[1] https://www.servicenow.com/docs/bundle/yokohama-it-business-...