Monorepo – Our Experience

My only counter argument here, is when those 4 things deploy independently. Sometimes, people will get tricked into thinking a code change is atomic because it is in one commit, when it will lead to a mixed fleet because of deployment realities. In that world, having them separate is easier to work with, as you may have to revert one of the deployments separately from the others.

I prefer microservices/microrepos _conceptually_, but we had the same experience as your quoted text - making changes to four repos, and backporting those changes to the previous two release branches, means twelve separate PRs to make a change.

Having a centralized configuration library (a shared Makefile that we can pull down into our repo and include into the local Makefile) helps, until you have to make a backwards-incompatible change to that Makefile and then post PRs to every branch of every repo that uses that Makefile.

Now we have almost the entirety of our projects back into one repository and everything is simpler; one PR per release branch, three PRs (typically) for any change that needs backporting. Vastly simpler process and much less room for error.

> IMO, this is a huge quality of life improvement and prevents a lot of mistakes from not having the right revision synced down across different repos. This alone is a HUGE improvement where a dev doesn't accidentally end up with one repo in this branch and forgot to pull this other repo at the same branch and get weird issues due to this basic hassle.

This scenario only happens if you're working with a distributed monolith, you care nothing about breaking existing APIs, and you own all producers and consumers.

When this happens, your problem is obviously not how you maintain projects independently in separate repos. Your problems is how you're failing to preserve API contracts, and instead you go to the extreme of insisting in breaking all contracts at once.

It's also a huge red flag when you talk about "accidentally end up with one repo in this branch". Don't you know what you are pushing to production?

As always, monorepo talks are just a thin veil of optimization placed over a huge mess of major operational and technical problems.

There’s nothing preventing you from having a single pull request in for merging branches over multiple repos. There’s nothing preventing you from having a parent repo with a lock file that gives you a single linear set of commits tracking the state of multiple repos.

That is, if you’re not tied to using just Github of course.

Big monorepos and multiple repo solutions require some tooling to deal with scaling issues.

What surprises me is the attitude that monorepos are the right solution to these challenges. For some projects it makes sense yes, but it’s clear to me that we should have a solution that allows repositories to be composed/combined in elegant ways. Multi-repository pull requests should be a first class feature of any serious source code management system. If you start two projects separately and then later find out you need to combine their history and work with them as if they were one repository, you shouldn’t be forced to restructure the repositories.

The issue you faced stemmed from the previous best practice of "everything in its own repository." This approach caused major issues. Such as versioning challenges and data model inconsistencies you mentioned. The situations it could lead to are comedy sketches, but it's a real pain especially when you’re part of a team struggling with these problems. And it’s almost impossible to convince a team to change direction once they’ve committed to it.

Now, though, it seems the pendulum has swung in the opposite direction, from “everything in its own repo” to “everything in one repo.” This, too, will create its own set of problems, which also can be comedic, but frustrating to experience. For instance, what happens when someone accidentally pushes a certificate or API key and you need to force an update upstream? Coordinating that with 50 developers spread across 8 projects, all in a single repo.

Instead we could also face the problems we currently face and start out wirn a balanced approach. Start with one repository, or split frontend and backend if needed. For data pipelines that share models with the API, keep them in the same repository, creating a single source of truth for the data model. This method has often led to other developers telling me about the supposed benefits of “everything in its own repo.” Just as I pushed back then, I feel the need to push back now against the monorepo trend.

The same can be said for monoliths and microservices, where the middle ground is often overlooked in discussions about best practices.

They all reminded me of the concept of “no silver bullet”[0]. Any decision will face its own unique challenges. But silver bullet solution can create artificial challenges that are wasteful, painful, and most of all unnecessary.

[0]https://en.m.wikipedia.org/wiki/No_Silver_Bullet

I thought one of the whole points behind separate (non-mono)repos was to help enforce loose coupling and if you came to a point where a single feature change required PRs on 4 separate repos then that was an indicator that your project needed refactoring as it was becoming to tightly coupled. The example in the article could have been interpreted to mean that they should refactor the functionality for interacting with the ML model into it's own repo so it could encapsulate this aspect of the project. Instead they doubled down on the tighter coupling by putting them in a monorepo (which itself encourages tighter coupling).

I felt the same, the author seemed to downplay the success while every effect listed in the article felt like a huge improvement.

ironically was gonna come and comment on that same second block of text.

We went from monorepo to multi-repo at work and it's been a huge set back and disappointment with the devs because it's what our contractors recommended.

I've asked for a code deploy and everything and it's failed in prod due to a missing check in

It's not as much of a pain if your tooling supports git repos as dependencies. For example a typical multi-repo PR for us with rust is 1) PR against library 2) PR against application that points dependency to PR's branch, makes changes 3) PR review 4) PR 1 is approved and merged 5) PR 2 is changed to point to new master branch of commit 6) PR 2 is approved and merged

Same idea if you use some kind of versioning and release system. It's still a bit of a pain with all the PRs and coordination involved, but at every step every branch is consistent and buildable, you just check it out and hit build.

This is obviously more difficult if you have a more loosely coupled architecture like microservices. But that's self-inflicted pain

It’s a game of tradeoffs. Yes the situation described does sometimes happen. If you are intentional about how you break apart your repos you can avoid this most of the time. They don’t need to be micro repos, they can just be regular sized and driven by the data model.

Some benefits include:

* Separate repos evolve better than a monorepo. Particularly in early stage companies where pivots are frequent. * They keep tools just doing one thing well and prevent tying everything in the codebase together. * Dependencies are more flexible, this is particularly useful with ML libraries. * A lot of repo tooling is simpler with medium sized single purpose projects * Stronger sense of ownership boundaries

> > In the previous, separate repository world, this would've been four separate pull requests in four separate repositories, and with comments linking them together for posterity.

Tooling issue. Bitbucket will happily link PRs if instructed so.

> prevents a lot of mistakes from not having the right revision synced down across different repos.

Tooling issue. Depending on setup, "tooling" could be few lines in your favorite scripting language.

---

Monorepo steps around some tooling issues, but sacrifices access control, checkout speed

This "huge quality of life improvement" is "minor" compared to the lack of significant differences in more important aspects of version control that become neither easier nor more complex: for example resolving conflicts (files that used to be in another repository remain uninvolved) or creating, merging and killing branches (still related to the same features, bugfixes, modules, schedules etc.)

Why can't we have one PR, many repos?

Jibes. Not jives. The word is jibes.

Since we are indulging in generalizations from our past. With separate repos you end up with 10 "cores" that are radioctive DMZ's everybody is afraid to touch. And those "disciplined" public API's will be universally hated by everyone who consumes them.

Neither a monorepo nor separate repos will result in people being disciplined. If you already have the discipline to do separate repositories correctly then you'll be fine with a monorepo.

So I guess it's six on one hand, half dozen in the other.

I would advise annotating your experience anecdotes, which are surely valuable, with some information about team size, company size, corporate structure (tech team in a non tech corp, vs pure tech), age etc.

The meat is in the detail, I find.

> Separate repos for each team means that when two teams own components that need to interact, they have to expose a "public" interface to the other team--which is the kind of disciplined engineering work that we should be striving for.

This pattern has its own set of problems. Strict ownerships separation creates strong dependencies and capacity constraints on the project organization. A single team is rarely able to deliver a complete feature, because it will mean changes in a couple of services. If you go with a model where teams are allowed to build features in "foreign" services, you will still come to the situation that the owning team doesn't feel that responsible for something they haven't built / don't really understand. You can tweak it, involve the owning team more etc. but it has trade-offs / friction.

The worst anti-pattern coming from this is "we have dependency on service X / team Y, but their code reviews are usually very long and pedantic, threatening our delivery. Can we do it locally in our service instead?" which results in people putting stuff where it doesn't belong, for organisational reasons.

The problem is that DRY principles have be cargo culted way past the point of being useful.

People see two pieces of duplicate, or even similar code in a monorepo, and they feel the urge create some sort of shared directory to de duplicate them. The problem, is this introduces more edges to your graph, which over time increases the monorepo complexity exponentially. And who is going to maintain that shared directory?

Even in a mono-repo, you should probably start off by copy and pasting code around, until you figure out how to design and find ownership for an actual well thought through public api. And maybe for some code, that never happens.

Also, like many things, people cargo cult big tech ideas without understanding why they work. In Google, the monorepo has two key features: 1) every directory has an owner by default 2) the owner can control build target visibility lists.

That means that the ownership team controls who depends on them. By default, everything is private.

Basically, every directory should either be a project, owned by a team. Or some common library code, owned by the library maintainers.

The other thing, is library maintainers need to be discerning with what APIs and code they support. Again, a common refrain I here is “when it doubt, it’s better to put it in the company’s common library to promote code reuse”. That’s the exact wrong way to think about it. Library teams should be very careful about what APIs they support because they will be on the hook maintaining them forever.

This is all to say, I can see how monorepos fail. Because people cherry-pick the ideas they like, but ignore the important things which make them work at Google: 1) default code owners 2) visibility lists that default to private 3) discerning library teams that aggressively reject code additions 4) don’t be too dry, if it can’t be exposed in a public api, just copy and paste, just like you would in a microservice.

You can have well-defined interfaces without splitting it into many repos just like how you can have well-defined interfaces without splitting it into microservices. In fact, I've seen enough garbage to know that forcing the issue via these mechanisms just makes bad software worse.

Even if people are disciplined about versioned interfaces with documented compatibility (which is something I've personally never seen, except perhaps in a small number of very high quality open source repos), you still end up with deployment headaches. Making any sort of backwards incompatible change requires a high level of coordination and opens up the potential for mistakes that break service. With a monorepo you can deploy a backwards incompatible change all at once, or at the very least you can see in a single commit/PR what the changes were, instead of split across multiple repos.

Any old codebase will have their own version of Nina, Katya, Masha or even just an open reactor, monorepo or not.

I once worked on a codebase that had had a directory that no one was allowed to touch because it was "about to be decommissioned". When I looked into it more closely, it had been "about to be decommissioned" for 7 years and it was holding so many things up.

The problem with giving each team a repo and an API surface is that you create API boundaries where your organizational boundaries are, not necessary where your service boundaries are. And as your organizational structure evolves over time, your repo and API boundaries lag behind since it’s so difficult to make large scale shifts to the code.

This seems like an org/process problem and not a tech problem.

You can end up in this situation with or without a monorepo.

Code ownership is a scam

I've seen teams take this even further and vendor all dependencies.

This way a commit hash contains even the exact third party code involved.

I don't see how that relates to monorepos. Even using submodules a single commit hash will specify the commit hashes of every submodule (usually; you can actually set up submodules to point to a branch instead of a commit but I've never seen anyone do this).

There are definitely huge benefits to monorepos but I don't see how this is one.

I've also seen this with branching strategies. IMO the best branching strategy for open source projects is generally the worst one for commercial projects, and vice versa.

this should be the top comment

Meta open-sourced their complete stack: https://github.com/facebook/sapling

Microsoft released Scalar (https://github.com/microsoft/scalar) although it's not a complete stack yet but it is already planning on releasing the backend components eventually.

Have you tried Sapling? It has EdenFS baked in so it'll only materialize the files you touch and operations are fast because it has a filesystem watcher for activity so it doesn't need to do a lot of work to maintain a view of what has been invalidated.

I’m secretly hoping that google releases piper (and Mondrian); the gaming industry would go wild.

Perforce is pretty brutal, and the code review tools are awful - but its still the undisputed king of mixed text and binary assets in a huge monorepo.

We at Meta use Sapling, which is open source. Using EdenFS is what makes it super fast at scale.

https://sapling-scm.com/

https://engineering.fb.com/2023/06/27/developer-tools/meta-d...

fsmonitor and untrackedCache doesn’t help?

I believe .codeowners does that

This is something I’ve been preaching at work and it just falls on deaf ears. We aren’t even doing a great job with monolithic, why people think modular is going to _improve_ things is beyond my comprehension. I’ve pretty much decided no one actually cares and they just see it as an opportunity for résumé building. Someone want to enlighten me?

You can still publish packages and pull them down as (pinned) dependencies all within a monorepo.

I have a monorepo and like it. I think at some size you get enough people maintaining the repo that it becomes good? I also never want yaml, fwiw.

I don’t think CI/CD should really be a big worry as far as mono-repositories go as you can setup different pipelines and different flows with different configurations. Something you’re probably already doing if you have multiple repos.

In my experience the article is right when it tells you there isn’t that big of a difference. We have all sorts of repositories, some of which are basically mono-repositories for their business domain. We tend to separate where it “makes sense” which for us means that it’s when what we put into repositories is completely separate from everything else. We used to have a lot of micro-repositories and it wasn’t that different to be honest. We grouped more of them together to make it easier for us to be DORA compliant in terms of the bureaucracy it adds to your documentation burden. Technically I hardly notice.

Most CI/CD platforms will allow specification of targeted triggers.

For example, in GitHub[0]:

    name: ".NET - PR Unit Test"
    
    on:
      ## Only execute these unit tests when a file in this directory changes.
      pull_request:
        branches: [main]
        paths: [src/services/publishing/**.cs, src/tests/unit/**.cs]

[0] https://docs.github.com/en/actions/writing-workflows/workflo...

Wouldn't CI be easier with a monorepo ? Testing integration across multiple repositories (triggered by changes in any of them) seems more complex than just adding another test suite to a single repo.

I think this is one of the big reasons so many companies end up using Bazel as part of scaling their monorepos. It has many faults but one thing it does perfectly is “build and test everything affected by this commit”

You use a build system that sandboxes dependencies (Bazel, Buck2, Please, etc.) so that you know there are no undeclared dependencies. Then you can simply query the build system "what could have been affected by this change" and only build/test those things.

It doesn't have to be that way.

You can have a mono-repo and deploy different parts of the repo as different services.

You can have a mono-repo with a React SPA and a backend service in Go. If you fix some UI bug with a button in the React SPA, why would you also deploy the backend?

This mirrors my own experience in the SaaS world. Anytime things move towards multiple artifacts/pipelines in one repo; trying to understand what change existed where and when seems to always become very difficult.

Of course the multirepo approach means you do this dance a lot more: - Create a change with backwards compatibility and tombstones (e.g. logs for when backward compatibility is used) - Update upstream systems to the new change - Remove backwards compatibility and pray you don't have a low frequency upstream service interaction you didn't know about

While the dance can be a pain - it does follow a more iterative approach with reduced blast radiuses (albeit many more of them). But, all in all, an acceptable tradeoff.

Maybe if I had more familiarity in mature tooling around monorepos I might be more interested in them. But alas not a bridge I have crossed, or am pushed to do so just at the moment.

I couldn't disagree more. DevOps probably likes it but from software dev standpoint it's not helpful at all. Had to stave that thinking off at my last job (because of devops). It was obvious it was going to be a huge waste of time with a continuing negative payback. It was trivial to essentially put a description file for our repo of 7 or so components.

Is that really an issue? Huge companies like Microsoft and Google use monorepos, and they hire tens of thousands of people and contractors all with access to the code.

I think it's a natural fear but the reality is that a) most people don't leak source code, and b) access to source code isn't really that valuable. Most source code is too custom to be useful to most other people, and most competitors (outside China at least) wouldn't want to steal code anyway.

Actually I did find this answer on how Google does it and apparently they do support some ACLs for directories in their monorepo. Microsoft uses Git though so I'm not sure what they do.

https://www.quora.com/If-Google-has-1-big-monorepo-how-do-th...

> a monorepo with N completely independent long-lived branches

That's not a monorepo.

I would be hard pressed to call the Linux Kernel a mono repo.

It’s all one kernel really.

The Kernel is a monolith, but that doesn’t make its repo a mono repo.

FreeBSD, on the other hand, is a mono-repo. It has the kernel, all the user mode tools, basically everything in a single repo.

That is very different from the Linux ecosystem as whole.

Linux is not a mono repo.

Monorepo is just a single repo. Yup.

Git submodules have some places where you can surprisingly lose branches/stashed changes.

> Does it just mean a single version-controlled repository of code?

Yeah- they idea is that all of your projects share a common repo. This has advantages and drawbacks. Google is most famous for this approach, although I think they technically have three now- one for Google, one for Android, and one for Chrome.

> They seem like a great solution to me

They don't work in a team context because they're extra steps that people don't do, basically. And did some reason a lot of people find them confusing.

> What the hell is a monorepo? I feel dumb for asking the question, and I feel like I missed the boat on understanding it, because no one defines it anymore. Yet I feel like every mention of monorepo is highly dependent on the context the word is used in. Does it just mean a single version-controlled repository of code?

In my mind, a mono repo is one company, one (or a very small number of) source code repository. When I started working at Yahoo, everything was in CVS on one hostname (backed by a NetApp Filer), that was a mono repo; when you got into the weeds, there were actually a couple separate repo; Yahoo Japan had a separate repo, and DNS and prod ops had a separate repo, and a couple more, but mostly everything in one; just organized by directories so most people only checked out part of the repo, because not many people needed to look at all the code (or had disk space for either). That evolved into separate SVN repos for each group that wanted to move to SVN. I assume they moved to git at some point after I left.

Same deal when I was at Whatsapp. When I started, we had one SVN repo that everyone shared --- that was a mono repo; when we moved to git, each client had their own repo, server had a repo, and there was a common repo for docs and other things that needed sharing. Facebook had a lot of repos, but one repo was very large and requires 'monorepo' style tools. As a first step for monorepo style tools; in a large company with a large git repo; you need something to sequence commits, because otherwise everyone gets stuck on the git pull; git push loop until they manage to win the race. This wasn't an issue with a large CVS repo, because commits are file based, and while you might have conflicts within your team, you didn't need a global lock; I don't remember having issues with it in SVN either, but my memory is fuzzy and the whole company SVN repo I had was a lot smaller than the whole company CVS repo.

Maybe, I'd say a monorepo is a large repo where the majority of users/developers aren't going to need or want most of the tree.

> Can these issues with sync'ing repos be solved with better use of `git submodule`? It seems to be designed exactly for this purpose. The author says "submodules are irritating" a couple times, but doesn't explain what exactly is wrong with them. They seem like a great solution to me, but I also only recently started using them in a side project

I don't use submodules often, and I'm not sure if some of the irritations have been fixed, but in my use I run into two things: a) git clone requires additional work to get a full checkout of all the submodules; b) git pull requires additional work to update the submodules. I'm sure there's some other issues with some git features; but I was actually fine with CVS and don't really care about git features :P

That's the thing. All the pitfalls of multi-repos also disappear with good tooling and regular maintenance.

Neither one has an actual edge. Yet you can find countless articles from people talking about their experience. Take those as a hint about what kind of tooling you need, not about their comparative qualities.

Not to mention when you need to make cross repo changes in development, and you have to set up a whole web of local repointing in package manifests. Repo is a hard boundary. Sometimes you need one. But to create a boundary when you don’t have to, on things that are deeply interconnected and have nowhere near a stable api? Utter madness imo.

Presumably your language has a package manager which can do that for you? (or you could distribute it as an image). I guess you do have to decide which version you should depend on, but putting that power in your hands is sort of the point.

> you should have one monorepo per project (api, frontend, backend, mobile, etc. as long as it's the same project)

that's not a monorepo!

Unless the singular "project" is stuff our company ships, the problem you have is of impedance mismatch between the projects, which is the problem that an actual monorepo solves. for swe's on individual projects who will never have the problem of having to ship a commit on all the repos at the "same" time, yeah that seems fine, and for them it is. the problem comes as a distributed systems engineer where, for whatever reason, many or all the repos need to be shipped at the ~same time. or worse - A needs to ship before B which needs ship before C but that needs to ship before A, and you have to unwind that before actually being able to ship the change.

> when a company puts completely different projects together inside a single repo.

This is google3. It was absolutely loved by the majority of devs. If you change a line of code, all dependencies are type checked and tested and also a bunch of other things. It keeps so many versioning issues out.

One of the big reasons why the JS ecosystem is so fragmented compared to Go or even Rust is the leftpad-sized packages with 10 config files that are out of date. Not to mention our friend peerDependencies, who needs no introduction.

> But it can also breed instability, as you can upgrade other people's stuff without them being aware

This is why Google embraced the principle that if somebody breaks your code without breaking your tests, it's your fault for not writing better tests. (This is sometimes known as the Beyonce rule: if you liked it, you should have put a test on it.)

You need the ability to upgrade dependencies in a hands-off way even if you don't have a monorepo, though, because you need to be able to apply security updates without scheduling dev work every time. You shouldn't need a careful informed eye to tell if upgrades broke your code. You should be able to trust your tests.

In my experience this turns to nightmare when (not if, when) there is need to make changes to the libraries and app at the same time. Especially with libraries it's often necessary to create a client for an API at the same time to really know that the interface is any good.

...just curious, what's your line of work where you're still using SVN?

Working on a monorepo where we have hundreds (possibly thousands) of projects each with a different version and release schedule. It actually works quite well, the dependencies are always in a good state, it's easy to see the ramifications of a change and to reuse common components.

No offense but I think you're doing monorepos wrong. We have more than 100 applications living in our monorepo. They share common core code, some common signals, common utility libs, and all of them share the same build.

We release everything weekly, and some things much more frequently.

If your testing is good enough, I don't see what the issue is?