This website uses cookies

Read our Privacy policy and Terms of use for more information.

The problem: one cluster, every feature, one blast radius

GitHub.com started the way most Rails products start: one application, one MySQL database. Over the years the team added separate databases for some features, read replicas, and ProxySQL to cut the number of connections hitting the primaries. Those are the usual moves, and they bought time. They did not change the shape of the problem.

That shape had a name: mysql1. One cluster still held the data behind the core of the product, meaning users, repositories, issues and pull requests. Its growth forced the team to keep scaling up to bigger machines, and every incident on mysql1 touched every feature stored there. In 2019, GitHub started a program to improve its tooling and its ability to partition relational databases, described by Thomas Maurer in the post this issue is built on.

The scale numbers frame the result. In 2019, mysql1 served about 950,000 queries per second on average, 900,000 of them on replicas and 50,000 on the primary. By 2021, after the split, the same data across several clusters served about 1,200,000 queries per second, with 1,125,000 on replicas and 75,000 on primaries. Total traffic went up by roughly a quarter, and average load per host went down by about half. A companion post from the same period puts the end state at 30 databases behind the Rails monolith: 15 primaries and 15 replicas, each primary with its own schema.

The interesting part is not that GitHub moved tables to new clusters. Everyone knows how to move tables. The interesting part is the order of operations.

The architecture before, and what actually broke

Before the program, the database was the integration layer. Any Rails model could join any other model, and any transaction could touch any set of tables, because they all lived on the same server. That freedom is exactly what makes a monolith productive, and it is also what makes a physical split dangerous. The day you move gists to another cluster, every query that joins gists to users fails, and every transaction that writes both becomes silently non-atomic.

So the hard problem was never the copy. It was finding, and then removing, every hidden coupling between groups of tables while hundreds of engineers kept shipping features on top of them. GitHub's answer was to make the split virtual first: separate the tables in the application, prove the separation holds in production, and treat the physical move as the last and least risky step.

Mechanism 1: schema domains and a query linter that blocks new coupling

The first mechanism is a map. GitHub grouped tables that are joined together or written in the same transactions into schema domains. The gists domain holds the gist tables, and so on. The map lives in the Rails app as a YAML file, db/schema-domains.yml, and a linter checks two invariants on it: the file matches the real database schema, and every table belongs to exactly one domain. A new table cannot land without someone deciding which domain owns it.

The shape is simple. Something like this, illustrative rather than copied from GitHub:

# db/schema-domains.yml (illustrative shape)
gists:
  - gists
  - gist_comments
repositories:
  - repositories
  - repository_topics

The map becomes useful through the second piece: a query linter that flags any query referencing tables from more than one domain. It runs only in development, test and CI, so it costs nothing in production, and CI blocks new violations. Existing violations can be exempted with an annotation comment on the query, and GitHub upstreamed an ActiveRecord method to make that annotation easy to add.

This is the ratchet. The exemption list is the backlog of coupling you still have to pay down, and the linter guarantees it can only shrink. Teams then removed exemptions in two ways. The first was to replace cross-domain joins with separate queries, for example preload instead of includes, and a disable_joins option on has_many :through associations that GitHub contributed to Rails 7:

class Dog < AnimalsRecord
  has_many :humans
  has_many :treats, through: :humans, disable_joins: true
end

With that flag, dog.treats stops being one SQL join and becomes two queries: one that selects the human IDs for the dog, and one that selects treats where human_id is in that list. The second way was to do the join in application code. GitHub reports that this sometimes improved performance, because MySQL's planner occasionally picks a poor plan for a join, while an application-side join has a more predictable cost. Risky rewrites shipped behind Scientist experiments, which run the old and new code paths side by side on a subset of requests and compare the results.

The trade-off is real. You trade one round trip for two or more, and you move work from the database into Ruby. The Rails post is explicit that ordering and limits across the split queries happen in memory, so the feature should not be used on result sets in the hundreds of thousands of rows. You also need the indexes that make an IN (...) lookup cheap. What you buy is a codebase where the domain boundary is a fact the build checks, not a diagram on a wiki.

Mechanism 2: a transaction linter that runs in production, sampled

Queries are the easy coupling to find, because they show up in tests. Transactions are harder. MySQL only guarantees atomicity inside one database, so any transaction that writes to two domains today will lose that guarantee the moment those domains live on different clusters, and nothing will fail loudly.

GitHub built a transaction linter for that case, and made a deliberate choice about where it runs: in production, with heavy sampling to keep the overhead low. Development and CI do not exercise the real mix of code paths that write data; production does. The output tells the team which code paths open cross-domain transactions, and that drives one of two fixes: change the code path so it no longer needs atomicity across domains, or change the data model.

The data model fix is the instructive one. A polymorphic table such as reactions, shared by several features, sits naturally in no single domain. Rather than keep a cross-domain transaction alive, GitHub sometimes copied the data into domain-specific tables so each feature could keep its writes atomic inside its own domain. That is duplication on purpose, chosen over a distributed transaction.

The cost here is subtle. A sampled linter gives you evidence, not proof: a rare code path may never get sampled. And each fix changes a consistency guarantee that some feature relied on, so every finding is a product conversation as much as a code change. The benefit is that you find those conversations before the move, not during an incident after it.

Mechanism 3: a write cutover measured in milliseconds, with Vitess as the other path

Only after the virtual partition holds does data move. GitHub used two approaches. The first was Vitess for vertical sharding, moving sets of tables without downtime: VTGate is the endpoint the application talks to and it speaks the MySQL protocol, VTTablet manages the MySQL instances, and VReplication copies the data between clusters.

The second was a write-cutover process the team built as a backup path, because Vitess adoption was still early in 2020 and they did not want a single tool on the critical path. The destination cluster joins the replication tree of the source, ProxySQL routes the traffic, and a script performs the switch: set the source primary read-only, record the last executed GTID, wait for the destination to catch up to it, stop replication, point ProxySQL at the new primary, and turn read-only off. Writes fail during that window. For the busiest tables it lasted a few tens of milliseconds, and GitHub ran it at the lowest-traffic time of day, which produced a handful of user-facing write errors.

That process is what split mysql1, moving 130 of its busiest tables in one operation. Vitess was not used everywhere because of deployment topology and read-your-writes requirements, though GitHub said it expected to use it for most future migrations.

The trade-off is between a tool that hides the cutover from the application and a script that you fully understand but that accepts a short write outage. Keeping both costs maintenance. It also means a stuck migration never blocks the program.

What this changes compared to DDD and microservice playbooks

Domain-Driven Design tells you to find bounded contexts and give each one its own model and storage. Most migration playbooks then jump to extraction: pull a context into a service, give it a database, and fix the breakage. GitHub's sequence inverts that. The bounded context was defined at the table level, enforced inside the monolith by linters, and validated in production before any process boundary or network hop existed. The monolith stayed a monolith. The databases became plural.

That matters because most of the risk in a split lives in the data, not in the code. Joins and transactions are where coupling hides, and they are invisible in a service dependency graph. A schema-domain map is a context map that the CI pipeline can verify. Compared to an event-driven extraction, there are no new events, no outbox, no eventual consistency to explain to product managers, except in the specific places where the transaction linter forced the choice.

It also changes what "done" means. In this model, a domain is ready to move when its exemption count is zero and the transaction linter is quiet for it. That is a measurable gate, which is rare in architecture work.

What to steal for your architecture

Adopt this when one database is your scaling limit and your incident blast radius, and when the application is still a monolith you intend to keep. Do not adopt it if your problem is a single hot table that needs horizontal sharding; GitHub treats that as a separate track with its own tooling. Do not adopt it either if your data volume fits comfortably on one primary for the next two years: the linters cost engineering time and the split adds operational surface.

For a single team, steal the map and the query linter. Write the domain file today, even with three domains, and fail CI on new cross-domain joins. It costs a day and stops the coupling from growing while you decide whether you will ever need a split.

For ten teams, add ownership and the ratchet. Each domain gets an owning team, the exemption list becomes a tracked metric per domain, and disable_joins or application joins become the default for anything crossing a boundary. Ship the rewrites behind a Scientist-style comparison so correctness is measured, not assumed.

For a hundred teams, add the sampled transaction linter in production and build the move tooling before you need it: a replication-based cutover you can rehearse, plus a proxy layer that lets you repoint a domain without a deploy. Treat the physical move as an operation run at low traffic with a known error budget, not as a project.

The lesson I take from GitHub's program is about sequencing. Splitting a database is mostly a code problem that looks like a data problem. Fix the code first, make the boundary a build failure, and the move itself shrinks to a few tens of milliseconds.

Sources