Most teams treat a schema migration as a deployment problem: write the migration, pick a quiet hour, run it, hope. Stripe's Atlas team treated it as an architecture problem instead. When Jacqueline Xu described how Stripe moved its Subscriptions data to a new model, the headline numbers were plain: hundreds of millions of objects, a table used across a large part of the codebase, thousands of lines of subscription logic spread over several services, and a company rule that infrastructure upgrades happen online, at full availability, with no maintenance window.
The article does the arithmetic that kills the naive plan. At one object per second, a sequential migration of 100 million records takes more than three years. You cannot stop the world for that, and you cannot run it as one script either. So the question becomes: how do you change the shape of data that every service reads and writes, while those services keep serving payments?
The problem: a data model that grew past its container
The Subscriptions model went through three stages, and the third one is the one many products end up in.
In the first stage, each Customer had at most one subscription, stored on the customer record itself. That is the obvious design for a product that sells one plan per account. In the second stage, customers could hold several subscriptions, so the single subscription field became a subscriptions array embedded in the Customer. Still reasonable: one read gives you the customer and everything they pay for.
The third stage is where the embedding started to cost real money. Updating any subscription meant rewriting the whole Customer record. Querying subscriptions meant scanning customer objects. Trials, coupons and invoices had accumulated on top of the model, and the product served customers such as DigitalOcean and Squarespace. The fix was clear: give subscriptions their own table. The hard part was getting there while thousands of lines of code assumed the array.
What broke, in other words, was not performance in one place. It was the coupling. Every service that touched billing had learned the shape of the Customer document. That is the real migration cost in any large system: not the bytes, but the number of code paths that encode the old shape.
The pattern: four phases, each one safe to stop
Stripe's answer is a four-phase dual-writing sequence. Dual write to the old and the new table. Move every read path to the new table. Move every write path to the new table only. Remove the old data.
Written that way it looks like textbook material, and it is. What makes the Stripe case worth studying is the discipline around each phase: every step was incremental, measured against production, and left the system in a consistent state if work stopped there. Changes reportedly never exceeded a few hundred lines at a time, against a body of subscription logic measured in thousands of lines.
Three mechanisms carry that discipline.
Mechanism 1: dual writes with three ways to reach the old data
The first phase creates the new table and starts writing new subscriptions to both stores. Stripe did not flip this on for everything at once. It gradually increased the percentage of objects being duplicated while watching operational metrics, because dual writes add write load to the production database.
Then came the existing data, and here the design is clever. Stripe used three paths to get old records into the new table. New writes went to both stores. Existing subscriptions were copied lazily whenever they were updated, which means the hottest objects migrated themselves through normal traffic. Everything left over was backfilled offline: database snapshots were loaded into a Hadoop cluster, a Scalding MapReduce job listed every subscription ID that still needed copying, and a multi-threaded migration copied them in parallel. The Scalding job was then run again to confirm nothing had been missed.
The trade-off is explicit. Offline discovery keeps expensive scans away from the production database, but it requires a data platform that can load snapshots and run batch jobs. Lazy copying spreads the work over normal traffic, but it means both stores coexist for longer, which extends the window where bugs can hide. And the percentage rollout costs calendar time in exchange for never surprising the database.
The cost most teams underestimate is the verification job itself. Rerunning the discovery query after the backfill is what turns "we think we copied everything" into a checked statement. If you cannot cheaply ask "which records are still missing?", you do not have a migration plan, you have a hope.
Mechanism 2: reads switched only after production agrees
Phase two moves read paths to the new table. Stripe did this with GitHub's Scientist library, which runs an experiment in production: the existing code path is the control, the new one is the candidate, both run, only the control's result is returned, and any mismatch raises an alert. Errors in the candidate path do not affect the rest of the application.
The Stripe article reports that Scientist alerted engineers as soon as even a single inconsistent record showed up. Only when results matched did reads switch to the new table.
GitHub's own account of Scientist, from Vicent Martí's "Move Fast and Fix Things", shows what this mechanism buys at scale. GitHub used it to replace the code behind the Merge button with an in-memory libgit2 implementation. Each run randomized the execution order of control and candidate, swallowed candidate exceptions, and logged mismatches and timings. Sampling started at 1% of requests and climbed to 100%. The experiment ran at full traffic for 24 hours with no mismatches or slow cases, after verifying tens of millions of merge commits, and the new implementation's 99th percentile ended up roughly equal to the old one's 95th. Along the way it found two merge bugs in Git itself and three major performance issues in the new code.
A minimal version of the pattern in Ruby looks like this:
class SubscriptionReader
include Scientist
def for_customer(customer_id)
science "subscriptions-read-path" do |e|
e.use { LegacyCustomerStore.subscriptions(customer_id) } # control, returned
e.try { SubscriptionsTable.where(customer_id: customer_id) } # candidate, compared
e.compare { |a, b| a.map(&:id).sort == b.map(&:id).sort }
end
end
end
The class names are illustrative; the use, try and compare blocks are Scientist's interface. The trade-off: every read during the experiment runs twice, so you pay extra latency and database load, which is why sampling matters. The cost you do not see in the code is the mismatch triage. GitHub's first mismatches were mostly old-path timeouts and timestamp differences, not real bugs, and someone had to filter that noise before the signal was usable. Budget for that work.
Mechanism 3: inverted writes and a tripwire on the old shape
Phase three moves write paths, and Stripe called it the hardest part: updates, prorations and renewals spread across thousands of lines and several services. The approach was to isolate each code path into the smallest possible unit and reverse the write order. Write to the new store first, then archive to the old store. At every incremental step, both stores stayed consistent, so any path could be paused or rolled back without a repair job. Scientist experiments checked these paths too.
Then Stripe added the mechanism I find most transferable. To catch every remaining use of the old field, it made the accessor fail loudly:
class Customer
def subscriptions
Opus::Error.hard("Accessing subscriptions array on customer")
end
end
This turns "did we miss a caller?" from a code-search question into a runtime fact. Any forgotten path raises immediately, with a stack trace pointing at the culprit. The trade-off is that a missed path now fails in front of someone, so this step only makes sense after read and write experiments have already shaken out the known callers. Used too early, it is an outage generator. Used at the right time, it is the cheapest completeness check you will ever write.
Phase four mirrors phase one. Once no code relied on the old field, writes to the old store stopped and the new table became the source of truth. The embedded array was emptied lazily whenever a subscription was loaded, and a final Scalding job found the remaining objects so the migration could delete them.
What this changes compared to the principles you already know
If you practice Domain-Driven Design, the Stripe story is a textbook aggregate split: Subscription stopped being part of the Customer aggregate and became its own. DDD tells you where the boundary should be. It says very little about how to move a live boundary that thousands of lines depend on. The four phases are the missing operational half.
If you know the expand-and-contract (parallel change) pattern, you will recognise the shape: expand by adding the new structure, migrate callers, contract by removing the old. Stripe adds two things on top. The first is evidence gates between phases: a read path does not switch on a code review, it switches when production stops reporting mismatches. The second is a deliberate asymmetry in how data moves, with live traffic migrating the hot records and batch jobs sweeping the cold ones.
It also differs from a strangler-fig migration, which usually puts a new service beside the old one and redirects traffic at the edge. Here there is no new service and no proxy. The change happens inside the same application, at the data access layer, which is why the accessor tripwire is possible at all.
What to steal for your architecture
Adopt this pattern when a data model change touches many code paths, when you cannot afford downtime, and when the cost of a silent inconsistency is high: billing, permissions, ledgers, anything a customer will notice being wrong. Do not adopt it for a small table with one owner and a tolerable maintenance window. Four phases, dual stores and production experiments are real overhead, and an offline migration plus a five-minute freeze is often the honest answer.
For a single team, the proposal is lightweight. Keep the four phases, but replace Hadoop with a plain SQL query that lists unmigrated IDs, run it before and after the backfill, and write a tiny comparison wrapper around the read path for a week. Add the raising accessor as the last step before deleting the old column. You get most of the safety for a few days of work.
For ten teams, the bottleneck becomes coordination, not code. Make the migration a tracked artefact: one owner, one dashboard showing percentage dual-written, mismatch rate per read path and number of remaining callers. Adopt a library such as Scientist (or its ports) as shared infrastructure so teams do not invent their own comparison harness. Require that each phase can be paused without a repair job, and write that into the migration's checklist.
For a hundred teams, invest in the platform Stripe implicitly relied on. You need snapshot loading into a batch system so discovery queries never hit production, a standard experiment framework with sampling and alerting, and a convention for hard-failing deprecated accessors in every language you run. At that size, a migration capability is a product. If every team builds its own, you will have a hundred slightly different ways to lose data.
The deeper lesson is that the safest migration is not the cleverest one. It is the one where every step produces evidence, and the next step waits for it.
