The problem and the scale
Discord stores every message ever sent in every channel, and the read pattern is brutal: random, close to 50/50 between reads and writes, and uneven, because large servers send orders of magnitude more messages than small ones. In January 2017 the team wrote that they were "well past 120 million" messages per day and had just moved from MongoDB to Cassandra after crossing 100 million stored messages, the point where data and indexes no longer fit in RAM. At that time the messages cluster ran on 12 Cassandra nodes.
By early 2022 the cluster called cassandra-messages had grown to 177 nodes holding trillions of messages. In March 2023 Bo Ingram, Senior Staff Software Engineer at Discord, described how that cluster became the team's main on-call burden and how they replaced it. The headline numbers are easy to quote: 72 ScyllaDB nodes instead of 177, p99 latency for historical message reads down from 40-125 ms to about 15 ms, p99 inserts down from 5-70 ms to a steady 5 ms. The interesting part is the order in which they did things.
The architecture before, and what broke
The data model was sound and is still worth copying. Every ID is a Snowflake, so IDs sort chronologically. The primary key is ((channel_id, bucket), message_id): the partition is a channel plus a static time window, and messages are sorted inside it by ID. The 2017 post explains the bucket choice: about 10 days of messages per bucket, so partitions stay under 100 MB even though Cassandra tolerates up to 2 GB. Since the bucket is derivable from a message ID or a timestamp, a "load history" query starts at the current bucket and walks backward until it has enough messages.
What the model cannot fix is skew. A partition is replicated on three nodes, and reads and writes use quorum consistency. When a big server posts an announcement with an @everyone mention, thousands of clients open the same channel and hit the same (channel_id, bucket) partition at the same moment. The node holding it slows down, and because quorum queries wait on that node, every other query that touches it slows down too. A hot partition in one community becomes latency for unrelated communities.
Cassandra's internals made this worse. Writes are cheap: they go to a commit log and a memtable. Reads may have to merge the memtable with several SSTables, so they cost more, and the team fell behind on compaction, which made each read touch even more SSTables. Maintenance became a manual "gossip dance": pull a node out of rotation, let it compact, put it back, let it catch up on hints. On top of that, Cassandra runs on the JVM, and garbage collection pauses produced latency spikes that sometimes required a manual reboot. The 2017 post already showed the tombstone trap: a channel mass-deleted from millions of messages down to one took 20 seconds to load with constant 10-second GC pauses while Cassandra scanned the tombstones.
The obvious answer was to change the database. ScyllaDB is Cassandra-compatible, written in C++, has no garbage collector, and isolates workloads with a shard-per-core design. By 2020 every other Discord database already ran on it. But migrating nearly 200 nodes of trillions of messages is the kind of project where a hot partition during the cutover becomes an incident. So they changed the traffic first.
Mechanism 1: partition by channel and time bucket
This one predates the migration, but everything else leans on it. Choosing (channel_id, bucket) as the partition key turns "a channel's history" into a bounded set of partitions, each one capped by time rather than by count. The query path never scans the cluster: it computes a bucket from a Snowflake, reads one partition, and moves to the previous bucket if it needs more.
The trade-off is that the bucket size is a bet on traffic. Ten days keeps a busy channel's partition small, but a quiet channel spreads a handful of messages across many empty buckets, and a history query has to step through them. Discord's 2017 fix was to track empty buckets so queries skip them. The cost is also on the write side of the model: Cassandra writes are upserts with last-write-wins per column, and writing a null column creates a tombstone. With 16 columns and about 4 set on an average message, Discord was writing around 12 useless tombstones per message until they changed the code to write only non-null values.
What it costs you: you must know your access pattern before you pick the key, and changing the partition key later means rewriting the table.
Mechanism 2: data services with request coalescing and consistent hash routing
This is the part that deserves the attention. Discord built "data services": intermediary services between the API monolith and the database clusters, written in Rust on Tokio. Each service exposes roughly one gRPC endpoint per database query and contains no business logic. Their job is narrow: limit the concurrency that reaches the database.
They do it with two techniques that only work together. Request coalescing means that if several requests for the same row arrive while a query for that row is in flight, only the first one spawns a worker task that queries the database. The others subscribe to that task and receive the same result. A thousand clients loading the same announcement produce one database read instead of a thousand.
Coalescing only merges requests that land on the same process, so Discord routes by key. Each request carries a routing key, the channel ID for messages, and consistent hashing sends all requests for a channel to the same data service instance. The thundering herd on one channel now converges on one instance and collapses into one query.
The shape of the idea fits in a few lines. This is not Discord's code (theirs is Rust), but Go's singleflight package expresses the same contract:
var group singleflight.Group
func (s *MessagesService) GetMessages(ctx context.Context, channelID int64, before int64) ([]Message, error) {
key := fmt.Sprintf("%d:%d", channelID, before)
v, err, _ := group.Do(key, func() (any, error) {
// Only one goroutine per key reaches the database at a time.
return s.db.FetchMessages(ctx, channelID, before, 50)
})
if err != nil {
return nil, err
}
return v.([]Message), nil
}
The router in front must hash channelID onto the instance ring, otherwise the group above only sees a fraction of the duplicates.
The trade-offs are real. Coalescing returns the same answer to everyone who asked during the in-flight window, so a request that arrives a few milliseconds after a write can get the pre-write result; for chat history that is acceptable, for a balance check it is not. A slow query now delays every subscriber, so timeouts and cancellation need care: one caller cancelling must not cancel the shared query. Consistent hashing concentrates a hot channel on one instance, which protects the database but makes that instance the new hot spot, so the service itself has to be cheap per request. Discord also notes the limit honestly: the data services reduced the spikes, but hot partitions and latency still happened on Cassandra, just less often. The layer bought time and safety; it did not fix the storage engine.
What it costs you: one more network hop, one more deployable per database, and a discipline of keeping business logic out of it.
Mechanism 3: a migrator built on the same library, checkpointed and shadow-validated
The first plan was conventional: dual-write new messages to ScyllaDB from a cutover time, then backfill history with ScyllaDB's Spark migrator. Estimate: three months.
Instead, the team extended the data service library into a migrator in an afternoon. It reads token ranges from Cassandra, checkpoints its progress in a local SQLite database, and writes into ScyllaDB. The new estimate was nine days, short enough that they dropped the dual-write cutover and migrated everything at once. At peak it moved 3.2 million messages per second.
It stalled at 99.9999% complete. The last token ranges held huge numbers of uncompacted tombstones from Cassandra, the same failure mode the 2017 post described. Compacting those ranges let the migration finish. Before switching, they sent a small percentage of reads to both databases and compared the results, then made ScyllaDB primary in May 2022.
The storage side mattered too. Discord ran the new cluster on what they call "super-disk": local SSDs mirrored with RAID to a persistent disk, to get local-disk read latency with the durability of network storage. Each ScyllaDB node holds about 9 TB, against an average of 4 TB per Cassandra node, which is how 177 nodes became 72.
The trade-off of migrating in one pass is that you lose the gradual rollback a dual-write gives you. Discord compensated with checkpoints (a crash resumes a range, it does not restart the job) and shadow reads (correctness is measured on live traffic before the switch). What it costs you: a migrator you own and must test, and a validation harness that compares results, not just row counts.
What this changes compared to the principles you already use
Most teams know DDD for boundaries, CQRS for separating read and write models, and caching for hot reads. Discord's design is none of those, and the difference is the useful lesson.
A cache would have helped the @everyone case, but a cache is a second copy with its own invalidation problem, and Discord's 2017 requirements explicitly excluded relying on Redis or Memcached for performance. Coalescing keeps one source of truth and only removes duplicate concurrent work. It sits closer to a bulkhead than to a cache: it caps the concurrency any single key can impose on the database.
The data services are also a counterpoint to the usual microservice split. They are not domain services. They have no business logic and map one-to-one with queries. That makes them closer to a database driver deployed as a fleet than to a bounded context, and it lets the API stay a monolith while the storage tier gets its own scaling and protection. The domain boundary stays where it was; a technical boundary is added below it.
Finally, the migration order inverts the instinct. The instinct says the database is slow, so replace the database. Discord first put a layer in front of it that made traffic predictable, then reused that layer's code to build the migrator. The same abstraction served steady state, migration and validation.
What to steal for your architecture
Adopt this when you have read skew on a small number of keys, when identical reads arrive in bursts (announcements, launches, live events, a popular product page), and when stale-by-milliseconds results are acceptable. Do not adopt it for reads that must reflect a write the same caller just made, for keys that are almost never read concurrently (coalescing adds overhead with no gain), or as a substitute for fixing a partition key that is simply wrong.
For one team, you do not need a new service. Put request coalescing inside your API process on the two or three hottest read paths (singleflight in Go, a shared in-flight promise map in Node, asyncio futures keyed by ID in Python), and measure database QPS during your next traffic spike. Fix nulls and deletes in your write path before they become tombstones or bloat in whatever engine you use.
For ten teams, extract a data access service for the one storage cluster that pages you, with an API that mirrors queries and has no business logic, and route to it by partition key with consistent hashing at your load balancer or service mesh. Make the routing key part of the request contract so every caller provides it.
For a hundred teams, treat this as a platform: a shared library for coalescing, routing and checkpointed range scans, owned by the storage team, used both for serving and for migrations. Require shadow-read comparison as the exit criterion of any database migration, and size node count by disk per node and tail latency, not by the cluster you happen to have.
