This website uses cookies

Read our Privacy policy and Terms of use for more information.

The problem: a block model that rewrites itself

Everything in Notion is a block: a paragraph, a to-do, a page, a database row. Each block is a row in Postgres, with a parent pointer and its own permissions. According to Notion's data platform team, the company's data grew tenfold in three years and doubles every 6 to 12 months. At the start of 2021 Postgres held more than 20 billion block rows. By mid-2024 it held more than two hundred billion blocks, hundreds of terabytes even compressed.

That volume was already split. In 2021 Notion sharded Postgres into 32 physical instances carrying 15 logical shards each; in 2023 it moved to 96 instances with 5 logical shards each. Both layouts keep the same 480 logical shards, which matters for everything that follows: the shard is the stable unit, the host is not.

The number that shaped the architecture is not the size, though. It is the write mix. Notion reports that 90% of its upserts are updates. Users edit blocks far more often than they create them. Analytics stacks are usually designed for the opposite profile: append events, query them later.

The architecture before, and what broke

In 2021 the pipeline was simple to describe. Fivetran read the Postgres write-ahead log and loaded Snowflake through 480 hourly connectors, one per shard, into 480 raw tables, which were then merged into one large table for analytics and machine learning.

It broke on three axes. Operability first: 480 connectors have to be monitored and re-synced every time a shard moves, Postgres is upgraded or maintenance runs, and re-sharding in 2023 multiplied those events. Freshness and cost second: a warehouse optimized for inserts pays heavily to apply a stream that is 90% updates, so ingestion became slower and more expensive as the data grew. Use cases third: deciding who can see a block means walking the tree up to its root, across hundreds of billions of rows. Notion's team writes that this kind of computation would simply time out in Snowflake.

The useful lesson is that none of these is a Snowflake bug. A warehouse is a fine serving layer for clean, modeled data. It is a poor first landing zone for a mutable, sharded, tree-shaped operational dataset.

Mechanism 1: Change data capture with one topic per table, not per shard

The new ingestion path starts with Debezium. Notion runs one Debezium connector per Postgres host, deployed on Kubernetes (EKS), reading the logical replication stream and publishing row changes to Kafka. As of May 2024 the setup handles tens of MB per second of row changes, and the team reports upgrading EKS and Kafka only a few times in two years.

The decision worth copying is the topic layout. Every connector, for every one of the 480 shards, writes to the same Kafka topic for a given table. There is one block topic, not 480 of them. The shard becomes a field in the event, not a piece of infrastructure.

What this buys: re-sharding no longer changes the shape of the pipeline. When 32 hosts became 96, connectors were redeployed per host, but consumers kept reading the same topics. The number of things to monitor scales with tables, not with tables multiplied by shards.

What it costs: a single topic per table concentrates the hottest table (block) into one stream, so partitioning inside that topic and consumer throughput must be sized for the largest table, and a misbehaving shard is harder to isolate than when it had its own connector and destination. You also take on Kafka and Debezium operations yourself, where Fivetran was a managed service.

Mechanism 2: An upsert-native table format, partitioned like the source

Kafka is a log; analytics needs current state. Notion uses Apache Hudi's Deltastreamer, a Spark job, to replay each topic into a Hudi table on S3. Most tables lag by a few minutes; the block table lags by up to two hours.

Hudi won the 2022 evaluation against Iceberg and Delta Lake for two stated reasons: it performed well on an update-heavy workload, and it had native, open-source Debezium integration. Iceberg had no ready Debezium consumer at the time, and Delta's Debezium support was not open source, so either would have required writing a custom consumer.

The configuration is where the design lives. Notion published the key settings:

# Settings quoted in Notion's post (table type: COPY_ON_WRITE, operation: UPSERT)
hoodie.datasource.write.partitionpath.field: db_schema_source_partition  # shard0001 ... shard0480
source-ordering-field: event_lsn   # last update position
hoodie.index.type: BLOOM

Each setting answers a specific property of the data. Partitioning by the Postgres shard (shard0001 to shard0480) means a batch of updates from one shard touches files in one partition, instead of rewriting files across the whole table. Ordering by event_lsn clusters recently updated blocks together; since recent blocks are the ones most likely to change again, files that hold only old blocks are never rewritten and can be pruned. The Bloom filter index lets the writer skip files that cannot contain a given key.

The trade-off is in the table type. Copy-on-write rewrites a whole file whenever one row in it changes. Reads are cheap and simple, which suits Spark jobs downstream, but write amplification is real, and it is only tolerable here because partitioning and ordering keep each batch of changes inside few files. Merge-on-read would write faster and read slower. If your updates are spread uniformly across all keys, the same settings will not save you.

Mechanism 3: Raw first, then Spark for what the warehouse cannot do

Notion lands raw Postgres data in S3 without transforming it on the way in. The raw Hudi tables are the single source of truth, which makes debugging a matter of comparing the lake with the source rather than reverse-engineering a pipeline. Only cleaned, business-critical data moves on to Snowflake and to product-facing stores.

Transformation runs in Spark: PySpark for most jobs, Scala for tree traversal and denormalization. Spark was chosen because it accepts user-defined functions and arbitrary logic, exposes control over partitioning, skew and resources, and is open source, so it could be trialed quickly. The permission computation that timed out in the warehouse is now a Spark job. Small shards are loaded fully into a task's memory; large shards that exceed memory fall back to disk reshuffling; the 480 shards are processed in parallel with multithreading. The shard key that organized ingestion also organizes compute.

Bootstrapping is the part most teams forget to design. Notion's sequence has three steps: start the Debezium connector first, take an RDS export-to-S3 snapshot at a timestamp t and convert it to Hudi with Spark, then run Deltastreamer from Kafka starting at t to apply the changes that happened during the snapshot. The three steps usually finish within 24 hours, which makes re-bootstrapping a table for a new use case, a Postgres upgrade or a re-shard a routine operation. A full snapshot on its own would take more than 10 hours and cost twice as much as incremental ingestion, so it is reserved for this bootstrap.

The cost of this mechanism is ownership. Notion now operates Debezium, Kafka, Hudi and Spark, where it previously paid for a connector service. The team built the lake between spring and fall 2022.

Results

Notion reports a net saving of more than a million dollars for 2022 from moving several large datasets, some of them tens of terabytes, with proportionally higher savings in 2023 and 2024. End-to-end ingestion from Postgres to S3 and Snowflake dropped from more than a day to a few minutes for small tables and up to a couple of hours for large ones. Re-syncs complete within 24 hours without overloading the live databases. The team credits the lake with enabling the rollout of Notion AI features in 2023 and 2024, because denormalized, permission-aware data became cheap to compute.

What changes compared with the principles you already apply

If you practice event-driven architecture, the CDC stream looks familiar, but it is not a domain event stream. Debezium emits row changes, not PageShared or BlockMoved. Notion uses them as replication, not as an integration contract between services, and that is the right scope: row-level CDC couples consumers to the table schema, which is acceptable for a lake that owns its own transformations and dangerous for business services.

If you apply CQRS, the lake is a read model at a different scale. The write side stays in sharded Postgres; the Hudi tables are a materialized replica; the Spark outputs are purpose-built read models for search, AI and analytics. The difference with textbook CQRS is that the projection is rebuilt from a snapshot plus a log, not from an event store, so replay is bounded to 24 hours rather than to the whole history.

If you shard by tenant or workspace, Notion shows the second dividend of a good shard key: once it is stable across topology changes (480 logical shards, whatever the host count), it can drive Kafka event fields, lake partitions and Spark parallelism without any extra mapping.

What to steal for your architecture

Adopt this when three conditions hold: your operational data is mostly updated rather than appended, analytics or AI needs joins or recursive logic your warehouse handles badly, and loading costs grow faster than your data. Do not adopt it when your data is append-only events (a warehouse or a plain object store with Parquet does the job), when you need second-level freshness (Notion explicitly scoped the lake to offline workloads tolerating minutes to hours), or when no one on the team will own Kafka and Spark in production.

For one team, do not build the stack. Keep the managed connector into your warehouse, but write down the shard or tenant key and make sure every analytics table carries it. Script the bootstrap procedure (snapshot at t, then replay from t) even if a vendor runs it, because the day you re-shard you will need it.

For ten teams, introduce CDC with one topic per table, keyed by the source shard, and land raw changes in an object store with an upsert-capable table format. Pick the format by testing your real update ratio and your existing CDC tooling, as Notion did, rather than by popularity. Keep the warehouse as the serving layer for curated tables only.

For a hundred teams, treat the lake as a platform product: a catalog of raw tables that mirror sources one to one, a documented bootstrap and re-sync runbook with a time budget (Notion's is 24 hours), and a rule that business events between services go through domain topics, never through CDC topics. That last rule keeps the replication pipeline from becoming an accidental integration bus.

Sources