On June 30, 2021, at 11:45am PDT, a network link connecting one availability zone (AZ) to the other AZs hosting Slack servers in the U.S. East Coast region started failing intermittently. The cloud provider pulled the link at 12:33pm, put it back after automated checks, and watched it fail the same way again at 5:22pm. It was removed for good at 5:31pm. Slack users saw slowness and errors in both windows.

That region matters because it is where most of Slack runs. Slack has a global edge network, but the core compute sits in one region, spread across several AZs. Slack sells a 99.99% availability SLA, which Cooper Bethea, Senior Staff Engineer on Slack's cloud team, translates into less than one hour of total unavailability per year. One bad afternoon can spend most of that budget.

The post-incident question was blunt: why does losing one AZ out of several produce user-visible errors at all? The whole point of running in multiple zones is that one of them can go away.

Before the change, Slack ran a classic multi-AZ deployment. Every service had instances in every zone, and any instance could call any upstream instance in any zone. That layout is resilient to a clean failure: an AZ goes dark, health checks fail, load balancers stop sending traffic there, and the rest carries the load.

June 30 was not a clean failure. Bethea calls it a gray failure: different components held different views of which backends were healthy, depending on which network path their traffic took. A frontend in zone A could reach a backend in zone B fine, while a frontend in zone C saw timeouts to the same backend. Health checks from one vantage point said "up", user requests from another said "down".

Two properties of Slack's stack made this worse. First, fan-out. A single user request can turn into hundreds of internal RPCs, so a small per-call failure rate compounds into a large per-request failure rate. Second, strong consistency in the main datastore. Slack uses Vitess, and each write needs the single primary of its shard to be reachable. If the primary sits behind the flaky link, writes to that shard fail until the link recovers or a replica is promoted.

Because services were spread across every zone, a failure inside one zone surfaced as errors in frontends in all zones. The blast radius of an AZ fault was the whole region.

The core decision is a routing rule, not a new component. A siloed service only receives traffic from its own AZ, and only sends traffic to upstream servers in its own AZ. In Bethea's words, each service effectively becomes N virtual services, one per zone.

The effect is that an AZ turns into a vertical slice of the whole backend. Traffic that enters zone B stays in zone B from the first hop to the last. If you stop sending user requests into zone B, every internal service in zone B quiesces on its own, because nothing else in the region is calling it. You do not need to tell each of the many RPC clients in the stack that zone B is unhealthy. You only need to stop the traffic at the top.

This is what the AWS Well-Architected guidance calls a single-AZ cell: a fully isolated deployment scoped to one zone, with a routing layer in front that decides which cell gets which request.

The trade-off is capacity. In a shared multi-AZ pool, a zone failure removes a fraction of capacity and the survivors absorb it through normal load balancing. In a siloed layout, each zone must be sized so that the remaining zones can take its full share when it is drained, and that headroom sits idle most of the time. The second cost is that some things cannot be siloed. A strongly consistent datastore with one primary per shard still lives in exactly one zone at a time, so cross-zone calls to it remain. Slack's post says plainly that services which cannot be siloed are a topic for a later installment, and does not claim to have removed every cross-zone dependency.

What it costs you to build: every service owner has to conform. Service discovery, client configuration and deployment topology all need a zone dimension. Slack describes the migration of its critical user-facing services as taking about a year and a half.

Once the backend is siloed, failover reduces to one question: how much user traffic goes into each zone? Slack answers it at the edge load balancers.

Those load balancers run Envoy. That itself was a multi-year migration away from HAProxy. A separate Slack post on moving millions of concurrent websockets to Envoy explains why: HAProxy needed a reload every time backend endpoints changed, each reload spawned new processes, and old processes had to live for hours so long-lived websockets could drain. Envoy takes endpoints dynamically and supports hot restarts. That migration moved traffic with weighted DNS in steps of 10%, 25%, 50%, 75% and 100% per region, ran both stacks in parallel (roughly doubling the AWS bill during the move) and finished in about six months with zero customer impact.

With Envoy at the edge, the drain uses two standard features: weighted clusters and dynamic weight assignment through RTDS, Envoy's runtime discovery service. Each AZ is a target cluster. Rotor, Slack's in-house xDS control plane, tells the edge Envoys to set the weight of the drained zone's cluster to zero. Envoy finishes the requests already in flight and sends new ones to the other zones.

Here is an illustrative Envoy route, not Slack's configuration, showing the shape of the idea. The weights are read from runtime keys that a control plane can overwrite through RTDS without touching the route:

Draining the first zone means the control plane pushes runtime values such as edge.az_weights.backend_use1_az1 = 0 and redistributes the rest. Because weights are integers over a total, Slack can move traffic in steps as small as 1%.

The trade-off: the edge becomes the single place where an AZ is declared healthy or not. That is a feature, since there is exactly one lever, but it also means the quality of the decision depends on the people pulling it and on the dashboards they look at. The cost is the Envoy migration itself, plus a control plane you now operate as a tier-zero dependency.

Slack wrote down four requirements before choosing a design. Remove as much traffic as possible from an AZ within five minutes. Cause no user-visible errors, so a drain can be used as a generic first response before anyone knows the root cause. Drain and undrain incrementally, down to 1% of traffic. And the mechanism must not depend on any resource inside the zone being drained, so it still works when that zone is completely offline.

The fourth requirement is what shaped the topology. The edge load balancers sit in different regions, and the control plane is replicated regionally so it survives the loss of any single AZ. Slack explicitly rejected a design where operators would SSH into servers in the bad zone to force their health checks to fail, because that requires the bad zone to be reachable.

The second requirement changed how the button is used. A drain that causes no errors is cheap to try. Operators can pull a zone at the first sign of a gray failure, watch whether error rates drop, and put it back if they guessed wrong. Slack's published graph of queries per second by AZ shows traffic leaving one zone for the other two in three steps and then returning to an even split, with sharp knees that reflect propagation in seconds.

Slack also made a deliberate choice about who presses the button. They considered automatic detection and remediation of gray failures and chose human judgment instead. Gray failures are exactly the cases where automated health signals disagree, so an automatic drain risks acting on the wrong signal, or flapping.

The cost is operational: someone on call must recognize the pattern, decide, and act within the five-minute target. The benefit is that the action is reversible, graded and safe, so the decision does not need certainty.

The rejected alternative is the one most teams would reach for first: per-client failure handling. Every RPC client learns to detect a bad zone and route around it. Slack ruled it out because of heterogeneity. Its services are written in Hack, Go, Java and C++, and service discovery runs through Envoy xDS, Consul and DNS. DNS has no notion of a zone or of a partial drain. Patching open-source datastores like Vitess would mean maintaining a fork or upstreaming changes. Implementing consistent failure detection N times, in N runtimes, against M discovery systems, is the kind of work that never finishes.

Siloing inverts the problem. Instead of making every client smart, it makes every client local, and puts the one smart decision at the edge.

Domain-driven design gives you boundaries by business capability. Event-driven architecture decouples producers from consumers in time. Neither says anything about failure domains. You can have perfectly bounded contexts that all call each other across three zones, and a single flaky link will still take down all of them.

Cell-based architecture adds an orthogonal boundary: a deployment boundary that matches the infrastructure failure domain. Slack's version is the smallest useful step in that direction. It does not partition customers into cells, it does not shard by tenant, and it accepts that some stateful dependencies still cross zones. It partitions by AZ, because that is the failure it actually had.

It also departs from the AWS guidance on one point. The Well-Architected paper states that cells were not designed as failover domains, and recommends multi-AZ cells unless your service is scoped to a single zone. Slack builds single-AZ cells precisely to fail over between them. The AWS document does allow this path, noting that evacuating a cell during a gray failure can beat living with it, at the price of higher infrastructure cost and the work of handling replication yourself. Slack paid that price on purpose.

Adopt it when three things are true: most of your traffic runs in one region across several zones, you have an availability target where a few hours of degradation per year is unacceptable, and your stack is heterogeneous enough that per-client resilience logic would have to be written many times. Add a fourth signal if you have already lived through a gray failure where health checks and users disagreed.

Do not adopt it when your service is small enough that a single zone's worth of headroom is a large share of your bill, when your critical path is dominated by a single-primary datastore you cannot replicate across zones (siloing the stateless tiers will not save writes), or when you lack the on-call maturity to make a drain decision in minutes.

For one team, do not build cells. Put a proxy you control at the edge, give it per-zone upstream pools with weights, and write a runbook that sets one pool to zero. Rehearse it once a quarter during business hours. That gives you the core of Slack's lever without the service-by-service migration.

For ten teams, make zone locality the default in your service discovery and client libraries, so a new service is siloed unless its owner opts out and justifies it. Keep an inventory of the services that cannot be siloed, with an owner and a plan for each. Put the weights behind a small control plane with an audit log, and drill a full-zone drain every month.

For a hundred teams, treat the drain as a product. Track conformance per service as a published metric, gate new tier-one services on it, and run drains in production on a schedule so the capacity headroom is tested, not assumed. At that size, the next question is the one Slack left open: whether to go from zone cells to customer cells, so a bad deploy or a poison tenant can be contained the same way a bad zone is.