The problem and the scale
Between 2019 and 2023 DoorDash moved from a monolith to cell-based microservices. By the time its service mesh was fully rolled out, it carried more than 80 million requests per second at peak across more than 1,000 migrated microservices, and the main stateless Kubernetes cluster ran around 2,000 nodes.
That architecture was built to survive the loss of one or more availability zones, so every cluster spans several AZs. The price of that resilience shows up in a line of the cloud bill that most teams never look at: cross-AZ data transfer, which is charged on both the sending and the receiving side. As the number of services and back-ends grew, so did that line. The question DoorDash had to answer was not "how do we stop crossing zones", because a multi-AZ design has to cross zones to stay balanced and available. The question was "which crossings are buying us resilience, and which ones are just accidents of the call path".
The architecture before, and what it did wrong by default
DoorDash runs cells. Each service has one Kubernetes deployment per cell, traffic between cells is not allowed, and singleton services live in a global cell reached through internal Envoy routers. A cell contains several Kubernetes clusters, each microservice runs in exactly one cluster per cell, and each cluster spans several AZs. With AWS-CNI the network is flat, so pods in different clusters of the same cell talk to each other directly.
Service discovery goes through DDSD, DoorDash's own data-center service discovery. A name like payment-service.service.prod.ddsd resolves to every pod IP behind the service, a bit like a Kubernetes headless service that works across clusters. Envoy sidecars intercept HTTP/1, HTTP/2 and gRPC traffic and are driven by a custom xDS control plane.
The default behavior of that setup is round-robin across all pods. With pods spread over three zones, a caller sends roughly two requests out of three to another zone, every time, for no reason other than that nobody told the load balancer where anyone lives. Nothing is broken. The system is correct and evenly loaded. It is simply paying for distance it does not need.
Mechanism 1: give the mesh the zone of every endpoint, then let Envoy decide
The first change was in the control plane. DDSD clusters in Envoy moved from STRICT_DNS to EDS, the endpoint discovery service. With DNS, Envoy only gets a list of IPs. With EDS, the control plane can attach locality to each endpoint, and that is the whole trick:
# shape of an EDS response (illustrative addresses)
cluster_name: payment-service.service.prod.ddsd
endpoints:
- locality: { zone: us-west-2a }
lb_endpoints:
- endpoint: { address: { socket_address: { address: 1.1.1.1, port_value: 80 } } }
- locality: { zone: us-west-2b }
lb_endpoints:
- endpoint: { address: { socket_address: { address: 2.2.2.2, port_value: 80 } } }
- locality: { zone: us-west-2c }
lb_endpoints:
- endpoint: { address: { socket_address: { address: 3.3.3.3, port_value: 80 } } }
With locality known, Envoy's zone-aware routing takes over. It is not "always stay local". Envoy compares the share of healthy hosts in the local zone for the calling cluster and for the called cluster. If the caller's local share is larger than the callee's, sending everything locally would overload the callee's pods in that zone, so Envoy computes the fraction that can go direct and sends the rest to other zones, picked by spare capacity. If the caller's local share is smaller, everything stays local. The feature also switches itself off when either cluster is in panic mode or when the two clusters do not span the same number of zones.
The trade-off is that routing becomes a function of where pods happen to land. The benefit DoorDash lists is that weights adjust on their own, with no manual tuning, and that callers using nearer callees see lower latency and a smaller blast radius when a zone degrades. The cost is a dependency: zone-aware routing is only as good as the endpoint metadata your control plane can deliver, which is why it only applied to DDSD names and not to anything resolved by plain DNS.
Mechanism 2: spread pods evenly, but refuse to pay for perfect symmetry
Zone-aware routing works best when callers and callees have the same distribution across zones. If a service has six pods in one zone and two in another, Envoy has to spill traffic across zones to keep the two pods from drowning. So DoorDash spread pods with a topology constraint:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
The interesting choice is the last line. DoNotSchedule would enforce the balance strictly and leave pods pending when a zone is short on capacity, which means idle nodes elsewhere and worse bin-packing. ScheduleAnyway treats the spread as a preference. DoorDash reports that with this policy about 10% of traffic still crosses zones in production. That 10% is a deliberate price: a small, permanent data-transfer cost in exchange for not reserving compute in every zone just to keep the counts equal.
This is the kind of decision that rarely gets written down. It is a cost trade between two cloud line items, compute and network, made in a scheduler setting that most teams copy from a blog without knowing which side it favors.
Mechanism 3: measure bytes by path, then delete hops instead of tuning them
The rollout worked. Envoy metrics and application logs confirmed that zone-aware routing behaved as designed, and cross-AZ charges went down. But they went down less than the team had projected, and the mesh metrics could not say why.
So DoorDash stacked three sources. VPC flow logs, with source and destination IPs mapped to subnets, gave a roll-up of traffic between production clusters, storage, caches and brokers. Envoy listener and cluster metrics (cx_rx_bytes_total, cx_tx_bytes_total) gave bytes per service on the mesh. A BPF agent on every production node covered services not yet on the mesh. Put together, the numbers were blunt: 90% of all traffic was HTTP, HTTP/2 or gRPC; 45% of that was already on the mesh; and 91% of the on-mesh portion went to one destination, Iguazu, the event ingestion pipeline. A single ELB in front of Iguazu accounted for about 20% of the largest cost category in the flow logs.
The path to Iguazu explained everything. Clients called the ELB's DNS name directly rather than a DDSD name, so their Envoy clusters stayed on STRICT_DNS and zone-aware routing never applied: requests were round-robined across the ELB nodes. Cross-zone load balancing from the ELB to worker nodes was disabled, and once traffic reached a worker node, externalTrafficPolicy: Cluster let iptables forward it to a random node, which may sit in another zone. Each hop was a fresh chance to cross a zone boundary, and the mesh could not see any of them.
The fix was not a better routing policy. It was fewer hops. DoorDash moved Iguazu into the same cell as its clients and configured the client sidecars to call Iguazu pods directly instead of the ELB, with no change to application code or configuration. Calls became pod to pod, eligible for zone-aware routing, and the ELB stopped carrying the traffic. The drop in data transfer and load balancer charges was large enough that the cloud provider asked DoorDash whether it was having a production incident.
What this changes compared with the principles you already apply
If you practice DDD, you draw boundaries around domains. If you run cells, you draw boundaries around blast radius. Neither tells you anything about the physical path a request takes between two services inside the same boundary, and that path is where this money went. The DoorDash case adds a third boundary that architecture diagrams usually omit: the zone. A call graph that is clean at the domain level can still cross zones two or three times per request because of a load balancer, a NAT rule or a DNS name that bypasses the mesh.
The second lesson is about where to put the intelligence. DoorDash did not ask service teams to become zone-aware. It put zone knowledge in the control plane and the routing decision in the sidecar, so a thousand services gained the behavior through configuration. That is the same move as a domain gateway in Uber's DOMA: centralize a cross-cutting rule in infrastructure the teams already depend on, instead of in a library each team must adopt.
The third lesson is about measurement. The first mechanism was correct and still underdelivered, because the team measured what the mesh could see and the biggest offender sat just outside it. Every efficiency project needs one data source that does not come from the thing being optimized. Here it was VPC flow logs.
What to steal for your architecture
Adopt zone-aware routing when you run in several AZs, your services talk to each other synchronously at volume, and you already have a mesh or a client-side load balancer that can carry locality. Do not adopt it when a service has only a handful of pods (the local zone may hold one pod and Envoy will either spill or overload it), when your zones are badly unbalanced in capacity, or when your traffic between services is small enough that data transfer is noise on the bill. And do not adopt it as a substitute for removing hops: it cannot route around a load balancer it does not know about.
For one team on a single cluster, skip the mesh. Turn on topology-aware routing in Kubernetes Services, add a zone spread constraint with ScheduleAnyway, and make sure internal callers use cluster-local names rather than a public or internal load balancer. Then look at one month of cost broken down by data transfer type; if inter-AZ is not in your top five, stop there.
For ten teams, write down the rule that internal calls go through service discovery, never through an ELB or a public hostname, and check it in review. Enable VPC flow logs on the subnets that matter and build one table of bytes between subnets. That table will show you your Iguazu, because every organization of that size has one: a shared ingestion or logging endpoint that everyone calls through a load balancer.
For a hundred teams, do what DoorDash did. Put locality in the control plane, deliver endpoints by EDS, make zone-aware routing the default for every mesh-managed name, and treat any STRICT_DNS cluster as a finding. Keep a standing cost view that joins flow logs, sidecar byte counters and node-level eBPF data, and review the top destinations by cross-zone bytes each quarter. The next step DoorDash lists, single-AZ routing for large transfers to PostgreSQL, CockroachDB, Redis and Kafka, is where the same thinking goes after the HTTP traffic is under control.
The question to ask in your next architecture review is simple: for the three highest-volume calls in the system, how many network hops does a request make, and how many of them are allowed to change zone?

