Pingora is the HTTP proxy Cloudflare wrote in Rust to sit between its network and customer origin servers. When Yuchen Wu and Andrew Hauck described it in September 2022, it was serving over 1 trillion requests a day and powering the CDN, Workers fetch, Tunnel, Stream and R2. Before Pingora, that path ran on NGINX, extended with Lua through OpenResty.

The interesting part of the story is not "Rust is fast". The authors say plainly that the old service already handled requests in the sub-millisecond range. The latency win came from somewhere else: the shape of the process that held the connections. That makes this a story about application architecture, and it applies far outside proxies. Any service that talks to a database, a cache or a downstream API holds a pool of expensive connections, and the scope of that pool is decided by how you split your work into processes and threads.

NGINX runs a fixed set of worker processes. Each incoming request lands on one worker and stays there for its whole life. The Cloudflare team listed five problems with that model, and two of them were architectural rather than cosmetic.

The first was load imbalance. Because a request is pinned to a worker, a CPU-heavy request or a blocking I/O call slows down every other request queued on the same worker, while neighbouring cores can sit idle. The second, which the authors call the most critical, was connection reuse. Connection pools in NGINX are per worker. A request on worker 3 can only reuse a connection that worker 3 opened earlier. Add more workers to absorb more traffic and you split the pool into more, smaller pools, so the reuse ratio drops. Lower reuse means more TCP and TLS handshakes to origins, which means slower time to first byte and more connections for both Cloudflare and its customers to keep alive.

That is the trap: the standard way to scale the system made the system slower per request. No configuration change fixes it, because the limit is the boundary between processes.

The other problems compounded the pressure. NGINX would not let Cloudflare retry or fail over a request to a different origin with different headers, a feature they needed. NGINX is written in C, which is not memory safe, and the business logic lived in Lua, which was slower and had no static types for complex code. The team ran a quarterly build-or-buy review with three options: keep investing in NGINX and maybe fork it, migrate to a third-party proxy such as Envoy, or build their own. They chose to build, on the bet that the return would justify the upfront cost.

Pingora uses multithreading instead of multiprocessing. Every thread can see every connection, so the pool is global to the process instead of private to a worker. This is the decision that moved the latency numbers.

The results Cloudflare published are specific. Pingora opens only a third as many new connections per second as the old service. For one major customer, the connection reuse ratio went from 87.1% to 99.92%, which cut new connections to that customer's origins by 160x. Cloudflare estimates the saving at 434 years of handshake time per day across its customers and their users. Median TTFB dropped by 5 ms and p95 by 80 ms. The p95 gap is the telling one: the long tail is where a request pays for a fresh TCP and TLS handshake, and a shared pool makes that the rare case.

The trade-off is that shared state between threads now needs synchronisation, and a bug in one thread can take down the whole process instead of one worker. Cloudflare paid for the first with Rust's ownership model, sharing data through atomic reference counters (Arc) instead of the mutex-locked shared memory NGINX uses between processes. That choice also shows up in the efficiency numbers: about 70% less CPU and 67% less memory for the same traffic. The authors attribute it to Rust running more efficiently than Lua, to avoiding the copying and garbage collection Lua needed for header access, to Arc replacing locked shared memory, and to fewer TLS handshakes, which are expensive. They paid for the second cost, the blast radius of a crash, with memory safety: since launch, Pingora served a few hundred trillion requests with no crash attributed to its service code. The crashes they did see came from a kernel bug and from hardware faults on a few machines.

Sharing a pool fixes reuse but not the imbalance problem. If requests were still pinned to the thread that accepted them, one slow request could still starve its neighbours. Pingora runs on the Tokio async runtime and relies on work stealing: an idle thread takes queued tasks from a busy one. Load spreads across cores at task granularity instead of at connection-accept time.

What it costs: work stealing moves tasks between threads, so every piece of per-request state must be safe to send across threads. In Rust the compiler enforces that, which is why the language choice and the runtime choice are one decision here, not two. In a language without that guarantee, the same design invites data races that a multiprocess model never had. That is the honest reason many teams stay with process-per-worker: it is slower to scale but harder to break.

The third decision is about extensibility, and it is where Pingora looks most like application architecture. Instead of a configuration language, Pingora exposes a programmable, event-based interface modelled on the NGINX and OpenResty "life of a request". The developer implements callbacks for named phases. A request filter phase runs when the headers arrive and can modify or reject the request. Other phases choose the upstream peer, rewrite the upstream request, and so on. Generic proxy logic stays in the framework; business logic lives in those callbacks.

This is what the open-source quick start looks like for a round-robin load balancer, condensed from the Pingora repository:

Two details matter. The load balancer itself is wrapped in an Arc, the same sharing primitive as mechanism 1, so every thread sees one set of upstreams. And the retry-with-different-headers feature that NGINX blocked becomes ordinary code in a phase callback rather than a patch to the proxy core. The authors report that adding HTTP/2 to upstreams took no major effort and enabled gRPC for customers shortly after, and that Cache Reserve, which uses R2 as a caching layer, was built on the same platform.

Cloudflare also wrote its own HTTP library instead of using hyper, because it sees large amounts of traffic that does not follow the RFCs, such as status codes above 599, which hyper rejected at the time. Hyper later accepted the change, but Cloudflare still wanted to control that behaviour itself. The cost of that choice is a protocol implementation to maintain forever. They took it because tolerating malformed traffic is part of their product.

Most architecture discussions talk about boundaries between services: Domain-Driven Design draws them around business language, event-driven architecture decides how services talk across them. Pingora is a reminder that there is a boundary inside every service too, between processes, and it has measurable cost. A connection, a cache, a compiled template or a rate-limit counter that lives behind a process boundary cannot be shared. Every time you scale by adding processes, you divide those resources.

The phase-based interface is also a familiar pattern applied in an unusual place. It is the template method, or a middleware pipeline, with a fixed lifecycle owned by the framework and extension points owned by the product teams. Uber's DOMA used extension points to let teams customise behaviour without editing a domain's core services. Pingora does the same inside one binary. The common idea is that the platform owns the order of operations and the product owns the decisions at each step.

Adopt the shared-pool idea when your service spends meaningful time on connection setup: TLS to downstream APIs, database connections behind a pooler, gRPC channels. The signal is a gap between median and p95 latency that grows when you add replicas or workers. Measure your connection reuse ratio before you touch anything; Cloudflare's whole case rests on going from 87.1% to 99.92%.

Do not adopt it when your runtime does not give you safe sharing, or when your workload is CPU-bound per request with little connection churn. In those cases the process model's isolation is worth more than the pool. Do not build your own proxy either: Cloudflare had 1 trillion requests a day, a quarterly build-or-buy review and a protocol edge that no off-the-shelf library tolerated. Pingora is now open source, so the build decision is already made for you.

A concrete proposal by size of organisation. For one team: check how many processes your app server forks (Gunicorn workers, PM2 cluster, Node cluster) and whether each one holds its own database and HTTP pool. Prefer fewer processes with more threads or async concurrency, and put PgBouncer or an equivalent pooler in front of the database so the pool is shared at the infrastructure level. For ten teams: give every service a standard client library with a process-wide pool and expose reuse ratio and handshake count as metrics in your shared dashboards, so the regression is visible when someone scales by replica count. For a hundred teams: build the edge or service mesh as a programmable platform with named phases, like Pingora's filters, owned by a platform team, and let product teams ship logic in those phases instead of forking the proxy or asking for configuration flags.

The lesson is short. Before you add a worker, ask what it will stop sharing.