Airbnb's payments platform, by its own 2016 account, moves money across 191 countries, over 70 currencies and over two dozen payment routes, from card networks to PayPal to Boletos, the cash invoices used in Brazil. When the engineering organization moved to a service-oriented architecture, the payments flow stopped being one database transaction and became a chain of network calls between internal services and external processors. Each of those calls can time out after the money moved but before the caller learned about it.
The cost of getting that wrong is concrete. In the words of the Orpheus post by Jon Chew and Ninad Khisti, idempotency is what prevents "multiple payouts to the host, and even worse, multiple charges to the guest." Their result: five nines of payment consistency since launch, during a period when annual payment volume doubled.
This issue looks at how they got there. The interesting part is not that Airbnb uses idempotency keys. Everyone says they do. The interesting part is where Airbnb put the idempotency logic, and which rules it imposes on the code around it.
The architecture before, and what broke
The 2016 platform post describes a system that had grown by accretion. Payment integrations were added one at a time without a shared framework, so every new processor was built from scratch. Core payment objects were mutable, audit trails came from database triggers, and financial reports were reverse engineered from those trails with complex SQL. As volume grew, the reporting database hit its limits and reports took many hours to run.
The answer was a set of dedicated services: a Billing Interface API, a Payments Gateway with an append-only data model, and a Financial Data Pipeline fed by Kafka events. That fixed mutability and reporting. It also created the classic distributed systems problem. Once a payment crosses several services, there is no single transaction to roll back, and two-phase commit across services and external banks is not an option.
The Orpheus post names three ways to converge back to a consistent state: read repair (fix on read), write repair (fix on every write by the client), and asynchronous repair (a background job reconciles). Airbnb uses all three in different parts of the system. Orpheus is the write repair piece: every client write, retried as many times as needed, must land exactly once.
The stated goal was one generic, configurable idempotency solution for all payments services rather than a custom fix per use case, with no compromise on consistency, ultra low latency, and product engineers shielded from the data integrity mechanics.
Mechanism 1: three phases, and a hard wall between network and database
Orpheus forces every payment request into three phases.
The Pre-RPC phase records the request details in the database. The RPC phase calls the external party, a processor or an acquiring bank, or runs an idempotent status query when the request is a retry. The Post-RPC phase records the outcome, including whether a failure is retryable.
Two rules make this work. No network calls are allowed in Pre-RPC or Post-RPC. No database access is allowed during RPC. The library wraps each database phase in a single transaction that it opens itself, so the request record and the application's own writes commit or fail together.
The second rule is not only about correctness. The authors report that network calls made inside database transactions had exhausted connection pools and degraded performance. Holding a database connection open while waiting on a bank's API is the fastest way to turn a slow processor into an outage of your own service. Splitting the phases removes that coupling by construction.
To merge the library's writes with the product code's writes in one commit, Orpheus takes Java functional interfaces (lambdas) for each phase. The team deliberately did not implement nested transactions. Here is a simplified sketch of the shape, written by me to illustrate the contract, not copied from Orpheus:
PaymentResult charge(ChargeRequest req, String idempotencyKey) {
return orpheus.execute(idempotencyKey, req,
// Pre-RPC: one DB transaction opened by the library, no network
tx -> tx.insert(new PaymentAttempt(req, Status.PENDING)),
// RPC: network only, no DB connection held
() -> processor.authorize(req.cardToken(), req.amountMinor()),
// Post-RPC: one DB transaction, no network
(tx, response) -> tx.update(PaymentAttempt.from(response))
);
}
The trade-off is explicit in the post. Developers must split every API call into three parts, keep the lambdas readable and pass dependencies correctly. For complex flows that is hard, so one Airbnb service models the flow as a finite state machine (StatefulJ) in which each transition is an idempotent step. Airbnb's later rebuild of payment orchestration pushes the same idea further: each major workflow is a directed acyclic graph of retryable, idempotent steps running on Orpheus, and Kafka's at-least-once delivery is turned into exactly-once processing by the same framework.
What it costs: a programming model that every payments engineer has to learn, and code that reads less naturally than a single method with a try block. What it buys: a transaction boundary that the library, not each developer, controls.
Mechanism 2: retry classification and leases
Idempotency keys only help if the system knows when a retry is safe. Orpheus classifies every failure as retryable or non-retryable and stores that classification in the Post-RPC phase.
The default mapping is simple. Network and infrastructure errors, the 5XX class, are retryable. Validation errors, the 4XX class, are non-retryable, for example an attempt to refund a refund. The post is direct about the asymmetry of mistakes. A transient error marked non-retryable fails a payment permanently, which is a support ticket. A non-retryable error marked retryable can produce a double payment, which is lost money and lost trust. Null values get special care: a null caused by a database connectivity issue is not the same as a null returned by a third party.
Concurrency is handled with leases. Each request takes an expiring lease on its key, and a retry is admitted only after the lease has expired. The lease duration must be longer than the RPC timeout, otherwise a retry could start while the original call is still in flight at the processor. A maximum retryable window caps how long a key stays retryable at all, as a safety net against rogue retries.
The client has duties too, and the post lists them: generate a unique key per new request and reuse it on every retry, persist the key before calling the service, clear it after consuming a successful response, never change the payload between retries, and back off exponentially with jitter to avoid a thundering herd.
Key design follows two patterns. Request-level keys are random UUIDs, which allows several distinct payments for the same reservation. Entity-level keys are deterministic, such as payment-1234-refund, which guarantees that a given refund can happen only once whatever the client does.
What it costs: every error path must be classified on purpose, and the classification becomes a correctness property that deserves tests. What it buys: retries become a normal tool instead of a risk.
Mechanism 3: read the primary, shard by the key
The last decision is about where the idempotency data lives. Orpheus reads and writes its tables only on the primary (master) database, never on a replica.
The reasoning is short. If a retry reads from a replica that has not yet received the recorded response, the retry looks like a new request and the money moves twice. Replication lag, normally a freshness problem, becomes a double charge problem. So the team accepted the load on the primary.
As volume grew, the primary was sharded by idempotency key. The authors note that these keys have high cardinality and an even distribution, which makes them good shard keys: no hot customer, no hot reservation, no rebalancing headache driven by business skew.
Recorded responses also let retries return fast without calling the processor again. That makes the table grow with throughput. Old rows can be pruned, but pruning too early breaks correctness for late retries, and response schemas must stay backward compatible because an old stored response may be returned by new code.
The team also rejected a standalone idempotency service. A separate service would have added a network hop on the critical path and, worse, would face the same consistency problem it was meant to solve: its own writes and the product's writes would again sit in two systems.
What it costs: primary capacity, a sharding layer, and a retention policy you must reason about. What it buys: idempotency guarantees that do not depend on replication timing.
What this changes compared with the usual principles
Domain-driven design tells you to keep a transaction inside one aggregate and to accept eventual consistency between aggregates. Event-driven architecture tells you consumers must be idempotent because brokers deliver at least once. Both are correct and both stop at the principle. Neither tells you where the idempotency record lives, who opens the transaction, or what happens when an external bank times out after charging.
Orpheus answers those questions with a library that owns transaction boundaries. That is closer to the outbox pattern than to a utility function: the record of intent and the business write share one local transaction, and the side effect sits outside it. The difference is that Orpheus generalizes the pattern to any outbound RPC, including synchronous calls to third parties, and adds retry semantics on top.
It also inverts a common reflex. When a concern is cross-cutting, many teams build a platform service. Airbnb chose an in-process library precisely because the concern is transactional, and a transaction cannot span a network hop.
What to steal for your architecture
Adopt this design when your service calls something that moves money, sends a message or changes state outside your database, and when a client can retry. Payments are the obvious case. Booking engines, inventory reservations, outbound emails and webhooks to partners are the same problem with lower stakes.
Do not adopt it in full when your writes stay inside one database: a unique constraint and a single transaction are enough. Do not adopt it when the external party already gives you idempotency you can trust end to end, as most modern payment providers do with their own idempotency key header. In that case, pass your key through and keep only the local record.
For a single team, the proposal is a table, not a framework: idempotency_keys(key primary key, request_hash, status, response, locked_until, created_at), written in the same transaction as the business row, a rule that no HTTP call happens inside a database transaction, and a lint or code review check for it. Reject a retry whose payload hash differs from the original.
For ten teams, extract that into a shared library with the three-phase contract, a default 5XX and 4XX mapping that teams can override, leases longer than the RPC timeout, and a retention job. Make the library the only way to call a processor, and route idempotency reads to the primary in its data access layer so nobody can point it at a replica by accident.
For a hundred teams, add what Airbnb added later: workflows expressed as graphs or state machines of idempotent steps, events consumed through the same library to get exactly-once processing on top of at-least-once delivery, idempotency tables sharded by key, and asynchronous reconciliation that compares your ledger with the processors' reports. At that size the five nines number is not a property of one service. It is the product of write repair, read repair and asynchronous repair working together.
