Last Saturday, a durable execution platform stopped scheduling work for roughly three hours because of a timeout. Not a missing one. A short, sensible, carefully chosen client-side deadline on a frequently used database lookup.
That is the finding buried in Inngest's incident report for October 9 and 10, and it deserves more attention than the network outage that shares the title. The thesis of this issue is narrow: when a client gives up on a query that is still waiting in a pooler's queue, the cancellation can cost more than the query would have. In the wrong configuration, every deadline you set is a small amount of damage, and under load the damage is paid in connections you cannot rebuild fast enough.
What actually happened
Two separate incidents, and it helps to keep them apart.
On October 9, from 21:05 to 21:49 UTC, a data center provider lost network connectivity to Inngest's other data centers in both directions, plus outbound internet access. The provider's preliminary explanation, as quoted in the report, is an error during a change to its DDoS protection configuration. That is 44 minutes, and the report is candid about why it spread: execution depends on all three data centers, run state for some accounts lives in a different data center from the services executing it, and accounts with no dependency on the broken site still shared an event pipeline with those that had one. Failing work built a backlog, and the backlog delayed everyone. Accounts on dedicated pipelines were not affected.
That is a classic shared-fate story, and worth its own essay. The second incident is the stranger one.
On October 10, the pooling layer collapsed twice, from 14:00 to 14:48 and from 15:30 to 16:25 UTC. Services reach the primary database through a pair of PgBouncer poolers running in AWS. Both poolers fell to nearly zero usable connections. Run scheduling could not look up accounts and functions, and queue processing stalled for most accounts.
The report gives the mechanism in five steps, and I am going to restate them slowly because each one is individually reasonable.
Several frequently used lookups have short client-side deadlines.
When a deadline expires while the query is still waiting inside PgBouncer for a server connection, the driver cancels the query and closes its client connection.
PgBouncer had nothing running to cancel. So it later ran the abandoned query anyway, and then closed the database connection entirely instead of returning it to the pool.
Each pooler opens new server connections one at a time, so connections were destroyed faster than they could be replaced.
Queued clients then hit their own deadlines, which produced more cancellations, which destroyed more connections.
Inngest says it reproduced this in a test environment. The triggers were also ordinary: cron-scheduled work at the top and bottom of each hour roughly triples the query rate, and both collapses began at exactly those moments, plus a new set of lookups had been rolled out to all accounts before the weekend and began failing in large numbers at the same time.
Why a timeout makes this worse
The intuition behind a client deadline is load shedding. If the database is slow, stop waiting, free the caller, and let the system recover. That intuition holds when cancelling a query is cheap and removes work from the system.
Here it did neither. The query was not running yet, so there was nothing to stop. It sat in the pooler's wait queue, and a cancel request for a query that is not assigned to a server has nothing to attach to. The work stayed in the system. The only thing the cancellation changed was the state of the client connection, and, by the report's account, the fate of the server connection that eventually ran the abandoned query.
This is the part I would want every backend engineer to internalize. A deadline has two jobs that people routinely conflate: ending the caller's wait, and ending the work. They are different operations with different costs. A client can stop waiting for free. Stopping the work requires cooperation from every layer between the client and the database, and a pooler in the middle is exactly the layer where that cooperation breaks.
The PgBouncer docs are explicit about the queueing side of this. The query_wait_timeout setting says that "if the query is not assigned to a server during that time, the client is disconnected," and that "if this is disabled, clients will be queued indefinitely." There is a dedicated cancel_wait_timeout for cancellation requests, described as the "maximum time cancellation requests are allowed to spend waiting for execution." The configuration reference treats cancellation as a first-class path with its own queue and its own timeout. That alone should tell you it is not a free operation. I am not claiming the exact failure Inngest saw is a documented PgBouncer behavior, because the report does not name a version or a setting. I am claiming the surface area is large enough that you should not assume it is benign.
Then there is the arithmetic of rebuilding. If a pooler opens connections one at a time and each open takes a handful of milliseconds to tens of milliseconds plus authentication, its replacement rate has a hard ceiling. Any process that destroys connections above that ceiling drains the pool, and a drained pool makes every queued client wait longer, which pushes more of them past their deadlines. That is the feedback loop. Positive feedback with a fixed repair rate has one stable state, and it is empty.
The report notes the practical consequence: restarting the poolers did not reliably recover them, and raising the pool size did not help, because opening connections was the bottleneck, not holding them. If you have ever responded to a connection exhaustion incident by bumping default_pool_size, this is the case where that instinct is exactly wrong. The recovery that worked, between 16:09 and 16:28, was moving all services to poolers run alongside the database by the database provider, which removed the slow path entirely. Error rates returned to normal at 17:10.
The fix was small, and that is the point
The shipped fix, in the report's words, is that a lookup's deadline "now ends the caller's wait without cancelling the underlying query." Inngest is also reviewing other database calls on the execution path and their timeouts.
That is a one-line change in concept. In sketch form, and this is my illustration of the pattern, not Inngest's code:
// Before: the deadline cancels the in-flight (or queued) query.
ctx, cancel := context.WithTimeout(parent, 50*time.Millisecond)
defer cancel()
row := db.QueryRowContext(ctx, q, id)
// After: the caller stops waiting, the query is left alone.
type result struct { row *sql.Row }
ch := make(chan result, 1)
go func() {
// detached from the caller's deadline
ch <- result{db.QueryRowContext(context.WithoutCancel(parent), q, id)}
}()
select {
case r := <-ch:
return r.row, nil
case <-time.After(50 * time.Millisecond):
return nil, ErrLookupSlow // caller moves on; query finishes on its own
}
I have not run this against a pooler, and I would not copy it blindly. Detaching the work means abandoned queries keep running, so you now need a separate bound on how many can pile up, otherwise you have traded connection churn for unbounded queue growth. The honest summary is that you are choosing which resource to protect, and the original code protected the caller at the expense of the pool.
The second shipped fix is a quieter lesson. During both incidents, internal failures counted against functions' configured retries, and a step that failed permanently because of an internal error left its run in Running instead of marking it failed. As a result, onFailure handlers and functions waiting through step.invoke were never notified. Those runs will not continue on their own. Customers must cancel and replay them, and replayed runs start from the beginning, so steps that already completed run again. Platform faults leaked into user-visible retry semantics, and user-visible state was left lying. A run that says Running and will never run again is worse than a failure, because nobody is paged by it.
The mitigation that bit back
One more detail, because it is the most transferable. The move to the new poolers routed traffic through network address translation gateways, which silently dropped some connections. Queries on dropped connections hung for up to 16 minutes before failing. Customers reported delays at 03:01 on October 11, and cron scheduling was moved back to the original poolers at 04:17. The report lists intermittent delays running from 21:07 on October 10 to 04:17 on October 11, though the timeline has no entry for 21:07 and Inngest does not say what started them.
The pattern is familiar to anyone who has fought an incident at 3 in the afternoon with a fix that has to ship now. The emergency path was less tested than the normal path, and its failure mode was a hang rather than an error. Hangs are the worst failure mode a distributed system has, because they pass health checks and eat timeouts.
The strongest objection
The reasonable pushback is that none of this is a timeout problem. It is a pooler problem, a PgBouncer behavior, and the correct response is to configure the pooler properly: set query_wait_timeout, size pools, add min_pool_size, put a circuit breaker in front of the hot lookups.
I agree with most of that. A circuit breaker would plausibly have cut the amplification, and the right pooler settings might have made the cancellation path cheaper. But notice what the objection concedes. It concedes that the interaction between a client deadline and a pooler queue is a configuration surface that has to be understood and tested, and almost nobody tests it. Deadlines get set per call site by whoever wrote the call, in milliseconds that felt right on a quiet day. The pooler gets configured once, by someone else, years ago. The failure lives in the gap between them, and neither owner sees it.
Also, to be fair to Inngest: the report is unusually specific for an incident write-up, and it reports one thing most do not, which is a mechanism they reproduced. The unknowns are real, though. The report says "most accounts" and "some runs" were affected without counts, and the provider's full report on the October 9 network event is still pending.
What to do on Monday
Find your three hottest database call sites and read their deadline code. For each one, answer a single question: when this deadline fires, does the work stop, or only the wait? If the work stops, find out what the layer in the middle does with the cancellation. If you run a pooler, check whether the abandoned query still executes and whether the server connection survives it.
Then run the experiment Inngest could not run in production. In a test environment, push your request rate to three times normal at the top of the hour, with deadlines tight enough to fire, and watch the pooler's connection count rather than its error rate. The error rate lags. The connection count is the number that falls first.
A timeout is a decision about who pays when the system is slow. Make sure you know who that is.
