Mike Cann

4 months ago

How to Prevent Queue Bottlenecks with Workpool

Say a third-party service fires off a burst of webhooks and a hundred "email delivered" notifications land in your app at once. None of them are mission-critical. But if your app schedules them the same way it schedules everything else, they can pile up in front of the one task that actually is time-sensitive, the chat response your user is sitting there waiting on.

That collision between "throughput" work and "latency" work is the problem the Workpool component was built to solve.

The scheduler is one big queue

Anyone getting started with Convex reaches for queries, mutations, and actions, usually wired up from the client through hooks. The moment you need something asynchronous, the primitive on offer is Convex's scheduler: runAfter or runAt. Those let you escape a mutation and hand off to an action, or chain one mutation into another.

Queues are great at absorbing bursts. The trouble is that Convex's scheduler gives you serverless queues out of the box, but it's just one queue for the whole deployment, and one big shared queue is rough on user latency. Say a user sends a chat message and you're generating an LLM response. That's a "thinking" bubble on screen, and the perceived latency matters. Normally you save the message in a mutation, enqueue an action to write the response back, and it's fast. But once a hundred webhook-triggered tasks land in that same queue, the user's response is stuck behind all of them. The scheduler has no concept of which tasks matter more.

Workpool exists to give you a bounded bucket for a given category of task. Think of it as carving that one big queue into several purpose-built serverless queues. Low-priority throughput work can't crowd out latency-sensitive work just by showing up in volume.

Why isolation at the component level doesn't fix this by itself

You might wonder whether Convex's component isolation already solves this. Components run in their own sandboxed environment, each with its own database, file storage, and even its own scheduler. But those per-component schedulers all still compete for the same underlying resource: the pool of isolates that execute your code. That pool is capped per deployment and varies by pricing tier. A component's scheduler can only read and cancel its own scheduled jobs, but every component's jobs ultimately land in the same execution queue. Isolation doesn't buy you prioritization.

Workpool solves this differently. Instead of asking the platform's scheduler to run many things and hoping it sorts out priority, Workpool collects work into its own table. Then it enforces a hard cap on how many of its tasks can be in flight at once. The underlying scheduler never sees more than that number of items from a given Workpool at any time. It doesn't need to understand priority, because Workpool has already throttled what it hands over.

There's no platform-level prioritization between components today. If you want the webhook-processing queue to stay out of the chat queue's way, you stand up two separate serverless queues as Workpools, size them independently, and route the webhook path to enqueue through one of them. A large backlog on the email Workpool doesn't touch the headroom reserved for the LLM Workpool, because they're enforcing separate concurrency limits against separate queues.

Workpool adds two things on top of what the raw scheduler gives you: a parallelism limit (cap it at ten concurrent tasks, for instance) and an onComplete handler. The scheduler will run your function, but it won't tell you anything about how it went without you polling for it. Workpool guarantees a callback that fires whether the task succeeded or failed, with enough context to know which.

The Workflow component, which handles durable multi-step processes, is itself built on top of Workpool, and it's probably one of the more fundamental components in that sense. But what does "built in" mean in convex, and where's the line between the core platform and the libraries built on top of it? Convex's answer leans, deliberately, toward the library model rather than a Firebase-style batteries-included one.

The scheduler is the primitive. Workpool, Workflow, rate limiting, and similar concerns are components layered on top, open source and forkable if you want different behavior.

What happens to a task when you enqueue it

Under the hood, enqueueing work writes into a work table. That table holds the function reference, its arguments, and its onComplete handler. Alongside it sit separate pending_start, pending_completion, and pending_cancelled tables that represent queue state.

Workpool Task Lifecycle

That split is deliberate. Convex stores documents as multi-version records, so writing to a document means writing a whole new copy of it. Documents also have a one-megabyte cap. If a status field lived directly on the work table, every status change would rewrite the entire document, arguments and all. That gets expensive fast when the payload is large. Keeping the pending-state tables separate and small lets the loop read, write, and delete them cheaply. It also narrows exactly who's reading and writing what, which matters for avoiding conflicts. The work table itself is written once, so it can be read without contention.

Enqueue also has to decide who kicks off the Workpool's core loop, since there's no daemon sitting around in a serverless environment. The loop reads from the pending tables to see what's changed and tracks its own state about what's in flight. An entry in pending_start gets grabbed and handed to the scheduler to run the user's function. An entry in pending_completion triggers a lookup of the original work entry and a call to its onComplete handler, if one exists. Cancellation gets its own pending_cancelled queue. A cancel request can race against a task that's already running, so the loop has to decide, correctly, whether to honor the cancel or admit the task already finished.

Every task runs inside a wrapper, a mutation or action that executes your function in a try/catch. Failure calls onComplete with the failure immediately. Success calls it with the return value. That's also why the design leans on pending_completion as a cue rather than a direct state mutation. It lets the wrapper say "you don't need to track me anymore" without touching the loop's internal state and risking a conflict.

Why the loop has to fight its own database

This is the part of Workpool's design that generalizes well beyond queuing.

Convex's serializability guarantee is stronger than a lot of engineers expect from optimistic concurrency control. A query doesn't just take a read dependency on the individual documents it returns; it takes a dependency on the range it queried. Ask for "the latest N pending tasks." If another process inserts a new task into that range while your transaction is still open, your transaction conflicts, even though it never touched that specific document. For a loop processing a busy queue, a naive "give me the newest N" query can conflict over and over as new items keep arriving. That stalls the loop instead of letting it make progress.

OCC Conflict Retry

The fix is to bound the time range the loop is responsible for on each run. Rather than asking "what's pending right now," the loop gets handed a fixed window, something like a 250-millisecond slice. It only reads within that slice. New writes landing outside the window can't invalidate the transaction, because the query never claimed a dependency on them. The mutation also pins its own notion of "now" so that retries hit the same segment instead of drifting to a new timestamp each time. Convex mutations have a maximum runtime and a bounded commit window. Starting the loop's window slightly in the future and looking slightly backward trades a bit of latency for far fewer retries.

The same pattern generalizes past queues. Say many producers all want to update one shared aggregate, a running total cost. Having each one mutate that aggregate directly means constant contention over the same document. The fix is the same shape. Producers write the operation they want applied into a table, and a single asynchronous worker (a Workpool, in practice) reads those operations and applies them serially. One writer owns the aggregate; everyone else just queues an intent.

Tombstones are a real, if obscure, tax on high-churn tables

With multi-version storage, deleting a document doesn't erase it outright. It leaves a tombstone, retained for roughly a four-minute window so older snapshots can still be read consistently. On a table where documents are inserted and deleted constantly, like Workpool's pending-state tables under load, tombstones accumulate inside that window. A scan starting from time=0 has to skip over all of them, which slows down as churn increases.

Tombstone Bounded Scan

The workaround is to remember the last timestamp you successfully read to and query forward from there, rather than re-scanning from the beginning each time. That skips the accumulated tombstones. You still need an occasional full scan to catch anything that arrived out of order. But bounded reads avoid the degenerate case where a hot queue's read performance degrades as it processes more items.

This is exactly the kind of problem that's invisible at low traffic and shows up right when an app starts gaining real usage. It lives below Convex, not something your application code would normally have to reason about.

What might change next

One direction under consideration is converting the loop from a mutation into an action. A query executes at a fixed timestamp, so it doesn't conflict with newer writes, and that's the motivation. An action that calls a query to figure out what work is pending could read without triggering the same OCC pressure the mutation-based loop manages around today. The action would then dispatch individual point-update mutations rather than one mutation contending over a shared range.

The tradeoff is that queries and mutations are no longer atomic with each other once you split them that way. A mutation-based loop can read and change state in the same transaction; an action-based loop can't, which opens a window for races, particularly around cancellation. The responsibility for checking "was this canceled?" would move into the wrapper that starts a task, right before it runs. That check still sits inside a transaction small enough to stay serializable, just no longer the same transaction that decided what to run.

The other direction on the table is support for multiple queues inside a single Workpool: one global concurrency limit with per-key or per-user limits nested inside it. Today, getting that kind of fairness (per-user caps, protecting against one noisy tenant, giving certain customers more throughput) means standing up separate Workpools for each case. That makes it hard to guarantee a single global limit while keeping the pool fully used. A shared pool with finer-grained limits inside it would behave more like a single well-managed line at an airport than several separate lines that go idle or overloaded on their own.

Rate limiting and token-based throttling are on the same wishlist. Think the existing Rate Limiter component, but folded into Workpool's own concurrency model, covering most cases by default while staying open source enough to fork if it doesn't fit yours.

None of this requires abandoning Convex's serverless queues for asynchronous work at scale, either. You can still reach for an external system, a managed queue or your own service called from an action, and use Convex purely as reactive glue between that system and your UI. But that gives up something specific. Convex's reactive queries mean subscriptions cascade automatically, so a frontend tracking an async job's progress updates without you wiring up any eventing yourself. For asynchronous work tied to UI state, that's the payoff a plain external queue doesn't hand you for free.