Convex's avatar
Convex
6 months ago

What ACTUALLY happens when you push to Convex?

Convex users often ask what the difference is between typing code in your editor and that code running when a user hits your app. There are really two journeys hiding in that question. The first is the trip your code takes from npx convex dev up to a live deployment. The second is the trip a single request takes from someone's browser, through our execution fleet, into the database, and back. Here's both of them, including the parts we haven't figured out yet.

From npx convex dev to a deployment

When you run npx convex dev for the first time, you get a fresh deployment. That deployment is the backend for your specific Convex project: your tables, your functions, your indexes. When you open the dashboard and look at your data, you're looking at that deployment. Every one of them is its own logical backend, which trips people up. A lot of developers picture "Convex" as one giant shared system that gets sliced into per-customer slivers, and that's not the model at all.

How a deployment is hosted has changed a lot, though. Two years ago every deployment ran as its own Nomad job with its own container and its own process. That was easy to reason about and expensive to run. Most deployments sit idle most of the time, so we were holding compute and memory for workloads that weren't happening. Multiply that by tens or hundreds of thousands of deployments, which is what you get in a world where people spin up backends from a prompt, and the waste adds up fast on any cloud platform.

Last year we moved to a multi-tenant service called Conductor that runs thousands of deployments inside a single container. More on why that mattered later.

Start push: type-checking your code in V8

Once you have a deployment, npx convex dev or npx convex deploy sends your code up in what we call a startPush request. Your deployment doesn't analyze that code itself. It hands it to a separate service called FunRun, which runs it through V8, the same JavaScript engine Chrome uses.

What FunRun is doing there is type-checking. We can't tell whether your code works, and we're not trying to. Type-checking is a strong enough signal that the code is probably fine, and it catches an enormous class of mistakes before anything touches real data. Once it type-checks, we store your modules in S3.

Where your code actually lives

The S3 part surprises people, because the thing everyone associates with Convex is the database, and that's not S3. Our database currently runs on PlanetScale, on Vitess, their sharded MySQL engine. It hasn't always. We've been on RDS, and we've moved between Postgres and MySQL more than once over the years.

None of that matters for a code push, though. Your modules, your files, and our text and vector indexes all live in S3, separate from the transactional data. So a push is mostly a conversation between the CLI, your deployment, FunRun, and an object store. The database only enters the picture when your schema does.

Schema and index workers, then an atomic cutover

If your push includes a schema, that same startPush kicks off two more workers in parallel. An index worker starts building any new indexes your schema defines. A schema validation worker checks the new schema against the data you already have, because a schema you can't actually satisfy with existing documents is a problem you want to hear about now.

Meanwhile the CLI sits there polling. Is it ready? Is it ready? Only when both workers have finished does it send a finishPush request. That's the moment your deployment stops running the old version of your code and starts running the new one.

A Convex pushA Convex push

That two-step shape is deliberate. We wait until the new indexes are fully built and the new schema is fully validated before cutting over, so there's never a window where your deployment is running new code against an old, incompatible schema. Anyone who has done manual deploys elsewhere has probably lived through the alternative. The database migration lands, the code doesn't ship for another minute, and for that minute your users are hitting a genuinely broken state.

That window doesn't exist here. Every change in a push applies at once, which is why we call it an atomic push, and it buys you something you might not notice until you go back to another system. You never have to write code that works against both the old schema and the new one. In most other setups that defensive, works-either-way code is unavoidable, it piles up over the years, and it's very easy to get subtly wrong.

This is a good example of what we mean by letting developers fall into the pit of success. We'd rather make the unsafe path impossible than write a docs page discouraging it.

Why V8, and where it runs out

We picked V8 back in 2021, when it was the most tried-and-tested JavaScript engine going, ahead of SpiderMonkey or the JavaScriptCore engine Bun uses. Honestly, there might be better options today. But swapping the execution engine under a production system is a mountain of work and a pile of compatibility risk, so it isn't something we'd do casually.

The engine choice does leak through to you in one place. Plenty of people want Node-specific functionality that our default runtime doesn't support, and those get routed to Node actions running on AWS Lambda instead. We keep closing gaps in what the default V8 environment can do, and there are newer runtimes we may eventually move to, but that's a bigger conversation for another day.

The three kinds of functions

Before tracing a request, it's worth naming what can actually be at the other end of one, because the runtime story is different for each. Convex gives you three kinds of functions. One of them splits in two, and there's a fourth entry point worth knowing about.

Function kindWhere it runsWhat it's for
QueryFunRun, default V8 runtimeReads your data at a fixed timestamp. Transactional and cacheable.
MutationFunRun, default V8 runtimeReads and writes transactionally. Returns a writeset rather than writing directly.
Action (default runtime)FunRun, default V8 runtimeTalks to the outside world, where transactions don't apply.
Action (Node)AWS LambdaSame job, for code that needs Node APIs the default runtime lacks.
HTTP actionFunRun, default V8 runtimeAn action reached by a raw HTTP request instead of a client call.

Queries and mutations are the core of Convex, and they're the interesting ones here, because they're what runs transactionally against the database. Everything below them in that table exists because the outside world refuses to participate in your transaction.

What happens when a client makes a request

Once your code is pushed and type-checked, here's what happens when someone's browser actually calls it. A client, usually in a web browser, opens a WebSocket connection to your deployment. There's a whole distributed system sitting behind that connection, but for this story you can treat it as one socket to one deployment.

Convex client requestConvex client request

When the client executes a query or a mutation, your deployment doesn't run it. It forwards the request to FunRun, and the request looks roughly like this:

1{
2  deployment: "happy-otter-123",
3  functionId: "todos:addTodo",
4  args: { text: "Buy milk" },
5  timestamp: 1732489200123
6}
7

Two of those fields are doing more work than they look like they are.

The timestamp decides which snapshot of the database FunRun loads. Your function then sees a consistent view of your data as of that exact moment, however long it takes to run and whatever anyone else is doing in the meantime.

The deployment name matters for a different reason. FunRun can run functions from any deployment, so it needs to be told which one this request belongs to. That one field is why FunRun can be a shared fleet at all, and it's what took the ceiling off how much load a single Convex app can handle.

The committer and optimistic concurrency control

Your deployment holds a component called the committer, and the committer is the only thing in the entire system that writes to the database. We treat the database roughly as an append-only log of documents and indexes, with the committer as its sole writer. FunRun gets read access and nothing more.

That's true even for mutations, which surprises people. The mutation code running on FunRun never writes anything. It reads, it works out what should change, and it hands that answer back.

What it hands back is a writeset: the list of changes the mutation wants to make. The committer takes that writeset and decides whether to apply it, and this is the point where optimistic concurrency control enters the picture. The committer looks at the timestamp your function ran against and asks one question. Has anything you read changed since then?

If the answer is no, the write commits. If the answer is yes, the commit fails, and we retry it for you by sending the function back to FunRun with a fresh timestamp. Most of the time you never find out any of this happened.

When OCC conflicts show up in your app

Sometimes you do find out. If you've ever seen an error along the lines of "data written while this function was running", you've met the conflict path described above. What it means is that two functions read and wrote overlapping data close enough together that one of them lost the race, and the retries didn't clear it. Your code is fine. The system is reporting contention, which is a load problem rather than a logic one.

This gets more likely in exactly one situation: a single document that's being written to very frequently while other functions are reading it. Every one of those writes invalidates every read of that document, so the more traffic you put through it, the more collisions you get.

The fix lives in your schema. Pull the field that changes constantly out into its own document, away from the fields that are read often and rarely change. A view counter living on the same document as a post's title and body means every view invalidates every read of the post. Move the counter somewhere of its own and the collisions mostly evaporate.

We've run into this plenty while building Convex itself, and the answer has been the same every time. It's also why Convex components are designed the way they are, with hot fields deliberately kept apart from cold ones. The architecture nudges you toward the layout that doesn't fight itself.

Splitting execution out into FunRun

FunRun exists because of one number. V8 can run roughly 128 concurrent functions in a single process, and that's a hard ceiling.

Three years ago, execution happened inside your deployment process, so that ceiling was your ceiling. Customers were outgrowing it. Splitting execution out into a standalone service is what fixed it. FunRun runs across a cluster of machines and isn't tied to any one deployment, so an app under heavy load can be served by many FunRun instances at once. That's thousands of concurrent queries, mutations, and actions instead of 128.

I want to be careful about how far to push that claim. The goal is a backend that scales to any workload you throw at it, and we aren't there yet. Thousands of concurrent functions is a real answer to a real problem, and it isn't the last one we'll need.

The 128 figure itself comes down to memory and CPU. Every V8 isolate costs real resources to spin up, and we cap isolate memory somewhere in the 100 to 500 megabyte range (closer to the top of that range in practice). A FunRun machine runs on 16 cores with roughly eight of them given over to V8, and a process can only host one V8 runtime, so concurrency is bounded by how many isolates fit inside that budget.

Some of that ceiling is there on purpose. We keep strict limits on what any one deployment can consume, because FunRun is shared. One app stuck in an infinite while loop shouldn't be able to starve everyone else on the machine, and noisy neighbors are a lot easier to prevent than to apologize for.

Module caching and routing

Your code doesn't get re-downloaded from S3 on every request. FunRun instances keep a local cache of modules and indexes, and we lean on that hard.

Routing is built around it. We send a given deployment's traffic to the same FunRun instances whenever we can, purely to keep that cache warm, and we only spread a deployment across more instances once its load genuinely demands it. Cache hits are the whole reason that routing rule exists.

Which raises the obvious question about pushes. When you deploy a new version of a function, what stops a FunRun instance from happily serving the old cached copy? There's a version attached to your code that keeps the two straight. Exactly where that version gets resolved, sent along with the request or looked up on the FunRun side, is the kind of detail I'd want to go read the code before answering confidently. Cache invalidation earned its reputation as one of computing's perennially hard problems, and this is a small reminder of why.

Conductor and multi-tenancy

FunRun started out doing only execution, meaning the queries, mutations, and actions themselves. Moving code analysis into it happened about a year ago, and that move was tied directly to the shift toward multi-tenant hosting.

Here's the connection. Before Conductor, every deployment ran V8 inside its own process to do that analysis, and V8 is memory-hungry. That memory was the single biggest thing standing between us and packing more deployments onto a machine. Life in that world was rough in a second way too: with every deployment as its own process, rolling out one of our own internal updates across the fleet took four, five, sometimes six hours.

Conductor fixed the density problem by hosting many deployments inside one multi-tenant container, and ripping V8 out of the deployment process is what let us pack them in properly. Early Conductor instances ran around 100 deployments each. Today we run about 3,000 per instance, our fleet-wide pushes finish in a fraction of the time they used to, and the cost per idle deployment dropped enough to fund the free plan.

Protecting the system: the mutation semaphore

Nothing about FunRun stops it from scaling forever, given enough machines. The real ceiling shows up well before that, and we put it there ourselves.

Say a client fires off 100,000 mutations at once. Nobody meant to run that. It's a bug in somebody's code. Passing all of them straight through to FunRun would put the whole service at risk for every other deployment sharing it, so we don't. A mutation semaphore caps each deployment at somewhere around 256 concurrent mutations. The first 256 go through and the rest wait their turn as the earlier ones come back.

Call it what it is, which is DDoS protection pointed inward. There's a second reason for it too. Every queued request holds memory while it waits, its arguments and its state, so an unbounded queue would eventually run the deployment out of memory even if FunRun never blinked.

What's still unsolved

The thing that's top of mind for me right now is FunRun utilization, and it comes straight out of that cache-affinity routing rule from earlier.

A handful of deployments carry far more load than everyone else, and cache hits keep sending them back to the same FunRun instances. So those instances run hot, sitting at maybe 12 of their 16 cores at steady state. Meanwhile plenty of other FunRun instances are using one or two cores out of 16. We're paying for all of them.

What I want is boring: a CPU graph where the lines sit in the middle, rather than a cluster pinned near the top and another cluster scraping along the bottom. Getting there means better load balancing between deployments and FunRun instances. A shared or global cache, something along the lines of memcached rather than per-instance memory, would give routing much more freedom, and we've thought about it. Cache locality is only part of the problem though, and it's the CPU distribution we actually want to fix.

This one is genuinely open. If we already knew the answer, we'd have shipped it.

We're a small team

All of this, FunRun and Conductor both, came out of a very small group of people. FunRun was me and about one and a half other engineers. Conductor was a three-person project. Convex's entire engineering team is around 11 people.

That's part of why we talk about this stuff publicly. We're fighting growth and improving the system at the same time, with a lot more scaling work ahead of us than hands to do it. If the open problems in here sound like your idea of a good time, we're hiring at convex.dev/jobs

All gas, no breakages

Convex is the reactive backend platform that keeps up with you and your agents. Database, functions, workflow, sync, search, file storage, and more. All TypeScript, zero glue.

Get started