The full, unabridged story on how the Convex internals work.

How Convex Works

Imagine you're a database best practices guru.

You know every feature and you've got everything *locked down*. You did the `ENABLE ROW LEVEL SECURITY` dance. You’ve used `REFERENCES` for confident foreign key constraints, your indexes were `CREATE`ed `UNIQUE`. Hell, you’ve even used some `CHECK` constraints to make sure field values are mostly sane!

Then a teammate pulls a DevOops, executes the wrong `sudo` command, and summarily wipes out your entire database—protections and all. Or someone commits a malicious change to your codebase and streams your entire userbase to an IRC server somewhere. Oops! Did you forget to `ENABLE DONT LEAK PII`?

# Safety in a world of everything-is-code

In software systems, risks come from *everywhere*. And for most of the Internet’s history, we’ve had various ways to armor against each and every one of them. Permission systems in *nix. ACLs in LDAP. IAM in AWS. And of course databases. Traditional databases encode a varied, somewhat useful, but still far too limited set of basic checks for your data and its safe access.

The inconsistent ways to specify rules for each of these systems are a product of them being developed by and for a particular group of people with particular technical norms. Groups such as DBAs, sysadmins, cloud architects, SREs, and network admins.

But as time has gone on, as [software has eaten the world](https://a16z.com/why-software-is-eating-the-world/), one by one those specialized rule sets have been replaced by *code*. Code with the same expressive power and management practices as the software we build. Instead of sometimes clicking things in an AWS UI, sometimes sending commands to a network switch over a serial port, and yet at other times running `ALTER TABLE` on a Postgres server somewhere–let’s make everything pull requests. Change management, auditing, rollbacks, tests, are all achieved with familiar software-esque methods.

So we have [software-defined networking](https://en.wikipedia.org/wiki/Software-defined_networking), [software-defined infrastructure](https://en.wikipedia.org/wiki/Infrastructure_as_code), and I’m sure lots of other software-defined stuff I don’t know about.

In that spirit, consider <QuickConvex /> the first software-defined database.

# Underpowered overpowered Convex

Out of the box, Convex doesn’t have row-level security, uniqueness constraints, or much of anything. And yet teams using Convex have had these features in their projects since even before Convex was 1.0. What's more, they’ve benefited from the same level of transactional integrity they’d have with an industrial-grade, decades-old, built-in solutions like those in Postgres.

This is possible because when you’re a Convex developer, your code is *in* the database.

<div style={{maxWidth: "480px", margin: "auto"}}>
![](https://cdn.sanity.io/images/ts10onj4/production/d65d6df44aa76030ee41404ee5fe6df6824ab106-716x500.jpg)
</div>

In essence, Convex is a new kind of “virtual machine” for persistent transactional (OLTP) computing. Convex query and mutation functions are its machine code. If you’re a Convex user, you’re a database engineer, and you’ve been writing database plugins all along.

For example, let’s say you want to ensure every database access goes through an authorization check. You want to encapsulate access to your tables and utilize a modular, consistent authorization scheme. No problem–just write a function that wraps your Convex tables and exposes methods that always authenticate before passing through table data. Encapsulation, abstraction, and modularity using regular old code. It’s a software-defined database.

# Examining the rock and the hard place

Are we just being lazy? Are we making you build our database for us? Surely, these tried-and-true built-in security/integrity features of traditional databases have been good enough for every growing software company thus far.

Actually… they haven’t. It might surprise you to learn that most sites with scale don’t use *any* of this stuff. They don’t use foreign key constraints. They don’t rely on in-database row-based permission systems. They often don’t even use the database's actual type system!

Why not?

- Performance. There’s nothing magical here. The database incurs extra reads and writes to provide these capabilities. Sometimes that makes sense to do, but at other times it doesn’t. Opting into these tradeoffs in a granular way is easier to manage in the application’s code.
- Sharding/scaling. Many relational database management system capabilities were designed for simpler times when the database primary could imagine it was a single machine. The first time you have to introduce horizontal scaling, you need to kiss a lot of this stuff goodbye!
- The semantics are too limited. For example, when it comes to valid field values or authorization policies, we want to express requirements so specific we might as well use a full programming language. So, yeah…
- Consolidation of all the rules. Since you end up encoding *some* of your rules in the application due to the aforementioned limitations, eventually you’d rather encode all of it there so you don’t go crazy reasoning about the cumulative behavior.

So as projects scale, more and more of these needs get pushed “up” a layer into the application. The bigger the system gets, the less it’s able to use the built-in database features. However building these capabilities into traditional applications is problematic, too. You’re further from the database so everything is slower, and it’s much more difficult to maintain transactional consistency with the rest of your data.

So which should we pick? The expressive power of programming languages, or the robustness of enforcing these rules in the transactional core of the database?

<div style={{width: "300px", margin:"auto"}}>
![](https://cdn.sanity.io/images/ts10onj4/production/d03b6916f9b85b9adbc9c013bc2fed78176d6436-300x397.jpg)
</div>

By removing the boundary between the application and the database, Convex developers have a database just as expressive and powerful as code without the limitations of living in the “external” calling application.

# Simple primitives, functional composition

Does this mean the Convex team thinks the platform is finished? Hell no!

But we think we’re almost done with the foundational layer of Convex, this new persistent transactional virtual machine and its machine code. We know these foundational building blocks are the part of Convex that our users *cannot* change and cannot opt out of. So we strive to keep that core small but elegant and composable. We want you to be able to easily augment Convex’s power in specific ways that accelerate your team and project using your tool of choice: code.

The good news is this isn’t just theoretical. One trailblazer of this software-defined database methodology has been Ian Macartney, an engineer on the Convex DevX team. Throughout 2023, Ian created modular solutions for [authentication](https://stack.convex.dev/wrappers-as-middleware-authentication), [rich argument validation](https://stack.convex.dev/typescript-zod-function-validation), [presence](https://stack.convex.dev/presence-with-convex), [rate limiting](https://stack.convex.dev/throttling-requests-by-single-flighting), [sessions](https://stack.convex.dev/track-sessions-without-cookies), [background job management](https://stack.convex.dev/background-job-management), [migrations](https://stack.convex.dev/migrating-data-with-mutations), [relational modeling](https://stack.convex.dev/functional-relationships-helpers), etc. 

As the year went on, we discovered that thousands of Convex developers were copying and pasting his code from those articles into all of their codebases. So Ian bundled up all the most useful stuff into an npm package called [convex-helpers](https://www.npmjs.com/package/convex-helpers). Consider, in that repo (currently a mere ~2500 lines of TypeScript), one engineer added 25 years of PostgreSQL features to Convex.

How about a Prisma-style ORM, with cascading deletes, easy joins, and access control rules built in? Convex engineer Michal Srb [made that too](https://labs.convex.dev/convex-ents), in a week or two.

Naturally, we want to double down on this library-oriented, code-oriented functional composition approach to building out the rest of Convex’s higher-level capabilities. And that’s where **Convex Components** come in.

# Introducing Convex Components

Convex Components are a way to formalize a library pattern that allows any Convex project to easily integrate sophisticated solutions to backend problems.

Some goals and properties of Convex Components:

- Make it dead simple to share implementations of ORMs, permission systems, workflow systems, and more.
- Allow these libraries to collaborate in the same transactional window with the application’s code for correctness and consistency.
- Give them their own namespaced and private tables so they can store persistent data as part of their workflows.
- Provide isolation so components can only read and write to your tables in ways you explicitly enable, and so resource usage can be traced appropriately.
- Eventually allow Convex projects to be implemented in other (non-JS/TS) programming languages.
- Speed up project compilation time by only type checking, recompiling, and re-bundling changed components instead of the entire project.
- Provide a path for larger companies to subdivide their internal work across teams.

We’ve been developing the component framework internally for several months. Soon, we’ll be releasing an early version of this framework to iterate on with the community and gather your feedback. We’ll also be releasing a series of Convex-authored components to take on some of the most urgent ergonomic needs.

So… one day will Convex have row-level security? Yes! There will be an open-source, MIT-licensed Convex Component implementing RLS. It will work just as well as any “built-in” solution from traditional databases. And if it doesn’t quite do what you need, fork it and make it your own. It's just TypeScript!

In fact, we’re looking forward to the day when most of the exciting new features on Convex are being written by Convex *users* instead of Convex employees. And with the software-defined database, that day may not be far away.

Which to choose, the expressive power of code, or the robustness of built-in database features? With Convex, you can have both. By eliminating the boundary between the application and the database, Convex provides a uniform and powerful way to model your entire backend data flow and security using plain ol' code.

Convex: The Software-Defined Database

Convex provides strong guarantees so developers usually don’t have to reason about retries or inconsistencies. Queries and mutations execute as database transactions that are automatically retried in a fully-consistent manner. This is not the case for Convex actions however, which execute non-transactionally and with best-effort execution guarantees.

Actions provide an escape hatch from the deterministic world of queries and mutations. You could run a several-minute-long action that talks to a bunch of third-party services and externalizes database state. If one of these actions were to fail it wouldn’t be safe for Convex just to automatically retry it - perhaps your action does something like posting a tweet (an X?) that you wouldn’t want to happen twice if the action is retried.

Many times you know that your action really *is* safe to retry though. In these cases you’re in luck because you can use Convex scheduling to automatically retry a failed action.

# Just gimme the code

If you want an out-of-the-box solution there is a Convex helper ready to go:
https://www.npmjs.com/package/convex-helpers#action-retries

You can directly import this helper and use it to retry an unreliable action until it succeds:

```tsx
 // in convex/utils.ts
 import { makeActionRetrier } from "convex-helpers/server/retries";

 export const { runWithRetries, retry } = makeActionRetrier("utils:retry");

 // in a mutation or action
 export const myMutation = mutation({
   args: {...},
   handler: async (ctx, args) => {
     //...
     await runWithRetries(ctx, internal.myModule.myAction, { arg1: 123 });
   }
 });
```

# How it works

<aside>
🧑‍💻 Follow along with the code in https://github.com/JamesCowling/convex-action-retrier/blob/main/convex/retrier.ts. This repository is a simpler version that predates the convex-helper module.
</aside>

Retries are a great use case to leverage Convex scheduling. Along the way we’ll also learn about using `db.system` to look up system table information and the `makeFunctionReference` helper to generate a reference to a function from its name as a string.

We can schedule a function from either a mutation or an action but since our goal is to build a reliable wrapper around an unreliable action we’re going to use mutations to do this. We’ll start with a mutation called `runAction` that takes in the name of an action and its args, then does some magic to make sure that action gets retried until it succeeds:

```tsx
export const runAction = internalMutation({
  args: {
    action: v.string(),
    actionArgs: v.any(),
  },
  handler: async (ctx, { action, actionArgs }) => {
		...
```

We make this an `internalMutation` for safety since it exposes access to any action via its name. We can use a public mutation to call it for a specific action. The version in `convex-helpers` provides type-safety, unlike the simple example here.

## The algorithm

We’ll use the following logic to retry an action until it succeeds:
![The backoff algorithm](https://cdn.sanity.io/images/ts10onj4/production/4523ce713f9881a60b23fd114bb610efce03fd2c-1186x1333.png)

In a production app it’s also a good idea just to give up eventually in case the action will never succeed. [The linked source code](https://github.com/JamesCowling/convex-action-retrier/blob/main/convex/retrier.ts) also includes a configurable `maxFailures` check to do so.

## Execute action

The first step of our algorithm is to actually execute the action. You’re probably familiar with triggering an action from a mutation using a scheduling command like:

```tsx
await ctx.scheduler.runAfter(0, api.example.unreliableAction, actionArgs);
```

It’s nice to have the type-safety provided by function references like `api.example.unreliableAction` but in this case we only have the name of the action as a string. Fortunately we can use the `makeFunctionReference` helper that will generate a function reference from its type (query/action/mutation) and name:

```tsx
// const action = "example:unreliableAction"
await ctx.scheduler.runAfter(
	0,
	makeFunctionReference<"action">(action),
	actionArgs
);
```

Be careful with code like this to ensure you don't expose functions that you don't intend to be public.

The last step is to record the job id for the scheduled function so we can check its status later:

```tsx
const job = await ctx.scheduler.runAfter(
	...
```

## Check job status

The job id we just recorded is actually the `Id` of a document in the Convex [system table](https://docs.convex.dev/database/advanced/system-tables) `_scheduled_functions`. There are a growing set of system tables in Convex that allow you to query internal system state from right within queries and mutations. You can read a system table with the `ctx.db.system.get()` command just like reading a regular table with `ctx.db.get()`. In particular we want to look up the `state.kind` field in the system table document for the job we just scheduled:

```tsx
const status = await ctx.db.system.get(job);
if (!status) {
	throw new Error(`Job ${job} not found`);
}
switch (status.state.kind) {
	case "pending":
	case "inProgress":
		...
		break;
	case "failed":
		...
		break;
	case "success":
		...
		break;
	case "canceled":
	  ...
		break;
}
```

Now it’s just a matter of deciding when to retry the action and how long to wait before doing so.

## Back off

You might find it odd that there are two different exponential backoffs in our flowchart above. If you’re not familiar with exponential backoff it basically just means waiting longer every time we retry. It’s a great tool when you don’t know exactly how long to wait but you also don’t want to retry thousands of times in a tight loop.

The first backoff is used to figure out when the action has finished. The wrapper doesn’t know how long the unreliable action is meant to take so it wouldn’t make sense to keep checking every millisecond if it usually takes 5 minutes. Instead we wait 10ms, then 20ms, 40ms, 80ms, 160ms, etc.

The second backoff determines how long to wait before retrying a function. Oftentimes an action will fail because a third-party service is temporarily down. We would want to be hammering that poor service constantly if it’s already drowning under load. Instead we wait 10ms, then 20, then… you get the picture.

The `convex-action-retrier` library allows configuring the specific parameters of the exponential backoff but the nice thing about exponential backoff is that the default parameters should be fine for almost everyone.

# Ready for actions

Now you can implement reliable actions by retrying them until they succeed, but hopefully now you can also build relatively sophisticated workflows by chaining mutations and scheduling. The mental model to maintain is that queries, mutations and scheduling are reliable and transactional but restrictive, whereas actions are the wild west where you trade transactional guarantees for the flexibility to talk to the outside world. Fortunately these components all work together seamlessly so you can have your cake and eat it too.

Hooray!

Learn how to automatically retry actions in Convex while also learning a little about scheduling, system tables, and function references.

Automatically Retry Actions

Operational maturity is the umbrella term I like when thinking about scalability, security, observability, and other important aspects of a serious product. Similar to scaling, it isn’t a destination, but a continual process. There is no one checklist that an app goes through once. Rather, you should understand where you are in the journey, what the biggest risks are, and what incremental steps are available. 

This post will cover various areas of operational maturity, and link to posts outlining steps to take as your app develops. The advice will specifically reference <QuickConvex /> but the concepts are generally applicable.

I’ve worked on teams and products all along this spectrum, from launching a new GCP product for startups like Clockwork, to greenfield products for established companies like The New York Times, to managing the Dropbox infrastructure responsible for file previews—involving hundreds of servers in multiple data centers handling millions of image requests per day targeting three nines of availability. I can assure you, all of these did not—and should not—have the same level of operational maturity.

# 1. Prototyping: YOLO

When you’re first building your app or bootstrapping your company, you want to move as quickly as possible. If you’re spending a lot of time thinking about load balancing, connection pooling, data architecture for future-proof sharding, or Kubernetes, then you likely aren’t thinking enough about the human problem you’re trying to solve. For reassurance, I have a heuristic that for every order of magnitude increase in users, an app often gets re-architected or re-written at some layer. Your database schema in your first commit to your git repository does not need to be the data model you launch with. Don’t let perfect be the enemy of good.

### [Click here for tips on getting to an MVP fast](https://stack.convex.dev/yolo-fast-mvp)

Tips include: version control, liberal logging, interactive database queries, auto-reload, auto-deploys, keeping your stack simple, snapshotting data, seed scripts, deferring auth, loose schemas, manual migrations, and more.

# 2. Observing your app

When your app is running, observability allows you to see what is happening, how your product is being used, what is going wrong, and help you debug the “why” behind it all. It is a critical piece of running an app in production, where you don’t have debug access to all of the devices interacting with your software. 

### [Learn more about observing your app in production](https://stack.convex.dev/observability-in-production)

You can start with simple logs, and incorporate dedicated tools over time like Axiom, Sentry, Plausible, PagerDuty, Databricks, and more.

# 3. Testing for peace of mind

From a pragmatic standpoint, testing allows you to validate behavior, catch regressions in performance or functionality, and ultimately give you peace of mind. When you have high confidence in your testing, you will feel confident shipping more frequently.

### [Learn more about testing patterns](https://stack.convex.dev/testing-patterns)

From end-to-end tests to unit tests, from manual to automated strategies, there are a lot of options to choose from when deciding what to up-level next. Often-overlooked aspects of testing are how you test subjective changes in production, and testing your app from outside of your own ecosystem. The latter helps to catch issues with hard-to-test parts of your stack like networking and configurations that only exist in production.

# 4. Protecting your app from yourself

Even if your code is well tested, you can still make mistakes in how you interact with the powerful tools at your disposal. The source of many major internet outages have been from someone mis-typing a command, for instance running a destructive—like deleting a table—in production when they meant to run it against a development instance. Over time, you’ll need to invest in safer processes around changing code, configuration, and data in production. 

Some areas of investment include:

**Deployments:** push-time checks for environment variable definitions, checking for accidental deletion of large indexes, isolating your production deployment from staging and development workflows, and avoiding breaking and inefficient schema changes.

**Migrations**: codifying mutations in code, verifying them against seed data, validating a dry run, and opt-in automation.

**Scoped data changes:** authenticated, authorized, audit-logged changes to production data through dedicated admin interfaces. 

# 5. Hardening your app

Your app needs protection from more than your own mistakes. When you launch to production, you’ll need to consider how clients might misbehave. A backend needs to protect against bad input, or requests that try to access or modify other users’ data. As your customers start to rely on your site, you’ll need to refine your authentication and authorization story.

Simple steps include: 

- [schema validation](https://docs.convex.dev/database/schemas) for data.[^2] 
- [argument validation](https://docs.convex.dev/functions/args-validation) for endpoints. 
- [Internal functions](https://docs.convex.dev/functions/internal-functions) for calling or scheduling functions from other server functions.
- [Authentication in functions](https://docs.convex.dev/auth/functions-auth) to identify users instead of passing up user identifiers.
- [Standardized authentication patterns](https://stack.convex.dev/custom-functions) for public functions with [lints to enforce usage](https://stack.convex.dev/eslint-setup).
- [Environment variables](https://docs.convex.dev/production/environment-variables) for secrets.
- [Shared secrets for cross-server requests](https://stack.convex.dev/custom-functions#consuming-a-function-argument-for-basic-api-key-auth).
- [Authorizing data access based on the user](https://stack.convex.dev/row-level-security).

[^2]: You can also use [Zod](https://stack.convex.dev/typescript-zod-function-validation) for finer-grained runtime validation.

# 6. Scaling

As your app grows from dozens to thousands to millions of users, the performance and reliability of your app become more important. This can include considerations for organic growth such as: 

- [Optimizing your database queries](https://stack.convex.dev/queries-that-scale) to avoid scanning too many documents.
- [Retrying functions](https://stack.convex.dev/retry-actions) that rely on unreliable services.
- [Dynamic client-side throttling](https://stack.convex.dev/throttling-requests-by-single-flighting)
- Using [scheduled functions](https://docs.convex.dev/scheduling/scheduled-functions) as a work queue when you care more about throughput than latency.
- Rate limiting expensive requests (such as requests to LLMs) per user.
- [Managing the state of asynchronous workloads](https://stack.convex.dev/background-job-management).
- Load testing your app to stay ahead of your users’ growth.

# Summary

Operational maturity is an ongoing process that covers a wide range of topics. We’ve touched on many ways to level up your app, but this list is neither exhaustive nor essential. The important decisions to make are:

- Where are your gaps?
- What is worth investing in next?
- When is the right time to take the next step and re-evaluate?

We’d love to hear from you in our [community Discord](https://convex.dev/community).

And if this has gotten you interested in how we think about the future of product development here at Convex, check out this video:

<Video videoId="Xjud1weG4z8" />

This post will cover various aspects of operational maturity, and steps to take as your app grows up.

Operational maturity for production

[Convex 0.5.0](https://blog.convex.dev/announcing-convex-0-5-0/) introduces built-in support for *actions* — arbitrary lambda functions that run in a Node.js environment on the Convex cloud. We use the term *action* to differentiate these functions from Convex *mutation* and *query* functions, which enable [transactional writes and dynamic subscriptions that update whenever data changes](https://www.youtube.com/watch?v=AdrO6IvHjRQ).

Queries and mutations are the core building blocks in Convex, but they’re required to be deterministic and free of side-effects. We leverage this determinism to efficiently schedule transactions, dynamically track changes to data, and automatically cache and distribute query results.

Actions are Convex’s first foray into executing code that has side-effects or isn’t purely oriented around reading or writing data. Data is the center of your application, and Convex has a custom-built and highly scalable database at its core, but Convex is not just a database. Convex is a platform designed to make backend engineering a thing of the past. We have a vision for the future where the vast majority of developers focus on building application code and leverage a small convenient set of abstractions that obviate the need for thinking about storage, execution, sync, queueing, or workflow.

There are a bewildering array of tools and technologies in the Infrastructure-as-a-Service landscape. These tools eliminate inconveniences but [they don’t make problems go away](https://blog.convex.dev/missing-link/)—developers still end up having to compose these tools into a platform that runs their application. Convex isn’t a bag of tools. We have a lot of features coming soon, but these aren’t  standalone building blocks that introduce another component to an already complex infrastructure. All features in Convex are designed to work together seamlessly: tying actions, mutations, queries, subscriptions and end-to-end reactivity together without developers dealing with glue code or data synchronization.

We think you’re going to like what’s coming soon but there’s a very good chance that Convex already meets your needs. [Give it a try](https://convex.dev) and say farewell to thoughts about infra overhead and data management.

Convex 0.5.0 introduces built-in support for actions — arbitrary lambda functions that run in a Node.js environment on the Convex cloud. We use the term action to differentiate these functions from Convex mutation and query functions, which enable transactional writes and dynamic subscriptions that update whenever data changes.

Ready for Actions