The full, unabridged story on how the Convex internals work.

How Convex Works

Convex has an [open source version](https://github.com/get-convex/convex-backend) of the backend. We can use this to do local development of an app built on Convex. With local development, you can develop with lower latency, faster push times, and when offline.

## Download the Convex backend binary

Grab the latest binary from [this link](https://github.com/get-convex/convex-backend/releases). You’ll have to pick the appropriate binary for your operating system and CPU architecture. If your architecture isn’t available precompiled, you’ll have to build from source, which we have a guide for [here](https://stack.convex.dev/building-the-oss-backend).

Unzip and run the binary from your terminal

```sh
./convex-local-backend
```

You will know it’s ready when you see a log line like this one:

```sh
2024-03-20T00:06:44.522791Z  INFO common::http: Listening on http://0.0.0.0:3210
```

Keep the backend running in the background and open up a new terminal for the next steps.

# Run `convex dev`

To hit the local OSS backend, `convex dev` needs to know the url and admin key:

```sh
npx convex dev
  --admin-key 0135d8598650f8f5cb0f30c34ec2e2bb62793bc28717c8eb6fb577996d50be5f4281b59181095065c5d0f86a2c31ddbe9b597ec62b47ded69782cd
  --url "http://127.0.0.1:3210"
```

**Note**: This is the default OSS backend admin key. It's ok to check it into source control (and post here 😅).

As a convenience, you can copy the [Justfile](https://github.com/get-convex/convex-backend/blob/main/Justfile) from the open-source repo into your app:

```sh
curl -O https://raw.githubusercontent.com/get-convex/convex-backend/main/Justfile
```

The `Justfile` will add the default URL and admin key to all `just convex` commands. If you don't have `just` installed, instructions [are here](https://github.com/casey/just) - for example `cargo install just` or `brew install just`. In my example, I have an app called `convex-chess`:

```sh
cd convex-chess
npm i
just convex dev
```

Here, we use `just convex dev` instead of `npx convex dev` but any command will work, such as `just convex env`. Note that it is using the version of the Convex CLI specified in the `package.json` of your project. 

Keep this running in the background and open up a new terminal for the next steps.

# Run your frontend

Run the appropriate frontend dev command for your application:

```sh
next dev
```

Voila! You’re up and running with local development. Your application will identify the local URL from the `.env.local` .

# Notes and Limitations

- Upgrading to a newer version of `local-backend` may require wiping your local database. You can do this with `rm convex_local_backend.sqlite3`
- You can run other CLI commands (eg `just convex run`  and `just convex env` ) just as you would via the cloud development flow.

# Related

If you like this, you may also want to check out our other OSS articles

- [https://stack.convex.dev/testing-with-local-oss-backend](https://stack.convex.dev/testing-with-local-oss-backend)
- [https://news.convex.dev/convex-goes-open-source/](https://news.convex.dev/convex-goes-open-source/)

Read this article to learn how you can develop against our open-source backend using our packaged binaries.

Developing with the Open-Source backend

Imagine you're a database best practices guru.

You know every feature and you've got everything *locked down*. You did the `ENABLE ROW LEVEL SECURITY` dance. You’ve used `REFERENCES` for confident foreign key constraints, your indexes were `CREATE`ed `UNIQUE`. Hell, you’ve even used some `CHECK` constraints to make sure field values are mostly sane!

Then a teammate pulls a DevOops, executes the wrong `sudo` command, and summarily wipes out your entire database—protections and all. Or someone commits a malicious change to your codebase and streams your entire userbase to an IRC server somewhere. Oops! Did you forget to `ENABLE DONT LEAK PII`?

# Safety in a world of everything-is-code

In software systems, risks come from *everywhere*. And for most of the Internet’s history, we’ve had various ways to armor against each and every one of them. Permission systems in *nix. ACLs in LDAP. IAM in AWS. And of course databases. Traditional databases encode a varied, somewhat useful, but still far too limited set of basic checks for your data and its safe access.

The inconsistent ways to specify rules for each of these systems are a product of them being developed by and for a particular group of people with particular technical norms. Groups such as DBAs, sysadmins, cloud architects, SREs, and network admins.

But as time has gone on, as [software has eaten the world](https://a16z.com/why-software-is-eating-the-world/), one by one those specialized rule sets have been replaced by *code*. Code with the same expressive power and management practices as the software we build. Instead of sometimes clicking things in an AWS UI, sometimes sending commands to a network switch over a serial port, and yet at other times running `ALTER TABLE` on a Postgres server somewhere–let’s make everything pull requests. Change management, auditing, rollbacks, tests, are all achieved with familiar software-esque methods.

So we have [software-defined networking](https://en.wikipedia.org/wiki/Software-defined_networking), [software-defined infrastructure](https://en.wikipedia.org/wiki/Infrastructure_as_code), and I’m sure lots of other software-defined stuff I don’t know about.

In that spirit, consider <QuickConvex /> the first software-defined database.

# Underpowered overpowered Convex

Out of the box, Convex doesn’t have row-level security, uniqueness constraints, or much of anything. And yet teams using Convex have had these features in their projects since even before Convex was 1.0. What's more, they’ve benefited from the same level of transactional integrity they’d have with an industrial-grade, decades-old, built-in solutions like those in Postgres.

This is possible because when you’re a Convex developer, your code is *in* the database.

<div style={{maxWidth: "480px", margin: "auto"}}>
![](https://cdn.sanity.io/images/ts10onj4/production/d65d6df44aa76030ee41404ee5fe6df6824ab106-716x500.jpg)
</div>

In essence, Convex is a new kind of “virtual machine” for persistent transactional (OLTP) computing. Convex query and mutation functions are its machine code. If you’re a Convex user, you’re a database engineer, and you’ve been writing database plugins all along.

For example, let’s say you want to ensure every database access goes through an authorization check. You want to encapsulate access to your tables and utilize a modular, consistent authorization scheme. No problem–just write a function that wraps your Convex tables and exposes methods that always authenticate before passing through table data. Encapsulation, abstraction, and modularity using regular old code. It’s a software-defined database.

# Examining the rock and the hard place

Are we just being lazy? Are we making you build our database for us? Surely, these tried-and-true built-in security/integrity features of traditional databases have been good enough for every growing software company thus far.

Actually… they haven’t. It might surprise you to learn that most sites with scale don’t use *any* of this stuff. They don’t use foreign key constraints. They don’t rely on in-database row-based permission systems. They often don’t even use the database's actual type system!

Why not?

- Performance. There’s nothing magical here. The database incurs extra reads and writes to provide these capabilities. Sometimes that makes sense to do, but at other times it doesn’t. Opting into these tradeoffs in a granular way is easier to manage in the application’s code.
- Sharding/scaling. Many relational database management system capabilities were designed for simpler times when the database primary could imagine it was a single machine. The first time you have to introduce horizontal scaling, you need to kiss a lot of this stuff goodbye!
- The semantics are too limited. For example, when it comes to valid field values or authorization policies, we want to express requirements so specific we might as well use a full programming language. So, yeah…
- Consolidation of all the rules. Since you end up encoding *some* of your rules in the application due to the aforementioned limitations, eventually you’d rather encode all of it there so you don’t go crazy reasoning about the cumulative behavior.

So as projects scale, more and more of these needs get pushed “up” a layer into the application. The bigger the system gets, the less it’s able to use the built-in database features. However building these capabilities into traditional applications is problematic, too. You’re further from the database so everything is slower, and it’s much more difficult to maintain transactional consistency with the rest of your data.

So which should we pick? The expressive power of programming languages, or the robustness of enforcing these rules in the transactional core of the database?

<div style={{width: "300px", margin:"auto"}}>
![](https://cdn.sanity.io/images/ts10onj4/production/d03b6916f9b85b9adbc9c013bc2fed78176d6436-300x397.jpg)
</div>

By removing the boundary between the application and the database, Convex developers have a database just as expressive and powerful as code without the limitations of living in the “external” calling application.

# Simple primitives, functional composition

Does this mean the Convex team thinks the platform is finished? Hell no!

But we think we’re almost done with the foundational layer of Convex, this new persistent transactional virtual machine and its machine code. We know these foundational building blocks are the part of Convex that our users *cannot* change and cannot opt out of. So we strive to keep that core small but elegant and composable. We want you to be able to easily augment Convex’s power in specific ways that accelerate your team and project using your tool of choice: code.

The good news is this isn’t just theoretical. One trailblazer of this software-defined database methodology has been Ian Macartney, an engineer on the Convex DevX team. Throughout 2023, Ian created modular solutions for [authentication](https://stack.convex.dev/wrappers-as-middleware-authentication), [rich argument validation](https://stack.convex.dev/typescript-zod-function-validation), [presence](https://stack.convex.dev/presence-with-convex), [rate limiting](https://stack.convex.dev/throttling-requests-by-single-flighting), [sessions](https://stack.convex.dev/track-sessions-without-cookies), [background job management](https://stack.convex.dev/background-job-management), [migrations](https://stack.convex.dev/migrating-data-with-mutations), [relational modeling](https://stack.convex.dev/functional-relationships-helpers), etc. 

As the year went on, we discovered that thousands of Convex developers were copying and pasting his code from those articles into all of their codebases. So Ian bundled up all the most useful stuff into an npm package called [convex-helpers](https://www.npmjs.com/package/convex-helpers). Consider, in that repo (currently a mere ~2500 lines of TypeScript), one engineer added 25 years of PostgreSQL features to Convex.

How about a Prisma-style ORM, with cascading deletes, easy joins, and access control rules built in? Convex engineer Michal Srb [made that too](https://labs.convex.dev/convex-ents), in a week or two.

Naturally, we want to double down on this library-oriented, code-oriented functional composition approach to building out the rest of Convex’s higher-level capabilities. And that’s where **Convex Components** come in.

# Introducing Convex Components

Convex Components are a way to formalize a library pattern that allows any Convex project to easily integrate sophisticated solutions to backend problems.

Some goals and properties of Convex Components:

- Make it dead simple to share implementations of ORMs, permission systems, workflow systems, and more.
- Allow these libraries to collaborate in the same transactional window with the application’s code for correctness and consistency.
- Give them their own namespaced and private tables so they can store persistent data as part of their workflows.
- Provide isolation so components can only read and write to your tables in ways you explicitly enable, and so resource usage can be traced appropriately.
- Eventually allow Convex projects to be implemented in other (non-JS/TS) programming languages.
- Speed up project compilation time by only type checking, recompiling, and re-bundling changed components instead of the entire project.
- Provide a path for larger companies to subdivide their internal work across teams.

We’ve been developing the component framework internally for several months. Soon, we’ll be releasing an early version of this framework to iterate on with the community and gather your feedback. We’ll also be releasing a series of Convex-authored components to take on some of the most urgent ergonomic needs.

So… one day will Convex have row-level security? Yes! There will be an open-source, MIT-licensed Convex Component implementing RLS. It will work just as well as any “built-in” solution from traditional databases. And if it doesn’t quite do what you need, fork it and make it your own. It's just TypeScript!

In fact, we’re looking forward to the day when most of the exciting new features on Convex are being written by Convex *users* instead of Convex employees. And with the software-defined database, that day may not be far away.

Which to choose, the expressive power of code, or the robustness of built-in database features? With Convex, you can have both. By eliminating the boundary between the application and the database, Convex provides a uniform and powerful way to model your entire backend data flow and security using plain ol' code.

Convex: The Software-Defined Database

From January to March, Convex hosted the Zero to One Virtual Hackathon with the challenge “Build fullstack apps at light speed with Convex.” The hackathon was completed with over 2400 participants and 79 full-stack apps submitted. After an intense round of judging, the top winner of the Best Overall App was Konrad Hylton, who built LaunchPie. LaunchPie represents a unique approach to improving equity distribution among startup employees. Below is an interview with Konrad on building LaunchPie and his Convex experience.


## **What is the core problem that LaunchPie solves for startups and investors?**

LaunchPie helps founders make informed equity decisions. It provides a framework for fair equity distribution that values each founder's contributions—whether time, expertise, or resources—in a balanced and equitable manner. This not only motivates teams but also provides investors with clear insights into the company's operations and commitment levels.

As startups mature and founders seek a more stable equity structure, LaunchPie facilitates this transition with cap tables and scenario modeling.

In other words, LaunchPie provides clear answers to questions like 'How much equity should a new co-founder receive?' or 'Will my equity dilute after a rights issue?' ensuring founders have the knowledge to make informed decisions.

## **What experiences led you to build LaunchPie?**

As a corporate attorney, I've seen more than my fair share of equity disputes. It's a frequent and ugly problem. Founders generally have good intentions but often fall into the same traps. Having witnessed these conflicts first-hand and experienced similar challenges myself, it's clear to me that much of this is preventable.

After a deep dive into equity management, I began working on LaunchPie. I designed it to preemptively address the root causes of equity-related tensions by providing a clear, automated framework for equity distribution.

## **How did you learn about Convex?**

The first time I heard about Convex was through AI Town. Not long after, I began seeing @YTCodeAntonio and @webdevcody using it in their projects. What stood out was the real-time dashboard, which allowed for live edits and instant updates. It looked like a good way to improve my workflow so I began checking out their docs.

## **What was your developer experience like building LaunchPie on Convex during the Zero to One Hackathon?**

I don’t say this lightly: Convex offers the best developer experience available for a backend as a service today. If I ran into trouble, the support from their Discord was instant and always helpful— though I rarely needed it, given the quality of Convex’s docs. Their docs have a well implemented fuzzy search that grabs relevant matches from their docs, stack blog and existing Discord discussions. This is complimented by their chatbot that had no trouble deciphering my vague, error riddled prompts and provided very useful answers. Regardless of your skill level, it’s impossible to feel lost when building on Convex.

Convex also simplifies many of the more time-consuming parts of backend development. Security, real-time updates, and caching are either ready out of the box or greatly simplified. However, the dashboard truly stands out. It gives you an intuitive GUI that lets you test functions, view logs, edit data, monitor cron jobs, and inspect uploaded files, all in one place. It’s hard to express just how much this streamlines development until you’ve experienced it firsthand.

## **What top Convex features help build LaunchPie?**

If I had to choose the three features from Convex that helped most in building LaunchPie, they would be:

1. Indexes: These allowed me to query documents quickly and efficiently, invaluable for managing a data-intensive application.
2. File Storage: I used this to store and serve important company documents as well as audio files from board meetings.
3. Vector Search: LaunchPie uses a lot of AI. Convex’s built in vector search and ability to define vector indexes made storing and retrieving embeddings pretty easy

## **What advice would you give to developers just learning about Convex to dive in to start building their app?**

First, go through the Quick Start in their docs, then jump into their Stack. It’s full of well-thought-out articles, walkthroughs and perspectives on how to use and think about Convex. Find an article that aligns with your interests and use it as a springboard for your next project.

If you get stuck, use the search feature in their docs or consult their chatbot.

## **What’s next for you and LaunchPie?**

I’m turning LaunchPie into a fully-fledged business. I spent the last few weeks speaking with entrepreneurs to make sure there’s a real demand. I’m also reaching out to potential investors and strategic partners who are interested in getting involved. If you’re interested in joining the waitlist or learning more about LaunchPie, check out [launchpie.com](http://launchpie.com).

Earlier this year we ran a hackathon to find who could use Convex's features the best to build an application from scratch. From 2400 participants, Konrad Hylton pulled it off with Launchpie.

LaunchPie’s Experience Winning the Zero-to-One Hackathon

There has been a lot changing with AI Town - here’s an overdue update to catch you up. 

Background: After launching https://convex.dev/ai-town mere weeks ago as an experiment, we were surprised at how much attention the project gained (over 5k stars on GitHub!). Interest grew in developing it further: for games, personal experimentation, and more. Some of you likely ran across notable clones of it, like [AI Silicon Valley](https://ai-silicon-valley.fly.dev/), [Cat Town](https://cat-town.fly.dev/), or my personal favorite: [Zaranova](https://zaranova.xyz/). We’ve been hard at work improving it since then. Let’s look at some recent changes, and what it means for anyone interested in extending the platform. As a reminder, the repo is open-source, and can be found here:

<GitHubRepo url="https://github.com/a16z-infra/ai-town" />

# Interactive

You can now walk around and interact with the AI citizens! Click to walk around, invite or be invited to chat with nearby citizens. We’ve had some fun internally spreading rumors amongst the citizens and seeing that spread and show up in later conversations with other people.

# Extensible game mechanics

We overhauled the serverless game engine to make it easier to extend. Be on the lookout for an upcoming Stack post on how I added activities, if you’re wanting to add more gameplay elements. Some exciting ones proposed by the community: let the citizens message on Discord or SMS, interact with objects and their environment 🔥, or even trade stonks 💹.

# Easy to mod

Don’t want to code, but want your own AI ant farm, Matrix, or USS Callister? 

Edit the name, identity, and other attributes of an agent (NPC) in the character data before creating your world, or in your [Dashboard](https://dashboard.convex.dev) in the `agents` table once it’s live. Their conversations will immediately start to reflect their new identities. To save this data, you can download a table snapshot from settings, and copy the identities into the seed data later when you want to recreate the world.

There’s also a map editor to make your own levels, with some community members using AI to make the map assets and character sprites. What a time to be alive! 

Want to contribute? Join [this Issue](https://github.com/a16z-infra/ai-town/issues/185) to make agent data easier to sync and share.

<VimeoPlayer video="879687042" />

# Join the party

Bring your wild ideas and busted worlds. The AI Town Discord is in #ai-town-dev in the AI Stack Devs [here](https://discord.gg/F8ApM3JX). Come chat about <QuickConvex />, the backend for AI Town, in our community Discord [here](http://discord.gg/convex). You can find the open source repo here: 

<GitHubRepo url="https://github.com/a16z-infra/ai-town" />

🤫 Keep your eyes out for an upcoming competition for best AI Town clone. May the best civilization win!

It's never been a better time to make your own town of AIs.

AI Town v2

Interactive

Extensible game mechanics

Easy to mod

Join the party