Jamstack TV

Today, a new wave of innovation is making databases approachable for developers without requiring a specialized skillset, opening the door to a whole new category of ambitious web experiences. In this panel, you’ll hear from the people pushing the boundaries of database technology and learn what’s possible — and what’s coming.

Jamstack Conf 2022 Panel: The Future of Databases

Yet Another Infrastructure Group

A roundtable on the future of streaming data featuring Convex CEO Jamie Turner. Jamie walks through how applications are changing in response to pervasive streaming, the looming problems with "the edge", and how Convex uniquely approaches caching and materialized views.

Hosted by Ian & Alex, with Alana (Readyset), Hojjat (Deltastream), Jamie (Convex).

Convex and the future of streaming infrastructure

Relational databases like MySQL and PostgreSQL have been the backbone of web
apps since the first interactive web apps existed. They power many of the most
popular web apps on the internet and have a proven track record of reliability.

At Convex, we have spent our entire careers working with these technologies and
understanding their strengths and weaknesses. Relational databases are a great
tool for storing and querying data but operate at the wrong level of abstraction
for managing global state in modern web applications.

Fundamentally, we believe that web developers should spend their time building
their products not managing their infrastructure or wrangling with SQL.

## Using the Database

Despite being a standard piece of many web app stacks, relational databases are
surprisingly difficult to use.

### Database Schema

Before you begin using a relational database, you must define your schema. This
involves answering questions like:

- What tables does my app need?
- What columns should my tables have?
- What types are these columns?
- What indexes will I need?

The problem here is that you must make all of these decisions _before you begin
building your app_. Correctly picking a schema means you need to plan your
entire data model before you begin using the database at all. Inevitably you'll
make mistakes and require time-consuming migrations to adjust your schema to
your app's actual needs.

Convex doesn't require you to define your schema upfront. Instead, you can
[immediately start storing and querying documents](https://docs.convex.dev/understanding/convex-fundamentals/storage).
No setup needed.

That being said, Convex also recognizes that schemas are a powerful way to
ensure data consistency in large applications. When you're ready, you can
[define a schema](https://docs.convex.dev/using/schemas) to codify your data modeling decisions.

### Query Language

MySQL and Postgres both use SQL as their query language. Reading and writing
data requires constructing SQL statements. This is additional work that has
nothing to do with building your app. Worse yet, if you make a mistake
constructing your SQL statements you may end up with
[SQL injection](https://owasp.org/www-community/attacks/SQL_Injection) security
bugs or surprisingly slow queries that fail to use the indexes you defined.

To solve this friction, many developers use object-relational mappings (ORMs)
like Prisma to convert between SQL rows and native objects. While ORMs can help
alleviate some of this friction they also commonly produce surprising SQL. Most
large web apps built with ORMs inevitably fall back to writing some queries with
raw SQL to ensure they execute efficiently.

On the other hand, [Convex database queries](https://docs.convex.dev/using/database-queries) are
written entirely in TypeScript and JavaScript. There is no additional language
to learn and no need for a clunky ORM.

Additionally, the Convex database query builder is designed to make the
performance of a query explicit. [Indexed queries](https://docs.convex.dev/using/indexes) both specify
what index they should be executed over and how they should use that index. This
is in sharp contrast to the MySQL or PostgreSQL query planners which try to
automatically decide which index to use and how, sometimes with surprising or
even devastating results.

## Managing the Infrastructure

Managing a relational database is a large undertaking. Traditionally this has
involved:

- Deciding on your database configuration.
- Picking a replication and backup plan.
- Provisioning and managing servers to run the database.

If you make a mistake at any step of this process, at best you'll have a
temporary outage and at worst you'll permanently lose or corrupt customer data.
There's a reason that many companies hire large teams of database admins and
site reliability engineers to manage their databases.

To alleviate these pains, there has been a recent explosion of managed database
products like Amazon RDS, Heroku Postgres, DigitalOcean MySQL, PlanetScale, and
more. These products remove a lot of the burden of getting set up on a
relational database.

Like these managed databases, Convex handles configuring, provisioning, and
replicating your database. Unlike these managed databases, Convex also handles a
lot more. As we'll see below, Convex also eliminates the need to manage separate
backend servers, build a cache layer, or bolt on reactivity.

### Backend Servers

Relational databases are not suited to be directly accessed by client
applications. This is for a few reasons:

- Security: End users should not have unfettered access to read and edit data in
  the database.
- Performance: Loading all the data that the client needs often requires many
  database queries. To build a fast web application all of this data should be
  sent to the client in a single round trip.
- Sharing Business Logic: There should be a central place to put code that is
  shared between your different platforms like web, mobile, and your public API.
- Protocol: SQL databases don't even speak a protocol that is usable from the
  browser.

To solve these concerns, most web architectures place some kind of backend
server between the client and database. Sometimes this is a web server that
renders HTML; other times it's a RESTful HTTP or GraphQL server.

Adding this additional layer creates a new set of complexities. Now you need to
configure, provision, and manage _another_ set of servers to host your backend.
Even if you opted for a managed database, you'll still have to make some hard
decisions about how to architect and scale your backend.

Convex obviates the need for backend servers because it's your backend too! Your
Convex [functions](https://docs.convex.dev/understanding/convex-fundamentals/functions) serve as your
backend's public API. As a developer, you still implement the backend logic that
is important to your app without any of the backend infrastructure management.

### Caching

Improving the performance of web apps built on relational databases is
difficult. You can eke some additional performance out of your database by
tweaking configuration, adding indexes, and using read replicas, but eventually
many teams add on an additional caching layer.

A common architecture is to use an in-memory data store like Redis or Memcached.
This is essentially a second database that you, as the developer, must keep in
sync with your existing MySQL or Postgres instance. This requires all of the
work to manage another database that we've already discussed plus an additional
problem: cache invalidation.

> There are only two hard things in Computer Science: cache invalidation and
> naming things.
>
> — Phil Karlton

Keeping your cache consistent with your relational database is notoriously
difficult and error-prone. Incorrect caching code creates subtle bugs where rare
race conditions and crashes corrupt the cache and send incorrect data to your
users. Tracking down caching bugs can take months of work for experienced
infrastructure engineers.

On Convex caching is completely automatic. Convex
[caches the results of your query functions](https://docs.convex.dev/understanding/convex-fundamentals/functions#caching)
and recomputes the values when the underlying data changes.

### Reactivity

Traditional relational databases are nonreactive. Developers can use SQL to
query the current state of the database, but there is no built-in ability to
subscribe to updates to those query results. Relational databases leave
implementing reactivity up to the developer.

The most common techniques for creating realtime, reactive applications on top
of relational databases are polling or pushing updates.

#### Polling

Polling requires no additional infrastructure; the client simply reruns its
queries at a regular interval. The problem with polling is that it requires
making a tradeoff between data freshness and database load. Frequent polling
will overwhelm your database with requests; polling rarely will leave your users
looking at stale data.

#### Pushing Updates

Scaling your application will normally require moving off of polling and pushing
updates to your clients instead.

There are two broad ways that teams push updates using a relational database:

- Using a relational database that has built-in Pub/Sub capabilities like
  PostgreSQL.
- Deploying a separate Pub/Sub cluster like Apache Kafka or Redis Pub/Sub.

Even if your database supports Pub/Sub that only solves part of the problem. The
database will give you a stream of events, but your application will still be
responsible for connecting those events to the SQL queries and API responses
used in your app and pushing updates over a WebSocket. Comprehensively
implementing realtime updates is a huge undertaking.

Realistically, most web developers never attempt this and instead settle for
showing their users stale, non-reactive data.

#### Convex

Compare this situation to Convex. In the
[`ConvexReactClient`](https://docs.convex.dev/api/classes/react.ConvexReactClient), all queries are
[fully reactive](https://docs.convex.dev/realtime)
and receive realtime updates. That's it.

Convex understands which query functions depend on what data. When the
underlying data changes, Convex reruns the query function and pushes the new
result all the way to the UI.

Reactivity shouldn't require making complex tradeoffs or deploying additional
infrastructure. On Convex, you get correct reactivity automatically.

## Bottom Line

Relational databases are powerful tools for persisting data, but using them to
build interactive web apps requires a lot of work. A web app architecture built
on a relational database is an entire cluster of databases, backends, caching
nodes, and Pub/Sub servers. And even after you invest in all of that
infrastructure you'll still need to explicitly define your database schema
upfront and write cumbersome SQL queries.

At Convex we believe that web developers shouldn't have to worry about any of
this. We can handle the servers, caching, and reactivity and you can focus on
your product.

What are you waiting for? [Get started with Convex!](https://docs.convex.dev/tutorial)


Relational databases are powerful tools for persisting data, but using them to build interactive web apps requires a lot of work. A web app architecture built on a relational database is an entire cluster of databases, backends, caching nodes, and Pub/Sub servers. And even after you invest in all of that infrastructure you'll still need to explicitly define your database schema upfront and write cumbersome SQL queries.

At Convex we believe that web developers shouldn't have to worry about any of this. We can handle the servers, caching, and reactivity and you can focus on your product.

Convex vs. Relational Databases


Convex and Google's Cloud Firestore are both platforms for managing database
state in modern serverless applications. Both platforms store documents and
notify applications of document changes in realtime without requiring
application developers to manage their own infrastructure.

At [Convex](https://convex.dev) we have a lot of respect for Firebase. We're building this product
because we also believe that developers deserve a comprehensive
backend-as-a-service tailored to their needs. Firebase was one of the first
teams to work on this problem and has done a lot of innovation in this space.

That being said, there are some important differences between the two platforms:

- Backend API
- React Integration
- End-to-End Correctness Philosophy


**Note:** *"Firebase" actually refers to two separate products: Firebase Realtime Database
and Cloud Firestore. This discussion focuses on Cloud Firestore because it's the
newer, more scalable Firebase offering.*


# Backend API: Documents or Functions?

The biggest architectural difference between Convex and Cloud Firestore is **how
your application interacts with its data.**

With Cloud Firestore, the client interacts with its data by loading documents
straight from the database. If you're building a chat application and you store
users and messages in Cloud Firestore, your application code will also interact
with those same user and message documents.

In Convex, all access to documents is mediated through
[Convex functions](https://docs.convex.dev/understanding/convex-fundamentals/functions). These functions can
return documents straight from the database, but also can return data derived
from these documents.

Having this additional layer on the server is important for 2 reasons:

1. Avoiding serial request waterfalls.
2. Encapsulating business logic.

## Avoiding Serial Request Waterfalls

_Serial request waterfalls_ happen when an application uses multiple, serial
requests to load its data. This harms page load performance because rendering
your site will now require multiple network round trips to the backend, one
after another, to render the core content.

With Cloud Firestore, a chat app might do one query from the client to load the
messages in the app and then a subsequent query for each message to load the
user that wrote it:

```typescript
// Client code in a Cloud Firestore chat app.
// This loads the messages and users using multiple round trips.
const querySnapshot = await getDocs(collection(db, "messages"));
const userSnapshots = await Promise.all(
  querySnapshot.docs().map(async messageSnapshot => {
    return await getDoc(docSnapshot.data().creator);
  })
);
```

The key thing to notice here is **this is happening client-side**. This code is
doing serial queries to load data over the user's slow network connection. In a
complex application, you will end up with many round trips and a slow,
multi-step loading experience.

Even if you model your Cloud Firestore data using subcollections (which require
your data to be hierarchical) you still can't do a deep query to load a document
and its subcollection documents.

In Convex you have full control over the data you send the client because it's
just the result of JavaScript/TypeScript functions. It's easy to bundle the
messages in a chat app along with the author that wrote each one. Our
[Users and Auth demo app](https://github.com/get-convex/convex-demos/tree/main/users-and-auth)
does exactly that!

```typescript
// Client code in a Convex chat app.
// This loads the messages and users all in a single round trip.
const messages = useQuery(api.messages.list);
```

Here the messages and users are all being loaded on the server inside of the
`list` query function in the `convex/messages.ts` module, so there is only a single round trip from the client.

In Cloud Firestore the only option to avoid request waterfalls is to load your
data in a Cloud Function but that involves sacrificing reactivity and optimistic
updates.

## Encapsulating Business Logic

Another important feature of Convex functions is that they can serve as a
business logic layer. While sending raw documents to the client can be a great
way to get started on an app, as your app grows in complexity you'll want to put
logic on the server. Here are a few common examples of where server-side logic
is necessary:

- Your mobile, web apps, and public API all need to share common logic like
  rendering notifications into a human-readable string.
- Some mutations require custom validation like checking that an email address
  is formatted correctly.
- You want to change your database schema to support more complex queries but
  want to present the same API to your apps to avoid migrating application code
  as well.

Convex functions serve as a natural place to put all of this logic. As a
developer, you can share logic between all of your platforms and ensure that
even malicious users only interact with your data through this layer.

Once again, with Cloud Firestore you _could_ put this code into Cloud Functions
and use them as a business logic layer, but it's messy and requires giving up
some of the platform's other features like reactivity and optimistic updates.

<BuildWithConvex />

# React Integration

**Convex was designed to be used with React.**

All of our [tutorials](https://docs.convex.dev/tutorial) are React apps. Our
[React Hooks](https://docs.convex.dev/client/react) make it dead simple to load and
edit data within React components.

**Cloud Firestore was not designed with React in mind.**

Cloud Firestore has a JavaScript SDK, but it leaves getting your data into your
React components up to the developer. In particular, Cloud Firestore supports
listening for realtime updates with `onSnapshot` but correctly binding those
snapshots to React components is complex enough that many developers give up on
end-to-end reactivity.

# End-to-End Correctness Philosophy

Convex has an almost fanatical obsession with correctness.

We don't just want it to be possible to build a correct app but we want it to be
impossible not to build your app correctly. Developers should
[fall into the pit of success](https://blog.codinghorror.com/falling-into-the-pit-of-success/).

Correctness doesn't stop at the backend API. At Convex we believe correctness
should extend all the way to the users of the application. Users should always
see consistent, up-to-date state. When users modify state, their modifications
should always occur transactionally and exactly once.

Here are a few places where our philosophy around correctness differs from Cloud
Firestore's:

## Reactivity

In the [`ConvexReactClient`](https://docs.convex.dev/api/classes/react.ConvexReactClient), all queries
are
[fully reactive](https://docs.convex.dev/realtime)
and receive realtime updates.

In Cloud Firestore, you _can_ receive realtime updates but it's extra work to
manage the `onUpdate` callbacks (especially in React) so developers normally
choose whether they want to load reactive or non-reactive data on a case-by-case
basis. This leads to applications that show users a mix of fresh and stale data.

## Transactions

In Convex, every mutation is a
[transaction](https://docs.convex.dev/understanding/convex-fundamentals/functions#transactions) running
on a consistent database snapshot.

In Cloud Firestore, you _can_ run transactions but they have complex
requirements. All of the reads in a transaction must precede all of the writes
and you must always remember not to edit application state in the transaction.
Often developers instead opt to use the simpler APIs to update documents and are
surprised down the road when one logical user action was only partially saved to
the database.

# Bottom Line

Convex is designed to both make it easy to get started on the platform and
support you as your app grows.

Our React integration makes it a breeze to start interacting with your global
shared state and loading fully reactive data. As your app grows, you'll
appreciate the ability to push logic into query and mutation functions and keep
your app fast by avoiding serial request waterfalls.

On Cloud Firestore, you'll have to do some acrobatics to integrate with React
and listen for realtime updates. Down the road, you'll find your app full of
slow loading experiences from request waterfalls and your database full of
half-committed mutations.

Convex is designed to help you evolve your application as it grows, not to push
you onto another platform. Soon we'll be adding support for enforcing
[database schemas](https://docs.convex.dev/understanding/deep-dive/schema-philosophy) and running
[database migrations](https://docs.convex.dev/understanding/state#schema-enforcement-and-migrations) so
that you'll be able to seamlessly grow your app on Convex.

What are you waiting for? [Get started with Convex!](https://convex.dev)

On Cloud Firestore, you'll have to do some acrobatics to integrate with React and listen for realtime updates. Down the road, you'll find your app full of slow loading experiences from request waterfalls and your database full of half-committed mutations.

Comparisons