Jamie Turner

a year ago

Searching for Sanity

While using one of your favorite apps, have you ever run into something like this?

It’s pretty reasonable to expect that once you add a document to your Notion workspace, the search box will find it! What’s going on here?

Unsavory search secrets

You may be surprised to learn that there’s nothing particularly unusual about what Notion is doing. In most systems, backend search architectures are structured just so:

In this common design, the system that stores your records is separate from the system that indexes them for search. Specialized systems are chained together, such as MySQL for the database and Elasticsearch for full-text search.

So there is inherently a delay in the pipeline that connects storage and search, and your users might experience a temporary inconsistency–a stored document not yet searchable, just as we saw with Notion. This inconsistency is often only a few seconds, but sometimes can be minutes or hours.

While the ease of leveraging several ready-to-go systems for each task may feel like a pragmatic decision for a software team, it really sucks that it “leaks” internal architectural choices in the form of a confusing user experience.

Fancy concepts, simple expectations

At Convex, we love deep infrastructure concepts like strong consistency and ACID. But not because distributed systems papers are riveting beach reads. Instead, because we believe the mark of a great platform is facilitating the easy creation of software that Just Works. Convex projects should automatically do what the developer and user expect.

And that’s why Convex’s search is transactional: search indexes in Convex guarantee that the very moment a document is committed to the database, that document shows up in search results. User expectations met!

And while we’re at it, Convex developers expect that everything in our platform is seamlessly reactive. So we made search reactive, too.

What “just works” looks like

Let’s use Convex to search through a sample of Wikipedia articles!

As a prerequisite, we retrofitted the Convex tutorial chat app by loading 100,000 random Wikipedia articles with the command line import tool. We assigned the page body to the body field and the title to the author field of the messages table.

Time to get searching! First, we need to create the search index using Convex’s schema definition. Let’s create an index called “search_body” that is indexing the body field. We’ll do this by using the searchIndex method on the schema object:

import { defineSchema, defineTable } from "convex/schema";
import { v } from "convex/values";

export default defineSchema({
  messages: defineTable({
    body: v.string(),
    author: v.string(),
  }).searchIndex("search_body", {
    searchField: "body",
  }),
});

Then, we can easily use this search index in our app’s query functions by calling our query’s withSearchIndex method.

export default query(async ({ db }, { bodyQuery }) => {
  const results = await db
    .query("messages")
    .withSearchIndex("search_body", q => q.search("body", bodyQuery))
    .take(5);
  return results;
});

Finally, as usual in React + Convex apps, we’ll leverage the Convex library’s useQuery hook to attach this backend query function to a component rendering a list of results:

This principled take on search is just one of the many ways in which Convex nudges your projects toward the pit of success. So start a new project today, try out transactional search, and come show us what you built!