# Are vector databases dead?

This year vector databases have sprung up [like mushrooms](https://twitter.com/jobergum/status/1706928630612275309?s=20) to enable applications to retrieve context based on semantic search. A large portion of these applications have used the retrieved context to augment the ability of large language models (LLMs) in a pattern known as RAG. On November 7th OpenAI released its [Assistants API](https://platform.openai.com/docs/assistants/overview), enabling the implementation of AI chat interfaces with context retrieval without needing a separate message store or vector database. Does this new API make vector databases obsolete?

Yes! And no.

This post concludes a [series](https://stack.convex.dev/ai-chat-using-openai-assistants-api) comparing three different implementations of AI chat backends to help answer this question. Read on for the final verdict.

## AI chat - what’s the point?

In the current age of ChatGPT, GPTs, Bards, Bings, and Claudes it might not be obvious why anyone should be building another AI-powered chat, but there are good reasons:

1. Proximity: People’s behavior is driven by convenience, and talking to an AI right in your product (website, app) might be more convenient than opening a separate chat product.
2. Control: Big LLMs have broad knowledge so they need specific prompting to get an answer about a specific topic. Having to provide this prompt again and again can be onerous for end-users. Your product can set up or augment the prompt automatically.
3. Context: Although LLMs retain a lot of knowledge from training, they might lack specific knowledge, either because it didn’t exist, wasn’t public, or wasn’t important enough at training time. Providing specific context as a part of the prompt leads to better accuracy and less hallucination.

Combining all three together can lead to integrations not possible in the frame of existing general chat interfaces. For example, we could provide the LLM with user-specific information, give it precise instructions, and have it take actions directly in our product. Think: going to Amazon and asking to reorder and double the order of an item you bought last week, and having your shopping cart update to reflect this conversation. We can render product information inline and have it be interactive without the user having to navigate between different websites or apps.

## AI chat case study: docs

This post discusses the tradeoffs and details of implementing this kind of AI system. A basic example of a custom AI-powered chat interface is the AI chatbot on the Convex docs site. You can play with it by clicking the chat bubble button at the top of the [page](https://docs.convex.dev).

![docs screenshot with ai chat bot](https://cdn.sanity.io/images/ts10onj4/production/627ee9c0759b8c43b3f43927cb9af70bf638f416-2000x1125.png)

To help answer the titular question, we implemented this AI chat backend in three ways. A guided walkthrough and the source code for each implementation is linked here:

1. [Using the new OpenAI Assistants API](https://stack.convex.dev/ai-chat-using-openai-assistants-api)
2. [Using Langchain with the Convex message and vector DB](https://stack.convex.dev/ai-chat-using-langchain-and-convex)
3. [Entirely on Convex](https://stack.convex.dev/ai-chat-with-convex-vector-search)

This example demonstrates the three reasons for building a custom AI chat:

- The interface is embedded in the docs site where developers look for technical information about Convex.
- The LLM is prompted to answer questions about Convex.
- The implementation passes the LLM the most relevant documentation pertaining to the question.

Even in this simple use case, the context is not static. The Convex team iterates on the docs and the product constantly, and the AI chat needs to reflect these changes. So when designing the implementations we considered how this contextual information will be updated over time.

## Assistants API

### Pros

The new Assistants API tries to provide a cohesive package, and this does have its strengths for us as the developers of the product:

- We don’t have to worry about the context data splitting (chunking) and search implementations.
- OpenAI’s implementation might improve over time without us having to invest in any effort.
- Other tools besides retrieval, currently function calling and code interpreter, are built-in, and OpenAI might add more tools in the future.

### Cons

Yet these come with tradeoffs, broadly around how much we control and how much we still have to do:

- The data splitting and search are a black box. It’s common (and we did so in our implementation) to tune the chunk size to get the best results from semantic search. But we can’t control this with the Assistants API, yet only we understand and can evaluate the quality of the context retrieval.
- The flip side to the API improving over time is that it might produce unexpected results in the future.
- We might want to build more complicated chains of context retrieval and LLM prompting, see [RAG with guardrails](https://www.pinecone.io/learn/fast-retrieval-augmented-generation/#RAG-with-Guardrails). But since we don’t control the implementation this is not possible.
- We still need a server to call the API from, to simplify the uploading and updating of contextual data, and to execute function calls from the LLM.
- It’s likely that we will still want to store message content for analytics, tight integration with our products, and abuse prevention. This leads to the need for syncing the messages to and from OpenAI, which can be more complex than having our server be the source of truth and only interact with OpenAI in a request/response fashion.
- Vendor lock-in: the implementations compared here all use OpenAI, as it performs well on both pricing and performance, especially considering the quality of results from GPT4. But adopting the Assistants API would make it much harder to move to a different LLM provider.

Additionally, the API has some rough edges at the moment. See the [implementation post](https://stack.convex.dev/ai-chat-using-openai-assistants-api) for more details.

Overall we’d suggest waiting before the API moves out of beta before adopting it. Even then, it should be carefully considered what kind of final experience one is trying to achieve, as more custom experiences will likely benefit from a custom implementation.

## LangChain

LangChain is an extremely popular and yet somewhat [controversial library](https://x.com/xixixao/status/1649930481834803200?s=20). It combines two seemingly opposing goals:

1. Make AI development more approachable.
2. Wrap, encapsulate, and expose sophisticated AI use cases.

### Pros

In the [second implementation](https://stack.convex.dev/ai-chat-using-langchain-and-convex), these two goals manifested themselves:

- The implementation is the shortest, as LangChain exposes very high level APIs tailored for exactly our use case. We didn’t even need to write a single system prompt.
- Under the hood the “chain” actually performs summarization before performing the context search and prompting to the LLM.
- Because LangChain abstracts the implementation, it would be trivial to swap in a different embedding or LLM model provider, or different vector database.
- It provides extensive “automatic” debug logging.

### Cons

Achieving these two goals though does have its downsides:

- LangChain defaults can be more complex than what the implementation really needs, and perform worse than simpler alternatives. This is especially true as LangChain tries to be agnostic to which LLM it interfaces with. Different LLMs will perform very differently and therefore might require different approaches to prompting and summarization.
- Once a developer wants to veer off of the default path, they have to understand the ways in which LangChain can be configured. This is tricky, and due to the high amount of abstraction, reading LangChain’s source code to figure out what it does under the hood can be difficult.
- The TypeScript version of LangChain is not perfectly typed, in part because the return values depend on configs nested deep in the chain.

Overall LangChain is great for prototyping and getting a system working quickly, but for a more sophisticated integration it’s a good idea to refactor into a fully custom implementation.

## Pure Convex  implementation

### Pros

Since Convex includes [vector search](https://docs.convex.dev/vector-search) the entire system can be implemented in it, only using external embedding and an LLM provider. In fact the LangChain implementation already used Convex under the hood for vector search, message storage and embedding caching.

Implementing the system without using a library like LangChain gives us the most control about exactly how the system works: 

- How and when data is split and how it is cached.
- When and how the data is embedded.
- How much context is used.
- In what format the data is passed to the LLM.
- What are the prompts involved.
- How are the responses processed and streamed back to our interface.

All of this is in plain sight in [our code](https://stack.convex.dev/ai-chat-with-convex-vector-search).

We have to admit that this comparison would be different if we weren’t using Convex, and instead had a more traditional server talk to an external relational/document databases and a vector database. With Convex, the custom implementation isn’t that much more complicated than the alternatives. And hence it’s easier to argue for the added up-front cost of developing a custom system to reap the benefits of total control. In turn, we get end-to-end type safety and full transparency.

## Conclusion

To answer the original question, does the new Assistants API make vector databases obsolete? We can argue, yes: If a use case aligns closely with what the API provides, and the alternative is shuffling data to an external vector database, then one might as well shuffle the data to the Assistants API instead.

But we would argue that integrated AI experiences should offer more than standalone ChatGPT to justify their existence, and to build such experiences it's best to retain control over the implementation by leveraging a database that can both store the product data and allow semantically searching it. This contextual data can then be used get the most out of current LLMs.

This year vector databases have sprung up like mushrooms to enable applications to retrieve context based on semantic search. A large portion of these applications have used the retrieved context to augment the ability of large language models (LLMs) in a pattern known as RAG. On November 7th OpenAI released its Assistants API, enabling the implementation of AI chat interfaces with context retrieval without needing a separate message store or vector database. Does this new API make vector databases obsolete?

Are Vector Databases Dead?

![](https://cdn.sanity.io/images/ts10onj4/production/c289de22f640759596268db735fd3208384ec951-2000x197.png?w=2000)

If you use PagerDuty and Slack you could probably benefit from automatically changing the topic for an #oncall Slack channel to display who’s currently on call. You can get this running in a manner of minutes [using this repo](https://github.com/get-convex/pagerduty-slack-sync). 

All you need to do is clone the repository, deploy the application to Convex, then enter your API keys and configuration entries.

```bash
git clone https://github.com/get-convex/pagerduty-slack-sync.git
cd pagerduty-slack-sync
npm install
npx convex deploy
# Set up the PagerDuty and Slack API keys on the dashboard. See README.md for details.
npx convex dashboard
```

And that’s it!

Read on to see how it’s built.

<GitHubRepo url="https://github.com/get-convex/pagerduty-slack-sync" description="Sync PagerDuty to Slack in minutes with Convex" />

At Convex, we use PagerDuty to handle our alerting needs so engineers can keep Convex running smoothly. PagerDuty has a powerful interface for creating multiple schedules letting engineers add temporary overrides, but this means that in the heat of an incident, it can be hard to quickly figure out who is actually on-call.

![PagerDuty’s scheduling interface](https://cdn.sanity.io/images/ts10onj4/production/c4ae3e38aa25652e49f34840312fcea8b29e0c1f-2000x840.png?w=2000)



Ideally, we wanted to see the current primary and secondary on-call engineer at a glance in Slack — where most incidents are discussed and resolved. While PagerDuty doesn’t have a built-in integration for this, we found [this repository](https://github.com/PagerDuty/pd-oncall-chat-topic) from the folks at PagerDuty which sets up a function in AWS Lambda to update your Slack channel topic with the current on-call status. It looked perfect for our needs — we followed the instructions to set up the CloudFormation deployment in AWS, set up the API Keys in AWS Systems Manager Parameter Store, stored the configuration in AWS DynamoDB, and a few hours later had PagerDuty updates streaming into our Slack channel.



![Architecture diagram from the PagerDuty repository](https://cdn.sanity.io/images/ts10onj4/production/d81add24ea9afc6a0c43b14da71aa612ed7ba026-2000x1523.png?w=2000)

Wait. A few hours? Surely we can do better than that!

The task at hand sounds simple enough: on a schedule, check who is on-call via the PagerDuty API. Send a request via the Slack API to update the channel topic with the latest state. This is a perfect fit for [Cron Job Scheduled Functions](https://docs.convex.dev/scheduling/cron-jobs) on Convex!

![](https://cdn.sanity.io/images/ts10onj4/production/ed00abff9abf938bcd668be8fe20dbb894226134-2000x1727.png?w=2000)

We can re-use the ideas from the original PagerDuty repository and write simple Convex functions to accomplish our task.

Let’s start with a function to fetch the current on-call user for a given PagerDuty schedule:

## `getOncallUser`
    
```tsx
// convex/sync.ts

async function getOncallUser(schedule: string): Promise<string> {
  const token = process.env.PAGERDUTY_API_KEY;
  if (token === undefined || token == "") {
    throw "PagerDuty API Key not set";
  }
  const headers = {
    Accept: "application/vnd.pagerduty+json;version=2",
    Authorization: `Token token=${token}`,
  };

  const normal_url = `https://api.pagerduty.com/schedules/${schedule}/users`;
  const override_url = `https://api.pagerduty.com/schedules/${schedule}/overrides`;
  const now = new Date();
  let since = new Date(now);
  // The PagerDuty API gets all users on-call for a schedule within a time bound. The easiest way to get the current on-call is to set a tight bound.
  since.setSeconds(since.getSeconds() - 5);
  let payload: Record<string, string> = {};
  payload["since"] = since.toISOString();
  payload["until"] = now.toISOString();
  let parameters = new URLSearchParams(payload);
  let query_string = `?${parameters.toString()}`;

  let normal_schedule = await fetch(normal_url + query_string, {
    headers: headers,
  });
  if (normal_schedule.status == 404) {
    throw `Invalid schedule ${schedule}`;
  }
  const response = await normal_schedule.json();
  const users = response["users"];
  const user = users[0];
  if (!user) {
    return "No one :panic:";
  }
  let username = user["name"];
  if (!username) {
    return "Deactivated user :panic:";
  }
  let override_schedule = await fetch(override_url + query_string, {
    headers: headers,
  });
  if ((await override_schedule.json())["overrides"].length > 0) {
    username += " (Override)";
  }
  return username;
}
```
    

We’ve chosen to store the PagerDuty API key in an [environment variable](https://docs.convex.dev/production/environment-variables) so it doesn’t need to be committed to your repository. 

Similarly, we’ll need to write a function that can set the topic of a given Slack channel. For extra style points, we’ll first get the current topic and preserve its contents when setting a new on-call user.

## `getSlackTopic` and `updateSlackTopic`
 
```tsx
// convex/sync.ts

async function getSlackTopic(channel: string): Promise<string> {
  const token = process.env.SLACK_API_KEY;
  if (token === undefined || token == "") {
    throw "Slack API Key not set";
  }
  const payload = {
    token: token,
    channel: channel,
  };
  let response = await fetch("https://slack.com/api/conversations.info", {
    method: "POST",
    body: new URLSearchParams(payload).toString(),
    headers: {
      "Content-Type": "application/x-www-form-urlencoded; charset=utf-8",
    },
  }).then((r) => r.json());
  try {
    const topic = response["channel"]["topic"]["value"];
    return topic;
  } catch (e) {
    console.error(e);
    console.info(response);
    throw `Could not find channel ${channel} on Slack; ensure the bot is in this channel`;
  }
}

const updateSlackTopic = async function (
  channel: string,
  proposed_update: string
): Promise<void> {
  const token = process.env.SLACK_API_KEY;
  if (token === undefined || token == "") {
    throw "Slack API Key not set";
  }
  let payload: Record<string, string> = {
    token: token,
    channel: channel,
  };

  let current_topic = await getSlackTopic(channel);
  if (current_topic == "") {
    current_topic = ".";
  }
  const pipe = current_topic.indexOf("|");
  if (pipe !== undefined) {
    let oncall_message = current_topic.substring(0, pipe).trimEnd();
    current_topic = current_topic.substring(pipe + 1).trimStart();
    if (oncall_message == proposed_update) {
      console.log("No topic update required");
      return;
    }
  }
  proposed_update = proposed_update + " | " + current_topic;
  payload["topic"] = proposed_update;
  if (process.env.DRY_RUN !== undefined) {
    console.log("Would update topic with request", payload);
    console.log("Exiting without sending request");
    return;
  }
  const response = await fetch("https://slack.com/api/conversations.setTopic", {
    method: "POST",
    headers: {
      "Content-Type": "application/x-www-form-urlencoded; charset=utf-8",
    },
    body: new URLSearchParams(payload).toString(),
  });
  if (response.status != 200) {
    const error = await response.text();
    throw `Failed to update topic: ${error}`;
  }
};   
```
    

Note that we also added a dry-run mode for testing without actually updating the topic. 

Next, we need a way to store our configuration such that it’s easy to add and remove Slack channels and PagerDuty schedules. Turns out Convex is pretty good at storing documents!

Let’s write a simple schema for our configuration documents:

```tsx
// convex/schema.ts
import { defineSchema, defineTable } from "convex/server";
import { v } from "convex/values";

export default defineSchema({
  // Each `config` defines a list of PagerDuty schedules (with names)
  // to fetch, and a Slack channel ID to update.
  configs: defineTable({
    channel: v.string(),
    schedules: v.array(v.object({ schedule: v.string(), name: v.string() })),
  }),
});
```

Lastly, we can wrap this all up in an [Action](https://docs.convex.dev/functions/actions) that iterates over the configs and performs the updates, then set up a cron job to call our function every minute.

## Action and Cron 

```tsx
// convex/sync.ts
export const getConfig = internalQuery({
  handler: async (ctx) => {
    return await ctx.db.query("configs").collect();
  },
});

async function performUpdate(config: Doc<"configs">) {
  console.log("Updating for config:", config);
  let topic = "";
  for (const schedule of config.schedules) {
    const oncall = await getOncallUser(schedule.schedule);
    if (topic != "") {
      topic += ", ";
    }
    topic += `${schedule.name}: ${oncall}`;
  }
  await updateSlackTopic(config.channel, topic);
}

export default action({
  handler: async (ctx) => {
    const configs = await ctx.runQuery(internal.sync.getConfig);
    await Promise.all(configs.map(performUpdate));
  },
});

// convex/crons.ts
import { cronJobs } from "convex/server";
import { api } from "./_generated/api";

const crons = cronJobs();
crons.interval("Update Slack topic", {minutes: 1}, api.sync.default);

export default crons;  
```
    

You can find the completed code for this integration [here](https://github.com/get-convex/pagerduty-slack-sync).

<GitHubRepo url="https://github.com/get-convex/pagerduty-slack-sync" description="Sync PagerDuty to Slack in minutes with Convex" />

And that’s it — deploying it is as simple as cloning this repository, setting up your Convex project, and configuring your API keys. You can add or remove configurations from the Convex dashboard and you’ll see changes immediately reflect in the linked Slack channel. No advanced degrees in AWS services required!

Syncing your Pagerduty instance with Slack is a good way to stay on top of your on-call schedule. We cover syncing Pagerduty information with a Slack channel.

Who's on Call? Learn to Sync Pagerduty with Slack

# AI Chat with Convex Vector Search

Convex is a full-stack development platform and cloud database, including built-in [vector search](https://docs.convex.dev/vector-search). In this third post in our series, we’ll build an AI-powered chat interface using Convex, with our own message storage and context retrieval.

All code can be found on [GitHub](https://github.com/get-convex/convex-ai-chat/).

<GitHubRepo url="https://github.com/get-convex/convex-ai-chat/" description="AI Chat implemented with Convex vector search" />

## ****Overview of the user experience****

We’ll build an AI chatbot for the [Convex docs site](https://docs.convex.dev/home). You can go to the docs now and interact with the bot there:

![docs screenshot with ai chat bot](https://cdn.sanity.io/images/ts10onj4/production/633b58f26a29b7e3de1a4d402aabc4ca0526b647-2400x1350.png)

## Overview of RAG chat

Here’s a graphical overview of the steps involved. Unlike in our previous posts, we will be implementing all steps ourselves, using Convex for storage and search, and OpenAI for creating embeddings and LLM chat completion.

![rag chat process overview](https://cdn.sanity.io/images/ts10onj4/production/4cb4a308b847724739ea0405f2f13825453efd7d-1730x724.png)

We need to load our data, optionally split it, and then “embed” it (run it through a model that encodes the data into an array of numbers). This step needs to happen at least once, but we probably want to run it periodically as the source data (in our example the Convex docs pages and other information about Convex) changes.

After we ingest data, we can serve traffic, in our case reply to user questions and follow-ups. Each question can be embedded (using the same model that we used during the ingest step), and then the embedding is used to look up the most relevant contextual data. We then pass on this data, the question, and any chat history to the LLM as a prompt, and return back to the user the LLM’s answer.

## Schema

We’ll start with our schema, which gives a good overview of what data will our database store ([source](https://github.com/get-convex/convex-ai-chat/blob/main/convex/schema.ts)):

```jsx
export default defineSchema({
  messages: defineTable({
    // Whether the message is from the AI or the human
    isViewer: v.boolean(),
    // Which conversation this message belongs to
    sessionId: v.string(),
    // Message content
    text: v.string(),
  }).index("bySessionId", ["sessionId"]),
  documents: defineTable({
    // The original page URL for the document
    url: v.string(),
    // The parsed document content
    text: v.string(),
  }).index("byUrl", ["url"]),
  chunks: defineTable({
    // Which document this chunk belongs to
    documentId: v.id("documents"),
    // The chunk content
    text: v.string(),
    // If the chunk has been embedded, which embedding corresponds to it
    embeddingId: v.union(v.id("embeddings"), v.null()),
  })
    .index("byDocumentId", ["documentId"])
    .index("byEmbeddingId", ["embeddingId"]),
  // the actual embeddings
  embeddings: defineTable({
    embedding: v.array(v.number()),
    chunkId: v.id("chunks"),
  })
    .index("byChunkId", ["chunkId"])
    .vectorIndex("byEmbedding", {
      vectorField: "embedding",
      dimensions: 1536,
    }),
});
```

We’ll refer to these tables as we go.

Since we’ll be using the OpenAI SDK, we’ll set our `OPENAI_API_KEY` in the [environment variable settings on Convex dashboard](https://docs.convex.dev/dashboard/deployments/deployment-settings).

## Ingest data: Loading & splitting data

How the data is loaded depends on what it is, whether webpages, PDFs, spreadsheets, scanned documents or any other source. This is the first complexity we encounter when building a system like this, and the choices we make here will have a downstream impact. For example, unless we coerce the LLM to format its answers in specific ways, it is likely to mimic the contextual data we provide it.

In the case of Convex docs, we would like the LLM to reply in markdown, which we can render into a rich text representation in the future. While we have the source of the docs, also in markdown, for our documentation chatbot we still opted to scrape our own website. Our docs include snippets and other content that does not show up until they are rendered into HTML.

You can see the 100 lines of code that perform the parsing from HTML to markdown [here](####). While we could use an existing library, writing this code ourselves gives us the most control over the output - a theme that will repeat throughout this post.

To test this scraping code out we build ourselves an endpoint that takes a single page URL, parses the page, splits it up into chunks, and saves it in the database ([source](https://github.com/get-convex/convex-ai-chat/blob/main/convex/ingest/load.ts)):

```jsx
export const fetchSingle = internalAction({
  args: {
    url: v.string(),
  },
  handler: async (ctx, { url }) => {
    const response = await fetch(url);
    const text = parsePage(await response.text());
    if (text.length > 0) {
      await ctx.runMutation(internal.ingest.load.updateDocument, { url, text });
    }
  },
});

export const updateDocument = internalMutation(
  async (ctx, { url, text }: { url: string; text: string }) => {
    const latestVersion = await ctx.db
      .query("documents")
      .withIndex("byUrl", (q) => q.eq("url", url))
      .order("desc")
      .first();

    const hasChanged = latestVersion === null || latestVersion.text !== text;
    if (hasChanged) {
      const documentId = await ctx.db.insert("documents", { url, text });
      const splitter = RecursiveCharacterTextSplitter.fromLanguage("markdown", {
        chunkSize: 2000,
        chunkOverlap: 100,
      });
      const chunks = await splitter.splitText(text);
      await map(chunks, async (chunk) => {
        await ctx.db.insert("chunks", {
          documentId,
          text: chunk,
          embeddingId: null,
        });
      });
    }
  }
);
```

Some noteworthy details:

- We don’t store pages with no text
- We’re using the `byUrl` index to ensure that we only store a new version of a given page if the content has changed
    - We do keep the existing data in the database since we don’t want to degrade the performance of the AI chat while we’re updating the scraped data. We can delete the stale data after successful update.

We also chunk the document text, for two reasons:

1. We want to make sure our contextual data fits in the LLM’s prompt.
2. Smaller chunks will be semantically more focused, and so we might get better results for specific user questions.

Since we have total control over this setup, we can easily optimize it. For example, instead of storing the full document text in both the `"documents"` table and split-up in the `"chunks"` table, we could store only a hash of the text in the `"documents"` table to determine whether the chunks need updating.

When we’re satisfied with the parsing we can use an action to scrape the whole website. Luckily our docs site has a [site map](https://en.wikipedia.org/wiki/Site_map), which will make it really easy ([source](https://github.com/get-convex/convex-ai-chat/blob/main/convex/ingest/load.ts)).

## Ingest data: Embedding data

For creating the embeddings for our documents we’ll use OpenAI’s API. This is our simple wrapper around the API ([source](https://github.com/get-convex/convex-ai-chat/blob/main/convex/ingest/embed.ts)):

```jsx
export async function embedTexts(texts: string[]) {
  if (texts.length === 0) return [];
  const openai = new OpenAI();
  const { data } = await openai.embeddings.create({
    input: texts,
    model: "text-embedding-ada-002",
  });
  return data.map(({ embedding }) => embedding);
}
```

Since we’ll want to run the embedding periodically, we structured our code to take in a list of document IDs, and we’ll first check which chunks need an embedding ([source](https://github.com/get-convex/convex-ai-chat/blob/main/convex/ingest/embed.ts)):

```jsx
export const chunksNeedingEmbedding = internalQuery(
  async (ctx, { documentId }: { documentId: Id<"documents"> }) => {
    const chunks = await ctx.db
      .query("chunks")
      .withIndex("byDocumentId", (q) => q.eq("documentId", documentId))
      .collect();
    return chunks.filter((chunk) => chunk.embeddingId === null);
  }
);
```

With this function in place, we can write the logic for embedding a list of documents ([source](https://github.com/get-convex/convex-ai-chat/blob/main/convex/ingest/embed.ts)):

```jsx
export const embedList = internalAction({
  args: {
    documentIds: v.array(v.id("documents")),
  },
  handler: async (ctx, { documentIds }) => {
    const chunks = (
      await map(documentIds, (documentId) =>
        ctx.runQuery(internal.ingest.embed.chunksNeedingEmbedding, {
          documentId,
        })
      )
    ).flat();

    const embeddings = await embedTexts(chunks.map((chunk) => chunk.text));
    await map(embeddings, async (embedding, i) => {
      const { _id: chunkId } = chunks[i];
      await ctx.runMutation(internal.ingest.embed.addEmbedding, {
        chunkId,
        embedding,
      });
    });
  },
});
```

With this action, we can again test that the embedding works as expected before we run the action for embedding all documents ([source](https://github.com/get-convex/convex-ai-chat/blob/main/convex/ingest/embed.ts)). And because we made sure to not re-embed text that’s already embedded, we don’t have to worry about how many times we run this action.

## Serving traffic: answering a question

When a user hits send in our little chat box, we’ll kick off our serving answering logic ([source](https://github.com/get-convex/convex-ai-chat/blob/main/convex/messages.ts)):

```jsx
export const send = mutation({
  args: {
    message: v.string(),
    sessionId: v.string(),
  },
  handler: async (ctx, { message, sessionId }) => {
    await ctx.db.insert("messages", {
      isViewer: true,
      text: message,
      sessionId,
    });
    await ctx.scheduler.runAfter(0, internal.aiChat.serve.answer, {
      sessionId,
    });
  },
});
```

The `scheduler.runAfter(0, …)` call is Convex parlance for “run an async job immediately”.

We implement the LLM-backed answering in one 80-line function ([source](https://github.com/get-convex/convex-ai-chat/blob/main/convex/serve.ts)). Let’s break it down:

```jsx
export const answer = internalAction({
  args: {
    sessionId: v.string(),
  },
  handler: async (ctx, { sessionId }) => {
```

We take in the `sessionId` argument which identifies which thread needs an answer. Then:

```jsx
    const messages = await ctx.runQuery(internal.serve.getMessages, {
      sessionId,
    });
    const lastUserMessage = messages.at(-1)!.text;

    const [embedding] = await embedTexts([lastUserMessage]);
```

Next, we find the messages for the thread and embed the last message. Since we expect this action to only be scheduled after a user sends a message, we can assume that the last message is indeed from the user. Then:

```jsx
    const searchResults = await ctx.vectorSearch("embeddings", "byEmbedding", {
      vector: embedding,
      limit: 8,
    });

    const relevantDocuments = await ctx.runQuery(internal.serve.getChunks, {
      embeddingIds: searchResults.map(({ _id }) => _id),
    });
```

We find the 8 most relevant documents for the given message (first performing a search that gives us the embeddings’ IDs, then mapping those to the chunk IDs). Let’s look at the rest of the function:

```jsx
    const messageId = await ctx.runMutation(internal.serve.addBotMessage, {
      sessionId,
    });

    try {
      const openai = new OpenAI();
      const stream = await openai.chat.completions.create({
        model: "gpt-4-32k",
        stream: true,
        messages: [
          {
            role: "system",
            content:
              "Answer the user question based on the provided documents " +
              "or report that the question cannot be answered based on " +
              "these documents. Keep the answer informative but brief, " +
              "do not enumerate all possibilities.",
          },
          ...(relevantDocuments.map(({ text }) => ({
            role: "system",
            content: "Relevant document:\n\n" + text,
          })) as ChatCompletionMessageParam[]),
          ...(messages.map(({ isViewer, text }) => ({
            role: isViewer ? "user" : "assistant",
            content: text,
          })) as ChatCompletionMessageParam[]),
        ],
      });
      let text = "";
      for await (const { choices } of stream) {
        const chunk = choices[0].delta.content;
        if (typeof chunk === "string" && chunk.length > 0) {
          text += choices[0].delta.content;
          await ctx.runMutation(internal.serve.updateBotMessage, {
            messageId,
            text,
          });
        }
      }
    } catch (error: any) {
      await ctx.runMutation(internal.serve.updateBotMessage, {
        messageId,
        text: "I cannot reply at this time. Reach out to the team on Discord",
      });
      throw error;
    }
```

First, we insert an empty message into the thread. This will enable us to show the “thinking” indicator back to the user.

We then call OpenAI’s GPT model. In this implementation, we use the `stream` argument to enable result streaming. This will lead to a better experience for our users, but we could have chosen not to use it as well (perhaps if our UX wasn’t quite the traditional chat app). With Convex, we don’t actually stream the results chunk by chunk to the client, instead, we accumulate the added chunks, store them in the message, and let Convex update the UI every time (via automatic reactivity). The resulting UX is just the same ([see this article for more details](https://stack.convex.dev/gpt-streaming-with-persistent-reactivity)).

Finally, we handle any errors by replacing the message text. We could have also decided to leave any partial message in place. We re-throw the error to get the best debugging experience.

## Serving traffic: listing messages

There’s only one more endpoint left to cover, and that’s the query that returns messages for a given session ([source](https://github.com/get-convex/convex-ai-chat/blob/main/convex/messages.ts)):

```jsx
export const list = query({
  args: {
    sessionId: v.string(),
  },
  handler: async (ctx, args) => {
    return await ctx.db
      .query("messages")
      .withIndex("bySessionId", (q) => q.eq("sessionId", args.sessionId))
      .collect();
  },
});
```

The client subscribes to this query and will get all messages for the session, automatically updating when new messages are added. Learn more about Convex reactivity [here](https://docs.convex.dev/tutorial/reactor).

And that’s it! The backend for our RAG chat is complete.

## Debugging tip

If you’re trying to debug the question-answering logic, it can be helpful to log the retrieved document chunks:

```tsx
// In convex/serve.ts, in the `answer` action:
console.log(lastUserMessage, relevantDocuments);
```

Then check out the [logs page on your Convex dashboard](https://docs.convex.dev/dashboard/deployments/logs) to see the context that was passed to the LLM.

## Frontend Implementation

You can find the front-end implementation of our chat box [here](https://github.com/get-convex/convex-ai-chat/blob/main/src/aiChat/index.tsx). It is only about 250 lines of React and Tailwind CSS, but it does pack a lot of functionality:

1. Realtime reactive updates with the LLM replies.
2. Preserving the session ID in `sessionStorage` (so it survives page reloads).
3. Scrolling to the bottom of the thread.
    1. Stop scrolling if the user scrolls up.
4. Opening the dialog in a React portal.
5. Disabling the send button when the input is empty.
6. Two different UIs: Small in the corner of the page and expanded.
7. Info icon with a tooltip for additional education/disclaimer.
8. Loading indicators
9. Dark mode

If you check out the [repo](https://github.com/get-convex/convex-ai-chat/blob/main/src/App.tsx) you can see that Convex is only used for the chat modal. This is a good example if you want to drop the chat component into an existing website.

## Conclusion

Sticking with Convex, instead of with the [Assistant API](https://stack.convex.dev/ai-chat-using-openai-assistants-api) or [LangChain](https://stack.convex.dev/ai-chat-using-langchain-and-convex) as we did in previous posts, we have full control over our AI system. We also have end-to-end type safety and consistency guarantees. We can log any data flowing through our system, and we can easily inspect the data stored in the database to debug why certain questions lead to given answers. Continue to the final post in this series to read more about the [tradeoffs between different AI product implementations](https://stack.convex.dev/are-vector-databases-dead).

Convex is a full-stack development platform and cloud database, including built-in vector search. In this third post in our [series](https://stack.convex.dev/ai-chat-using-openai-assistants-api), we’ll build an AI-powered chat interface using Convex, with our own message storage and context retrieval.

Build AI Chat with Convex Vector Search

# AI Chat using LangChain and Convex

[Langchain](https://www.langchain.com/) is a very popular library for building AI applications in Python or JavaScript/TypeScript. In this second post in [our series](https://stack.convex.dev/ai-chat-using-openai-assistants-api), we’ll build an AI-powered chat interface using LangChain and its new Convex integration.

All code can be found on [GitHub](https://github.com/get-convex/convex-ai-chat-langchain/).

<GitHubRepo url="https://github.com/get-convex/convex-ai-chat-langchain/" description="AI Chat implemented with LangChain and Convex." />

## ****Overview of the user experience****

We’ll build an AI chatbot for the [Convex docs site](https://docs.convex.dev/home). You can go to the docs now and interact with the bot there:

![docs screenshot with ai chat bot](https://cdn.sanity.io/images/ts10onj4/production/633b58f26a29b7e3de1a4d402aabc4ca0526b647-2400x1350.png)

We’re not just exposing the standard LLM model (such as GPT4), but we’re also retrieving the relevant documents for the question and passing them to the LLM as context ([RAG](https://js.langchain.com/docs/use_cases/question_answering/)).

## Overview of RAG chat

Here’s a graphical overview of the steps involved. LangChain will lead to a very short implementation, as almost the entirety of the process will be handled by the library.


![rag chat process overview with langchain](https://cdn.sanity.io/images/ts10onj4/production/7434678c58f8afb661ea5126c15cff68615f2ab7-1714x694.png)

We need to load our data, optionally split it, and then “embed” it (run it through a model that encodes the data into an array of numbers). This step needs to happen at least once, but we probably want to run it periodically as the source data (in our example the Convex docs pages and other information about Convex) changes.

After we ingest data, we can serve traffic, in our case reply to user questions and follow-ups. Each question can be embedded (using the same model that we used during the ingest step), and then the embedding is used to look up the most relevant contextual data. We then pass on this data, the question, and any chat history to the LLM as a prompt, and return back to the user the LLM’s answer.

The LangChain implementation we’ll be using performs a rephrasing step before searching data:

![serve step summarization diagram](https://cdn.sanity.io/images/ts10onj4/production/8288fedee8944ef1edc5a50d8041a37bb6ee0a7e-1338x650.png)

This is helpful for getting more useful documents retrieved for follow-up questions, but it does mean that the LLM is not seeing the conversation as a conversation, but instead is only seeing a single prompt that hopefully summarizes the previous discussion. It is possible to configure LangChain to not do this, but we’ll stick to the defaults in this post.

## Our Server Setup & Schema

We’ll be using the following schema, based on the LangChain Convex integrations docs [for cache](https://js.langchain.com/docs/modules/data_connection/text_embedding/how_to/caching_embeddings#usage-convex), [for documents](https://js.langchain.com/docs/integrations/vectorstores/convex), and [for messages](https://js.langchain.com/docs/integrations/chat_memory/convex) ([source](https://github.com/get-convex/convex-ai-chat-langchain/blob/main/convex/schema.ts)):

```tsx
export default defineSchema({
  // Simple cache to avoid recomputing embeddings
  cache: defineTable({
    // content
    key: v.string(),
    // embedding
    value: v.any(),
  }).index("byKey", ["key"]),
  // one row for each chunk of a document
  documents: defineTable({
    embedding: v.array(v.number()),
    text: v.string(),
    metadata: v.any(),
  }).vectorIndex("byEmbedding", {
    vectorField: "embedding",
    dimensions: 1536,
  }),
  messages: defineTable({
    // Which conversation this message belongs to
    sessionId: v.string(),
    message: v.object({
      // The message author, either AI or human
      type: v.string(),
      data: v.object({
        // The text of the message
        content: v.string(),
        role: v.optional(v.string()),
        name: v.optional(v.string()),
        additional_kwargs: v.optional(v.any()),
      }),
    }),
  }).index("bySessionId", ["sessionId"]),
});
```

We’ll also add `langchain/db.ts` file with the following export:

```tsx
export * from "langchain/util/convex";
```

This declares internal functions used by LangChain to interact with our tables.

Since we’ll be using the OpenAI SDK, we’ll set our `OPENAI_API_KEY` in the [environment variable settings on the Convex dashboard](https://docs.convex.dev/dashboard/deployments/deployment-settings).

## Ingest data: Loading & splitting & embedding data

We’ll lean on LangChain to help us scrape our docs, split them, and embed them for semantic search. We won’t have much control over the exact formatting of the documents using LangChain, but can implement manual parsing which we’ll cover in the [third post](https://stack.convex.dev/ai-chat-with-convex-vector-search) in this series.

Let’s look at the function responsible for ingesting a single page ([source](https://github.com/get-convex/convex-ai-chat-langchain/blob/main/convex/ingest/load.ts)):

```tsx
export const fetchAndEmbedSingle = internalAction({
  args: {
    url: v.string(),
  },
  handler: async (ctx, { url }) => {
    const loader = new CheerioWebBaseLoader(url);
    const data = await loader.load();
    const textSplitter = new RecursiveCharacterTextSplitter({
      chunkSize: 1000,
      chunkOverlap: 200,
    });

    const splitDocs = await textSplitter.splitDocuments(data);

    const embeddings = new CacheBackedEmbeddings({
      underlyingEmbeddings: new OpenAIEmbeddings(),
      documentEmbeddingStore: new ConvexKVStore({ ctx }),
    });

    await ConvexVectorStore.fromDocuments(splitDocs, embeddings, { ctx });
  },
});
```

The `CheerioWebBaseLoader` parses the HTML and returns just the text on the page with no formatting.

Then the `RecursiveCharacterTextSplitter` splits the documents into 1000-character chunks, with 200-character overlap. The size of the chunk depends on the context limits of the model we’re intending to use, we want to make sure that each chunk can fully fit in the context window. If we make chunks smaller we’ll make it more likely that multiple documents will be used as context, which can be better or worse, it really depends on our data and the kinds of questions the system is intended to answer.

Finally, we use `ConvexVectorStore` to run each chunk through OpenAI’s embedding API, caching them using `ConvexKVStore` in the `“cache”` table, and storing the chunks with the embeddings in the `“documents”` table.

## Serving traffic: answering a question

Just like with ingesting data, LangChain really streamlines the process of answering a message ([source](https://github.com/get-convex/convex-ai-chat-langchain/blob/main/convex/serve.ts)):

```tsx
export const answer = internalAction({
  args: {
    sessionId: v.string(),
    message: v.string(),
  },
  handler: async (ctx, { sessionId, message }) => {
    const vectorStore = new ConvexVectorStore(new OpenAIEmbeddings(), { ctx });

    const model = new ChatOpenAI({ modelName: "gpt-4-32k" });
    const memory = new BufferMemory({
      chatHistory: new ConvexChatMessageHistory({ sessionId, ctx }),
      memoryKey: "chat_history",
      outputKey: "text",
      returnMessages: true,
    });
    const chain = ConversationalRetrievalQAChain.fromLLM(
      model,
      vectorStore.asRetriever(),
      { memory }
    );

    await chain.call({ question: message });
  },
});
```

First, we set up the `ConvexVectorStore` using the same OpenAI embeddings API.

Next, we initialize  `ConvexChatMessageHistory` which will both read and write to our `"messages"` table, given the current `sessionId`.

And finally, we run the `ConversationalRetrievalQAChain`, which will write both the user message and the LLM answer to the `“messages”` table.

## Serving traffic: listing messages

The only other function we need is a query for listing current messages given a `sessionId` ([source](https://github.com/get-convex/convex-ai-chat-langchain/blob/main/convex/messages.ts)):

```tsx
export const list = query({
  args: {
    sessionId: v.string(),
  },
  handler: async (ctx, args) => {
    return (
      await ctx.db
        .query("messages")
        .withIndex("bySessionId", (q) => q.eq("sessionId", args.sessionId))
        .collect()
    ).map(({ message: { data, type }, ...fields }) => ({
      ...fields,
      isViewer: type === "human",
      text: data.content,
    }));
  },
});
```

We’re doing a little transformation on each message to return it in the same format as in our [first post](https://stack.convex.dev/ai-chat-using-openai-assistants-api).

## Debugging tip ☝️

If you’re trying to debug a chain, enable the verbose mode like this:

```tsx
    const chain = ConversationalRetrievalQAChain.fromLLM(
      model,
      vectorStore.asRetriever(),
      { memory, verbose: true }
    );
```

Then check out the [logs page on your Convex dashboard](https://docs.convex.dev/dashboard/deployments/logs) to see all the inputs and outputs for each step in the chain.

## Frontend Implementation

You can find the front-end implementation of our chat box [here](####). It is only about 250 lines of React and Tailwind CSS, but it does pack a lot of functionality:

1. Realtime reactive updates with the LLM replies.
2. Preserving the session ID in `sessionStorage` (so it survives page reloads).
3. Scrolling to the bottom of the thread.
    1. Stop scrolling if the user scrolls up.
4. Opening the dialog in a React portal.
5. Disabling the send button when the input is empty.
6. Two different UIs: Small in the corner of the page and expanded.
7. Info icon with a tooltip for additional education/disclaimer.
8. Loading indicators.
9. Dark mode.

If you check out the [repo](https://github.com/get-convex/convex-ai-chat-langchain/blob/main/src/aiChat/index.tsx) you can see that Convex is only used for the chat modal. This is a good example if you want to drop the chat component into an existing website.

## Conclusion

LangChain packs a lot of power into a simple library interface, and with very little code gets us a fairly sophisticated implementation. On the flip side, we do lose some type-safety, and figuring out what exactly the library does under the hood, and how to configure it, can be tricky. Continue on to our [third post](https://stack.convex.dev/ai-chat-with-convex-vector-search) if you’re interested in having total control over how your AI chat app is implemented. Continue to the final post in this series to read more about the [tradeoffs between different AI product implementations](https://stack.convex.dev/are-vector-databases-dead).

In this second post in our series, we’ll build an AI-powered chat interface using LangChain and its new Convex integration.

Build AI Chat with LangChain and Convex

Code With Antonio

Hi all 👋 In this 8 hour tutorial you will learn how to create an end-to-end fullstack notion clone, all with proper notion-style editor, cover images, nested documents, publishing documents to public, real-time database and more.

Fullstack Notion Clone: Next.js 13, React, Convex, Tailwind | Full Course 2023

Web Dev Cody


Build a text adventure game with Next.js, Convex, OpenAI, and DALL-E. Full stack walkthrough.

How to code an AI powered Text Adventure Game (Next.js, Convex, OpenAI, DALL-E)

Jack Herrington

Check out Convex, an incredibly well-designed Firebase alternative that provides a realtime database, object storage, server functions, http API hosting, cron jobs and more!

The Next Level Firebase for Modern Developers

Yet Another Infrastructure Group

Hosted by Ian & Alex, with Alana (Readyset), Hojjat (Deltastream), Jamie (Convex).

Convex and the future of streaming infrastructure

InfoQ

Alex Cole discusses dynamic web apps, how serverless solutions compare to in-house stacks, and how product development changes when individual engineers can own features end-to-end.


Everyone Can Be a Full-Stack Engineer