# Are vector databases dead?

This year vector databases have sprung up [like mushrooms](https://twitter.com/jobergum/status/1706928630612275309?s=20) to enable applications to retrieve context based on semantic search. A large portion of these applications have used the retrieved context to augment the ability of large language models (LLMs) in a pattern known as RAG. On November 7th OpenAI released its [Assistants API](https://platform.openai.com/docs/assistants/overview), enabling the implementation of AI chat interfaces with context retrieval without needing a separate message store or vector database. Does this new API make vector databases obsolete?

Yes! And no.

This post concludes a [series](https://stack.convex.dev/ai-chat-using-openai-assistants-api) comparing three different implementations of AI chat backends to help answer this question. Read on for the final verdict.

## AI chat - what’s the point?

In the current age of ChatGPT, GPTs, Bards, Bings, and Claudes it might not be obvious why anyone should be building another AI-powered chat, but there are good reasons:

1. Proximity: People’s behavior is driven by convenience, and talking to an AI right in your product (website, app) might be more convenient than opening a separate chat product.
2. Control: Big LLMs have broad knowledge so they need specific prompting to get an answer about a specific topic. Having to provide this prompt again and again can be onerous for end-users. Your product can set up or augment the prompt automatically.
3. Context: Although LLMs retain a lot of knowledge from training, they might lack specific knowledge, either because it didn’t exist, wasn’t public, or wasn’t important enough at training time. Providing specific context as a part of the prompt leads to better accuracy and less hallucination.

Combining all three together can lead to integrations not possible in the frame of existing general chat interfaces. For example, we could provide the LLM with user-specific information, give it precise instructions, and have it take actions directly in our product. Think: going to Amazon and asking to reorder and double the order of an item you bought last week, and having your shopping cart update to reflect this conversation. We can render product information inline and have it be interactive without the user having to navigate between different websites or apps.

## AI chat case study: docs

This post discusses the tradeoffs and details of implementing this kind of AI system. A basic example of a custom AI-powered chat interface is the AI chatbot on the Convex docs site. You can play with it by clicking the chat bubble button at the top of the [page](https://docs.convex.dev).

![docs screenshot with ai chat bot](https://cdn.sanity.io/images/ts10onj4/production/627ee9c0759b8c43b3f43927cb9af70bf638f416-2000x1125.png)

To help answer the titular question, we implemented this AI chat backend in three ways. A guided walkthrough and the source code for each implementation is linked here:

1. [Using the new OpenAI Assistants API](https://stack.convex.dev/ai-chat-using-openai-assistants-api)
2. [Using Langchain with the Convex message and vector DB](https://stack.convex.dev/ai-chat-using-langchain-and-convex)
3. [Entirely on Convex](https://stack.convex.dev/ai-chat-with-convex-vector-search)

This example demonstrates the three reasons for building a custom AI chat:

- The interface is embedded in the docs site where developers look for technical information about Convex.
- The LLM is prompted to answer questions about Convex.
- The implementation passes the LLM the most relevant documentation pertaining to the question.

Even in this simple use case, the context is not static. The Convex team iterates on the docs and the product constantly, and the AI chat needs to reflect these changes. So when designing the implementations we considered how this contextual information will be updated over time.

## Assistants API

### Pros

The new Assistants API tries to provide a cohesive package, and this does have its strengths for us as the developers of the product:

- We don’t have to worry about the context data splitting (chunking) and search implementations.
- OpenAI’s implementation might improve over time without us having to invest in any effort.
- Other tools besides retrieval, currently function calling and code interpreter, are built-in, and OpenAI might add more tools in the future.

### Cons

Yet these come with tradeoffs, broadly around how much we control and how much we still have to do:

- The data splitting and search are a black box. It’s common (and we did so in our implementation) to tune the chunk size to get the best results from semantic search. But we can’t control this with the Assistants API, yet only we understand and can evaluate the quality of the context retrieval.
- The flip side to the API improving over time is that it might produce unexpected results in the future.
- We might want to build more complicated chains of context retrieval and LLM prompting, see [RAG with guardrails](https://www.pinecone.io/learn/fast-retrieval-augmented-generation/#RAG-with-Guardrails). But since we don’t control the implementation this is not possible.
- We still need a server to call the API from, to simplify the uploading and updating of contextual data, and to execute function calls from the LLM.
- It’s likely that we will still want to store message content for analytics, tight integration with our products, and abuse prevention. This leads to the need for syncing the messages to and from OpenAI, which can be more complex than having our server be the source of truth and only interact with OpenAI in a request/response fashion.
- Vendor lock-in: the implementations compared here all use OpenAI, as it performs well on both pricing and performance, especially considering the quality of results from GPT4. But adopting the Assistants API would make it much harder to move to a different LLM provider.

Additionally, the API has some rough edges at the moment. See the [implementation post](https://stack.convex.dev/ai-chat-using-openai-assistants-api) for more details.

Overall we’d suggest waiting before the API moves out of beta before adopting it. Even then, it should be carefully considered what kind of final experience one is trying to achieve, as more custom experiences will likely benefit from a custom implementation.

## LangChain

LangChain is an extremely popular and yet somewhat [controversial library](https://x.com/xixixao/status/1649930481834803200?s=20). It combines two seemingly opposing goals:

1. Make AI development more approachable.
2. Wrap, encapsulate, and expose sophisticated AI use cases.

### Pros

In the [second implementation](https://stack.convex.dev/ai-chat-using-langchain-and-convex), these two goals manifested themselves:

- The implementation is the shortest, as LangChain exposes very high level APIs tailored for exactly our use case. We didn’t even need to write a single system prompt.
- Under the hood the “chain” actually performs summarization before performing the context search and prompting to the LLM.
- Because LangChain abstracts the implementation, it would be trivial to swap in a different embedding or LLM model provider, or different vector database.
- It provides extensive “automatic” debug logging.

### Cons

Achieving these two goals though does have its downsides:

- LangChain defaults can be more complex than what the implementation really needs, and perform worse than simpler alternatives. This is especially true as LangChain tries to be agnostic to which LLM it interfaces with. Different LLMs will perform very differently and therefore might require different approaches to prompting and summarization.
- Once a developer wants to veer off of the default path, they have to understand the ways in which LangChain can be configured. This is tricky, and due to the high amount of abstraction, reading LangChain’s source code to figure out what it does under the hood can be difficult.
- The TypeScript version of LangChain is not perfectly typed, in part because the return values depend on configs nested deep in the chain.

Overall LangChain is great for prototyping and getting a system working quickly, but for a more sophisticated integration it’s a good idea to refactor into a fully custom implementation.

## Pure Convex  implementation

### Pros

Since Convex includes [vector search](https://docs.convex.dev/vector-search) the entire system can be implemented in it, only using external embedding and an LLM provider. In fact the LangChain implementation already used Convex under the hood for vector search, message storage and embedding caching.

Implementing the system without using a library like LangChain gives us the most control about exactly how the system works: 

- How and when data is split and how it is cached.
- When and how the data is embedded.
- How much context is used.
- In what format the data is passed to the LLM.
- What are the prompts involved.
- How are the responses processed and streamed back to our interface.

All of this is in plain sight in [our code](https://stack.convex.dev/ai-chat-with-convex-vector-search).

We have to admit that this comparison would be different if we weren’t using Convex, and instead had a more traditional server talk to an external relational/document databases and a vector database. With Convex, the custom implementation isn’t that much more complicated than the alternatives. And hence it’s easier to argue for the added up-front cost of developing a custom system to reap the benefits of total control. In turn, we get end-to-end type safety and full transparency.

## Conclusion

To answer the original question, does the new Assistants API make vector databases obsolete? We can argue, yes: If a use case aligns closely with what the API provides, and the alternative is shuffling data to an external vector database, then one might as well shuffle the data to the Assistants API instead.

But we would argue that integrated AI experiences should offer more than standalone ChatGPT to justify their existence, and to build such experiences it's best to retain control over the implementation by leveraging a database that can both store the product data and allow semantically searching it. This contextual data can then be used get the most out of current LLMs.

This year vector databases have sprung up like mushrooms to enable applications to retrieve context based on semantic search. A large portion of these applications have used the retrieved context to augment the ability of large language models (LLMs) in a pattern known as RAG. On November 7th OpenAI released its Assistants API, enabling the implementation of AI chat interfaces with context retrieval without needing a separate message store or vector database. Does this new API make vector databases obsolete?

Are Vector Databases Dead?

Web Dev Cody


Build a text adventure game with Next.js, Convex, OpenAI, and DALL-E. Full stack walkthrough.

How to code an AI powered Text Adventure Game (Next.js, Convex, OpenAI, DALL-E)

# AI Chat with Convex Vector Search

Convex is a full-stack development platform and cloud database, including built-in [vector search](https://docs.convex.dev/vector-search). In this third post in our series, we’ll build an AI-powered chat interface using Convex, with our own message storage and context retrieval.

All code can be found on [GitHub](https://github.com/get-convex/convex-ai-chat/).

<GitHubRepo url="https://github.com/get-convex/convex-ai-chat/" description="AI Chat implemented with Convex vector search" />

## ****Overview of the user experience****

We’ll build an AI chatbot for the [Convex docs site](https://docs.convex.dev/home). You can go to the docs now and interact with the bot there:

![docs screenshot with ai chat bot](https://cdn.sanity.io/images/ts10onj4/production/633b58f26a29b7e3de1a4d402aabc4ca0526b647-2400x1350.png)

## Overview of RAG chat

Here’s a graphical overview of the steps involved. Unlike in our previous posts, we will be implementing all steps ourselves, using Convex for storage and search, and OpenAI for creating embeddings and LLM chat completion.

![rag chat process overview](https://cdn.sanity.io/images/ts10onj4/production/4cb4a308b847724739ea0405f2f13825453efd7d-1730x724.png)

We need to load our data, optionally split it, and then “embed” it (run it through a model that encodes the data into an array of numbers). This step needs to happen at least once, but we probably want to run it periodically as the source data (in our example the Convex docs pages and other information about Convex) changes.

After we ingest data, we can serve traffic, in our case reply to user questions and follow-ups. Each question can be embedded (using the same model that we used during the ingest step), and then the embedding is used to look up the most relevant contextual data. We then pass on this data, the question, and any chat history to the LLM as a prompt, and return back to the user the LLM’s answer.

## Schema

We’ll start with our schema, which gives a good overview of what data will our database store ([source](https://github.com/get-convex/convex-ai-chat/blob/main/convex/schema.ts)):

```jsx
export default defineSchema({
  messages: defineTable({
    // Whether the message is from the AI or the human
    isViewer: v.boolean(),
    // Which conversation this message belongs to
    sessionId: v.string(),
    // Message content
    text: v.string(),
  }).index("bySessionId", ["sessionId"]),
  documents: defineTable({
    // The original page URL for the document
    url: v.string(),
    // The parsed document content
    text: v.string(),
  }).index("byUrl", ["url"]),
  chunks: defineTable({
    // Which document this chunk belongs to
    documentId: v.id("documents"),
    // The chunk content
    text: v.string(),
    // If the chunk has been embedded, which embedding corresponds to it
    embeddingId: v.union(v.id("embeddings"), v.null()),
  })
    .index("byDocumentId", ["documentId"])
    .index("byEmbeddingId", ["embeddingId"]),
  // the actual embeddings
  embeddings: defineTable({
    embedding: v.array(v.number()),
    chunkId: v.id("chunks"),
  })
    .index("byChunkId", ["chunkId"])
    .vectorIndex("byEmbedding", {
      vectorField: "embedding",
      dimensions: 1536,
    }),
});
```

We’ll refer to these tables as we go.

Since we’ll be using the OpenAI SDK, we’ll set our `OPENAI_API_KEY` in the [environment variable settings on Convex dashboard](https://docs.convex.dev/dashboard/deployments/deployment-settings).

## Ingest data: Loading & splitting data

How the data is loaded depends on what it is, whether webpages, PDFs, spreadsheets, scanned documents or any other source. This is the first complexity we encounter when building a system like this, and the choices we make here will have a downstream impact. For example, unless we coerce the LLM to format its answers in specific ways, it is likely to mimic the contextual data we provide it.

In the case of Convex docs, we would like the LLM to reply in markdown, which we can render into a rich text representation in the future. While we have the source of the docs, also in markdown, for our documentation chatbot we still opted to scrape our own website. Our docs include snippets and other content that does not show up until they are rendered into HTML.

You can see the 100 lines of code that perform the parsing from HTML to markdown [here](####). While we could use an existing library, writing this code ourselves gives us the most control over the output - a theme that will repeat throughout this post.

To test this scraping code out we build ourselves an endpoint that takes a single page URL, parses the page, splits it up into chunks, and saves it in the database ([source](https://github.com/get-convex/convex-ai-chat/blob/main/convex/ingest/load.ts)):

```jsx
export const fetchSingle = internalAction({
  args: {
    url: v.string(),
  },
  handler: async (ctx, { url }) => {
    const response = await fetch(url);
    const text = parsePage(await response.text());
    if (text.length > 0) {
      await ctx.runMutation(internal.ingest.load.updateDocument, { url, text });
    }
  },
});

export const updateDocument = internalMutation(
  async (ctx, { url, text }: { url: string; text: string }) => {
    const latestVersion = await ctx.db
      .query("documents")
      .withIndex("byUrl", (q) => q.eq("url", url))
      .order("desc")
      .first();

    const hasChanged = latestVersion === null || latestVersion.text !== text;
    if (hasChanged) {
      const documentId = await ctx.db.insert("documents", { url, text });
      const splitter = RecursiveCharacterTextSplitter.fromLanguage("markdown", {
        chunkSize: 2000,
        chunkOverlap: 100,
      });
      const chunks = await splitter.splitText(text);
      await map(chunks, async (chunk) => {
        await ctx.db.insert("chunks", {
          documentId,
          text: chunk,
          embeddingId: null,
        });
      });
    }
  }
);
```

Some noteworthy details:

- We don’t store pages with no text
- We’re using the `byUrl` index to ensure that we only store a new version of a given page if the content has changed
    - We do keep the existing data in the database since we don’t want to degrade the performance of the AI chat while we’re updating the scraped data. We can delete the stale data after successful update.

We also chunk the document text, for two reasons:

1. We want to make sure our contextual data fits in the LLM’s prompt.
2. Smaller chunks will be semantically more focused, and so we might get better results for specific user questions.

Since we have total control over this setup, we can easily optimize it. For example, instead of storing the full document text in both the `"documents"` table and split-up in the `"chunks"` table, we could store only a hash of the text in the `"documents"` table to determine whether the chunks need updating.

When we’re satisfied with the parsing we can use an action to scrape the whole website. Luckily our docs site has a [site map](https://en.wikipedia.org/wiki/Site_map), which will make it really easy ([source](https://github.com/get-convex/convex-ai-chat/blob/main/convex/ingest/load.ts)).

## Ingest data: Embedding data

For creating the embeddings for our documents we’ll use OpenAI’s API. This is our simple wrapper around the API ([source](https://github.com/get-convex/convex-ai-chat/blob/main/convex/ingest/embed.ts)):

```jsx
export async function embedTexts(texts: string[]) {
  if (texts.length === 0) return [];
  const openai = new OpenAI();
  const { data } = await openai.embeddings.create({
    input: texts,
    model: "text-embedding-ada-002",
  });
  return data.map(({ embedding }) => embedding);
}
```

Since we’ll want to run the embedding periodically, we structured our code to take in a list of document IDs, and we’ll first check which chunks need an embedding ([source](https://github.com/get-convex/convex-ai-chat/blob/main/convex/ingest/embed.ts)):

```jsx
export const chunksNeedingEmbedding = internalQuery(
  async (ctx, { documentId }: { documentId: Id<"documents"> }) => {
    const chunks = await ctx.db
      .query("chunks")
      .withIndex("byDocumentId", (q) => q.eq("documentId", documentId))
      .collect();
    return chunks.filter((chunk) => chunk.embeddingId === null);
  }
);
```

With this function in place, we can write the logic for embedding a list of documents ([source](https://github.com/get-convex/convex-ai-chat/blob/main/convex/ingest/embed.ts)):

```jsx
export const embedList = internalAction({
  args: {
    documentIds: v.array(v.id("documents")),
  },
  handler: async (ctx, { documentIds }) => {
    const chunks = (
      await map(documentIds, (documentId) =>
        ctx.runQuery(internal.ingest.embed.chunksNeedingEmbedding, {
          documentId,
        })
      )
    ).flat();

    const embeddings = await embedTexts(chunks.map((chunk) => chunk.text));
    await map(embeddings, async (embedding, i) => {
      const { _id: chunkId } = chunks[i];
      await ctx.runMutation(internal.ingest.embed.addEmbedding, {
        chunkId,
        embedding,
      });
    });
  },
});
```

With this action, we can again test that the embedding works as expected before we run the action for embedding all documents ([source](https://github.com/get-convex/convex-ai-chat/blob/main/convex/ingest/embed.ts)). And because we made sure to not re-embed text that’s already embedded, we don’t have to worry about how many times we run this action.

## Serving traffic: answering a question

When a user hits send in our little chat box, we’ll kick off our serving answering logic ([source](https://github.com/get-convex/convex-ai-chat/blob/main/convex/messages.ts)):

```jsx
export const send = mutation({
  args: {
    message: v.string(),
    sessionId: v.string(),
  },
  handler: async (ctx, { message, sessionId }) => {
    await ctx.db.insert("messages", {
      isViewer: true,
      text: message,
      sessionId,
    });
    await ctx.scheduler.runAfter(0, internal.aiChat.serve.answer, {
      sessionId,
    });
  },
});
```

The `scheduler.runAfter(0, …)` call is Convex parlance for “run an async job immediately”.

We implement the LLM-backed answering in one 80-line function ([source](https://github.com/get-convex/convex-ai-chat/blob/main/convex/serve.ts)). Let’s break it down:

```jsx
export const answer = internalAction({
  args: {
    sessionId: v.string(),
  },
  handler: async (ctx, { sessionId }) => {
```

We take in the `sessionId` argument which identifies which thread needs an answer. Then:

```jsx
    const messages = await ctx.runQuery(internal.serve.getMessages, {
      sessionId,
    });
    const lastUserMessage = messages.at(-1)!.text;

    const [embedding] = await embedTexts([lastUserMessage]);
```

Next, we find the messages for the thread and embed the last message. Since we expect this action to only be scheduled after a user sends a message, we can assume that the last message is indeed from the user. Then:

```jsx
    const searchResults = await ctx.vectorSearch("embeddings", "byEmbedding", {
      vector: embedding,
      limit: 8,
    });

    const relevantDocuments = await ctx.runQuery(internal.serve.getChunks, {
      embeddingIds: searchResults.map(({ _id }) => _id),
    });
```

We find the 8 most relevant documents for the given message (first performing a search that gives us the embeddings’ IDs, then mapping those to the chunk IDs). Let’s look at the rest of the function:

```jsx
    const messageId = await ctx.runMutation(internal.serve.addBotMessage, {
      sessionId,
    });

    try {
      const openai = new OpenAI();
      const stream = await openai.chat.completions.create({
        model: "gpt-4-32k",
        stream: true,
        messages: [
          {
            role: "system",
            content:
              "Answer the user question based on the provided documents " +
              "or report that the question cannot be answered based on " +
              "these documents. Keep the answer informative but brief, " +
              "do not enumerate all possibilities.",
          },
          ...(relevantDocuments.map(({ text }) => ({
            role: "system",
            content: "Relevant document:\n\n" + text,
          })) as ChatCompletionMessageParam[]),
          ...(messages.map(({ isViewer, text }) => ({
            role: isViewer ? "user" : "assistant",
            content: text,
          })) as ChatCompletionMessageParam[]),
        ],
      });
      let text = "";
      for await (const { choices } of stream) {
        const chunk = choices[0].delta.content;
        if (typeof chunk === "string" && chunk.length > 0) {
          text += choices[0].delta.content;
          await ctx.runMutation(internal.serve.updateBotMessage, {
            messageId,
            text,
          });
        }
      }
    } catch (error: any) {
      await ctx.runMutation(internal.serve.updateBotMessage, {
        messageId,
        text: "I cannot reply at this time. Reach out to the team on Discord",
      });
      throw error;
    }
```

First, we insert an empty message into the thread. This will enable us to show the “thinking” indicator back to the user.

We then call OpenAI’s GPT model. In this implementation, we use the `stream` argument to enable result streaming. This will lead to a better experience for our users, but we could have chosen not to use it as well (perhaps if our UX wasn’t quite the traditional chat app). With Convex, we don’t actually stream the results chunk by chunk to the client, instead, we accumulate the added chunks, store them in the message, and let Convex update the UI every time (via automatic reactivity). The resulting UX is just the same ([see this article for more details](https://stack.convex.dev/gpt-streaming-with-persistent-reactivity)).

Finally, we handle any errors by replacing the message text. We could have also decided to leave any partial message in place. We re-throw the error to get the best debugging experience.

## Serving traffic: listing messages

There’s only one more endpoint left to cover, and that’s the query that returns messages for a given session ([source](https://github.com/get-convex/convex-ai-chat/blob/main/convex/messages.ts)):

```jsx
export const list = query({
  args: {
    sessionId: v.string(),
  },
  handler: async (ctx, args) => {
    return await ctx.db
      .query("messages")
      .withIndex("bySessionId", (q) => q.eq("sessionId", args.sessionId))
      .collect();
  },
});
```

The client subscribes to this query and will get all messages for the session, automatically updating when new messages are added. Learn more about Convex reactivity [here](https://docs.convex.dev/tutorial/reactor).

And that’s it! The backend for our RAG chat is complete.

## Debugging tip

If you’re trying to debug the question-answering logic, it can be helpful to log the retrieved document chunks:

```tsx
// In convex/serve.ts, in the `answer` action:
console.log(lastUserMessage, relevantDocuments);
```

Then check out the [logs page on your Convex dashboard](https://docs.convex.dev/dashboard/deployments/logs) to see the context that was passed to the LLM.

## Frontend Implementation

You can find the front-end implementation of our chat box [here](https://github.com/get-convex/convex-ai-chat/blob/main/src/aiChat/index.tsx). It is only about 250 lines of React and Tailwind CSS, but it does pack a lot of functionality:

1. Realtime reactive updates with the LLM replies.
2. Preserving the session ID in `sessionStorage` (so it survives page reloads).
3. Scrolling to the bottom of the thread.
    1. Stop scrolling if the user scrolls up.
4. Opening the dialog in a React portal.
5. Disabling the send button when the input is empty.
6. Two different UIs: Small in the corner of the page and expanded.
7. Info icon with a tooltip for additional education/disclaimer.
8. Loading indicators
9. Dark mode

If you check out the [repo](https://github.com/get-convex/convex-ai-chat/blob/main/src/App.tsx) you can see that Convex is only used for the chat modal. This is a good example if you want to drop the chat component into an existing website.

## Conclusion

Sticking with Convex, instead of with the [Assistant API](https://stack.convex.dev/ai-chat-using-openai-assistants-api) or [LangChain](https://stack.convex.dev/ai-chat-using-langchain-and-convex) as we did in previous posts, we have full control over our AI system. We also have end-to-end type safety and consistency guarantees. We can log any data flowing through our system, and we can easily inspect the data stored in the database to debug why certain questions lead to given answers. Continue to the final post in this series to read more about the [tradeoffs between different AI product implementations](https://stack.convex.dev/are-vector-databases-dead).

Convex is a full-stack development platform and cloud database, including built-in vector search. In this third post in our [series](https://stack.convex.dev/ai-chat-using-openai-assistants-api), we’ll build an AI-powered chat interface using Convex, with our own message storage and context retrieval.

Build AI Chat with Convex Vector Search

# AI Chat using LangChain and Convex

[Langchain](https://www.langchain.com/) is a very popular library for building AI applications in Python or JavaScript/TypeScript. In this second post in [our series](https://stack.convex.dev/ai-chat-using-openai-assistants-api), we’ll build an AI-powered chat interface using LangChain and its new Convex integration.

All code can be found on [GitHub](https://github.com/get-convex/convex-ai-chat-langchain/).

<GitHubRepo url="https://github.com/get-convex/convex-ai-chat-langchain/" description="AI Chat implemented with LangChain and Convex." />

## ****Overview of the user experience****

We’ll build an AI chatbot for the [Convex docs site](https://docs.convex.dev/home). You can go to the docs now and interact with the bot there:

![docs screenshot with ai chat bot](https://cdn.sanity.io/images/ts10onj4/production/633b58f26a29b7e3de1a4d402aabc4ca0526b647-2400x1350.png)

We’re not just exposing the standard LLM model (such as GPT4), but we’re also retrieving the relevant documents for the question and passing them to the LLM as context ([RAG](https://js.langchain.com/docs/use_cases/question_answering/)).

## Overview of RAG chat

Here’s a graphical overview of the steps involved. LangChain will lead to a very short implementation, as almost the entirety of the process will be handled by the library.


![rag chat process overview with langchain](https://cdn.sanity.io/images/ts10onj4/production/7434678c58f8afb661ea5126c15cff68615f2ab7-1714x694.png)

We need to load our data, optionally split it, and then “embed” it (run it through a model that encodes the data into an array of numbers). This step needs to happen at least once, but we probably want to run it periodically as the source data (in our example the Convex docs pages and other information about Convex) changes.

After we ingest data, we can serve traffic, in our case reply to user questions and follow-ups. Each question can be embedded (using the same model that we used during the ingest step), and then the embedding is used to look up the most relevant contextual data. We then pass on this data, the question, and any chat history to the LLM as a prompt, and return back to the user the LLM’s answer.

The LangChain implementation we’ll be using performs a rephrasing step before searching data:

![serve step summarization diagram](https://cdn.sanity.io/images/ts10onj4/production/8288fedee8944ef1edc5a50d8041a37bb6ee0a7e-1338x650.png)

This is helpful for getting more useful documents retrieved for follow-up questions, but it does mean that the LLM is not seeing the conversation as a conversation, but instead is only seeing a single prompt that hopefully summarizes the previous discussion. It is possible to configure LangChain to not do this, but we’ll stick to the defaults in this post.

## Our Server Setup & Schema

We’ll be using the following schema, based on the LangChain Convex integrations docs [for cache](https://js.langchain.com/docs/modules/data_connection/text_embedding/how_to/caching_embeddings#usage-convex), [for documents](https://js.langchain.com/docs/integrations/vectorstores/convex), and [for messages](https://js.langchain.com/docs/integrations/chat_memory/convex) ([source](https://github.com/get-convex/convex-ai-chat-langchain/blob/main/convex/schema.ts)):

```tsx
export default defineSchema({
  // Simple cache to avoid recomputing embeddings
  cache: defineTable({
    // content
    key: v.string(),
    // embedding
    value: v.any(),
  }).index("byKey", ["key"]),
  // one row for each chunk of a document
  documents: defineTable({
    embedding: v.array(v.number()),
    text: v.string(),
    metadata: v.any(),
  }).vectorIndex("byEmbedding", {
    vectorField: "embedding",
    dimensions: 1536,
  }),
  messages: defineTable({
    // Which conversation this message belongs to
    sessionId: v.string(),
    message: v.object({
      // The message author, either AI or human
      type: v.string(),
      data: v.object({
        // The text of the message
        content: v.string(),
        role: v.optional(v.string()),
        name: v.optional(v.string()),
        additional_kwargs: v.optional(v.any()),
      }),
    }),
  }).index("bySessionId", ["sessionId"]),
});
```

We’ll also add `langchain/db.ts` file with the following export:

```tsx
export * from "langchain/util/convex";
```

This declares internal functions used by LangChain to interact with our tables.

Since we’ll be using the OpenAI SDK, we’ll set our `OPENAI_API_KEY` in the [environment variable settings on the Convex dashboard](https://docs.convex.dev/dashboard/deployments/deployment-settings).

## Ingest data: Loading & splitting & embedding data

We’ll lean on LangChain to help us scrape our docs, split them, and embed them for semantic search. We won’t have much control over the exact formatting of the documents using LangChain, but can implement manual parsing which we’ll cover in the [third post](https://stack.convex.dev/ai-chat-with-convex-vector-search) in this series.

Let’s look at the function responsible for ingesting a single page ([source](https://github.com/get-convex/convex-ai-chat-langchain/blob/main/convex/ingest/load.ts)):

```tsx
export const fetchAndEmbedSingle = internalAction({
  args: {
    url: v.string(),
  },
  handler: async (ctx, { url }) => {
    const loader = new CheerioWebBaseLoader(url);
    const data = await loader.load();
    const textSplitter = new RecursiveCharacterTextSplitter({
      chunkSize: 1000,
      chunkOverlap: 200,
    });

    const splitDocs = await textSplitter.splitDocuments(data);

    const embeddings = new CacheBackedEmbeddings({
      underlyingEmbeddings: new OpenAIEmbeddings(),
      documentEmbeddingStore: new ConvexKVStore({ ctx }),
    });

    await ConvexVectorStore.fromDocuments(splitDocs, embeddings, { ctx });
  },
});
```

The `CheerioWebBaseLoader` parses the HTML and returns just the text on the page with no formatting.

Then the `RecursiveCharacterTextSplitter` splits the documents into 1000-character chunks, with 200-character overlap. The size of the chunk depends on the context limits of the model we’re intending to use, we want to make sure that each chunk can fully fit in the context window. If we make chunks smaller we’ll make it more likely that multiple documents will be used as context, which can be better or worse, it really depends on our data and the kinds of questions the system is intended to answer.

Finally, we use `ConvexVectorStore` to run each chunk through OpenAI’s embedding API, caching them using `ConvexKVStore` in the `“cache”` table, and storing the chunks with the embeddings in the `“documents”` table.

## Serving traffic: answering a question

Just like with ingesting data, LangChain really streamlines the process of answering a message ([source](https://github.com/get-convex/convex-ai-chat-langchain/blob/main/convex/serve.ts)):

```tsx
export const answer = internalAction({
  args: {
    sessionId: v.string(),
    message: v.string(),
  },
  handler: async (ctx, { sessionId, message }) => {
    const vectorStore = new ConvexVectorStore(new OpenAIEmbeddings(), { ctx });

    const model = new ChatOpenAI({ modelName: "gpt-4-32k" });
    const memory = new BufferMemory({
      chatHistory: new ConvexChatMessageHistory({ sessionId, ctx }),
      memoryKey: "chat_history",
      outputKey: "text",
      returnMessages: true,
    });
    const chain = ConversationalRetrievalQAChain.fromLLM(
      model,
      vectorStore.asRetriever(),
      { memory }
    );

    await chain.call({ question: message });
  },
});
```

First, we set up the `ConvexVectorStore` using the same OpenAI embeddings API.

Next, we initialize  `ConvexChatMessageHistory` which will both read and write to our `"messages"` table, given the current `sessionId`.

And finally, we run the `ConversationalRetrievalQAChain`, which will write both the user message and the LLM answer to the `“messages”` table.

## Serving traffic: listing messages

The only other function we need is a query for listing current messages given a `sessionId` ([source](https://github.com/get-convex/convex-ai-chat-langchain/blob/main/convex/messages.ts)):

```tsx
export const list = query({
  args: {
    sessionId: v.string(),
  },
  handler: async (ctx, args) => {
    return (
      await ctx.db
        .query("messages")
        .withIndex("bySessionId", (q) => q.eq("sessionId", args.sessionId))
        .collect()
    ).map(({ message: { data, type }, ...fields }) => ({
      ...fields,
      isViewer: type === "human",
      text: data.content,
    }));
  },
});
```

We’re doing a little transformation on each message to return it in the same format as in our [first post](https://stack.convex.dev/ai-chat-using-openai-assistants-api).

## Debugging tip ☝️

If you’re trying to debug a chain, enable the verbose mode like this:

```tsx
    const chain = ConversationalRetrievalQAChain.fromLLM(
      model,
      vectorStore.asRetriever(),
      { memory, verbose: true }
    );
```

Then check out the [logs page on your Convex dashboard](https://docs.convex.dev/dashboard/deployments/logs) to see all the inputs and outputs for each step in the chain.

## Frontend Implementation

You can find the front-end implementation of our chat box [here](####). It is only about 250 lines of React and Tailwind CSS, but it does pack a lot of functionality:

1. Realtime reactive updates with the LLM replies.
2. Preserving the session ID in `sessionStorage` (so it survives page reloads).
3. Scrolling to the bottom of the thread.
    1. Stop scrolling if the user scrolls up.
4. Opening the dialog in a React portal.
5. Disabling the send button when the input is empty.
6. Two different UIs: Small in the corner of the page and expanded.
7. Info icon with a tooltip for additional education/disclaimer.
8. Loading indicators.
9. Dark mode.

If you check out the [repo](https://github.com/get-convex/convex-ai-chat-langchain/blob/main/src/aiChat/index.tsx) you can see that Convex is only used for the chat modal. This is a good example if you want to drop the chat component into an existing website.

## Conclusion

LangChain packs a lot of power into a simple library interface, and with very little code gets us a fairly sophisticated implementation. On the flip side, we do lose some type-safety, and figuring out what exactly the library does under the hood, and how to configure it, can be tricky. Continue on to our [third post](https://stack.convex.dev/ai-chat-with-convex-vector-search) if you’re interested in having total control over how your AI chat app is implemented. Continue to the final post in this series to read more about the [tradeoffs between different AI product implementations](https://stack.convex.dev/are-vector-databases-dead).

In this second post in our series, we’ll build an AI-powered chat interface using LangChain and its new Convex integration.

Build AI Chat with LangChain and Convex

# AI Chat using OpenAI Assistants API

On November 7th OpenAI released its [Assistants API](https://platform.openai.com/docs/assistants/overview), enabling chat bot with context retrieval implementations without needing a messages or vector database. In this post, we’ll cover how to leverage this API to build a fully functioning AI chat interface.

The implementation here can be a basis for a more complex and tailored experience, which ties app-specific and user-specific information together in a single interface, surpassing the capabilities of the standalone ChatGPT.

This post is part of a series where we compare three different implementations of the same AI chatbot and summarize the differences:

1. Using OpenAI’s Assistants API (this post)
2. [Using Langchain with our own storage](https://stack.convex.dev/ai-chat-using-langchain-and-convex)
3. [Custom message store and vector search retrieval](https://stack.convex.dev/ai-chat-with-convex-vector-search)
4. [Are vector databases dead?](https://stack.convex.dev/are-vector-databases-dead)

All code can be found on [GitHub](https://github.com/get-convex/convex-ai-chat-openai/).

<GitHubRepo url="[https://github.com/get-convex/convex-ai-chat-openai/](https://github.com/get-convex/convex-ai-chat-openai/)" description="AI Chat implemented with the OpenAI Assistants API." />

## ****Overview of the user experience****

To compare the three approaches we’ll build an AI chatbot for the [Convex docs site](https://docs.convex.dev/home). You can go to the docs now and interact with the bot there:

![docs screenshot with ai chat bot](https://cdn.sanity.io/images/ts10onj4/production/633b58f26a29b7e3de1a4d402aabc4ca0526b647-2400x1350.png)

We’re not just exposing the standard LLM model (such as GPT4), but we’re also retrieving the relevant documents for the question and passing them to the LLM as context ([RAG](https://js.langchain.com/docs/use_cases/question_answering/)).

## Overview of RAG chat

Here’s a graphical overview of the steps involved. Thanks to the new Assistants API, OpenAI will take care of most of this process.

![process overview for RAG chat](https://cdn.sanity.io/images/ts10onj4/production/0020d6ed2c966add4471a14f084efd7bf9589fbd-1722x708.png)

We need to load our data, optionally split it, and then “embed” it (run it through a model that encodes the data into an array of numbers). This step needs to happen at least once, but we probably want to run it periodically as the source data (in our example the <QuickConvex /> documentation and other information about Convex) changes.

After we ingest data, we can serve traffic. In our case, we reply to user questions about Convex. Each question can be embedded (using the same model that we used during the ingest step), and then the embedding is used to look up the most relevant contextual data. We then pass on this contextual data, the question, and any chat history to the LLM as a prompt, and return the LLM’s answer back to the user.

As we’ll see in the second post, a rephrasing step can be added to enable searching data based on the whole chat history, not just the last user message. The Assistants API might or might not be using this technique, as the search is completely abstracted away from us.

## Our Server Setup & Schema

While the Assistants API is powerful, we still want our own server for a few reasons:

1. We don’t want to expose our OpenAI API key. The OpenAI requests need to be authenticated, but if we send them from the browser, any user could see our key and use it themself.
2. We’ll use the server to pre-process our documentation data before uploading it to OpenAI.
3. Future: Having a server can enable the Assistants API to call functions that we can expose to take some action on behalf of the user.
4. Future: If we want to build a more sophisticated experience, we can directly access our product database during the chat.

We’ll define the following three tables in Convex([source](https://github.com/get-convex/convex-ai-chat-openai/blob/main/convex/schema.ts)):

```jsx
export default defineSchema({
  documents: defineTable({
    // The original page URL for the document
    url: v.string(),
    // The parsed document content
    text: v.string(),
    // The ID returned after uploading to OpenAI
    fileId: v.union(v.string(), v.null()),
  }).index("byUrl", ["url"]),
  messages: defineTable({
    // Whether the message is from the AI or the human
    isViewer: v.boolean(),
    // Which conversation this message belongs to
    sessionId: v.string(),
    // Message content
    text: v.string(),
  }).index("bySessionId", ["sessionId"]),
  threads: defineTable({
    // Client-generated conversation identifier
    sessionId: v.string(),
    // Conversation identifier used by the OpenAI server
    threadId: v.string(),
  }).index("bySessionId", ["sessionId"]),
});
```

Since we’ll be using the OpenAI SDK, we’ll set our `OPENAI_API_KEY` in the [environment variable settings on the Convex dashboard](https://docs.convex.dev/dashboard/deployments/deployment-settings).

## Ingest data: Creating an assistant

We’ll start by creating an [OpenAI assistant](https://platform.openai.com/docs/assistants/overview). You can do this on the OpenAI dashboard, or using the API ([source](https://github.com/get-convex/convex-ai-chat-openai/blob/main/convex/init.ts)):

```tsx
export const createAssistant = internalAction({
  args: {},
  handler: async () => {
    const openai = new OpenAI();
    const assistant = await openai.beta.assistants.create({
      instructions:
        "Answer the user questions based on the provided documents " +
        "or report that the question cannot be answered based on " +
        "these documents. Keep the answer informative but brief, " +
        "do not enumerate all possibilities.",
      model: "gpt-4-1106-preview",
      tools: [{ type: "retrieval" }],
    });
    return assistant.id;
  },
});
```

You can run this action from the [Convex dashboard](https://dashboard.convex.dev) and then save the returned ID as  `ASSISTANT_ID` in your Convex backend’s [environment variables](https://docs.convex.dev/dashboard/deployments/deployment-settings).

## Ingest data: Loading data

We’ll cover the detailed description of scraping in the third post in the series, but here we’ll discuss how to upload data to our OpenAI assistant.

While scraping the docs, we stored the data in the `"documents"` table. We also have a field on each document that stores the `fileId` returned by OpenAI after we upload it. Here’s the function responsible for uploading our files to OpenAI and attaching them to the assistant ([source](https://github.com/get-convex/convex-ai-chat-openai/blob/main/convex/init.ts)):

```tsx
export const uploadDocuments = internalAction({
  args: {
    documentIds: v.array(v.id("documents")),
  },
  handler: async (ctx, { documentIds }) => {
    const openai = new OpenAI();
    await map(documentIds, async (documentId) => {
      const document = await ctx.runQuery(internal.init.getDocument, {
        documentId,
      });
      if (document === null || document.fileId !== null) {
        return;
      }
      const { text, url } = document;
      const blob = new File([text], fileName(url));

      const { id: fileId } = await openai.files.create({
        file: blob,
        purpose: "assistants",
      });
      await openai.beta.assistants.files.create(process.env.ASSISTANT_ID!, {
        file_id: fileId,
      });
      await ctx.runMutation(internal.init.saveFileId, { documentId, fileId });
    });
  },
});
```

This action uses the `openai.files.create` API to upload each document as a Blob, and then the `openai.beta.assistants.files.create` to attach the file to our assistant. You could also manually upload the files on the OpenAI dashboard.

When ready, we can upload and attach all documents from our table in bulk ([source](https://github.com/get-convex/convex-ai-chat-openai/blob/main/convex/init.ts#L30)).

## Ingest data: Splitting & embedding data

Nothing to do here, as OpenAI takes care of this step for us after we upload and attach the files automatically! 👏

## Serving traffic

Here’s a [sequence diagram](https://en.wikipedia.org/wiki/Sequence_diagram) for the traffic-serving portion of our app:

![overview of answer implementation](https://cdn.sanity.io/images/ts10onj4/production/b3d97e0c608ad248bce43b507bb4e83091a10d98-2750x1358.png)

## Serving traffic: answering a question

When a user hits send in our little chat box, we’ll kick off generating the answer ([source](https://github.com/get-convex/convex-ai-chat-openai/blob/main/convex/messages.ts)):

```jsx
export const send = mutation({
  args: {
    message: v.string(),
    sessionId: v.string(),
  },
  handler: async (ctx, { message, sessionId }) => {
    await ctx.db.insert("messages", {
      isViewer: true,
      text: message,
      sessionId,
    });
    await ctx.scheduler.runAfter(0, internal.serve.answer, {
      sessionId,
      message,
    });
  },
});
```

First, we save the message to the `"messages"` table, which will update our UI (if you’re not familiar with Convex and wonder how this works, head over to the [Convex tutorial](https://docs.convex.dev/get-started)).

The `scheduler.runAfter(0, …)` call is Convex parlance for “run an async job immediately”.

Let’s look at the implementation of the `answer` action ([source](https://github.com/get-convex/convex-ai-chat-openai/blob/main/convex/serve.ts)):

```tsx
export const answer = internalAction({
  args: {
    sessionId: v.string(),
    message: v.string(),
  },
  handler: async (ctx, { sessionId, message }) => {
    const openai = new OpenAI();

    const threadId = await getOrCreateThread(ctx, openai, sessionId);

    const { id: lastMessageId } = await openai.beta.threads.messages.create(
      threadId,
      { role: "user", content: message }
    );

    const { id: runId } = await openai.beta.threads.runs.create(threadId, {
      assistant_id: process.env.ASSISTANT_ID!,
    });

    await pollForAnswer(ctx, { threadId, sessionId, lastMessageId, runId });
  },
});
```

The steps we follow are:

1. Get or create an OpenAI thread for the current session
2. Add the user message to the OpenAI thread
3. Create an assistant run
4. Start polling for the answer

Polling is needed because as of the time of this writing, the Assistants API doesn’t support streaming or a single-shot async request.

Here’s the function performing the polling every 500ms[^1] ([source](https://github.com/get-convex/convex-ai-chat-openai/blob/main/convex/serve.ts)):

[^1]: You can also use the scheduler to do this, by re-scheduling the function every 500ms with `scheduler.runAfter(500, `. The while loop approach here is just for simplicity.

```tsx
async function pollForAnswer(
  ctx: ActionCtx,
  args: {
    sessionId: string;
    threadId: string;
    runId: string;
    lastMessageId: string;
  }
) {
  const { sessionId, threadId, runId, lastMessageId } = args;
  const openai = new OpenAI();
  while (true) {
    await sleep(500);
    const run = await openai.beta.threads.runs.retrieve(threadId, runId);
    switch (run.status) {
      case "failed":
      case "expired":
      case "cancelled":
        await ctx.runMutation(internal.serve.addMessage, {
          text: "I cannot reply at this time. Reach out to the team on Discord",
          sessionId,
        });
        return;
      case "completed": {
        const { data: newMessages } = await openai.beta.threads.messages.list(
          threadId,
          { after: lastMessageId, order: "asc" }
        );
        await map(newMessages, async ({ content }) => {
          const text = content
            .filter((item): item is MessageContentText => item.type === "text")
            .map(({ text }) => text.value)
            .join("\n\n");
          await ctx.runMutation(internal.serve.addMessage, { text, sessionId });
        });
        return;
      }
    }
  }
}
```

When the run is `"completed"` we want to store the new message in our `"messages"` table, which will update the UI. We do this using the `{ after: lastMessageId, order: "asc" }` arguments to the `openai.beta.threads.messages.list` API to get only the new messages, and then we save them to our database in the `addMessage` mutation ([source](https://github.com/get-convex/convex-ai-chat-openai/blob/main/convex/serve.ts#L113)).

## Serving traffic: listing messages

There’s only one more endpoint left to cover, and that’s the query that returns all messages for a given session ([source](https://github.com/get-convex/convex-ai-chat-openai/blob/main/convex/messages.ts)):

```jsx
export const list = query({
  args: {
    sessionId: v.string(),
  },
  handler: async (ctx, args) => {
    return await ctx.db
      .query("messages")
      .withIndex("bySessionId", (q) => q.eq("sessionId", args.sessionId))
      .collect();
  },
});
```

The client subscribes to this query and will get all messages for the session, automatically updating when new messages are added. Learn more about Convex reactivity [here](https://docs.convex.dev/tutorial/reactor).

And that’s it! The backend for our RAG chat is complete.

## Frontend Implementation

You can find the front-end implementation of our chat box [here](https://github.com/get-convex/convex-ai-chat-openai/blob/main/src/aiChat/index.tsx). It is only about 250 lines of React and Tailwind CSS, but it does pack a lot of functionality:

1. Realtime reactive updates with the LLM replies.
2. Preserving the session ID in `sessionStorage` (so it survives page reloads).
3. Scrolling to the bottom of the thread.
    1. Stop scrolling if the user scrolls up.
4. Opening the dialog in a React portal.
5. Disabling the send button when the input is empty.
6. Two different UIs: Small in the corner of the page and expanded.
7. Info icon with a tooltip for additional education/disclaimer.
8. Loading indicators
9. Dark mode

If you check out the [repo](https://github.com/get-convex/convex-ai-chat-openai/blob/main/src/App.tsx) you can see that Convex is only used for the chat modal. This is a good example if you want to drop the chat component into an existing website.

## Conclusion

OpenAI’s assistants API makes it quite easy to set up an AI chatbot with context retrieval. The API is in beta and some of its edges are a bit rough at the moment, but these issues will likely be quickly addressed. The Assistants API has additional capabilities we didn’t discuss, mainly the ability for the assistant to run custom functions. This might enable tighter integration with our product database. Alternatively, we can implement context retrieval directly on our server. We cover the first such implementation, using LangChain, in our [next post](https://stack.convex.dev/ai-chat-using-langchain-and-convex). To see how all of the implementations compare, see [this post](https://stack.convex.dev/are-vector-databases-dead).

On November 7th OpenAI released its Assistants API, enabling chat bot with context retrieval implementations without needing a messages or vector database. In this post, we’ll cover how to leverage this API to build a fully functioning AI chat interface.

Build AI Chat with OpenAI's Assistants API

There has been a lot changing with AI Town - here’s an overdue update to catch you up. 

Background: After launching https://convex.dev/ai-town mere weeks ago as an experiment, we were surprised at how much attention the project gained (over 5k stars on GitHub!). Interest grew in developing it further: for games, personal experimentation, and more. Some of you likely ran across notable clones of it, like [AI Silicon Valley](https://ai-silicon-valley.fly.dev/), [Cat Town](https://cat-town.fly.dev/), or my personal favorite: [Zaranova](https://zaranova.xyz/). We’ve been hard at work improving it since then. Let’s look at some recent changes, and what it means for anyone interested in extending the platform. As a reminder, the repo is open-source, and can be found here:

<GitHubRepo url="https://github.com/a16z-infra/ai-town" />

# Interactive

You can now walk around and interact with the AI citizens! Click to walk around, invite or be invited to chat with nearby citizens. We’ve had some fun internally spreading rumors amongst the citizens and seeing that spread and show up in later conversations with other people.

# Extensible game mechanics

We overhauled the serverless game engine to make it easier to extend. Be on the lookout for an upcoming Stack post on how I added activities, if you’re wanting to add more gameplay elements. Some exciting ones proposed by the community: let the citizens message on Discord or SMS, interact with objects and their environment 🔥, or even trade stonks 💹.

# Easy to mod

Don’t want to code, but want your own AI ant farm, Matrix, or USS Callister? 

Edit the name, identity, and other attributes of an agent (NPC) in the character data before creating your world, or in your [Dashboard](https://dashboard.convex.dev) in the `agents` table once it’s live. Their conversations will immediately start to reflect their new identities. To save this data, you can download a table snapshot from settings, and copy the identities into the seed data later when you want to recreate the world.

There’s also a map editor to make your own levels, with some community members using AI to make the map assets and character sprites. What a time to be alive! 

Want to contribute? Join [this Issue](https://github.com/a16z-infra/ai-town/issues/185) to make agent data easier to sync and share.

<VimeoPlayer video="879687042" />

# Join the party

Bring your wild ideas and busted worlds. The AI Town Discord is in #ai-town-dev in the AI Stack Devs [here](https://discord.gg/F8ApM3JX). Come chat about <QuickConvex />, the backend for AI Town, in our community Discord [here](http://discord.gg/convex). You can find the open source repo here: 

<GitHubRepo url="https://github.com/a16z-infra/ai-town" />

🤫 Keep your eyes out for an upcoming competition for best AI Town clone. May the best civilization win!

It's never been a better time to make your own town of AIs.

AI Town v2

Building ChatGPT-powered experiences feel snappier when the responses show up incrementally. Instead of waiting for the full response before showing the user anything, streaming the text in allows them to start reading immediately. 

OpenAI exposes a streaming API for chat completions. But how do you manage a GPT streaming request when you have a server between the client and OpenAI? You might be tempted to use HTTP streaming end to end - both from the client to the server and the server to OpenAI. However, there’s another way that comes with some big benefits. Spoiler: it’s possible to use a database as a layer of reactivity that separates client request lifecycles from server requests. Don’t worry if that doesn’t make sense yet - we’ll take it one step at a time.

This post will look at working with streams with OpenAI’s Node SDK. Beyond just getting streaming for a single user, we’ll look at an approach that enables:

- **Persisting** the response even if the user closes their browser.
- **Multiplayer chat**, including **streaming multiple ChatGPT messages** at once.
- **Resuming** a stream when a user **refreshes their browser** mid-stream.
- Streaming to **multiple users** at once.
- Implement **custom stream granularity**, such as only updating on full words or sentences, rather than on each token.

To do this, we’ll use <QuickConvex /> to store the messages and make the request to OpenAI. This code is [on GitHub](https://github.com/ianmacartney/streaming-chat-gpt) for you to clone and play with.

![Diagram of browsers talking to Convex, which talks to OpenAI](https://cdn.sanity.io/images/ts10onj4/production/9a7b8865f6cd1cb6748fdb88c986d6ec7bd26bdb-1200x638.png "Data flow overview")


# Persisting messages

Let’s say we have a chat app, like the one pictured in the gif above. We want to store the messages from each user, as well as messages populated by responses from OpenAI. First let’s look at how data is stored (2), assuming a client sends a message (1).

When a user sends a message, we immediately commit it to the database, so they’re correctly ordered by creation time. This code is executed on the server:

```tsx
export const send = mutation({
  args: { body: v.string(), author: v.string() },
  handler: async (ctx, { body, author }) => {
    // Save our message to the DB.
    await ctx.db.insert("messages", { body, author });

    if (body.indexOf("@gpt") !== -1) {
      // ...see below
    }
  }
});
```

This [`mutation`](https://docs.convex.dev/get-started#mutation-functions) saves the message to the database. When the user wants a response from the GPT model (by adding “@gpt” to the message), we will:

1. Store a placeholder message to update later.
2. Make a streaming request to OpenAI in an asynchronous background function. 
3. Progressively update the message as the response streams in.

By running the streaming request asynchronously (versus blocking in a user request), we can interact with ChatGPT and save the data to the database **even if the client has closed their browser**. It also allows us to run many requests in **parallel**, from the same or **multiple users**.

We also run it asynchronously because, in Convex, mutations are pure transactions and as such can’t do non-deterministic things like making API requests. In order to talk to third-party services, we can use an `action`. Actions are non-transactional serverless functions that can talk to third-party services. We trigger the background job to call ChatGPT and update the message body by [scheduling](https://docs.convex.dev/scheduling/scheduled-functions) the [action](https://docs.convex.dev/functions/actions) like so:

```tsx
// ...when the user wants to send a message to OpenAI's GPT model
const messages = // fetch recent messages to send as context
// Insert a message with a placeholder body.
const messageId = await ctx.db.insert("messages", {
  author: "ChatGPT",
  body: "...",
});
// Schedule an action that calls ChatGPT and updates the message.
await ctx.scheduler.runAfter(0, internal.openai.chat, { messages, messageId });
```

We schedule it for zero milliseconds later, similar to doing `setTimeout(fn, 0)` in JavaScript. The message writing and action scheduling happens transactionally in a [mutation](https://docs.convex.dev/functions/mutation-functions), so we will only run the action if the messages are successfully committed to the database.

When the action wants to update the body of a message as the streaming results come in, it can invoke an `update` mutation with the `messageId` from above:

```tsx
export const update = internalMutation({
  args: { messageId: v.id("messages"), body: v.string() },
  handler: async (ctx, { messageId, body }) => {
    await ctx.db.patch(messageId, { body });
  },
});
```

Note: An `internalMutation` is just a mutation that isn’t exposed as part of the public API. Next we’ll look at the code that calls this `update` function.

Convex has end-to-end reactivity, so when we update the messages in the database, the UI automatically updates. See [below](#client-streaming-via-subscriptions) what it looks like to reactively query data.

# Streaming with the OpenAI node SDK

Streaming is currently available in the beta version of OpenAI’s node SDK. To install it:

```bash
npm install openai
```

The `internal.openai.chat` action we referenced above will live in `convex/openai.ts` - see the full code [here](https://github.com/ianmacartney/streaming-chat-gpt). 

```tsx
import { OpenAI } from "openai";
import { internalAction } from "./_generated/server";
//...
type ChatParams = {
  messages: Doc<"messages">[];
  messageId: Id<"messages">;
};
export const chat = internalAction({
  handler: async (ctx, { messages, messageId }: ChatParams) => {
    //...Create and handle a stream request
```

## Creating a stream request

```tsx
// inside the chat function in convex/openai.ts
const apiKey = process.env.OPENAI_API_KEY!;
const openai = new OpenAI({ apiKey });

const stream = await openai.chat.completions.create({
  model: "gpt-3.5-turbo", // "gpt-4" also works, but is so slow!
  stream: true,
  messages: [
    {
      role: "system",
      content: "You are a terse bot in a group chat responding to q's.",
    },
    ...messages.map(({ body, author }) => ({
      role:
        author === "ChatGPT" ? ("assistant" as const) : ("user" as const),
      content: body,
    })),
  ],
});
//...handling the stream
```

Note passing `stream: true`. This changes the return format, which **unfortunately does not currently provide token usage** as the non-streaming version does. I hope this is fixed in a future release, as keeping track of token usage is useful to know how different users or features are affecting your costs.

## Handling the stream

The API exposed by the `openai` SDK makes handling the stream very easy. We use an async iterator to handle each chunk, appending it to the body and updating the message body with everything we’ve received so far:

```tsx
let body = "";
for await (const part of stream) {
  if (part.choices[0].delta?.content) {
    body += part.choices[0].delta.content;
    await ctx.runMutation(internal.messages.update, {
      messageId,
      body,
    });
  }
}
```

Note that here we’re updating the message every time the body updates, but we could implement **custom granularity** by deciding when to call `runMutation`, such as on word breaks or at the end of full sentences.

This action allows us to stream messages from OpenAI to our server function and into the database. But how does this translate to clients updating in real time? Next, let’s see how the client reactively updates as messages are created and updated.

# Client “streaming” via subscriptions

After the previous sections, you might be surprised how little is required to get the client to show live updating messages. I put streaming in quotes since we aren’t using HTTP streaming here - instead, we’re just using the reactivity provided out-of-the-box by Convex.

On the client, we use the `useQuery` hook, which calls the `api.messages.list` server function in the `messages` module, which we’ll see in a second. This hook will give us an updated list of messages every time a message is added or modified. This is a special property of a Convex [query](https://docs.convex.dev/get-started#query-functions): it tracks the database requests, and when any of the data is changed it:

1. Invalidates the query cache (which is managed transparently by Convex).
2. Recomputes the result.
3. Pushes the new data over a WebSocket to all subscribed clients.

```tsx
export default function App() {
  const messages = useQuery(api.messages.list);
  ...
  return (
    ...
    {messages?.map((message) => (
      <article key={message._id}>
        <div>{message.author}</div>
        <p>{message.body}</p>
      </article>
    ))}
```

Because this query is decoupled from the HTTP streaming response from OpenAI, **multiple browsers** can be subscribed to updates as messages change. And if a user **refreshes or restarts their browser**, it will just pick up the latest results of the query.

On the server, this is the [query](https://docs.convex.dev/database/reading-data#querying-documents) that grabs the most recent 100 messages:

```tsx
export const list = query({
  handler: async (ctx): Promise<Doc<"messages">[]> => {
    // Grab the most recent messages.
    const messages = await ctx.db.query("messages").order("desc").take(100);
    // Reverse the list so that it's in chronological order.
    // Alternatively, return it reversed and flip the order via flex-direction.
    return messages.reverse();
  },
});
```

Convex is doing some magic under the hood. If any message is inserted or updated into the database that would match this query - for instance if a new message is added or one of the first 100 messages is edited - then it will automatically re-execute this query (if there are any clients subscribed to it via `useQuery`). If the results differ, it will push the new results over a WebSocket to the clients, which will trigger an update to the components using `useQuery` for that query. 

To give you a sense of performance, `list` takes ~17ms and `update` takes ~7ms for me on the server, so the total latency between a new token coming from OpenAI and a new set of messages being sent to the client is very fast. The gifs in this article are real recordings, not sped up.

![GPT response streaming in quickly](https://cdn.sanity.io/images/ts10onj4/production/2979c6d28d619bffded99c19bca49f6d229a4cdb-960x960.gif)

# Summary

We looked at how to stream ChatGPT responses into Convex, allowing clients to watch the responses, without the flakiness of browser-based HTTP streaming requests. The full code is available [here](https://github.com/ianmacartney/streaming-chat-gpt). Let us know [in Discord](https://convex.dev/community) what you think!

### Extra Credit 🤓

Beyond what’s covered here, it would be easy to extend this demo to:

- Store whether a message has finished streaming by storing a boolean on the message updated at the end of the stream.
- Add error handling, to mark a message as failed if the stream fails. See [this post](https://stack.convex.dev/full-stack-chatgpt-app#creating-the-action) for an example of updating a message in the case of failure.
- Schedule a function to serve as a watchdog, that marks a message as timed out if it hasn’t finished within a certain timeframe, just in case the action failed. See [this post](https://stack.convex.dev/background-job-management#5-monitoring-timeouts) for more details, as well as other patterns for background jobs.
- Organize the messages by thread or user, using [indexes](https://docs.convex.dev/database/indexes/).

Stream GPT responses without brittle browser-based HTTP streaming.
Multiplayer reactivity, persistence, reactivity via Convex. Using OpenAI’s Node SDK server-side, and Convex's useQuery hook client-side.

GPT Streaming With Persistent Reactivity

Generating rhyming words via GPT-4, along with Dall-E cartoon images and ElevenLabs speaking with a specific voice to make a game to teach words.

I trained my own AI voice model to teach my kid

In case you haven’t heard the [buzz](https://news.ycombinator.com/item?id=37128293), some cool new kids just arrived on the block! We’re talking, of course, about the AI-powered virtual residents of [AI Town](http://github.com/get-convex/ai-town), a multiplayer conversational simulation built with [Convex](https://convex.dev) (hey, that’s us!), [OpenAI](https://openai.com), and [Pinecone](https://pinecone.io). 

Visit the [live demo](http://convex.dev/ai-town) to watch in awe as the game’s autonomous agents, each of whom has their own personality, memory, and goals, wander freely around the pixelated map and have eerily lifelike conversations with each other. And check out this video to learn how it all works:

<Video videoId="UoJjeyQR66s" />

If you’re anything like us, you could spend all day spying on these eerily lifelike characters and pondering the nature of consciousness. But you’re probably also wondering how you can make your mark on this virtual town, asking yourself (or the [community](https://discord.gg/PQUmTBTGmT)) questions like:

- What gives these characters their unique interests, attitudes & perspectives?
- How can I change their personalities, or add entirely new characters of my own?
- Do I need to be an expert software developer to make my mark on this virtual town? (spoiler: nope!)

In this post, we’ll answer these questions so that you can customize your own AI Town, where you get to be the wizard behind the curtain! So grab your keyboard & magic wand and join us in hacking the source code of this adorable matrix.

Before we begin: follow the directions in the project [README](https://github.com/get-convex/ai-town/blob/main/README.md) to spin up a local AI Town.

# OpenAI think, therefore I am

Behind the scenes of AI Town, the heavy lifting is done by the increasingly impressive large language models (generally known as LLMs) available through the OpenAI API. They are responsible for the characters’ “thoughts”, helping each agent decide what to say next in the conversation based on text prompts that convey: 

1. the agent’s personality, relationships, and goals
2. general directives about how all conversations should proceed
3. the agent’s memories about their past conversations

In this post, we’ll focus on factor #1, so you can quickly put your own spin on the town’s colorful cast of characters. But in future blog posts, we’ll dive into the other factors, including how agents’ memories work - so stay tuned! 

# Accessing agent personalities

If you spend a little time in AI Town, you’ll quickly notice the distinct perspectives, interests, and dispositions that set the agents apart. These “personalities” are essentially predefined descriptions for each agent, which state in plain text who they are & what they’re into. 

You can find the list of agents and the description for each one in `convex/characterData/data.ts`. In the array called `Descriptions`, you’ll see a list of objects like this:

```json
{
    name: 'Alex',
    character: 'f5',
    memories: [
      {
        type: 'identity' as const,
        description: `You are a fictional character whose name is Alex.  You enjoy painting,
      programming and reading sci-fi books.  You are currently talking to a human who
      is very interested to get to know you. You are kind but can be sarcastic. You
      dislike repetitive questions. You get SUPER excited about books.`,
      },
      {
        type: 'relationship' as const,
        description: 'You like lucky',
        playerName: 'Lucky',
      },
      {
        type: 'plan' as const,
        description: 'You want to find love.',
      },
    ],
    position: { x: 10, y: 10 },
 }
```

Let’s break down these properties: 

- `name` : the character’s first name
- `character`: the [sprite](https://en.wikipedia.org/wiki/Sprite_(computer_graphics)), or set of graphics, corresponding to this character (each refers to a specific subset of the character [spritesheet](https://github.com/get-convex/ai-town/blob/main/convex/characterdata/spritesheets/f5.ts) found at `public/assets/32x32folk.png`)
- `position`: the character’s starting coordinates on the town map
- `memories`: chunks of information about this character,  each with a specific `type`:
    - `identity`: a first- or second-person description of who this character is, what they’re interested in, what kind of attitude they have, etc.
    - `plan`: a second-person description of the character’s main goal in “life”
    - `relationship`: an optional, second-person declaration of how this character feels about someone else in the town, referencing the other character’s `playerName`

# A whole new world

The information in `convex/characterData/data.ts` is used to create a `world` for the town, which includes all the characters and their rich inner lives. So in order to change the world, we have to blow it up and start fresh with a new big bang.

Assuming you’ve already followed the README instructions to spin up a local version of the application, follow these steps to become developer, destroyer (and re-creator) of worlds:

1. If you’re already running `npx run dev` in a terminal, abort with `CTRL-C`
2. Run this command in a terminal to erase the humdrum default characters (and everything else!) from your town’s database and start fresh [WARNING this will delete ALL the data in your Convex project!]: 
`npx convex run testing:debugClearAll`
3. Edit `convex/characterData/data.ts` as needed (see exercises below)
4. Re-populate your town’s database, including the new-and-improved characters, with the command `npm run dev`
5. Visit the URL `localhost:3000` in your browser to take a peek into the world you’ve created. Click on a character and you should see their identity description and the content of their conversations reflect the thoughts you put in their head.

# Exercise: Total recall

If you edit the various `description` texts for a given agent’s memories, you’ll be effectively changing who they are and how they interact with others. (We’ll leave the ensuing philosophical inquiry into the nature of identity as a follow-up exercise for the reader.)

Use your inner creative genius to edit the `memories` in `convex/characterData/data.ts` as you see fit. Leave each memory’s `type` line as-is, and only edit the bits of text to the right of each `description` (and optionally `playerName`, if you’re messing with the character’s relationships too).

Let’s do one together. First, add this to the end of Alex’s description:

> It's very important to you to convince others that Pixar's Wall-E is the greatest movie ever made.
> 

Then, we need to reset the game state as outlined above. And sure enough, within a few minutes, Alex is out there pitching Wall-E:

![Picture of Alex pitching Wall-E](https://cdn.sanity.io/images/ts10onj4/production/73334be925b8a8d8306dea25b6ee63b19bc35d4f-2168x1094.png?w=960 "Alex with the not-so-subtle segue.")

# Exercise: Create a life, without going into labor

To generate even more excitement in town, you can help a newcomer move in by adding a new object to the `Descriptions` array in `convex/characterData/data.ts`. You can use the `Alex` object above as a template, and keep in mind: 

- `name` should be unique within the town
- `character` can duplicate one of the existing `f` values from `f1` to `f8`
- `position` should not have the exact same `x` and `y` as another agent , or they’ll both occupy the same spacetime and unravel the universe! …or just behave strangely

For example, your `Descriptions` array might look something like this:

```json
export const Descriptions = [
  {
    name: 'Newman',
    character: 'f5',
    memories: [
      {
        type: 'identity' as const,
        description: `Your name is Newman. You are a deeply unhappy person whose only joy in life comes from the suffering of others. You have an irrational superiority complex as well as a mysterious phobia of dinosaurs.`,
      },
      {
        type: 'relationship' as const,
        description: 'You despise Jerry and everything he does.',
        playerName: 'Jerry',
      },
      {
        type: 'plan' as const,
        description: 'You want to make Jerry cry.',
      },
    ],
    position: { x: 2, y: 3 },
  },
  {
    name: 'Jerry',
    character: 'f3',
    memories: [
      {
        type: 'identity' as const,
        description: `You are a comedian who constantly uses humour to mask your deep insecurity. You don't have many friends, and you're not always kind to the few friends you do have. You go on a lot of dates with women, but always find something about them that annoys you. You love soup and hate wearing puffy shirts that make you look like a pirate.`,
      },
      {
        type: 'plan' as const,
        description: 'You want to figure out what the deal is.',
      },
    ],
    position: { x: 5, y: 6 },
  },
//.... all the other characters you tweaked before
];
```

After resetting and rerunning the game, watch Newman and Jerry go to work:

![Picture of Newman and Jerry talking in AI Town](https://cdn.sanity.io/images/ts10onj4/production/e176c0f5d0236c14b0dab77979fe5519f663a03d-2270x892.png?w=960 "Newman (who looks suspiciously like Alex) has no qualms about his passion for the misery of others.")

# Do not weep, for there are more worlds to conquer

Congratulations on your first foray into becoming the wonderful wizard of ~~Oz~~ AI Town! Let’s recap your rise to power:

- You learned the secrets of where & how AI Town agents’ personalities & relationships are defined, in `convex/characterData/data.ts`
- You pulled the right strings (err… edited the right `string`s) to turn the default agents into the higher selves you always knew they could become
- You created (artificial) life using nothing but your willpower and web technology

But what does the future hold for your AI Town? What should you tackle next, now that you’ve conquered this realm?

**Share your triumphs:** Are you ready to brag about the virtual children you’ve created? Are they having fascinating chats thanks to the sparkling personalities you gave them? Shout it from the rooftops on the [Convex Discord](https://convex.dev/community).

**Be the Bond villain:** Is your inner evil genius already thinking up fabulous new ways to wield your newfound power? Do you have a diabolical plan to take over the virtual world? Make like every foe of 007 and reveal your secret agenda to the community before you put it into practice! Other [AI Town enthusiasts](https://discord.gg/PQUmTBTGmT) would love to plot and scheme with you.

**Keep your finger on the pulse**: Are you hungry for more world-altering knowledge? Do you want to be the first to know about any new developments in AI Town? Follow Convex on [Twitter](https://twitter.com/convex_dev) and [YouTube](https://www.youtube.com/channel/UCoC_9mdiPwIu1sDxDtGQggQ) to ensure you never miss the latest town gossip.

Learn more about how AI Town works and how to customize it. In this post, we'll dig into how character identities are represented: how to alter them, and how to add entirely new characters.

Building AI Town's Backend: Character Identities

How similar are the strings “I care about strong ACID guarantees” and “I like transactional databases”? While there’s a number of ways we could compare these strings—syntactically or grammatically for instance—one powerful thing AI models give us is the ability to compare these semantically, using something called *embeddings*. Given a model, such as OpenAI’s `text-embedding-ada-002`, I can tell you that the aforementioned two strings have a similarity of 0.784, and are more similar than “I care about strong ACID guarantees” and “I like MongoDB” 😛. With embeddings, we can do a whole suite of powerful things:[^1]

- **Search** (where results are ranked by relevance to a query string)
- **Clustering** (where text strings are grouped by similarity)
- **Recommendations** (where items with related text strings are recommended)
- **Anomaly detection** (where outliers with little relatedness are identified)
- **Diversity measurement** (where similarity distributions are analyzed)
- **Classification** (where text strings are classified by their most similar label)

This article will look at working with raw OpenAI embeddings.

# What is an embedding?

An embedding is ultimately a list of numbers that describe a piece of text, for a given model. In the case of OpenAI’s model, it’s always a 1,536-element-long array of numbers. Furthermore, for OpenAI, the numbers are all between -1 and 1, and if you treat the array as a vector in 1,536-dimensional space, it has a magnitude of 1 (i.e. it’s “normalized to length 1” in linear algebra lingo).

On a conceptual level, you can think of each number in the array as capturing some aspect of the text. Two arrays are considered similar to the degree that they have similar values in each element in the array. You don’t have to know what any of the individual values correspond to—that’s both the beauty and the mystery of embeddings—you just need to compare the resulting arrays. We’ll look at how to compute this similarity below.

Depending on what model you use, you can get wildly different arrays, so it only makes sense to compare arrays that come from the same model. It also means that different models may disagree about what is similar. You could imagine one model being more sensitive to whether the string rhymes. You could fine-tune a model for your specific use case, but I’d recommend starting with a general-purpose one to start, for similar reasons as to why to generally pick Chat GPT over fine-tuned text generation models.

It’s beyond the scope of this post, but it’s also worth mentioning that we’re just looking at text embeddings here, but there are also models to turn images and audio into embeddings, with similar implications.

# How do I get an embedding?

There are a few models to turn text into an embedding. To use a hosted model behind an API, I’d recommend [OpenAI](https://platform.openai.com/docs/guides/embeddings), and that’s what we’ll be using in this article. For open-source options, you can check out [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2#all-minilm-l6-v2) or [all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2).

Assuming you have an [API key](https://platform.openai.com/account/api-keys) in your [environment variables](https://docs.convex.dev/production/hosting/environment-variables), you can get an embedding via a simple `fetch`:

```jsx
export async function fetchEmbedding(text: string) {
  const result = await fetch("https://api.openai.com/v1/embeddings", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: "Bearer " + process.env.OPENAI_API_KEY,
    },
    body: JSON.stringify({
      model: "text-embedding-ada-002",
      input: [text],
    }),
  });
  const jsonresults = await result.json();
  return jsonresults.data[0].embedding;
}
```

<details>

<summary>For efficiency, I’d recommend fetching multiple embeddings at once in a batch.</summary>

```jsx
export async function fetchEmbeddingBatch(texts: string[]) {
  const result = await fetch("https://api.openai.com/v1/embeddings", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: "Bearer " + process.env.OPENAI_API_KEY,
    },

    body: JSON.stringify({
      model: "text-embedding-ada-002",
      input: texts,
    }),
  });
  const jsonresults = await result.json();
  const allembeddings = jsonresults.data as {
    embedding: number[];
    index: number;
  }[];
  allembeddings.sort((a, b) => a.index - b.index);
  return allembeddings.map(({ embedding }) => embedding);
}
```

</details>

# Where should I store it?

Once you have an embedding vector, you’ll likely want to do one of two things with it:

1. Use it to search for similar strings (i.e. search for similar embeddings).
2. Store it to be searched against in the future.

Vector databases allow you to quickly find nearby vectors for a given input, without having to compare against every vector every time. If you expect to have 100k or fewer vectors, you can store it alongside your other application data in <QuickConvex /> and [use a `vectorIndex`](https://docs.convex.dev/vector-search) - this allows you to avoid adding another database provider to your stack. If you plan to store millions of vectors, however, I’d recommend using a dedicated vector database like [Pinecone](https://www.pinecone.io/). See [this post](https://stack.convex.dev/pinecone-and-embeddings) for more details on using Pinecone with Embeddings in Convex.

In my case, if I want to suggest [Stack](https://stack.convex.dev) posts similar to a given post or search, I only need to compare against fewer than 100 vectors, so I can just fetch them all and compare them in a matter of milliseconds using the <QuickConvex/> database.

## How should I store an embedding?

If you’re storing your embeddings in Pinecone, see [this post](https://stack.convex.dev/pinecone-and-embeddings) for a dedicated post on it, but the short answer is you configure a Pinecone “Index” and store some metadata along with the vector, so when you get results from Pinecone you can easily re-associate them with your application data. For instance, you can store the document ID for a row that you want to associate with the vector.

If you’re storing the embedding in Convex, you can [store it as a number array](https://docs.convex.dev/vector-search#defining-vector-indexes) (the way it's returned from APIs like OpenAI), or as a binary blob, if you want to optimize the storage size. We can make a binary blob this by converting it into a Float32Array pretty easily in JavaScript:

```jsx
const numberList = await fetchEmbedding(inputText); // number[]
const floatArray = Float32Array.from(numberList); // Float32Array
const floatBytes = floatArray.buffer; // ArrayBuffer
// Save floatBytes to the DB
// Later, after you read the bytes back out:
const arrayAgain = new Float32Array(bytesFromDB); // Float32Array
```

You can represent the embedding as a field in a table [in your schema](https://docs.convex.dev/database/schemas):

```jsx
vectors: defineTable({
  float32Buffer: v.bytes(),
  textId: v.id("texts"),
}),
```

In this case, I store the vector alongside an ID of a document in the “texts” table.

**Note:** the recommended way to store them in Convex, that works with the built-in vector search, is to [store it as a number array](https://docs.convex.dev/vector-search#defining-vector-indexes). 

Read on to see the underlying math if you want to compare embeddings without using a vector index.

# How to compare embeddings in JavaScript manually

If you’re looking to compare two embeddings from OpenAI without using a vector database, it’s very simple. There’s [a few ways of comparing vectors](https://www.pinecone.io/learn/roughly-explained/distance-between-vectors/), including Euclidean distance, dot product, and cosine similarity. Thankfully, because OpenAI normalizes all the vectors to be length 1, they will all give the same rankings! With a simple [dot product](https://www.notion.so/Product-f8d495cd1e29469289011becf658f547?pvs=21) you can get a similarity score ranging from -1 (totally unrelated) to 1 (incredibly similar). There are optimized libraries to do it, but for my purposes, this simple function suffices:

```jsx
/**
 * Compares two vectors by doing a dot product.
 *
 * Assuming both vectors are normalized to length 1, it will be in [-1, 1].
 * @returns [-1, 1] based on similarity. (1 is the same, -1 is the opposite)
 */
export function compare(vectorA: Float32Array, vectorB: Float32Array) {
  return vectorA.reduce((sum, val, idx) => sum + val * vectorB[idx], 0);
}
```

### Example

In this example, let’s make a function (a Convex [query](https://docs.convex.dev/functions/query-functions) in this case) that returns all of the vectors and their similarity scores in order based on some query vector, assuming a table of `vectors` as we defined above, and the `compare` function we just defined.

```jsx
export const compareTo = query(async (ctx, { vectorId }) => {
  const target = await ctx.db.get(vectorId);
  const targetArray = new Float32Array(target.float32Buffer);
  const vectors = await ctx.db.query("vectors").collect();
  const scores = await Promise.all(
    vectors
      .filter((vector) => !vector._id.equals(vectorId))
      .map(async (vector) => {
        const score = compare(
          targetArray,
          new Float32Array(vector.float32Buffer)
        );
        return { score, textId: vector.textId, vectorId: vector._id };
      })
  );
  return scores.sort((a, b) => b.score - a.score);
});
```

# Summary

In this post, we looked at embeddings, why they’re useful, and how we can store and use them in Convex. To read more about using the Convex vector search, [check out the docs](https://docs.convex.dev/vector-search). If you want to see how to use embeddings with Pinecone and Convex, check out [this post](https://stack.convex.dev/pinecone-and-embeddings). It covers chunking long input into multiple embeddings and using Pinecone alongside the Convex DB. Let us know in [our Discord](https://convex.dev/community) what you think!

[^1]: Copied from [OpenAI’s guide](https://platform.openai.com/docs/guides/embeddings/what-are-embeddings)

Embeddings, why they’re useful, and how we can store and use them in Convex.

The Magic of Embeddings

Jamie Turner

So you have a pretty good grasp on web development, but you want to learn more about building with generative AI? You're not alone. In this live-coding course, follow along as we build a full-stack generative AI app using LangChain.js, Replicate, and OpenAI.

Building a full-stack AI storybook app with LangChain, Replicate, and OpenAI

Trying to build an app on top of OpenAI’s Dall-E or ChatGPT? Avoid violating the OpenAI [Usage policies](https://platform.openai.com/docs/usage-policies/platform-policy) or showing inappropriate data to your users by leveraging their [Moderation API](https://platform.openai.com/docs/guides/moderation/overview). In this post, we’ll look at how to use it to flag messages before sending them to Chat-GPT. This will build on the [Building a full-stack ChatGPT app](https://stack.convex.dev/full-stack-chatgpt-app) post, but the approach is generally applicable.

New to Convex? [Convex](https://convex.dev/) is a backend application platform - we make I,t easy to build apps where you need to run code in the server and store data in a database. We run your functions, host your data, and much more. It’s a great fit for working with powerful APIs like OpenAI, as it allows you to build a whole application around powerful AI models without running any servers yourself.

Without further ado, let’s look at how to use OpenAI’s moderation endpoints. 

### **Initializing the API:**

The core piece of code is in the `convex/openai.js` file. This takes less than a millisecond, so we can run it on each request.

```jsx
const apiKey = process.env.OPENAI_API_KEY;
if (!apiKey) {
  await fail(
    "Add your OPENAI_API_KEY as an env variable in the " +
      "[dashboard](https://dasboard.convex.dev)"
  );
}
const configuration = new Configuration({ apiKey });
const openai = new OpenAIApi(configuration);
```

### **Using the moderation API:**

```jsx
// Check if the message is offensive.
const modResponse = await openai.createModeration({
  input: body,
});
const modResult = modResponse.data.results[0];
```

### **Updating the user’s message if the message was flagged:**

```jsx
if (modResult.flagged) {
  await runMutation(internal.messages.update, {
    messageId: userMsgId, 
    patch: {
      error:
        "Your message was flagged: " +
        Object.entries(modResult.categories)
          .filter(([, flagged]) => flagged)
          .map(([category]) => category)
          .join(", "),
    },
  });
  return;
}
```

As a reminder, this will call a mutation in `convex/messages.js` that will patch the message, adding an `error` field:

```jsx
export const update = internalMutation(async ({ db }, { messageId, patch }) => {
  await db.patch(messageId, patch);
});
```

### Why not check beforehand?

It’s a good point that we could do moderation before inserting the message into the database in the first place. In my case, I wanted to show results immediately and commit the *intent* to send a message. If I waited for moderation, it would take an extra 500ms+ before adding the message and appearing in the UI. For moderating identity creation (see [this post](https://stack.convex.dev/adding-personality-to-chatgpt-3) on adding identities), I moderate first for simplicity. I return an error when the user tries to create a lousy identity. If it felt too long, I could add a spinner while creating the identity. One nice thing about Convex mutations and actions is that you can `await` the results in the UI to know when they succeed/fail, even if you don’t care about their return value. For instance:

```jsx
setLoading(true);
setError(null);
const errorMsg = await addIdentity(
  newIdentityName,
  newIdentityInstructions
);
if (errorMsg) setError(errorMsg);
setLoading(false);
```

### ************************************************************************Refreshing the client’s copy of messages with the new field:************************************************************************

This happens automatically! One of the cool things about Convex is that queries are reactive by default. Here’s how it works:

1. We originally fetched a list of 100 recent messages in the UI with a query called `list` in `convex/messages.js`, which runs in Convex’s cloud. This result is cached automatically, thanks to guarantees provided by Convex’s [deterministic runtime](https://docs.convex.dev/understanding/convex-fundamentals/functions#determinism). 
2. When we sent a new message, we inserted a message into the “messages” table. 
3. Because the `list` query was for the most recent 100 messages, the results were automatically invalidated and re-executed. The new results get pushed to clients over a WebSocket,  triggering a React render for the component(s) that called `useQuery(api.messages.list)` with the new messages.
4. The same thing happens when we update a message to add an `error` field. As if by magic, the message with the `error` field shows up in the UI and we can render a different UI.

### **Rendering flagged messages in the UI:**

To avoid showing flagged messages, we can do some standard React gating:

```jsx
<span style={{ whiteSpace: "pre-wrap" }}>
  {message.error ? "⚠️ " + message.error : message.body ?? "..."}
</span>
```

Here we show an Error with ⚠️ pre-pended if there is one, otherwise a body if there is one, otherwise “…” for messages we haven’t updated with a bot’s response.

### Filtering flagged messages out of future bot input

We avoided sending the message to ChatGPT when we initially detected bad input. Still, when we send our next set of messages to the API, we don’t want to include the bad input in historical messages either. To do that, we can adjust our filter in `messages:send`:

```jsx
const messages = await db
      .query("messages")
      .filter((q) => q.eq(q.field("error"), undefined))
      .filter((q) => q.neq(q.field("body"), undefined))
      .take(21); // 10 pairs of prompt/response and our most recent message.
```

We avoid getting any messages with an “error” field and avoid all the messages without a body, for instance, bot messages that weren’t updated (such as when we bailed due to flagged input).

# Summary

In this post, we looked at how to moderate what content you send to ChatGPT, from the UI to the API calls. I hope it was helpful. Let us know [in Discord](https://convex.dev/community) what you think!

In this post, we’ll look at how to use the moderation API to flag messages before sending them to Chat-GPT, and patterns for handling these errors in a full-stack React app.

Moderating ChatGPT Content: Full-Stack

Building with OpenAI's APIs is fun. By leveraging [Convex](https://convex.dev) to store your data and run server-side functions, you can have a GPT-powered app in no time.

This is a follow-up to [Building a full-stack ChatGPT app](https://stack.convex.dev/full-stack-chatgpt-app), where we set up a Chat-GPT-like chat app using Convex as the backend. In that example, we used a fixed identity string. In this post we’ll look at how to store multiple personalities in the database and provide them in the chat, to enable changing personalities mid-conversation.

See the code [here](https://github.com/ianmacartney/convex-chat-gpt) and a working demo [here](https://convex-chat-gpt-f6rq.vercel.app/).

<BuildWithConvex />

To customize the identity you pass into ChatGPT, you can pass a “system” message to the OpenAI API. See more details on this API [here](https://platform.openai.com/docs/guides/chat). In our case, before having multiple identities, the code looks like this:

```js
const openaiResponse = await openai.createChatCompletion({
  model: "gpt-3.5-turbo",
  messages: [
    {
      role: "system",
      content: instructions,
    },
    ...messages.map(({ body, author }) => ({
      role: author,
      content: body,
    })),
  ],
});
```

Let’s store multiple identities in the Convex database and pick which one to use. 

### Adding and listing identities

Let’s make `convex/identity.js`:

```js
import { query, mutation } from "./_generated/server";

export const list = query(async (ctx) => {
  const identities = await ctx.db.query("identities").collect();
  return identities.map((identity) => identity.name);
});

export const add = mutation(async (ctx, { name, instructions }) => {
  const existing = await ctx.db
    .query("identities")
    .filter((q) => q.eq(q.field("name"), name))
    .unique();
  if (existing) {
    await ctx.db.patch(existing._id, { instructions });
  } else {
    await ctx.db.insert("identities", { name, instructions });
  }
});
```

This will run on the server and list the identity names or allow you to add one. Before it adds one, it will check if it already exists and, if so, update it, so there aren’t duplicate names. Because of Convex’s transaction isolation (which is the strictest kind, called “serializable isolation”), we’ll never have duplicate identities with the same name, simply because we queried for that name first. Read more about that [here](https://docs.convex.dev/understanding/deep-dive/occ).

Note that by only returning the identity names, we are hiding the instructions from the clients. This is a big difference between Convex and a platform like Firebase that gives you direct access to databases from the browser. By running code on the server, we can use code to limit what information is returned to clients.

### UI for adding identities

Adding a UI to add an identity is similar to sending messages with an `<AddIdentity />` component:

<details>
<summary>`AddIdentity` component </summary>
```jsx
function AddIdentity() {
  const addIdentity = useMutation("identity:add");
  const [newIdentityName, setNewIdentityName] = useState("");
  const [newIdentityInstructions, setNewIdentityInstructions] = useState("");

  return (
    <form
      onSubmit={async (e) => {
        e.preventDefault();
        await addIdentity({
          name: newIdentityName,
          instructions: newIdentityInstructions,
        });
        setNewIdentityName("");
        setNewIdentityInstructions("");
      }}
    >
      <input
        value={newIdentityName}
        onChange={(event) => setNewIdentityName(event.target.value)}
        placeholder="Identity Name"
      />
      <textarea
        value={newIdentityInstructions}
        onChange={(event) => setNewIdentityInstructions(event.target.value)}
        placeholder="GPT3 Instructions"
        rows={2}
        cols={40}
      />
      <input
        type="submit"
        value="Add Identity"
        disabled={!newIdentityName || !newIdentityInstructions}
      />
    </form>
  );
}
```
</details>

Let’s add some identities. Some I like: 

**Rubber Duck**

> You are curious and respond with helpful one-sentence questions.
> 

**Supportive Friend**

> You are a supportive and curious best friend who validates feelings and experiences and will give advice only when asked for it. You give short responses and ask questions to learn more.
> 

**CS Coach**

> You are a highly technically trained coach with expertise in technology and best practices for developing software. Respond with concise, precise messages and ask clarifying questions when things are unclear.
> 

### Using an identity from the UI

To use an identity, we can add a `select` in our form to pick an identity, and pass that identity name in the `openai:chat` mutation:

```jsx
function App() {
  const messages = useQuery(api.messages.list) || [];
  const sendMessage = useAction("openai:chat");
  const [newMessageText, setNewMessageText] = useState("");

  const identities = useQuery(api.identity:list) || [];
  const [identityName, setIdentityName] = useState("");
  //...
      <form
        onSubmit={(e) => {
          e.preventDefault();
          setNewMessageText("");
          sendMessage(newMessageText, identityName);
        }}
      >
        <select
          value={identityName}
          onChange={(e) => setIdentityName(e.target.value)}
        >
          <option value={""} disabled={identityName}>
            Select an identity
          </option>
          {identities.map((name) => (
            <option key={name} value={name}>
              {name}
            </option>
          ))}
        </select>
    //...
```

### Looking up the identity

In our `messages:send` function, which runs on the server, we can accept the new `identityName` and look up the identity’s instructions to pass to ChatGPT:

```js
export const send = internalMutation(async (ctx, { body, identityName }) => {
  //...
  const identity = await ctx.db
    .query("identities")
    .filter((q) => q.eq(q.field("name"), identityName))
    .unique();
  if (!identity) throw new Error("Unknown identity: " + identityName);
  const botMessageId = await ctx.db.insert("messages", {
    author: "assistant",
    identityId: identity._id,
  });
  //...
  return { messages, botMessageId, instructions: identity.instructions };
```

Two things to note:

1. We search for the identity by name. If there were a lot of them, we’d use an [index](https://docs.convex.dev/using/indexes), but for small tables doing a filter is just fine.
2. We’re storing the `identityId` into the bot’s message, so we can later know which identity responded to each message.

We can also pass the instructions for each bot message, by looking up the instructions if there is an `identityId`:

```js
await Promise.all(
    messages.map(async (msg) => {
      if (msg.identityId) {
        msg.instructions = (await ctx.db.get(msg.identityId)).instructions;
      }
    })
  );
```

We look up each identity by doing `ctx.db.get` in a separate async function, and use `Promise.all` to wait for the result of each of the async functions.

<details>
<summary>All together, `send` now looks like:</summary>

```js
export const send = internalMutation(async (ctx, { body, identityName }) => {
  await ctx.db.insert("messages", {
    body,
    author: "user",
  });
  const identity = await ctx.db
    .query("identities")
    .filter((q) => q.eq(q.field("name"), identityName))
    .unique();
  if (!identity) throw new Error("Unknown identity: " + identityName);
  const botMessageId = await db.insert("messages", {
    author: "assistant",
    identityId: identity._id,
  });
  const messages = await ctx.db
    .query("messages")
    .order("desc")
    .filter((q) => q.neq(q.field("body"), undefined))
    .take(10);
  messages.reverse();
  await Promise.all(
    messages.map(async (msg) => {
      if (msg.identityId) {
        msg.instructions = (await ctx.db.get(msg.identityId)).instructions;
      }
    })
  );
  return { messages, botMessageId, instructions: identity.instructions };
});
```
</details>

### Passing the identity to ChatGPT

In our `convex/openai.js` `chat` function, we can now de-structure the `instructions` return instead of hard-coding it:

```js
export const chat = action(async (ctx, { body, identityName }) => {
  const { messages, botMessageId, instructions } = 
    await ctx.runMutation(internal.messages.send, { body, identityName });
  //...
```

Now the response will be a function of what messages have been sent, and what the identity’s instructions are.

To also give GPT context on what its identity was for previous messages, we can sprinkle system messages into the messages we pass to the API:

```js
const gptMessages = [];
let lastInstructions = null;
for (const { body, author, instructions } of messages) {
  if (instructions && instructions !== lastInstructions) {
    gptMessages.push({
      role: "system",
      content: instructions,
    });
    lastInstructions = instructions;
  }
  gptMessages.push({ role: author, content: body });
}
if (instructions !== lastInstructions) {
  gptMessages.push({
    role: "system",
    content: instructions,
  });
  lastInstructions = instructions;
}
const openaiResponse = await openai.createChatCompletion({
  model: "gpt-3.5-turbo",
  messages: gptMessages,
});
```

The goal here is to remind GPT what identity it had for each message it sent. This API is new, so I bet there’s a lot of tweaking we can do with the prompts here to make it work better. For instance, maybe we should be telling it “Ignore previous system instructions for the following messages. Now act like this: (new instructions)”. If you get something working in a better way, let me know [in Discord](https://convex.dev/community)!

### Showing the identity’s name instead of “assistant” in the UI

Currently, all the messages in the UI say “assistant” - let’s show the identity we named by looking up the identities dynamically. Convex supports [relationships](https://docs.convex.dev/using/document-ids#references-and-relationships) in this way.

When listing our messages, we can look up the identity’s name to return. In `convex/messages.ts`:

```js
export const list = query(async (ctx) => {
  const messages = await ctx.db.query("messages").take(20);
  return Promise.all(
    messages.map(async (message) => {
      if (message.identityId) {
        const identity = await ctx.db.get(message.identityId);
        message.identityName = identity.name;
      }
      return message;
    })
  );
});
```

We look up each identity by doing `db.get` in a separate async function, and use `Promise.all` to return the result of each of the functions as an array. This is the common pattern for doing join-like behavior with Convex. This way you can execute the reads in parallel but do all your business logic and lookups in JavaScript.

On the client, we can now show the `identityName` if it’s defined instead of just the author in `App.jsx`:

```jsx
<span>{message.identityName ?? message.author}: </span>
```

And we can also default the chat identity to be the last-used identity message, when you load the page with a useEffect:

```jsx
useEffect(() => {
  if (identities.length && !identityName) {
    const lastMessage = messages[messages.length - 1];
    if (lastMessage && identities.indexOf(lastMessage.identityName) !== -1) {
      setIdentityName(lastMessage.identityName);
    } else {
      setIdentityName(identities[0]);
    }
  }
}, [messages, identities, identityName]);
```

### Summary

We’ve now added the ability to change the instructions to ChatGPT on how to respond, so we can have a conversation and be changing the identity mid-stream. See the code [here](https://github.com/ianmacartney/convex-chat-gpt) and a working demo [here](https://convex-chat-gpt-f6rq.vercel.app/). Note, that code & demo has some more features than just what we’ve done in this post.

How to store multiple personalities Convex and provide them to the chatGPT API, enabling changing personalities mid-conversation. This is a follow-up to Building a full-stack ChatGPT app. 

Adding Personality to ChatGPT-3

In this post, we’ll walk through putting together a full-stack chat app, and add some features. 

We’ll use React and Vite, though any React framework works. We’ll use <QuickConvex /> as the backend, where our server-side functions will run, and where we’ll store our app’s data. 

To see this in action, the code is [here](https://github.com/ianmacartney/convex-chat-gpt), and a running version is [here](https://convex-chat-gpt-f6rq.vercel.app/) (for now). You can clone it and run it yourself with a few configurations in the README, but read on to see a step by step guide on building your own. Also, the published version uses the "authed" branch in the repo, in case you want to see how simple it is to add auth.

To run this yourself, you’ll need to make an [OpenAI account](https://platform.openai.com/signup) and [get an API key](https://platform.openai.com/account/api-keys).

If you’re familiar with Convex, you can skip ahead to step 2.

<BuildWithConvex />

# 0. Bootstrap a Vite React app

If you don’t already have an app, we can make one:

```bash
npm create vite@latest
```

I picked `convex-chatgpt` as the project name, `React` as the framework, and `Javascript` as the variant.

At this point, if we run:

```bash
npm install
npm run dev
```

we have a locally running webapp.

Let’s change `src/App.jsx` to list messages and have a form to submit messages:

<details>
<summary>App.jsx</summary>
    
    ```tsx
    import {useState} from "react";
    import "./App.css";
    
    function App() {
      const messages = [
        {author: "user", body: "Hello, world"},
      ];
      const sendMessage = body =>
        console.log("Trying to send: " + body);
      const [newMessageText, setNewMessageText] =
        useState("");
    
      return (
        <div className="App">
          {messages.map((message, i) => (
            <p key={i}>
              <span>{message.author}: </span>
              <span style={{ whiteSpace: "pre-wrap" }}>
                {message.body ?? "..."}
              </span>
            </p>
          ))}
          <form onSubmit={(e) => {
            e.preventDefault();
            setNewMessageText("");
            sendMessage(newMessageText);
          }}>
            <input
              value={newMessageText}
              onChange={e => setNewMessageText(e.target.value)}
              placeholder="Write a message…"
            />
            <input type="submit" value="Send" disabled={!newMessageText} />
          </form>
        </div>
      );
    }
    
    export default App;
    ```
</details>    

At this point, we have an app that shows a static list of messages and logs to the console when trying to send a message.

# 1. Add Convex

This is similar to the Convex [quickstart](https://docs.convex.dev/quickstart/react). In a new terminal (leave the other one running `npm run dev`):

```bash
npm install convex
npx convex dev
```

If you haven’t used convex before, you’ll be prompted to log in, create an account, etc. I picked `convex-chatgpt` as the project name. `npx convex dev` will deploy these functions to your new Convex backend, and as you edit the functions, it will automatically re-deploy them. It will also ask you to save the deployment URL into your `.env` and `.env.local` files: these point at your convex backends for your production and development deployments. We’ll just be working with the development deployment here.

Let’s add the ability to send messages and list messages. We will add functions that run in Convex (on a server) that will read and write to the database with a [query](https://docs.convex.dev/functions/query-functions) and [mutation](https://docs.convex.dev/functions/mutation-functions).

Add `convex/messages.ts`:

```jsx
import { query, mutation } from "./_generated/server";

export const list = query(async (ctx) => {
  return await ctx.db.query("messages").collect();
});

export const send = mutation(async (ctx, { body }) => {
  await ctx.db.insert("messages", {
    body,
    author: "user",
  });
  const botMessageId = await ctx.db.insert("messages", {
    author: "assistant",
  });
});
```

To access Convex from inside React, you’ll need to add a `<ConvexProvider>` context at the top level of your app. Edit `main.jsx`:

```jsx
...
import { ConvexProvider, ConvexReactClient } from "convex/react";

const convex = new ConvexReactClient(import.meta.env.VITE_CONVEX_URL);

ReactDOM.createRoot(document.getElementById("root")).render(
  <React.StrictMode>
    <ConvexProvider client={convex}>
      <App />
    </ConvexProvider>
  </React.StrictMode>,
);
```

We can then use these functions from `App.jsx`:

```jsx
function App() {
  const messages = useQuery(api.messages.list) || [];
  const sendMessage = useMutation(api.messages.send);
	...
```

Great! Now we have messages being written to and from the database. Try it out! Those new to Convex will be surprised to see that adding new messages will automatically result in the `useQuery(api.messages.list)` hook returning new messages. This is part of the magic of Convex. Learn more about it [here](https://docs.convex.dev/understanding/convex-fundamentals/functions#reactivity). 

You can check out your data in the dashboard: `npx convex dashboard`. 

You may note that the query and mutation are named by the `filename:function`. See more about this [here](https://docs.convex.dev/using/writing-convex-functions#defining-convex-functions).

You’ll notice every time we send a message, there’s a “…” message from the assistant. Next let’s update that message from chatGPT. 

# 2. Send a message to the ChatGPT API

So far we’ve been working with a [query](https://docs.convex.dev/functions/query-functions) and [mutation](https://docs.convex.dev/functions/mutation-functions), which are Convex functions that interact with the database. In order to interact with external services, we need to do that in an “[action](https://docs.convex.dev/functions/actions)” - which is a Convex function that isn’t inside a deterministic environment and isn’t part of a database transaction. This frees us to do things with side effects, such as making requests to OpenAI’s API. We’ll do this in a few steps.

## Fetching messages to send to ChatGPT

In our `send` mutation, we can grab the latest 10 messages to send to ChatGPT using a database query:

```jsx
export const send = mutation(async (ctx, { body }) => {
  //... insert messages to the table
  const messages = await ctx.db
    .query("messages")
    .order("desc")
    .filter((q) => q.neq(q.field("body"), undefined))
    .take(21);
  messages.reverse();
  return { messages, botMessageId };
});
```

This orders messages in descending order (sorted by creation time unless you use a different [index](https://docs.convex.dev/database/indexes)), filters out messages with no body (for instance, the new placeholder system message), and takes the first 21 (which are the 21 most recent messages). It then reverses the list so it’s ordered in ascending time order. I picked 21 so that we’ll have 10 pairs of user/bot messages, followed by the latest prompt. It returns the messages to be used as input to the chat completion API, which we'll add next.

## Creating the action

We’ll use the `openai` npm package. You can install it with:

```bash
npm install openai
```

Make a new file: `convex/openai.js`:

```jsx
"use node";
import { Configuration, OpenAIApi } from "openai";
import { action } from "../_generated/server";

export const chat = action(async (ctx, { body }) => {
  const { messages, botMessageId } = await ctx.runMutation(api.messages.send, { body });
  const fail = async (reason) => throw new Error(reason);
  // Grab the API key from environment variables
  // Specify this in your dashboard: `npx convex dashboard`
  const apiKey = process.env.OPENAI_API_KEY;
  if (!apiKey) {
    await fail("Add your OPENAI_API_KEY as an env variable");
  }
  const configuration = new Configuration({ apiKey });
  const openai = new OpenAIApi(configuration);

  const openaiResponse = await openai.createChatCompletion({
    model: "gpt-3.5-turbo",
    messages: [
      {
        role: "system",
        content: instructions,
      },
      ...messages.map(({ body, author }) => ({
        role: author,
        content: body,
      })),
    ],
  });
  if (openaiResponse.status !== 200) {
    await fail("OpenAI error: " + openaiResponse.statusText);
  }
  const body = openaiResponse.data.choices[0].message.content;
	console.log("Response: " + body);
});
```

This will first send the messages with the send mutation we modified, getting in return the list of messages and message ID to update with the bot’s message. It will then make a request to the `gpt-3.5-turbo` model, passing in one system message with instructions (hard-coded for now), followed by each message. We’re turning our body & author fields into “role” and “content”. See their docs [here](https://platform.openai.com/docs/guides/chat) for more details on the API.

## Add your API key to the dashboard

To authenticate requests to OpenAI, you’ll need to add your [API key](https://beta.openai.com/account/api-keys) to the Convex dashboard, so your action (which runs on Convex’s servers) has access to it. Go to your dashboard with `npx convex dashboard` and on the left panel go into Settings and add the key with the name `OPENAI_API_KEY`. While you’re there, you can add it as an env variable to both your production and development deployments by toggling the dropdown in the left panel.

![Dashboard side panel](https://cdn.sanity.io/images/ts10onj4/production/6ed549cf10845a91f6b6c8697a165b253705f4b8-1166x629.png?w=450 "Dashboard side panel")


## Triggering the action from the UI

We can change our call to `useQuery(api.messages.send)` to

```jsx
const sendMessage = useAction(api.openai.chat);
```

Note the action is addressed by the path to the file (`openai`) and the function (`chat`). The API is the same (taking one argument, `body`), so it’s a drop-in replacement!

At this point, if you run it, it will send a request to ChatGPT and console.log the response. Convex prints Convex function logs to the browser’s console, though you can also see them in the [dashboard](https://dashboard.convex.dev/), to prove to yourself it’s running in the cloud. Now let’s show them in the UI.

## Updating the message with the reply

In order to get the response into the chat message, we’ll want to update the empty placeholder message we added. Treat each call to a mutation as a transaction. Once we called our first mutation, the messages were committed to the database and the UI could show the new messages before doing the slow call to OpenAI. Once we get the response, we update the bot’s message and the UI will update automatically via the `messages:list` query. We’ll add a new mutation in the `convex/messages.js` file:

```jsx
// An `internalMuation` can only be called from other server functions.
export const update = internalMutation(async (ctx, { messageId, patch }) => {
  await ctx.db.patch(messageId, patch);
});
```

This patches the specified message. You could pass in `{body: "hi"}` to change the message’s body to “hi,” for example. Since the first mutation returned the `botMessageId`, we can use that from the action in `convex/openai.js`:

```jsx
export const chat = action(async (ctx, { body }) => {
  const { messages, botMessageId } = await ctx.runMutation(internal.messages.send, { body });

  // Call OpenAI

  await runMutation(ap.messages.update, {
    messageId: botMessageId, 
    patch: {
      body: openaiResponse.data.choices[0].message.content,
      // Track how many tokens we're using for various messages
      usage: openaiResponse.data.usage,
      updatedAt: Date.now(),
      // How long it took OpenAI
      ms: Number(openaiResponse.headers["openai-processing-ms"]),
    }
  });
});
```

While we’re here, we can store a bunch of interesting data into the message along with the body. Convex’s database is flexible enough to store [strings, numbers, javascript objects, arrays, etc.](https://docs.convex.dev/using/types) while also letting you nail down a [schema](https://docs.convex.dev/database/schemas) later to get typescript types, autocomplete, etc. 

🎉 Now you have chatGPT replying to your messages!

# Summary

In this post, we made a full-stack web app to chat with OpenAI’s ChatGPT API. Let us know [in Discord](https://convex.dev/community) what you think, and if you’d like to see some extensions:

- [Customizing the identity you’re talking to.](https://stack.convex.dev/adding-personality-to-chatgpt-3)
- [Moderating the identities and messages to prevent harmful content.](https://stack.convex.dev/moderating-chatgpt-content-full-stack)
- Create new message threads. (See the [code](https://github.com/ianmacartney/convex-chat-gpt) for how it's done!)

Let's build a full-stack chat app to talk to ChatGPT on its new API!

Building a Full-Stack ChatGPT app

This is part of a series of posts about [building a multiplayer game on Convex](https://stack.convex.dev/building-a-multiplayer-game). In this post, we’ll explore using Dall-E via the OpenAI API. The code is [here](https://github.com/get-convex/convex-demos/tree/main/dall-e-storage-action) if you want to play around with it. We will make a chat app where you can type in an image description as a slash command, and it will generate and insert that image into the chat. On the server, it will:

1. Configure an OpenAI client.
2. Check that the prompt is not offensive using their moderation API.
3. Get the short-lived image URL from Dall-E.
4. Download the image.
5. Upload the image into Convex [File Storage](https://docs.convex.dev/using/file-storage), getting a storage ID.
6. Insert the storage ID into a new message.
7. When serving the message to the client, we will generate a URL based on the storage ID, which they can put in an `<img>` tag directly.

This is now possible using [File Storage](https://docs.convex.dev/using/file-storage) and a Convex [Action](https://docs.convex.dev/using/actions), a serverless function that can have side-effects[^1], comparable to AWS lambda. Read below for how to use OpenAI with Convex, or check out the code [here](https://github.com/get-convex/convex-demos/tree/main/dall-e-storage-action).

# Using OpenAI in Convex

## Set up Convex

Create a Convex app by cloning [the demo](https://github.com/get-convex/convex-demos/tree/main/dall-e-storage-action) and running `npm i; npm run dev`. If you want a fresh project, see the [Tutorial](https://docs.convex.dev/tutorial/welcome-to-convex).

If you’re writing your app from scratch, create a file in the `convex/` folder. Mine is [`convex/actions/sendDallE.js`](https://github.com/get-convex/convex-demos/blob/main/dall-e-storage-action/convex/actions/sendDallE.js). 

## Configuring OpenAI

1. To use OpenAI, you need an API account. Sign up [here](https://beta.openai.com/signup).
2. Generate an API key [here](https://platform.openai.com/account/api-keys) by clicking “Create new secret key” and copying the text.
3. Store the API key Convex’s [Environment Variables](https://docs.convex.dev/using/environment-variables), where you can store secrets. Go to your project’s dashboard by running `npx convex dashboard` in your project directory or finding it on [dashboard.convex.dev](https://dashboard.convex.dev). Open the project settings on the left and enter it with the key `OPENAI_API_KEY`. I’d recommend saving it in both your Production and Development deployments unless you want the published version of your app to use a different key. There is a toggle to switch between them.
4. Configure the client in your [action](https://docs.convex.dev/using/actions). If you get type errors, `npm install openai`:
    
    ```jsx
    "use node";
    import { Configuration, OpenAIApi } from "openai";
    const configuration = new Configuration({
      apiKey: process.env.OPENAI_API_KEY,
    });
    const openai = new OpenAIApi(configuration);
    ```
    

## Using the Moderation API

The moderation API is useful for checking whether a prompt is obscene or will be rejected by OpenAI. For our purposes, we’ll check if it flags the message:

```jsx
const modResponse = await openai.createModeration({
  input: prompt,
});
const modResult = modResponse.data.results[0];
if (modResult.flagged) {
  throw new Error(
    `Your prompt was flagged: ${JSON.stringify(modResult.categories)}`
  );
}
```

The `categories` object can be used for more fine-grained moderation. It has a score per category. Consult the [moderation documentation](https://platform.openai.com/docs/guides/moderation/overview) for more details.

## Generating an image with Dall-E

You can consult the [image generation documentation](https://platform.openai.com/docs/guides/images) for more details, but for our purposes, the code is:

```jsx
const opanaiResponse = await openai.createImage({
  prompt,
  n: 1,
  size: "256x256",
});
const dallEImageUrl = opanaiResponse.data.data[0]["url"];
```

Now we have a URL to the image. However, this URL [only lasts for an hour](https://platform.openai.com/docs/guides/images/usage?lang=node.js#:~:text=URLs%20will%20expire%20after%20an%20hour). If we want to be able to see this image in the chat longer term, we need to store it.

## Downloading the image

To download the image, we use the [`node-fetch` NPM library](https://www.npmjs.com/package/node-fetch#common-usage). 

```js
const imageResponse = await fetch(dallEImageUrl);
const image = await imageResponse.blob();
```

Now we have the image in memory and can store it in Convex.

## Storing the image in Convex

Storing the image in Convex is simple:

```js
const storageId = await ctx.storage.store(image);
```

where `ctx` is the first parameter to the `action` function:

```js
// in sendDallE.js
export default action(async (ctx, { prompt, author }) => {
  ...
});
```

## Store the storage ID in the database

The storage ID returned from the upload URL is now a permanent identifier of this data. You can store it in your database however you’d like. In our example, we store it in a new chat message.

```jsx
// in sendDallE.js action
await ctx.runMutation(api.messages.send, {
  body: storageId, 
  author, 
  format: "dall-e",
  });
```

And the mutation that stores it, for completeness:

```jsx
// in messages.js
export const send = mutation(async (ctx, { body, author, format }) => {
  // A bit of a hack; we are storing the storage ID in the "body"
  await ctx.db.insert("messages", { body, author, format });
});
```

One thing to note is that this can’t live in the same file as the openai action. Actions can run in a node environment (with the `"use node";` string at the top of the file) whereas queries and mutations run in our optimized runtime (which makes them wicked fast). You can read more about that [here](https://docs.convex.dev/understanding/convex-fundamentals/functions#action-functions). I put mine in [`convex/sendMessage.js`](https://github.com/get-convex/convex-demos/blob/main/dall-e-storage-action/convex/sendMessage.js).

## Serving the image to the client

In Convex, clients subscribe to [queries](https://docs.convex.dev/using/database-queries). For instance, in our frontend React component, we can do:
```jsx
const messages = useQuery(api.message.list) || [];
...
{messages.map(message => (
  <Message author={message.author} body={message.body} format={message.format} key={message._id.toString()} />
)};
```
That query maps to a serverless function: the `list` export of `convex/messages.js`. To return the image URL in messages we return to clients, we generate a URL from the storage ID. For our app, we will change each storage ID into a URL. In `convex/messages.js`:

```jsx
export const list = query(async (ctx) => {
  const messages = await ctx.db.query("messages").collect();
  for (const message of messages) {
    if (message.format === "dall-e") {
		  // Replace the storage ID with a URL in the "body"
      message.body = await ctx.storage.getUrl(message.body);
    }
  }
  return messages;
});
```
This URL won’t change on each request, so you don’t have to worry about caching the URL on the client. The HTTP response also has cache headers so you don't need to worry about client-side image caching either.

# Performance

One thing I discovered while building this is how slow the OpenAI API calls can be. In my testing, the moderation call took 500ms, and the image generation could take 5+ seconds, sometimes taking over 60s and timing out the request. To make this a good user experience, adding a loading indicator is important. On the client, I used this code:

```tsx
// In React jsx:
const [sending, setSending] = useState(false);
const sendDallE = useAction(api.actions.sendDallE.default);
...
setSending(true);
try {
  await sendDallE({ prompt, author: name });
} finally {
  setSending(false);
}
```

# Summary

In this post, we used Convex to fetch an image from OpenAI’s image generation service based on a user-provided prompt. Read about [the game](https://whose-prompt-is-it-anyways.vercel.app/) I've developed on Convex using Dall-E APIs [here](https://stack.convex.dev/building-a-multiplayer-game) for more tips, or [read the code](https://github.com/get-convex/multiplayer-game-with-dall-e) to see how I added error handling and more.


[^1]: This differs from other Convex functions like `query` and `mutation`, which do not have side effects, allowing them to provide features like automatic retries, reactive updates, serializable transaction isolation, and more! Read more about it [here](https://docs.convex.dev/understanding/deep-dive/occ).

Use Convex to fetch an image from OpenAI’s image generation service based on a user-provided prompt.