Last week, I launched [notesGPT](https://usenotesgpt.com/), a full-stack voice note taking app that got 35,000 visitors, 7,000 users, and over 1,000 GitHub stars so far in the last week. It allows you to record a voice note, transcribes it uses [Whisper](https://github.com/openai/whisper), and uses Mixtral via [Together](https://together.ai/) to extract action items and display them in an action items view. It’s also [fully open source](https://github.com/nutlope/notesgpt) and comes equipped with authentication, storage, vector search, action items, and is fully responsive on mobile for ease of use.

I’m going to to walk you through exactly how I built it.

<GitHubRepo url="https://github.com/nutlope/notesGPT" />

## Architecture and tech stack

This is a quick diagram for the architecture. We’ll be discussing each piece in more depth and also showing code examples as we go.

![The frontend calls the Convex server functions, which communicate with the DB, kick off background actions, etc.](https://cdn.sanity.io/images/ts10onj4/production/712cd6587c252973538b1a929cf5a365c27041fc-1890x1373.png?w=1000)

Here’s the overall tech stack I used:

- [Convex](https://convex.dev/) for the database and cloud functions
- Next.js [App Router](https://nextjs.org/docs/app) for the framework
- [Replicate](https://replicate.com/) for Whisper transcriptions
- [Mixtral](https://mistral.ai/news/mixtral-of-experts/) with [JSON mode](https://docs.together.ai/docs/json-mode) for the LLM
- [Together.ai](http://Together.ai) for inference and embeddings
- [Convex File Storage](https://docs.convex.dev/file-storage) for storing voice notes
- [Convex Vector search](https://docs.convex.dev/vector-search) for vector search
- [Clerk](https://clerk.dev/) for user authentication
- [Tailwind CSS](https://tailwindcss.com/) for styling

## Landing Page

The first piece of the app is the landing page you see when you navigate to notesGPT.
![Landing page with an image of it on mobile & web, along with a Get Started button](https://cdn.sanity.io/images/ts10onj4/production/bc5ea0a4318de8aebc2447460017fa04508d2147-2000x1839.png?w=1000)

The first things users see is this landing page which along with the rest of the app, was built with Next.js and with Tailwind CSS for styling. I enjoy using Next.js since it makes it it easy to spin up web apps and just write React code. Tailwind CSS is great too since it allows you to iterate quickly on your web pages while staying in the same file as your JSX.

## Authentication with Clerk and Convex

When the user clicks either of the buttons on the homepage, they get directed to the sign in screen. This is powered by Clerk, an easy authentication solution that integrates well with Convex, which is what we’ll be using for our entire backend including cloud functions, database, storage, and vector search.

![Clerk sign in UI](https://cdn.sanity.io/images/ts10onj4/production/fd2b5543ce06a9f7e23041e4fbb259a5a5f2999a-2000x1161.png?w=1000)

Clerk and Convex are both easy to setup. You can simply create an account on both services, install their npm libraries, run `npx convex dev` to setup your convex folder, and create a `ConvexProvider.ts` file as seen below to wrap your app with.

```tsx
'use client';

import { ReactNode } from 'react';
import { ConvexReactClient } from 'convex/react';
import { ConvexProviderWithClerk } from 'convex/react-clerk';
import { ClerkProvider, useAuth } from '@clerk/nextjs';

const convex = new ConvexReactClient(process.env.NEXT_PUBLIC_CONVEX_URL!);

export default function ConvexClientProvider({
  children,
}: {
  children: ReactNode;
}) {
  return (
    <ClerkProvider
      publishableKey={process.env.NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY!}
    >
      <ConvexProviderWithClerk client={convex} useAuth={useAuth}>
        {children}
      </ConvexProviderWithClerk>
    </ClerkProvider>
  );
}
```

Check out the [Convex Quickstart](https://docs.convex.dev/quickstart/nextjs) and the [Convex Clerk](https://docs.convex.dev/auth/clerk) auth section for more details.

## Setting up our schema

You can use Convex with or without a schema. In my case, I knew the structure of my data and wanted to define it so i did so below. This also gives you a really nice type-safe API to work with when interacting with your database. We’re defining two tables – a `notes` table to store all voice note information and `actionItems` table for extracted action items. We’ll also define indexes to be able to quickly query the data by `userId` and `noteId`.

```ts
import { defineSchema, defineTable } from 'convex/server';
import { v } from 'convex/values';

export default defineSchema({
  notes: defineTable({
    userId: v.string(),
    audioFileId: v.string(),
    audioFileUrl: v.string(),
    title: v.optional(v.string()),
    transcription: v.optional(v.string()),
    summary: v.optional(v.string()),
    embedding: v.optional(v.array(v.float64())),
    generatingTranscript: v.boolean(),
    generatingTitle: v.boolean(),
    generatingActionItems: v.boolean(),
  })
    .index('by_userId', ['userId'])
    .vectorIndex('by_embedding', {
      vectorField: 'embedding',
      dimensions: 768,
      filterFields: ['userId'],
    }),
  actionItems: defineTable({
    noteId: v.id('notes'),
    userId: v.string(),
    task: v.string(),
  })
    .index('by_noteId', ['noteId'])
    .index('by_userId', ['userId']),
});
```

## Dashboard

Now that we have our backend and authentication setup along with our schema, we can take a look at fetching data. After signing into the app, users can view their dashboard which lists all of the voice notes they’ve recorded.

![NotesGPT Dashboard](https://cdn.sanity.io/images/ts10onj4/production/4bd97587e0925dc222e9248d94f6f4189911fe35-2000x1205.png?w=1000)

To do this, we first define a query in the convex folder that uses auth to take in a `userId`, verify it’s valid, and returns all the notes that match a user’s `userId`.

```ts
export const getNotes = queryWithUser({
  args: {},
  handler: async (ctx, args) => {
    const userId = ctx.userId;
    if (userId === undefined) {
      return null;
    }
    const notes = await ctx.db
      .query('notes')
      .withIndex('by_userId', (q) => q.eq('userId', userId))
      .collect();

    const results = Promise.all(
      notes.map(async (note) => {
        const count = (
          await ctx.db
            .query('actionItems')
            .withIndex('by_noteId', (q) => q.eq('noteId', note._id))
            .collect()
        ).length;
        return {
          count,
          ...note,
        };
      }),
    );

    return results;
  },
});
```

After this, we can call this `getNotes` query with a user’s authentication token via a function that convex provides to display all the user’s notes in the dashboard. We’re using server side rendering to fetch this data on the server then passing it into the `<DashboardHomePage />` client component. This also ensures that the data stays up to date on the client as well.

```tsx
import { api } from '@/convex/_generated/api';
import { preloadQuery } from 'convex/nextjs';
import DashboardHomePage from './dashboard';
import { getAuthToken } from '../auth';

const ServerDashboardHomePage = async () => {
  const token = await getAuthToken();
  const preloadedNotes = await preloadQuery(api.notes.getNotes, {}, { token });

  return <DashboardHomePage preloadedNotes={preloadedNotes} />;
};

export default ServerDashboardHomePage;
```

## Recording a voice note

Initially, users won’t have any voice notes on their dashboard so they can click the “record a new voice note” button to record one. They’ll see the following screen that will allow them to record.
![UI to record a voice note on NotesGPT](https://cdn.sanity.io/images/ts10onj4/production/4ca7220e1d224ad27c21a400602a68f0aa3291b2-2000x1232.png?w=1000)
This will record a voice note using native browser APIs, save the file in Convex file storage, then send it to Whisper through Replicate to be transcribed. The first thing we do is define a `createNote` mutation in our convex folder that will take in this recording, save some information in the Convex database, then call the whisper action.

```tsx
export const createNote = mutationWithUser({
  args: {
    storageId: v.id('_storage'),
  },
  handler: async (ctx, { storageId }) => {
    const userId = ctx.userId;
    let fileUrl = (await ctx.storage.getUrl(storageId)) as string;

    const noteId = await ctx.db.insert('notes', {
      userId,
      audioFileId: storageId,
      audioFileUrl: fileUrl,
      generatingTranscript: true,
      generatingTitle: true,
      generatingActionItems: true,
    });

    await ctx.scheduler.runAfter(0, internal.whisper.chat, {
      fileUrl,
      id: noteId,
    });

    return noteId;
  },
});
```

The whisper action is seen below. It uses Replicate as the hosting provider for Whisper.

```tsx
export const chat = internalAction({
  args: {
    fileUrl: v.string(),
    id: v.id('notes'),
  },
  handler: async (ctx, args) => {
    const replicateOutput = (await replicate.run(
      'openai/whisper:4d50797290df275329f202e48c76360b3f22b08d28c196cbc54600319435f8d2',
      {
        input: {
          audio: args.fileUrl,
          model: 'large-v3',
          translate: false,
          temperature: 0,
          transcription: 'plain text',
          suppress_tokens: '-1',
          logprob_threshold: -1,
          no_speech_threshold: 0.6,
          condition_on_previous_text: true,
          compression_ratio_threshold: 2.4,
          temperature_increment_on_fallback: 0.2,
        },
      },
    )) as whisperOutput;

    const transcript = replicateOutput.transcription || 'error';

    await ctx.runMutation(internal.whisper.saveTranscript, {
      id: args.id,
      transcript,
    });
  },
});
```

Also, all these files can be seen in the Convex dashboard under “Files”.
![Convex Dashboard UI to see uploaded files](https://cdn.sanity.io/images/ts10onj4/production/af403ac53fbd7d1ed9c766b27a503bc6019fd2b4-2000x1813.png?w=1000)

## Generating action items

After the user finishes recording their voice note and it gets transcribed via whisper, the output is then passed into Together AI. We show this loading screen in the meantime.

![A page showing loading indicators for the title, action items, transcription, and summary](https://cdn.sanity.io/images/ts10onj4/production/406ce202f9caa1643f7bdc76bc4658be0ca9efc7-2000x1332.png?w=1000)

We first define a schema that we want our output to be in. We then pass this schema into our Mixtral model hosted on Together.ai with a prompt to identify a summary of the voice note, a transcript, and generate action items based on the transcript. We then save all this information to the Convex database. To do this, we create a Convex action in the convex folder.

```ts
// convex/together.ts

const NoteSchema = z.object({
  title: z
    .string()
    .describe('Short descriptive title of what the voice message is about'),
  summary: z
    .string()
    .describe(
      'A short summary in the first person point of view of the person recording the voice message',
    )
    .max(500),
  actionItems: z
    .array(z.string())
    .describe(
      'A list of action items from the voice note, short and to the point. Make sure all action item lists are fully resolved if they are nested',
    ),
});

export const chat = internalAction({
  args: {
    id: v.id('notes'),
    transcript: v.string(),
  },
  handler: async (ctx, args) => {
    const { transcript } = args;
	  const extract = await client.chat.completions.create({
	    messages: [
	      {
	        role: 'system',
	        content:
	          'The following is a transcript of a voice message. Extract a title, summary, and action items from it and answer in JSON in this format: {title: string, summary: string, actionItems: [string, string, ...]}',
	      },
	      { role: 'user', content: transcript },
	    ],
	    model: 'mistralai/Mixtral-8x7B-Instruct-v0.1',
	    response_model: { schema: NoteSchema, name: 'SummarizeNotes' },
	    max_tokens: 1000,
	    temperature: 0.6,
	    max_retries: 3,
	  });
	  const { title, summary, actionItems } = extract;
	
	  await ctx.runMutation(internal.together.saveSummary, {
	    id: args.id,
	    summary,
	    actionItems,
	    title,
	  });
});
```

When Together.ai responds, we get this final screen which lets users toggle between their transcript and a summary on the left, and see and check off the action items on the right.

![Example note](https://cdn.sanity.io/images/ts10onj4/production/0b589546cc40260f90195d1b39c46b0d79da436a-2000x1325.png?w=1000)

## Vector Search

The final piece of the app is vector search. We’re using Together.ai embeddings to embed the transcripts and make it possible for folks to search in the dashboard based on the semantic meaning of the transcripts.

We do this by creating a `similarNotes` action in the convex folder that takes in a user’s search query, generates an embedding for it, and finds the most similar notes to display on the page.

```tsx
export const similarNotes = actionWithUser({
  args: {
    searchQuery: v.string(),
  },
  handler: async (ctx, args): Promise<SearchResult[]> => {
    // 1. Create the embedding
    const getEmbedding = await togetherai.embeddings.create({
      input: [args.searchQuery.replace('/n', ' ')],
      model: 'togethercomputer/m2-bert-80M-32k-retrieval',
    });
    const embedding = getEmbedding.data[0].embedding;

    // 2. Then search for similar notes
    const results = await ctx.vectorSearch('notes', 'by_embedding', {
      vector: embedding,
      limit: 16,
      filter: (q) => q.eq('userId', ctx.userId), // Only search my notes.
    });

    return results.map((r) => ({
      id: r._id,
      score: r._score,
    }));
  },
});
```

## Conclusion

Just like that, we’ve built a production-ready full-stack AI app ready with authentication, a database, storage, and APIs. Feel free to check out [notesGPT](https://usenotesgpt.com/) to generate action items from your notes or the [GitHub repo](https://github.com/nutlope/notesGPT) for reference. And if you had any questions, [shoot me a DM](https://twitter.com/nutlope) and I’d be more than happy to answer it!

I recently built a full-stack app called notesGPT. It allows you to record a voice note, transcribes it, and extract action items and display them as action items. It’s fully open source and comes equipped with authentication, storage, vector search, action items, and is fully responsive on mobile for ease of use.



Hassan El Mghari

All posts by Hassan