Mike Cann

a month ago

We made AI on Convex way simpler

Hey guys, good news today as something landed in Convex that I've been asking for for a while now. You can now make LLM calls directly from Convex actions with no manual setup for OpenAI, Anthropic or OpenRouter. Here's a little demo that my agent built, where I ask a model a question and get an answer back, and there isn't a single provider API key anywhere in it:

It's a new feature called AI Gateway and it basically lets you make OpenRouter calls but without an actual OpenRouter account. Convex just takes care of all that account management nastiness for us, and we have this single unified billing interface.

So make sure you grab yourself a lovely steaming hot cup of tea, and let's take a closer look.

Let's look at the code

AI Gateway is basically just a super convenient way to make AI calls on Convex, because you don't need to set up or subscribe to another third party service. You can just do it all within a Convex action.

Actually you know what, let's take a look inside the code itself. Oh man, it's been some time since I've been inside an IDE. But reading code is like riding a bike, surely I will not forget how to do this 😅

First you install the AI SDK and the Convex provider for it:

1npm install ai @convex-dev/ai-sdk-provider
2

Then here's roughly what that demo is doing. We have a Convex action that takes a prompt and a choice of the different models we want to use (Luna, Haiku and Flash), and then we're just using the normal generateText from the AI SDK and passing in convexGateway:

1// convex/text.ts
2import { action } from "./_generated/server";
3import { v } from "convex/values";
4import { generateText } from "ai";
5import { convexGateway } from "@convex-dev/ai-sdk-provider";
6
7export const generate = action({
8  args: {
9    text: v.string(),
10    model: v.union(
11      v.literal("openai/gpt-5.6-luna"),
12      v.literal("anthropic/claude-haiku-4.5"),
13      v.literal("google/gemini-3.7-flash"),
14    ),
15  },
16  handler: async (ctx, { text, model }) => {
17    const result = await generateText({
18      model: convexGateway(model),
19      prompt: text,
20    });
21    return result.text;
22  },
23});
24

And that's it! There's no key in there and nothing to add to your environment variables, you just change the model name to change the model or the provider. The models page has the full list. You will need convex 1.45 or later for this tho.

The nice thing about this as well is that it works with anything that supports the OpenAI-compatible spec, so you can point the OpenAI SDK at it too. You can also use it with the Convex Agent component if you're writing an agentic application on top of Convex. Once you have the component installed that's just:

1import { Agent } from "@convex-dev/agent";
2import { convexGateway } from "@convex-dev/ai-sdk-provider";
3import { components } from "./_generated/api";
4
5const agent = new Agent(components.agent, {
6  name: "Support agent",
7  languageModel: convexGateway("openai/gpt-4o-mini"),
8  instructions: "You answer questions about our product.",
9});
10

Super cool!

What else can it do?

So basically under the hood Convex is effectively just a proxy to OpenRouter. That means we can do the OpenRouter things you'd expect, such as generating images and embeddings, and we can even do the brand new decisions model kind of thing in the form of Jev. Let's have a look at some of that in the demo.

That clip from the start was me asking Luna to explain why the sky is blue to a five-year-old in two sentences. Very appropriate, as my son was asking me that exact question the morning I recorded this. "Sunlight is made of many colors". Very good.

Next up we can do embeddings, so that's some text going in and a vector coming out that you can use for things like semantic search:

1import { embed } from "ai";
2
3const { embedding } = await embed({
4  model: convexGateway.embeddingModel("openai/text-embedding-3-small"),
5  value: text,
6});
7

The demo's Embeddings tab: the text "I can't sign in to my account." has become a vector of 1,536 dimensions

Lovely. Then there's Jev, from TypeSafe AI, which you use to classify or score things. Instead of chat text you give it a question and some criteria and it gives you back an answer with probabilities. In the demo it decides what sort of support priority a message should get, as you type. Watch the priority on the right:

So "im feeling a little tired" is somewhere between low and normal, and "oh no my house is on fire" is, hopefully, a more urgent request. Here's the shape of that call, with the instructions and criteria trimmed down a bit:

1import { experimental_evaluate as evaluate } from "ai";
2
3const decision = await evaluate({
4  model: convexGateway.evaluationModel("typesafe/jev-1.13"),
5  state: { message: text },
6  questions: {
7    priority: {
8      type: "choice",
9      instructions: "Classify the support request based on its actual impact",
10      criteria: {
11        low: "A cosmetic preference, feature suggestion or general question",
12        normal: "A problem affecting one person without a critical business impact",
13        urgent: "An outage, data loss or security incident",
14      },
15    },
16  },
17});
18

And finally image generation:

1import { generateImage } from "ai";
2
3const { images } = await generateImage({
4  model: convexGateway.imageModel("openai/gpt-image-1"),
5  prompt,
6});
7

Here's the little robot painting a sunset that I got back:

The demo's Image generation tab showing a clay animation style robot painting a sunset on an easel

So again, this is all built directly on Convex. I didn't need to sign up for OpenRouter, I didn't need to sign up for OpenAI or any of that. I just get all these things straight away, and it all just works.

By the way, Jev decisions and image generation are in alpha, so their request and response formats may change. The same goes for video and voice, which you can also do through the gateway now.

The Heroku model

I think that's the beauty of Convex to me, or at least that's what people were telling me when I was in San Francisco for our Convex Abstract event. They were saying that the power of Convex in the age of AI is not the lovely syntax, it's the holistic thing. You can just point your agent at it and step back and be confident that it will be fine, because it has everything it needs to do a good job.

So I think this is why AI Gateway fits so naturally into Convex, it's just another thing that your agent can leverage right out of the box.

And this is kind of like the Heroku model. You remember Heroku, right? It was this pretty magical service at the time where you could use it for most of your backend web hosting and database needs, and then you could kind of just plug third party services into it. The best part about that was the unified billing. You only had to set up the billing once for Heroku, and then Heroku would work out how much to pay each of the third party services you used and take care of all that for you.

Anyway, this is what I have been asking for in Convex since I joined about two years ago now, this single unified backend, so I don't ever have to think about external service X, Y or Z. I just kind of get that functionality and it just works. But anyways, enough waffle about that.

Who can use it

Now one thing to know is that it's only on paid plans. Starter counts tho, and that's $0 a month and pay as you go.

It works on your cloud deployments, and on local deployments as long as they're linked to a project on a paid team. It doesn't work on anonymous local deployments. When I had my agent run that action on one, this is what came back:

1Uncaught Error: `getServiceToken("ai-gateway")` requires an authenticated local deployment. Run `npx convex login`, then restart `npx convex dev`.
2

So log in, link it to your project and you're away.

Keeping an eye on the cost

What else can I say about AI Gateway? Well there's a really nice set of reporting features built into the dashboard. If you go up into your team settings, the usage page has an AI Gateway section where you can see your spend by project or by model:

The AI Gateway section of the team usage page: one day of spend at about 6 cents, broken down by model, with openai/gpt-image-1 at $0.0551 and everything else under a cent each

That's my demo project on the day I recorded this, and image generation was by far the most expensive part. You can also see which functions spent the money, in the AI tab of the breakdown by function further down that page.

The pricing is the same as OpenRouter with no gateway markup on top, so check out OpenRouter's models page for what each one costs. It then shows up as its own line on your Convex invoice.

Then you can go into each individual deployment and set usage limits, monthly or daily, under Settings and then Usage Limits. AI Gateway is the row at the bottom:

The Usage Limits page in deployment settings, with Monthly and Daily tabs and a warning and disable threshold column for each metric. The AI Gateway row shows 0.06 USD of current usage

Every deployment can have a disable threshold, and production and preview deployments can have a warning threshold as well.

Now just be careful, if you do set a disable threshold then it's not going to disable just the AI Gateway feature, it's going to disable that entire deployment. So if you hit your AI Gateway spend limit, it's going to shut down your deployment until the window rolls over or you raise the limit. So just be aware of that. It's still a very nice way of limiting how much a given deployment is going to be able to spend tho. The same goes for team spending limits, which AI Gateway spend counts towards, except a disable threshold there disables all the projects in your team.

Conclusion

So just to recap: you make your calls from a Convex action using convexGateway, with no provider accounts or keys to deal with, and it all ends up on your Convex bill at OpenRouter's rates. You can see what you've spent in the dashboard and set limits, just remember that a disable threshold switches off the whole deployment.

I think it's a really nice addition to the Convex feature set, and just another one of those annoying paper cuts that has now been removed. If you want to get started then just point your agent at the docs and let it rip.

Until next time, cheerio! 👋