Mike Cann

5 months ago

Does AI still need Convex guidelines?

Since February of last year, every project created with npm create convex@latest has automatically shipped a set of guidelines for AI models to use. The idea was to plug the gaps in a model's Convex knowledge. Convex is newer and smaller in the training data than something like Postgres or Express, so those gaps are real. I talked about this at the time.

A lot has happened in Model Land since then. Models have gotten a lot smarter, and so has the tooling around them. So I had a question. Do we still need to ship that 6,000-ish tokens of Convex guidelines with every prompt, or is there now enough Convex code on the public web that models just get it on their own?

The answer's interesting but predictable. It depends.

The evals project

If you're new to Convex, and there are a lot of new folks lately, you may not have heard about our evals project. It's a set of rigorous tests that measure how well models understand and write Convex code. The evals split into seven categories: fundamentals, data modeling, queries, mutations, actions, idioms, and client. Each one holds a handful of evals that test a specific slice of Convex knowledge.

Take the queries category. One eval, 009 text_search, gives the model an explicit schema and this task: write a query named searchArticles in convex/pub.ts that takes arguments searchTerm and author, searches for all published articles that match the search term, and returns the top 10 matching articles with title, author, preview, and the first 100 characters of content with no extra characters or tags.

Notice how explicit that task is about names and types. That's deliberate: when we grade the model's output, we check for those exact names and types. This is a recent change. A bunch of our evals used to be a little loosey-goosey with their tasks, so we'd fail models unfairly when the task itself wasn't spelled out clearly enough.

The expected answer for that eval is a query that takes searchTerm and author, uses a full text search index to look up the term, returns 10 results, and truncates the preview to 100 characters. We give the model the task and ask it to one-shot generate an answer, write the files to disk, run the TypeScript type checker and a linter, and then run unit tests. The tests first check that the schema matches what we asked for. Then functional tests confirm the code does what it's supposed to. Because we match on exact names and types, a model that gets even a little creative will fail here, even if its logic is sound.

1import { v } from "convex/values";
2import { query } from "./_generated/server";
3
4export const searchArticles = query({
5  args: {
6    searchTerm: v.string(),
7    author: v.string(),
8  },
9  returns: v.array(
10    v.object({
11      title: v.string(),
12      author: v.string(),
13      preview: v.string(),
14      tags: v.array(v.string()),
15    }),
16  ),
17  handler: async (ctx, args) => {
18    const articles = await ctx.db
19      .query("articles")
20      .withSearchIndex("search_articles", (q) =>
21        q
22          .search("content", args.searchTerm)
23          .eq("author", args.author)
24          .eq("isPublished", true),
25      )
26      .take(10);
27    return articles.map((article) => {
28      let preview = article.content;
29      if (preview.length > 100) {
30        preview = preview.slice(0, 100);
31      }
32      return {
33        title: article.title,
34        author: article.author,
35        preview,
36        tags: article.tags,
37      };
38    });
39  },
40});
41

Alongside the task, we hand the model the guidelines we've handcrafted over many iterations to plug its knowledge gaps. Those are what eventually become the .rules files shipped into every npm create convex template and listed in the Convex docs. And that's just one eval out of 66. We run all of them daily against a range of models and collect the results onto a leaderboard.

Run-to-run variance and a results visualizer

I'd long suspected there was real run-to-run variance on the same eval. So instead of overwriting each day's result, I started storing every run and computing the standard deviation across them. On average, the run-to-run variance isn't too bad, though some models are noticeably noisier on certain categories than others.

While digging through the results, I built a results visualizer that lets you drill into each run, category, and eval to see exactly what failed and why. It's Convex-powered. The model-generated output gets zipped, uploaded to Convex file storage, and downloaded again on the client, so you can inspect the logs against what was expected. This would be a natural place to add an agent that analyzes a given eval or a given model's history over time and looks for patterns in what's failing. I spent more time than intended getting this far, so I'm leaving that idea for now. The convex-evals repo is open if anyone wants to pick it up; I'd take the PR.

Do models still need the guidelines?

Back to the leaderboard. Newer models do better at Convex. Part of that is raw intelligence, and part is that there's simply more Convex code and discussion online now to train on. Many top models are pushing over 90% on the evals regularly, and we're still stuffing 6K tokens of guidelines into every prompt to get there. So I ran the obvious experiment: models with guidelines, and models without.

The results were underwhelming, in the sense that they didn't resolve the question cleanly. Opus is still on top, then Gemini, and it gets messier below that. In general, if a model is good with guidelines, it's also good without them, though not exactly on par. Opus without guidelines scores about 6% worse. Gemini 2.5 Pro is only about 2.7% worse without them.

Because I was still worried about run-to-run noise, I added error bars and made it possible to click a row and see results graphed over time, filtered by category. Some models only have a couple of data points so far (Kimi K2.5, for instance). I wouldn't draw firm conclusions on those until there's more data behind them. With that caveat, models generally do quite well even without the guidelines.

So should we stop automatically including guidelines with every project created via npm create convex? I haven't reached a definitive conclusion. A 6% difference isn't huge, but when a model does make a mistake, it's a genuinely annoying one to debug. The size of that gap isn't consistent, either. Most of the top models show a small delta, while older or smaller models further down the leaderboard show a bigger one and would benefit more from keeping the guidelines around. For now I'm leaning toward keeping them in. Let me know what you think in the comments.

Actions are still a weak spot

One thing stood out regardless of guidelines. Most models do noticeably worse on actions, with or without them. When performance is bad across the board like that, it's worth checking whether the tasks themselves are poorly defined or the tests too strict. I looked into this and didn't land on a clear conclusion. It might just be that models are genuinely worse at writing Convex actions than at the rest of the API surface.

Shrinking the guidelines instead of removing them

Rather than removing the guidelines outright, another option is to shrink them. Instead of 6,000 tokens on every prompt, what's the minimal number of tokens that still gets a model performing at least as well as the full guideline version?

My idea was an agent-swarm system. Start with no guidelines, have an agent run the eval benchmark, analyze what failed, and propose a guideline addition that would fix it. If that hypothesis held up on the next run, it gets committed to the guideline set. I spent a few days on this and made fair progress, then paused. I wasn't confident it was the right place to keep sinking time.

One-shot evals versus agentic reality

Right now, our evals ask the model to one-shot the answer: give it the task and the guidelines, ask it to generate everything at once, then grade the output. That was a reasonable way to test things in late 2024 and early 2025. Since then, things have gotten a lot more agentic. Agents run multiple steps, make multiple tool calls, search the web, hit HTTP servers, reason, and use skills.

That raises a real question about how we should test models on Convex at all. A more realistic test would measure agentic capability given real tools, not just the model's raw Convex knowledge. We could let models call the web, look up the Convex documentation, and work the way an agent actually would in practice. Do we need to spell everything out in the guidelines, or could we point to specific documentation topics and let the model make tool calls to grab what it needs? For full text search, for instance, we could point at a URL for the feature instead of pre-loading the guidance into every prompt.

Vercel's eval team has been asking a similar question. In their post on AGENTS.md outperforming skills in their own agent evals, they found that a compressed, always-present index of documentation beat retrieval-triggered skill files. The skill approach depends on the model deciding to invoke it, and that decision point is itself a source of failure. Their advice to framework authors is to compress aggressively. You don't need the full docs in context; an index pointing to retrievable files works just as well.

That might be the shape of what Convex needs too. Instead of 6,000 tokens baked into every prompt, you'd give the model an index into the Convex docs and a system prompt that favors retrieval over leaning purely on pre-training. That direction feels more promising to me right now than the agent-swarm, self-improving guidelines idea, even though that one sounds cooler. There aren't enough hours in the day for both.

Where this leaves things

I now have a solid foundation for running further eval-based experiments, and the first no-guidelines experiment has already produced results worth sitting with. I don't have a final answer on whether to drop the guidelines, shrink them, or swap them for a retrieval-based index. But the evals repo is public if you want to dig into the data yourself or contribute an eval.

If you want the longer history here, I did a video a year ago about evals and what I saw at the time as a bit of a crisis in AI-assisted software development. I sound a little more certain in that one than I feel today.