Since February of last year, every project that's been created with MPM create convex at latest has automatically included a set of guidelines for AI models to use. Now, I talked about this at the time, but basically the idea behind the rules is to plug in the gaps in the models convex knowledge. I'm not sure if you noticed, but a lot has happened in Model Land in the past year. Models have gotten a whole lot smarter, and so has the tooling around them. So, the question I had is, do we still need to include that 6,000 tokens or so of convex guidelines with every single prompt, or is there now enough innate training material on the web that the models just get convex without us needing to stuff in that extra knowledge? Well, the answer to that question is both very interesting, but also kind of predictable. It depends. But before we get there, make sure you stay nicely hydrated with a lovely cup of tea. Drop me a like and sub and let's get into it. All right, so if you're new to Convex, and yes, there are a lot of new folks uh to Convex lately, then you may not have heard about our evals project. It's a set of rigorous tests that we've created that's designed to test the model model's knowledge and ability to write convex code. The EVAR is split into seven categories. Fundamentals, data modeling, queries, mutations, actions, idioms, and client. Within each category, for example, this queries one, we have a number of evals that test some specific part of convex knowledge. For example, if we open up this one, the 009 text search, we can see that the task we give the model is uh saved here in the task.txt. So, inside the task.txt, tst we um tell the first thing we do is tell it the model to use an explicit schema. Now other evalued uh for the model in this particular eval we just give them the schema. Then we say write a query named search articles in convex/pub.ts TS that takes arguments, search term, author, searches for all published articles that match the search term, returns the top 10 matching articles with title, author, preview, and we just want the first first 100 characters of content with no extra characters and tags. And just note that we are being very explicit with the names and types here. Um, this is important because I'll show you later that when we come to grade the model, we'll be looking for those exact names and expecting those exact types. This is a change I had to make recently because we had a number of evals previously that were being a little bit loosey goosey with their tasks, which meant we were failing the model's output unfairly when it was actually the task that wasn't specified clearly enough. Anyway, now that the model has the task, this is what we expect the answer to look like. It's a query that takes a search term and an author. It then uses the full text search index to look up the search term and then returns 10 results. Then finally mapping over those results and ensuring that the preview is not longer than 100 characters. So this is what we expect the model to generate. Then what we do is we give it the task and we ask it to one shot generate the answer and we then take that answer and we manually write all the files out to disk and then we run TypeScript type checker and a llinter on it before running it through the unit tests. Uh and we do that for each eval. So for this eval that we're looking at um this is what the tests look like. So the first one is we're just checking that the schema that we asked for does indeed match the one that's in the answer. And then we do our usual unit test things, making sure it does what it's supposed to do. And this is why the exact names of types and things are important because if the model gets even a little bit creative with its output, then it will cause this to fail as the names and types will be incorrect. Now it's important to note that when we give the model the task, we also give them this set of guidelines that we have handcrafted over a series of iterations to plug in the knowledge gaps and get the model to pass the eval. And it's this set of guidelines that is actually what forms eventually the output of the system, the dot rules files that we ship into every npm create convex template and list on the convex docs website. So that was just one evval. We have 66 of them in total and we run them daily against um a number of different models and then we collect all the results and display them on this leaderboard. This leaderboard is great but I have long suspected that there was a bit of runto-run variance between the model runs. So I decided that instead of simply replacing the model's previous result each day, I would then I would store the run and then use that to work out what the stand standard deviation is between the different runs. So that's what these little plus negative numbers are here. It shows what the standard deviation is between the runs on average and one for each of the eval categories. And I guess in conclusion, it seems like on average the run-to- variance isn't too bad. Um some models do have a really high variance for certain categories, but in general the average run variance is not as bad as I thought. [music] Now, while I was doing this, I spent way too much time digging through the results and attempting to refine our set of tasks and guidelines. To help me with doing that, I built this results visualizer that lets you drill down into each run and each category and each val and see exactly what failed in each step. And obviously this is all convex powered and uses convex files storage to upload the model generated output uh which is zipped up and then downloaded again on the client which allows you to inspect the logs and see what went wrong versus what you expected from the task. I think this would be a perfect place to add an agent here. So I could ask it to analyze one eval here or even better look at a given model over time and work out if there's any pattern to eval failing. I think this would be super powerful, but I've spent way more time than I intended on this. So, I'm just going to leave this one here for now. If anybody wants to pick this this task up of adding an agent to the visualizer, uh then I've left a link to the ConX evals repo down below. I would be very much open to a PR. But anyways, back to the leaderboard. Now, I noticed that newer models would do better at Convex, and presumably this is because of increased model intelligence. But also I think that because convex is growing in popularity, there is just more convex code and more convex discussions floating around on the internet for training. So models are getting better at convex over time. Um I mean like a bunch of the top models are pushing over 90% on the evals regularly, but we are still stuffing 6K worth of tokens into every prompt to get to that point. So this naturally had me questioning, do we still need to use all those rules and guidelines? How would a model do if we didn't include them? So I [clears throat] cursor I mean did a bunch of work and now in addition to running models with guidelines, we could also now run models without guidelines. So without further ado and without further teasing, here's the result of that. ah kind of underwhelming. Opus is still the top and then Gemini. Uh but then it gets a little bit messier below that. But in general, it appears that if models are good with guidelines, they're also going to be good without guidelines. That's not to say they're exactly on par. I mean, that's what this column here shows. It shows the change in the average versus the version with guidelines and without. So you can see that Opus without guidelines is about 6% worse whereas Gemini 2 uh 5 Pro for example is only like 2.7% worse. Now I am still concerned about runto- run variance here. So I have also added the error bars here uh and made it so you can click a row and see the results graphed over time and you can also change different categories as well. I should just note here that some of these models have only got a limited amount of data. So for example, Kimmy K2.5, which has only just come out, uh only has a couple of data points. So it's a good idea not to make any definitive conclusions until we this has been run a few times and there's been sufficient amount of data built up. Having said that though, it does seem like in general models do really really well even without the guidelines. So I guess now the question is should we stop automatically including the guidelines with every project created with mpm create convex or not and to be honest I don't know 6% difference isn't that large but when it does make a mistake it can be really annoying. Also the difference is not consistent across the board. It appears that most of these models at the top have quite a small delta, whereas some of the older models or the smaller models lower down here do have a larger delta, which means that they will benefit more from having guidelines. So, as for should we stop including the guidelines, I haven't come to a definitive conclusion yet. I think it depends upon your model use that you're using. So, therefore, I think I'm going to lean towards keeping them in for now. But let me know down in the comments below. I'm genuinely interested to hear what your thoughts are on this. By the way, I also noticed that most models seem to do quite poorly on actions both with and without the guidelines. I did spend a bit of time investigating this because I suspect that when you see across the board poor performance like this, then it's something to do with what we're doing. Um, maybe perhaps the tasks are poorly defined or the tests are being too strict or something. I spent some time looking into this, but unfortunately I didn't really come to any clear conclusions on this one. It might just be morals are just poor at doing convex actions. My next thought was perhaps we could instead of just completely removing the guidelines, we could try and reduce the amount of guidelines we provide. So rather than 6,000 tokens every prompt, maybe we we we reduce that a bit. So the question is what is the minimal number of tokens that we can get away with and still have the model perform at least as good or maybe even better than the full guideline version. My idea to try and solve this is perhaps we could design an agent swarm type of system where we start off with no guidelines whatsoever and then we have an agent run the eval benchmark, [music] analyze the results, work out what failed and then come up with a hypothesis for guidelines that could be added to improve the next run and then it would run it again and see whether the hypothesis was correct. if it was they would be committed into the set of uh guidelines that improve the model and I spent probably again too long investigating this a day or two or three um and I made a fair bit of progress but I decided to pause that work as I wasn't quite sure if it was the right direction I should spend more time on you see my thinking for that is right now what we do is we ask the model to oneshot the answer we basically just give it a bunch of input so which is the task and the guideline and ask it to generate everything at once and then we just grade it on that output. And this was fine back in late 2024, early 2025. Um, but since then things have gotten much more agentic. Now it's much more common for agents to run multiple steps, make multiple tool calls, search the web, use MCP servers, reason skills, etc. So, is it still valid to evaluate the models like we're currently doing, or should we allow models to call the web, look up the convex documentation, etc. It seems to me like a much more realistic scenario these days is instead of testing the [music] model's raw convex knowledge, we should also test the agentic capability of a model by putting it through a more realistic scenario, giving it realistic tools, and then see how it fares. I mean, do we need to explicitly tell a model what to do in the guidelines themselves? Or could we just offer links to specific topics in the documentation? For example, we could say [music] just go to this page at this URL if you've been asked to build a full text search functionality. Then the model could just make subsequent tool calls to get that information. Now this kind of question is something that the eval team over at Visel have also asked recently. In this blog post they made some really interesting observations around how successful a model is on their set of evals when given um skills files versus the information just being dumped directly into the context. In particular this section under what this means for framework authors. It says compress aggressively. You don't need full docs in context. an index pointing to retrievable files works just as well. So maybe that's what we need to do. Instead of having 6,000 tokens or guidelines included in every prompt, we instead have an index of the convex docs which the model is highly encouraged to use via this magical prompt. Important prefer retrievalled reasoning over pre-training reasoning. [music] So yeah, um I think this might be a more promising direction to take my to spend my time looking into rather than the fairly messy and timeconsuming agent swarm [music] um self-improving based guidelines approach. Though having just said that, it does sound pretty cool still. It's [laughter] just not enough hours in the day. So anyways, in conclusion, I [music] think now I have a good foundation to which to run other eval based experiments on [music] and the first no guidelines based experiment has yielded some very interesting results. [music] But I think for now I'm just going to leave us there. I hope you enjoy this video and if you did, you might want to check out that video I did a year ago where I was talking about emails for the first time and what I what I perceived as a crisis in software development. Oh, look at that guy. He looks so young and naive. Anyways, that's it for me for today. Until next time, thanks for watching. Cheerio.
Since February of last year, every project created with npm create convex@latest has automatically shipped a set of guidelines for AI models to use. The idea was to plug the gaps in a model's Convex knowledge. Convex is newer and smaller in the training data than something like Postgres or Express, so those gaps are real. I talked about this at the time.
A lot has happened in Model Land since then. Models have gotten a lot smarter, and so has the tooling around them. So I had a question. Do we still need to ship that 6,000-ish tokens of Convex guidelines with every prompt, or is there now enough Convex code on the public web that models just get it on their own?
The answer's interesting but predictable. It depends.
The evals project
If you're new to Convex, and there are a lot of new folks lately, you may not have heard about our evals project. It's a set of rigorous tests that measure how well models understand and write Convex code. The evals split into seven categories: fundamentals, data modeling, queries, mutations, actions, idioms, and client. Each one holds a handful of evals that test a specific slice of Convex knowledge.
Take the queries category. One eval, 009 text_search, gives the model an explicit schema and this task: write a query named searchArticles in convex/pub.ts that takes arguments searchTerm and author, searches for all published articles that match the search term, and returns the top 10 matching articles with title, author, preview, and the first 100 characters of content with no extra characters or tags.
Notice how explicit that task is about names and types. That's deliberate: when we grade the model's output, we check for those exact names and types. This is a recent change. A bunch of our evals used to be a little loosey-goosey with their tasks, so we'd fail models unfairly when the task itself wasn't spelled out clearly enough.
The expected answer for that eval is a query that takes searchTerm and author, uses a full text search index to look up the term, returns 10 results, and truncates the preview to 100 characters. We give the model the task and ask it to one-shot generate an answer, write the files to disk, run the TypeScript type checker and a linter, and then run unit tests. The tests first check that the schema matches what we asked for. Then functional tests confirm the code does what it's supposed to. Because we match on exact names and types, a model that gets even a little creative will fail here, even if its logic is sound.
Alongside the task, we hand the model the guidelines we've handcrafted over many iterations to plug its knowledge gaps. Those are what eventually become the .rules files shipped into every npm create convex template and listed in the Convex docs. And that's just one eval out of 66. We run all of them daily against a range of models and collect the results onto a leaderboard.
Run-to-run variance and a results visualizer
I'd long suspected there was real run-to-run variance on the same eval. So instead of overwriting each day's result, I started storing every run and computing the standard deviation across them. On average, the run-to-run variance isn't too bad, though some models are noticeably noisier on certain categories than others.
While digging through the results, I built a results visualizer that lets you drill into each run, category, and eval to see exactly what failed and why. It's Convex-powered. The model-generated output gets zipped, uploaded to Convex file storage, and downloaded again on the client, so you can inspect the logs against what was expected. This would be a natural place to add an agent that analyzes a given eval or a given model's history over time and looks for patterns in what's failing. I spent more time than intended getting this far, so I'm leaving that idea for now. The convex-evals repo is open if anyone wants to pick it up; I'd take the PR.
Do models still need the guidelines?
Back to the leaderboard. Newer models do better at Convex. Part of that is raw intelligence, and part is that there's simply more Convex code and discussion online now to train on. Many top models are pushing over 90% on the evals regularly, and we're still stuffing 6K tokens of guidelines into every prompt to get there. So I ran the obvious experiment: models with guidelines, and models without.
The results were underwhelming, in the sense that they didn't resolve the question cleanly. Opus is still on top, then Gemini, and it gets messier below that. In general, if a model is good with guidelines, it's also good without them, though not exactly on par. Opus without guidelines scores about 6% worse. Gemini 2.5 Pro is only about 2.7% worse without them.
Because I was still worried about run-to-run noise, I added error bars and made it possible to click a row and see results graphed over time, filtered by category. Some models only have a couple of data points so far (Kimi K2.5, for instance). I wouldn't draw firm conclusions on those until there's more data behind them. With that caveat, models generally do quite well even without the guidelines.
So should we stop automatically including guidelines with every project created via npm create convex? I haven't reached a definitive conclusion. A 6% difference isn't huge, but when a model does make a mistake, it's a genuinely annoying one to debug. The size of that gap isn't consistent, either. Most of the top models show a small delta, while older or smaller models further down the leaderboard show a bigger one and would benefit more from keeping the guidelines around. For now I'm leaning toward keeping them in. Let me know what you think in the comments.
Actions are still a weak spot
One thing stood out regardless of guidelines. Most models do noticeably worse on actions, with or without them. When performance is bad across the board like that, it's worth checking whether the tasks themselves are poorly defined or the tests too strict. I looked into this and didn't land on a clear conclusion. It might just be that models are genuinely worse at writing Convex actions than at the rest of the API surface.
Shrinking the guidelines instead of removing them
Rather than removing the guidelines outright, another option is to shrink them. Instead of 6,000 tokens on every prompt, what's the minimal number of tokens that still gets a model performing at least as well as the full guideline version?
My idea was an agent-swarm system. Start with no guidelines, have an agent run the eval benchmark, analyze what failed, and propose a guideline addition that would fix it. If that hypothesis held up on the next run, it gets committed to the guideline set. I spent a few days on this and made fair progress, then paused. I wasn't confident it was the right place to keep sinking time.
One-shot evals versus agentic reality
Right now, our evals ask the model to one-shot the answer: give it the task and the guidelines, ask it to generate everything at once, then grade the output. That was a reasonable way to test things in late 2024 and early 2025. Since then, things have gotten a lot more agentic. Agents run multiple steps, make multiple tool calls, search the web, hit HTTP servers, reason, and use skills.
That raises a real question about how we should test models on Convex at all. A more realistic test would measure agentic capability given real tools, not just the model's raw Convex knowledge. We could let models call the web, look up the Convex documentation, and work the way an agent actually would in practice. Do we need to spell everything out in the guidelines, or could we point to specific documentation topics and let the model make tool calls to grab what it needs? For full text search, for instance, we could point at a URL for the feature instead of pre-loading the guidance into every prompt.
Vercel's eval team has been asking a similar question. In their post on AGENTS.md outperforming skills in their own agent evals, they found that a compressed, always-present index of documentation beat retrieval-triggered skill files. The skill approach depends on the model deciding to invoke it, and that decision point is itself a source of failure. Their advice to framework authors is to compress aggressively. You don't need the full docs in context; an index pointing to retrievable files works just as well.
That might be the shape of what Convex needs too. Instead of 6,000 tokens baked into every prompt, you'd give the model an index into the Convex docs and a system prompt that favors retrieval over leaning purely on pre-training. That direction feels more promising to me right now than the agent-swarm, self-improving guidelines idea, even though that one sounds cooler. There aren't enough hours in the day for both.
Where this leaves things
I now have a solid foundation for running further eval-based experiments, and the first no-guidelines experiment has already produced results worth sitting with. I don't have a final answer on whether to drop the guidelines, shrink them, or swap them for a retrieval-based index. But the evals repo is public if you want to dig into the data yourself or contribute an eval.
If you want the longer history here, I did a video a year ago about evals and what I saw at the time as a bit of a crisis in AI-assisted software development. I sound a little more certain in that one than I feel today.
All gas, no breakages
Convex is the reactive backend platform that keeps up with you and your agents. Database, functions, workflow, sync, search, file storage, and more. All TypeScript, zero glue.
