Mike Cann's avatar
Mike Cann
5 months ago

Comparing 9 Code Review Tools - Which is Best?

Let's play a game. Here's a Convex query. See if you can spot the bug.

1export const getShared = query({
2  args: { taskId: v.id("tasks") },
3  handler: async (ctx, args) => {
4    const userId = await getAuthUserId(ctx);
5    if (!userId) return null;
6
7    const task = await ctx.db.get("tasks", args.taskId);
8    if (!task) return null;
9
10    const project = await ctx.db.get("projects", task.projectId);
11
12    let assigneeName: string | undefined;
13    if (task.assigneeId) {
14      const assignee = await ctx.db.get("users", task.assigneeId);
15      assigneeName = assignee?.name ?? "Unknown";
16    }
17
18    return {
19      ...task,
20      projectName: project?.name ?? "Unknown",
21      assigneeName,
22    };
23  },
24});
25

It takes a task ID, grabs the signed-in user, then loads the task, its project, and the assignee's name. Spotted it? No? Neither did I.

The query confirms that somebody is logged in. It never confirms that the task belongs to a project that person is a member of. So any authenticated user can read anyone else's task by guessing an ID. That's a textbook IDOR, and it's exactly the kind of hole that disappears into the noise when you're deep in the vibe coding zone.

I've been letting agents rip on my Convex projects lately, merging their work without much of a security pass. Things were falling through the cracks. So I took that bug, plus nine more I built to probe Convex-specific blind spots, and ran all ten past nine of the best AI code review tools on the market.

The results surprised me. One tool got nearly everything right. One very famous tool face-planted.

Jump to:

The results at a glance

Ten tests, four points available on each, so 40 is a perfect run. Every per-test score is published in the evaluation playground repo if you want to check my working.

#ToolTierScoreStrongest showingWeakest showing
1QodoGold32/40Caught nine of ten primary issuesFell for the fake N+1
2GitHub CopilotGold31/40Most bonus points in the fieldFell for the fake N+1
3CubicSilver26/40Only tool to knowingly pass the N+1 trapMissed the unbounded .collect()
4CodeRabbitSilver24/40Nailed the document size limit testReview engine failed on one PR
5GreptileSilver23/40Full marks on the internal-function and authorization testsCalled valid array code a "critical bug"
6CodeAnt AIBronze13/40Strong on the two auth testsFalse positives on two internal queries
7SourcerySkip9/40Caught the unbounded .collect()Sequence diagrams on every single PR
8MacroscopeSkip9/40Flagged the intro bug as CriticalSilent on six of the ten PRs
9Graphite AISkip3/40NothingZero comments on nine of ten PRs

How I tested them

I built a Trello clone on React, Vite, and Convex Auth as the baseline. You log in, create a project, assign members, and they create tasks on a kanban board. Tasks carry comments, and an activity log keeps an audit trail behind the scenes.

It's a basic app, but it has the shape of a real SaaS product: nested related data, plus genuine authentication and authorization boundaries. That's enough surface area to hide real bugs in.

From there I designed ten PRs, which are really evals for AI code review tools. Each one started as a markdown document spelling out what a correct review should catch, what it should not flag, and why. I wrote those before opening a single PR, so I couldn't grade toward whatever a bot happened to say. They never went onto the branches under test. They're on the main branch now so you can run the whole thing yourself.

Keeping each PR realistic while testing one thing at a time took some care. My main trick was to mark new queries as internal and add a comment saying they'd be run from the Convex dashboard. Convex developers do this all the time, so it let me add code that isn't wired into the UI without a bot crying dead code. I also kept the repo and the PR descriptions clean of anything that would tip a bot off. I don't want a Volkswagen emissions situation on my hands.

Every bot ran on its default configuration. I didn't tune a thing. I'm not an expert in nine separate products, and configuring them all correctly would have been fiddly and error-prone. I also watched for any bot referencing another bot's comments and never saw it happen, probably because they all trigger the moment a PR opens and grab their context at the same time.

Grading nine bots across ten PRs by hand would have taken forever, so I had an agent do the scoring. Claude Opus did the first pass. Then Codex 5.3 ran the whole thing again in a clean context as a cross-check, and it mostly agreed, catching a couple of things Opus missed.

Here's the scoring:

  • 3 points for catching the primary issue
  • 1 point for a mixed or partial result
  • 0 points for missing it
  • +1 bonus for finding something else genuinely useful
  • -1 penalty for false positives

That penalty matters to me. Noisy false alarms are their own kind of cost, and I don't want to spend my week arguing with a bot.

Here's the first test, since it comes up in every entry below. It's an activity feed query that filters in memory instead of using an index:

1// Reporting queries for use via the Convex Dashboard
2
3export const getActivityForUser = internalQuery({
4  args: {
5    projectId: v.id("projects"),
6    userId: v.id("users"),
7  },
8  handler: async (ctx, args) => {
9    const activity = await ctx.db
10      .query("activityLog")
11      .order("desc")
12      .filter((q) =>
13        q.and(
14          q.eq(q.field("projectId"), args.projectId),
15          q.eq(q.field("userId"), args.userId),
16        ),
17      )
18      .take(50);
19
20    return activity;
21  },
22});
23

That .filter() pulls the whole activityLog table through the server and throws most of it away. The right answer is an index. The guidance is plastered all over the Convex docs, and I'd also put it in the project's .cursor/rules directory. Part of the test was seeing whether any bot would go read those rules. Spoiler: none of them did.

1. Qodo

Score: 32/40. Tier: gold. Best for: raw accuracy, if you can stomach the dashboard.

Qodo topped the table. It caught the .filter() versus .withIndex() issue and nine of the ten primary issues overall, which nothing else came close to. It was also one of only two tools that caught the unbounded array test.

It wasn't flawless. It falsely flagged the N+1 pattern that Convex's collocated compute and database make a non-issue, and it hedged an auth concern on an internal query.

What makes the win remarkable is the dashboard, which was the worst of any tool here. Buggy, confusing, and repeatedly unable to show repositories I'd already connected. I got frustrated enough that I deleted my entire account, and it kept submitting PR reviews anyway. Free, accurate, and apparently impossible to get rid of.

2. GitHub Copilot

Score: 31/40. Tier: gold. Best for: most people, because you're probably already paying for it.

GitHub Copilot finished one point behind. It caught the indexing issue and attached a genuinely useful inline suggestion, and it racked up more bonus points than anything else in the lineup. It kept spotting real secondary bugs nobody asked it to look for, including a dueDate truthy check that treats zero as falsy.

It fell for the N+1 non-issue like most of the field, so it isn't immune to Convex's quirks.

What tips it into my personal pick is practical. It's already built into GitHub, so there's no new billing relationship and nothing to install.

3. Cubic

Score: 26/40. Tier: silver. Best for: teams who want the fewest false alarms.

Cubic was the standout on the test I cared about most, which was the fake N+1. Here's the bait:

1export const getProjectOverview = internalQuery({
2  args: { projectId: v.id("projects") },
3  handler: async (ctx, args) => {
4    // Intentional cap: we only show up to 50 tasks
5    const tasks = await ctx.db
6      .query("tasks")
7      .withIndex("by_projectId", (q) =>
8        q.eq("projectId", args.projectId))
9      .take(50);
10
11    const enrichedTasks = [];
12    for (const task of tasks) {
13      const assignee = task.assigneeId
14        ? await ctx.db.get("users", task.assigneeId)
15        : null;
16      // ... labels fetched per task the same way
17    }
18  },
19});
20

Train a model on traditional serverless code and that loop looks like a pile of database round-trips waiting to happen. Most tools flagged it as a performance problem on sight. Cubic saw nothing wrong, because it understood that Convex runs the whole thing as one transaction rather than a chain of network calls.

Two other tools technically passed that test by saying nothing at all. Cubic passed it on purpose, which is a different thing entirely.

It also never took a single false-positive penalty across all ten PRs, and it went and read the Convex docs mid-review to confirm the OCC anti-pattern. Its blind spot was the unbounded .collect() test, which it missed completely. Of the nine, Cubic showed the clearest sign of reasoning about Convex's architecture instead of pattern-matching generic JavaScript.

4. CodeRabbit

Score: 24/40. Tier: silver. Best for: deep dives on data modeling.

CodeRabbit caught the indexing issue without trouble and was the strongest tool on the document size limit test, naming the 1MB ceiling and recommending a separate table. That's exactly the right fix.

It also fell hard for the N+1 false positive and took a scoring penalty for it, one of only two tools to go negative on that test. Its "critical" finding was that the two-argument ctx.db.get form is wrong API usage. It isn't.

Reliability bit it once too. On the internal-function PR, its review engine errored out and posted nothing at all. A broken review still counts as a miss.

5. Greptile

Score: 23/40. Tier: silver. Best for: catching what the diff implies, not just what it says.

Greptile tracked closely with CodeRabbit. It caught the indexing problem, and it was one of the few tools to name the OCC hot path directly, describing the contention precisely on a test that nothing in the field had touched in its subtler form.

Then it false-flagged the N+1 non-issue, and on the array test it described perfectly valid mutable array operations as "critical logic bugs." That one earned a penalty, because a confidently wrong critical is worse than silence.

It's a capable middle-of-the-pack reviewer that knows Convex's basics but not its deeper guarantees.

6. CodeAnt AI

Score: 13/40. Tier: bronze. Best for: nothing I'd recommend it for yet.

CodeAnt AI started badly. It missed the indexing issue, gesturing vaguely at indices without committing, and then flagged auth and data-leak problems on a query I'd explicitly marked internal and documented as dashboard-only. Internal queries don't need auth checks, because the dashboard is what runs them. It did the same thing on the N+1 test, which means it went negative on both of its first two PRs.

It recovered later. It caught the unbounded .collect() that three better-scoring tools missed, and it was solid on both authorization tests, including the intro bug.

That's the shape of it. Real capability, undermined by a weak grasp of what Convex's internal functions are for.

7. Sourcery

Score: 9/40. Tier: skip. Best for: nobody who dislikes sequence diagrams.

Sourcery had the most verbose PR summaries of anything I tried. It bolts a full sequence diagram onto every single PR, which adds noise rather than insight.

On substance it managed half credit on the indexing test, suggesting an index "if not already indexed" without ever checking the schema to see whether one existed. Then it false-flagged the N+1 like most of the field, and it repeated CodeRabbit's mistake about the two-argument ctx.db.get form, which cost it another penalty.

It missed the intro authorization bug entirely. The one issue it found on that PR was a clipboard API guard sitting in the UI code next door.

8. Macroscope

Score: 9/40. Tier: skip. Best for: teams who prefer their reviewers quiet.

Macroscope said nothing at all on six of the ten PRs. It missed the indexing test, offering a generic passing check instead of a review, then "passed" the N+1 test by not flagging anything. I gave it the points, since not flagging a non-issue is technically correct, but I doubt it reflects any understanding of Convex's execution model.

To its credit, it woke up twice late in the suite. It caught the out-of-sync aggregate, and it flagged the intro authorization bug as Critical with a proper evidence trail. Two real catches out of ten isn't enough to build a workflow on.

9. Graphite AI

Score: 3/40. Tier: skip. Best for: nothing, on this evidence.

Graphite AI has real brand recognition, which made this the most surprising result of the whole test. It scored zeros almost straight across the board. It left zero comments on nine of the ten PRs.

Its only points came from silence on the N+1 test, the same technical pass Macroscope got. Everywhere else it produced a completed check run and nothing else.

I logged back into the dashboard afterward and checked, double-checked, and triple-checked that it was properly connected. It was. It just doesn't find anything.

What the other tests showed

The indexing and N+1 tests get the most airtime because they're the cleanest signal, but the other eight are where the Convex-specific gaps show up. Here's the whole suite:

TestWhat it probedCaught it
Nested ctx.db lookupsA fake N+1. Does the bot know Convex runs the query as one transaction?3 of 9 stayed quiet, only Cubic on purpose
.filter() over .withIndex()Basic Convex query hygiene, documented everywhere6 of 9, one of those only half credit
Missing authenticationAn export function doing real work without checking for a signed-in user5 of 9
Public mutation that should be internalA cron-triggered mutation left callable by anyone4 of 9
Unbounded .collect()Reading a table that grows forever4 of 9
.collect() used to countShould be a denormalized counter or the aggregate component7 of 9
Unbounded array on a documentA checklist field heading for the document size limit2 of 9 clean, 3 partial
Component left out of syncA mutation that changes the task count without updating the aggregate7 of 9
OCC hot pathEvery mutation in the app writing one shared counter row5 of 9 clean, 3 partial
Authorization bypassThe intro bug: authentication present, membership check missing7 of 9

A few of those deserve a note.

The authorization bug from the intro was caught by seven of the nine, which was reassuring. Authorization is a well-understood concept in every framework, so the bots have plenty to pattern-match against. Making a cron-triggered mutation internal rather than public is far more Convex-specific, and only four of nine got there.

The performance tests split the field in an odd way. Unbounded .collect() calls on a table that grows without limit tripped up five of the nine, including Cubic and CodeRabbit. But using .collect() just to count documents, where the fix is a denormalized counter or the aggregate component, was caught by seven. The difference is that the counting version was a public query wired into the UI, so the bots could see the useQuery call and reason about reactive re-execution.

The document size test was the hardest of the lot. I added a checklist array to tasks and waited to see who'd notice that an unbounded array risks Convex's per-document size limits and belongs in its own table. Only CodeRabbit and Qodo said so clearly.

Optimistic concurrency control was the test I expected nobody to pass. I added a platformStats table with a single totalMutations field, then wrote to it from every mutation in the app, for every user. That guarantees write conflicts under any real load, since Convex's OCC model retries a mutation whenever something it read gets written first.

My first attempt at this test was subtler, tracking activity per project, and it scored a clean zero across all nine bots. Making the anti-pattern blatant changed the picture: five tools named the hot document and the retry storm directly, and three more got partial credit for spotting a concurrency problem without framing it as contention. Cubic went and searched the Convex docs to confirm it. That's the gold standard for how a review bot should behave.

Tools I didn't test

Aikido was going to be the tenth bot until I found out it wanted $350 a month before I could run a single comparison. Thanks, but no thanks.

I also left out Cursor's Bugbot and the newer review tooling from OpenAI and Anthropic. People suggested all three when I posted that I was working on this, and they're good suggestions. Nine tools was already a lot for one round of testing, so those are going into a follow-up along with GitHub's agentic workflows, which I suspect might be the real answer here.

What these tools should fix

A few things came up across all nine that have nothing to do with review quality.

Onboarding varied wildly. Some tools wanted a credit card before I could try anything, which puts me off immediately. The ones I liked opened with a short wizard asking what I wanted out of it, PR summaries or code reviews or both, and set up sensible defaults from there.

Dashboards were worse. Graphite's main nav is icons with no text, and finding the settings that actually configure the review bot took me three guesses. It isn't alone in that.

The bigger miss is context. None of these bots read the .cursor/rules files sitting in the repo, and those rules contain most of the Convex knowledge the bots kept failing on. Some of them may be reading AGENTS.md or CLAUDE.md files, which this project didn't have. The smarter move would be to detect a Convex project, go find the ESLint config and the framework docs, and keep that in the project's knowledge for every future review. Nobody's doing that yet.

My pick

I'm going with Copilot for day-to-day work. It did well on Convex-specific code, it's already part of GitHub, and there's no new billing relationship to set up.

If accuracy is the only thing you're optimizing for and you can tolerate a rough dashboard, Qodo is hard to argue with, and it's free. If false positives are what you can't stand, take Cubic, which was the only tool here that never cried wolf.

The broader lesson from the exercise is that the best AI code review tools are the ones that know something about your stack. Every bot in this lineup handles generic JavaScript fine. The gap between first place and last was almost entirely about whether the tool understood how Convex actually executes your code.

All gas, no breakages

Convex is the reactive backend platform that keeps up with you and your agents. Database, functions, workflow, sync, search, file storage, and more. All TypeScript, zero glue.

Get started