Which is best for coding, claw code or codeex? This is a question people have been asking since the dawn of time. Well, a few months anyway. Man, has it only been a few months? Anyway, to try and answer that question, I built the same app six times. Three times with claw code and three times with codeex. To add a little bit of variance, each run was either using superbase, firebase or convex. So, which agent produced the best quality code? Which produced the most secure code? And which backend is best for AI? Well, stick around as finding this out actually took me quite a lot longer than I expected. So, I have got a lot to share with you guys. But first, make sure you grab yourself a lovely cup of tea, drop me a like and sub, and then we're going to get into it. All right. First, I have to explain my methodology a little bit. Now, don't click off just yet. I know you guys hate this bit, but I do have to try and explain how I try to keep things as fair as possible. There's some really interesting learnings in here, but I promise I'll try and keep it as brief as possible. So, the way it works is from a high level perspective, we have firstly the build step um and then we have a functional evaluation step and then we have a security evaluation step and then a code quality. So, let's take a look at the build step first. So, um as I said built it six times, three times with clawed code, opus 4.8 8 high and three times with codeex GPT 5.5 high and for each claud code one time was with firebase one time with superbase one time was with convex now to isolate my system my locally running PC here from this agent so they don't you know get any pollution from skills or memories or anything like that these run using the API in an isolated docker container and they were [music] given this prompt so we've told the agent to build a collaborative project management web app from scratch using react vit typescript and superbase. Now the only thing that changes between each run is just typescript and convex typescript and firebase. So you can see the only thing that's going to change is just the backend [music] name that we're choosing. So there's no other kind of like information that we're giving it based upon each backend. So, I chose this project management app um because I think it it fairly stresses all three of them, all the three different backends fairly and is a realistic enough complex enough app that hopefully I should be able to get some good signal. So, rather than go through all of these different things that it's going to have, I might as well just show you. So, here's an example. I think this one's using Convex. Um but they basically all look roughly the same. a bit of differences between the two, but the idea is you've got authentication. So, you have to log in and you have roles [snorts] and you have organizations and organizations can have members again like I said with different roles, admin, member, viewer and then you can have projects within those organizations. Projects can have boards and boards are very much just like canban style things. So, it's kind of a very much like a Trello style app if you familiar with that. and tasks can have descriptions, comments, due dates, and file uploads. So hopefully this had enough depth to the schema that it would realistically test each model's ability to generate code use given the given back end that it was given. For each docker container, we gave it a pre-authenticated CLI. So it had a pre-authenticated convex CLI, pre-authenticated Superbase, pre-authenticated Firebase. I needed to do that because I needed to be able to have a repeatable way of running these and then destroying the artifacts after afterwards. So, I needed to basically pre-create these projects for them. Now, each agent for each of these back ends, each of these runs had 45 minutes in which to build this project, which I thought was going to be plenty, but spoiler, it apparently was not in some cases. But anyway, once it's finished building, we moved on to the actual evaluation part. um starting off with a functional evaluation. So the idea with this is I wanted to make sure that it actually worked. It actually did what we said it was supposed to be doing in the prompt. And the way that we did that is we had a manual evaluation process. So that's me personally. I went through a checklist of 40 different items. And this is what that checklist looks like. So for example, app opens without fatal runtime error. Um, a signed in user can create a board, can move tasks, uh, a viewer can read organization content. Uh, you can move things around with real-time collaboration. So, I wanted to check whether um, we could actually do real-time collaboration. So, I could see when I when user A updates something that immediately reflects on users B uh, users B's screen. Um, file uploads um, was important. Activity feeds uh, just another dimension. So, this is the kind of checklist I went through. There's 40 different items in here and I would give it a pass or fail or is blocked because something else um was broken beforehand. Now, if I couldn't actually get to do the checklist because maybe the app um didn't run, for example, for some reason, then I gave each agent a possibility to repair. And repair is basically where I say I give it back to the agent under test inside the Docker container down here. And I told and I tell it from where it is, this is the thing that's wrong. can you repair it and fix it please? And then I had had an opportunity to come back in again and then I would try and do the functional evaluation tech checklist again. Now you might be wondering why am I doing this in a manual way? Why am I as a human going through and checking this? I'll talk more about that a bit later. But it was basically turns out to be very hard to reliably um automate this process. [music] But anyway, once we've done the functional evaluation, we moved on to the security evaluation which was automated. So again, I used uh isolated docking containers different from the containers that the uh agents were building it within [music] and they would be given the source code that the agent built and also it would be given security best practices grounded in the documentation for each of the backends [music] convex, firebase and superbase. And then I had Claude and Codeex do two runs of security evaluation based upon those inputs. And if there are any disagreements between the runs or between Claude and Codeex, then it would come to a human adjudication part which is me. There were some um disagreements but genuinely it was around about severity of the issue. Most of the time, uh, Claude and Codex found the same things, but they didn't agree on the severity. And when I was doing the human adjudication, we'll talk more about that in a little bit, but there definitely were some kind of like I don't know whether that's a B or a C or a half B half C kind of thing. But anyway, once we've done that, we then ended up with a final result score from A to F. And the final evaluation I wanted to do was code quality. So we have the functional, does it work? Security, is it secure? code quality. Is this code going to be maintainable in the future? And is it going to be able to grow and scale properly? And we're not going to end up with a big bowl of spaghetti after like a week of working on this. So, this one works very similar to the security one. Uh we have an isolated docker container. It's given um some instructions to things to look out for and it's given [music] the source. So, there's things to to look out for is like architecture, modularity, backend, idioms, type safety, data flow, front end state, error handling, etc., et etc etc and then again we run it twice for claw twice for codeex and if there's any disagreements have a human adjudicate it and we get a final result at the end. Oh and one little trick that I did both for the code quality and for the security evaluation [music] is I forced the models to enumerate all of the files in the built source code and then go through each file one by one. This is a trick that I saw Mythos do um to be able to uncover security issues. It used each file as a seed, like an entry point to start its investigations from. And I think that gave me more confidence that it had checked every single code flow properly by forcing it to audit as a starting point each file. But anyway, um, all of the results and evaluations and inputs and everything and human adjudications, what I was adjudicating against is all on here. So, if you don't believe me in any of this, this is all going to open source. I've left a link to in the description down below. You can go and check this out and run this yourself. All right, that's enough jibber jabber. Let's take a look at the results now. Woo, that's uh that's a lot of dimensions to to process at once. So, let's make it easy. Let's go over here and do these one by one. So, starting off with just Convex Claude code. Convex scored a perfect perfect functional score 40 out of 40 with no repairs and security grade of B and code quality of B which is great. It took 27 minutes and cost $649. and codeex got pretty much the same 40 out of 40 B and its code quality was slightly worse than Claude with a C and but it did take significantly less time 10 minutes less time and [music] cost a little bit more actually $9.56. One issue with um the cost is it is um estimated in the codeex runs because they don't give us exact costs after the fact uh unlike the clawed ones that do. Okay, moving on. So, we have clawed code on superbase. Again, perfect functional score 40 out of 40. Uh, repairs zero. Security grade C. So, a little bit worse than convex there. And code quality again B. And this one took 40 minutes, a long time to generate and cost $6.24. And then Superbase on codeex was again perfect functional score. Everything worked. Although it did take one repair. So it didn't work on the first initial run [music] that it it was missing some environment variables I think if I remember correctly. U but then passed it back in again and it fixed it and got a perfect functional 40 out of 40. Um security-wise again agreed with clawed code here with a C and code quality again a little bit worse than clawed codes but it did only take 26 minutes and cost $5.68. And finally we have claw code on Firebase. Unfortunately, this one completely timed out. So, it had 45 minutes to build it and it didn't complete it in that 45 minutes. It did complete it eventually. So, we were able to give it a security score and a code quality score, which was D and B, but functionally I was unable to um [music] to verify whether it did what it was supposed to do. And it cost $9.85 again because it was taking so long. it was iterating in a in a looper if I remember correctly. And finally on codeex uh Firebase got a 34 out of 40, but it did take three repairs to get to that point. So, uh there was lots of things wrong when I was going through. I couldn't get to the next stage because something was broken. So, I [music] had to feed it back into the model to get it to repair. But eventually, it did uh work. And the reason why it didn't get 40 is because it still made mistakes on some of the file uploading things and it just didn't work as it should have done. Security-wise, got uh a D and code quality was a C. Uh it took 34 minutes 25 seconds and cost $9.74. All right, so now we've seen an overview of the results. Let's dig into some of the details that I thought were particularly interesting. So firstly, both Claude and Codex agreed on the security score for all three backends. They disagreed a little bit on the severity of some things that they ran [music] into, which is why I had to step in and adjudicate, but it was the actual issue details themselves that really interested me because many of the the high severity ones were due to like Firebase security rules or lowlevel security issues on Superbase, which is [music] topics that I've complained about and talked about frequently on this channel. For example, take a look at this high severity issue that Claude coded up for Firebase. Here we can see a bad security rule. If the [music] original organization creator is removed from the organization, they can then bypass that removal by basically allowing themselves to become admin again because the rule only checks that the org created by ID is equal to my UID. So effectively, there's an issue here in the security rule. But Claude is not alone in making this mistake because Codex coded up something similar. So it's the same problem again that initial admin, the one that created the organization, if they are removed from being an admin, then they can remake themselves an admin. So to me, it seems like AI has a really hard time keeping security rules [music] well secure. It just seems really easy for these kind of subtle security issues to just to [music] creep in. So, because of these high severity issues and many other security [music] issues, um, Firebase ended up with a D-grade. Superbase wasn't as bad, scoring C overall. It didn't have any really high severity issues, but it did had a whole bunch of medium ones from both Claude and Codeex. For example, on tasks, you can add comments, and the schema for comments has a task ID that it belongs to, which can be changed. But the role level security update policy only [music] checks that you are authored for the comment. That means that a malicious attacker if they know another tasks you uh ID, [music] they can move a comment onto that task even if it's in another organization which is not great. It's not as bad as gaining admin access, but it's still not good. Codex made a similar mistake on Superbase as well. It created a security definer function. Then it inserts an activity event [music] using the caller control target IDs, which means that a signed in user can potentially inject fake activity logs into organizations that that they are not a member of. [music] And again, I wasn't really surprised to see issues like these pop up because they're exactly the kind of mistakes that are easy to make with rowle security and and security rules in general. And I I've discussed them in depth in one of my recent videos on the topic. Now while Convex didn't have any high severity issues, it h however didn't come away with a [music] perfect Agrade landing itself only a B. The reason why is the same mistake was basically made by both claude and codeex around file uploading and serving. You see, no checks were done when files were uploaded to the convex server and then those files were served from the comics back end as bytes which were then turned into blob URLs [music] which then was allowed to be opened in a new browser tab. And this means that an attacker within the same organization can potentially upload a HTML document that contains malicious JavaScript in it which they could convince somebody else on their team to open which might then allow stealing of different things because it's within the same uh origin. [music] Now, I personally felt like this was a bit of a stretch, but Codeex, who I was working with this on, was very insistent that no, this should definitely knock it down from an A to a B. And a better way to do it would be to have the browser force download the contents rather than allowing it to attempt to render it. Okay. Now, code quality. So, for each backend, Claude's generation consistently scored one grade higher than Codeex. And primarily this was due to cord splitting up code into more manageable pieces rather than dumping everything into one file like codeex seemed to want to do. For example, in this convex generation, you can see that Claude has split things out nicely into different components and different files. But here you can see codeex has just dumped them all into this app.tsx. Another thing I noticed is that they both do quite a bit of reloading of data on Superbase, but Codeex does seem to [music] do more of it, which might be the result of shoving everything together. Codex also did seem to take a few more shortcuts with typing as well, whereas Claude was a little bit more diligent, for example, around the usage of any in this comx query here. Okay, time and cost. Now, I wanted to get a little bit more data on how long it takes to build each project and how much it costs. So I actually had the implementing agent build each of the six projects again another two times for a grand total of 18 different runs. And here are the results. So Codex was on average 2 to three times faster at generating projects. As you can see here 13 12 11 minutes [music] versus 23 31 and 32 minutes. Costwise they're pretty close. In total, if you add the average cost together, Claude Code was $1953 [music] and Codeex was $1589. Another interesting data point here is that from a back-end perspective, Codex seems to take about the same amount of time to generate either convex code, superbase code, or Firebase code. Whereas Claude Code seems to have a bit of an easier time with Convex versus Superbase or Firebase. All right. Finally, before we get to the conclusion, I just wanted to mention here that I originally wanted to make this entire benchmarking process automated [music] to remove any human bias from the system and to make it faster and repeatable. I wanted to be able to fire this off overnight and wake up to some nicely formatted scores. That way, we could also iterate on it at Convex and hopefully try and get better scores over time, which would hopefully result in better user experience for you guys. Well, as they say, no plan survives the battle, and it took me two weeks of battling before I realized that this plan wasn't going to work. Basically, the TLDDR is that I just couldn't get the confidence that I needed with the automated evaluation. I was just getting so much flakiness from the functional evaluator from run to run. I think a lot of this is because I was using Playright, so I was getting the model to then click buttons and stuff like a human would. and then maybe each build and each run was just slightly different or there was just a little bit of a non-determinism going on. Anyway, eventually I decided enough was enough and I just decided, right, I've spent too long on this and I'm just going to make some parts of it manual as you've seen me demonstrate in this video. And actually, when I was doing the manual evaluation, I noticed that there was lots of subtleties that the automated evaluator wasn't picking up, such as, you know, full page refreshes or other download jankiness. But overall, I'm kind of disappointed that I couldn't get it automated. Maybe future models will be smarter and more reliable, so maybe I can revisit this. So, conclusion time. [music] In the intro, past Mike asked which agent produced the best quality code. Well, I think I would have to say that Claude code has slightly more maintainable code. It took less shortcuts and typing and it split things up into more maintainable files and boundaries, but it did take a lot longer to get to that point. So, which produced the most secure code? Well, both of them actually kind of had the same result. There was however a little bit of disagreement between the security review agents, hence why I needed the manual adjudication. But in the end, both agents produced code that from a security perspective was about on par with the big difference being that AI found it much much harder to write secure Firebase or superbase code than convex code. Which backend is best for AI? Well, this is going to be a bit of a subjective one and you obviously have to take what I say with a big pinch of salt here, but after looking at the evidence, I would say that Convex is probably best for AI at this point. At least certainly for this particular project management application, not just because of the security issues that we talked about, although that is a big part. I just think that AI has an easier time writing reactive UI front front-end code using convex. When you look through the superbase or firebase code for example, you find all kinds of like callbacks and refs and effects just listed throughout the code and it just makes me shudder about the kind of hidden bugs that are just lurking in there. Not to mention like the the Superbase and Firebase apps don't consistently use atomic backend transactions uh which would mean that you can definitely run into like data corruption or performance issues in the future. Now, in Complex, these issues just don't exist because transactions are by default, and the real-time updating queries just make the code on the front end significantly easy to read and maintain, not just for us fleshy meat bags, but for AI overlords, too. And on that, I think I'll just leave this one here. I hope you enjoyed this video. And if you did, you might want to check out this one that I did a few months back where I compared clawed code to cursor. But that's about it for me for today. Until next time, thanks for watching. Cheerio.
Which agent is better at coding, Claude Code or Codex? To find out, I ran a Claude Code vs. Codex comparison the hard way. I built the same app six times, three runs with Claude Code and three with Codex. For each agent I built once on Convex, once on Supabase, and once on Firebase. I wanted to know three things: which agent produced the best-quality code, which produced the most secure code, and which backend is actually best for AI agents to work with. Finding out took a lot longer than I expected.
Methodology
The pipeline had four stages: build, functional evaluation, security evaluation, and code quality evaluation.
The benchmark methodology as a pipeline
For the build step, each agent built a collaborative project management web app from scratch, once per backend, using React, Vite, and TypeScript. That's Claude Code on Opus 4.8 high and Codex on GPT-5.5 high. To keep the comparison fair, the only thing that changed between runs was the backend name in the prompt. Each agent ran through the API inside an isolated Docker container, so no local skills or memories could leak in. Each container came with a pre-authenticated CLI for Convex, Supabase, or Firebase, which kept builds repeatable and disposable. Every run got 45 minutes, which I assumed would be plenty. It wasn't, in some cases.
I picked a project management app because it stresses all three backends at once: authentication, roles, organizations with members (admin, member, viewer), projects, Kanban-style boards, tasks with descriptions and due dates, comments, and file uploads. That's enough surface area to get real signal on how each model handles a backend it's given.
Once a build finished, I ran a manual functional evaluation against a 40-item checklist. The kinds of things it checked:
Does the app open without a fatal error?
Can an assigned user create a board and move tasks?
Can a viewer read but not edit?
Does real-time collaboration actually work (user A updates something and user B sees it without refreshing)?
Do file uploads work?
Does the activity feed populate?
Each item got a pass, fail, or blocked if something upstream was broken. If a build didn't run at all, I gave the agent one repair attempt. I handed it the container state, described what was broken, asked it to fix it, then re-ran the checklist.
I evaluated functional behavior manually because I couldn't get automated evaluation to work reliably, which I'll come back to.
Security and code quality evaluation were both automated, using isolated Docker containers separate from the ones the agents built in. I had Claude and Codex each run two security reviews grounded in the official Convex, Firebase, and Supabase documentation, auditing for the kind of access-control mistakes that show up in real apps. When the two runs or two models disagreed, I adjudicated by hand. That mostly meant resolving disagreements about severity, not about whether an issue existed at all. That produced a letter grade from A to F. Code quality worked the same way: an isolated container graded architecture, modularity, backend idioms, type safety, data flow, front-end state, and error handling, run twice per model with human adjudication on disagreements.
One thing helped both evaluations. I forced the grading models to enumerate every file in the generated source tree and walk through them one by one, using each file as an entry point to trace data flow. That surfaced issues a surface-level pass would have missed.
Here's how all six Claude Code vs. Codex builds scored:
Backend
Agent
Functional
Security
Code quality
Time
Cost
Convex
Claude Code
40/40, no repairs
B
B
27 min
$6.49
Convex
Codex
40/40
B
C
~17 min
~$9.56 (est.)
Supabase
Claude Code
40/40, no repairs
C
B
40 min
$6.24
Supabase
Codex
40/40, 1 repair (env vars)
C
C
26 min
$5.68
Firebase
Claude Code
timed out, unverified
D
B
>45 min
$9.85
Firebase
Codex
34/40, 3 repairs (file uploads)
D
C
34m 25s
$9.74
Convex and Supabase held up cleanly, and every agent hit a perfect 40/40 on both. Firebase is where things fell apart. Claude Code blew past its 45-minute window and never gave me a run I could verify functionally, iterating the whole time, and Codex only reached 34/40 after three repair attempts. Both landed at a D on security. One caveat on the cost column: Codex doesn't report exact spend after a run, so its numbers are estimated from token usage.
Where the security bugs actually came from
Claude and Codex agreed on the security grade for every backend. They sometimes disagreed on severity for a given finding, which is what drove most of the human adjudication.
Firebase produced the worst results, and the pattern was the same across both agents: security rules that looked reasonable but had a gap. Take one high-severity example. If the original creator of an organization is removed, the rule that's supposed to stop them from acting as admin only checks whether their UID matches the stored creator ID on the org document. Nothing in the rule re-checks current membership, so a removed creator can write themselves back in as admin. Codex made an equivalent mistake in its own Firebase run. Roughly, the flawed pattern looked like this:
The rule checks identity, not current membership, so removing a user from an organization doesn't actually revoke what that rule allows them to do. Both models wrote a version of this. Writing a security rule that's correct under every mutation path is a narrow, unforgiving kind of correctness. Both agents struggled with it in the same way.
Supabase scored a C. There were no high-severity findings, but both agents produced a cluster of medium-severity ones, mostly row-level security policies that checked identity without checking scope. In one case, a comment-update policy verified that the caller was the comment's author, but never checked that the comment's task actually belonged to an organization the caller was a member of. Anyone who knew another task's ID, including one in a completely different organization, could move a comment onto it. Roughly:
1create policy "authors can edit their comments"2on comments forupdate3using(auth.uid()= author_id);4
That's true but insufficient, because author_id says who wrote the comment, not whether the comment's parent task is still in scope for that user. Codex made a related mistake elsewhere in the same app. A Postgres function marked security-definer inserted activity-log events using a caller-supplied target ID, which let a signed-in user inject fake activity events into organizations they weren't a member of. Both are classic row-level security failure modes. It's the kind of thing Convex's own take on RLS argues you can avoid by keeping authorization logic in server functions instead of policies attached to the table.
Convex was the only backend without a high-severity finding. It still landed on a B rather than an A, because both agents made the same mistake around file uploads. Neither checked what was uploaded. Both served attachments by pulling the file down as raw bytes and turning it into a blob URL that opens directly in the browser. Roughly:
That URL, opened directly, renders whatever the file contains in the same origin as the app, including an HTML document with embedded JavaScript. An attacker inside the same organization could upload one and get someone else to open it. That's a real, if narrow, path to data theft. I initially thought this was a stretch, but Codex's evaluator made the case that it was enough to knock the grade from an A to a B, and I came around. Convex's file-serving docs note that a stored file can be served as a URL directly from storage.getUrl. The fix keeps that API. You just set the response to force a download, so the browser saves the file instead of rendering arbitrary uploaded content inline.
Code quality: modularity versus speed
For every backend, Claude Code's output graded one letter higher than Codex's. The difference wasn't subtle once you looked at the file tree. Claude split logic into smaller, purpose-specific files. Codex tended to dump more into fewer files, and in the Convex builds it piled components into app.tsx rather than breaking them out. Both agents did some unnecessary data reloading in their Supabase builds, but Codex did more of it. That tracked with its tendency to lump related logic together instead of separating concerns. Codex also took more shortcuts with typing. Claude was more consistent about avoiding any and keeping types flowing end to end from schema to component.
Time and cost
I reran all six build configurations two more times each, 18 runs total, purely to measure timing and cost without evaluation or repairs. Codex was consistently faster, averaging roughly 11 to 13 minutes per build against Claude Code's 23 to 32 minutes. Total average cost across all three backends came out close: Claude Code at about $19.53 and Codex at about $15.89. The backend mattered more for Claude Code's timing than for Codex's. Codex took roughly the same amount of time regardless of backend. Claude Code was noticeably faster on Convex than on Supabase or Firebase.
Why I couldn't fully automate this
I spent about two weeks trying to make the whole pipeline, including functional evaluation, run unattended so I could kick it off overnight and wake up to scored results. I couldn't get there. The functional evaluator used Playwright to simulate clicks and reads, and it was flaky from run to run. I came to believe that's because simulated UI interaction adds its own non-determinism on top of whatever the generated app is doing. I ended up doing that stage by hand instead.
The manual pass caught things the automated evaluator missed entirely. For example: a full page refresh where the app should have updated reactively, or a download that technically worked but felt janky in a way a script wouldn't flag. Automated security and code-quality review worked fine, because those are closer to static analysis. Automated behavioral testing of a UI that a different model just generated from scratch turned out to be much harder than I expected going in.
Conclusion: which backend is best for AI agents
Across this Claude Code vs. Codex comparison, the tradeoff was consistent. Claude Code produced more maintainable code: fewer typing shortcuts, better file separation, at the cost of taking noticeably longer to generate it. Codex was two to three times faster and priced close to Claude Code overall, but its code consistently graded a letter lower on quality.
On security, the two agents ended up roughly on par. One of the more interesting findings here was that the backend mattered more than the agent. Both agents found Firebase security rules and Supabase row-level security genuinely hard to get right, and both made structurally similar mistakes independently. Neither agent had that problem with Convex.
Subjectively, based on this one project, Convex came out ahead as a backend for AI agents to build against, and not only because of the security results. Both agents wrote more reactive front-end code against Convex than against Supabase or Firebase. On those two, the generated code was more likely to lean on callbacks, refs, and effects to keep the UI in sync, which is exactly the kind of code that hides subtle bugs. Neither Supabase nor Firebase generation consistently used atomic backend transactions either, which opens the door to data corruption or performance problems under concurrent writes. Convex sidesteps both issues by default. Transactions aren't something you have to remember to reach for, and reactive queries push updates to the client automatically, which made the resulting front-end code easier for both the agents and me to read.
If you want to check any of this yourself, the full benchmark repo has every prompt, container config, scorecard, and adjudication used to produce these results.
All gas, no breakages
Convex is the reactive backend platform that keeps up with you and your agents. Database, functions, workflow, sync, search, file storage, and more. All TypeScript, zero glue.
The benchmark methodology as a pipeline