This is Let's Build It. It's a Convexpowered mobile app designed to let kids build apps and games using just their voice. I've been building it with my son Joshy for the past year or so, and I'm really quite proud of it. Along the way, I learned a bunch about building Agentic applications that I think would be worth sharing with you guys. But first, I think the best way to show you what the app actually does is this little video that Josh and I put together when we launched it. Roll the clip. Hi guys, I'm Mike and this is my son Josh. Hi. And for the past year or so, we've been building an app together called Let's Build It. So, he can make his own apps and games just by talking to his iPad because he can't actually read yet, can you? No. So, Josh, just before we jump in, what's the favorite thing you've made on Let's Build It so far? A [snorts] snake math game. Cool. Shall we show them it? Yeah. So what happens in this? You drive a snake around and collecting points [music] and then what happens if you hit somebody else? Hit another snake. You have to answer math question. Cool. Do you want to show them that? Okay. I know you don't want to hit another snake, but [laughter] I'm going to get 1 + 3 equals what? Oh, that's an easy one. That question is too easy. You want to try hitting a snake again and see if you get a harder question? Did I get you? Got you. No. H 20 / 4 equals what? Oh, that's a tricky one. What does divide do, Dad? Yeah, maybe divide is too tricky. Maybe. Hey, you know what? One of the great things I love about Let's Build It is that he can make edits to his own applications and games at any time just by going into the edits. [music] Do you want to go do that now and make maybe get rid of divides cuz it's too hard? Hey Josh, what should we change in Snake Math Rescue? We should change the divide on it because it it was too tricky. Should we use just addition [music] and multiplication for the math questions? Yes, please. Using only addition and multiplication for the math questions. Let me build [music] it. Yeah. So, that's the bit I love. he could just, you know, if he has an idea or wants to make a change, then he can just do it and then he's off playing again while it's building in the background. And yeah, he's always got plenty of ideas. And so I just wanted to give him a way that he can, you know, exercise that creativity without me, without me having to type something into Claude a codeex for him every single time. And I couldn't find anything like this out there, so I just decided to build it myself. And yeah, I am a builder at heart. I love building things. And I just wanted him to have a [music] that a little bit of a sense of that, you know, joy from building things himself at 6 [music] years old. And when we were testing an early version of this, uh, some of his cousins, him and his friends, I've got video footage I'll overlay here, were actually fighting over the iPad of whose turn it was because they were loving building these games [music] and apps themselves so much. He did. He's made it. He made Turbo races. That's cool. Don't Don't touch. And there's all kinds of things they've built. They've built interactive stories, quizzes, games, drawing apps, music games, all kinds of things. Just so much creativity. It's It's awesome to see. And as a parent, to be honest, I'm not entirely keen on some of the stuff that goes on in games like Roblox. I mean, this is an actual ad that I saw for it on X a little while back. And when I saw that, I was like, "Oh my god, I really don't want my kid playing a game like this, to be honest, at 5 or 6 years old." So, with Let's Build It, there are no ads and I can set up his profile exactly how I want it inside there. Um, make sure that it's into his interests and I can control exactly the kinds of things that he builds. And yeah, so I'd much rather spend the screen time that he gets being creative and building things than, you know, passively consuming TV or watching content on his iPad. But anyway, Let's Build It is out right now for iPhone and iPad, and I'm actively working on the Android one as we speak. If you fancy trying it out with your kids, you can go to let'sbuildit.app. I'd love to see what you guys make together. Anyway, that's enough for us for today. Well, guys, see you later, Josh. See you. See you guys. Bye, Dad. What? One again. [laughter] So, that's Build It. So, for this video, I thought it would be good to take you guys through what's going on a little bit behind the scenes and then give you a few quick lessons of what I learned along the way. So, [music] hopefully you can learn from my fails. So, let's take a look. So, the way it works is that the kid, Josh in this case, talks to the friendly assistant. And when it's got enough to be able to build what it needs to build, it [music] says, "Let me build it." What that does then is then it kicks off a job on Convex, which runs his idea through a few different AI models one after the other. [music] Firstly, the conversation goes into a designer step which is running using GPTC 6 soul at the moment. Uh, which takes everything that Josh said and then turns that into a little brief for the game. The reason why I chose GPT6 souls is not just on a whim. It's because I actually ran it between like in a in a double blinded test between it [music] and various other models and I just I couldn't tell them apart to be honest. So, I went with Soul because it's the right price and it's also quick. [music] Anyway, after the designer step, I have an image model that draws a quick mockup of what the game could or should look like. [music] So, it's kind of almost like a inverse screenshot of what the funniest thing should look like. I found that by doing that that giving the model that image, it helps it write the code better, particularly when combined with those designer notes. Because often when a kid describes the thing that they want, the game or the app, they don't often use enough language for the coding model to do a good job, particularly these lower kind of like lower level coding models that I have been using, the cheaper ones. [music] Anyways, once I've done the designer step, the screenshot step, it then goes on to [music] the actual coding step. So all that gets fed in uh along with the conversation into the coding model, which actually then goes ahead and builds the game [music] or app. I'll talk a little bit more about this later, but as I said, I've been I went through a lot of different models and right now I found that Soul is got the right cost, speed, quality kind of tradeoffs. As for the output from the coding model, it's just HTML, CSS, and JS right now. So, it's pretty simple for the Convex agent to just [music] then output it to Convex file storage, which then in turn just gets pushed down to the phone when you [music] click to load one of these apps. The whole process is pretty quick. It takes about a minute from when the kids finished talking to him or her being able to play it. Okay, now I do want to say that getting the UI to look like this was actually quite a painful process. [music] I'm pretty happy with how it looks now, but getting there was yeah, was a bit of [music] a nightmare. You see, back in June when I was doing a lot of this UI work, I was using codeex a lot of the time and I asked it to restyle the whole app to match some mock-ups that I generated [music] and it went off and, you know, generated something like 20 pull requests. And when I asked it how close it actually was, it reckoned it [music] was like maybe 35 to 45% of the way there. So, I just bidded the whole lot. [music] Yeah. I I think the lesson at the time was there was just that like coding models were just not good enough at design. At least the open AI open AI ones weren't. [music] It'd be interesting to see how well like Opus or like this current batch of models does because I think they would do way way better. But anyway, back to the story. What I ended up uh doing was [music] taking one screen, one bit of the UI at a time, but putting it together in Storybook [music] and then just iterating on like smaller chunks like that using screenshots from [music] an image generated model to see what I actually wanted at the end of it. And eventually after a lot of iteration and time, I got to the state that I was happy happy with. But anyways, that was a quick look at behind the scenes of Let's Build It. Now, I promised a few lessons are the lessons that I learned along the way. So, the first one is write down what your app is supposed to do [music] in plain English and then test that. You see, when I started this project like a year ago now, I was finding that the agents were often drifting away from what I originally asked them to build. [music] So, I decided to pull out an old tool from my toolbox called BDD. BDD or behavior-driven development, if you hadn't heard of it, is a is a process where every feature that you want inside your application starts life as a story like this one, which is like semistructured natural [music] language. I probably should do a whole video on this at some point, but the idea is that you write down what you expect your app to do in natural language. Then you have your tests like these enforce each line of those behaviors. [music] So you might be like bells going off in your head here because it's kind of like a prompt, a prompt that we give to an AI except you can deterministically enforce it. [music] So you can have it run on the CI or whatever to ensure that your features and your requirements are being still being adhered to much further down in your development process. [music] But yeah, as I said, I could probably do a whole video just on this topic alone. So drop me a comment down below if you'd like to hear more about this. But anyway, just for now, let's take a look at this quick example. So, um, when Josh asks for a change to his game, he is, [music] uh, the feature requirement is that he's able to keep playing the old one until the new one is ready. So, that requirement, that story gets turned into [music] a test that clicks its way through the app in a browser. So, an end to-end test. [music] Now, you as a human should be writing those behaviors, but the tests can be written by agents. But the question is, [music] do those tests actually test what they should be testing to enforce your behavior? And the answer is most of the time, but not always. For example, we had a test here which [music] said create a kid called Max with a robot avatar. And it passed, but if you actually looked at it, the kid it made was a unicorn. So the agent had written the test. So that ignored the [music] bit about the robot. So it was just the test was happily passing without actually checking the avatar at all. So to fix that, you know, we we now have a check that runs before every commit. And if a test ignores parts of its own story, it [music] fails. The second lesson from the building process of Let's Build It is something that I've talked about a little bit before on this channel, but it's worth talking about again, and that is the power of state machines. One of the things that I kept running into like in the early development of this of [music] Let's Build it was that uh projects that the kid was building were getting stuck like forever like in a spinning loop. And a lot of the time that happened, it it came down to how the agents were just storing things in the comics database. they would end up adding like these optional fields and flags that would end up disagreeing with each other. So you would end up in like this half quasi state between [music] two different things happening. And the solution as often is is to [music] use state machines or in this case discriminated unions. So now this comic schema for a game [music] is either in its building ready failed states and it can only ever be in one of these states at any one time. It can never be in a kind of like failed but building state for example. And because it's right there in the schema, convex won't let the agent save something in between and also won't let us do a migration that could potentially [music] break existing uh rows. And then the the thing that I also really really like about state machines [music] is because you can get the agent to generate a diagram of of the your actual system and where the data is inside the system [music] because it just is the code. Oh, and I am working on a state machine component specifically for convex so you guys can be able to take leverage a lot of the power of the state machines without suffering many of the pains that I encounter along the way. [music] So keep an eye out for that as I'll hopefully have a lot more to share on it soon. So make sure you subscribe to the Convex YouTube channel so that you uh you know when that video drops. [music] Okay. And the last thing I want to share with you guys is about how I picked which AI model you know actually [music] builds the game. I hinted at this earlier, but getting the cost right for this uh application, which is a paid for application, was up on my priority list because I didn't want to have uh kids building lots of games which costing a lot of money and therefore them running out of lots of credits very quickly. But I also didn't want the quality of the generated projects to suffer either. So getting that cost to quality to build time balance just right was was very important [music] to me. and it feels like there's a new model coming out every week or so. So, back in July, I think I changed the model something like seven times in one week. It was getting a bit ridiculous. So, in the end, what I did is set up a big comparison where I took a bunch of game ideas and pitted [music] them against different models against each other. And I did it in a blind way. So, I had the agent like kick out HTML document where I would test them against each other in a blind way. So, I didn't even know which one was it which. [music] So, what I ended up with and what it's currently running right now is GPT6, which I mentioned before has got [music] just the right balance trade-off for me between cost, quality, and build speed. It costs about 16 cents per project um at [music] the moment to to generate one of these projects, but I'm always keeping an eye on that and seeing [music] the the right trade-off to make for as new models come out. So, yeah, that's a few things I learned about Let's Build It. [music] It's free on the App Store right now. If you want to give it a go with your own kids, I'll put a link down in the description down below. But anyways, I think that's enough for me for now. I hope you enjoy this video, and if you did, please do drop me a like and sub. And until next time, thanks for watching. Cheerio.
This is Let's Build It. It's a Convex powered iPhone and iPad app designed to let kids build apps and games using just their voice.
Josh's project shelf in Let's Build It, with the apps and games he has made
I've been building it with my son Josh for the past year or so and I'm really quite proud of it. Josh can't actually read yet, so the whole idea is that he makes his own apps and games just by talking to his iPad. He's always got plenty of ideas, and I wanted to give him a way to exercise that creativity without me having to type something into Claude or Codex for him every single time.
And if he has an idea or wants to make a change later on he just says so, and then he's off playing again while it's building in the background:
The assistant in Let's Build It saying "Adding multiplication questions to the math challenge. Let me build it!"
Now along the way I learnt a bunch about building an agentic application that I think would be worth sharing with you guys. So I thought it would be good to take you through what's going on a little bit behind the scenes, and then give you a few quick lessons, so hopefully you can learn from my fails.
Let's take a look.
How it works behind the scenes
So the way it works is that the kid, Josh in this case, talks to the friendly assistant. When it's got enough to go on it says "Let me build it!", and that kicks off a job on Convex which runs his idea through a few different AI models, one after the other:
An iPad with the assistant saying "Let me build it!", and an arrow to Convex running four steps: Designer, Mockup, Design notes and Coder
First the conversation goes into a designer step, which is running on GPT-6 Sol at the moment. It takes everything Josh said and turns it into a little brief for the game.
I didn't pick Sol on a whim by the way. I ran it in a blind test against various other models and to be totally honest I just couldn't tell them apart, so I went with Sol because it's the right price and it's also quick.
After the designer step an image model draws a quick mockup of what the game could or should look like. So it's kind of like an inverse screenshot of the finished thing:
The mockup the image model drew of a cake stacking game on the left, and the game that got built from it on the right
Sol then writes a few design notes to go with it. I found that giving the coding model that image helps it write the code better, particularly when combined with those notes. You see, often when a kid describes the thing they want they don't use enough language for the coding model to do a good job, particularly the cheaper models that I've been using. Here's everything that was asked for in one of the test games on the left, and the brief the designer wrote from it on the right:
The five short lines asking for a cake stacking game in a test, next to the much longer brief the designer step wrote from them
Then all of that gets fed in, along with the conversation, to the coder model, which goes ahead and builds the game or app. That's Sol again. I'll talk about this a bit more later, but I went through a lot of different models and right now I've found Sol has got the right trade-off between cost, speed and quality.
As for the output from the coding model, it's just HTML, CSS and JS right now. So it's pretty simple for the job to save it to Convex file storage, and then it gets pushed down to the phone when you tap to load one of these apps:
File Storage in the Convex dashboard, with a game's HTML file and its mockup image
The whole process is pretty quick too. It takes about a minute from when the kid has finished talking to them being able to play it.
The UI was a bit of a nightmare
Now I do want to say that getting the UI to look like this was actually quite a painful process. I'm pretty happy with how it looks now, but getting there was a bit of a nightmare.
You see, back in June when I was doing a lot of this UI work I was using Codex a lot of the time, and I asked it to restyle the whole app to match some mockups that I had generated. It went off and opened something like 20 pull requests:
A list of closed draft pull requests from June, each one called "Apply Warm Maker styling to" another part of the app
And when I asked it how close it actually was, it reckoned it was maybe 35 to 45 percent of the way there:
Codex's reply: "I'd call the current auth slice maybe 35-45% there visually. It has the palette and some surface language, but not the composition."
So I just binned the whole lot š
I think the lesson at the time was that coding models were just not good enough at design, or at least the OpenAI ones weren't. It would be interesting to see how well Opus or the current batch of models does, because I think they would do way, way better.
Anyways, what I ended up doing was taking one screen, one bit of the UI at a time, and putting it together in Storybook. Then I could iterate on smaller chunks like that, with a mockup from an image model to show what I actually wanted at the end of it:
My message to Codex with a mockup of the voice screen attached, next to the speech bubble component it built in Storybook
And eventually, after a lot of iteration and time, I got it to a state I was happy with.
Lesson 1: BDD
Now I promised a few lessons, and the first one is to write down what your app is supposed to do in plain English, and then test that.
You see, when I started this project about a year ago I was finding that the agents were often drifting away from what I originally asked them to build. So I decided to pull out an old tool from my toolbox called BDD.
BDD, or behaviour driven development if you haven't heard of it, is a process where every feature you want in your app starts life as a story written in semi-structured natural language. I could probably do a whole video just on this topic alone, but the idea is that you write down what you expect your app to do, and then you have tests that enforce each line of it. Here's one of those stories from Let's Build It:
Feature: Project Editing
As a parent helping a kid update a project
I want to describe changes with voice or text
So the project can be updated without losing the current version too early
Scenario: Editing keeps the current version live until the new version is ready
Given a parent is authenticated
And a kid "Luna" exists for the parent
And a project "Space Game" exists with 1 ready current version
When the parent opens the edit screen for "Space Game"
And the parent submits an edit request
Then the project keeps the current version live while the next version builds
And the new version is offered to the parent
So that's the requirement that when Josh asks for a change to his game, he's able to keep playing the old one until the new one is ready.
And here's the test behind that "Then" line (the app uses playwright-bdd for these):
// e2e/steps/project-editing.steps.tsThen("the project keeps the current version live while the next version builds",async({ page, scenarioState })=>{awaitexpect(page).toHaveURL(newRegExp(`/project/${getProjectId(scenarioState)}/${getOriginalVersionId(scenarioState)}$`,),);awaitexpect( page.frameLocator('[data-testid="project-html-iframe"]').getByText("Original version"),).toBeVisible();},);
It checks that we're still on the original version's page and that the original game is still the one showing (the stand-in game in the test just says "Original version" on it). Put the lines together and that story becomes a test that clicks its way through the app in a browser, so an end-to-end test:
The scenario ticked off line by line on the left. On the right the test's browser still shows the original Space Game, with a "Your new version is ready!" message
Now you might have bells going off in your head here, because it's kind of like a prompt that we give to an AI. Except you can deterministically enforce it, so you can have it run on the CI or wherever, to make sure your features and requirements are still being adhered to much further down in your development process.
A commit on main with all its checks passed: the unit tests, Storybook and seven shards of end-to-end tests
Now you as a human should be writing those behaviours, but the tests can be written by agents. So the question is, do those tests actually test what they should be testing to enforce your behaviour? And the answer is most of the time, but not always.
For example, we had a scenario with this line in it:
When the parent creates a kid with name "Max" and avatar "robot"
And it passed. But if you actually looked at it, the kid it made was a unicorn:
The home screen that scenario ended on, with kids called Luna and Max who both have the unicorn avatar
The agent had written the test so that it ignored the bit about the robot, so it was happily passing without actually checking the avatar at all.
To fix that we now have a check that runs before every commit, and if a test ignores part of its own story, it fails:
$ bun run test:bdd-check
BDD feature / step verification failed:
e2e/steps/_common.steps.ts:269 When /^the parent creates a kid with name "(.*)" and avatar "(.*)"$/ declares 2 capture(s), but the callback only binds 1 step argument(s), so 1 captured value(s) are dropped. Bind the missing parameter(s) and use them, or bind intentionally ignored values with a leading underscore and add a comment explaining why.
error: script "test:bdd-check" exited with code 1
So the line asks for a name and an avatar, the test's code only took the name, and now that gets caught before it's ever committed. Noice.
One of the things I kept running into in the early development of Let's Build It was that the projects a kid was building were getting stuck, like forever in a spinning loop. A lot of the time it came down to how the agents were storing things in the Convex database. They would end up adding these optional fields and flags that could disagree with each other, so you'd end up in this half state between two different things happening.
Here's what the projects table looked like back in June:
// convex/projects/schema.ts in June, without its indexesexportconst projectsTable =defineTable({ kidId: v.id("kids"), title: v.string(), lifecycle: projectLifecycleValidator, currentVersionId: v.optional(v.id("projectVersions")), currentVersionStatusKind: v.optional(projectCurrentVersionStatusKindValidator), pendingVersionId: v.optional(v.id("projectVersions")), pendingVersionStatusKind: v.optional(projectCurrentVersionStatusKindValidator), nextVersionNumber: v.number(), icon: v.optional(projectIconValidator),});
So a project had an optional version id, and then a separate optional flag for what state that version was in. There's nothing there to stop one being set without the other, or the two of them disagreeing.
And the solution, as it often is, is to use state machines, or in this case discriminated unions:
So now a game is either building, ready or failed (or there's nothing in that slot yet), and it can only ever be in one of those states at any one time. It can never be in a kind of failed but building state, for example.
Each build of a game is a version, and a version's own status is a union too, so the data sits with the state it belongs to. Only a ready version has the HTML file in file storage, and only an error one has a message:
And because it's right there in the schema, Convex won't let the agent save something in between, and it also won't let us push a schema change that would break the rows that are already there. Here's a little throwaway project with the same kind of union to show you. It has a row in the error state, and the schema being pushed has dropped error from the union:
$ npx convex dev
ā Schema validation failed.
Document with ID "j571c6x5233fhbqyh3fvxz2gnd8j3fcj"in table "projectVersions" does not match the schema: Value does not match validator.
Path: .status
Value: {kind: "error", message: "Build timed out"}Validator: v.union(v.object({kind: v.literal("generating")}), v.object({html: v.string(), kind: v.literal("ready")}))
So Convex is going to yell at you before anything gets broken.
The other thing I really like about state machines is that you can get your agent to generate a diagram of your actual system, and where the data is inside it, because the diagram just is the code:
A state diagram an agent drew from the schema: create or edit goes to Generating, then to Ready when the HTML is saved or to Error when generation fails
Oh and I'm working on a state machine component for Convex at the moment, so you guys can leverage a lot of the power of state machines without suffering many of the pains I ran into along the way. Hopefully I'll have a lot more to share on that soon.
Lesson 3: Picking a model
Okay, the last thing I want to share is how I picked which AI model actually builds the games.
I hinted at this earlier, but getting the cost right was high up on my priority list. The app runs on credits, and I didn't want kids building lots of games that cost a lot of money and then running out of credits very quickly:
The card a kid sees when the credits run out: "Time for a recharge! Your building battery is empty. A grown-up can recharge it."
But I also didn't want the quality of the generated projects to suffer either. So getting that balance between cost, quality and build time just right was very important to me.
The trouble is it feels like there's a new model coming out every week or so, and back in July I think I changed the model something like seven times in one week:
The git log for the model config in early July, with the seven commits that switched model highlighted
It was getting a bit ridiculous. So in the end I set up a big comparison, where I took a bunch of game ideas and pitted the different models against each other. And I did it in a blind way, so I had my agent kick out an HTML page where I could play the games side by side as A and B, without knowing which one was which:
A page asking "Which game is better?", with two builds of the same drawing app labelled A and B and buttons for A is better, About the same and B is better
What I ended up with, and what's running right now, is GPT-6 Sol. It costs about 16 cents per project at the moment, but I'm always keeping an eye on that as new models come out.
Conclusion
So just to recap: a kid talks to the assistant, a job on Convex runs their idea through a designer, a mockup, some design notes and a coder, and about a minute later there's a game sat in file storage ready to play. And the three lessons were to write down what your app should do in plain English and then test that, to keep your state in a union so it can only ever be one thing at a time, and to pick your model blind.
Let's Build It on the App Store, where it's free with in-app purchases, for iPhone and iPad
I'm a builder at heart, and I just wanted Josh to have a little bit of that joy of building things himself. If you want to give it a go with your own kids, Let's Build It is free on the App Store and there's more about it over at letsbuildit.app.
Until next time, cheerio! š
All gas, noĀ breakages
Convex is the reactive backend platform that keeps up with you and your agents. Database, functions, workflow, sync, search, file storage, and more. All TypeScript, zero glue.