Can your database do this? Ep 3: Zero-downtime, type-safe migrations

so I'm a big fan of movies especially long [Music] dramas these are the fanciest of movies in my mind these movies deserve a specialized website well beyond the Internet Movie Database so I built it this is the fancy Movie Database built on convex it lists every single long drama since 1990 lets you search for your favorite and it jumps right to the spot so it also kind of gives these fancy movies the treatment they deserve including nice fancy script for all of the details I was really happy with this site I mean five stars but then the other day is I was re-watching my favorite Great American fancy [Music] movie featuring two Canadian Actors as leads I noticed something [Applause] alarming I forgot that fancy movies use fancy Roman numerals for copyright years my stupid years is just regular old decimal so I found an npm package that can resolve this for me now some people might argue I could just change my query function to convert the year to Roman numerals like right when my app displays it but this is a fancy Movie Database not just a fancy movie website so I want to make sure I'm storing the fancy Roman numerals in my actual database I want my database to be fancy too so the first thing I tried it's just going into my schema file and changing the year from a number into a string but then I got some type errors in my backend code it ends up that decimals are pretty good for sorting stuff like sorting my movies into Pages ordered by year uh Roman numerals aren't so good at sorting so let's leave year a number and instead let's add a new field we're going to call fancy year this one could be a string let's check that our schema synced with convex all right schema validation is failing H all right well it's saying that there are documents in the table movies that don't match the schema because fancy years required field in fact convex tells us what we need to do here just wrap the field in optional and then we should be good to go let's do that real fast v. optional then in the background convex functions ready all right that seemed to work and after making that change to our schema convexes background Cod gen should have automatically updated our model so for example here in our component where we're passing along what values from each movie to use for to display on the page we can just tell the component to prefer the fancy ear if it exists okay was double checking our app is still working but we are showing the boring non-fancy years and that's because speaking of existing although the field is now existent in our database it's not set it's empty for every single record we marked it optional and no records have it it's time to run a migration to backfill all these values and turn our standard years into fancy years all right the first first move we need to do is head over to convex dodev components there's a whole bunch of components for convex that are these Standard Building Blocks to solve all kinds of full stack problems and backend needs but the one we're going to use today is migrations so there's a few instructions here about how to like install it we're going to do that and then we're also going to update our convex config to uh add it to our app now all we have left to do is create a migration to backfill all these fancy values I'll show you how I did it all right time to write our backfill job using the convex migrations component so the first thing we need to do speaking of that component is create a migrations object which we do by passing that mounted component to the migrations class and this is going to create a migration system that kind of manages all the background jobs uh using my data model operating on the tables in my data model so now for a specific migration we are going to backfill fancy year um and this is a defined migration that's going to operate on the movies table so this is telling the migration component we want you to make sure to call my transformation function for every single record in the movies table and this is here comes the transformation function right here it's called migrate one and it's going to be an asynchronous function and the asynchronous function takes convex as usual context object and then it's passed a single document that's given to it but from the migration component so we what we can do is we can just think about how we want to transform that one document and in this case we're going to return fields that we want to be changed in the document we want to set fancy year instead of it being undefined we want to set it equal to the kind of Roman numeral equivalent of the Year field on that dock that boring decimal year we now we going to set the fancy year a string to the output from this Romans npm package and that's it so that's our job and then the third and final step create the migrations create this kind of job step function definition and then we actually create a runnable convex function it's going to be called run backfill and we create it by calling migration. Runner and we pass it the handle to our internal um backfill fancy year job specification that's it we should be good to go now all we have to do is run it now you can run functions from the command line like we could run run backfill from there but a lot of times I like just come into the dashboard and use the function Runner here so I'm just going to choose run backfill and kick off the mutation that runs the background uh migration and let's just see what happens okay I'm starting to see some values just flow in here and everything in convex is automatically kept in sync so looks great and seem to work we can double check that the migrations done running by coming into the migrations component and calling its function get status you can see backfill fancy year State success it's processed all 3,446 of my records so now now that it's completely done we also should be able to come back into our schema file and close everything out here's our schema. TS we don't need this to be optional anymore because we know the migration is finished and there are no more missing records so if we page over to our terminal it succeeded great okay migrations done all my fancy numbers exist and we ran the whole thing while the app just kept working the type system kept us safe and guided us through it so next time you want to run a hassle-free type safe zero downtime migration grab convex and grab this migration component if it works for the notebook sure it works just fine for your app