Bobby Alv
a day ago
MongoDB to Convex: A Step-by-Step Data Migration Script
Ah, data migration. The unsung hero of the developer’s life. It’s like moving houses, but instead of furniture, you’re hauling bits and bytes from one database to another. And just like moving, it’s always more complicated than you think. In this article, I’ll walk you through how I wrote a script to migrate data from MongoDB to Convex (shoutout to convex.dev), and hopefully, you’ll learn something — or at least get a good laugh.
The Setup: MongoDB and Convex Sitting in a Tree First, let’s set the stage. I had a MongoDB database with over 30 tables and over 2M documents that relate to one another. But MongoDB, while great for many things, wasn’t cutting it for our new app’s real-time needs. Enter Convex, a backend-as-a-service that promised to make my life easier with its real-time capabilities and serverless architecture. The only problem? Getting all that data from MongoDB into Convex.
So, I rolled up my sleeves, brewed a pot of coffee, and got to work.
The Code: A Tale of Two Databases Here’s the script I wrote to migrate the data. It’s a bit of a beast, but I’ll break it down for you. (You can find the full code at the end of this article, but let’s walk through the highlights.)
Step 1: Connecting to MongoDB and Convex
First things first: I needed to connect to both databases. MongoDB was easy — I just used the MongoClient from the mongodb package. Convex, on the other hand, required a bit more setup, including an auth token.
1const mongoClient = new MongoClient(MONGODB_URI);
2const convexClient = new ConvexHttpClient(CONVEX_URL);
3
4await mongoClient.connect();
5convexClient.setAuth(authToken);
6Pro tip: If your auth token is invalid, Convex will throw an error faster than you can say “unauthorized.” So, make sure you’ve got the right token.
Step 2: Filtering and Counting Documents Next, I needed to filter the team/organization/workspace object in MongoDB that I wanted to migrate. I created a filter only include documents from specific teams and to make sure that the object was not deleted.
1const filter = {
2 team: { $in: [
3 new ObjectId('628...'),
4 new ObjectId('628...')
5 ]},
6 isDeleted: { $ne: true }
7};
8
9const contactsCount = await contactsCollection.countDocuments(filter);
10const taskCount = await tasksCollection.countDocuments(filter);
11// ...and so on
12I console logged This gave me a sense of how much data I was dealing with. Spoiler: It was a lot.
Step 3: The Migration Loop Now, the fun part: the migration loop. I processed the data in batches of 100 to avoid overwhelming the system (Convex only allows for a little over 16k). For each contact, I checked if it had already been merged (using a Set to track processed IDs) and skipped it if it had.
1for (let skip = skipCount; ; skip += batchSize) {
2 const batch = await contactsCollection.find(filter).skip(skip).limit(batchSize).toArray();
3 if (batch.length === 0) break;
4
5 for (const oldContact of batch) {
6 if (processedMongoIds.has(oldContact._id.toString())) {
7 console.log(`Contact already merged: ${oldContact.firstName} ${oldContact.lastName}`);
8 continue;
9 }
10 // ... process the contact ...
11 }
12}
13This loop is the heart of the script. It’s where the magic (and the headaches) happen.
Step 4: Formatting Data for Convex I had changed my schema in Convex, so I had to reformat the MongoDB data to fit. This included mapping phone numbers, emails, and addresses into my new structure.
1const phoneNumbers = [];
2if (oldContact.phoneNumbers?.length > 0) {
3 phoneNumbers.push(...oldContact.phoneNumbers.map(p => ({
4 label: p.phoneLabel || "Other",
5 number: p.phone,
6 isBad: p.isBadNumber || false,
7 isPrimary: p.isPrimary || false
8 })));
9}
10I also had to handle edge cases, like contacts with no first or last name. (Yes, those exist. No, I don’t know why.)
Step 5: Creating Contacts and Tasks in Convex Once the data was formatted, I used Convex mutations to create new records, tasks, tags, activities… and so on.
1const result = await convexClient.mutation('contacts:create', newContact);
2Step 6: Error Handling and Retries Of course, nothing ever goes perfectly. I added error handling and a retry mechanism to deal with hiccups like network issues or cursor timeouts. Of course, nothing ever goes perfectly. I added error handling and a retry mechanism to deal with hiccups like network issues or cursor timeouts
1async function retryOperation(operation, maxRetries = 3, delay = 1000) {
2 for (let attempt = 1; attempt <= maxRetries; attempt++) {
3 try {
4 return await operation();
5 } catch (error) {
6 if (attempt === maxRetries) throw error;
7 console.log(`Attempt ${attempt} failed, retrying in ${delay}ms...`);
8 await new Promise(resolve => setTimeout(resolve, delay));
9 delay *= 2; // Exponential backoff
10 }
11 }
12}
13
14The Aftermath: Lessons Learned After hours of debugging, coffee, and the occasional existential crisis, the migration was complete. Here’s what I learned:
- Batch processing is your friend. It keeps things manageable and prevents your script from crashing under its own weight.
- Error handling is non-negotiable. Things will go wrong, so plan for it.
- Log everything. When something breaks (and it will), you’ll want to know where and why.
- Convex is awesome. Once the data was in, Convex made real-time updates a breeze.
The Code Here’s the full script for your reading pleasure. Feel free to adapt it for your own migrations — just don’t forget the coffee.
1import { MongoClient, ObjectId } from 'mongodb';
2import { ConvexHttpClient } from 'convex/browser';
3
4const MONGODB_URI = 'YOUR MONGO URI';
5const CONVEX_URL = 'CONVEX CLOUD URL';
6const createdContacts = new Map();
7async function migrateProperties() {
8 const mongoClient = new MongoClient(MONGODB_URI);
9 const convexClient = new ConvexHttpClient(CONVEX_URL);
10 try {
11 console.log("Connecting to MongoDB...");
12 await mongoClient.connect();
13 const db = mongoClient.db('propbear');
14 console.log("Connected to MongoDB.");
15 console.log("Authenticating with Convex...");
16
17 const authToken = "YOUR AUTH TOKEN";
18 convexClient.setAuth(authToken);
19 const user = await convexClient.query('users:viewer');
20 if (!user) {
21 throw new Error("Authentication failed");
22 }
23 console.log("Authenticated with Convex.");
24 const propertiesCollection = db.collection('propertydetails');
25 const contactsCollection = db.collection('contactdetails');
26 const tasksCollection = db.collection('tasks');
27 const linkedPropertyDetailsCollection = db.collection('linkedpropertydetails');
28 const filter = {
29 team: { $in: [
30 new ObjectId('628...'),
31 new ObjectId('627...')
32 ]}
33 };
34 const propertyCount = await propertiesCollection.countDocuments(filter);
35 const contactCount = await contactsCollection.countDocuments(filter);
36 const taskCount = await tasksCollection.countDocuments(filter);
37 const linkedPropertyDetailsCount = await linkedPropertyDetailsCollection.countDocuments(filter);
38
39 console.log("Mongo Properties:", propertyCount);
40 console.log("Mongo Contacts:", contactCount);
41 console.log("Mongo Tasks:", taskCount);
42 console.log("Mongo Linked Property Details:", linkedPropertyDetailsCount);
43 console.log("Starting migration...");
44 let migratedCount = 0;
45 let errorCount = 0;
46 let skippedCount = 0;
47 const batchSize = 100;
48 let lastId = null;
49 const skipCount = 6800;
50 const skipCursor = propertiesCollection.find(filter)
51 .sort({ _id: 1 })
52 .skip(skipCount)
53 .limit(1);
54
55 const lastSkippedDoc = await skipCursor.next();
56 if (lastSkippedDoc) {
57 lastId = lastSkippedDoc._id;
58 }
59 while (true) {
60 const query = lastId ? { ...filter, _id: { $gt: lastId } } : filter;
61 const cursor = propertiesCollection.find(query).sort({ _id: 1 }).limit(batchSize);
62 let batch = await retryOperation(async () => await cursor.toArray());
63 if (batch.length === 0) {
64 console.log("No more properties to process. Migration completed.");
65 break;
66 }
67 let allSkipped = true;
68 for (const oldProperty of batch) {
69 try {
70 async function recordExistsByMongoId(client, table, mongoId) {
71 try {
72 const result = await client.query(`${table}:getByMongoId`, { mongoId });
73 return result ? result._id : null;
74 } catch (error) {
75 console.error(`Error checking if ${table} record exists:`, error);
76 return null;
77 }
78 }
79 try {
80 const existingPropertyId = await recordExistsByMongoId(convexClient, 'properties', oldProperty._id.toString());
81
82 if (existingPropertyId) {
83 console.log(`Property already exists: ${oldProperty.name}`, existingPropertyId);
84 skippedCount++;
85 continue;
86 }
87 allSkipped = false;
88 let contactId = null;
89 if (oldProperty.owner) {
90 const oldContact = await contactsCollection.findOne({ _id: oldProperty.owner });
91 if (oldContact) {
92 contactId = await recordExistsByMongoId(convexClient, 'contacts', oldContact._id.toString());
93 if (!contactId) {
94 const newContact = {
95 mongoId: oldContact._id.toString(),
96 recordType: "contacts",
97 firstName: oldContact.firstName || "",
98 lastName: oldContact.lastName || "",
99 fullName: `${oldContact.firstName || ""} ${oldContact.lastName || ""}`.trim(),
100 orgId: mapOrgId(oldContact.team?.toString()) || "nd70p3ngxkh8n7chaddb81tjpx7166mr",
101 phone: (() => {
102 let phoneNumbers = [];
103 if (oldContact.phoneNumbers && oldContact.phoneNumbers.length > 0) {
104 phoneNumbers = oldContact.phoneNumbers.map(p => ({
105 label: p.phoneLabel || "Other",
106 number: p.phone,
107 isPrimary: p.isPrimary || false,
108 isBad: oldContact.phoneDetails?.isBadNumber || false
109 }));
110 }
111 if (oldContact.phone && !phoneNumbers.some(p => p.number === oldContact.phone)) {
112 phoneNumbers.push({
113 label: oldContact.phoneDetails?.phoneLabel || "Other",
114 number: oldContact.phone,
115 isPrimary: oldContact.phoneDetails?.isPrimary || phoneNumbers.length === 0,
116 isBad: oldContact.phoneDetails?.isBadNumber || false
117 });
118 }
119 return phoneNumbers.length > 0 ? phoneNumbers : undefined;
120 })(),
121 email: (() => {
122 let emails = [];
123 if (oldContact.emails && oldContact.emails.length > 0) {
124 emails = oldContact.emails.map(e => ({
125 label: e.emailLabel || "Other",
126 address: e.email,
127 isPrimary: e.isPrimary || false,
128 isBad: oldContact.emailDetails?.isBadEmail || false
129 }));
130 }
131 if (oldContact.email && !emails.some(e => e.address === oldContact.email)) {
132 emails.push({
133 label: oldContact.emailDetails?.emailLabel || "Other",
134 address: oldContact.email,
135 isPrimary: oldContact.emailDetails?.isPrimary || emails.length === 0,
136 isBad: oldContact.emailDetails?.isBadEmail || false
137 });
138 }
139 return emails.length > 0 ? emails : [{ label: "Other", address: "", isPrimary: true, isBad: false }];
140 })(),
141 address: oldContact.address ? [{
142 label: "Home",
143 street: oldContact.address,
144 city: oldContact.city,
145 state: oldContact.state,
146 zip: oldContact.zipCode ? parseFloat(oldContact.zipCode) : undefined,
147 }] : undefined,
148 summary: oldContact.stickyNote,
149 };
150 Object.keys(newContact).forEach(key => newContact[key] === undefined && delete newContact[key]);
151 try {
152 let contactId;
153 if (createdContacts.has(oldContact._id.toString())) {
154 contactId = createdContacts.get(oldContact._id.toString());
155 await convexClient.mutation('contacts:updateContact', {
156 id: contactId,
157 });
158 } else {
159 contactId = await convexClient.mutation('contacts:createContactWithoutEnrichment', newContact);
160 createdContacts.set(oldContact._id.toString(), contactId);
161 }
162 if (oldContact.tags && oldContact.tags.length > 0) {
163 for (const tag of oldContact.tags) {
164 if (tag !== "Import") {
165 try {
166 const tagId = await createOrGetTag(
167 convexClient,
168 tag.name || tag,
169 "contacts",
170 contactId,
171 newContact.orgId,
172 await mapUserId(oldContact.createdBy?.toString()),
173 tag._id ? tag._id.toString() : undefined
174 );
175 if (tagId) {
176 console.log(`Created or got tag: ${tag.name || tag}`, tagId);
177 } else {
178 console.log(`Failed to create or get tag: ${tag.name || tag}`);
179 }
180 } catch (error) {
181 console.error(`Error processing tag ${tag.name || tag} for contact:`, error);
182 }
183 }
184 }
185 }
186 console.log(`Created contact: ${newContact.firstName} ${newContact.lastName}`, contactId);
187 // Create tasks for the contact
188 if (oldContact.tasks && oldContact.tasks.length > 0) {
189 for (const taskId of oldContact.tasks) {
190 const oldTask = await tasksCollection.findOne({ _id: taskId });
191 if (oldTask) {
192 const existingTaskId = await recordExistsByMongoId(convexClient, 'tasks', oldTask._id.toString());
193 if (!existingTaskId) {
194 const newTask = {
195 mongoId: oldTask._id.toString(),
196 orgId: newContact.orgId,
197 title: oldTask.task,
198 description: oldTask.description,
199 dueDate: oldTask.endDate ? new Date(oldTask.endDate).toISOString() : undefined,
200 linkedRecord: contactId,
201 assignedTo: await mapUserId(oldTask.createdForUserId?.toString()),
202 priority: mapPriority(oldTask.taskPriority),
203 column: mapStatus(oldTask.status),
204 };
205 Object.keys(newTask).forEach(key => newTask[key] === undefined && delete newTask[key]);
206 try {
207 const taskResult = await convexClient.mutation('tasks:createTask', newTask);
208 console.log(`Created contact task: ${newTask.title}`, taskResult);
209 } catch (error) {
210 console.error(`Error creating contact task ${newTask.title}:`, error);
211 }
212 } else {
213 console.log(`Task already exists: ${oldTask.task}`, existingTaskId);
214 }
215 }
216 }
217 }
218 // Create related contacts
219 if (oldContact.relatedContacts && oldContact.relatedContacts.length > 0) {
220 for (const relatedContact of oldContact.relatedContacts) {
221 const newRelatedContact = {
222 label: "Related Contact",
223 firstName: relatedContact.name.split(' ')[0] || "",
224 lastName: relatedContact.name.split(' ').slice(1).join(' ') || "",
225 email: [{
226 label: "Other",
227 address: relatedContact.email,
228 isBad: false,
229 isPrimary: true
230 }],
231 phone: [{
232 label: "Other",
233 number: relatedContact.phone,
234 isBad: false,
235 isPrimary: true
236 }],
237 orgId: newContact.orgId,
238 recordId: contactId
239 };
240 try {
241 const relatedContactResult = await convexClient.mutation('relatedContacts:createRelatedContact', newRelatedContact);
242 console.log(`Created related contact: ${newRelatedContact.firstName} ${newRelatedContact.lastName}`, relatedContactResult);
243 } catch (error) {
244 console.error(`Error creating related contact ${newRelatedContact.firstName} ${newRelatedContact.lastName}:`, error);
245 }
246 }
247 }
248 } catch (error) {
249 console.error(`Error creating contact ${newContact.firstName} ${newContact.lastName}:`, error);
250 }
251 } else {
252 console.log(`Contact already exists: ${oldContact.firstName} ${oldContact.lastName}`, contactId);
253 }
254 }
255 }
256 // Create the property
257 const newProperty = {
258 mongoId: oldProperty._id.toString(),
259 name: oldProperty.name,
260 recordType: "properties",
261 image: oldProperty.propertyImages && oldProperty.propertyImages.length > 0 ? oldProperty.propertyImages[0] : undefined,
262 orgId: mapOrgId(oldProperty.team?.toString()) || "nd70p3ngxkh8n7chaddb81tjpx7166mr",
263 propertyType: oldProperty.type || undefined,
264 address: {
265 street: oldProperty.address ? toProperCase(oldProperty.address) : toProperCase(oldProperty.name),
266 city: oldProperty.city ? toProperCase(oldProperty.city) : undefined,
267 state: oldProperty.state ? oldProperty.state.toUpperCase() : undefined,
268 zip: oldProperty.zipCode ? parseFloat(oldProperty.zipCode) : undefined,
269 },
270 location: oldProperty.coordinates && oldProperty.coordinates.lng && oldProperty.coordinates.lat ? {
271 type: "Point",
272 coordinates: [
273 parseFloat(oldProperty.coordinates.lng),
274 parseFloat(oldProperty.coordinates.lat)
275 ],
276 } : { type: "Point", coordinates: [0, 0] }, // Default coordinates if not available
277 isDeleted: false,
278 tags: oldProperty.tags ? oldProperty.tags.map(tag => tag._id?.toString()) : [],
279 yearBuilt: oldProperty.propertyData?.yearBuilt ? parseFloat(oldProperty.propertyData.yearBuilt) : undefined,
280 squareFootage: oldProperty.propertyData?.acres ? oldProperty.propertyData.acres * 43560 : undefined,
281 units: oldProperty.units ? parseFloat(oldProperty.units) : undefined,
282 price: oldProperty.propertyData?.forsale?.price ? parseFloat(oldProperty.propertyData.forsale.price) : undefined,
283 parcelNumber: oldProperty.parcelNo || oldProperty.propertyData?.parcelNo || undefined,
284 saleDate: oldProperty.lastSold ? new Date(parseFloat(oldProperty.lastSold)).toISOString() : undefined,
285 salePrice: oldProperty.propertyData?.soldPrice ? parseFloat(oldProperty.propertyData.soldPrice) : undefined,
286 landValue: oldProperty.landValue ? parseFloat(oldProperty.landValue) : undefined,
287 buildingValue: oldProperty.propertyData?.bldgValue ? parseFloat(oldProperty.propertyData.bldgValue) : undefined,
288 status: oldProperty.status,
289 primaryUse: oldProperty.propertyData?.occupancy || undefined,
290 construction: oldProperty.propertyData?.construction || undefined,
291 lotSize: oldProperty.propertyData?.acres ? parseFloat(oldProperty.propertyData.acres) : undefined,
292 zoning: oldProperty.propertyData?.zoning || undefined,
293 meterType: oldProperty.propertyData?.meterType || undefined,
294 class: oldProperty.propertyData?.class || undefined,
295 structures: oldProperty.propertyData?.structures ? parseInt(oldProperty.propertyData.structures) : undefined,
296 parking: oldProperty.propertyData?.parking || undefined,
297 };
298 // Remove createdBy field as it's not allowed
299 delete newProperty.createdBy;
300 Object.keys(newProperty).forEach(key => newProperty[key] === undefined && delete newProperty[key]);
301 // Ensure location is always present
302 if (!newProperty.location || isNaN(newProperty.location.coordinates[0]) || isNaN(newProperty.location.coordinates[1])) {
303 newProperty.location = { type: "Point", coordinates: [0, 0] };
304 }
305 try {
306 const result = await convexClient.mutation('properties:createPropertyWithoutEnrichment', newProperty);
307 console.log(`Created property: ${newProperty.name}`, result);
308 // Process tags for the property
309 if (oldProperty.tags && oldProperty.tags.length > 0) {
310 for (const tag of oldProperty.tags) {
311 if (tag !== "Import") {
312 try {
313 const tagId = await createOrGetTag(
314 convexClient,
315 tag.name || tag,
316 "properties",
317 result,
318 newProperty.orgId,
319 await mapUserId(oldProperty.creator?.toString()),
320 tag._id ? tag._id.toString() : undefined
321 );
322 if (tagId) {
323 console.log(`Created or got tag: ${tag.name || tag}`, tagId);
324 } else {
325 console.log(`Failed to create or get tag: ${tag.name || tag}`);
326 }
327 } catch (error) {
328 console.error(`Error processing tag ${tag.name || tag} for property:`, error);
329 }
330 }
331 }
332 }
333 // Create tasks for the property
334 if (oldProperty.tasks && oldProperty.tasks.length > 0) {
335 for (const taskId of oldProperty.tasks) {
336 const oldTask = await tasksCollection.findOne({ _id: taskId });
337 if (oldTask) {
338 const existingTaskId = await recordExistsByMongoId(convexClient, 'tasks', oldTask._id.toString());
339 if (!existingTaskId) {
340 const newTask = {
341 mongoId: oldTask._id.toString(),
342 orgId: newProperty.orgId,
343 title: oldTask.task,
344 description: oldTask.description,
345 dueDate: oldTask.endDate ? new Date(oldTask.endDate).toISOString() : undefined,
346 linkedRecord: result,
347 assignedTo: await mapUserId(oldTask.createdForUserId?.toString()),
348 priority: mapPriority(oldTask.taskPriority),
349 column: mapStatus(oldTask.status),
350 };
351 Object.keys(newTask).forEach(key => newTask[key] === undefined && delete newTask[key]);
352 try {
353 const taskResult = await convexClient.mutation('tasks:createTask', newTask);
354 console.log(`Created property task: ${newTask.title}`, taskResult);
355 } catch (error) {
356 console.error(`Error creating property task ${newTask.title}:`, error);
357 }
358 } else {
359 console.log(`Task already exists: ${oldTask.task}`, existingTaskId);
360 }
361 }
362 }
363 }
364 if (contactId) {
365 await convexClient.mutation('contactLinkedProperties:linkPropertyFromMongo', {
366 contactId: contactId,
367 propertyId: result,
368 relation: 'Owner',
369 orgId: newProperty.orgId,
370 });
371 console.log(`Linked contact ${contactId} as Owner to property ${result}`);
372 }
373 // After creating the main property and owner contact
374 await processLinkedProperties(convexClient, oldProperty, result, newProperty.orgId, {
375 linkedPropertyDetailsCollection,
376 contactsCollection
377 }, createdContacts);
378 migratedCount++;
379 } catch (error) {
380 console.error(`Error migrating property ${newProperty.name}:`, error);
381 errorCount++;
382 }
383 } catch (error) {
384 console.error(`Error migrating property ${oldProperty.name}:`, error);
385 errorCount++;
386 }
387 } catch (error) {
388 if (error.code === 43 && error.codeName === 'CursorNotFound') {
389 console.error('Cursor not found, restarting migration from the last successful point');
390 // You might want to implement a way to track the last successfully migrated property
391 // and restart from there instead of from the beginning
392 break;
393 }
394 throw error; // Rethrow other errors
395 }
396 // Update lastId after processing each property
397 lastId = oldProperty._id;
398 }
399 if (allSkipped) {
400 console.log(`All ${batchSize} properties in this batch were already migrated. Moving to next batch.`);
401 }
402 console.log(`Processed batch. Total migrated: ${migratedCount}, Skipped: ${skippedCount}, Errors: ${errorCount}`);
403 }
404 console.log(`Migration completed. Total migrated: ${migratedCount}, Skipped: ${skippedCount}, Errors: ${errorCount}`);
405 } catch (error) {
406 console.error('Error during migration:', error);
407 } finally {
408 await mongoClient.close();
409 }
410}
411async function processLinkedProperties(convexClient, oldProperty, propertyId, orgId, collections, createdContacts) {
412 const { linkedPropertyDetailsCollection, contactsCollection } = collections;
413 const batchSize = 10; // Reduced batch size
414 let skip = 0;
415 while (true) {
416 const linkedPropertyDetails = await linkedPropertyDetailsCollection.find({ property: oldProperty._id })
417 .skip(skip)
418 .limit(batchSize)
419 .toArray();
420 if (linkedPropertyDetails.length === 0) break;
421 for (const link of linkedPropertyDetails) {
422 await processLinkedProperty(convexClient, link, propertyId, orgId, contactsCollection, createdContacts);
423 await new Promise(resolve => setTimeout(resolve, 100)); // 100ms delay between each link
424 }
425 skip += batchSize;
426 await new Promise(resolve => setTimeout(resolve, 1000)); // 1 second delay between batches
427 }
428}
429async function processLinkedProperty(convexClient, link, propertyId, orgId, contactsCollection, createdContacts) {
430 try {
431 const oldContact = await contactsCollection.findOne({ _id: link.contact });
432 if (oldContact && oldContact._id) {
433 let linkedContactId;
434 if (createdContacts.has(oldContact._id.toString())) {
435 linkedContactId = createdContacts.get(oldContact._id.toString());
436 } else {
437 // Create the linked contact
438 const newLinkedContact = {
439 mongoId: oldContact._id.toString(),
440 firstName: oldContact.firstName || "",
441 lastName: oldContact.lastName || "",
442 fullName: `${oldContact.firstName || ""} ${oldContact.lastName || ""}`.trim(),
443 email: oldContact.email ? [{
444 label: "Other",
445 address: oldContact.email,
446 isBad: false,
447 isPrimary: true
448 }] : [{ label: "Other", address: "", isPrimary: true, isBad: false }],
449 orgId: orgId,
450 recordType: "contacts",
451 };
452 // Add company name if it exists
453 if (oldContact.company) {
454 delete oldContact.company;
455 }
456 // Use company name as fullName if no individual name is provided
457 if (!newLinkedContact.fullName && newLinkedContact.company) {
458 newLinkedContact.fullName = newLinkedContact.company;
459 }
460 // Skip truly empty contacts
461 if (!newLinkedContact.fullName.trim() && !newLinkedContact.email[0].address.trim() && !newLinkedContact.company) {
462 console.log(`Skipping empty linked contact for property ${propertyId}`);
463 return;
464 }
465 try {
466 linkedContactId = await convexClient.mutation('contacts:createContactWithoutEnrichment', newLinkedContact);
467 console.log(`Created linked contact: ${newLinkedContact.fullName || newLinkedContact.company}`, linkedContactId);
468 await convexClient.mutation('contactLinkedProperties:linkPropertyFromMongo', {
469 contactId: linkedContactId,
470 propertyId: propertyId,
471 relation: link.relation || "Other",
472 orgId: orgId,
473 });
474 console.log(`Linked contact ${linkedContactId} as ${link.relation || "Other"} to property ${propertyId}`);
475 createdContacts.set(oldContact._id.toString(), linkedContactId);
476 } catch (error) {
477 console.error(`Error creating linked contact ${newLinkedContact.fullName || newLinkedContact.company}:`, error);
478 return;
479 }
480 }
481 } else {
482 console.error(`Invalid linked contact found for property ${propertyId}:`, link);
483 }
484 } catch (error) {
485 console.error(`Error processing linked property for property ${propertyId}:`, error);
486 }
487}
488/**
489 * @param {string | undefined} oldId
490 * @returns {string | null}
491 */
492function mapOrgId(oldId) {
493 if (!oldId) return "nd70p3ngxkh8n7chaddb81tjpx7166mr";
494 const orgIdMap = {
495 '628..': 'nd7b...', // first is mongoId second is convex _id
496 '627...': 'nd7d...'
497 };
498 return orgIdMap[oldId] || null;
499}
500function mapUserId(oldId) {
501 if (!oldId) return null;
502
503 const userIdMap = {
504 '628c...': 'jx7...', // first is mongoId second is convex _id
505 ...
506 };
507 return userIdMap[oldId] || null;
508}
509function mapPriority(oldPriority) {
510 const priorityMap = {
511 'Low': 'low',
512 'Medium': 'medium',
513 'High': 'high'
514 };
515 return priorityMap[oldPriority] || 'no-priority';
516}
517function mapStatus(oldStatus) {
518 const statusMap = {
519 'In Progress': 'in-progress',
520 'Completed': 'done',
521 'Not Started': 'todo'
522 };
523 return statusMap[oldStatus] || 'todo';
524}
525function toProperCase(str) {
526 return str.replace(/\w\S*/g, function(txt) {
527 if (isNaN(txt)) {
528 return txt.charAt(0).toUpperCase() + txt.substr(1).toLowerCase();
529 } else {
530 return txt;
531 }
532 });
533}
534async function retryOperation(operation, maxRetries = 3, delay = 1000) {
535 for (let attempt = 1; attempt <= maxRetries; attempt++) {
536 try {
537 return await operation();
538 } catch (error) {
539 if (attempt === maxRetries) throw error;
540 console.log(`Attempt ${attempt} failed, retrying in ${delay}ms...`);
541 await new Promise(resolve => setTimeout(resolve, delay));
542 delay *= 2; // Exponential backoff
543 }
544 }
545}
546async function createOrGetTag(convexClient, tagName, recordType, recordId, orgId, createdBy, mongoId = undefined) {
547 try {
548 let existingTag;
549
550 // Try to get the tag by mongoId if it exists
551 if (mongoId) {
552 existingTag = await convexClient.query('tags:getTagByMongoId', { mongoId });
553 }
554 if (existingTag) {
555 // If the tag exists, return its ID
556 return existingTag._id;
557 } else {
558 // If the tag doesn't exist, create a new one
559 const newTag = {
560 name: tagName,
561 recordType,
562 orgId,
563 createdBy,
564 mongoId
565 };
566 const tagId = await convexClient.mutation('tags:createTag', newTag);
567 // Link the tag to the record
568 await convexClient.mutation('tags:linkTagToRecord', {
569 tagId,
570 recordId
571 });
572 return tagId;
573 }
574 } catch (error) {
575 console.error(`Error creating or getting tag ${tagName}:`, error);
576 return null;
577 }
578}
579
580migrateProperties().catch(console.error);
581
582Final Thoughts Data migration isn’t glamorous, but it’s a necessary evil. With the right tools and a bit of patience, you can move mountains of data without losing your sanity. And if all else fails, remember: there’s always more coffee.
More Examples: https://github.com/robertalv/merge-mongodb-convex
Happy coding! 🚀
Build in minutes, scale forever.
Convex is the sync platform with everything you need to build your full-stack project. Cloud functions, a database, file storage, scheduling, search, and realtime updates fit together seamlessly.
Get started