Michal Srb

3 months ago

Convex Ents: Manage your document relationships

Convex Ents: Manage your document relationships

Note: This article assumes some familiarity with Convex. If you’re not familiar with it, check out the Convex tutorial.

Convex Ents is a library for Convex providing a bunch of useful functionality:

1. Simpler ways to model and query related documents
2. Ability to easily map and filter documents retrieved from the database
3. Enforcing unique document field values
4. Defining default values for easier document shape evolution
5. Propagating deletion to related documents
6. Soft and scheduled document deletion
7. And more

While all of these can be achieved without Convex Ents, the library makes them really easy. If you’re familiar with Prisma or Drizzle ORM, you’ll find yourself at home. Let’s look at each item on the list in more detail.

Simpler ways to model and query related documents

You can store IDs of other documents in Convex documents, just like in any other relational database. These can represent 1:1 and 1:many relationships between documents, which in the Ents parlance are called “edges”:

In vanilla Convex:

// schema.ts
users: defineTable({
  name: v.string(),
}),
messages: defineTable({
  text: v.string(),
  userId: v.id("users")
})
  .index("userId", ["userId"])

// myFunctions.ts
// args: userId
const messages = await ctx.db
  .query("messages")
  .withIndex("userId", (q) => q.eq("userId", userId))
  .collect();

In this example we have two tables, users and messages, and messages have a required userId field. We also defined an index on this field, so that we can efficiently retrieve just the messages related to a given userId. Which is exactly what we did in the example query.

Now let’s look at the equivalent with Convex ents:

// schema.ts
users: defineEnt({
  name: v.string(),
})
  .edges("messages", { ref: true }),
messages: defineEnt({
  text: v.string()
})
  .edge("user")

// myFunctions.ts
// args: userId
const messages = await ctx.table("users")
  .getX(userId)
  .edge("messages");

While there are a bunch of differences in the code between this version and the “vanilla” Convex code, the semantics are exactly the same.

First, we define two “ents” (short for “entity”): users and messages. The message ents are declared to have a unique edge to the users table. This translates to the exact same code you saw above: a userId field, and an associated index. Additionally, the user ents are declared to have 1:many edges to the messages table (ref: true means that the edge is stored as a “reference” in a field - the field name is inferred). This information doesn’t affect the Convex schema, but it allows you to query the relevant messages “from” the user ent.

And that’s exactly what we do in the example query. Instead of ctx.db.query we use ctx.table. We then ask for the ent with the given userId - but we don’t retrieve it. Instead we immediately ask to traverse the 1:many “messages” edge. This performs the same indexed retrieval as the vanilla code.

Many to many relationships

So far we have saved a little bit of code, but Convex Ents shine even more when it comes to modeling many to many relationships. Let’s look at vanilla Convex example first:

// schema.ts
roles: defineTable({
  name: v.string(),
}),
permissions: defineTable({
  name: v.string(),
})
roles_to_permissions: defineTable({
  rolesId: v.id("roles"),
  permissionsId: v.id("permissions")
})
  .index("rolesId", ["rolesId", "permissionsId"])
  .index("permissionsId", ["permissionsId"])

// myFunctions.ts
// args: roleId
const rolePermissions = await Promise.all(
  await ctx.db
    .query("roles_to_permissions")
    .withIndex("rolesId", (q) => q.eq("rolesId", roleId))
    .collect(),
  (doc) => ctx.db.get(doc.permissionId),
);
// args: roleId, permissionId
const hasPermission = (await ctx.db
  .query("roles_to_permissions")
  .withIndex("rolesId", (q) =>
    q.eq("rolesId", roleId).eq("permissionId", permissionId),
  )
  .first()) !== null;

To model a many to many relationship in a relational database, you usually define another table to store the relationship, like the roles_to_permissions table in this example. You need 2 indexes on it, one for each “foreign key”, so that you can efficiently retrieve related documents from either “side” of the relationship.

Then when you do this retrieval you have to first find the relevant documents representing the relationship, and then you have to map over them to retrieve the document from the other table, this is how we get rolePermissions.

In this example we also showcase how to use one of the indexes to answer the common question: “Does this document have given relationship with this other document?”, to get hasPermission.

Now let’s look at the equivalent with Convex ents:

// schema.ts
roles: defineEnt({
  name: v.string(),
})
  .edges("permissions"),
permissions: defineEnt({
  name: v.string(),
})
  .edges("roles")

// myFunctions.ts
// args: roleId
const rolePermissions = await ctx.table("roles")
  .getX(roleId)
  .edge("permissions");
// args: roleId, permissionId
const hasPermission = await ctx.table("roles")
  .getX(roleId)
  .edge("permissions")
  .has(permissionId);

As before, this code is semantically equivalent to the vanilla Convex code, but is perhaps more clearly aligned with our intent 💡.

Let’s say that you also need to retrieve the role document itself in the previous example. This is easy with Ents:

// myFunctions.ts
const role = await ctx.table("roles").getX(roleId)
const rolePermissions = await role.edge("permissions");

All we had to do is split the chained call and await the result of the getX (get or throw) method call.

This brings us to our second item:

Ability to easily map and filter documents retrieved from the database

You’ve already seen that Convex Ents use chained method calls, similar to the built-in ctx.db API. Ents have one trick up their sleeve though: all methods are await-able. This makes the API even more fluent:

// myFunctions.ts
const allUsers = await ctx.table("users");
const user = await ctx.table("users").getX(userId);
const messages = await ctx.table("users").getX(userId).edge("messages");

This is achieved via “lazy” Promises. Unlike normal JavaScript Promises, which kick off work immediately when they’re created, the ctx.table method and methods chained to it return a lazy promise, which doesn’t perform any work until it is awaited.

This also allows ents to have extra helper methods which help with retrieving documents, performing “joins” and returning filtered data from Convex functions:

return await ctx.table("users")
  .getX(userId)
  .edge("messages")
  .map((message) => {
    const attachments = await message.edges("attachments");
    return {
      _id: message._id,
      text: message.text,
      numAttachments: attachments.length,
    };
  });

There are two main things happening in this example, using the map method:

1. We query the related attachments for given message
2. We only return the fields we want to return to the client

This is totally possible with vanilla Convex, it’s just a bit more code:

return await Promise.all(
  (
    await ctx.db
      .query("messages")
      .withIndex("userId", (q) => q.eq("userId", userId))
      .collect()
  ).map((message) => {
    const attachments = await ctx.db
      .query("attachments")
      .withIndex("messageId", (q) => q.eq("messageId", message._id))
      .collect();
    return {
      _id: message._id,
      text: message.text,
      numAttachments: attachments.length,
    };
  }),
);

We’ll pick up the pace and cover the next two points quickly:

Unique field values

In databases fields there are often “unique” fields which serve as “secondary” keys by which documents can be retrieved. In Convex we can achieve this by:

1. Defining an index on the field
2. Ensuring that a document with a given value doesn’t already exist, anywhere we write given documents

// schema.ts
users: defineTable({
  email: v.string(),
}),
  .index("email", ["email"])

// myFunctions.ts
// Before every insert, patch or replace using the `email` field:
const existing = await ctx.db
  .query("users")
  .withIndex("email", (q) => q.eq("email", email))
  .first();
if (existing !== null) {
  throw new Error(
    `In table "users" cannot create a duplicate document with field "email" of value \`${email}\`, existing document with ID "${
      existing._id as string
    }" already has it.`,
  );
}

Convex Ents have a built-in shortcut for this:

// schema.ts
users: defineEnt({}),
  .field("email", { unique: true })

// myFunctions.ts
// The uniqueness check is performed automatically

No extra code is required when writing to the users table.

Default field values

When you evolve your schema over time you’ll probably add more fields. But existing documents in the database won’t have any values for these fields yet. The easiest approach is to add an optional field:

// schema.ts
posts: defineTable({
  // ... other fields
  contentType: v.optional(v.union(v.literal("text"), v.literal("video")))
}),

In this example we added a contentType field, and made it optional. Everywhere we read posts, we can manually include a default value, in vanilla Convex:

// myFunctions.ts
return (await ctx.db.query("posts")).map((post) => ({
  ...post,
  contentType: post.contentType ?? "text",
}));
  

Usually you want to always specify the new field when writing the document. It’s not possible to automatically require this with the built-in schema validation, you have to make sure you write the value yourself.

If the default value is just a simple value like in this example, you can achieve this more easily with Convex Ents:

// schema.ts
posts: defineEnt({
  // ... other fields
})
  .field(
    "contentType",
    v.union(v.literal("text"), v.literal("video")),
    { default: "text" }
  )

// myFunctions.ts
// The "contentType" is not optional, and defaults to "text"
return await ctx.table("posts");

Since contentType is not an optional field in the document type, TypeScript can ensure that you’re always providing it when writing to the database.

Cascading deletes, soft deletion and scheduled deletion

In vanilla Convex, when a document is deleted other documents can still include “references” to it by storing the deleted document’s ID. This is a great, simple and scalable model. When querying the ID Convex will return null, and this can be handled (or ignored) by your code.

However, relationships are often required, and it can be easier to reason about your data model without “dangling references” in your documents. For this reason, Convex Ents do not support dangling references in the edges declared via edge and edges. Convex already makes this easy when writing data to the database, simply by declaring the field which stores the “foreign key” as NOT optional.

This makes deletion in general more challenging though. You can easily have a scenario where a document’s ID is stored in 1000s or even more other documents. Deleting all of these documents in a single mutation, which is within a single transaction, is simply impossible, as it would require a long-lived transaction, grinding the whole database to a halt (something Convex does not allow, instead failing the mutation).

Convex Ents include 3 deletion behaviors:

1. The default one deletes all related documents that require the existence of a given document - cascading deletions, in a single transaction. This is a fine behavior that preserves the “no dangling references” invariant, as long as you don’t expect to have many related documents.
2. The soft deletion behavior doesn’t actually delete the document, but instead sets a deletionTime field on it. It’s up to you to make sure that soft delete documents are not shown when they should not be. For example you might want to show the “group posts” of deleted users, because the posts really belong to the “group”, but you don’t show the user’s “profile”.
3. The scheduled deletion behavior combines the two: First it performs only soft deletion, and then, with an optional delay, performs the cascading delete, over possibly many scheduled mutations to make sure that each individual mutation doesn’t read or write too many documents. The deletion is performed depth first, so that no dangling references are created in the process.

Learn more about the different deletion behaviors in Cascading Deletes documentation.

Conclusion

We hope you find the library interesting, both for its own merits and as an example of an abstraction that can be built on top of the powerful Convex base. Notably, Ents is built entirely on top of vanilla Convex, and you can contribute to it or fork it to meet your own needs or preferred API ergonomics. The library is still in its early experimental stage, without the stability or quality guarantees built-in Convex provides. If it does seem promising to you, please give it a try and let us know your feedback on Discord.

Check out these links to learn more:

• Convex Ents docs
• Convex Ents repo
• Convex Ents issues
• Prisma vs Convex examples