Jamal Lyons

20 days ago

Building Type-Safe Rust Applications with Convex: Introducing convex-typegen

Building Type-Safe Rust Applications with Convex: Introducing convex-typegen is a guest Stack Post from Convex community member CodingWithJamal

If you've been following the backend-as-a-service landscape, you've likely heard of Convex. This innovative platform has been turning heads by offering a unique combination of developer experience, serverless functions, and real-time subscriptions, all wrapped in a developer-friendly package. What makes Convex particularly interesting is that under the hood, it's powered by Rust – a language choice that speaks volumes about its commitment to performance and reliability.

Speaking of Rust, it's fascinating to see how this systems programming language has found its way into backend development. While traditionally associated with low-level programming, Rust has become increasingly popular for building backend services, and for good reason. Its zero-cost abstractions, memory safety guarantees, and fearless concurrency make it an excellent choice for building high-performance, reliable services. Companies like Discord, Dropbox, and Cloudflare have all embraced Rust for critical backend components, proving its worth in production environments.

The relationship between Convex and Rust runs deep. Not only is Convex's core written in Rust, but they also provide first-class support for Rust developers through their official client library. This makes perfect sense – if you're building applications that demand both real-time capabilities and high performance, the combination of Convex's infrastructure and Rust's efficiency is hard to beat.

However, there was one piece missing from this otherwise perfect puzzle: type safety between your Convex backend and Rust frontend. While TypeScript developers enjoyed seamless end-to-end type safety with Convex, Rust developers had to manually maintain their types. That's where our journey begins – creating convex-typegen, a tool that brings the same level of type safety and developer experience to the Rust ecosystem.

Pain Points

When building applications with Convex, type safety is crucial for catching errors early and providing a great developer experience. TypeScript developers have it easy – Convex's TypeScript SDK automatically generates type definitions that perfectly match their backend schema and functions. This means they get real-time type checking, autocomplete suggestions, and compile-time guarantees that their code will work as expected.

For Rust developers, however, the story was different. Despite Rust being one of the most type-safe languages available, integrating with Convex required manually defining and maintaining types that mirror the backend schema. This manual process was not only tedious but also error-prone. Imagine updating a field in your Convex schema and then having to track down every Rust type that needs updating – it's the kind of maintenance burden that can slow down development and introduce subtle bugs.

Let's look at a concrete example. Consider this simple Convex schema:

import { defineSchema, defineTable } from "convex/server";
import { v } from "convex/values";

export default defineSchema({
  messages: defineTable({
    body: v.string(),
    userId: v.id("users"),
  }),
  users: defineTable({
    name: v.string(),
    tokenIdentifier: v.string(),
  }).index("by_token", ["tokenIdentifier"]),
});

Before convex-typegen, Rust developers had to manually maintain corresponding types:

#[derive(Debug, Clone)]
struct MessageDocument
{
    body: String,
    userId: String,
}

#[derive(Debug, Clone)]
struct UserDocument
{
    name: String,
    tokenIdentifier: String,
}

This manual mapping becomes even more complex when dealing with:

• Nested objects and arrays
• Optional fields
• Union types
• Custom validators
• Function arguments and return types

What we needed was a tool that could automatically generate these Rust types, keeping them in perfect sync with the Convex backend, while respecting Rust's strong type system and idioms. This would not only eliminate the manual maintenance burden but also provide the same level of type safety that TypeScript developers enjoy.

Options I Considered

Before building convex-typegen, I explored using a generic JSON approach to manage type maintenance. This method leverages the flexibility of JSON to dynamically handle data without predefined types. In Rust, this can be achieved using the Value type from the serde_json crate, which allows you to work with JSON data in a type-agnostic manner.

use serde_json::Value;

However, when working with Convex, you have the option to use Convex's own Value type, which is specifically designed to represent the various data types supported by Convex's API. This type provides a more seamless integration with Convex's functions and can be a more idiomatic choice when interacting with Convex services.

use convex::Value as ConvexValue;

Example: Building a Message with Convex's Value Type

Consider the following MessageDocument struct that matches your schema:

#[derive(Debug, Clone)]
struct MessageDocument
{
    body: String,
    userId: String,
}

To construct a message using Convex's Value type, you need to create a BTreeMap<String, ConvexValue>:

use convex::Value as ConvexValue;
use std::collections::BTreeMap;

let create_message_args = BTreeMap::from([
    ("body".to_string(), ConvexValue::String("Crabs can walk in all directions, but mostly walk and run sideways".to_string())),
    ("userId".to_string(), ConvexValue::String("j97b03kebvey8gnqmdzhvwf69974e6c6".to_string())),
]);

client.mutation("createMessage", create_message_args).await?;

This BTreeMap represents a structured message that aligns with the MessageDocument schema, allowing you to pass it to Convex functions seamlessly.

Trade-offs While both approaches can simplify initial development by eliminating the need to define and update Rust types manually, they come with significant trade-offs:

• Loss of Type Safety: By using generic JSON or Convex's Value, you forfeit Rust's strong type guarantees. Errors related to data structure mismatches will only surface at runtime, rather than being caught at compile time.

• Reduced Developer Experience: Without explicit types, you lose the benefits of Rust's powerful type system, such as autocomplete suggestions and compile-time checks. This can lead to a more error-prone and less efficient development process.

In summary, while the generic JSON approach and Convex's Value type can be quick fixes for avoiding manual type maintenance, they sacrifice the robustness and efficiency that Rust is known for. This underscores the need for a tool like convex-typegen, which can automatically generate Rust types that are perfectly in sync with your Convex backend, preserving both type safety and developer productivity.

Building convex-typegen

Embarking on the development of convex-typegen was a thrilling adventure that took me deep into the realms of Abstract Syntax Trees (ASTs) and the art of code generation. My mission was clear: to craft a tool that seamlessly converts Convex's JavaScript schema definitions into robust Rust types. This would not only enhance type safety but also liberate developers from the tedium of manual type maintenance. Here's a glimpse into how I brought this vision to life:

Step 1: Parsing JavaScript with Oxc

The first step in my journey was to parse the JavaScript schema definitions into a format that I could work with programmatically. This is where Abstract Syntax Trees (ASTs) come into play. An AST is a tree representation of the abstract syntactic structure of source code. Each node in the tree denotes a construct occurring in the source code.

To parse JavaScript into an AST, I used the Oxc JavaScript Oxidation Compiler. Oxc is a high-performance toolchain written in Rust, known for its speed and efficiency. Here's a simplified example of how I used it, inspired by the Oxc documentation:

use oxc::{
    allocator::Allocator,
    parser::{Parser, ParserReturn},
    span::SourceType,
};

let source_text = r#"
import { defineSchema, defineTable } from "convex/server";
import { v } from "convex/values";

export default defineSchema({
  messages: defineTable({
    body: v.string(),
    userId: v.id("users"),
  }),
  users: defineTable({
    name: v.string(),
    tokenIdentifier: v.string(),
  }).index("by_token", ["tokenIdentifier"]),
});
"#;

// Memory arena where AST nodes are allocated.
let allocator = Allocator::default();
// Infer source type (TS/JS/ESM/JSX/etc) based on file extension
let source_type = SourceType::from_path("schema.js").unwrap();
let mut errors = Vec::new();

let parser_result = Parser::new(&allocator, source_text, source_type).parse();

// The program is the root of the AST.
let program = parser_result.program;

This AST provided a structured representation of the JavaScript code, which I could traverse and analyze to extract the necessary information for generating Rust types.

Step 2: Analyzing the AST

Once I had the AST, the next step was to analyze it to identify the schema definitions and their corresponding types. This involved traversing the AST and looking for specific patterns that matched Convex's schema definitions.

For example, I looked for defineSchema and defineTable calls and extracted the field names and types. This information was crucial for generating the corresponding Rust types.

fn find_define_schema(body: &[JsonValue]) -> Option<&JsonValue>
{
    for node in body {
        // Check if this is an export default declaration
        if let Some(declaration) = node.get("declaration") {
            // Check if this is a call expression
            if declaration["type"].as_str() == Some("CallExpression") {
                // Check if the callee is defineSchema
                if let Some(callee) = declaration.get("callee") {
                    if callee["type"].as_str() == Some("Identifier") && callee["name"].as_str() == Some("defineSchema") {
                        // Return the declaration node
                        return Some(declaration);
                    }
                }
            }
        }
    }
    None
}

Step 3: Constructing a Code Generator

With the AST parsed and analyzed, the next step was to save the schema and function information into Rust data types. This involved creating structured representations of the Convex schema, tables, columns, and functions.

use serde::Value as JsonValue;

/// The convex schema.
#[derive(Debug, Serialize, Deserialize)]
struct ConvexSchema
{
    tables: Vec<ConvexTable>,
}

/// A table in the convex schema.
#[derive(Debug, Serialize, Deserialize)]
struct ConvexTable
{
    name: String,
    columns: Vec<ConvexColumn>,
}

/// A column in the convex schema.
#[derive(Debug, Serialize, Deserialize)]
struct ConvexColumn
{
    name: String,
    data_type: JsonValue,
}

/// A list of all known convex functions.
type ConvexFunctions = Vec<ConvexFunction>;

/// Convex functions (Queries, Mutations, and Actions)
#[derive(Debug, Serialize, Deserialize)]
struct ConvexFunction
{
    name: String,
    params: Vec<ConvexFunctionParam>,
    type_: String,
    file_name: String,
}

/// A parameter in a convex function.
#[derive(Debug, Serialize, Deserialize)]
struct ConvexFunctionParam
{
    name: String,
    data_type: JsonValue,
}

These data structures allowed me to store information in a way that was organized and easy to manipulate. By serializing and deserializing these structures, I could seamlessly convert between the parsed AST and the Rust code I needed to generate.

Achieving Type Safety in Convex Functions One key challenge was ensuring type safety in Convex functions. This required parsing the function arguments and generating corresponding Rust code that maintained type integrity. By analyzing the AST, I extracted function definitions, including their parameters and types, and mapped them to Rust's type system.

For each Convex function, I generated Rust code that defined the function's parameters and function query key, ensuring they matched the schema's expectations. This involved creating Rust structs and enums that mirrored the JavaScript types, providing compile-time guarantees of type safety.

Here's a conceptual example of how I approached this:

fn generate_function_code(function: ConvexFunction) -> String
{
    let mut code = String::new();

    // Generate the args struct name
    let struct_name = format!("{}Args", capitalize_first_letter(&function.name));

    // Generate struct with derive macros
    code.push_str("#[derive(Debug, Clone, Serialize, Deserialize)]\n");
    code.push_str(&format!("pub struct {} {{\n", struct_name));

    // Generate fields for each parameter
    for param in &function.params {
        let rust_type = convex_type_to_rust_type(&param.data_type, None, None);
        code.push_str(&format!("    pub {}: {},\n", param.name, rust_type));
    }

    code.push_str("}\n\n");

    // Add implementation block with static FUNCTION_PATH method
    code.push_str(&format!("impl {} {{\n", struct_name));
    code.push_str("    pub const FUNCTION_PATH: &'static str = ");
    code.push_str(&format!("\"{}:{}\";\n", function.file_name, function.name));
    code.push_str("}\n\n");

    // Generate From implementation to convert to BTreeMap
    code.push_str(&format!(
        "impl From<{}> for std::collections::BTreeMap<String, serde_json::Value> {{\n",
        struct_name
    ));
    code.push_str(&format!("    fn from(_args: {}) -> Self {{\n", struct_name));

    // Only create map and insert values if there are parameters
    if function.params.is_empty() {
        code.push_str("        std::collections::BTreeMap::new()\n");
    } else {
        code.push_str("        let mut map = std::collections::BTreeMap::new();\n");
        // Convert each field to a serde_json::Value and insert into map
        for param in &function.params {
            code.push_str(&format!(
                "        map.insert(\"{}\".to_string(), serde_json::to_value(_args.{}).unwrap());\n",
                param.name, param.name
            ));
        }
        code.push_str("        map\n");t6
    }

    code.push_str("    }\n");
    code.push_str("}\n\n");

    code
}

There are many more details to this process, but this gives you a good idea of how I approached it.

Putting it All Together

To start using convex-typegen, follow these steps:

Step 1: Setting Up build.rs First, configure your build.rs file to generate types from your Convex schema. This setup ensures that your types are always up-to-date with any changes in your schema.

use std::path::PathBuf;
use convex_typegen::{generate, Configuration};

// Auto-rebuild the types if the schema or messages files change
println!("cargo:rerun-if-changed=convex/schema.ts");
println!("cargo:rerun-if-changed=convex/messages.ts");

let config = Configuration {
    // Tell the parser where to find the function definitions
    function_paths: vec![PathBuf::from("convex/messages.ts")],

    ..Default::default()
};

match generate(config) {
    Ok(_) => {}
    Err(e) => panic!("Typegen failed: {}", e),
};

This script leverages convex-typegen to parse your schema and generate corresponding Rust types, automatically rebuilding them whenever your schema changes. The default file generated is convex_types.rs, but you can change this in the output_file field of the Configuration struct. Make sure to add this file to your src/main.rs so cargo can find it.

Step 2: Writing Type-Safe Code With the types generated, you can now write type-safe code that interacts with your Convex backend. Here's how you can perform the same message mutation with type safety:

use convex::ConvexClient;
use convex_typegen::convex::ConvexClientExt;

// Import the generated types
use convex_types::CreateMessageArgs;

let client = ConvexClient::new("your-convex-url").await?;

let message_args_map = client.prepare_args(CreateMessageArgs {
    body: "Crabs are omnivores, meaning they eat both plants and animals.".to_string(),
    userId: "j97b03kebvey8gnqmdzhvwf69974e6c6".to_string(),
});

let message_result = client.mutation(CreateMessageArgs::FUNCTION_PATH, message_args_map).await?;

In this version, CreateMessageArgs is a Rust struct generated by convex-typegen, ensuring the arguments you pass are type-checked at compile time. This not only prevents runtime errors but also enhances the developer experience with features like autocomplete and inline documentation. The FUNCTION_PATH constant is a string that points to the function you're calling, and it's generated by convex-typegen based on the function's location in your convex backend directory.

Closing Thoughts

Building convex-typegen was more than just solving a technical problem—it was an opportunity to embrace challenges, learn deeply, and reaffirm key principles of software development.

As developers, we often face scenarios where existing tools don't quite meet our needs. In these moments, the ability to create something new becomes crucial. Developing convex-typegen required not only a thorough understanding of Convex and Rust but also a willingness to innovate and tackle problems without a predefined path.

This project highlighted the critical role of type safety in modern software development. By ensuring type consistency across the backend and frontend, we can identify errors early, minimize runtime issues, and improve overall reliability. convex-typegen brings the robust type safety of TypeScript to the Rust ecosystem, providing compile-time guarantees that boost developer confidence.

Additionally, it underscores the transformative power of automation in streamlining workflows. Automating the generation of Rust types from Convex schemas saves time, reduces human error, and enables developers to focus on what truly matters—building features and solving complex problems. This focus not only enhances productivity but also leads to higher-quality software.

Ultimately, the journey of building convex-typegen reflects the importance of innovation, the power of type safety, the value of community collaboration, and the efficiency of automation. I hope this tool empowers other Rust developers to build more robust and reliable applications with ease.

Footnotes

1. GitHub Repository: View the source code and examples on GitHub.
2. YouTube Demo: Watch a demonstration of convex-typegen in action on YouTube.
3. Convex: Learn more about Convex and its capabilities on their official website.
4. Rust: Discover why Rust is a popular choice for backend development on the Rust official site.
5. Oxc JavaScript Oxidation Compiler: Explore the Oxc documentation for more details on parsing JavaScript with Rust.
6. serde_json: Check out the serde_json documentation for working with JSON in Rust.