Ian Macartney

25 days ago

Reimplementing Mastra Workflows: Lessons Learned

I spent a week making a component to take Mastra Workflows and replicate its behavior on Convex, using a durable function primitive I’ve made recently. Wait a sec, that was a lot of jargon. What does that actually mean?

• Mastra is a hot new framework for building and composing “agentic workflows,” which are a combination of Agents, which call LLMs with tools they can leverage, and define graphs of what should happen in what order. This post is not a dig on them, but rather to surface mistakes I made. The Mastra team has been a pleasure to work with and responsive to my various questions and comments, and supportive of hosting integrations. Big ups to them 👏.
• Convex is a backend with a reactive database, including running serverless functions in the background (and a whole lot more not as relevant to this article). I work here.
• Components in Convex are like micro-services on steroids. They have their own isolated database tables and expose an API, but commit changes in the same transaction as whoever is calling them. I wrote more about them here.
• Durable functions are functions that will survive server restarts and provide guarantees of always making progress and eventually terminating.

My component let users define their graph of Steps (functions that could call LLMs, etc) using Mastra syntax, then extract that graph and run each Step asynchronously in its own isolated function (like an AWS lambda function), with configurable parallelism limits. I pulled a lot of late nights to make it happen, cut an alpha, and was about to release it more broadly. Then I decided to stop.

What happened?

The risks with deep integrations

There’s a lot of technical risk with coupling technologies. There’s a few classes of “integrations,” each with their own pro’s and con’s

A: Reimplement the internals

The approach I took, where I try to mimic the behavior of Mastra Workflows, with my own implementation of everything but the definition DSL¹.

Pros

• Leverage the familiarity and existing traction of a known solution. In this case, letting users reference the Mastra syntax, type-safety, and docs, while also running “natively” in Convex, using first-class primitives.
• You have the opportunity to provide extra features and guarantees.

Cons

• You have to mimic every feature (and quirk) to maintain the illusion that it’s indistinguishable from running Mastra elsewhere.
• If your architectures differ significantly, you may have painted yourself into a corner.
• You are bound to their feature roadmap, and end up duplicating much of their work.

B: Wrap the API

Leverage the hard work of some product and expand on it, or expose a simpler API. In my case, if I were to make my own syntax and interface for defining workflows, but then map those to Mastra calls and delegate the actual work and running the state machine to Mastra.

Pros

• This is often the easiest path when the platform has all the primitives you need.
• An example of this is the R2 component which gives some convenience and ergonomics, while delegating the “hard” work to Cloudflare.

Cons

• You’re limited to the API of the underlying platform. While it’s easy to simplify the API of what you’re wrapping, it’s hard to make it more expressive or change its behavior, unless the primitives to do so are already exposed.
• These are often mere stepping stones for users until they are willing to interact with the underlying API directly.

C: Follow their lead

Platforms that want integrations will provide guidance and standards. In this case, to provide a simple storage provider so Mastra’s engine can persist and re-hydrate state themselves. This way, Convex users using Mastra don’t have to find another provider to store the Mastra state, while still getting the guarantees Mastra provides.

Pros

• It’s the “blessed” path and will have the most support from their team.
• The responsibilities of each are well defined and relatively isolated. Running workflows is fully delegated to Mastra.

Cons

• As the schema changes and underlying architecture changes over time, you still need to keep up, or strand users on an older version. However, as Mastra makes changes, they are likely to consider the impact to integrations and iterate on the integration API in a backwards-compatible way.

You can probably tell where I’m going with this, but the biggest mistake I made was to do too much. By coupling my component’s custom implementation with the features Mastra continues to put out, I signed up to keep up with a whole company’s worth of evolving capabilities, while not being able to say much more than “you can do that with mine too” - I’d have to keep up while only adding other capabilities “around the edges” - for instance, passing in the parallelism limits alongside the defined workflow, not configuring it in Workflow itself.

So… why did I go that route?

How could you be so naive

It’s easy in hindsight for me to scratch my head and wonder what I was thinking. With the standard revisionist history caveat, I think this is how it happened. I hope I can help you avoid falling into the same trap:

1. I got excited about the idea of Mastra + Convex. Co-locating agentic workflows with a reactive database and the rest of your business logic was and still is really compelling to me, in my somewhat-biased opinion. Running functions asynchronously with live subscriptions via Queries is a great DX & UX. Me excitedly advocating for idiomatic dataflow

2. I starting building the storage engine for Mastra and letting workflows run start to finish in a single (possibly asynchronous background) function. This was fine, but it wasn’t actually leveraging all the things I was excited about with my Workpool component.

   Me excited about the hidden internals

3. I decided that I should just do the routing and orchestration in the component, and delegate “the rest” to Mastra Workflows. I underestimated how much that would entail.

   Expectation vs reality for reimplementation

4. I then got so excited that I disappeared into execution mode, fixated on the next step. While this is useful at time (constant questioning can paralyze progress), it’s important to check in at critical decision points before over-committing.

   Where's Ian? Oh, he's in a code hole. Should we tell him his idea is risky? No, I mean his head is literally in a hole coding. He's an ostrich, did you know that?

… Which brings me to now.

Where do I go from here

Update: I've since released two Convex-native components to handle Durable Workflows with Inngest/Temporal-style syntax and Agent memory. This allows me to control my own feature roadmap and API evolution, while leveraging Convex-native primitives.

For the Mastra component, now that I’ve re-evaluated (🎶with a little help from my friends🎶), I’m going to:

1. Finish up a “blessed” integration path, so folks can use Mastra with Convex for persistence when using Mastra’s hosting. The component will handle snapshot storage, agent message history and vector search, but won't manage the graph of steps or provide the retries and durable execution.
2. Document best practices and patterns for how to use Mastra and Convex together.
3. Think about a Convex-native API that isn’t as ambitious and opinionated as Mastra, but makes it more ergonomic to use the Workflow or Workpool to chain together asynchronous tasks for agent-like use cases. Done!

Advice in hindsight

If I were to go back, I would have done it all the same way because of hard determinism… I kid (sort of). If I could go back and change something about who I was that would avoid some of the “wasted” work, I would:

• Write up a design doc enumerating the responsibilities and integration points between Mastra code and my component.
• List the assumptions I was making about how Mastra worked, and stack-rank them in order of risk. I found out too late that some code of theirs I expected to leverage was buried in private functions in classes I wasn’t using.
• Instead of reading their whole workflows code cover to cover to learn how it works, use my existing storage interface code and run real scenarios to see how it behaves. I took too long trying to understand every code abstraction instead of doing more black-box evaluation.
• Start with matching their data model, instead of deciding to roll my own. With this, I may have ended up being able to leverage more of their abstractions.
• Get to a simplified prototype faster, instead of iterating on the perfect data model first. I ended up rewriting it multiple times anyways.

Takeaway

While it’s impossible to tell the future (or is it?), there’s always things to learn from the past.

1. Do less. When building an integration, spend time weighing your options for feasibility, maintainability, and user benefit.
2. Do more. If you start going down the route of forking and replacing internals of a project, consider that it might be simpler to control experience end-to-end rather than starting with their (evolving) interface as a constraint. If you’re building something like MS Word in the cloud, don’t copy the whole UI and try to make yours behave the same way (like Google Docs). Instead, make your UI bespoke to the capabilities you care about (like Notion or Quip).
3. Do anything. While I’ve been beating myself up for wasting time, I’m glad I didn’t spend the whole week deliberating instead. I’ve come to a better strategy through experiencing an alternative instead of merely speculating.

Best of luck! Let me know what you think in Discord or on Bluesky / X.