Nipunn Koorapati

a year ago

Adding Data Integration to Your Convex Application with Fivetran

Convex now integrates with Fivetran via a brand new source connector! This unlocks lots of data engineering use cases for apps built on Convex. Essentially, Convex apps can now replicate data to external data warehouses like Snowflake, Databricks, Google Big Table, and many more from Fivetran’s destination catalog.

Check out Fivetran's post on their new integrations and SDKs here.

OLTP and OLAP Databases

The database at the core of Convex is best suited for Online Transaction Processing (OLTP) use-cases like running a live site and subscribing to realtime queries. Convex is designed to optimize for these use cases so you can build realtime, reactive applications. Data warehouses, on the other hand, are optimized for Online Analytical Processing (OLAP) use-cases like deep analysis on past data. These two types of databases have fundamentally different design.

OLTP databases (like Convex) typically optimize for small, predictable indexed queries. OLTP thrives on point queries (fetching/writing a single document) and indexed range queries over small ranges. OLTP databases often use Optimistic Concurrency Control (OCC) to allow for greater throughput in independent mutations by executing multiple mutations at the same time optimistically, failing them on write if they conflict. OLTP databases often used journaled row-based data structures with auxiliary indexes. This design lends itself to small, fast queries.

OLAP databases (like Snowflake, Databricks, and BigTable) are best suited for large and possibly unpredictable queries like table scans. OLAP databases are typically organized in a column-oriented structure that enables parallel analysis over large amounts of data. High transaction rates and low latency are typically not a priority.

It can even be dangerous to run large analytics queries against OLTP databases. They can result in unpredictable load on the system that could adversely affect your live traffic. Predictability is reliability’s best friend.

Connecting Convex to Data Warehouses with Fivetran

With the new Convex source connector for Fivetran, you can get the best of both OLTP and OLAP. Build your live application on Convex, then use Fivetran to get your data to your Data Warehouse and Business Intelligence (BI) tools of choice for analysis.

For those who want to just get started quickly, Fivetran with a managed Google BigQuery offering which works nicely with Google Looker Studio. If you want more control of the setup, you can always choose your own destination.

Getting Started with Convex+Fivetran

You’ll need a Convex Professional Plan. Learn about the different plans here. Then head to your convex dashboard here.

1. Grab your Convex Credentials

Grab your Deployment Credentials from the Convex deployment dashboard by going to the Settings section on the bottom of the left nav bar. Your deployment credentials will be the first screen that shows up.

2. Set up your Fivetran Connector

Create a Fivetran account and create a new connector. Select the destination of your choice. Select the “Convex” source connector. From there, you drop your credentials in and get syncing.

Next Steps

During the early access period, you’ll have to contact us in our community Discord or through support@convex.dev to get started, but it’ll be freely available soon.

We'd love your feedback on the integration as we continue polishing it with the Fivetran team for a beta release.