[JSON Lines](https://jsonlines.org/) is a file format that stores one JSON object per line in a file. This is more scalable than storing it as a JSON array, since most JSON parsers require loading the full stringified array into memory before it can parse it. With JSON Lines, your code can load one line at a time. If you don't hold onto each object (for instance, you do some operation and save it elsewhere), your code can theoretically read an infinitely long file without running out of memory. It also allows you to easily read a stream of data. 

To make these files, each object needs to be on its own line. and the file extension is .json**l**

For <QuickConvex /> [import](https://docs.convex.dev/database/import-export/import), we allow you to specify either format: a `.json` or `.jsonl`. However, we limit the `.json` size to 8MB. So one question that comes up is **how can you transform your `.json` file into `.jsonl`?**

# Using `jq` to convert .json to .jsonl
[`jq`](https://jqlang.github.io/jq/) is a common tool for working with JSON from the commandline.

Here's the one-liner to convert a `.json` file into `.jsonl` using `jq`:

```sh
jq -c '.[]' ./mydata.json > mydata.jsonl
```

# Example

**Input: `./mydata.json`**

```json
[
    {
        "foo": "bar"
    },
    {
        "easy_as": 123
    }
]
```

**Output: `./mydata.jsonl`**

```js
{"foo":"bar"}
{"easy_as":123}
```

JSON Lines stores one JSON object per line in a file, which is more scalable. Here's the one-liner to convert your .json file to .jsonl

Convert your .json array to a .jsonl (JSON  Lines)

# Knowledge Base: Comparing Realtime Databases

## Comparing Realtime Databases

Realtime databases are a popular choice for developers looking to build applications that require instant updates. They are particularly beneficial for applications that involve collaborative features, messaging systems, live tracking, and real-time analytics. To contextualize this, here are a bunch of examples of applications that greatly benefit from a realtime backend layer:

1. Chat and Messaging Applications: Realtime databases are ideal for chat and messaging applications where instant message delivery and synchronization are crucial. Users can see messages in real time, enabling seamless and interactive conversations.
2. Collaborative Tools: Applications that involve real-time collaboration, such as project management tools, document editing platforms, or whiteboarding applications, can greatly benefit from a realtime database. It allows multiple users to work together simultaneously, seeing each other's changes in real time.
3. Live Tracking and Location-Based Services: Applications that require live tracking, such as ride-sharing services, delivery tracking, or asset management systems, rely on realtime databases to provide accurate and up-to-date location information to users.
4. Real-Time Analytics and Dashboards: Applications that require real-time data visualization and analytics, such as financial platforms, monitoring systems, or IoT (Internet of Things) applications, can leverage realtime databases to display live data and generate real-time insights.
5. Multiplayer Games: Realtime databases are essential for multiplayer games, enabling real-time interactions between players, synchronization of game states, and instant updates of scores and achievements.
6. Auction and Bidding Platforms: Applications that involve auctions, bidding, or real-time bidding systems benefit from realtime databases to ensure that bids are instantly reflected, providing a seamless and fair experience for users.
7. Social Media Feeds: Social media platforms heavily rely on realtime databases to deliver real-time updates of posts, comments, likes, and notifications to users, creating an engaging and interactive user experience.

Three tools that often appear in the conversation of realtime databases are Convex, Supabase, and Firebase. Here, we'll look past the surface level details and go over the actual developer experience when trying to implement realtime on each of these platforms.

### Convex

Convex stands out by providing realtime capabilities out of the box. With Convex, developers get reliable realtime updates without the need for additional configuration or manual intervention. This is achieved through Convex's intelligent query dependency tracking system. When a dependency in a query changes, Convex automatically updates all relevant queries, ensuring that the data remains up to date in real time.

In addition to its built-in realtime capabilities, Convex also offers excellent integration with React. Developers can leverage Convex's React support to seamlessly incorporate realtime updates into their React applications. This abstraction of the realtime functionality simplifies the development process and allows developers to focus on building their applications rather than dealing with the intricacies of realtime data synchronization.

### Supabase

Supabase does technically offer realtime capabilities. However, it requires developers to manually handle refetching data when a row changes. On the other hand, Convex automates this process by updating all queries when a dependency changes.

Some developers also mention that Postgres has realtime capabilities, but it can be challenging to set up and manage. Supabase's realtime feature is built on top of Postgres and Phoenix PubSub, but it still requires effort to configure. However, there are frameworks emerging that simplify the process and offer similar functionality to what Convex provides.

### Firebase

Firebase also provides realtime updates, but it follows a similar approach to Supabase, where developers need to handle row-oriented updates.

When comparing Convex and Firebase, Convex offers better integration with React and abstracts the realtime updates in a more efficient manner. Firebase, on the other hand, pushes the subscription predicate logic to the client, which may not be ideal for all applications. It is worth noting that Convex recommends putting this logic on the server for better security and easier authorization management.

### Wrap-up

Essentially, consider factors such as ease of setup, automatic updates, integration with React, and the level of control over subscription logic. Supabase, Convex, and Firebase each have their strengths and weaknesses in these area. We're biased, but we believe that Convex offering this out of the box is ultimately a superior approach.

Learn about the existing options for databases that come with realtime elements and find out how they compare.

Comparing Realtime Databases

`npm run dev` is the standard for "run my website locally," but how does it work? How can we expand its functionality? In this post we'll look at:
- How to configure what `npm run dev` does.
- How to decompose complex commands into granular units.
- How to run multiple commands in parallel.
- How to run pre-requisites without losing normal `Ctrl-C` behavior.
- How to add seed data (if none exists) when starting up a <QuickConvex /> backend.

As a motivating example, here are some scripts defined in the [convex-helpers](https://github.com/get-convex/convex-helpers) example app. We'll cover what each piece does

```json
  "scripts": {
    "dev": "npm-run-all --parallel dev:backend dev:frontend",
    "build": "tsc && vite build",
    "dev:backend": "convex dev",
    "dev:frontend": "vite",
    "predev": "convex dev --until-success",
    "test": "vitest"
  },
```


# How and where they're defined

`npm run` runs commands that are defined in your `package.json` in your project's workspace. These commands are often pre-configured when you start your repo from a command like `npm create vite@latest` with commands for:
- `dev`: Run a development environment. This often includes autor-reloading the UI when files change. For [Vite](https://vitejs.dev/) this is `vite` and [Next.js](https://nextjs.org/) is `next dev`.
- `build`: Build the website for deployment. This will generally compile and bundle all your html, css, and javascript. For Vite this is `vite build` and Next.js is `next build`.
- `test`: Run tests - if you're using [Jest](https://jestjs.io), it's just `"test": "jest"` or `vitest` for [Vitest](https://vitest.dev/).

Here's a basic example from Next.js:
```js
// in package.json
{
// ...
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
  },
//...
```

Here you can run `npm run dev` or `npm run lint` etc.

You can learn more about `npm run` in [the docs](https://docs.npmjs.com/cli/v10/commands/npm-run-script).


# Why use scripts?

It's a fair question why one would put commands that already so simple into package scripts. Why not just call `jest` or `vite` or `next build`? There's a few good reasons:

1. You can save the default parameters for commands so you don't have to remember or document the "standard" way of starting something. We'll see below how you can configure it to chain commands and run others in parallel.
2. It allows you to easily run commands that are installed by `npm` but not globally accessible from your shell (terminal).[^1] When you install things like `npm install -D vitest`, it installs `vitest` into `node_modules/.bin`.[^2] You can't run `vitest` directly in your shell,[^3] but you can have a config like: `"scripts": { "test": "vitest" }` and `npm run test` will run `vitest`.
3. It always runs with the root of the package folder as the "current directory" even if you're in a subdirectory. So you can define a script like `"foo": "./myscript.sh"` and it will always look for `myscript.sh` in the package root (in the same directory as package.json). Note: you can access the current directory where it was called via the `INIT_CWD` environment variable.
4. You can reference variables in the `package.json` easily when the script is run from `npm run`. For instance, you can access the "version" of your package with the `npm_package_version` environment variable, like `process.env.npm_package_version` in js or `$npm_package_version` in a script.
5. If you have multiple workspaces (many directories with their own package.json configured into a parent package.json with a "workspaces" config), you can run the same command in all workspaces with `npm test --workspaces` or one with `npm run lint --workspace apps/web`. 

[^1]: The way your shell finds which command to run when you type `npm` is to check the shell's `PATH` environment variable (on unix machines anyways). You can see your own with `echo "$PATH"`. It checks all the places specified in `$PATH` and uses the first one. 

[^2]: Technically you can override & specify where npm installs binaries.

[^3]: If you really want to, you can run `npm exec vitest`, `npx vitest` for short, `./npm_modules/.bin/vitest` directly, or add `.npm_modules/.bin` to your PATH.

# Does `npm run dev` work with yarn / pnpm / bun?

Yes! Even if you install your dependencies with another package manager, you can still run your package scripts with npm.

```sh
yarn # similar to `npm install`
npm run dev # still works!
```
You don't have to remember that `npm run dev` maps to `yarn dev` (or `yarn run dev`). The same goes for `npx`: `npx convex dev` works regardless of what package manager you used to install things.

# Running commands in parallel

There are a couple packages you can use to run commands concurrently:[^4]

[^4]: Some people use a bare `&` to run one task in the background, but that is not supported on Windows, and interrupting one command won't necessarily kill the other.

1. [`npm-run-all`](https://github.com/mysticatea/npm-run-all)
2. [`concurrently`](http://npmjs.org/package/concurrently)

We'll just look at `npm-run-all` here. Consider our example:
```json
  "scripts": {
    "dev": "npm-run-all --parallel dev:backend dev:frontend",
    "dev:backend": "convex dev",
    "dev:frontend": "vite",
  },
```
This defines three scripts. 
1. `npm run dev:backend` runs `convex dev`.
2. `npm run dev:frontend` runs `vite`. 
3. `npm run dev` runs both `convex dev` and `vite` in parallel via `npm-run-all`.

Both outputs are streamed out, and doing Ctrl-C will interrupt both scripts.

# predev? postbuild?

You can specify commands to run before (pre) or after (post) another command (say, X) by naming your command `preX` or `postX`. In the example:
```json
  "scripts": {
    "dev": "npm-run-all --parallel dev:backend dev:frontend",
    "dev:backend": "convex dev",
    "dev:frontend": "vite",
    "predev": "convex dev --until-success",
  },
```

This will run `convex dev --until-success`, before the "dev" command of `npm-run-all --parallel dev:backend dev:frontend`.

## Chaining with "&&"

For those used to shell scripting, you can run two commands in sequence if the previous one succeeds with `commandA && commandB`. This works on both Windows and Unix (mac / linux). 

However, there's a couple advantages to just using `pre`-scripts:
1. You can run either command with `npm run dev --ignore-scripts` to not do the "predev" script, or `npm run predev` to  explicitly only do the "predev" step.
2. The Ctrl-C behavior is more predictable in my experience. In different shell environments, doing Ctrl-C (which sends an interrupt signal to the current process) would sometimes kill the first script but still run the second script. After many attempts we decided to switch to "predev" as the pattern.

# Run interactive steps first

For Convex, when you first run `npx convex dev` (or `npm run dev` with the above scripts), it will ask you to log in if you aren't already, and ask you to set up your project if one isn't already set up. This is great, but interactive commands that update the output text don't work well when the output is being streamed by multiple commands at once. This is the motivation for running `npx convex dev --until-success` before `npx convex dev`. 

- `convex dev` syncs your functions and schema whenever it doesn't match what you have deployed, watching for file changes. 
- The `--until-success` flag syncs your functions and schema only until it succeeds once, telling you what to fix if something is wrong and retrying automatically until it succeeds or you Ctrl-C it. 
- By running `npx convex dev --until-success`, we can go through the login, project configuration, and an initial sync, all before trying to start up the frontend and backend. 
- The initial sync is especially helpful if it catches issues like missing environment variables which need to be set before your app can function. 
- This way the frontend doesn't start until the backend is ready to handle requests with the version of functions it expects.

# Seeding data on startup

If you change your "predev" command for <QuickConvex/> to include `--run` it will run a server-side function before your frontend has started.
```json
  "scripts": {
	  //...
    "predev": "convex dev --until-success --run init",
		//...
  },
```

The `--run init` command will run a function that is the default export in `convex/init.ts`. You could also have run `--run myFolder/myModule:myFunction`. See [docs on naming here](https://docs.convex.dev/functions/query-functions#query-names). See [this post](https://stack.convex.dev/seeding-data-for-preview-deployments) on seeding data but the gist is that you can define an [internalMutation](https://docs.convex.dev/functions/internal-functions) that checks if the database is empty, and if so inserts a collection of records for testing / setup purposes.

# tsc?

If you use TypeScript, you can run a type check / compile your typescript files with a bare `tsc`. If your `tsconfig.json` is configured to emit types, it will write out the types. If not, it will just validate the types. This is great to do as part of the build, so you don't build anything that has type errors. This is why the above example did:
```js
    "build": "tsc && vite build",
```

# How to pass arguments?

If you want to pass arguments to a command, for instance passing arguments to your testing command to specify what test to run, you can pass them **after** a `--` to separate the command from the argument. Technically you don't need `--` if your arguments are positional instead of `-`-prefixed, but it doesn't hurt to always do it in case you forget which to do it for.

```sh
npm run test -- --grep="pattern"
```

# Summary

We looked at some ways of using package.json scripts to simplify our workflows. Who knew how much power could rest behind a simple `npm run dev`? Looking at our original example:
```json
  "scripts": {
    "dev": "npm-run-all --parallel dev:backend dev:frontend",
    "build": "tsc && vite build",
    "dev:backend": "convex dev",
    "dev:frontend": "vite",
    "predev": "convex dev --until-success",
    "test": "vitest"
  },
```
- `dev` runs the frontend and backend in parallel, after `predev`.
- `build` does type checking via `tsc` before building the static site.
- `dev:backend` continuously deploys the backend functions to your development environment as you edit files.
- `dev:frontend` runs a local frontend server that auto-reloads as you edit files.
- `predev` runs before `dev` and does an initial deployment, handling login, configuration, and an initial sync as necessary.
- `test` uses Vitest to run tests. Note: `npm test` is shorthand for `npm run test` along with other commands, but they're special cases. `npm run test` is the habit I suggest.

With a simple `npm run dev` we can sign up a user for Convex, add seed data, and run the frontend and backend (including database) in parallel. Here's how we do it, and how to undo it too.

Supercharge `npm run dev` with package.json scripts

# What is a Database Transaction?

## Definition

A database transaction refers to a logical unit of work that consists of one or more database operations. It is a fundamental concept in database management systems, ensuring the integrity and consistency of data. A transaction allows multiple database operations to be treated as a single, indivisible unit, either all succeeding or all failing. This guarantees that the database remains in a consistent state, even in the event of failures or concurrent access by multiple users.

## What is ACID?

Transactions have the following four qualities, which is usually communicated as the acronym, ACID. Note that different implementations of transactions will provide guarantees of differing “strength.”

### Atomicity

Atomicity is one of the key properties of a database transaction. It ensures that all the operations within a transaction are treated as a single, indivisible unit. In other words, either all the operations within the transaction are successfully completed, or none of them are. This guarantees that the database remains in a consistent state, even in the face of failures or concurrent access by multiple users. By maintaining atomicity, database transactions provide reliability and predictability in data manipulation.

### Consistency

Consistency is a crucial property of a database transaction. It ensures that the database moves from one consistent state to another consistent state after the completion of a transaction. In simpler terms, it means that the data in the database adheres to a set of predefined rules or constraints. If any operation within a transaction violates these rules, the entire transaction is rolled back, and the database remains unchanged. By enforcing consistency, database transactions maintain the integrity and reliability of the data, ensuring that it remains accurate and valid throughout the transaction process. Traditionally (and in Convex), this is “immediate” consistency - in that once you write data to the database, you can immediately read the value you wrote. Some systems offer a weaker “eventual” consistency, which means you may not be able to observe the changes you made until a later point.

### Isolation

Isolation is an essential property of a database transaction that ensures concurrent access to the database does not interfere with the integrity of the data. It allows multiple transactions to execute concurrently without impacting each other's results. Each transaction is isolated from others, creating a virtual environment where it appears to be the only transaction running. There are varying levels of isolation. Convex offers “serializable” isolation by default, which is the strongest isolation. This prevents issues such as dirty reads, non-repeatable reads, and phantom reads. Most systems will default to a weaker level, for instance Postgres defaults to “read committed” which can allow non-repeatable reads and phantom reads. By providing isolation, database transactions maintain data integrity and ensure that the results of concurrent transactions are consistent and reliable. 

### Durability

Durability is a critical property of a database transaction that guarantees the permanence of its changes. Once a transaction is committed, its effects are durable and will persist even in the event of system failures or crashes. This is achieved through the use of transaction logs and other mechanisms that ensure the changes are written to non-volatile storage. By providing durability, database transactions ensure that the data remains intact and recoverable, maintaining the reliability and consistency of the database over time.

By providing ACID properties, database transactions ensure reliable and predictable data manipulation.

## Convex and Transactions

When using Convex, all of your reads and writes with your database within a server function are **automatically transactional** and **ACID compliant**, even though Convex isn't a SQL database.

This may come to a surprise for developers, as most document stores are not transactional, as they optimize for performance, sacrificing key reliability features. Unlike traditional relational databases, NoSQL databases prioritize horizontal scalability and high-speed data processing, often sacrificing transactional capabilities. The absence of transactions in NoSQL databases allows for greater flexibility and agility in handling large volumes of data**.** However, this lack of transactional support can lead to potential data inconsistencies or integrity issues in certain scenarios.

NoSQL databases typically adopt a "eventual consistency" model, where data consistency is achieved over time rather than instantaneously. This means that changes made to the database may take some time to propagate across all nodes in a distributed system, resulting in temporary inconsistencies.

While some NoSQL databases offer limited transactional support for specific use cases, such as single-document operations, they generally lack the comprehensive ACID properties provided by traditional relational databases.

Understanding database transactions is essential for full-stack developers. So, what is a database transaction and how does it work?

What is a Database Transaction?

## Definition

A client is a program or device that requests services or resources from a server in a client-server architecture. It initiates communication, sends requests, and interacts with the server to obtain the desired information or perform specific actions.

## Client-Server Architecture

In the client-server architecture, the client typically initiates communication by sending a request to the server. This request can be for various services or resources, such as data retrieval, file uploads, or authentication. The server, on the other hand, responds to these requests by providing the requested services or resources.

The communication between the client and the server typically occurs over a network, using protocols such as HTTP or TCP/IP. The client sends requests to the server, specifying the desired actions or data, and the server responds with the requested information or performs the requested operations.

This allows for a distributed and scalable approach to software development. It enables the separation of concerns, where the client is responsible for the user interface and user interactions, while the server handles the business logic and data management. This separation allows for easier maintenance, scalability, and flexibility in building complex software systems.

In TypeScript, you can create clients using various approaches and frameworks, depending on your specific requirements and project preferences. One popular approach is to use the Fetch API, which is built into modern web browsers. The Fetch API provides a powerful and flexible way to make HTTP requests from the client to the server.

## What are WebSockets?

When you're using Convex, client-server architecture is facilitated through WebSockets, which act as intermediaries between the client and server. They provide bidirectional communication, operating over a single TCP connection, which is ideal for low-latency, high-performance exchange. Unlike the typical request-response model, WebSockets enable real-time, full-duplex communication between the client and the server.

WebSockets act as a communication channel that allows the server to push data to the client without the need for the client to continuously send requests. This persistent connection eliminates the overhead of establishing new connections for each request, resulting in reduced latency and improved efficiency.

Convex takes advantage of WebSockets to make your application realtime, as the established WebSocket between your database/cache and application will update your app with new data without needing to refresh the page. This enables live chat functionality, collaborative editing, and general multiplayer capabilities. Whenever new information is available, the app server will update your user's client.

## Examples

Here are some examples of clients commonly used in TypeScript development:

1. Web Browsers: Web browsers, such as Chrome, Firefox, and Safari, are the most common type of client in web development. They send HTTP requests to servers and receive responses to display web pages.
2. Mobile Applications: Mobile apps developed using TypeScript, such as those built with frameworks like React Native or Ionic, act as clients. They communicate with servers to fetch data, send user inputs, and receive updates.
3. Desktop Applications: These applications are installed and run on a user's computer and communicate with servers through APIs, allowing users to fetch data, submit forms, and perform various tasks directly from their desktop.
4. Command Line Interfaces (CLIs): TypeScript can be used to build command-line tools or CLIs that act as clients. These clients interact with servers through APIs, enabling developers to perform various tasks from the command line.
5. Server-side Applications: In some cases, TypeScript applications can act as both clients and servers. For example, a TypeScript server application may make requests to external APIs or services, acting as a client to retrieve data or perform certain actions.

Learn what a client is in the context of fullstack TypeScript development and understand the client's role in the client-server architecture.