Ian Macartney

8 months ago

Convert your .json array to a .jsonl (JSON Lines)

JSON Lines is a file format that stores one JSON object per line in a file. This is more scalable than storing it as a JSON array, since most JSON parsers require loading the full stringified array into memory before it can parse it. With JSON Lines, your code can load one line at a time. If you don't hold onto each object (for instance, you do some operation and save it elsewhere), your code can theoretically read an infinitely long file without running out of memory. It also allows you to easily read a stream of data.

To make these files, each object needs to be on its own line. and the file extension is .jsonl

For Convex import, we allow you to specify either format: a .json or .jsonl. However, we limit the .json size to 8MB. So one question that comes up is how can you transform your .json file into .jsonl?

Using jq to convert .json to .jsonl

jq is a common tool for working with JSON from the commandline.

Here's the one-liner to convert a .json file into .jsonl using jq:

jq -c '.[]' ./mydata.json > mydata.jsonl

Example

Input: ./mydata.json

[
    {
        "foo": "bar"
    },
    {
        "easy_as": 123
    }
]

Output: ./mydata.jsonl

{"foo":"bar"}
{"easy_as":123}