Convert your .json array to a .jsonl (JSON Lines)
JSON Lines is a file format that stores one JSON object per line in a file. This is more scalable than storing it as a JSON array, since most JSON parsers require loading the full stringified array into memory before it can parse it. With JSON Lines, your code can load one line at a time. If you don't hold onto each object (for instance, you do some operation and save it elsewhere), your code can theoretically read an infinitely long file without running out of memory. It also allows you to easily read a stream of data.
To make these files, each object needs to be on its own line. and the file extension is .jsonl
For Convex import, we allow you to specify either format: a .json
or .jsonl
. However, we limit the .json
size to 8MB. So one question that comes up is how can you transform your .json
file into .jsonl
?
Using jq
to convert .json to .jsonl
jq
is a common tool for working with JSON from the commandline.
Here's the one-liner to convert a .json
file into .jsonl
using jq
:
1jq -c '.[]' ./mydata.json > mydata.jsonl
2
Example
Input: ./mydata.json
1[
2 {
3 "foo": "bar"
4 },
5 {
6 "easy_as": 123
7 }
8]
9
Output: ./mydata.jsonl
1{"foo":"bar"}
2{"easy_as":123}
3
Convex is the sync platform with everything you need to build your full-stack project. Cloud functions, a database, file storage, scheduling, search, and realtime updates fit together seamlessly.