What is ndJSON?
ndJSOn is a variation on JSON. Instead of a single monolithic object, an ndJSON file contains multiple valid JSON objects, delimited by newlines, hence newline-delimited JSON.
A simple ndJSON file might look like this:
{"name":"John Doe","address":"1, The Street, Someplace, Somewhere", "phone":"1-555-234-5678"}
{"name":"Jane Doe","address":"2, The Avenue, Anyplace, Anywhere", "phone":"1-555-876-5432"}
At first glance this looks similar to JSON array, but it lacks the square brackets and comma delimiters. As a whole this is not valid JSON, but each line is valid JSON. It is valid ndJSON.
Why do we need it?
The monolithic nature of JSON implies that to add an element we must first acquire the entire JSON object in working space, add the new element, then return the updated object to permanent storage. For a file this is an open-read-update-write-close operation on the entire file. This is clearly not suitable for data logging or similar applications where data is acquired from a stream and appended to a file, an open-update-close operation.
Common solutions to this requirement include comma-separated values, tab-separated values, or some fixed format layouts. Whilst effective, these formats have limitations that ndJSON overcomes. With ndJSON there's no need for a fixed number of fields, or fixed length records. Data can be structured with arrays and objects, and elements can appear in any order, facilitating changes to the included data in a way that doesn't break existing downstream applications.
Parsing ndJSON with Javascript
Here's a Javascript snippet that parses a file retrieved with fetch()
let jsonData = [];
fetch("ndJSON.json")
.then(resp=> {
// Extract the text from the response
resp.text()
.then(ndjson => {
// Split the file into an array of lines
ndjson = ndjson.split("\n");
// Iterate over the array, parsing each line
ndjson.forEach(el => {
// Skip empty lines
if (el.length !== 0) {
jsonData.push(JSON.parse(el));
}
});
// Process array of JSON data here.
console.log(jsonData);
});
});Parsing ndJSON with PHP
Two options for parsing ndJSON with PHP. The first example reads the entire ndJSON file into memory before processing. This risks exhausting memory in the case of a large file.
// Option 1 - read entire file into an array
$filename = 'ndJSON.json';
// read file into an array of lines
$jsonData = file($filename);
// Parse each line in turn to produce an array of objects
$data = array_map('json_decode', $jsonData);
// Do something with the data
echo nl2br(print_r($data,true));
// Option 2 - Read file line by line
$filename = 'ndJSON.json';
$fh = fopen($filename,'r');
$data = [];
// Read the file line by line
while ($json = fgets($fh)) {
// Do something with the data. Here we're just adding it to an array
$data[] = json_decode($json);
}
// Close file, display data
fclose($fh);
echo nl2br(print_r($data,true));
Writing ndJSON with PHP
A very simple example. While it may seem trivial here to write a JSON array, remember that the source data may be gathered at different times, and appended to the file as it arrives. Note the newline appended to each line of JSON as it is written.
$filename = 'ndJSON.json';
$object1 = (object)["name"=>"John Doe","address"=>"1, The Street, Someplace, Somewhere", "phone"=>"1-555-234-5678"];
$object2 = (object)["name"=>"Jane Doe","address"=>"2, The Avenue, Anyplace, Anywhere", "phone"=>"1-555-876-5432", "gender"=>"Female"];
// Append the line of JSON to the file. Note the newline appended to the JSON data.
file_put_contents($filename, json_encode($object1)."\n", FILE_APPEND);
file_put_contents($filename, json_encode($object2)."\n", FILE_APPEND);