On Sun, May 29, 2016 at 1:53 PM, Stephen Tetley
<stephen.tetley@gmail.com> wrote:
Isn't this a problem of JSON rather than it's parsers?
I can understand that a file with multiple JSONs is not a legal "JSON text". But... isn't that issue separate from whether parsers expect terminated strings, or, conversely, are tolerant of arbitrary text following the JSON expr? Scheme "read" functions from time immemorial would read the first expression off a handle without worrying about what followed it! It doesn't mean the whole file needs to be valid JSON, just that each prefix chewed off the front is valid JSON.
Thanks to Nikita for the links to json-stream and json-incremental-decoder. My understanding is that if I use a top-level array to wrap the objects, then these approaches will let me retain a streaming/incremental IO. I'm not sure yet how to use this to stream output from a monadic computation.
Let me be specific about the scenario I'm trying to handle:
Criterion loops over benchmarks, and after running each, it writes the report out to disk appending it to a file:
This way, the report doesn't sit in memory affecting subsequent benchmarks. (I.e. polluting the live set for major GC.) When all benchmarks are complete, the reports are read back from the file.
There are bugs in the binary serialization used in the linked code. We want to switch it to dump and read back in JSON instead.
In this case, we can just write an initial "[" to the file, and then serialize one JSON object at a time, interspersed with ",". That's ok... but it's kind of an ugly solution -- it requires that, we, the client of the JSON serialization API, make assumptions about the serialization format and reimplement a tiny tiny fraction of it.
Cheers,
-Ryan