Re: [Haskell-cafe] JSON parser that returns the rest of the string that was not used

On Sun, May 29, 2016 at 1:53 PM, Stephen Tetley wrote:

Isn't this a problem of JSON rather than it's parsers?

I can understand that a file with multiple JSONs is not a legal "JSON text". But... isn't that issue separate from whether parsers expect terminated strings, or, conversely, are tolerant of arbitrary text following the JSON expr? Scheme "read" functions from time immemorial would read the first expression off a handle without worrying about what followed it! It doesn't mean the whole file needs to be valid JSON, just that each prefix chewed off the front is valid JSON. Thanks to Nikita for the links to json-stream and json-incremental-decoder. My understanding is that if I use a top-level array to wrap the objects, then these approaches will let me retain a streaming/incremental IO. I'm not sure yet how to use this to stream output from a monadic computation. Let me be specific about the scenario I'm trying to handle: Criterion loops over benchmarks, and after running each, it writes the report out to disk appending it to a file: https://github.com/bos/criterion/blob/fb815c928af2cb089cea9399503304530e2788... This way, the report doesn't sit in memory affecting subsequent benchmarks. (I.e. polluting the live set for major GC.) When all benchmarks are complete, the reports are read back from the file. There are bugs in the binary serialization used in the linked code. We want to switch it to dump and read back in JSON instead. In this case, we can just write an initial "[" to the file, and then serialize one JSON object at a time, interspersed with ",". That's ok... but it's kind of an ugly solution -- it requires that, we, the client of the JSON serialization API, make assumptions about the serialization format and reimplement a tiny tiny fraction of it. Cheers, -Ryan On Sun, May 29, 2016 at 1:53 PM, Stephen Tetley wrote:

Hi Ryan

Isn't this a problem of JSON rather than it's parsers?

That's too say I believe (but could easily be wrong...) that a file with multiple JSON objects would be ill-formed; it would be well-formed if the multiple objects were in a single top-level array.

...

As someone who spent many years putting data in S-expression format, it seems natural to me to write multiple S-expressions (or JSON objects) to a file, and expect a reader to be able to read them back one at a time.

This seems comparatively uncommon in the JSON world. Accordingly, it looks like the most popular JSON parsing lib, Aeson, doesn't directly provide

On 29 May 2016 at 18:09, Ryan Newton wrote: this

...

functionality. Functions like decode just return a "Maybe a", not the left-over input, meaning that you would need to somehow split up your multi-object file before attempting to parse, which is annoying and error prone.

It looks like maybe you can get Aeson to do what I want by dropping down to the attoparsec layer and messing with IResult.

But is there a better way to do this? Would this be a good convenience routine to add to aeson in a PR? I.e. would anyone else use this?

Thanks, -Ryan

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe