Re: [Haskell-cafe] Is it possible to have constant-space JSON decoding?

On Tue, Dec 04, 2012 at 03:58:24PM +0100, Herbert Valerio Riedel wrote:

Iustin Pop writes:

[...]

...

Let's say we have a simple JSON message: an array of 5 million numbers. I would like to parse this in constant space, such that if I only need the last element, overall memory usage is low (yes, unrealistic use, but please bear with me for a moment).

Using aeson, I thought the following program will be nicely-behaved:

part of the problem is that aeson builds an intermediate JSON parse-tree which has quite an overhead for representing a list of numbers on the heap as each numbers requires multiple heap objects (see also [1]). This is an area where e.g. Python has a significantly smaller footprint (mostly due to a more efficient heap representation).

Ah, I see. Thanks for the link, so that's from where the 'S' constructor was coming from in the -hd output. And indeed, I was surprised as well that Python has a more efficient representation for this.

[...]

...

It seems that the Array constructor holds a vector, and this results in too much strictness?

btw, using a list on the other hand would add an overhead of 2 words (due to the ':' constructor) for representing each JSON array element in the parse-tree, that's probably why aeson uses vectors instead of lists.

Ack. thanks, iustin