Hi, The test.data is very repetitive: {"1":["a","b","c","d","e","f","g","h","i","j","k","l","m","n","o","p","q","r","s","t","u","v","w","x","y","z"],"10":["a","b","c","d","e","f","g","h","i","j","k","l","m","n","o","p","q","r","s","t","u","v","w","x","y","z"],...} Perhaps (after parsing (which might fuse enough to avoid a memory spike), otherwise during parsing might require modifications to aeson?) you could compress it by interning the symbols using a `Map Text Text` to generate one canonical `Text` object for each unique string. `pack "a" == pack "a"` under `Eq` but they might be different `Text` objects. You might also need to `copy` the `Text` objects, which might be slices referencing the input. On 08/08/18 21:24, Vanessa McHale wrote:
I'm not 100% sure it's applicable in your case, but perhaps trying threadscope would be a good idea?
Based on the heap profile, I'm guessing that most of the allocations are due to slicing ByteStrings up. What happens when you try to read a map of Integers?
I'd also note that your example uses evaluate and performGC which isn't entirely realistic and thus may not actually correspond to the performance of the application in practice. I gather it *was* a problem in practice but nonetheless it's always good to benchmark the right thing.
On 08/08/2018 02:54 PM, Lana Black wrote:
Hello Cafe,
Recently while trying to optimize some code, I noticed that aeson requires quite a substantial amount of memory when parsing map-like structures. The quick example I crafted [1] is quite surprising. In order to parse an 11 megabyte file aeson requires around 300 megabytes of memory (according to the heap profiler), and the total memory footprint (according to htop) jumps to around 700 megabytes.
Is there a way to reduce this number to something less insane? I suspect that I might be doing something wrong here, but I can't exactly see what.
Claude -- https://mathr.co.uk