Re: [Haskell-cafe] Re: Mining Twitter data in Haskell and Clojure

braver wrote:

On Jun 14, 11:40 am, Don Stewart wrote:

...

Oh, you'll want insertWith'.

You might also consider bytestring-trie for the Graph, and IntMap for the AdJList ?

Yeah, I saw jsonb using Trie and thought there's a reason for it. But it's very API-poor compared with Map, e.g. there's not even a fold -- should one toListBy first?

I find that surprising. Have you looked in Data.Trie.Convenience? The API of Data.Map is rather bloated so I've pushed most of it out of the main module in order to clean things up. There are only a small number of functions in the Data.Map interface I haven't had a chance to implement yet. For folding, the `foldMap`, `foldr`, and `foldl` functions are provided via the Data.Foldable interface. The Data.Traversable class is also implemented if you need to make changes to the trie along the way. These all give generic folding over the values stored in the trie. If you need access to the keys during folding you can use `foldrWithKey`, though it has to reconstruct the keys, which doesn't sound good for your use case. `toListBy` is a convenience wrapper around `foldrWithKey` which supports list fusion, so it has the same advantages and disadvantages compered to the Foldable/Traversable functions. If there's a particular function you still need, let me know and I can add an implementation for it. In terms of optimizing your code, one thing you'll surely want to do is to construct an intern table (Trie Int, IntMap ByteString) so that you only have to deal with Ints internally rather than ByteStrings. I haven't looked at your code yet to see how this would fit in, but it's almost always a requisite trick for handling large text corpora. -- Live well, ~wren