Oren Ben-Kiki wrote:
The code is in http://hpaste.org/1314#a1 if anyone at all is willing to take pity on me and explain what I'm doing wrong.
There is an important point to note about streaming, namely that it conflicts with knowing whether the input is syntactically correct or not. In other words, evaluating (rResult reply) parses the hole input to the bitter end. For example, take your input data cycle "a\n" = "a\na\na\na\n..." Can we determine whether (rResult reply) is Nothing or Just after reading the first 'a'? No, we can't because the hole parse could fail far to the end resulting in Nothing. You've probably noticed this already and introduced rToken for that reason but I'm not sure.
Chasing down my memory leak I got into a weird situation where adding a magic manual SCC section and compiling with -prof makes the leak disappear.
That sounds fishy. Note that manual SCC annotations + optimization currently is a bit buggy in GHC, in the sense that optimization disturbs correct cost attribution. Does the program eat the output of yes 'a' without memory explosion showed by "top"? If so, it could perhaps be a problem of the optimization phase which maybe expands D.append parsed (case result of ...) to case result of { Nothing -> D.append parsed D.empty Just .. -> D.append parsed (D.singleton...) } The latter won't stream because of the reason noted above. But it's highly unlikely that GHC performs a transformation that changes semantics this radically.
I can achieve the results I want in very short elegant code...
In my opinion, your streaming parser is not as elegant as it could be which makes it hard to read the code. With monad transformers, we almost have Parser a ~= StateT State (WriterT (D.DList Token) Maybe) a ~= State -> (D.DList Token, Maybe (a,State)) Your Parser is slightly different but I think that the monad instances behave exactly the same way. The power functional programming is to assemble big things from small things. Monad transformers supply tons of small monads that you can reuse to build big monads. In a sense, the many pattern matches on Maybe have already been written down in general form by someone else. Also, I'm really missing type signatures, especially for many. When reading the code, I expected many :: Parser a -> Parser [a] but had to conclude many :: Parser a -> Parser () by inspecting its code. Regards, apfelmus