On 20/06/16 06:42, martin wrote:
as the simulation progresses, I collect "primary" log entries, each one associated with a timestamp. These entries carry the information I am interested in (and not so much the final state). Currently I am using a simple list to collect log entries. Since (:) is so much easier than (++) I prepend new entries to the head of the list. So my log runs backwards in time with the oldest entries at the tail end.
Because the log-entries are typically too fine-grained to be useful, I want to aggregate several of them into a single log entry in a secondary log. I want to do this "as I go" and discard primary log entries which are already aggregated. Thus I can keep the primary log reasonably small.
I thought that aggregation is just a special case of parsing. And indeed, I could write an applicative parser
PrimaryLog -> (PrimaryLog, SecondaryLog)
which does the trick, except I had to reverse the ordering of the primary log. This is because the parser/aggregator must parse from past to future, because there may be new log entries coming up in the future. I need to be able to resume parsing where I left off whenever new primary log entries are entered, becuase I may then have just enough unconsumed primary log entries to create another secondary entry.
You are dealing with the classic "streaming problem" when you keep a growing "Log" that you can't reduce until the end of the process; instead of it being returned incrementally as when you `map a list`. Luckily this problem has two standards solution namely pipes/conduit. They let you recover the streaming behaviour you want and will make your parsing go left-to-right. -- -- Ruben Astudillo