Hi Cafe,<div><br></div><div>In the last couple of days I completed my quest of making my graphing utility timeplot ( <a href="http://jkff.info/software/timeplotters">http://jkff.info/software/timeplotters</a> ) not load the whole input dataset into memory and consequently be able to deal with datasets of any size, provided however that the amount of data to *draw* is not so large. On the go it also got a huge speedup - previously visualizing a cluster activity dataset with a million events took around 15 minutes and a gig of memory, now it takes 20 seconds and 6 Mb max residence.</div>
<div>(I haven&#39;t yet uploaded to hackage as I have to give it a bit more testing)</div><div><br></div><div>The refactoring involved a number of interesting programming patterns that I&#39;d like to share with you and ask for feedback - perhaps something can be simplified.</div>
<div><br></div><div>The source is at <a href="http://github.com/jkff/timeplot">http://github.com/jkff/timeplot</a></div><div><br></div><div>The datatype of incremental computations is at <a href="https://github.com/jkff/timeplot/blob/master/Tools/TimePlot/Incremental.hs">https://github.com/jkff/timeplot/blob/master/Tools/TimePlot/Incremental.hs</a> .</div>
<div>Strictness is extremely important here - the last memory leak I eliminated was lack of bang patterns in teeSummary.</div><div>There&#39;s an interesting function statefulSummary that shows how closures allow you to achieve encapsulation over an unknown piece of state - curiously enough, you can&#39;t define StreamSummary a r as StreamSummary { init :: s, insert :: a-&gt;s-&gt;s, finalize :: s-&gt;r } (existentially qualified over s), as then you can&#39;t define summaryByKey - you don&#39;t know what type to store in the states map.</div>
<div><br></div><div>It is used to incrementally build all plots simultaneously, shown by the main loop in makeChart at <a href="https://github.com/jkff/timeplot/blob/master/Tools/TimePlot.hs">https://github.com/jkff/timeplot/blob/master/Tools/TimePlot.hs</a></div>
<div><br></div><div>Incremental building of plots of different types is at <a href="https://github.com/jkff/timeplot/blob/master/Tools/TimePlot/Plots.hs">https://github.com/jkff/timeplot/blob/master/Tools/TimePlot/Plots.hs</a></div>
<div>There are also a few interesting functions in that file - e.g. edges2eventsSummary, which applies a summary over a stream of &quot;long&quot; events to a stream of rise/fall edges.</div><div>This means that you can define a &quot;stream transformer&quot; (Stream a -&gt; Stream b) as a function (StreamSummary b -&gt; StreamSummary a), which can be much easier. I have to think more about this idea.</div>
<div><div><br></div>-- <br>Eugene Kirpichov<br>Principal Engineer, Mirantis Inc. <a href="http://www.mirantis.com/" target="_blank">http://www.mirantis.com/</a><br>Editor, <a href="http://fprog.ru/" target="_blank">http://fprog.ru/</a><br>

</div>