tplot and splot - analyst's swiss army knifes for visualizing log files - Haskell-Cafe

tplot and splot - analyst's swiss army knifes for visualizing log files

Eugene Kirpichov

16 Dec 2010 16 Dec '10

10:02 a.m.

Hi cafe, I've published a large presentation about two Haskell-based tools of mine - tplot and splot. Their motto is "visualize system behavior from logs with a shell one-liner". Based on my experience, they usually seem to live up to this motto. http://www.slideshare.net/jkff/two-visualization-tools [attention attractor: the presentation has *really a lot* of pictures] hackage: http://hackage.haskell.org/package/timeplot http://hackage.haskell.org/package/splot other docs: http://www.haskell.org/haskellwiki/Timeplot http://www.haskell.org/haskellwiki/Splot -- Eugene Kirpichov Senior Software Engineer, Grid Dynamics http://www.griddynamics.com/

Show replies by date

Henning Thielemann

17 Dec 17 Dec

4:02 a.m.

New subject: tplot and splot - analyst's swiss army knifes for visualizing log files

Eugene Kirpichov schrieb:

...

Hi cafe,

I've published a large presentation about two Haskell-based tools of mine - tplot and splot.

Their motto is "visualize system behavior from logs with a shell one-liner". Based on my experience, they usually seem to live up to this motto.

http://www.slideshare.net/jkff/two-visualization-tools

[attention attractor: the presentation has *really a lot* of pictures]

... and complete TeX code attached! :-) However can I also view a simple PDF document of the presentation?

Eugene Kirpichov

4:08 a.m.

New subject: tplot and splot - analyst's swiss army knifes for visualizing log files

Hello! Actually the presentation was created in PowerPoint, not in TeX :) You can download the PDF here - http://www.slideshare.net/jkff/two-visualization-tools/download (however one has to be logged in to Slideshare, for example with a facebook acct., for this link to work) Just in case, I'm also attaching a PDF of the current version to this email, but visiting the link is preferable, since I'll be updating the contents. 2010/12/17 Henning Thielemann :

...

Eugene Kirpichov schrieb:

...
Hi cafe,

I've published a large presentation about two Haskell-based tools of mine - tplot and splot.

Their motto is "visualize system behavior from logs with a shell one-liner". Based on my experience, they usually seem to live up to this motto.

http://www.slideshare.net/jkff/two-visualization-tools

[attention attractor: the presentation has *really a lot* of pictures]

... and complete TeX code attached! :-) However can I also view a simple PDF document of the presentation?

-- Eugene Kirpichov Senior Software Engineer, Grid Dynamics http://www.griddynamics.com/

Brandon S Allbery KF8NH

1 Jan 1 Jan

8:43 p.m.

New subject: tplot and splot - analyst's swiss army knifes for visualizing log files

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 still catching up on inbox... On 12/17/10 04:08 , Eugene Kirpichov wrote:

...

Just in case, I'm also attaching a PDF of the current version to this email, but visiting the link is preferable, since I'll be updating the contents.

So lessee, I need to get *yet another* account to be able to get updated versions of your document? I expect I won't be updating, then. (No, I'm not on Facebook, and won't be; they aren't going to fix their security and privacy issues because they're where their profit comes from. If I want to give away my vital information to the universe I'll just post my social security/US ID number somewhere, thanks.) - -- brandon s. allbery [linux,solaris,freebsd,perl] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk0f2C8ACgkQIn7hlCsL25WbZACgoRAafE2xKTx7z5JxuypZdwRP JXsAnAkjx67HqBFYdobhk0zl9P34Vb95 =bp7O -----END PGP SIGNATURE-----

Eugene Kirpichov

4 Jan 4 Jan

1:26 p.m.

New subject: tplot and splot - analyst's swiss army knifes for visualizing log files

Hi Brandon, I would happily use a better online presentation sharing tool if I knew of one - I am a novice in this respect. Which do you suggest? 02.01.2011, в 3:43, Brandon S Allbery KF8NH написал(а):

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

still catching up on inbox...

On 12/17/10 04:08 , Eugene Kirpichov wrote:

...
Just in case, I'm also attaching a PDF of the current version to this email, but visiting the link is preferable, since I'll be updating the contents.

So lessee, I need to get *yet another* account to be able to get updated versions of your document? I expect I won't be updating, then. (No, I'm not on Facebook, and won't be; they aren't going to fix their security and privacy issues because they're where their profit comes from. If I want to give away my vital information to the universe I'll just post my social security/US ID number somewhere, thanks.)

- -- brandon s. allbery [linux,solaris,freebsd,perl] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk0f2C8ACgkQIn7hlCsL25WbZACgoRAafE2xKTx7z5JxuypZdwRP JXsAnAkjx67HqBFYdobhk0zl9P34Vb95 =bp7O -----END PGP SIGNATURE-----

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Henning Thielemann

3:51 p.m.

New subject: tplot and splot - analyst's swiss army knifes for visualizing log files

Eugene Kirpichov schrieb:

...

I would happily use a better online presentation sharing tool if I knew of one - I am a novice in this respect. Which do you suggest?

For me a plain PDF file, say on code.haskell.org or projects.haskell.org is the best way.

Eugene Kirpichov

4:30 p.m.

New subject: tplot and splot - analyst's swiss army knifes for visualizing log files

Thanks Hennig, this sounds very good indeed! I haven't thought of this. 04.01.2011, в 22:51, Henning Thielemann написал(а):

...

Eugene Kirpichov schrieb:

...
I would happily use a better online presentation sharing tool if I knew of one - I am a novice in this respect. Which do you suggest?

For me a plain PDF file, say on code.haskell.org or projects.haskell.org is the best way.

Lars Viklund

3 Jan 3 Jan

6:08 a.m.

New subject: tplot and splot - analyst's swiss army knifes for visualizing log files

On Fri, Dec 17, 2010 at 12:08:53PM +0300, Eugene Kirpichov wrote:

...

Hello!

Actually the presentation was created in PowerPoint, not in TeX :) You can download the PDF here - http://www.slideshare.net/jkff/two-visualization-tools/download (however one has to be logged in to Slideshare, for example with a facebook acct., for this link to work)

Just in case, I'm also attaching a PDF of the current version to this email, but visiting the link is preferable, since I'll be updating the contents.

Does the list have any policies on not attaching large files? I'd rather not have a sizeable chunk of my mail store consumed by files I am completely uninterested in. As for storing files online, I'm sure that there's download hosting services that do not require any signing in, like say archiv.to or something. -- Lars Viklund | zao@acc.umu.se

Eugene Kirpichov

4 Jan 4 Jan

3:44 p.m.

New subject: tplot and splot - analyst's swiss army knifes for visualizing log files

Thanks for pointing out the issue, Lars - though I personally do not think that 2mb is a sizeable chunk by measures of most modern mail hostings, and though the offending letter can simply be deleted, I agree that for some purposes attaching megabyte-sized files to wide mailing list postings is inconvenient. I will consider archive.to in the future. 03.01.2011, в 13:08, Lars Viklund написал(а):

...

On Fri, Dec 17, 2010 at 12:08:53PM +0300, Eugene Kirpichov wrote:

...
Hello!

Actually the presentation was created in PowerPoint, not in TeX :) You can download the PDF here - http://www.slideshare.net/jkff/two-visualization-tools/download (however one has to be logged in to Slideshare, for example with a facebook acct., for this link to work)

Just in case, I'm also attaching a PDF of the current version to this email, but visiting the link is preferable, since I'll be updating the contents.

Does the list have any policies on not attaching large files? I'd rather not have a sizeable chunk of my mail store consumed by files I am completely uninterested in.

As for storing files online, I'm sure that there's download hosting services that do not require any signing in, like say archiv.to or something.

-- Lars Viklund | zao@acc.umu.se

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Lars Viklund

6:47 p.m.

New subject: tplot and splot - analyst's swiss army knifes for visualizing log files

On Tue, Jan 04, 2011 at 10:44:10PM +0200, Eugene Kirpichov wrote:

...

Thanks for pointing out the issue, Lars - though I personally do not think that 2mb is a sizeable chunk by measures of most modern mail hostings, and though the offending letter can simply be deleted, I agree that for some purposes attaching megabyte-sized files to wide mailing list postings is inconvenient. I will consider archive.to in the future.

In order to provide a sense of scale, the allowed size for my mail spool is in the vincinity of tens of megs, and the home directory I can use procmail to sort into mailboxes with is a few gigs, out of which 1.5 gigabytes is taken up by mailing list archives. Not everyone uses gmail, you know, particularly as their threading support is abysmal. -- Lars Viklund | zao@acc.umu.se

Ferenc Wagner

17 Mar 17 Mar

1:39 p.m.

New subject: tplot and splot - analyst's swiss army knifes for visualizing log files

Eugene Kirpichov writes:

...

2010/12/17 Henning Thielemann :

...
Eugene Kirpichov schrieb:

...
I've published a large presentation about two Haskell-based tools of mine - tplot and splot.

Their motto is "visualize system behavior from logs with a shell one-liner". Based on my experience, they usually seem to live up to this motto.

http://www.slideshare.net/jkff/two-visualization-tools

[attention attractor: the presentation has *really a lot* of pictures]

... and complete TeX code attached! :-) However can I also view a simple PDF document of the presentation?

You can download the PDF here - http://www.slideshare.net/jkff/two-visualization-tools/download (however one has to be logged in to Slideshare, for example with a facebook acct., for this link to work)

Just in case, I'm also attaching a PDF of the current version to this email, but visiting the link is preferable, since I'll be updating the contents.

Please, if at all possible, link an up-to-date downloadable PDF from the documentation (http://hackage.haskell.org/package/timeplot) or from the homepage (http://haskell.org/haskellwiki/Timeplot) to make our life easier! Anyway, your tools look very interesting, I gave tplot a shot. Unfortunately, I hit various strange failures: $ head -4 or.log Mar 8 18:55:11 =overrun 1 Mar 8 18:55:13 =overrun 6 Mar 8 18:55:15 =overrun 13 Mar 8 18:55:16 =overrun 3 $ wc -l or.log 466 or.log $ ls -l or.log overruns466.log lrwxrwxrwx 1 wferi wferi 15 Mar 17 14:45 or.log -> overruns466.log -rw-rw-r-- 1 wferi wferi 12587 Mar 17 14:35 overruns466.log $ tplot -if or.log -tf 'date %b %e %T' -o overruns.png -k 'overrun' 'sum 10' This worked just fine. However, when given the same file with a longer name, tplot does not terminate: $ tplot -if overruns466.log -tf 'date %b %e %T' -o overruns.png -k 'overrun' 'sum 10' ^C while doing the same the other way around still works: $ cat overruns466.log | tplot -if - -tf 'date %b %e %T' -o overruns.png -k 'overrun' 'sum 10' Choosing any other extension (svg, pdf or ps) also results in nontermination (or at least unbearable runtime and memory consumption). Adding a simple no-op statement, like: diff -ur ../timeplot-0.2.19/Tools/TimePlot.hs ./Tools/TimePlot.hs --- ../timeplot-0.2.19/Tools/TimePlot.hs 2011-03-09 11:36:24.000000000 +0100 +++ ./Tools/TimePlot.hs 2011-03-17 16:42:57.247625607 +0100 @@ -627,6 +627,7 @@ when (null args || args == ["--help"]) $ showHelp >> exitSuccess case (readConf args) of Conf conf -> do + putStr "" let render = case (outFormat conf) of { PNG -> \c w h f -> const () `fmap` renderableToPNGFile c w h f; PDF -> renderableToPDFFile ; also results in nontermination, even in the previously working case. Something is clearly wrong here, seemingly in the runtime IO system. -- Thanks, Feri. GHC 6.12.1 Chart-0.14 bytestring-0.9.1.5 bytestring-lexing-0.2.1 cairo-0.11.0 colour-2.3.1 containers-0.3.0.0 data-accessor-0.2.1.3 data-accessor-template-0.2.1.7 haskell98-1.0.1.1 regex-tdfa-1.1.4 strptime-1.0.1 time-1.1.4

Eugene Kirpichov

18 Mar 18 Mar

5:06 a.m.

New subject: tplot and splot - analyst's swiss army knifes for visualizing log files

Hello Ferenc, Thank you for reporting the bug - it's very curious, I will look into it in the nearest couple of days. Could you please send me privately the actual input file on which the program crashes? I have added a downloadable link to the presentation to all places where the slideshare link was present (finally making some use of my hosting...). Here it is: http://jkff.info/presentations/two-visualization-tools.pdf 2011/3/17 Ferenc Wagner :

...

Eugene Kirpichov writes:

...
2010/12/17 Henning Thielemann :

...
Eugene Kirpichov schrieb:

...
I've published a large presentation about two Haskell-based tools of mine - tplot and splot.

Their motto is "visualize system behavior from logs with a shell one-liner". Based on my experience, they usually seem to live up to this motto.

http://www.slideshare.net/jkff/two-visualization-tools

[attention attractor: the presentation has *really a lot* of pictures]

... and complete TeX code attached! :-) However can I also view a simple PDF document of the presentation?

You can download the PDF here - http://www.slideshare.net/jkff/two-visualization-tools/download (however one has to be logged in to Slideshare, for example with a facebook acct., for this link to work)

Just in case, I'm also attaching a PDF of the current version to this email, but visiting the link is preferable, since I'll be updating the contents.

Please, if at all possible, link an up-to-date downloadable PDF from the documentation (http://hackage.haskell.org/package/timeplot) or from the homepage (http://haskell.org/haskellwiki/Timeplot) to make our life easier!

Anyway, your tools look very interesting, I gave tplot a shot. Unfortunately, I hit various strange failures:

$ head -4 or.log Mar 8 18:55:11 =overrun 1 Mar 8 18:55:13 =overrun 6 Mar 8 18:55:15 =overrun 13 Mar 8 18:55:16 =overrun 3

$ wc -l or.log 466 or.log

$ ls -l or.log overruns466.log lrwxrwxrwx 1 wferi wferi 15 Mar 17 14:45 or.log -> overruns466.log -rw-rw-r-- 1 wferi wferi 12587 Mar 17 14:35 overruns466.log

$ tplot -if or.log -tf 'date %b %e %T' -o overruns.png -k 'overrun' 'sum 10'

This worked just fine. However, when given the same file with a longer name, tplot does not terminate:

$ tplot -if overruns466.log -tf 'date %b %e %T' -o overruns.png -k 'overrun' 'sum 10' ^C

while doing the same the other way around still works:

$ cat overruns466.log | tplot -if - -tf 'date %b %e %T' -o overruns.png -k 'overrun' 'sum 10'

Choosing any other extension (svg, pdf or ps) also results in nontermination (or at least unbearable runtime and memory consumption).

Adding a simple no-op statement, like:

diff -ur ../timeplot-0.2.19/Tools/TimePlot.hs ./Tools/TimePlot.hs --- ../timeplot-0.2.19/Tools/TimePlot.hs 2011-03-09 11:36:24.000000000 +0100 +++ ./Tools/TimePlot.hs 2011-03-17 16:42:57.247625607 +0100 @@ -627,6 +627,7 @@ when (null args || args == ["--help"]) $ showHelp >> exitSuccess case (readConf args) of Conf conf -> do + putStr "" let render = case (outFormat conf) of { PNG -> \c w h f -> const () `fmap` renderableToPNGFile c w h f; PDF -> renderableToPDFFile ;

also results in nontermination, even in the previously working case. Something is clearly wrong here, seemingly in the runtime IO system. -- Thanks, Feri.

GHC 6.12.1 Chart-0.14 bytestring-0.9.1.5 bytestring-lexing-0.2.1 cairo-0.11.0 colour-2.3.1 containers-0.3.0.0 data-accessor-0.2.1.3 data-accessor-template-0.2.1.7 haskell98-1.0.1.1 regex-tdfa-1.1.4 strptime-1.0.1 time-1.1.4

-- Eugene Kirpichov Principal Engineer, Mirantis Inc. http://www.mirantis.com/

Ferenc Wagner

7:37 a.m.

New subject: tplot and splot - analyst's swiss army knifes for visualizing log files

Eugene Kirpichov writes:

...

2011/3/17 Ferenc Wagner :

...
Eugene Kirpichov writes:

...
2010/12/17 Henning Thielemann :

...
Eugene Kirpichov schrieb:

...
I've published a large presentation about two Haskell-based tools of mine - tplot and splot.

Their motto is "visualize system behavior from logs with a shell one-liner". Based on my experience, they usually seem to live up to this motto.

http://www.slideshare.net/jkff/two-visualization-tools

[attention attractor: the presentation has *really a lot* of pictures]

... and complete TeX code attached! :-) However can I also view a simple PDF document of the presentation?

You can download the PDF here - http://www.slideshare.net/jkff/two-visualization-tools/download (however one has to be logged in to Slideshare, for example with a facebook acct., for this link to work)

Just in case, I'm also attaching a PDF of the current version to this email, but visiting the link is preferable, since I'll be updating the contents.

Please, if at all possible, link an up-to-date downloadable PDF from the documentation (http://hackage.haskell.org/package/timeplot) or from the homepage (http://haskell.org/haskellwiki/Timeplot) to make our life easier!

Anyway, your tools look very interesting, I gave tplot a shot. Unfortunately, I hit various strange failures:

$ head -4 or.log Mar 8 18:55:11 =overrun 1 Mar 8 18:55:13 =overrun 6 Mar 8 18:55:15 =overrun 13 Mar 8 18:55:16 =overrun 3

$ wc -l or.log 466 or.log

$ ls -l or.log overruns466.log lrwxrwxrwx 1 wferi wferi 15 Mar 17 14:45 or.log -> overruns466.log -rw-rw-r-- 1 wferi wferi 12587 Mar 17 14:35 overruns466.log

$ tplot -if or.log -tf 'date %b %e %T' -o overruns.png -k 'overrun' 'sum 10'

This worked just fine. However, when given the same file with a longer name, tplot does not terminate:

$ tplot -if overruns466.log -tf 'date %b %e %T' -o overruns.png -k 'overrun' 'sum 10' ^C

while doing the same the other way around still works:

$ cat overruns466.log | tplot -if - -tf 'date %b %e %T' -o overruns.png -k 'overrun' 'sum 10'

Choosing any other extension (svg, pdf or ps) also results in nontermination (or at least unbearable runtime and memory consumption).

Adding a simple no-op statement, like:

diff -ur ../timeplot-0.2.19/Tools/TimePlot.hs ./Tools/TimePlot.hs --- ../timeplot-0.2.19/Tools/TimePlot.hs 2011-03-09 11:36:24.000000000 +0100 +++ ./Tools/TimePlot.hs 2011-03-17 16:42:57.247625607 +0100 @@ -627,6 +627,7 @@ when (null args || args == ["--help"]) $ showHelp >> exitSuccess case (readConf args) of Conf conf -> do + putStr "" let render = case (outFormat conf) of { PNG -> \c w h f -> const () `fmap` renderableToPNGFile c w h f; PDF -> renderableToPDFFile ;

also results in nontermination, even in the previously working case. Something is clearly wrong here, seemingly in the runtime IO system. [...]

Thank you for reporting the bug - it's very curious, I will look into it in the nearest couple of days.

Great, thanks!

...

Could you please send me privately the actual input file on which the program crashes?

Well, it compresses to 1511 bytes, so I risk attaching it here for simplicity.

...

I have added a downloadable link to the presentation to all places where the slideshare link was present (finally making some use of my hosting...). Here it is: http://jkff.info/presentations/two-visualization-tools.pdf

Wonderful, thanks a lot! -- Regards, Feri.

5178

Age (days ago)

5270

Last active (days ago)

List overview

Download

12 comments

5 participants

participants (5)

Brandon S Allbery KF8NH
Eugene Kirpichov
Ferenc Wagner
Henning Thielemann
Lars Viklund