April 2011 - Haskell-Cafe

Hexpat: Lazy I/O problem with huge input files
by Aleksandar Dimitrov 10 Dec '11

10 Dec '11

Hello Haskell Cafe, I really hope this is the right list for this sort of question. I've bugged the folks in #haskell, they say go here, so I'm turning to you. I want to use Hexpat to read in some humongous XML files (linguistic corpora,) since it's the only Haskell XML library (I could find) that takes ByteStrings as input. I stumbled on a problem when using one of the examples from the docs of Text.XML.Expat.Tree. The "cookbook recipe" there suggests *first* processing the data, and only then looking into the parser error to see if there has been an error. I understand this should prevent the parse tree from being fully evaluated before use. Unfortunately, that is not what happens on my system (ghc 6.12.1, if that's of importance.) This is the code from the docs, that I modified to read files: > import Text.XML.Expat.Tree > import System.Environment (getArgs) > import Control.Monad (liftM) > import qualified Data.ByteString.Lazy as C >· > -- This is the recommended way to handle errors in lazy parses > main = do > f <- liftM head getArgs >>= C.readFile > let (tree, mError) = parse defaultParseOptions f > print (tree :: UNode String) >· > -- Note: We check the error _after_ we have finished our processing > -- on the tree. > case mError of > Just err -> putStrLn $ "It failed : "++show err > Nothing -> putStrLn "Success!" Given a 42M test file, an invocation like this: % ghc --make -O2 Hexpat.hs % ./Hexpat input.xml > dump.xml will gobble up some 2Gigs of RAM (at least. I usually kill it before it starts thrashing the swap space, since that almost crashes my entire machine.) If I remove the last 3 lines: > import Text.XML.Expat.Tree > import System.Environment (getArgs) > import Control.Monad (liftM) > import qualified Data.ByteString.Lazy as C > > main = do > f <- liftM head getArgs >>= C.readFile > let (tree, mError) = parse defaultParseOptions f > print (tree :: UNode String) the same invocation and input file barely uses a megabyte or two of RAM and finishes really quickly. Why is that? Is this a mistake in the Hexpat docs, or am I doing something wrong? Lazy IO has always been a little bit of a mystery to me, and just when I thought I had it... Thanks for any help on the matter! Aleks

3 3

Re: [Haskell-cafe] FFI for a beginner
by Jason Dagit 14 Jul '11

14 Jul '11

I don't know how to make ghc load them without using either hsc2hs or c2hs. I've had better experiences with hsc, but your mileage may vary. On Apr 8, 2011 3:34 AM, "Andrew Pennebaker" <andrew.pennebaker(a)gmail.com> wrote: ncurses is proving too difficult to setup, so I'm working on a new library called charm. The C code works by itself, but I can't compile a Haskell wrapper for it. While the tutorials at HaskellWiki<http://www.haskell.org/haskellwiki/FFI_complete_examples>are helpful, they're outdated. Argh! The docs<http://www.haskell.org/ghc/docs/latest/html/users_guide/ffi-ghc.html#glasgo…>say that -#include pragmas no longer work, but fail to explain how to load code without them. Suffice to say I have no recourse but trial and error. GitHub: charm <https://github.com/mcandre/charm> and hscharm<https://github.com/mcandre/hscharm> <https://github.com/mcandre/charm>$ make cp /usr/include/charm.c . ghc --make -fforce-recomp -o hellocharm hellocharm.hs charm.hs charm.c -I/usr/include -dylib-install-name /usr/lib/libcharm.dynlib [1 of 2] Compiling Charm ( charm.hs, charm.o ) [2 of 2] Compiling Main ( hellocharm.hs, hellocharm.o ) Linking hellocharm ... ld: duplicate symbol _Charm_getWidth_info in charm.o and charm.o collect2: ld returned 1 exit status make: *** [hellocharm] Error 1 Cheers, Andrew Pennebaker www.yellosoft.us _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe(a)haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

4 4

Weird multi-threading runtime behaviour of single-threaded program with GHC-7.0.3
by Herbert Valerio Riedel 24 Jun '11

24 Jun '11

Hello, I'm experiencing an unexpected behaviour with GHC-7.0.3/x86_64's multithreading runtime when running some more demanding single-thread computations. I've isolated the part of the larger application which exposes the problem down to the parsing of a ~8MiB JSON file which causes quite a bit of memory allocations (source at end of email). The program is compiled with ghc --make -O2 -threaded -eventlog -rtsopts parse_aeson.hs and I ran it with and w/o the -N12 option on a 12-core machine (the problem is visible on a 4-core machine as well): $ time ./parse_aeson test.json 0.000392s after readFile 0.001016s parse was a success... 6.050836s NFed parse-tree... 6.050968s NFed parse-tree... no exception real 0m6.100s user 0m5.680s sys 0m0.419s $ time ./parse_aeson test.json +RTS -N12 0.001249s after readFile 0.001777s parse was a success... 10.048607s NFed parse-tree... 10.048738s NFed parse-tree... no exception real 0m10.125s user 0m57.496s sys 0m39.927s When observing top(1)'s output, I saw that all HECs consumed a substantial amount of cpu cycle, so I loaded up threadscope to see what the remaining 11 HECs were doing all the time. I've put the resulting eventlog files and screenshots of threadscope's visualization online at http://www.narf.at/~hvr/Haskell/parse_aeson.N0.eventlog.png http://www.narf.at/~hvr/Haskell/parse_aeson.N0.eventlog.xz http://www.narf.at/~hvr/Haskell/parse_aeson.N12.eventlog.png http://www.narf.at/~hvr/Haskell/parse_aeson.N12.eventlog.xz What's happening there? The actual processing work seems to be done in a single HEC... but what are the remaining 11 HECs doing exactly? Am I doing something wrong? What does it mean when the GC-bars in threadscope becomes green as in the middle of the screenshot below? http://www.narf.at/~hvr/Haskell/parse_aeson.N4.eventlog.pdf ------------------------------------------------------------------------- -- also available at http://www.narf.at/~hvr/Haskell/parse_aeson.hs module Main where import Control.DeepSeq import Control.Exception import Data.Aeson import Data.Time import System.Environment import qualified Data.Attoparsec.Lazy as APL import qualified Data.ByteString.Lazy.Char8 as LB parseIt :: FilePath -> IO () parseIt fn = do t0 <- getCurrentTime let putStrLn' !s = do t0' <- getCurrentTime putStrLn $ show (t0' `diffUTCTime` t0) ++ " " ++ s json_text <- LB.readFile fn putStrLn' "after readFile" let Success p = decode json_text :: Result Value putStrLn' "parse was a success..." putStrLn' $ p `deepseq` "NFed parse-tree..." putStrLn' $ p `deepseq` "NFed parse-tree..." decode :: FromJSON a => LB.ByteString -> Result a decode = either Error fromJSON . APL.eitherResult . APL.parse json main :: IO () main = do fns <- getArgs case fns of [fn] -> do res <- try $ parseIt fn case res of Left e -> putStrLn $ "caught: " ++ show (e :: SomeException) Right () -> putStrLn "no exception" _ -> putStrLn "usage: parse_aeson <json-file>"

2 3

Parsing Haskell in Parsec
by Alex Rozenshteyn 16 Jun '11

16 Jun '11

I'm trying to write a parser for a small functional language in Parsec, as part of a larger project of porting this<http://www.cs.jhu.edu/%7Escott/pl/book/dist/>from ocaml to Haskell. I was wondering if there was a parser for Haskell written in Parsec that I could use as a reference. -- Alex R

4 5

fishing for ST mutable Vector examples
by brad clawsie 17 May '11

17 May '11

hi all i was wondering if anyone could post some minimal examples on using mutable Vectors in the ST monad. i've been digging around in the usual places but haven't been able to find anything to get me over the hump thanks in advance brad

8 10

Re: [Haskell-cafe] Asynchronous Arrows need Type Specialization - Help!
by Paul L 10 May '11

10 May '11

I now understand where you are coming from, but I don't quite get your motivation to develop new classes for arrows. Tupling is Haskell is of course very lazy, it does not evaluate any of its element. As for the spatial and logical concepts, don't you think they are over-specifying what is really necessary for your system model? So I guess what I'm having trouble with is your arr f example. If f is sufficiently lazy, it does not evaluate the elements in its input tuple at all. Most pure functions we use to wire up arrows are like this: arr fst, arr snd, arr swap, arr (\x -> (x, x)), etc. I also don't quite follow your example of passing a time stamp together with value. Do you intend to keep the time stamp always updated? If you are getting unwanted results, it is perhaps due to wrong computation, rather than the composition to a pure functions. Maybe you want to give a specific example of "f". Regards, Paul Liu On Fri, Apr 1, 2011 at 1:55 PM, David Barbour <dmbarbour(a)gmail.com> wrote: > On Fri, Apr 1, 2011 at 11:47 AM, Paul L <ninegua(a)gmail.com> wrote: >> On Sun, Mar 20, 2011 at 10:18 PM, David Barbour <dmbarbour(a)gmail.com> wrote: >> >>> The (***) and (&&&) operations, as specified in Control.Arrow, are >>> inherently synchronization points. >>> >>> Ideally one could do something like: >>> >>> (a1 *** a2) >>> first a3 >>> >>> and the output from a1 would be piped directly as input to a3, without >>> touching a2. However, arrows also allow: >>> >>> (a1 *** a2) >>> arr f >>> >>> And, in this case, the current state of a1 and a2 must be combined >>> into a stream of pairs so that f may be mapped over the stream. >>> Obtaining both streams at a single place (a single vat) and time is a >>> synchronizing operation. The synchronization operations are severely >>> undermining the intended scalability and performance of this agent >>> abstraction. >> >> If what you mean by "synchronization" is that for every output of a1 >> there must be an output of a2, then YES. >> >> But if you mean that both the outputs of a1 and a2 have to be >> available (fully evaluated) due to the tupling, then NO, at least not >> in Haskell where things are evaluated lazily. > > What I mean by synchronization is that the outputs from a1 and a2 must > occur at the same logical or physical space and time. > > Assume 'a1' and 'a2' represent external, asynchronous resources. > * 'a1' loads a file, with latency 40 milliseconds. (input is filename) > * 'a2' is a database query, with a latency of 100 milliseconds. (input is query) > > (a1 *** a2) both loads a file and performs a query, then tuples the > results as (r1,r2). > > (a1 *** a2 >>> first a3) further processes the file state r1, > generating (r3,r2). > > Logically, a pair is a spatial-temporal concept: if you have a pair, > you have access to a fst and a snd value at the same place and the > same time. This synchronization is obvious if you pass values into an > 'arr' arrow. The 'physical' synchronization issue is that the naive > way to implement (a1 *** a2) is to wait for both resources to be > available (time) in the same thread (space), then generate a tuple > that couples the responses. > > For my programming model, however, I must avoid both forms of > synchronization. The 'logical' synchronization is minimized. You might > model this, roughly, by pairing the responses with their latencies: > ((40,r1),(100,r2)). This would essentially be a 'futures' passing > style. The idea is that '(a1 *** a2) >>> first a3' might generate > something like ((80,r3),(100,r2)). Explicit synchronization might > bring this to ((100,r3),(100,r2)). > > Futures passing might work well enough for the problem as I've > explained it to you. My programming model is operating under several > additional constraints to support reactivity, concurrency, > parallelism, scalability, distribution, and open systems. Under the > more stringent requirements, I've been unable to make futures fit the > problem. However, in the time since my last Haskell-cafe post, I have > found a way to leverage GADTs to at least keep elements of a product > distinct under-the-hood and minimize physical synchronization. > > data A a d r where > Aprim :: a d r -> A a d r > Aid :: A a r r > Acomp :: A a d' r -> A a d d' -> A a d r > Aarr :: (d -> r) -> A a d r > Aprod :: A a d r -> A a d' r' -> A (a,a') (d,d') > Adup :: A a (a,a) > Afst :: A a (b,c) b > Asnd :: A a (b,c) c > ... > >> >> BTW, according to arrow laws, (a1 *** a2) >>> first a3 is equivalent >> to (a1 >>> a3) *** a2, does the latter look more appealing to you? If >> yes, I don't see where is the problem, unless you intentionally want >> to make them different, and then it is no longer arrow. > > While both expressions should have equivalent observable properties, > neither Haskell nor Arrow laws assure they will possess the same > operational characteristics. In a naive implementation, the latter > will generally be the more efficient option. > > For efficiency reasons, I have also abandoned Ross's Control.Arrow. > The main problem with it is Ross's use of pragmas: > > {-# RULES > "first/arr" forall f . first (arr f) = arr (first f) > "second/arr" forall f . second (arr f) = arr (second f) > "product/arr" forall f g . arr f *** arr g = arr (f *** g) > "fanout/arr" forall f g . arr f &&& arr g = arr (f &&& g) > #-} > > These rules go the wrong direction, making the program more opaque to > the runArrow operation, and actually change both the physical and > logical synchronization characteristics for the worse. I.e. if I pass > ((80,r3),(100,r2)) to 'arr f', the output becomes available at logical > time 100 rather than time 80. > > Peter Gammie suggested use of Adam Megacz's Generalized Arrows, which > would avoid this problem by use of an opaque product type that can > only be converted to a pair by an explicit operation (i.e. 'synch :: a > (b**c) (b,c)' for opaque product type (**)). I'm still debating > whether to take this approach. > -- Regards, Paul Liu

2 1

ANN: timeplot-0.3.0 - the analyst's swiss army knife for visualizing ad-hoc log files
by Eugene Kirpichov 06 May '11

06 May '11

Hello fellow haskellers, I announce the release of timeplot-0.3.0, the "analyst's swiss army knife for visualizing ad-hoc log files". Links: * http://jkff.info/presentation/two-visualization-tools - a presentation saying what the tools are all about and giving plenty of graphical examples on cluster computing use cases. At the end of the presentation there's also a couple of slides about installation. It is a little bit outdated, it corresponds to versions just before 0.3.0. * http://hackage.haskell.org/package/timeplot * http://github.com/jkff/timeplot * The sibling tool, splot - for visualizing the activity of many concurrent processes - http://hackage.haskell.org/package/splot and http://github.com/jkff/splot . It has also gotten a couple of new features since my last announcement. The major new feature of tplot is the introduction of subplots, the <<'within' plots>>. It allows one to plot data from several sub-tracks on one track of the graph: - several line- or dot-plots - several plots of sums or cumulative sums, perhaps stacked (to see how the sub-tracks contribute to the total sum - e.g. if your log speaks about different types of overhead and you wish to see how they contribute to the total) - stacked "activity count" plot - a generalization of the previous "activity count" plot, which allows you to, given a log saying like "Machine started servicing job JOB1 ... Machine finished servicing job JOB1" etc, plot how many machines are servicing each job at any moment, in a stacked fashion - so, how loads by different jobs contribute to the whole cluster's load. The "activity frequency" plot plots the same on a relative scale. The syntax is, for example: "within[.] dots" or "within[.] acount" or even "within[.] duration cumsum stacked" etc. Note that these are of course just example use cases and the tool is universal, it is not in any sense specialized to clusters, jobs, overheads or actually even to logs. I'd like to encourage you to give it a try and look around for a use case :) If you do give the tool a try, please tell me if something goes wrong, be it an installation problem or a bug (the version is fresh released, so this is quite possible). -- Eugene Kirpichov Principal Engineer, Mirantis Inc. http://www.mirantis.com/ Editor, http://fprog.ru/

2 2

How often is the planet updated?
by Magnus Therning 05 May '11

05 May '11

I see that Planet Haskell hasn't been updated since April 26. Is something wrong with it, or does it really not update more often than that? /M -- Magnus Therning OpenPGP: 0xAB4DFBA4 email: magnus(a)therning.org jabber: magnus(a)therning.org twitter: magthe http://therning.org/magnus

3 4

Can't install haskellnet with ghc7
by Charles-Pierre Astolfi 04 May '11

04 May '11

Hi -cafe, Did anybody managed to install haskellnet from hackage with ghc7? I tried on windows and mac and I get the following type error: [ 4 of 11] Compiling Network.HaskellNet.SMTP ( Network/HaskellNet/SMTP.hs, dist/build/Network/HaskellNet/SMTP.o ) Network/HaskellNet/SMTP.hs:269:25: Couldn't match expected type `Data.Text.Internal.Text' with actual type `[Char]' Expected type: Data.Text.Internal.Text Actual type: String In the first argument of `simpleMail', namely `to' In a stmt of a 'do' expression: myMail <- simpleMail to from subject plainBody htmlBody attachments Am I missing something? -- Cp

4 4

ANN: Leksah 0.10.0
by Hamish Mackenzie 03 May '11

03 May '11

Yesterday we uploaded our official 0.10.0 release (0.10.0.4) to Hackage and we have Windows and OS X installers at http://leksah.org/download.html Anyone using an earlier version should upgrade. This release is mainly small improvements and bug fixes. Some of the improvements include: * Better messages for some common problems. * Improved grep (now searches all source directories in your workspace) * Updated build system for more reliable rebuilding of dependent packages in your workspace We have automated the building of the Leksah installers and this should allow us to do more frequent releases in the future. There is a lot more to do and we would love to hear from anyone who has time to help out. Worth mentioning because they can be hard to find: * <Ctrl> R adds missing imports * <Ctrl> <Enter> evaluates current line or selection in GHCi * Right click menu in the editor has other debug functions * Comment and uncomment are now <Ctrl> D and <Ctrl> <Shift> D To build Leksah yourself: * Install Gtk and GtkSourceView2 (C libraries and headers) * Install The Haskell Platform (or GHC >= 6.12.3) * cabal update * cabal install gtk2hs-buildtools * cabal install leksah * leksah Making a Gtk application look nice on Windows and OS X is not easy so we recommend using one of these prebuilt binaries: Windows users using GHC 7.0.3 http://leksah.org/packages/leksah-0.10.0.4-ghc-7.0.3.exe Windows users using GHC 6.12.3 http://leksah.org/packages/leksah-0.10.0.4-ghc-6.12.3.exe OS X 10.6 users using GHC 7.0.3 http://leksah.org/packages/leksah-0.8.0.6.dmg (These installers do not install Gtk2Hs packages on your system) Please consult http://leksah.org/download.html for more details about installation! Background information can be found in the Leksah manual http://leksah.org/leksah_manual.pdf. Known bugs and problems: * Leksah does not run with GHC 6.12.2 (http://http://hackage.haskell.org/trac/ghc/ticket/4038) Have fun Jürgen & Hamish See you on #leksah IRC

11 19