May 2011 - Haskell-Cafe

*GROUP HUG*
by Gregory Crosswhite 04 Jun '11

04 Jun '11

Hey everyone, Okay, this will sound silly, but I ventured into the Scala mailing list recently and asked an ignorant question on it, and I was shocked when people reacted not by enlightening me but by jumping on me and reacting with hostility. I bring this up not to badmouth the Scala community (they are apparently going through growing pains and will hopefully mature with time!) but just because it made me appreciate just how awesome you guys are, so I just feel the need to publicly express my admiration and thank to everyone on this list for having fostered such an incredibly professional, fanatically nonhostile, and generally pleasant place to talk about Haskell!!! *GROUP HUG* Okay, I'm done now. :-) Cheers, Greg

30 53

ANN: unordered-container 0.1.3.0
by Johan Tibell 04 Jun '11

04 Jun '11

Hi all, I've just uploaded a new version of the unordered-containers package, a package of fast hashing-based container types. Version 0.1.3.0 [1] adds: * the ability to take the union of two maps, * lower memory overhead per key/value pair (contributed by Jan-Willem Maessen), and * the beginning of a `Data.HashSet` module (contributed by Bryan O'Sullivan). If you want to contribute, you can get the source from the git repository: git clone https://github.com/tibbe/unordered-containers.git Alternatively, just fork the project on GitHub [2]. 1. http://http://hackage.haskell.org/package/unordered-containers-0.1.3.0 2. https://github.com/tibbe/unordered-containers

3 3

How on Earth Do You Reason about Space?
by Aleksandar Dimitrov 03 Jun '11

03 Jun '11

Dear Cafe, (Excuse the probably very ranty email; I am, unfortunately, at the end of my wits, and I hope that as fellow programmers, you will understand that this is among the most dreadful situations for our kind to be in.) Say, we have an input file that contains a word per line. I want to find all unigrams (unique words) in that file, and associate with them the amount of times they occurred in the file. This would allow me, for example, to make a list of word frequencies in a given text. Simple enough task. Here's an implementation using iteratees (lazy IO is evil) and unordered-containers' Data.HashMap.Strict, which enforces WHNF in values and keys: > import qualified Data.ByteString.Char8 as S > import qualified Data.Iteratee as I > import Data.Iteratee.IO > > import qualified Data.HashMap.Strict as T > > import Data.Iteratee.Char > import Data.List (foldl') > import Data.Char (toLower) > > import Data.Ord (comparing) > import Data.List (sortBy) > import System.Environment (getArgs) > > type Wordcounts = T.HashMap S.ByteString Int > > f' :: Monad m => I.Iteratee S.ByteString m Wordcounts > f' = I.joinI $ enumLinesBS (I.liftI $ step T.empty) > where step t (I.Chunk str) = I.liftI (step $! foldl' maybeIncrement t str) > step t stream = I.idone t stream > maybeIncrement t s > | s == S.empty = t > | otherwise = {-# SCC "m-I-other" #-} T.insertWith (+) s 1 t > > main :: IO () > main = getArgs >>= fileDriverVBuf 65536 f'.head >>= print.prettyList > where prettyList = -- sortBy (comparing snd) . T.toList > T.keys Some lines are empty, and I don't want them to be recorded, so that's why maybeIncrement is necessary. hpaste of this code: http://hpaste.org/47300/spaceleak (ignore convert, that's yet another issue.) Now, here's some observations: on a 75M input file (minuscule, compared to what I actually need) this program will eat 30M of heap space (says profiling) and return in 14 secs. I have two problems with that: a) that's too much heap space, b) the actual memory residency is *much* worse. ad b) average memory residency is at 38MB (this is OK, given heap consumption) but max residency is at 130MB, which is unacceptable to me (remember that I need to run this on files *much* bigger than just 75M.) <<ghc: 9013546200 bytes, 16876 GCs, 38232524/130572448 avg/max bytes residency (58 samples), 306M in use, 0.00 INIT (0.00 elapsed), 6.12 MUT (6.28 elapsed), 7.45 GC (7.50 elapsed) :ghc>> In fact, htop reports total memory residency of the program at around 320 MB at the end of its life-cycle (compiled and ran without profiling options.) I tried running this program on a 1.4GB file, and had to kill it when at 3.5GB memory consumption, my computer started paging. The actual hash-map, however, shouldn't be much bigger than in the 75MB case (I expect about twice the size,) since the input is natural language. I redirected the output of the program (showing a list of assoc pairs that were in the hash map) to a file, and that file measured 11MB in the case of a 75MB corpus, and 40MB when I ran the program on a 1.4GB corpus (I had to use a machine with 24GB of RAM to be able to run this.) ad a) heap consumption is too high for two reasons: firstly, the actual data I care about is much less than there's data on the heap. Secondly, about half the heap space is in LAG state. Here are profiles that will illustrate this: http://imgur.com/wBWmJ&XN1mW&ltYNR. - The first image shows 50% of the heap space being gobbled up with data that shouldn't be there anymore (LAG) - The second image shows the types that are in LAG state: ByteString and HashMap. So, it seems I'm keeping around hash maps? - The third image blames the cost centre m-i-other for the LAG. - Then there's the retainer profile: I have no idea how to read it. I thought it had something to do with iteratee reading in chunks of data, but after setting the chunk size to 4096 bytes and actually getting *less* spikes than before, I doubt that really is the case. It seems like half of the data this program puts on the heap is not cleaned up properly, and the otherwise-branch of maybeIncrement (i.e. the only really important line is this program) is to blame. I'm thinking there's too little strictness somewhere in the T.insertWith clause, but I have no idea how to make it better. Honestly, sprinkling around magic seq-dust every time something goes wrong doesn't feel good. I cannot reason about Haskell's space-performance even a little bit, and I have no idea how to change that. Some more remarks: - I tried using lazy IO first, but that was a total disaster. Iteratee is nice, and works very well, but maybe I don't know how to use it? - In terms of map-implementation, I observed that unordered-containers < Data.Map < bytestring-trie. It surprised me to see that Data.Map would perform better than bytestring-trie for this task (usually, Data.Map would be around twice as fast.) - I should probably look into using a mutable data structure as a map. I would find it a bit disenchanting to have to use a mutable data structure this way, but maybe that's just reality for you. RANT I have tried and tried again to avoid writing programs in Haskell that would leak space like BP likes to leak oil. However, I have yet to produce a single instance of a program that would do anything at all and at the same time consume less memory than there is actual data in the input file. It is very disconcerting to me that I seem to be unable, even after quite some practice, to identify space leaks in trivial programs like the above. I know of no good resource to educate myself in that regard. I have read the GHC manual, RWH's chapter on profiling, also "Inside T5"'s recent series on the Haskell heap, but no dice. Even if I can clearly see the exact line where at least some of the leaking happens (as I can in this case,) it seems impossible for me to prevent it. *thank you very much* for reading this far. This is probably a mostly useless email anyhow, I just had to get it off my chest. Maybe, just maybe, someone among you will have a crucial insight that will save Haskell for me :-) But currently, I see no justification to not start my next project in Lua, Python or Java. Sure, Haskell's code is pretty, and it's fun, but if I can't actually *run* it, why bother? (Yes, this isn't the first time I've ran into this problem …) Thanks, and kindest regards to you all, Aleks

9 19

Re: [Haskell-cafe] Status of Haskell + Mac + GUIs & graphics
by John Lato 02 Jun '11

02 Jun '11

> > Message: 17 > Date: Fri, 20 May 2011 15:59:51 -0700 > From: Evan Laforge <qdunkan(a)gmail.com> > Subject: Re: [Haskell-cafe] Status of Haskell + Mac + GUIs & graphics > To: Erik Hesselink <hesselink(a)gmail.com> > Cc: haskell-cafe(a)haskell.org > Message-ID: <BANLkTi=74mM6ORtU2e192jtoOT9g49fzTQ(a)mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1 > > > Note that it is supposed to be possible to build gtk2hs with gtk+osx, > > which will not use X11 but use the native OS X GUI. I've not been able > > to get this to work, but it's been a while since I tried. The Haskell > > wiki mentions it doesn't support Glade, but does support Cairo. If > > this were to work, gtk2hs would be a serious option as well. > > I've tried this 3 or 4 times, and failed every time. It's crazy > complicated. > I used to use this combination regularly. IME the difficulties are primarily with getting a working gtk+osx build. Once that was accomplished, gtk2hs was a straightforward install, provided you build from the src repo with the -quartz flag. Recently I've switched to using macports gtk2 with the quartz, no-x11 variant, which also uses native cocoa. This is much more reliable than gtk+osx. I don't know if it supports Glade though. My biggest problem with wx is that there's no support for building 64-bit wx applications on OS X. If that were fixed I might prefer it. John Lato

3 6

Re: [Haskell-cafe] Names for pretty-printing combinators
by Ivan Lazar Miljenovic 02 Jun '11

02 Jun '11

(Changing the subject as it's going off-topic from the original email :p) On 25 May 2011 22:45, Simon Meier <iridcode(a)gmail.com> wrote: > 2011/5/25 Ivan Lazar Miljenovic <ivan.miljenovic(a)gmail.com>: >> >> Also, by clashes with Applicative, are you referring to empty and <$> >> ? I'm not sure if a better name than "empty" can be found; as for >> <$>, maybe using pretty's notation of $$ and $+$ rather than <$> and >> <$$> ? > > What about 'emptyDoc'? It's a bit more of a mouthful, and getting a bit close to Hungarian Notation, isn't it? :p Also, empty is already used a fair amount in other libraries as it stands (admittedly, most of those are in modules that are typically imported qualified...). > Moreover, if you are changing the names of > combinators, then moving them away from Applicative and Arrow would be > a good idea; i.e., don't use <+>, as it already used by ArrowPlus. *sigh* We really need to get a list of used operators somewhere so that libraries can stake a claim :p > Moreover, if you can make a Monoid instance such that `mappend` equals > <>, you would also make the library compatible to a future > introduction of (<>) = mappend. I was thinking about that actually... -- Ivan Lazar Miljenovic Ivan.Miljenovic(a)gmail.com IvanMiljenovic.wordpress.com

4 3

Haskell School of Expression (graphics)
by michael rice 01 Jun '11

01 Jun '11

I installed the Haskell Platform (7.0.2) earlier, and I just downloaded package soegtk with cabal, apparently unsuccessfully. Any suggestions? Michael [michael@sabal ~]$ cabal updateConfig file path source is default config file.Config file /home/michael/.cabal/config not found.Writing default configuration to /home/michael/.cabal/configDownloading the latest package list from hackage.haskell.org[michael@sabal ~]$ cabal list SOE* soegtk Synopsis: GUI functions as used in the book "The Haskell School of Expression". Default available version: 0.11.1 Installed versions: [ Not installed ] Homepage: http://www.haskell.org/gtk2hs/ License: LGPL-2.1 [michael@sabal ~]$ cabal install soegtkResolving dependencies...Downloading cairo-0.11.1... /tmp/cairo-0.11.12906/cairo-0.11.1/Gtk2HsSetup.hs:25:2: warning: #warning Setup.hs is guessing the version of Cabal. If compilation of Setup.hs fails use -DCABAL_VERSION_MINOR=x for Cabal version 1.x.0 when building (prefixed by --ghc-option= when using the 'cabal' command) [-Wcpp][1 of 2] Compiling Gtk2HsSetup ( /tmp/cairo-0.11.12906/cairo-0.11.1/Gtk2HsSetup.hs, /tmp/cairo-0.11.12906/cairo-0.11.1/dist/setup/Gtk2HsSetup.o ) /tmp/cairo-0.11.12906/cairo-0.11.1/Gtk2HsSetup.hs:190:70: Couldn't match expected type `[PackageDB]' with actual type `PackageDB' Expected type: PackageDBStack Actual type: PackageDB In the sixth argument of `registerPackage', namely `packageDb' In the expression: registerPackage verbosity installedPkgInfo pkg lbi inplace packageDbDownloading glib-0.11.2... /tmp/glib-0.11.22906/glib-0.11.2/Gtk2HsSetup.hs:25:2: warning: #warning Setup.hs is guessing the version of Cabal. If compilation of Setup.hs fails use -DCABAL_VERSION_MINOR=x for Cabal version 1.x.0 when building (prefixed by --ghc-option= when using the 'cabal' command) [-Wcpp][1 of 2] Compiling Gtk2HsSetup ( /tmp/glib-0.11.22906/glib-0.11.2/Gtk2HsSetup.hs, /tmp/glib-0.11.22906/glib-0.11.2/dist/setup/Gtk2HsSetup.o ) /tmp/glib-0.11.22906/glib-0.11.2/Gtk2HsSetup.hs:190:70: Couldn't match expected type `[PackageDB]' with actual type `PackageDB' Expected type: PackageDBStack Actual type: PackageDB In the sixth argument of `registerPackage', namely `packageDb' In the expression: registerPackage verbosity installedPkgInfo pkg lbi inplace packageDbcabal: Error: some packages failed to install:cairo-0.11.1 failed during the configure step. The exception was:ExitFailure 1gio-0.11.1 depends on glib-0.11.2 which failed to install.glib-0.11.2 failed during the configure step. The exception was:ExitFailure 1gtk-0.11.2 depends on glib-0.11.2 which failed to install.pango-0.11.2 depends on glib-0.11.2 which failed to install.soegtk-0.11.1 depends on glib-0.11.2 which failed to install.[michael@sabal ~]$

6 9

Re: [Haskell-cafe] How on Earth Do You Reason about Space?
by John Lato 01 Jun '11

01 Jun '11

From: "Edward Z. Yang" <ezyang(a)MIT.EDU> > > Hello Aleksandar, > > It is possible that the iteratees library is space leaking; I recall some > recent discussion to this effect. Your example seems simple enough that > you might recompile with a version of iteratees that has -auto-all enabled. > Unfortunately, it's not really a safe bet to assume your libraries are > leak free, and if you've pinpointed it down to a single line, and there > doesn't seem a way to squash the leak, I'd bet it's the library's fault. > > Edward > I can't reproduce the space leak here. I tried Aleksander's original code, my iteratee version, the Ngrams version posted by Johan Tibell, and a lazy bytestring version. my iteratee version (only f' has changed from Aleksander's code): f' :: Monad m => I.Iteratee S.ByteString m Wordcounts f' = I.joinI $ (enumLinesBS I.><> I.filter (not . S.null)) $ I.foldl' (\t s -> T.insertWith (+) s 1 t) T.empty my lazy bytestring version > import Data.Iteratee.Char > import Data.List (foldl')import Data.Char (toLower) > > import Data.Ord (comparing) > import Data.List (sortBy) > import System.Environment (getArgs) > import qualified Data.ByteString.Lazy.Char8 as L > import qualified Data.HashMap.Strict as T > > f'2 = foldl' (\t s -> T.insertWith (+) s 1 t) T.empty . filter (not . L.null) . L.lines > > main2 :: IO () > main2 = getArgs >>= L.readFile .head >>= print . T.keys . f'2 None of these leak space for me (all compiled with ghc-7.0.3 -O2). Performance was pretty comparable for every version, although Aleksander's original did seem to have a very small edge. As someone already pointed out, keep in mind that this will use a lot of memory anyway, unless there's a lot of repetition of words. I'd be happy to help track down a space leak in iteratee, but for now I'm not seeing one. Best, John Lato

5 6

maybe a GHC bug but not sure
by Edward Amsden 01 Jun '11

01 Jun '11

When building the Haskell Objective C bindings tool from <http://code.google.com/p/hoc> Using GHC 7.0.3 for Haskell Platform 2011.2.0.1, OS X 64 bit HOC compiles HOC_cbits.o before trying to build it into the rest of the program (I don't say link because this came up while trying to compile: > [18 of 31] Compiling HOC.StdArgumentTypes ( HOC/HOC/StdArgumentTypes.hs, dist/build/HOC/StdArgumentTypes.o ) > [loading packages, like you'd see from ghci...] > Loading object (static) dist/build/HOC_cbits.o ... ghc: dist/build/HOC_cbits.o: Bad magic. Expected: feedfacf, got: feedface. GHC says "the impossible happened" and tells me to report it as a GHC bug, but based on the odd behavior I think the HOC build script might be doing something odd. I just don't have any idea where to start looking. For instance, why the heck is it looking at the magic bits for an object file while compiling a source file? Any insight appreciated. -- Edward Amsden Student Computer Science Rochester Institute of Technology www.edwardamsden.com

1 1

Enabling GADTs breaks Rank2Types code compilation - Why?
by dm-list-haskell-cafe＠scs.stanford.edu 01 Jun '11

01 Jun '11

I'm using GHC 7.0.2 and running into a compiler error that I cannot understand. Can anyone shed light on the issue for me? The code does not make use of GADTs and compiles just fine without them. But when I add a {-# LANGUAGE GADTs #-} pragma, it fails to compile. Here is the code: ==== {-# LANGUAGE Rank2Types #-} {-# LANGUAGE GADTs #-} mkUnit :: (forall t. t -> m t) -> m () mkUnit f = f () data Const b a = Const { unConst :: b } sillyId :: r -> r sillyId r = unConst (mkUnit mkConst_r) -- This line can't compile with GADTs where mkConst_r t = mkConst r t mkConst :: b -> t -> Const b t mkConst b _ = Const b ==== And the error I get is: Couldn't match type `t0' with `t' because type variable `t' would escape its scope This (rigid, skolem) type variable is bound by a type expected by the context: t -> Const r t The following variables have types that mention t0 mkConst_r :: t0 -> Const r t0 (bound at /u/dm/hack/hs/gadt.hs:11:11) In the first argument of `mkUnit', namely `mkConst_r' In the first argument of `unConst', namely `(mkUnit mkConst_r)' In the expression: unConst (mkUnit mkConst_r) I've found several workarounds for the issue, but I'd like to understand what the error message means and why it is caused by GADTs. Thanks in advance for any help. David

2 3

Lazy Evaluation in Monads
by Scott Lawrence 31 May '11

31 May '11

I was under the impression that operations performed in monads (in this case, the IO monad) were lazy. (Certainly, every time I make the opposite assumption, my code fails :P .) Which doesn't explain why the following code fails to terminate: iRecurse :: (Num a) => IO a iRecurse = do recurse <- iRecurse return 1 main = (putStrLn . show) =<< iRecurse Any pointers to a good explanation of when the IO monad is lazy? === The long story === I wrote a function unfold with type signature (([a] -> a) -> [a]), for generating a list in which each element can be calculated from all of the previous elements. unfold :: ([a] -> a) -> [a] unfold f = unfold1 f [] unfold1 :: ([a] -> a) -> [a] -> [a] unfold1 f l = f l : unfold1 f (f l : l) Now I'm attempting to do the same thing, except where f returns a monad. (My use case is when f randomly selects the next element, i.e. text generation from markov chains.) So I want unfoldM1 :: (Monad m) => ([a] -> m a) -> [a] -> m [a] My instinct, then, would be to do something like: unfoldM1 f l = do next <- f l rest <- unfoldM1 f (next : l) return (next : rest) But that, like iRecurse above, doesn't work.

9 13