August 2009 - Glasgow-haskell-users

Compiling large source files
by Günther Schmidt 07 Aug '09

07 Aug '09

Hi all, I'm having trouble compiling very large source files, the compiler eats 2GB and then dies. Is there a way around it? Günther

6 9

GHC 6.10.2 consuming lots of memory while compiling - help?
by Justin Bailey 06 Aug '09

06 Aug '09

I apologize in advance for the vagueness of my report here - it's one of those situations I'm not sure how to cut it down to size yet. I have a module that uses HaskellDB and Template Haskell together. The module itself depends on 23 other modules, each of which give a type definition for a particular database table or view. I only mention that to emphasize that the module depends on some "big" types (HList records w/ 20+ members) and on compile-time generated code. My problem is this - when GHC compile the module, it consumes 1.2 GB of memory, takes about 10 minutes, and finally produces an object file. The memory usage seems related to template haskell, but I'm not positive. I've attached verbose output from compiling the module in question. The command line I used was: ghc -v --make -c DeliveryManagementQueries.hs -XEmptyDataDecls -XTypeSynonymInstances -XTemplateHa skell Now for my question - please ignore the specifics of haskelldb/template haskell - any suggestions for figuring out what GHC is doing, besides tried-and-true divide and conquer? Justin

4 6

What is the mutator?
by Jason Dusek 06 Aug '09

06 Aug '09

I've been reading a little about GC latency and have run across statements like this: One solution to the GC synchronisation problem would be to implement a concurrent garbage collector. Typically, however, concurrent GC adds some overhead to the mutator, since it must synchronise with the collector.some thunks are never “black-holed”, so giving a potential performance win. Unfortunately, in the parallel setting, it substantially enlarges the time window in which two or more duplicate threads might evaluate the same think, and thus -- Comparing and Optimising Parallel Haskell Implementations for Multicore Machines What is the mutator? -- Jason Dusek

3 2

Data.List permutations
by Slavomir Kaslev 05 Aug '09

05 Aug '09

A friend mine, new to functional programming, was entertaining himself by writing different combinatorial algorithms in Haskell. He asked me for some help so I sent him my quick and dirty solutions for generating variations and permutations: > inter x [] = [[x]] > inter x yys@(y:ys) = [x:yys] ++ map (y:) (inter x ys) > perm [] = [[]] > perm (x:xs) = concatMap (inter x) (perm xs) > vari 0 _ = [[]] > vari _ [] = [] > vari k (x:xs) = concatMap (inter x) (vari (k-1) xs) ++ vari k xs After that I found out that nowadays there is a permutation function in the Data.List module: > permutations :: [a] -> [[a]] > permutations xs0 = xs0 : perms xs0 [] > where > perms [] _ = [] > perms (t:ts) is = foldr interleave (perms ts (t:is)) (permutations is) > where interleave xs r = let (_,zs) = interleave' id xs r in zs > interleave' _ [] r = (ts, r) > interleave' f (y:ys) r = let (us,zs) = interleave' (f . (y:)) ys r > in (y:us, f (t:y:us) : zs) I was surprised to find that not only my version is much simpler from the one in Data.List but it also performs better. Here are some numbers from my rather old ghc 6.8.1 running ubuntu on my box: *Main> length $ permutations [1..10] 3628800 (10.80 secs, 2391647384 bytes) *Main> length $ perm [1..10] 3628800 (8.58 secs, 3156902672 bytes) I would like to suggest to change the current implementation in Data.List with the simpler one. Also, it would be nice to add variations and combinations in the Data.List module. Cheers. -- Slavomir Kaslev

5 8

Working with GHC HEAD
by Antoine Latter 04 Aug '09

04 Aug '09

Folks, I was trying to see what GHC head was like, but I've run into a few snags compiling packages. My existing binary for cabal-install can install quite a few packages, but then starts giving me strange errors eventually: >>> $ cabal --version cabal-install version 0.6.2 using version 1.6.0.1 of the Cabal library $ ghc --version The Glorious Glasgow Haskell Compilation System, version 6.11.20090724 $ cabal install zlib Resolving dependencies... Configuring zlib-0.5.2.0... ghc-stage2: /home/alatter/.ghc/x86_64-linux-6.11.20090724/package.conf:6:163: parse error on input `Nothing' cabal: Error: some packages failed to install: zlib-0.5.2.0 failed during the configure step. The exception was: exit: ExitFailure 1 <<< If I then do: >>> $ mv /home/alatter/.ghc/x86_64-linux-6.11.20090724/package.conf /home/alatter/.ghc/x86_64-linux-6.11.20090724/package.conf.back $ cabal install zlib <<< Everything works great. After looking at the package.conf being complained about, the last line looks like so: >>> InstalledPackageInfo {package = PackageIdentifier {pkgName = PackageName "cpphs", pkgVersion = Version {versionBranch = [1,7], versionTags = []}}, license = LGPL Nothing, copyright = "2004-8, Malcolm Wallace", maintainer = "Malcolm Wallace <Malcolm.Wallace(a)cs.york.ac.uk>", author = ... <<< Not the license field of the package info - we have "license = LGPL Nothing, copright ...". I thought that maybe since GHC head ships with the new dev version of Cabal, I need the dev version of cabal-install to properly install packages. However cabal-install (and most of the things I want to test out) require the 'network' package, which doesn't build against GHC head (System.Posix.Internals doesn't exist, and the functionality that 'network' uses from there now lives in GHC.IO.* and is much changed). So a few things: - Should I expect the stable version of cabal-install built against GHC 6.10 to work for head? Are the errors I'm getting expected? - Does anyone have a version of 'network' which builds against GHC head? I could bludgeon in the new GHC.IO.FD.FD type myself, but I'd thought I'd ask around first. Thanks, Antoine

4 4

Current GHC core documentation.
by Richard Kelsall 04 Aug '09

04 Aug '09

This page http://www.haskell.org/ghc/documentation.html has a link to the September 2001 (Draft for GHC 5.02) document describing GHC Core (in what is for me user-hostile .ps.gz format.) And this page http://www.haskell.org/ghc/docs/latest/html/users_guide/ext-core.html promises an easier format PDF document, but the link is broken. I did eventually find the 1st April 2009 GHC 6.10 document here http://www.haskell.org/ghc/docs/6.10.2/html/ext-core/core.pdf and a bit on this page http://www.haskell.org/ghc/docs/latest/html/users_guide/options-debugging.h… about GHC core. I haven't read these yet, but could I ask whether they constitute the complete current documentation for GHC core? (I'm just curious to get a flavour of what core does.) Richard.

3 2

equivalent of EXPLAIN PLAN with GHC?
by Joshua Haberman 03 Aug '09

03 Aug '09

Hello, I'm quite new to Haskell, but experienced in other languages (C, Python, Ruby, SQL, etc). I am interested in Haskell because I've heard that the language is capable of lots of optimizations based on laziness, and I want to learn more about that. I dug in with Project Euler problem #1, and wrote: main = print (show (sum [x | x <- [3..999], x `mod` 3 == 0 || x `mod` 5 == 0])) So far so good, but I want to have some way of observing what optimizations GHC has performed. Most notably, in this example I want to know if the list was ever actually constructed in memory. The "sum" function only needs the elements one at a time, in order, so if Haskell and GHC are everything I've heard about them, I would fully expect the list construction to be optimized out. :) Unfortunately I was not able to see any way of examining ghc's output to determine whether it had performed this optimization. The C it produced with '-fvia-C -C' was totally unreadable -- it looked like something from the IOCCC. :( I couldn't find any way to match up any of the code it had generated with code I had written. My attempts to objdump the binaries was similarly unproductive. Is there any kind of intermediate form that a person can examine to see how their code is being optimized? Anything like EXPLAIN PLAN in SQL? It would make it much easier to understand the kinds of optimizations Haskell can perform. I'm not looking so much for profiling -- obvious this program is trivial and takes no time. I just want to better understand what kind of optimizations are possible given Haskell's language model. Thanks! Josh

4 7

Re to `optimization for list'
by Serge D. Mechveliani 02 Aug '09

02 Aug '09

Joshua Haberman <joshua(a)reverberate.org> writes on 2 Aug 2009 > Hello, I'm quite new to Haskell, but experienced in other languages (C, > Python, Ruby, SQL, etc). I am interested in Haskell because I've heard > that the language is capable of lots of optimizations based on laziness, > and I want to learn more about that. > > I dug in with Project Euler problem #1, and wrote: > > main = > print (show (sum [x | x <- [3..999], x `mod` 3 == 0 || x `mod` 5 == 0])) > > So far so good, but I want to have some way of observing what > optimizations GHC has performed. Most notably, in this example I want > to know if the list was ever actually constructed in memory. The "sum" > function only needs the elements one at a time, in order, so if Haskell > and GHC are everything I've heard about them, I would fully expect the > list construction to be optimized out. :) > [..] Your example is rather complex to start with, because `sum' has rather a complex definition in the Haskell library. Consider first a more pure example: main = putStr (shows (or' ys) "\n") where n = 10^8 ys = replicate n False or' :: [Bool] -> Bool or' [] = False or' (x: xs) = x || (or' xs) (the functions `||' (disjunction) and `replicate' are of the library). Compile it with ghc --make -O (for any occasion), and then, it will be evident that it takes a constant in n space (and stack) when running (run it like this: `./T +RTS -M1m -RTS' ). I think, this is not due to optimization but due to the very definition of evaluation in Haskell -- Head Normal Form. Compute this by hand: or' (replicate 9 False) = or' (False: (replicate 8 False)) = False || (or' (replicate 8 False)) = or' (replicate 8 False) = -- by definition of || ... or' [] = False. This is just the definition of evaluation in the Haskell language. In this example, the list under or' never expands further than for 2 members. Consider a more complex example (and more close to your example): main = putStr (shows (sum2 ys) "\n") -- sum' where n = 10^7 ys = [1 .. n] sum2 :: [Int] -> Int sum2 xs = sm xs 0 where sm [] s = s sm (x: xs) s = sm xs (s+x) sum' :: [Int] -> Int sum' [] = 0 sum' (x: xs) = x + (sum' xs) (the function [n .. m] is of the library). Compile it with ghc --make -O, and then, it will be evident that sum2 takes a constant in n space (and stack) when running (run it like this: `./T +RTS -M1m -RTS' ). Let us investigate this. Furst, compute this by hand with treating (+) as not of the library: sm [1 .. 4] 0 = sm (1:[2 .. 4]) 0 = sm [2 .. 4] (1+0) = sm (2: [3 .. 4]) (1+0) = sm [3 .. 4] (2+(1+0)) = ... It must accumulate an intermediate stack of size O(n). Why ghc-6.10.4 computes it in a constant space (and stack) ? The following is my guess. And let the Haskell experts correct me, please. In the Haskell-98 library it is defined that (+) is strict on the type of Int (right?). Therefore, the compiler optimizes (1+0) by replacing it immediately with 1. Taking in account that all other parts are computed lazily, the evaluation of the compiled program is like this: sm [1 .. 4] 0 = sm (1:[2 .. 4]) 0 = sm [2 .. 4] (1+0) = sm [2 .. 4] 1 = sm (2: [3 .. 4]) 1 = sm [3 .. 4] (2+1) = sm [3 .. 4] 3 = ... So, it is the result of combining of the evaluation definition with a certain compiler's optimization related to the strictness info for Int.(+). Right? As to sum', it still takes O(n) space -- in ghc-6.10.4. I think, this is because it is difficult for a compiler to guess how strictness of Int.(+) may correlate to the definition of sum'. Probably, it needs some kind of inductive reasoning about sum' to optimize it (for example) into sum2. Right ? Regards, ----------------- Serge Mechveliani mechvel(a)botik.ru

1 0