April 2009 - Glasgow-haskell-users

Threads and memory management
by Johannes Waldmann 30 Apr '09

30 Apr '09

Dear all, I was wondering what is the current status of the ghc RTS with respect to threading. Is it true that the allocator and deallocator (garbage collector) are still single-threaded? I made this example: import Control.Concurrent import Control.Concurrent.QSemN primes1 = sieve [ 2 .. ] primes2 = 2 : sieve [ 3, 5 .. ] sieve (x : ys) = x : sieve ( filter ( \ y -> 0 < y `mod` x ) ys ) main = do s <- newQSemN 0 forkIO $ do print $ sum $ take 10000 primes1 ; signalQSemN s 1 forkIO $ do print $ sum $ take 10000 primes2 ; signalQSemN s 1 waitQSemN s 2 Between the two computations, I see absolutely no data dependencies. So the naive expectation is that using two threads (in the RTS, on a multicore machine) gives half the execution time. I compiled with ghc-6.10.2 -o Par -O2 Par.hs -threaded and when I run with +RTS -N2 (instead of N1), I get my CPU load at 140 % (instead of 100) but the total run time (wall clock) stays nearly constant (i.e. CPU time goes up). For reference, I'm pretty sure a similar thing happens with the current mono implementation (they use Boehm GC and that does not know about C#/CIL threads ?), and this is preventing multi-core speedups in "straightforward" programs (that have no data dependencies, but lots of object creation, e.g. via (hidden) Enumerators and such.) Well, then, if the two Haskell threads are (nearly) completely independent like the above, it would be better to compile and run two separate executables and have them communicate via the OS (pipe or port). But that shouldn't be! (the OS being better than Haskell) Is there was a way of partitioning the memory (managed by the ghc RTS) in totally independent parts that each have their stand-alone memory management. Of course then all communication had to go via some Control.Concurrent.Chan, but that should be fine, if there is little of them. Well, just some thought. This idea can't be new? Tell me why it couldn't possibly work ... J.W.

5 7

Compiler modes / FFI / forcing CPP
by Philip Hölzenspies 30 Apr '09

30 Apr '09

Dear GHC-ers, From earlier questions, it seems no one wants to get down and dirty with linker tricks in ghc. I'm now looking over an alternative solution. I'm currently trying to use SDL. There's a wrapper on hackage, which works fine under Linux, but it takes some tinkering under Mac OS X, because OS X needs instantiated Cocoa stuff for the SDL entry point. Long story short, to use SDL with ghc on the OS X, a little wrapper c- file needs to be written, because the linker will find a 'main' function in libSDLmain.a which expects a function 'SDL_main' as an entry point into the user code. I have written a small c file that can be used to wrap around any haskell module (see below), based on the macro (cpp) definition of MODNAME. I can compile this c file with gcc, for, let's say a module Foo with entry point Foo.bar (so, if we could still have ghc make the entry point, we would say ghc -main-is Foo.bar): gcc -c -DMODNAME=Foo -D"ENTRYPOINT=bar();" -o Foo_entry.o entry.c However, this only works after I explicitly add ghc's "include" dir to my search path so that HsFFI.h can be found. Before I had this pre- processor trick, I could simply feed the c file to ghc, together with all my haskell files. Ghc then simply adds the correct includes and feeds the c file to the c compiler. Unfortunately, giving ghc the above -D options doesn't do much good. The problem seems to lie with the suffix recognition ghc does. I took this from [1]. With the -cpp switch, the c pre-processor is used, but only when the file has a .hs suffix. The c compiler is only run when the file has a .c suffix. Using -x .hs puts the file correctly through the c pre-processor, but then fails to compile it, because it's not haskell code. Using -x .c doesn't do pre-processing (even with -cpp). I tried to compile this wrapper using this command (assume now that we have Foo.hsMain as an entry point): ghc -cpp -pgmP cc -optP-E -DMODNAME=Foo -o Foo_entry.o entry.c (Notice that you can't just use cpp, because on OS X that's a shell script that doesn't provide CPPs full strength, so you want "cc -E".) This doesn't work: entry.c:27:2: error: #error MODNAME undefined! Trying to be more explicit as to where the -D should go: ghc -cpp -pgmP cc -optP-E -optP-DMODNAME=Foo -o Foo_entry.o entry.c does exactly the same. Now faking .hs: ghc -cpp -pgmP cc -optP-E -optP-DMODNAME=Foo -o Foo_entry.o -x hs entry.c which produces the error (coming from stdio.h inclusion): /usr/include/i386/_types.h:37:28: parse error on input `;' So, I guess my question is: Is it possible to let a c input file of ghc pass through *both* the c pre-processor *and* the c compiler? Kind regards, Philip [1] GHC User's Guide: Section 5.4: Modes of operation http://www.haskell.org/ghc/docs/latest/html/users_guide/modes.html entry.c: #ifdef MODNAME #ifndef ENTRYPOINT #define ENTRYPOINT hsMain(); #endif #include <stdio.h> #include "HsFFI.h" #define __HS_mangle(name) __stginit_ ## name #define __Xpand_HS_mangle(nm) __HS_mangle(nm) #define __hs_mangled_module __Xpand_HS_mangle( MODNAME ) #define __pack_quotes(x) #x #define __calc_include(x) __pack_quotes(x##_stub.h) #define __Xpand_calc_include(x) __calc_include(x) #include __Xpand_calc_include(MODNAME) extern void __hs_mangled_module ( void ); int SDL_main(int argc, char *argv[]) { hs_init(&argc, &argv); hs_add_root(__hs_mangled_module); ENTRYPOINT hs_exit(); return 0; } #else /* MODNAME */ #error MODNAME undefined! #endif /* MODNAME */

2 1

Multicore Haskell user experiences
by Scott Michel 29 Apr '09

29 Apr '09

[Sorry if this is the wrong place to hold this discussion... I'll move it if told to do so...] Simon P-J (et. al.) did recently publish a paper on making the Haskell runtime more multicore friendly, addressing the GC issue (among many things.) More broadly, though, there are two opportunities for multicore Haskell presentations: a) An IEEE conference in Pasedena, CA, July 19-21. The major theme is multicore in space (if you're familiar with OPERA/MAESTRO), but as a one of the conference-within-a-conference organizers, I'm interested in multicore programmability. Work on runtime environments, new language features, good-bad-and-ugly experiences, and "hard technical topics" are welcome. Looking for one or two presentations for this. b) Supercomputing '09 in Portland, OR next November. Focus here is primarily on user experience programming multicore with up-and-coming languages/environments. Any suggestions or presentation pitches are welcome. -scooter On Wed, Apr 29, 2009 at 3:01 AM, Johannes Waldmann <waldmann(a)imn.htwk-leipzig.de> wrote: >> PS: Anyone got a multicore Haskell experience they want to share at >> Supercomputing this year? The guys from Galois did a great job >> shocking the audience last year. > > well, this is my shocking experience: > http://comments.gmane.org/gmane.comp.lang.haskell.glasgow.user/16809 > > where of course the conclusion is that parallelism > does not work "out of the box". No surprise? > While I completely would buy this statement for Java, C# etc > I hoped it was different for Haskell. > I don't want to rewrite my programs to make them less depend on GC. > Sure it could be done but then I could use Java from the start. > > J.W. > >

2 1

Chimeric syntax
by Scott Michel 28 Apr '09

28 Apr '09

I've been hacking along on a NetBeans Haskell plugin (*) Looking at Parser.y.pp, because both Eclipse and NetBeans work with antlr, it seems like there are interesting cases in which chimeric constructions parse correctly. Here's an example: class ParsedModule m where let { a = 1; b = 2; } in a + b :: Int :: Int This is mostly accepted by ghc, which complains with an invalid type signature. This got me to thinking that either ghc has issues or I have some fundamental misunderstanding of Haskell syntax. Or, maybe I should use someone else's grammar. -scooter (*) Don't tell me about eclipsefp2: I know already. It's the Monty Python parrot of Haskell IDE support. And the code is about as Teutonic as one can get. :-)

3 3

6.10.3 plans
by Ian Lynagh 27 Apr '09

27 Apr '09

Hi all, Due to some problems with the handling of ^C in ghci, we are planning to do a 6.10.3 release. The idea is that, rather than being a full-blown release process, including all the various fixes that have been made since 6.10.2, we will only do what is necessary to resolve the ^C issue. By keeping the changes to a minimum, we will minimise the amount of testing, release-note writing, etc that is necessary. An exception to this rule is that we will probably also rebundle time in the bindists, as that has little chance of breaking anything else. There are actually two problems with ^C. One is due to the changes in the signal handling in ghci, and we will fix that. The other is that, on some platforms (including x86/Linux), the interaction between editline and GHC means that pressing ^C causes a segfault. Given all the other issues that people have had with editline, we plan to use haskeline in 6.10.3 instead. We already planned to do this for 6.12, but we think that it makes sense to do it sooner rather than later. The plan is: * Add haskeline and deps as boot packages * Move the interactive modules from the ghc package to the ghc-bin package. This means that the ghc package will not depend on the new boot packages, so there will not be any problems with upgrading them. Any comments? Thanks Ian

9 14

Re: Threads and memory management
by Jost Berthold 27 Apr '09

27 Apr '09

> Message: 8 > Date: Fri, 24 Apr 2009 19:20:46 +0200 > From: Johannes Waldmann <waldmann(a)imn.htwk-leipzig.de> > Subject: Threads and memory management > To: "glasgow-haskell-users(a)haskell.org" > <glasgow-haskell-users(a)haskell.org> > Message-ID: <49F1F4EE.8070306(a)imn.htwk-leipzig.de> > Content-Type: text/plain; charset="iso-8859-1" > > Dear all, > > I was wondering what is the current status of the ghc RTS > with respect to threading. Is it true that the allocator > and deallocator (garbage collector) are still single-threaded? > > > I made this example: > ... > Well, then, if the two Haskell threads are (nearly) completely > independent like the above, it would be better to compile and run > two separate executables and have them communicate via the OS (pipe or > port). But that shouldn't be! (the OS being better than Haskell) > > Is there was a way of partitioning the memory (managed by the ghc RTS) > in totally independent parts that each have their stand-alone > memory management. Of course then all communication > had to go via some Control.Concurrent.Chan, > but that should be fine, if there is little of them. > > Well, just some thought. This idea can't be new? > Tell me why it couldn't possibly work ... > > J.W. Hello everybody, Since I did not see any other replies, I thought I might give you some pointers to more information. Other people will perhaps follow up with more details. In all, quite a few things are recently going on about threading in the GHC world, and it is even difficult to keep the oversight. - GHC has undergone a big overhaul with respect to threading support since last September. I have started this work at Microsoft Research last summer, and a lot more has been done by Simon Marlow since. There are forthcoming papers (mainly one submitted to ICFP) about this work. The main focus here was the shared-heap implementation of the Glasgow-parallel Haskell programming model (see http://www.macs.hw.ac.uk/~dsg/gph/ , mainly the paper here: http://www.macs.hw.ac.uk/~dsg/gph/papers/html/Strategies/strategies.html) This programming model is supported in GHC versions since 2004 already, but should now deliver much better performance. That said, the vast majority of the latest GHC optimisations carried out in the GHC HEAD aim at improving thread performance (the explicit basis of the more implicit programming model), and thus relate directly to what you observe. The version you have tested with is GHC-6.10.2, which does not include substantial threading optimisations. - a related question is tool support for parallel performance tuning. Very recently, GHC (HEAD) supports a visual post-mortem analysis, and a graphical tool "ThreadScope" has been developed by Satnam Singh and others. http://raintown.org/?page_id=132 There is a related posting in glasgow-haskell-users (11 March). - GHC has parallel garbage collection since 2007 already. However, recent work showed that this parallel GC sometimes hampers performance, in particular on Linux systems. See this thread for more: http://www.haskell.org/pipermail/glasgow-haskell-users/2009-April/017050.ht… - about your remark that separate OS processes should communicate, rather than having the Haskell RTS manage all threads: yes, there are programming models around which aim at parallel Haskell with distributed heap. Aside from the older GpH cluster implementations, you might have a look at the language Eden: http://www.mathematik.uni-marburg.de/~eden/?content=paper (and personally, I might apologise for the suboptimal web presence...) Using Eden (and its implementation) would be a way to realise the model you propose, communicating processes with separate heaps, but managed by a common parallel runtime system and programmed in one single program. Let me add that Eden also provides a tool with features very similar to what recently became "ThreadScope". Interestingly, a number of recent experiments have shown that Eden has competitive performance to the shared heap GpH implementation when executed on multicores, delegating all communication to the underlying middleware (most commonly PVM). Early papers discussing alike results have been published recently or are in preparation. http://www-fp.cs.st-and.ac.uk/~kh/mainICPP09.pdf I strongly support the idea to collect all this related information systematically! Kickoff is already present here: http://haskell.org/haskellwiki/Performance/Parallel However, the question is where to start... the field is indeed very broad, and the page above will surely focus on GHC rather than general ideas. So I hesitate in dropping all this information into the wiki and rather send a mail for now. Cheers Jost Berthold

2 1

Re: Newby question re quicksort
by leledumbo 27 Apr '09

27 Apr '09

I think you miss something important here: quickslop :: (Ord a) => [a] ->[a] quickslop [] = [] quickslop (x:xs) = quickslop [a | a <- xs, a <= x] ++ [x] ++ quickslop [b | b <- xs, b > x] -- View this message in context: http://www.nabble.com/Newby-question-re-quicksort-tp23250159p23250345.html Sent from the Haskell - Glasgow-haskell-users mailing list archive at Nabble.com.

1 0

Some suggestion for "mutually recursive modules" problem.
by Andy Stewart 26 Apr '09

26 Apr '09

Dear GHC developers: First thanks for your work with GHC! And i have suggestion for "mutually recursive modules" problem: Now GHC use .hs-boot fix "mutually recursive modules" problem, yes it can work, but it not pretty. So GHC can fix this problem automatically at next release version? And programmer not need write .hs-boot file, then programmer's life will be more easier. Thanks! -- Andy

2 1

Static library to redefine entrypoint
by Philip K.F. Hölzenspies 25 Apr '09

25 Apr '09

Dear GHCers, I am trying to write a wrapper library for lab work to give to students. My problem is, that the libraries I use require initialization that I really want to hide from our students. The wrapper I'm writing is compiled to a static library and installed with cabal, so students can just "ghc --make" or "ghci" their sources. Here comes the problem: I want to define main and let students just define labMain as an entry point to their program. How can I have my library use "labMain" without a definition? Keep in mind that I want to give them a cabalized library that they can just link to, so I can't give them a template file that they fill and compile together with my code. Is it at all possible to have "external" functions and letting the linker sort stuff out? Regards, Philip

3 3

writeFile/readFile on multiple threads leads to error
by Neil Mitchell 24 Apr '09

24 Apr '09

Hi, I've got a multi-threaded application which occasionally generates failures in openFile. I haven't been able to reproduce the errors reliably, the code is way too large to send over, and any small attempts at invoking the same problem don't seem to work. Despite the uselessness of the bug report, I thought I'd share what I've seen and how I fixed it. I have many threads, which read and write files. Every so often one thread will write a file, then another thread will read the same file - but fail during the open call. There are locks to ensure that the write call finishes before the read call begins. I modified the code to give: do print ("READ START",x) ; res <- readFile x ; print ("READ STOP",x) ; return res do print ("WRITE START",x); writeFile x src ; print ("WRITE STOP",x) I then get on the console: WRITE START foo WRITE STOP foo READ START foo openFile doesn't have permission to open foo. The writeFile/readFile are happening in different threads, and they usually succeed - but not always. The bug seems to go away when I add performGC just after writeFile. My guess is that something in the openFile/hClose pair isn't really closed until a garbage collection happens. All this is using GHC 6.10.2 on XP through Cygwin. I'm happy to supply more details if you can think of anything that will help, Thanks Neil

6 11