February 2012 - Glasgow-haskell-users

Unpack primitive types by default in data
by Johan Tibell 01 Mar '12

01 Mar '12

Hi all, I've been thinking about this some more and I think we should definitely unpack primitive types (e.g. Int, Word, Float, Double, Char) by default. The worry is that reboxing will cost us, but I realized today that at least one other language, Java, does this already today and even though it hurts performance in some cases, it seems to be a win on average. In Java all primitive fields get auto-boxed/unboxed when stored in polymorphic fields (e.g. in a HashMap which stores keys and fields as Object pointers.) This seems analogous to our case, except we might also unbox when calling lazy functions. Here's an idea of how to test this hypothesis: 1. Get a bunch of benchmarks. 2. Change GHC to make UNPACK a no-op for primitive types (as library authors have already worked around the lack of unpacking by using this pragma.) 3. Run the benchmarks. 4. Change GHC to always unpack primitive types (regardless of the presence of an UNPACK pragma.) 5. Run the benchmarks. 6. Compare the results. Number (1) might be what's keeping us back right now, as we feel that we don't have a good benchmark set. I suggest we try with nofib first and see if there's a different and then move on to e.g. the shootout benchmarks. I imagine that ignoring UNPACK pragmas selectively wouldn't be too hard. Where the relevant code? Cheers, Johan

8 29

New STM?
by wren ng thornton 29 Feb '12

29 Feb '12

Hello all, I had some patches committed to STM a while back[1], and for some reason I was expecting them in the new GHC release. Does STM only ship with the Platform? If not, then any idea when the next version of STM will be released? [1] http://hackage.haskell.org/trac/ghc/ticket/5104 http://www.haskell.org/pipermail/cvs-libraries/2011-April/012914.html -- Live well, ~wren

2 2

Understanding the -A and the -H flags
by Johan Tibell 29 Feb '12

29 Feb '12

Hi! I'm trying to understand the interaction between the -A and -H RTS flags. The documentation at http://www.haskell.org/ghc/docs/7.4.1/html/users_guide/runtime-control.html says that if you use -H (with or without an argument) it implicitly implies some value of -A. However, it's not clear to me what value -A will get and how that value is related to the value of -H. For example, if I set the suggested heap size to 1G, using -H1G, surely the size of the nursery (-A) won't be "whatever is left over," but something more reasonable e.g. the size of the L2 cache? Perhaps it would make sense to document the actual algorithm used to set -A given -H (with and without argument.) Cheers, Johan

3 6

Re: Records in Haskell
by Anthony Clayden 28 Feb '12

28 Feb '12

Whoa! suddenly a deluge over the DORF proposal. I don't have time to reply fully now, but I must say: Barney you have got it all wrong. No, DORF does not attach one class to each label. There is only one class 'Has', with methods get and set. Each record decl generates an instance for the combination of record/field. You can't mix declared and free-standing labels in the same record. The switch for DORF is at the module level: in a module either all records and labels use DORF, or none do (that is, they use H98 style with each field name being unique). AntC ----- Original Message Follows ----- > > My objection is that I'm not sure if there is ever a > > case where "you really want things to be polymorphic > over all records". > > Well, I don't have a simple, really convincing example, > but there are certainly things I want to play with. More > importantly, DORF automatically attaches one class to each > label, but this is often not what you want. For example, > if you have two fields "firstname" and "lastname" the > associated classes are less useful: what you really want > is > > > class (Has r "firstname" String, Has r "lastname" > String) => HasPersonalName r > > so that you can define > > > fullname :: HasPersonalName r => r -> String > > fullname r = r.firstname ++ " " ++ r.lastname > > You may also want to define subclasses to express more > specific conditions. In general, the compiler cannot > automatically deduce what is semantically important: you > need to define it yourself. The Has class is the base on > which you can build. > > > It doesn't seem like the > > Haskell way to have the less safe thing as the one > > that's default and convenient, and to allow the > > programmer to layer a more-safe thing on top of it if he > > or she wants to. It seems more like the Haskell way to > have the safer thing be the default and to require extra > > work if you want to do something less safe*. > > I think you are using the word "safe" in a slightly > misleading way. None of this is mathematically unsafe, > because projections are natural (truly polymorphic). The > "safety" that is broken here is nothing to do with the > semantics of the language, it is to do with the semantics > of the system being implemented, and that is something the > compiler cannot infer. As my example above shows, it > doesn't always correspond one to one with the labels. > > The Haskel way is to make things as polymorphic as is > mathematically safe, even when this goes beyond the > programmers original intention. You can then restrict this > polymorphism by giving explicit less general types in the > same way as in my examples. I think my approach is more > Haskel like. > > Another important Haskel design consideration is to reuse > parts of the language where possible, rather than > introduce new structures. Type classes were originally > introduced to deal with equality and numeric functions, > but were reused for many things including monads. My > approach achieves the same as DORF (and more), but using > existing language features instead of introducing new > ones. > > Barney. > > > _______________________________________________ > Glasgow-haskell-users mailing list > Glasgow-haskell-users(a)haskell.org > http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

6 6

Increasing number of worker tasks in RTS (GHC 7.4.1) - how to debug?
by Sanket Agrawal 28 Feb '12

28 Feb '12

I wrote a program that uses a timed thread to collect data from a C producer (using FFI). The number of threads in C producer are fixed (and created at init). One haskell timer thread uses threadDelay to run itself on timed interval. When I look at RTS output after killing the program after couple of timer iterations, I see number of worker tasks increasing with time. For example, below is an output after 20 iterations of timer event: MUT time (elapsed) GC time (elapsed) Task 0 (worker) : 0.00s ( 0.00s) 0.00s ( 0.00s) Task 1 (worker) : 0.00s ( 0.00s) 0.00s ( 0.00s) .......output until task 37 snipped as it is same as task 1....... Task 38 (worker) : 0.07s ( 0.09s) 0.00s ( 0.00s) Task 39 (worker) : 0.07s ( 0.09s) 0.00s ( 0.00s) Task 40 (worker) : 0.18s ( 10.20s) 0.00s ( 0.00s) Task 41 (worker) : 0.18s ( 10.20s) 0.00s ( 0.00s) Task 42 (worker) : 0.18s ( 10.20s) 0.00s ( 0.00s) Task 43 (worker) : 0.18s ( 10.20s) 0.00s ( 0.00s) Task 44 (worker) : 0.52s ( 10.74s) 0.00s ( 0.00s) Task 45 (worker) : 0.52s ( 10.75s) 0.00s ( 0.00s) Task 46 (worker) : 0.52s ( 10.75s) 0.00s ( 0.00s) Task 47 (bound) : 0.00s ( 0.00s) 0.00s ( 0.00s) After two iterations of timer event: MUT time (elapsed) GC time (elapsed) Task 0 (worker) : 0.00s ( 0.00s) 0.00s ( 0.00s) Task 1 (worker) : 0.00s ( 0.00s) 0.00s ( 0.00s) Task 2 (worker) : 0.07s ( 0.09s) 0.00s ( 0.00s) Task 3 (worker) : 0.07s ( 0.09s) 0.00s ( 0.00s) Task 4 (worker) : 0.16s ( 1.21s) 0.00s ( 0.00s) Task 5 (worker) : 0.16s ( 1.21s) 0.00s ( 0.00s) Task 6 (worker) : 0.16s ( 1.21s) 0.00s ( 0.00s) Task 7 (worker) : 0.16s ( 1.21s) 0.00s ( 0.00s) Task 8 (worker) : 0.48s ( 1.80s) 0.00s ( 0.00s) Task 9 (worker) : 0.48s ( 1.81s) 0.00s ( 0.00s) Task 10 (worker) : 0.48s ( 1.81s) 0.00s ( 0.00s) Task 11 (bound) : 0.00s ( 0.00s) 0.00s ( 0.00s) Haskell code has one forkIO call to kick off C FFI - C FFI creates 8 threads. Runtime options are "-N3 +RTS -s". timer event is kicked off after forkIO. It is for the form (pseudo-code): timerevent <other arguments> time = run where run = do threadDelay time >> do some work >> run where <other variables defined for run function> I also wrote a simpler code using just timer event (fork one timer event, and run another timer event after that), but didn't see any tasks in RTS output. I tried searching GHC page for documentation on RTS output, but didn't find anything that could help me debug above issue. I suspect that timer event is the root cause of increasing number of tasks (with all but last 9 tasks idle - I guess 8 tasks belong to C FFI, and one task to timerevent thread), and hence, memory leak. I will appreciate pointers on how to debug it. The timerevent does forkIO a call to send collected data from C FFI to a db server, but disabling that fork still results in the issue of increasing number of tasks. So, it seems strongly correlated with timer event though I am unable to reproduce it with a simpler version of timer event (which removes mvar sync/callback from C FFI).

2 4

Enormous FFI speedup observed
by Joachim Breitner 28 Feb '12

28 Feb '12

Hi, as I wrote on http://www.joachim-breitner.de/blog/archives/546-GHC-7.4.1-speeds-up-arbtt-… I observe an enormous speed up (22×) in arbtt, and it seems to be related to FFI function calls. The release notes do not give a hint what has changed to result in this speed up, so I’m curious whether someone here knows the reason, or can make a good guess? Thanks, Joachim -- Joachim "nomeata" Breitner mail(a)joachim-breitner.de | nomeata(a)debian.org | GPG: 0x4743206C xmpp: nomeata(a)joachim-breitner.de | http://www.joachim-breitner.de/

2 1

parallel build fixes for 7.4.2
by Conrad Parker 28 Feb '12

28 Feb '12

Hi, recently we've been tweaking our internal build system at Tsuru to handle parallel builds of both cabal packages via 'cabal-sort --makefile' and our local code tree via 'ghc -M'. In addition to the recompilation checker fixes of #5878, the following would be great to have in 7.4.2: 1) http://hackage.haskell.org/trac/ghc/ticket/5891 -- the patch fixes a race condition in creating parent directories for built object files 2) master commit b6f94b5 "Compile link .note section separately from main.c" -- I think this is the patch that fixes link errors we've seen during parallel builds (via ghc -M) with 7.4.1, such as: /x/y/z.o: file not recognized: File truncated and: /usr/bin/ld: BFD (GNU Binutils for Ubuntu) 2.20.51-system.20100908 internal error, aborting at ../../bfd/merge.c line 872 in _bfd_merged_section_offset Will everything currently in master already be included in the next release or is it a separate branch? (If it's a separate branch I'll do some more testing to confirm that b6f94b5 is the patch that fixes the link error). Conrad.

3 2

Re: Records in Haskell
by Anthony Clayden 27 Feb '12

27 Feb '12

> Actually, I looked at the SORF page again: > http://hackage.haskell.org/trac/ghc/wiki/Records/OverloadedRecordFields > There is now an 'Alternative Proposal' section that claims > to solve polymorphic update. > If anyone has comments on this please let us know! Yes, Greg the quasifunctor stuff has been there for some time. You seem to be confused between polymorphic fields vs type-changing update. Quasifunctor is talking about type-changing update. (Look at the example given for `data R a`: there's no polymorphic field.) DORF already has a working approach to type-changing update. I make reference to quasifunctor in the DORF proposal. Essentially, if quasifunctor can be made to work for SORF, it can also be made to work for DORF. (Because what underlies both is to use a Has class with methods get/set.) No, I don't think anybody has a satisfactory approach to updating polymorphic/higher-ranked fields. (DORF mentions one, but it's a ghastly hack. In addition to higher-ranked types, SPJ is also concerned about h-r's with constraints. I've added a very speculative piece in DORF's comparison to SORF that considers using Constraint Kinds. (It's probably piling ghastliness upon ghastliness.) > ... and not automatically > abstract over fields. This leaves all of our future > options open while satisfying the narrow issue at hand. > What on earth do you mean by "not automatically abstract over fields"? AntC

4 4

parallelizing ghc
by Evan Laforge 27 Feb '12

27 Feb '12

I recently switched from ghc --make to a parallelized build system. I was looking forward to faster builds, and while they are much faster at figuring out what has to be rebuilt (which is most of the time for a small rebuild, since ld dominates), compilation of the whole system is either the same or slightly slower than the single threaded ghc --make version. My guess is that the overhead of starting up lots of individual ghcs, each of which has to read all the .hi files all over again, just about cancels out the parallelism gains. So one way around that would be parallelizing --make, which has been a TODO for a long time. However, I believe that's never going to be satisfactory for a project involving various different languages, because ghc itself is never going to be a general purpose build system. So ghc --make provides two things: a dependency chaser and a way to keep the compiler resident as it compiles new files. Since the dependency chaser will never be as powerful as a real build system, it occurs to me that the only reasonable way forward is to split out the second part, by adding an --interactive flag to ghc. It would then read filenames on stdin, compiling each one in turn, only exiting when it sees EOF. Then a separate program, ghc-fe, can wrap ghc and acts like a drop-in replacement for ghc. It would be nice if ghc could atomically read one line from the input, then you could just start a bunch of ghcs behind a named pipe and each would steal its own work. But I don't think that's possible with unix pipes, and of course there are still a few non-unix systems out there. And I guess ghc-fe has to wait for the compilation to finish, so I guess ghc has to print a status line when it completes (or fails) a module. But it can still be done with an external distributor program that acts like a server: starts up n ghcs, distributes src files between them, and shuts them down then given the command: data Ghc = Ghc { status :: Free|Busy, in :: Handle, out :: Handle, pid :: Int } main = do origFlags <- getArgs ghcs <- mapM (startup origFlags) [0..cpus] socket <- accept while $ read socket >>= \case of Quit -> return False Compile ghcFlags src -> forkIO $ assert $ ghcFlags == origFlags result <- bracket (findFreeAndMarkBusy ghcs) markFree $ \ghc -> do tellGhc ghc src readResult ghc write socket result return True mapM_ shutdown ghcs The ghc-fe then starts a distributor if one is not running, sends a src file and waits for the response, acting like a drop-in replacement for the ghc cmdline. Build systems just call ghc-fe and have an extra responsibility to call ghc-fe --quit when they are done. And I guess if they know how many files they want to rebuild, it won't be worth it below a certain threshold. So I'm wondering, does this seem reasonable and feasible? Is there a better way to do it? Even if it could be done, would it be worth it? If the answers are "yes", "maybe not", and "maybe yes", then how hard would this be to do and where should I start looking? I'm assuming start at GhcMake.hs and work outwards from there... I'm not entirely sure it would be worth it to me even if it did make full builds, say 1.5x faster for my dual core i5, but it's interesting to think about all the same.

5 16

Removal of #include <HsFFI.h> from template-hsc.h breaks largefile support on 32bit Linux
by Eugene Crosser 27 Feb '12

27 Feb '12

Hello all, I am new here, but I want to report what I suspect may be a problem. I ran into it while using some third-party package from hackage on a 32bit Linux with ghc 7.4.1. I discovered that off_t fields in the .hsc files in the package where interpreted as 32bit words. I suspected that 64bit offsets should be used because even 32bit Linux has "largefile" support with 64bit offsets. I found that earlier versions of hsc2hs included HsFFI.h into the generated C code, and HsFFI.h in turn indirectly includes ghcautoconf.h which has #define _FILE_OFFSET_BITS 64 in it. So, if I build the .hsc files like this: hsc2hs -i HsFFI.h filename.hsc then off_t is 64bit and 64bit file manipulation syscalls are used. I did not check it but I think that earlier versions of hsc2hs where creating largefile-aware version of the code by default, because HsFFI.h was included in the code by default. This is a simple test program: ==== checktypes.hsc ==== -- run like this: hsc2hs checktypes.hsc && runhaskell checktypes.hs module Main where #include <sys/types.h> main = do putStrLn $ show (#size off_t) ======================== $ hsc2hs checktypes.hsc && runhaskell checktypes.hs 4 $ hsc2hs -i HsFFI.h checktypes.hsc && runhaskell checktypes.hs 8 As I understand, this situation means that while the ghc itself and haskell programs compiled by it are largefile-capable, any third party modules that contain .hsc files are not. If I am right, this is probably not a good thing. Please can some guru take a look at this issue? See also: http://www.haskell.org/pipermail/glasgow-haskell-users/2009-February/016606… http://www.haskell.org/pipermail/cvs-ghc/2011-September/065848.html Thanks, Eugene

4 10