October 2015 - Glasgow-haskell-users

Discovery of source dependencies without --make
by Lars Hupel 14 Dec '15

14 Dec '15

(cross-posted from [haskell-cafe]) Hi, I've got a problem in our – admittedly complex – build process. We're running an automated grading system for student-submitted homework exercises. The compilation proceeds in stages: 1) Compile a library file, marked as trustworthy 2) Compile student submissions using Safe Haskell, able to access a whitelisted set of packages and the previously compiled library files 3) Compile the test suite 4) Link everything together into an executable The invocation is similar to this: ghc -c -outputdir "$OUT" -XTrustworthy Library.hs ghc -c -outputdir "$OUT" -i"$OUT" -XSafe "$SUBMISSION" ghc -c -outputdir "$OUT" -i"$OUT" Test_Suite.hs ghc -outputdir "$OUT" -i"$OUT" -o "$OUT/runner" The second stage works well under the assumption that there's just one single submitted file. If there's more than one, I need to specify them in topological order wrt their module dependencies. Consider two simple modules: A.hs > module A where > > import B > import Library B.hs > module B where Invoking GHC with "A.hs" "B.hs" will fail: A.hs:3:1: Failed to load interface for ‘B’ My naive attempt at solving that problem was to just insert "--make" into the compiler flags. No luck, because according to [1, §4.7.3], "--make" will only look at source files and will ignore the "Library.hi" file from the previous compilation stage. If I give the path to "Library.hs" as well, GHC insists on recompiling it with changed flags ("-XSafe" instead of "-XTrustworthy"). Reading a bit further, I discovered the '-M' flag, however, not only does it output Makefile-formatted output, it also ignores .hi files. Is there any way to get the dependency discovery of '--make' without the rest? I know I could probably make it work if I made a package out of the first build stage, but I really want to avoid that in order to not increase the build complexity even further. Cheers Lars [1] <https://downloads.haskell.org/~ghc/7.6.3/docs/html/users_guide/separate-com…>

8 15

type error formatting
by Evan Laforge 04 Nov '15

04 Nov '15

Here's a typical simple type error from GHC: Derive/Call/India/Pakhawaj.hs:142:62: Couldn't match type ‘Text’ with ‘(a1, Syllable)’ Expected type: [([(a1, Syllable)], [Sequence Bol])] Actual type: [([Syllable], [Sequence Bol])] Relevant bindings include syllables :: [(a1, Syllable)] (bound at Derive/Call/India/Pakhawaj.hs:141:16) best_match :: [(a1, Syllable)] -> Maybe (Int, ([(a1, Syllable)], [(a1, Sequence Bol)])) (bound at Derive/Call/India/Pakhawaj.hs:141:5) In the second argument of ‘mapMaybe’, namely ‘all_bols’ In the second argument of ‘($)’, namely ‘mapMaybe (match_bols syllables) all_bols’ I've been having more trouble than usual reading GHC's errors, and I finally spent some time to think about it. The problem is that this new "relevant bindings include" section gets in between the expected and actual types (I still don't like that wording but I've gotten used to it), which is the most critical part, and the location context, which is second most critical. Notice the same effect in the previous sentence :) After I see a type error the next thing I want to see is the where it happened, so I have to skip over the bindings, which can be long and complicated. Then I usually know what to do, and only look into the bindings if something more complicated is going on, like wonky inference. So how about reordering the message: Derive/Call/India/Pakhawaj.hs:142:62: Couldn't match type ‘Text’ with ‘(a1, Syllable)’ Expected type: [([(a1, Syllable)], [Sequence Bol])] Actual type: [([Syllable], [Sequence Bol])] In the second argument of ‘mapMaybe’, namely ‘all_bols’ In the second argument of ‘($)’, namely ‘mapMaybe (match_bols syllables) all_bols’ Relevant bindings include syllables :: [(a1, Syllable)] (bound at Derive/Call/India/Pakhawaj.hs:141:16) best_match :: [(a1, Syllable)] -> Maybe (Int, ([(a1, Syllable)], [(a1, Sequence Bol)])) (bound at Derive/Call/India/Pakhawaj.hs:141:5) After this, why not go one step further and set off the various sections visibly to make it easier to scan. The context section can also be really long if it gets an entire do block or record: Derive/Call/India/Pakhawaj.hs:142:62: * Couldn't match type ‘Text’ with ‘(a1, Syllable)’ Expected type: [([(a1, Syllable)], [Sequence Bol])] Actual type: [([Syllable], [Sequence Bol])] * In the second argument of ‘mapMaybe’, namely ‘all_bols’ In the second argument of ‘($)’, namely ‘mapMaybe (match_bols syllables) all_bols’ * Relevant bindings include syllables :: [(a1, Syllable)] (bound at Derive/Call/India/Pakhawaj.hs:141:16) best_match :: [(a1, Syllable)] -> Maybe (Int, ([(a1, Syllable)], [(a1, Sequence Bol)])) (bound at Derive/Call/India/Pakhawaj.hs:141:5) Or alternately, taking up a bit more vertical space: Derive/Call/India/Pakhawaj.hs:142:62: Couldn't match type ‘Text’ with ‘(a1, Syllable)’ Expected type: [([(a1, Syllable)], [Sequence Bol])] Actual type: [([Syllable], [Sequence Bol])] ----------------------------- In the second argument of ‘mapMaybe’, namely ‘all_bols’ In the second argument of ‘($)’, namely ‘mapMaybe (match_bols syllables) all_bols’ ----------------------------- Relevant bindings include syllables :: [(a1, Syllable)] (bound at Derive/Call/India/Pakhawaj.hs:141:16) best_match :: [(a1, Syllable)] -> Maybe (Int, ([(a1, Syllable)], [(a1, Sequence Bol)])) (bound at Derive/Call/India/Pakhawaj.hs:141:5) Thoughts? It seems simple enough that I could do myself, but of course not without buy-in.

9 15

Fwd: GHC 7.10.2 and 7.8.4 - optimization flags do not force recompilation.
by Mateusz Kłoczko 31 Oct '15

31 Oct '15

Hello I would like to ask one question about optimization flags. For different n, executing ghc --make -On MyCode.hs doesn't force recompilation for both GHC 7.10.2 and 7.8.4. Is this intended behaviour ? Or am I missing something ? Regards, Mateusz Kłoczko

2 1

Core libraries
by Paul van der Walt 16 Oct '15

16 Oct '15

Hello GHC users, For various reasons i'm trying to package something which is compatible with/equivalent to the Haskell Platform. I've been looking at the list posted at [0], and it turns out that GHC 7.10.2 doesn't bundle all the packages that are promised. (Or it's a case of PEBKAC and i apologise!) When i do `ghc-pkg list`, the output under ../package.conf.d provided by GHC notably has the following differences with the website list: $ diff website-list <(ghc-pkg list | ..prettify..) 2a3,4 > binary > bin-package-db 10c12,14 < haddock --- > ghc-prim > haskeline > hoopl 12,13c16 < old-locale < old-time --- > integer-gmp 15a19 > rts 16a21 > terminfo What surprises me is the lack of haddock, old-time and old-locale. Is this intentional, or a glitch on the website? I notice that these "missing" libraries are repeated in the `Additional Platform Libraries' and `Programs and Tools' sections. I guess this means that as a GHC packager, i should add these items separately? I guess what i'm saying is that i believe there is a bug in that list [0], but i'm not sure enough that i understand the situation to report it as such :). Sorry about the verbiage, and kind regards, p. 0. <https://www.haskell.org/platform/contents.html>

1 0

Handling multiple fds with GHC
by Markus Ongyerth 07 Oct '15

07 Oct '15

Hi, the last few days, I tried to get an IO-Event system running with GHC i.e. trigger an IO action when there is data to read from a fd. I looked at a few different implementations, but all of them have some downside. * using select package - This uses the select syscall. select is rather limited (fd cannot be >1024) * using GHC.Event - GHC.Event is broken in 7.10.1 (unless unsafeCoerce and a hacky trick are used) - GHC.Event is GHC internal according to hackage - Both Network libraries I looked at (networking (Network.Socket) and socket (System.Socket)) crash the application with GHC.Event - with 7.8+ I didn't see a way to create your own EventManager, so it only works with -threaded * using forkIO and threadWaitRead for each fd in a loop - needs some kind of custom control structure around it - uses a separate thread for each fd - might become pretty awkward to handle multiple events * using poll package - blocks in a safe foreign call - needs some kind of wrapper >From the above list, GHC.Event isn't usable (for me) right now. It would require some work for my usecase. The select option is usable, but suffers from the same problems as poll + the limitation mentioned, so it is strictly worse. This leaves me with two options: poll and forkIO + blocking. Those are based on two completely different approaches to event handling.. poll can be used in a rather classic event handling system with a main loop that blocks until an event occurs (or a timeout triggers) and handles the event in the loop. forkIO + blocking is closer to registering an action later that should be triggered by an event. My main questions right now are: 1. How bad is it for the (non-threaded) runtime to be blocking in a foreign call most of the time? 2. How significant will the overhead be for the forkIO version? 3. Is there a *good* way to use something like threadWaitRead that allows to wake up on other events as well? 4. Is there a better way to handle multiple fds that may get readable data at any time, in Haskell/with GHC right now? Thanks in advance, Ongy

3 5

using nmitchell's space leak detection technique
by Evan Laforge 02 Oct '15

02 Oct '15

Neil Mitchell wrote an article about finding space leaks by limiting the stack size: http://neilmitchell.blogspot.fr/2015/09/detecting-space-leaks.html I'm giving it a try, but the results don't make any sense to me. My understanding is that the too-deep stack indicates that someone created too many thunks, so when they get evaluated you have to descend too many times. And if the stack trace seems to be a reasonable size (say 100 stack frames somehow consuming 1mb of stack) then it means there is a recursive call in there somewhere that ghc is helpfully eliding. And lastly, that the thing about "Stack space overflow: current size 33568 bytes." always showing 33568 bytes is due to a ghc bug and it should actually be whatever limit I gave it. Is all this accurate? The stack trace jumps around a lot, to places which are not lexically present in the caller. I assume this is just lazy evaluation, so e.g. maybe 'f' doesn't call 'g', but if 'f' forces a value returned from 'g' which has not yet been forced, the stack will go to 'g'. Also sometimes things which seem like they should be run consecutively wind up as caller and callee... but perhaps this is due to using a monad with a success continuation, and without the SCCs would be eliminated since they are tail calls... or maybe they are always eliminated and I only see them in the stack when the elimination failed for some reason. At first I got a stack trace that ended in a function which is recursive, but should be productive guarded recursion: events_of :: [LEvent d] -> [d] events_of [] = [] events_of (Event e : rest) = e : events_of rest events_of (Log _ : rest) = events_of rest It's my understanding that while this isn't tail recursive, it should require only constant space. Of course the consumer could retain it all, but that wouldn't show up as too-deep recursion on that particular function, would it? Since ghc elides self-calls from the stack, the culprit could be one of the functions above, and the one immediately above was part of transforming a large list which could well be insufficiently strict somewhere. So I tried to break that out into its own call without the distraction of the continuation monad etc., just a function from a list to a list. The result is even more confusing: % build/profile/RunProfile-SpaceLeak +RTS -K2905K -xc -RTS ... force - *** Exception (reporting due to +RTS -xc): (THUNK_STATIC), stack trace: Derive.PSignal.signal, called from Derive.DeriveTest.mkevent_scale, called from Derive.DeriveTest.mkevent, called from Derive.SpaceLeak_profile.profile_cancel.make, called from Derive.SpaceLeak_profile.profile_cancel.f, called from Derive.SpaceLeak_profile.profile_cancel, called from Derive.SpaceLeak_profile.CAF Somehow these 7 calls are consuming almost 3mb of stack, so there must be a ton of elided recursion hiding in one of them. The confusing thing is that the profile_cancel function is basically just: rnf $ suspicious_function $ map make [0 .. 1024*50] So the stack is going into the 'make' function which is just a bunch of constructors to produce the values that suspicious_function will transform. As far as I can tell, the only recursion is in 'map', and the thunks from 'mkevent' down to 'PSignal.signal' only go 3 deep. But suspicious_function is definitely suspicious, because if I remove it, I no longer get an exception. But how does it manage to not be in the stack? If suspicious_function is not a good consumer and is accumulating thunks, how can I get 3 stack entries consuming 3mb? If I rnf the result of 'make' before running suspicious_function, I get a completely different stack trace, and this one with '--> evaluated by:' entries, and I don't really know what those mean. But this one at least has some recursive calls within suspicious_function. I can keep debugging via trial and error and successive rnf's as Neil describes in another article, but it would be nice to know what's going on with the stack. The stack seems fairly straightforward in strict languages, is there some documentation on how to understand the ghc version? Any clues appreciated!

1 0

Removing latency spikes. Garbage collector related?
by Will Sewell 01 Oct '15

01 Oct '15

Hi, I was told in the #haskell IRC channel that this would be a good place to ask this question, so here goes! We’re writing a low-latency messaging system. The problem is we are getting a lot of latency spikes. See this image: http://i.imgur.com/GZ0Ek98.png (yellow to red is the 90th percentile), which shows end-to-end latency of messages through the system. I have tried to eliminate the problem by removing parts of the system that I suspected to be expensive, but the spikes are still there. I’m now thinking that it’s the GC. As you can see in this output from ghc-events-analyze, work on the GC thread (red) seems to be blocking work on the main program thread (green) http://i.imgur.com/4YO5q4U.png (x axis is time, darkness of buckets is % CPU time). *Note: the graphs are not of the same run, but are typical* Do you think the GC is the most likely culprit? Is there anything I can do to confirm this hypothesis? (I looked into turning off the GC, but this seems tricky) If it is the GC, then is there anything that can be done about it? Thanks! Will

3 5