- ghc-devs - Haskell.org

Add Functor/Applicative instances for all Monads in GHC
by David Luposchainsky 08 May '13

08 May '13

Hello GHC devs, due to the formation of the Haskell Prime Committee [1], I am motivated to propose another go at the "Applicative => Monad" issue. Regardless of the future of that proposal, I think some changes required by it are beneficial in any case, namely ensuring that all Monads defined in GHC get Applicative/Functor instances. I've successfully modified toy version of 7.6.3 to include these instances to see how it goes, and as you can imagine the changes are numerous but simple; I don't assume HEAD won't be much different in that respect. However, since this would by my first involvement in a project as large as GHC, I am not quite sure of the procedure to follow, even after reading the corresponding Trac/Wiki pages. I'm assuming the policy here is the same as for the libraries mailing list (hence this email), and upon positive feedback I can open a Trac issue, take ownership, clone, modify, send pull request etc. If there's anything wrong with that approach here please tell me. Greetings, David [1] http://thread.gmane.org/gmane.comp.lang.haskell.general/19796

2 1

[PATCH] GHC should be able to list all possible command line options (#7843)
by Jan Stolarek 07 May '13

07 May '13

I'm attaching a patch that resolves ticket #7843. It adds new command line option --show-options that displays all GHC command line flags that are longer than two characters. This allows to have auto completion of command line options in bash. Janek

1 0

Re: Per-generation lists of weak pointers
by Akio Takano 06 May '13

06 May '13

On Sat, May 4, 2013 at 8:50 AM, Edward Z. Yang <ezyang(a)cs.stanford.edu>wrote: > Akio, your derefWeak WHITEHOLE fix looks really weird. I don't > know what the right pattern is, but it seems like asking for trouble > when there are multiple concurrent derefs: > > if (info == stg_WHITEHOLE_info) { > ("ptr" info) = ccall lockClosure(w "ptr"); > unlockClosure(w, info); > } > > I don't see what the problem is, could you elaborate? The purpose of the fix was to prevent a sequence like this: - w is a dead weak pointer. - Thread A: finalizeWeak# w. - Thread A: finalizeWeak# calls lockClosure(w), overwriting w->info with stg_WHITEHOLE_info. - Thread B: deRefWeak# w - Thread B: deRefWeak# sees stg_WHITEHOLE_info, and since it's not the same as stg_DEAD_WEAK_info, it thinks w is alive. The problem was that if deRefWeak# saw stg_WHITEHOLE_info, it was not clear whether the weak pointer was alive or not. So my fix adds a call to lockClosure, which never returns stg_WHITEHOLE_info. > > addForeignPtrFinalizer retries in this case. > > This can't be right; a dead weak pointer always stays dead, so won't > this infinite loop? > No. When addForeignPtrFinalizer retries, it will use a new Weak# object, because foreignPtrFinalizer must have been replaced the content of the IORef. > > > I haven't got around to looking at this, but I see Edward is on the case > > with some code review. Do you think I should look at it before it goes > in? > > Now you're asking for it :) I would always be interested in seeing if I > missed anything. > > Cheers, > Edward >

2 2

Re: Per-generation lists of weak pointers
by Akio Takano 06 May '13

06 May '13

Thank you for your comments, I updated the patches on the ticket (comment changes only). http://hackage.haskell.org/trac/ghc/ticket/7847 On Sun, May 5, 2013 at 7:50 PM, Edward Z. Yang <ezyang(a)cs.stanford.edu>wrote: > > The purpose of the fix was to prevent a sequence like this: > > > > - w is a dead weak pointer. > > - Thread A: finalizeWeak# w. > > - Thread A: finalizeWeak# calls lockClosure(w), overwriting w->info with > > stg_WHITEHOLE_info. > > - Thread B: deRefWeak# w > > - Thread B: deRefWeak# sees stg_WHITEHOLE_info, and since it's not the > same > > as stg_DEAD_WEAK_info, it thinks w is alive. > > > > The problem was that if deRefWeak# saw stg_WHITEHOLE_info, it was not > clear > > whether the weak pointer was alive or not. So my fix adds a call to > > lockClosure, which never returns stg_WHITEHOLE_info. > > OK, but you should add a comment about what this code is doing. > Done. > (You will also unnecessarily lock when GET_INFO snags a transient > whitehole), > > > > > addForeignPtrFinalizer retries in this case. > > > > > > This can't be right; a dead weak pointer always stays dead, so won't > > > this infinite loop? > > > > > > > No. When addForeignPtrFinalizer retries, it will use a new Weak# object, > > because foreignPtrFinalizer must have been replaced the content of the > > IORef. > > Sorry, I still don't understand. Tracing the code execution, no change > is made to the IORef when Finalizers is CFinalizers? > When addCFinalizerToWeak fails, some other thread must have called foreignPtrFinalizer, and it must have replaced the content of the IORef before calling finalizeWeak#. I updated the comment to mention this logic. > > Edward >

1 0

GHC plugin without registering the plugin
by Erik de Castro Lopo 06 May '13

06 May '13

Hi all, I'm messing about with writing a GHC plugin and looking at the documentation here: http://www.haskell.org/ghc/docs/7.6.3/html/users_guide/compiler-plugins.html which states: Plugins can be specified on the command line with the option -fplugin=module where module is a module in a registered package that exports a plugin. Is there anyway I can have a plugin in my local source tree rather than being installed and registered? Not having to cabal install it would make development and debugging somewhat easier. Cheers, Erik -- ---------------------------------------------------------------------- Erik de Castro Lopo http://www.mega-nerd.com/

3 2

link problem
by Kazu Yamamoto 04 May '13

04 May '13

Hi, GHC head of today cannot link installed libraries. When I tried to install "lifted-base", the following error occurred: ---- [6 of 6] Compiling Control.Concurrent.Lifted ( Control/Concurrent/Lifted.hs, dist/build/Control/Concurrent/Lifted.o ) /usr/bin/ld: cannot find -lHSmonad-control-0.3.2.1-ghc7.7.20130430 /usr/bin/ld: cannot find -lHStransformers-base-0.4.1-ghc7.7.20130430 /usr/bin/ld: cannot find -lHSbase-unicode-symbols-0.2.2.4-ghc7.7.20130430 collect2: ld returned 1 exit status cabal: Error: some packages failed to install: lifted-base-0.2.0.4 failed during the building phase. The exception was: ExitFailure 1 ---- Each library names followed by "-ghc7.7.20130430". Is this correct? Note that "lifted-base" was able to be installed in a few days ago. --Kazu

1 2

Vaildate fails
by Simon Peyton-Jones 03 May '13

03 May '13

Johan Did you validate before committing -funbox-strict-fields being the default? I'm getting this: Unexpected failures: perf/should_run T4474a [stat not good enough] (normal) perf/should_run T4474b [stat not good enough] (normal) perf/should_run T4474c [stat not good enough] (normal) simplCore/should_compile T7360 [stderr mismatch] (optasm) The last is certainly caused by -funbox-strict-fields being the default. The first three almost certainly are too, since there is a data type data Tree = Leaf !Int | Fork !Tree !Tree deriving Show In all three cases we allocate 25% more: bytes allocated value is too high: Expected bytes allocated: 3766493912 +/-5% Lower bound bytes allocated: 3578169216 Upper bound bytes allocated: 3954818608 Actual bytes allocated: 4831890456 Ironically Trac #4874 is a performance bug that you reported yourself, saying that there is too much unnecessary boxing! Would you like to investigate? Maybe there is another bug to fix? I think it's because "fulllTree" shares all the (I# 1) constants at all the leaves, whereas the strict version makes a tree with a 1# at every leaf, and allocates an I# box for each of them when it flattens the tree. Maybe that is too much a special case to worry about. Now this may be fine, but it's certainly worth noting. And if you decide it's acceptable (and you are the Peformance Tsar after all), you need to adjust the test bounds. Simon

2 2

Re: Per-generation lists of weak pointers
by Simon Marlow 03 May '13

03 May '13

I haven't got around to looking at this, but I see Edward is on the case with some code review. Do you think I should look at it before it goes in? Cheers, Simon On 01/05/13 23:02, Edward Z. Yang wrote: > Thanks for being patient. > > I think there is still a race condition for finalizeWeak and addCFinalizer. > The race goes as follows: > > finalizeWeak: lockClosure > finalizeWeak: unlockClosure - weak pointer is now dead > addCFinalizer: lockClosure > addCFinalizer: weak pointer is already dead! > addCFinalizer: unlockClosure > addCFinalizer: run newest finalizer > finalizeWeak: run list of finalizers > > However, addCFinalizer's finalizer should be run after the list. > > The obvious fix is to keep the closure locked until all the finalizers > are done running, but maybe there is something more clever you can do > to avoid having to have the lock taken out while runnig arbitrary C code. > > Minor style nits: > > - Why isn't the type: runCFinalizers(StgCFinalizerList*)? (Yes, a > stg_NO_FINALIZER_closure is not a StgCFinalizerList, but it should > be treated as a "null" type value; see END_TSO_QUEUE in includes/rts/storage/TSO.h > for an example). > - I don't think finalizer lists will ever be tagged, so it should > not be necessary to untag them. (Tagging wouldn't help, because > the only other "constructor" is stg_NO_FINALIZER_closure, which is > static so we can test for it using a pointer comparison). > - Some of your w != NULL tests are abbreviated to just w; I think we > prefer to have the explicit '!= NULL' afterwards > - You can probably add some ASSERTs for the linked list manipulation > in collectDeadWeakPtrs (e.g. ASSERT(dead_weak_ptrt_list_tail != > NULL) in the latter conditional branch) > - In tidyWeakList, you have 'new_gen = Bdescr((P_)w)->gen;' A brief > comment about what this is doing would be good (it is the primary > semantic change for the algorithm from the old version) > > The rest looks good; after it validates I'll be happy to push. > > Cheers, > Edward > > Excerpts from Akio Takano's message of Thu Apr 25 03:42:10 -0700 2013: >> Thank you for your comments! >> >> On Wed, Apr 24, 2013 at 8:23 PM, Edward Z. Yang <ezyang(a)cs.stanford.edu>wrote: >> >>> Great, good to see. >>> >>> Some questions: >>> >>> - You seem to be stubbing out cfinalizers with stg_WEAK_IS_DEAD_closure >>> when a weak pointer gets finalized; your locking is predicated on >>> operations on this field being cas'es. Another design is to lock >>> the closure itself, via the info table header (using lockClosure). >>> I think both strategies should be sound, but I suspect lockClosure >>> is bound to be safer, in case you end up mutating other fields. >>> >> >> You are right, I updated the patches to use lockClosure. I was not aware of >> the existence of lockClosure when I wrote the previous patches, which is >> why I used cas(). >> >>> - Typo "Kill a weak poitner" >>> >> >> Now the comment is gone. >> >>> - There was a comment about LDV which was removed. Why is that comment >>> irrelevant in the new world order? >>> >> >> In that version of the patches finalizeWeak# didn't modify the infoptr, so >> the comment didn't apply. It is back in the latest version of the patches, >> though. >> >>> >>> Edward >>> >>> Excerpts from Akio Takano's message of Wed Apr 24 02:12:42 -0700 2013: >>>> Thank you very much for your help, I just updated the patches on the >>> ticket. >>>> >>>> On Sun, Apr 21, 2013 at 10:50 AM, Edward Z. Yang <ezyang(a)cs.stanford.edu >>>> wrote: >>>> >>>>> In your ticket, you mention this patch introduces a race condition. >>> One >>>>> possible fix is to have addCFinalizerToWeak# check if the pointer is >>>>> already >>>>> dead, and just run the finalizer immediately if it is. I think this >>>>> preserves the semantics, but this needs to be checked closely. >>>>> >>>>> Edward >>>>> >>>>> Excerpts from Akio Takano's message of Fri Apr 19 02:58:50 -0700 2013: >>>>>> I removed the invariant by adding a new primop, >>> addCFinalizerToWeak#. I >>>>>> opened a ticket for the issue. >>>>>> >>>>>> http://hackage.haskell.org/trac/ghc/ticket/7847 >>>>>> >>>>>> - Akio >>>>>> >>>>>> On Thu, Mar 21, 2013 at 2:40 PM, Simon Marlow <marlowsd(a)gmail.com> >>>>> wrote: >>>>>> >>>>>>> On 11/03/13 03:17, Akio Takano wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I'm working on implementing per-generation lists of weak pointers >>> to >>>>>>>> speed up garbage collection in programs that allocate a lot of >>> weak >>>>>>>> pointers. I have a patch [1] that validates and gives a 3x speed >>> up on >>>>>>>> a benchmark. However I'd like to ask for some advise before >>> finishing >>>>>>>> and submitting the patch. >>>>>>>> >>>>>>>> [1] https://github.com/takano-**akio/ghc/commit/** >>>>>>>> c7345c68eaa1e7f9572e693b5e352e**386df7d680< >>>>> >>> https://github.com/takano-akio/ghc/commit/c7345c68eaa1e7f9572e693b5e352e386… >>>>>> >>>>>>>> >>>>>>>> The problem is that since my patch splits the weak pointer list >>>>>>>> between generations, it no longer maintains the right order of >>> weak >>>>>>>> pointers. This could cause finalizers added with >>>>>>>> addForeignPtrFinalizer to run in the wrong order. >>>>>>>> >>>>>>>> I can think of one way to fix it; to make sure that when a WEAK >>> object >>>>>>>> gets promoted, it is always added to the front of the new list. >>> So my >>>>>>>> questions are: >>>>>>>> >>>>>>>> - Would it be a correct fix? >>>>>>>> - If so, is it an acceptable fix? For example, is it too fragile a >>>>>>>> reasoning to rely on? >>>>>>>> >>>>>>> >>>>>>> I don't like the way that we rely on the ordering of the weak >>> pointer >>>>> list >>>>>>> right now. I think this invariant arose accidentally when the >>> support >>>>> for >>>>>>> C finalizers was added. It was wrong for some time, see: >>>>>>> >>>>>>> http://hackage.haskell.org/**trac/ghc/ticket/7160< >>>>> http://hackage.haskell.org/trac/ghc/ticket/7160> >>>>>>> >>>>>>> and as per my comments in that commit log, I think we should do it >>>>>>> differently. I don't know how hard that would be though. >>>>>>> >>>>>>> Incidentally, I implemented per-generation weak pointers in the >>>>> local-gc >>>>>>> branch, but didn't get around to porting it back over into the >>>>> mainline (I >>>>>>> still have a ToDo for that). My version probably has the ordering >>>>> bug, but >>>>>>> you could always look at the branch to see how my approach >>> compares to >>>>>>> yours. >>>>>>> >>>>>>> Cheers, >>>>>>> Simon >>>>>>> >>>>>>> >>>>> >>>

1 0

list of unexpecteds expected on OS X?
by Nicolas Frisby 03 May '13

03 May '13

I'm switching my GHC development over to a Mac, and so I'm establishing a validation baseline, since I don't expect all tests to pass on my setup. After a ./sync-all pull (as of Johan's ade1ae97), and a ./validate --slow, I see unexpected results for these tests. TEST="T149 T5313 jules_xref launchbury layout008 jl_defaults T7360 T1969 T5631 T3064 T4801 T3294 jules_xref2 ImpSafeOnly08 safePkg01 ImpSafeOnly02 ImpSafeOnly03 ImpSafeOnly01 ImpSafeOnly06 ImpSafeOnly07 ImpSafeOnly04 ImpSafeOnly05 ImpSafeOnly09 ImpSafeOnly10 T4474a T4474c T4474b space_leak_001 conc058 conc068 conc065 conc066 conc020 conc035 conc017 rittri lex T2507 haddock.Cabal T7478 jq_readsPrec IntegerConversionRules integerConstantFolding signals004 T7170 dynamic003 length001 echo001 cgrun071 hSetBuffering004 hSetBuffering002 hClose002 andy_cherry qq007 qq008 recomp006" (See below for more of the summary info.) Question 1: To other GHC developers using OS X: do you recognize these tests as "unexpecteds expected on OS X"? Question 2: Has anyone diagnosed these particular tests? IE is there an OS X tsar? (I am not necessarily volunteering here.) Thanks. ----- T149 became a pass with one my patches about a month ago, but I was seeing nondeterminism and was hoping someone more familiar with the T149 test If I run the testsuite manually (cd testsuite/tests/; make TESTS=<as above>), the "ImpSafeOnly*" tests pass as expected. The 'conc*' tests don't even compile because block and unblock were replaced by mask, but I'm weary of directly changing this because it seems like these should have shown up on all platforms. I've lightly investigated some of the others, and opened a few tickets (#7876 #7877, comment on #7684), but I'm asking here for more information before I flood the ticket system. A common culprit seems to be the IO library's behavior wrt to closed handles. Other tests look to involve some dynamic linking issues. Unexpected passes: driver T5313 (normal,hpc,optasm,ghci,threaded1,threaded2,dyn) perf/should_run T149 (normal) Unexpected failures: ../../libraries/base/tests dynamic003 [bad exit code] (ghci) ../../libraries/base/tests echo001 [bad exit code] (ghci) ../../libraries/base/tests length001 [bad exit code] (ghci) ../../libraries/base/tests/IO hClose002 [bad exit code] (ghci,threaded1,threaded2) ../../libraries/base/tests/IO hSetBuffering002 [bad exit code] (ghci) ../../libraries/base/tests/IO hSetBuffering004 [bad stdout or stderr] (ghci) ../../libraries/unix/tests signals004 [bad exit code] (ghci,threaded1,threaded2) codeGen/should_run cgrun071 [bad exit code] (normal,hpc,optasm,ghci,threaded1,threaded2,dyn,g1) concurrent/should_run conc017 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn) concurrent/should_run conc017 [bad stdout or stderr] (ghci) concurrent/should_run conc020 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn) concurrent/should_run conc020 [bad stdout or stderr] (ghci) concurrent/should_run conc035 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn) concurrent/should_run conc035 [bad stdout or stderr] (ghci) concurrent/should_run conc058 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn) concurrent/should_run conc058 [bad stdout or stderr] (ghci) concurrent/should_run conc065 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn) concurrent/should_run conc066 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn) concurrent/should_run conc068 [bad exit code] (ghci) concurrent/should_run conc068 [exit code non-0] (normal,hpc,optasm,threaded1,dyn) driver T2507 [bad stderr] (normal) driver/recomp006 recomp006 [bad stderr] (normal) ffi/should_run T7170 [bad stdout or stderr] (ghci) ghc-api/T7478 T7478 [bad exit code] (normal) layout layout008 [bad stdout] (normal) lib/integer IntegerConversionRules [bad stderr] (normal) lib/integer integerConstantFolding [bad stderr] (normal) perf/compiler T1969 [stat not good enough] (normal) perf/compiler T3064 [stat not good enough] (normal) perf/compiler T3294 [stderr mismatch] (normal) perf/compiler T4801 [stat too good] (normal) perf/compiler T5631 [stat not good enough] (normal) perf/haddock haddock.Cabal [stat not good enough] (normal) perf/should_run T4474a [stat not good enough] (normal) perf/should_run T4474b [stat not good enough] (normal) perf/should_run T4474c [stat not good enough] (normal) perf/space_leaks space_leak_001 [stat too good] (hpc,optasm,dyn) programs/andy_cherry andy_cherry [bad exit code] (ghci) programs/jl_defaults jl_defaults [bad exit code] (ghci) programs/jq_readsPrec jq_readsPrec [bad exit code] (ghci) programs/jules_xref jules_xref [bad exit code] (ghci) programs/jules_xref2 jules_xref2 [bad exit code] (ghci) programs/launchbury launchbury [bad exit code] (ghci) programs/lex lex [bad exit code] (ghci) programs/rittri rittri [bad exit code] (ghci) quasiquotation/qq007 qq007 [exit code non-0] (normal,hpc,optasm) quasiquotation/qq008 qq008 [exit code non-0] (normal,hpc,optasm) safeHaskell/check/pkg01 ImpSafeOnly01 [exit code non-0] (normal) safeHaskell/check/pkg01 ImpSafeOnly02 [exit code non-0] (normal) safeHaskell/check/pkg01 ImpSafeOnly03 [stderr mismatch] (normal) safeHaskell/check/pkg01 ImpSafeOnly04 [exit code non-0] (normal) safeHaskell/check/pkg01 ImpSafeOnly05 [stderr mismatch] (normal) safeHaskell/check/pkg01 ImpSafeOnly06 [exit code non-0] (normal) safeHaskell/check/pkg01 ImpSafeOnly07 [stderr mismatch] (normal) safeHaskell/check/pkg01 ImpSafeOnly08 [stderr mismatch] (normal) safeHaskell/check/pkg01 ImpSafeOnly09 [stderr mismatch] (normal) safeHaskell/check/pkg01 ImpSafeOnly10 [exit code non-0] (normal) safeHaskell/check/pkg01 safePkg01 [bad exit code] (normal) simplCore/should_compile T7360 [stderr mismatch] (optasm)

3 2

questions about validating in the presence of known failures
by Nicolas Frisby 02 May '13

02 May '13

[You can generalize this email from OS X to other architectures/operating systems.] Question 1: Is a list of "unexpecteds expected on OS X" maintained somewhere? The comments on #5757 seemed like a start, but they're 1.5 years old. The Trac "Custom Query" for OS X isn't helpful for my actual question of "can I ignore these wrt to validating my patch?". A wiki page collating tickets for any "known unexpecteds" for OS X (and other OSs) would be helpful. Question 2: Can we add another bit to unexpected results marking them as known/unknown? That would yield an immediacy for the validation workflow in the cases where we have long-standing known bugs, eg on various platforms. I think #5757 is on the right path, but marking a test that should pass as an expected failure seems disappointing. Is there a traditional workflow for handling this? Question 3: What's the status on the build bot farm? Thanks for your time, everyone.

2 2