- ghc-devs - Haskell.org

Reconfigure required for head.
by Phyx 31 Mar '18

31 Mar '18

Hi all, I have just committed 4de585a which lifts the MAX_PATH restrictions for Haskell programs compiled with GHC 8.6 for Windows users. However when you update, you *MUST* run configure again. This counts for ALL platforms otherwise your build will suffer a not so friendly failure. If you're using Hadrian to build, you must manually update the submodule to HEAD of master as the Hadrian submodule hasn't been bumped yet. Thanks, Tamar

1 0

Phabricator new behavior regarding submitting patches for reviews
by Ömer Sinan Ağacan 30 Mar '18

30 Mar '18

Hi, One of the changes with the recent Phabricator update is that we can no longer submit a patch for reviews until the build bot successfully builds it. I can't even ping people in the comment section until the patch builds. It says: > These changes have not finished building yet and may have build failures. This > revision is currently a draft. You can leave comments, but no one will be > notified until the revision is submitted for review. The "submit" button now says "submit quietly". This is really annoying because - It takes several days for build bot to build a patch (I have a patch that has been in the queue for 3 days now and it's still counting) - I can validate a patch on my laptop in an hour. (slow validate takes about 2-3 hours) Previously I could get approvals, and then test locally and push. Now I can't do that unless I email people about the patch. - In the previous version I could submit an incomplete patch for comments, now I can't do that because no one will be notified and there's no way to ping people to explicitly draw attention (again unless I email people). So if possible (without downgrading it) could we bring back the old behavior (perhaps there's a setting about this?) Thanks, Ömer

2 4

Re: Memory usage exploding for complex pattern matching
by Victor Miraldo (UU) 30 Mar '18

30 Mar '18

Hello, >>>> I have just tried compiling my code with 8.4.2 and using >>>> -fmax-pmcheck-iterations=0, >>>> I gave GHC 12GB of ram and it still ran out (through `ulimit >>>> -v$((1024*1024*12))`). >>>> >>> Hmmm, I'm a bit confused. Why are our results so different? How >>> precisely are you invoking GHC? >> >> Here I meant my whole code, not just the repro. I could have been more clear. >> Nevertheless, I'm calling it through stack: >> > I'll admit that I am a bit lost; Minimal.hs compiles for me with a > maximum residency of ~3.5 GBytes with both -O1 and the PM check enabled > using GHC 8.4.1. Is this not the repro you are referring to? I get the same behavior as you for Minimal.hs. The "my code" above referred to the whole library that I'm developping. In fact, the Minimal.hs file contains a distilled version of that library with a template haskell splice that we are trying to use in one of our fully fledged examples. Do you want me to produce a larger "NotSoMinimal.hs" closer to the real thing? Cheers, Victor

2 1

Re: Memory usage exploding for complex pattern matching
by Victor Miraldo (UU) 30 Mar '18

30 Mar '18

Hello, >> I have just tried compiling my code with 8.4.2 and using >> -fmax-pmcheck-iterations=0, >> I gave GHC 12GB of ram and it still ran out (through `ulimit >> -v$((1024*1024*12))`). >> > Hmmm, I'm a bit confused. Why are our results so different? How > precisely are you invoking GHC? Here I meant my whole code, not just the repro. I could have been more clear. Nevertheless, I'm calling it through stack: stack build --ghc-options="-fmax-pmcheck-iterations=0" I have just ran the repro with and without "-fmax-pmcheck-iterations=0" and got similar results to yours. > Let's start with a fresh ticket; #11195 has a rather long history. Do > you think you could open it and attach your repro? Sure thing, will do it tomorrow! Cheers, Victor

2 1

Re: Memory usage exploding for complex pattern matching
by Victor Miraldo (UU) 29 Mar '18

29 Mar '18

Hello, @Ben > I tried 8.4 and indeed the renamer/typechecker takes dramatically less > memory (yay!), leaving desugar as the largest allocator (allocating > around 3GB) while compiling the minimal example. Furthermore, disabling > the pattern match checker (via -fmax-pmcheck-iterations=0) dramatically > reduces desugarer allocations (to around 60MByte). It looks like you are > right. I have just tried compiling my code with 8.4.2 and using -fmax-pmcheck-iterations=0, I gave GHC 12GB of ram and it still ran out (through `ulimit -v$((1024*1024*12))`). I saw many warnings about the pattern match checker exceeding the iterations I specified: warning: Pattern match checker exceeded (0) iterations in a case alternative. (Use -fmax-pmcheck-iterations=n to set the maximun number of iterations to n) Which makes me wonder, is the option really stopping the checker or just displaying something when it exceeds the given number? > Unfortunately, it's not clear what can be done to work around this short > of > disabling the match checker for the entire module, as suggested by > mpickering. Ideally we would have some notion of local warning flags > (e.g. #602), allowing us to disable this (rather expensive) warning for > generated code. Sadly, we aren't there yet. Perhaps you would like to > propose such a mechanism? You guys are much more indicated to propose mechanisms to fix this. If you think this is a sensible mechanism to implement, sure! I'd be happy to propose it. @Simon > Let's make a Trac ticket for this (since there is a nice small reproducer). Or, if it seems a dup, attach the reproducer to some existing ticket. Should I be the one creating it or adding the reproducer? The closest I've seen is: https://ghc.haskell.org/trac/ghc/ticket/11195 > Maybe we should revert the (recent) patch that makes TH code behave like source code, until this pattern-match check business is solved. Let me know when this happens, I'll be happy to test my code again! Thanks! Cheers, Victor

2 1

Re: Memory usage exploding for complex pattern matching
by Victor Miraldo (UU) 29 Mar '18

29 Mar '18

Hello Ben, >> We came up with a "minimal" example that at least is self contained and >> triggers the problem; `runghc Minimal.hs` should finish with about 5GB of RAM. >> > Hmm, with 8.2 I'm seeing a peak memory usage of 3.5 GB, with that peak > occurring during renaming, not desugaring (when exhaustiveness checking > happens). Are you certain that the exhaustiveness checker is the > responsible party? Not at all! In fact, I ran some comparisons with "-ddump-phases" and plotted those. I'm attaching the graphs here. From base to "added E" we added 16 more pattern matches, then each "added" phase is an additional 16 pattern matches. The simplifier stands out there. I couldn't run a bigger case as I had hit the ceiling of my machine. Cheers, Victor

3 2

Re: Memory usage exploding for complex pattern matching
by Victor Miraldo (UU) 29 Mar '18

29 Mar '18

Hello! > You can turn off the pattern match checker, is that an option? How? > Can you post your code which shows the poor behaviour. It could be > good to profile that part of GHC to see if there are any easy gains > but it is known to be slow. I attached a minimal example in the original e-mail. The code that triggers the really bad case is not much bigger. It is divided in many more modules, though. Cheers, Victor

1 0

Memory usage exploding for complex pattern matching
by Victor Miraldo (UU) 29 Mar '18

29 Mar '18

Dear ghc-devs, We are working on a library and came across quite high memory consumption when compiling some code (generated by TH). This memory issue seems to come from the exhaustiveness checker as we found in: https://ghc.haskell.org/trac/ghc/ticket/11163 https://ghc.haskell.org/trac/ghc/ticket/11195 We came up with a "minimal" example that at least is self contained and triggers the problem; `runghc Minimal.hs` should finish with about 5GB of RAM. Our actual non-minimal scenario runs out of memory on a 16GB RAM machine. We tried with ghc 8.4.2, 8.0.2 and 7.10.3 and ghc-HEAD; all of those showed a similar problem, although the older ghc's used less memory. Do you have any suggestions to make our code finish compiling without running out of memory? Thanks!! Victor

3 2

Question about indirectees of BLACKHOLE closures
by Ömer Sinan Ağacan 29 Mar '18

29 Mar '18

Hi, I've been looking at BLACKHOLE closures and how the indirectee field is used and I have a few questions: Looking at evacuate for BLACKHOLE closures: case BLACKHOLE: { StgClosure *r; const StgInfoTable *i; r = ((StgInd*)q)->indirectee; if (GET_CLOSURE_TAG(r) == 0) { i = r->header.info; if (IS_FORWARDING_PTR(i)) { r = (StgClosure *)UN_FORWARDING_PTR(i); i = r->header.info; } if (i == &stg_TSO_info || i == &stg_WHITEHOLE_info || i == &stg_BLOCKING_QUEUE_CLEAN_info || i == &stg_BLOCKING_QUEUE_DIRTY_info) { copy(p,info,q,sizeofW(StgInd),gen_no); return; } ASSERT(i != &stg_IND_info); } q = r; *p = r; goto loop; } It seems like indirectee can be a TSO, WHITEHOLE, BLOCKING_QUEUE_CLEAN, BLOCKING_QUEUE_DIRTY, and it can't be IND. I'm wondering what does it mean for a BLACKHOLE to point to a - TSO - WHITEHOLE - BLOCKING_QUEUE_CLEAN - BLOCKING_QUEUE_DIRTY Is this documented somewhere or otherwise could someone give a few pointers on where to look in the code? Secondly, I also looked at the BLACKHOLE entry code, and it seems like it has a different assumption about what can indirectee field point to: INFO_TABLE(stg_BLACKHOLE,1,0,BLACKHOLE,"BLACKHOLE","BLACKHOLE") (P_ node) { W_ r, info, owner, bd; P_ p, bq, msg; TICK_ENT_DYN_IND(); /* tick */ retry: p = StgInd_indirectee(node); if (GETTAG(p) != 0) { return (p); } info = StgHeader_info(p); if (info == stg_IND_info) { // This could happen, if e.g. we got a BLOCKING_QUEUE that has // just been replaced with an IND by another thread in // wakeBlockingQueue(). goto retry; } if (info == stg_TSO_info || info == stg_BLOCKING_QUEUE_CLEAN_info || info == stg_BLOCKING_QUEUE_DIRTY_info) { ("ptr" msg) = ccall allocate(MyCapability() "ptr", BYTES_TO_WDS(SIZEOF_MessageBlackHole)); SET_HDR(msg, stg_MSG_BLACKHOLE_info, CCS_SYSTEM); MessageBlackHole_tso(msg) = CurrentTSO; MessageBlackHole_bh(msg) = node; (r) = ccall messageBlackHole(MyCapability() "ptr", msg "ptr"); if (r == 0) { goto retry; } else { StgTSO_why_blocked(CurrentTSO) = BlockedOnBlackHole::I16; StgTSO_block_info(CurrentTSO) = msg; jump stg_block_blackhole(node); } } else { ENTER(p); } } The difference is, when the tag of indirectee is 0, evacuate assumes that indirectee can't point to an IND, but BLACKHOLE entry code thinks it's possible and there's even a comment about why. (I don't understand the comment yet) I'm wondering if this code is correct, and why. Again any pointers would be appreciated. Thanks, Ömer

5 9

Cabal trouble
by Simon Peyton Jones 28 Mar '18

28 Mar '18

Trying to build the repro case in Trac #14208, I get this cabal install --with-ghc=/home/simonpj/5builds/ghc-8.4-branch/inplace/bin/ghc-stage2 Warning: cannot determine version of /home/simonpj/.cabal/bin/c2hs : "" Resolving dependencies... cabal: Could not resolve dependencies: trying: exceptions-0.8.3 (dependency of streamly) next goal: template-haskell (dependency of exceptions) rejecting: template-haskell-2.13.0.0/installed-2.1... (conflict: exceptions => template-haskell>=2.2 && <2.13) rejecting: template-haskell-2.12.0.0, template-haskell-2.11.1.0, template-haskell-2.11.0.0, template-haskell-2.10.0.0, template-haskell-2.9.0.0, template-haskell-2.8.0.0, template-haskell-2.7.0.0, template-haskell-2.6.0.0, template-haskell-2.5.0.0, template-haskell-2.4.0.1, template-haskell-2.4.0.0, template-haskell-2.3.0.1, template-haskell-2.3.0.0, template-haskell-2.2.0.0 (constraint from non-upgradeable package requires installed instance) After searching the rest of the dependency tree exhaustively, these were the goals I've had most trouble fulfilling: template-haskell, exceptions, streamly, transformers, base This is with my in-place build of GHC-8.4 (in-place but not modified in any way). I've done a 'cabal upate'. What am I doing wrong? Thanks Simon

2 1