- ghc-devs - Haskell.org

T7050 failing
by Richard Eisenberg 02 Apr '18

02 Apr '18

Hi Simon, Your recent "unpack datacons" patch (https://phabricator.haskell.org/rGHC9187d5fb1d3d38a4e607b0d61784c21447c8195b <https://phabricator.haskell.org/rGHC9187d5fb1d3d38a4e607b0d61784c21447c8195b>) causes typecheck/should_compile/T7050 to loop. This was harder to discover because I had broken validation previously... Richard

1 0

Testsuite failures
by Simon Peyton Jones 02 Apr '18

02 Apr '18

Ben I'm getting these unexpected failures on my Linux box (64 bit) Unexpected failures: /tmp/ghctest-g01j3x6z/test spaces/./cabal/ghcpkg05.run ghcpkg05 [bad stderr] (normal) Unexpected stat failures: /tmp/ghctest-g01j3x6z/test spaces/./perf/compiler/T5837.run T5837 [stat too good] (normal) /tmp/ghctest-g01j3x6z/test spaces/./perf/compiler/T5321FD.run T5321FD [stat too good] (normal) I would LOVE it if the testsuite framework didn't print those long prefix paths! I'd prefer Unexpected failures: cabal/ghcpkg05.run ghcpkg05 [bad stderr] (normal) Unexpected stat failures: perf/compiler/T5837.run T5837 [stat too good] (normal) perf/compiler/T5321FD.run T5321FD [stat too good] (normal) The stat failures look like this: =====> T5321FD(normal) 1 of 1 [0, 0, 0] cd "./T5321FD.run" && "/5playpen/simonpj/HEAD-1/inplace/test spaces/ghc-stage2" -c T5321FD.hs -no-user-package-db -rtsopts -fno-warn-missed-specialisations -fshow-warning-groups -fdiagnostics-color=never -fno-diagnostics-show-caret -dno-debug-output +RTS -V0 -tT5321FD.comp.stats --machine-readable -RTS bytes allocated value is too low: (If this is because you have improved GHC, please update the test so that GHC doesn't regress again) Expected T5321FD(normal) bytes allocated: 415136648 +/-10% Lower bound T5321FD(normal) bytes allocated: 373622983 Upper bound T5321FD(normal) bytes allocated: 456650313 Actual T5321FD(normal) bytes allocated: 373272448 Deviation T5321FD(normal) bytes allocated: -10.1 % *** unexpected stat test failure for T5321FD(normal) =====> T5837(normal) 1 of 1 [0, 0, 0] cd "./T5837.run" && "/5playpen/simonpj/HEAD-1/inplace/test spaces/ghc-stage2" -c T5837.hs -no-user-package-db -rtsopts -fno-warn-missed-specialisations -fshow-warning-groups -fdiagnostics-color=never -fno-diagnostics-show-caret -dno-debug-output -freduction-depth=50 +RTS -V0 -tT5837.comp.stats --machine-readable -RTS bytes allocated value is too low: (If this is because you have improved GHC, please update the test so that GHC doesn't regress again) Expected T5837(normal) bytes allocated: 55813608 +/-7% Lower bound T5837(normal) bytes allocated: 51906655 Upper bound T5837(normal) bytes allocated: 59720561 Actual T5837(normal) bytes allocated: 51558808 Deviation T5837(normal) bytes allocated: -7.6 % *** unexpected stat test failure for T5837(normal) In both cases, just a bit past the threshold. What is odd is that the latter doesn't fail on our CI system (I asked you this before). The former is new, maybe to do with Richard's recent patches. Question: are the numbers 'centred' for the CI infrastructure. Maybe they are just under 10% and 7% resp, and something local pushes them a tiny bit lower? The ghcpkg05 failure is quite consistent, despite from-clean rebuild. =====> ghcpkg05(normal) 1 of 1 [0, 0, 0] cd "./ghcpkg05.run" && $MAKE -s --no-print-directory ghcpkg05 Actual stderr output differs from expected: diff -uw "./ghcpkg05.run/ghcpkg05.stderr.normalised" "./ghcpkg05.run/ghcpkg05.run.stderr.normalised" --- ./ghcpkg05.run/ghcpkg05.stderr.normalised 2018-04-02 11:52:51.407023104 +0100 +++ ./ghcpkg05.run/ghcpkg05.run.stderr.normalised 2018-04-02 11:52:51.407023104 +0100 @@ -10,6 +10,13 @@ cannot find any of ["C/D.hi","C/D.p_hi","C/D.dyn_hi"] cannot find any of ["C/E.hi","C/E.p_hi","C/E.dyn_hi"] cannot find any of ["libtestpkg-2.0-XXX.a","libtestpkg-2.0-XXX.p_a","libtestpkg-2.0-XXX-ghc<VERSION>.so","libtestpkg-2.0-XXX-ghc<VERSION>.dylib","testpkg-2.0-XXX-ghc<VERSION>.dll"] on library path +Warning: include-dirs: /5playpen/simonpj/HEAD-1/compiler/stage2/build/utils doesn't exist or isn't a directory +Warning: include-dirs: /5playpen/simonpj/HEAD-1/compiler/stage2/build/../rts/dist/build doesn't exist or isn't a directory +Warning: include-dirs: /5playpen/simonpj/HEAD-1/compiler/stage2/build/stage2 doesn't exist or isn't a directory +Warning: include-dirs: /5playpen/simonpj/HEAD-1/libraries/haskeline/dist-install/build/includes doesn't exist or isn't a directory +Warning: include-dirs: /5playpen/simonpj/HEAD-1/libraries/text/dist-install/build/include doesn't exist or isn't a directory +Warning: include-dirs: /5playpen/simonpj/HEAD-1/libraries/containers/dist-install/build/include doesn't exist or isn't a directory +Warning: include-dirs: /5playpen/simonpj/HEAD-1/libraries/bytestring/dist-install/build/include doesn't exist or isn't a directory The following packages are broken, either because they have a problem listed above, or because they depend on a broken package. *** unexpected failure for ghcpkg05(normal) Any ideas? Thanks SImon

1 0

Haskell Colonectomy
by John Leo 01 Apr '18

01 Apr '18

A serious proposal: https://github.com/halfaya/ghc-proposals/blob/master/proposals/0000-colonec… John

2 1

Proposal: Professionalizing GHC Development
by Gershom B 01 Apr '18

01 Apr '18

Fellow Haskellers, Recently there has been much work into creating a better and more professional GHC development process, including in the form of DevOps infrastructure, scheduled releases and governance, etc. But much remains to be done. There continues to be concern about the lack of use of industry-standard tools. For example, GHC development is tied to Phabricator, which is a custom product originally developed for in-house use by an obscure startup. GHC development is documented on a wiki still -- ancient technology, not appropriate for 2018. Wiki syntax for documentation needs to be replaced by the only modern standard -- github flavored markdown. Trac itself is ancient technology, dating to 2003, well before anybody knew how to program real software. It provides no support for all the most important aspects of software development -- Kanban boards, sprint management, or even burndown charts. What is necessary is an integrated solution that holistically addresses all aspects of development, fostering a DevOps culture, embracing cloud-first, agile-first, test-first, disrupt-first principles, and with an ironclad SLA. Rather than homegrown solutions, we need a GHC development process that utilizes tools and procedures already familiar to regular developers. Cross-sectional feature comparison analysis yields a clear front-runner -- Visual Studio Team Services. VSTS is a recognized Leader in the Gartner Magic Quadrant for Enterprise Agile Planning tools. It lets us migrate from custom git hosting to a more reliable source control system -- Team Foundation Version Control. By enforcing the locking of checked-out files, we can prevent the sorts of overlap between different patches that occur in the current distributed version management system, and coordinate tightly between developers, enabling and fostering T-shaped skills. Team Build also lets us migrate from antiquated makefiles to modern, industry-standard technology -- XML descriptions of build processes that integrate automatically with tracking of PBIs (product backlog items), and one-button release management. In terms of documentation, rather than deal with the subtleties of different markdown implementations and the confusing world of restructured text, we can utilize the full power of Word, including SharePoint integration as well as Office 365 capabilities, and integration with Microsoft Teams, the chat-based workspace for collaboration. This enables much more effective cross-team collaboration with product and marketing divisions. One of the most exciting features of VSTS is powerful extensibility, with APIs offered in both major programming paradigms in use today -- JVM and .NET. The core organizational principle for full application lifecycle management is a single data construct -- the "work item" which documentation informs us "represents a thing," which can be anything that "a user can imagine." The power of work items comes through their extensible XML representation. Work items are combined into a Process Template, with such powerful Process Templates available as Agile, Scrum, and CMMI. VSTS will also allow us to analyze GHC Developer team performance with an integrated reporting data warehouse that uses a cube. Pricing for up to 100 users is $750 a month. Individual developers can also purchase subscriptions to Visual Studio Professional for $45 a month. I suggest we start directing resources towards a transition. I imagine all work to accomplish this could be done within a year, and by next April 1, the GHC development process will be almost unrecognizable from that today. Regards, Gershom

5 4

Reconfigure required for head.
by Phyx 31 Mar '18

31 Mar '18

Hi all, I have just committed 4de585a which lifts the MAX_PATH restrictions for Haskell programs compiled with GHC 8.6 for Windows users. However when you update, you *MUST* run configure again. This counts for ALL platforms otherwise your build will suffer a not so friendly failure. If you're using Hadrian to build, you must manually update the submodule to HEAD of master as the Hadrian submodule hasn't been bumped yet. Thanks, Tamar

1 0

Phabricator new behavior regarding submitting patches for reviews
by Ömer Sinan Ağacan 30 Mar '18

30 Mar '18

Hi, One of the changes with the recent Phabricator update is that we can no longer submit a patch for reviews until the build bot successfully builds it. I can't even ping people in the comment section until the patch builds. It says: > These changes have not finished building yet and may have build failures. This > revision is currently a draft. You can leave comments, but no one will be > notified until the revision is submitted for review. The "submit" button now says "submit quietly". This is really annoying because - It takes several days for build bot to build a patch (I have a patch that has been in the queue for 3 days now and it's still counting) - I can validate a patch on my laptop in an hour. (slow validate takes about 2-3 hours) Previously I could get approvals, and then test locally and push. Now I can't do that unless I email people about the patch. - In the previous version I could submit an incomplete patch for comments, now I can't do that because no one will be notified and there's no way to ping people to explicitly draw attention (again unless I email people). So if possible (without downgrading it) could we bring back the old behavior (perhaps there's a setting about this?) Thanks, Ömer

2 4

Re: Memory usage exploding for complex pattern matching
by Victor Miraldo (UU) 30 Mar '18

30 Mar '18

Hello, >>>> I have just tried compiling my code with 8.4.2 and using >>>> -fmax-pmcheck-iterations=0, >>>> I gave GHC 12GB of ram and it still ran out (through `ulimit >>>> -v$((1024*1024*12))`). >>>> >>> Hmmm, I'm a bit confused. Why are our results so different? How >>> precisely are you invoking GHC? >> >> Here I meant my whole code, not just the repro. I could have been more clear. >> Nevertheless, I'm calling it through stack: >> > I'll admit that I am a bit lost; Minimal.hs compiles for me with a > maximum residency of ~3.5 GBytes with both -O1 and the PM check enabled > using GHC 8.4.1. Is this not the repro you are referring to? I get the same behavior as you for Minimal.hs. The "my code" above referred to the whole library that I'm developping. In fact, the Minimal.hs file contains a distilled version of that library with a template haskell splice that we are trying to use in one of our fully fledged examples. Do you want me to produce a larger "NotSoMinimal.hs" closer to the real thing? Cheers, Victor

2 1

Re: Memory usage exploding for complex pattern matching
by Victor Miraldo (UU) 30 Mar '18

30 Mar '18

Hello, >> I have just tried compiling my code with 8.4.2 and using >> -fmax-pmcheck-iterations=0, >> I gave GHC 12GB of ram and it still ran out (through `ulimit >> -v$((1024*1024*12))`). >> > Hmmm, I'm a bit confused. Why are our results so different? How > precisely are you invoking GHC? Here I meant my whole code, not just the repro. I could have been more clear. Nevertheless, I'm calling it through stack: stack build --ghc-options="-fmax-pmcheck-iterations=0" I have just ran the repro with and without "-fmax-pmcheck-iterations=0" and got similar results to yours. > Let's start with a fresh ticket; #11195 has a rather long history. Do > you think you could open it and attach your repro? Sure thing, will do it tomorrow! Cheers, Victor

2 1

Re: Memory usage exploding for complex pattern matching
by Victor Miraldo (UU) 29 Mar '18

29 Mar '18

Hello, @Ben > I tried 8.4 and indeed the renamer/typechecker takes dramatically less > memory (yay!), leaving desugar as the largest allocator (allocating > around 3GB) while compiling the minimal example. Furthermore, disabling > the pattern match checker (via -fmax-pmcheck-iterations=0) dramatically > reduces desugarer allocations (to around 60MByte). It looks like you are > right. I have just tried compiling my code with 8.4.2 and using -fmax-pmcheck-iterations=0, I gave GHC 12GB of ram and it still ran out (through `ulimit -v$((1024*1024*12))`). I saw many warnings about the pattern match checker exceeding the iterations I specified: warning: Pattern match checker exceeded (0) iterations in a case alternative. (Use -fmax-pmcheck-iterations=n to set the maximun number of iterations to n) Which makes me wonder, is the option really stopping the checker or just displaying something when it exceeds the given number? > Unfortunately, it's not clear what can be done to work around this short > of > disabling the match checker for the entire module, as suggested by > mpickering. Ideally we would have some notion of local warning flags > (e.g. #602), allowing us to disable this (rather expensive) warning for > generated code. Sadly, we aren't there yet. Perhaps you would like to > propose such a mechanism? You guys are much more indicated to propose mechanisms to fix this. If you think this is a sensible mechanism to implement, sure! I'd be happy to propose it. @Simon > Let's make a Trac ticket for this (since there is a nice small reproducer). Or, if it seems a dup, attach the reproducer to some existing ticket. Should I be the one creating it or adding the reproducer? The closest I've seen is: https://ghc.haskell.org/trac/ghc/ticket/11195 > Maybe we should revert the (recent) patch that makes TH code behave like source code, until this pattern-match check business is solved. Let me know when this happens, I'll be happy to test my code again! Thanks! Cheers, Victor

2 1

Re: Memory usage exploding for complex pattern matching
by Victor Miraldo (UU) 29 Mar '18

29 Mar '18

Hello Ben, >> We came up with a "minimal" example that at least is self contained and >> triggers the problem; `runghc Minimal.hs` should finish with about 5GB of RAM. >> > Hmm, with 8.2 I'm seeing a peak memory usage of 3.5 GB, with that peak > occurring during renaming, not desugaring (when exhaustiveness checking > happens). Are you certain that the exhaustiveness checker is the > responsible party? Not at all! In fact, I ran some comparisons with "-ddump-phases" and plotted those. I'm attaching the graphs here. From base to "added E" we added 16 more pattern matches, then each "added" phase is an additional 16 pattern matches. The simplifier stands out there. I couldn't run a bigger case as I had hit the ceiling of my machine. Cheers, Victor

3 2