November 2018 - ghc-devs

Considerations for Control flow hint implementation.
by Andreas Klebinger 01 Dec '18

01 Dec '18

Hello Devs, I've started thinking about the implementation of https://github.com/ghc-proposals/ghc-proposals/pull/182 recently. (Add control flow hint pragmas.) For this purpose I've rebased D4327 "WIP: Add likelyhood to alternatives from stg onwards" which already does a lot of the work at the Cmm/Stg level. The issue I ask you for feedback now is how to best attach branch weights to case alternatives in core. My prefered approach would be to expand core data types to include them unconditionally. While this is quite far reaching in the amount of code it touches it would be rather straight forward to implement: Alternative 1: Putting the weights directly into the case alternative tuple: + It it's trivial to check which places manipulate case alternatives as they will initially fail to compile. + It's very mechanical, almost all use sites won't actually change the weight. + It's easy to keep this working going forward as any new optimizations can't "forget" they have to consider them. - It will introduce a cost in compiler performance. - New optimization who don't have to care about branchweights still have to at least pipe them through. - While syntactically heavy in terms of real complexity it's a simply approach. Alternative 2: Putting the weights into the case constructor. + Might give better compiler performance as I expect us to rebuild cases less often than alternatives. - Seems kind of clunky. - Weaker coupling between case alternatives and their weights. Or we could use ticks: + There is some machinery already there + Can be turned off for -O0 + Can be ignored when convenient. - Can be ignored when convenient. - Very weak coupling between case alternatives and their weights. - The existing machinery doesn't exactly match the needs of this. - We would have to extend tick semantics to a degree where complexity might grow too large for me to successfully implement this. - If new optimizations end up just removing these ticks because they are allowed to then the whole exercise becomes rather pointless. - Makes it harder to ensure all relevant code paths in GHC are actually updated. In particular there is currently no tick category which can stick to case alternatives but just get's removed in case it get's in the way of optimizations. The closest match is SoftScope which allows ticks to be floated up, something that could impact performance quite badly in this case. As then we might float something intended to mark a branch as unlikely into another branch that is actually along the hot path. I think the core variant(s) mostly stand and fall with the actual compile time impact. For -O0 the impact would be negligible as the compile time is already dominated by codegen and typechecking. For the rest it's hard to say. So I'm looking for feedback on this. Maybe you have other suggestions I haven't considered? How much compile time cost increase would be acceptable for what kind of performance boost? Cheers, Andreas Klebinger

2 2

Writing a simple Core evaluator, having trouble with name lookups
by Christopher Done 01 Dec '18

01 Dec '18

Hi all, I'm attempting to make a simple evaluator for GHC core, but I'm not clear on how to reliably looking up names. I'm compiling each of ghc-prim, integer-simple and base with a patched version of GHC which performs an extra output step with the Core AST to a file for each module. Later, I load those files in. So for an input Haskell file like this: module Main (main,Foo(..)) where class Foo a where foo :: a -> Int instance Foo Int where foo x = x * x instance Foo Char where foo x = 99 main = print (foo (123 :: Int)) I have an output set of bindings like this: https://gist.github.com/chrisdone/cb05a77d3fcb081a4580b5f85289674a One thing that I immediately notice is that the names of things are completely non-unique, especially in generated names. So here are two implementations of the method "foo" for the class "Foo": ( Id {idStableName = "main:Main:$cfoo", idUnique = Unique 6989586621679010917}, ...) -- Int ( Id {idStableName = "main:Main:$cfoo", idUnique = Unique 6989586621679010923}, ...) -- Char So e.g. the instance for "Foo Int" refers to the above method implementation via its Unique (6989586621679010923): ( Id {idStableName = "main:Main:$fFooInt", idUnique = Unique 8214565720323784705} , CastE (VarE (Id { idStableName = "main:Main:$cfoo" , idUnique = Unique 6989586621679010923 <---- HERE }))) At first, I thought I would use the Unique associated with every Name to make a lookup. This is completely reliable within one GHC compilation, but I've read in the docs that it's not stable across multiple invocations? What does that mean for my case? Another thing I notice is that type-class methods are not generated at the core level. I have, for example, this method call that provides it the instance dictionary, (AppE (AppE (VarE (Id { idStableName = "main:Main:foo" , idUnique = Unique 8214565720323784707 <---- MISSING })) (TypE (Typ "Int"))) (VarE (Id { idStableName = "main:Main:$fFooInt" , idUnique = Unique 8214565720323784705 }))) But the "main:Main:foo" (8214565720323784707) is not produced in the CoreProgram, it seems. My compile step is very simple: compile :: GHC.GhcMonad m => GHC.ModSummary -> m [CoreSyn.Bind GHC.Var] compile modSummary = do parsedModule <- GHC.parseModule modSummary typecheckedModule <- GHC.typecheckModule parsedModule desugared <- GHC.desugarModule typecheckedModule let binds = GHC.mg_binds (GHC.dm_core_module desugared) pure binds It simply gets the bindings and that's all from the ModGuts. mg_binds :: !CoreProgram Two questions: 1) How do I recognize class methods when I see one, like the "main:Main:foo" above? Maybe this? isClassOpId_maybe :: Id -> Maybe Class Is an "op" what GHC calls type-class methods? 2) If I compile e.g. ghc-prim and that generates a binding Name with ID 123, and then I compile base -- will the ID 123 be re-used by base for something else, or will any reference to 123 in the compiled Names for base refer ONLY to that one in ghc-prim? In other words, when GHC loads the iface for ghc-prim, does it generate a fresh set of names for everything in ghc-prim, or does it load them from file? Cheers!

4 11

make slow test suite failures (at least on OSX ) for 8.6 branch
by Carter Schonwald 30 Nov '18

30 Nov '18

to reproduce cd testsuite make slow TEST="EtaExpandLevPoly ManyAlternatives ManyConstructors MultiLayerModules T10370 T11535 T12707 T13379 T13701 T13719 T14683 T14697 T14936 T15349 T4334 T6132 T7702 T9203 T9630 ghci063 haddock.Cabal haddock.base haddock.compiler hpc_fork signals004 space_leak_001" -------------------- Unexpected results from: TEST="EtaExpandLevPoly ManyAlternatives ManyConstructors MultiLayerModules T10370 T11535 T12707 T13379 T13701 T13719 T14683 T14697 T14936 T15349 T4334 T6132 T7702 T9203 T9630 ghci063 haddock.Cabal haddock.base haddock.compiler hpc_fork signals004 space_leak_001" SUMMARY for test run started at Thu Nov 29 21:03:34 2018 EST 0:55:09 spent to go through 26 total tests, which gave rise to 137 test cases, of which 20 were skipped 0 had missing libraries 63 expected passes 0 expected failures 4 caused framework failures 0 caused framework warnings 6 unexpected passes 6 unexpected failures 38 unexpected stat failures Unexpected passes: runghc/T6132.run T6132 [unexpected] (normal) runghc/T6132.run T6132 [unexpected] (hpc) runghc/T6132.run T6132 [unexpected] (optasm) runghc/T6132.run T6132 [unexpected] (profasm) typecheck/should_run/EtaExpandLevPoly.run EtaExpandLevPoly [unexpected] (profasm) typecheck/should_run/EtaExpandLevPoly.run EtaExpandLevPoly [unexpected] (profthreaded) Unexpected failures: deriving/should_run/T11535.run T11535 [bad exit code] (ghci) ghci/scripts/ghci063.run ghci063 [bad stderr] (ghci) simplCore/should_compile/T7702.run T7702 [exit code non-0] (profasm) ../../libraries/base/tests/T15349.run T15349 [bad exit code] (ghci) ../../libraries/hpc/tests/fork/hpc_fork.run hpc_fork [bad heap profile] (profasm) ../../libraries/unix/tests/signals004.run signals004 [bad exit code] (threaded2) Unexpected stat failures: perf/compiler/T10370.run T10370 [stat not good enough] (optasm) perf/compiler/T12707.run T12707 [stat not good enough] (hpc) perf/compiler/T12707.run T12707 [stat not good enough] (optasm) perf/compiler/T12707.run T12707 [stat not good enough] (profasm) perf/compiler/T13379.run T13379 [stat not good enough] (hpc) perf/compiler/MultiLayerModules.run MultiLayerModules [stat not good enough] (optasm) perf/compiler/ManyConstructors.run ManyConstructors [stat not good enough] (hpc) perf/compiler/ManyConstructors.run ManyConstructors [stat not good enough]

1 0

perf.haskell.org functional again
by Joachim Breitner 30 Nov '18

30 Nov '18

Hi, after a system upgrade to avoid weird linker errors, and after some fixes in the nofib submodule, http://perf.haskell.org/ghc is running again. I am collecting instruction counts instead of runtime, because the latter was just too often varying too wildly. I hope this will yield less false alarms. I am also running nofib with mode=fast. This way, building GHC, running the testsuite and nofib takes a bit over one hour. I hope this can keep up with y'all's commits (when it took 2h it couldn't). Now nothing of the setup requires a quiet dedicated machine, so if there is a need, we could move these builds into the cloud or into some CI infrastructure - but no changes are immediately planned. Enjoy, Joachim

5 11

Windows test failures
by Simon Peyton Jones 30 Nov '18

30 Nov '18

Ben, Phyx, and other CI folk 'sh validate -fast' still gives a lot of failures on Window. The output is below. I would be so good to get this to zero! (another minor thing is that it would be great to eliminate the long path at the beginning - this is platform independent - I have to delete manually many times each day. Thanks Simon Unexpected passes: /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/rts/T9405.run T9405 [unexpected] (normal) Unexpected failures: /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/driver/T14452.run T14452 [bad stdout] (normal) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/ghci/linking/ghcilink001.run ghcilink001 [bad exit code] (normal) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/ghci/prog010/ghci.prog010.run ghci.prog010 [bad exit code] (ghci) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/ghci/scripts/ghci050.run ghci050 [bad exit code] (ghci) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/ghci/should_run/T14963a.run T14963a [bad exit code] (ghci) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/ghci.debugger/scripts/print012.run print012 [bad exit code] (ghci) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/ghci.debugger/scripts/print016.run print016 [bad exit code] (ghci) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/ghci.debugger/scripts/break019.run break019 [bad exit code] (ghci) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/ghci.debugger/scripts/break028.run break028 [bad exit code] (ghci) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/ghci.debugger/scripts/dynbrk002.run dynbrk002 [bad exit code] (ghci) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/ghci.debugger/scripts/T2740.run T2740 [bad exit code] (ghci) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/indexed-types/should_fail/T7102a.run T7102a [bad exit code] (ghci) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/plugins/plugin-recomp-change.run plugin-recomp-change [bad exit code] (normal) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/rts/T10672/T10672_x64.run T10672_x64 [bad exit code] (normal) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/th/TH_spliceGuard.run TH_spliceGuard [exit code non-0] (normal) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/th/TH_overlaps.run TH_overlaps [exit code non-0] (normal) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/th/T5886.run T5886 [exit code non-0] (normal) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/th/T10697_decided_3.run T10697_decided_3 [exit code non-0] (normal) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/th/T11629.run T11629 [exit code non-0] (normal) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/libraries/base/tests/T4006.run T4006 [bad stdout] (normal) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/libraries/base/tests/IO/environment001.run environment001 [bad stdout] (normal) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/ghci/should_run/T15633b.run T15633b [bad exit code] (ghci) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/plugins/plugins07.run plugins07 [bad exit code] (normal) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/plugins/plugins09.run plugins09 [bad stdout] (normal) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/plugins/plugins10.run plugins10 [bad stdout] (normal) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/plugins/plugins11.run plugins11 [bad stdout] (normal) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/plugins/T10420.run T10420 [bad exit code] (normal) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/plugins/plugin-recomp-pure.run plugin-recomp-pure [bad exit code] (normal) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/plugins/plugin-recomp-impure.run plugin-recomp-impure [bad exit code] (normal) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/plugins/plugin-recomp-flags.run plugin-recomp-flags [bad exit code] (normal) Framework failures: /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/perf/compiler/MultiLayerModules.run MultiLayerModules [runTest] (Unhandled exception during cleanup: Unable to remove folder '/c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/perf/compiler/MultiLayerModules.run': [Errno 13] Permission denied: '/c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/perf/compiler/MultiLayerModules.run' Unable to start current test.) /c/Users/simonpj/AppData/Local/Temp/ghctest-i30xogs3/test spaces/libraries/base/tests/IO/T3307.run T3307 [normal] ('latin-1' codec can't encode characters in position 24-25: ordinal not in range(256))

3 2

Kind inference
by Simon Peyton Jones 29 Nov '18

29 Nov '18

I have been working for longer than I care to admit on a significant refactoring of kind inference. I've just pushed it. It's a pretty big patch, which Richard and I have been discussing for some weeks. It validates, and fixes at least nine troublesome open tickets. But of course I could have made a mistake. Be vigilant. My: there will almost certainly be conflicts with your visible kind application patch, especially in tcInferApps. But that function has gotten substantially simpler in my patch so I don't think it'll be hard to fix things up. Phew! Simon

1 0

llvm way only tests?
by Gabor Greif 29 Nov '18

29 Nov '18

Hi all, what is the magic incantation for running tests only when the llvm tools (e.g. `opt`) are around? I am currently declaring the test as test('T15155l', [ only_ways(llvm_ways), ], run_command, ['$MAKE -s --no-print-directory T15155l']) but it won't be included neither in the optasm builds (which is okay) nor in the llvm builds (gets skipped unexpetedly) I looked at the other test which are declared similarly, and they behave just like mine. What I am doing wrong? Thanks, Gabor

2 2

condensed summary of vanilla mojave ghc 8.6.2 errors with clang
by Carter Schonwald 27 Nov '18

27 Nov '18

the vast majority of the errors i previously shared seem to be "can't find atom for stabs BNSYM at 0000AF80 in /Users/carter/dev-checkouts/ghc-tree/ghc-8.6.X-checkout-clang-build/libraries/base/dist-install/build/libHSbase-4.12.0.0.a(Internals.o)" which corresponds with this code in the apple linker https://github.com/llvm-mirror/lld/blob/master/lib/ReaderWriter/MachO/MachO… i dont know to what to make of this

1 2

Running IO function from RTS
by lonetiger＠gmail.com 26 Nov '18

26 Nov '18

Hi all, I’m looking for a way to initiate a call from the RTS into Haskell (base function) running and evaluating an IO function using the non-threaded RTS. For the threaded RTS I can use rts_evalIO with the RTS capability, but for the non-threaded I can’t find a way that doesn’t trigger the if (cap->in_haskell) { errorBelch("schedule: re-entered unsafely.\n" " Perhaps a 'foreign import unsafe' should be 'safe'?"); stg_exit(EXIT_FAILURE); } Clause in the scheduler. Taking a look at how the non-threaded I/O manager currently works on Windows, it seems that re-uses the TSO from the blocked thread doing the I/O call to give notifications back. This approach doesn’t work for me since the new I/O manager doesn’t block doing a foreign call boundary, instead It blocks in Haskell on an MVar, and I only want to wake up very specific threads. In short, does anyone know of a way to initialize an I/O operation from the non-threaded RTS? Thanks, Tamar

1 0

git stats
by Simon Peyton Jones 26 Nov '18

26 Nov '18

David I’m still getting all these fatal errors when making a single test. Plus it takes several seconds for them all to happen. Any thoughts about supressing this stuff? I know you made some progress here. Thanks Simon simonpj@cam-05-unx:~/5builds/HEAD-4/testsuite/tests/typecheck/should_compile$ make stage=1 TEST=T13871 PYTHON="python3" "python3" ../../../driver/runtests.py -e "ghc_compiler_always_flags='-dcore-lint -dcmm-lint -no-user-package-db -rtsopts -fno-warn-missed-specialisations -fshow-warning-groups -fdiagnostics-color=never -fno-diagnostics-show-caret -Werror=compat -dno-debug-output'" -e config.compiler_debugged=True -e ghc_with_native_codegen=True -e config.have_vanilla=True -e config.have_dynamic=True -e config.have_profiling=False -e ghc_with_threaded_rts=True -e ghc_with_dynamic_rts=True -e config.have_interp=False -e config.unregisterised=False -e config.have_gdb=False -e config.have_readelf=True -e config.ghc_dynamic_by_default=False -e config.ghc_dynamic=False -e ghc_with_smp=True -e ghc_with_llvm=False -e windows=False -e darwin=False -e config.in_tree_compiler=True -e config.cleanup=True -e config.local=True --rootdir=. --config-file=../../../config/ghc -e 'config.platform="x86_64-unknown-linux"' -e 'config.os="linux"' -e 'config.arch="x86_64"' -e 'config.wordsize="64"' -e 'config.timeout=int() or config.timeout' -e 'config.exeext=""' -e 'config.top="/5playpen/simonpj/HEAD-4/testsuite"' --config 'compiler="/5playpen/simonpj/HEAD-4/inplace/test spaces/ghc-stage1"' --config 'ghc_pkg="/5playpen/simonpj/HEAD-4/inplace/test spaces/ghc-pkg"' --config 'haddock=' --config 'hp2ps="/5playpen/simonpj/HEAD-4/inplace/test spaces/hp2ps"' --config 'hpc="/5playpen/simonpj/HEAD-4/inplace/test spaces/hpc"' --config 'gs="gs"' --config 'timeout_prog="../../../timeout/install-inplace/bin/timeout"' -e "config.stage=1" \ --only=T13871 \ \ \ \ \ \ Timeout is 300 fatal: Not a git repository (or any parent up to mount point /5playpen) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). Failed to get allowed metric changes from the HEAD git commit message. 0 Found 1 .T files... Beginning test run at Mon Nov 26 16:26:18 2018 GMT ====> Scanning ./all.T =====> T13871(normal) 1 of 1 [0, 0, 0] cd "T13871.run" && "/5playpen/simonpj/HEAD-4/inplace/test spaces/ghc-stage1" -c T13871.hs -dcore-lint -dcmm-lint -no-user-package-db -rtsopts -fno-warn-missed-specialisations -fshow-warning-groups -fdiagnostics-color=never -fno-diagnostics-show-caret -Werror=compat -dno-debug-output -fno-warn-incomplete-patterns SUMMARY for test run started at Mon Nov 26 16:26:18 2018 GMT 0:00:01 spent to go through 1 total tests, which gave rise to 3 test cases, of which 2 were skipped 0 had missing libraries 1 expected passes 0 expected failures 0 caused framework failures 0 caused framework warnings 0 unexpected passes 0 unexpected failures 0 unexpected stat failures Appending 0 stats to git notes. fatal: Not a git repository (or any parent up to mount point /5playpen) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). fatal: Not a git repository (or any parent up to mount point /5playpen) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). fatal: Not a git repository (or any parent up to mount point /5playpen) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). fatal: Not a git repository (or any parent up to mount point /5playpen) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). fatal: Not a git repository (or any parent up to mount point /5playpen) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). An error occured while writing the performance metrics to git notes. This is usually due to a lock-file existing somewhere in the git repo. simonpj@cam-05-unx:~/5builds/HEAD-4/testsuite/tests/typecheck/should_compile$

1 0