September 2013 - ghc-devs

Fixing 8266: How do I get a list of all dist-install/build directories?
by Christiaan Baaij 21 Sep '13

21 Sep '13

Hello everyone, I'm trying to write a patch for #8266. Here's what I have so far: https://gist.github.com/christiaanb/6648192 It changes the name of the dynamic library to a relative path instead of an absolute one. However, now I'm getting an error while building the DPH packages (probably due to the use of TH): Loading package vector-0.9.1 ... <command line>: can't load .so/.DLL for: /Users/christiaan/Documents/devel/ghc/libraries/vector/dist-install/build/libHSvector-0.9.1-ghc7.7.20130918.dylib (dlopen(/Users/christiaan/Documents/devel/ghc/libraries/vector/dist-install/build/libHSvector-0.9.1-ghc7.7.20130918.dylib, 9): Library not loaded: @loader_path/../primitive-0.4.0.1/libHSprimitive-0.4.0.1-ghc7.7.20130918.dylib Referenced from: /Users/christiaan/Documents/devel/ghc/libraries/vector/dist-install/build/libHSvector-0.9.1-ghc7.7.20130918.dylib Reason: image not found) Of course the problem is, is that the packages are not installed yet, so the relative paths don't make sense yet. My solution is to update rules/shell-wrapper.mk, and to add DYLD_FALLBACK_LIBRARY_PATH = <ALL_LIBRARY_BUILD_DIRS> to the inplace/bin shell wrappers. So that during building of ghc, dynlibs of all the packages can still be resolved to the place where they were build. So my question is: how do I get the list <ALL_LIBRARY_BUILD_DIRS> within the build system? Cheers, Christiaan

1 0

Re: llvm calling convention matters
by Carter Schonwald 20 Sep '13

20 Sep '13

hey all, first let me preface by saying I am in favor of breaking and updating/modernizing the GHC ABI. I just think that for a number of reasons, it doesn't make sense to do it for the 7.8 release, but rather start work on it in another month or so, so we can systematically have a better set of ABI, and keep all the code gens are first class citizens. (also work out the type system changes need to be able to correctly use SIMD shuffles, which are currently inexpressible correctly with GHC's type system. Simd Shuffles are crucial for interesting levels of SIMD performance!) the reason I don't want to make the ABI change right now is because then we'd have to wait until after llvm 3.4 gets released in like 6 months before giving them another breaking change! (OR start baking a LLVM into GHC, which is a leap we're not 100% on, though theres clear good reasons for why! ). Basically, if we make breaking changes to the ABI now (and thus have split ABI for llvm 3.4HEAD vs earlier), and then we do fixups or more breakage for 7.10, then when 7.10 rolls around (perhaps late next spring or sometime in the summer, perhaps?), the only supported llvm version for 7.10 would be LLVM HEAD / 3.5 (which won't be released till some time thereafter)! Unless we go ahead and break the 3.4 ABI to 7.10 rather than 7.8 abi (whatever that would entai, which would ). This is assuming the ~ 7-8 months between major version releases cycle that LLVM has done of late additionally, as Johan remarked today on a pending patch of mine, having operations only work on the llvm backend, and not on the native code gen is pretty problematical! see http://ghc.haskell.org/trac/ghc/ticket/8256 tl;dr : Unless we're throwing away native code gen backend next month, we probably want to actually not increase their capability gap / current ABI incompatibility right before 7.8 release. I am willing to help explore modernizing the native code gens so that they have parity with the llvm backends. Additionally, boxing ourselves in a corner where for 7.10 the only llvm with the right ABI will be llvm 3.5 seems totally unacceptable from an end users / distribution package managers standpoint, and a huge support headache for the community. I've had to help deal with the support headache of the xcode5 clang + ghc issues on OS X, A LOT, in the past 2 months, I'm not keen on deliberately creating similar support disasters for myself and others. that said: I absolutely agree that we should fix up the ABI, have a clear story for XMM, YMM, and ZMM registers, and if you've been following trac tickets at all, you'll see theres even a type system issue in properly handling the SIMD shuffles! i briefly sketch out the issue in http://ghc.haskell.org/trac/ghc/ticket/8107 (last comment) that said: i'm open to being convinced i'm wrong, and I absolutely understand your motivations for wanting it now, but I really believe that doing so right now will create a number of problems that are better off evaded to begin with cheers -Carter On Wed, Sep 11, 2013 at 5:49 PM, Geoffrey Mainland <mainland(a)cs.drexel.edu>wrote: > Hi Carter, > > On 09/06/2013 03:24 PM, Carter Tazio Schonwald wrote: > > Hey Geoff, > > > I'm leary about doing a calling convention change right before the ghc > > release (and I"m happy to elaborate more on the phone some time) 1) > > I'd rather we test the patches on llvm locally ourselves before going > > upstream 2) doing that AVX change on the calling convention now, would > > make it harder to make a more systematic exploration of calling > > convention changes post 7.8 release, because we would face either > > breaking the llvm head/3.4 changes, or having to wait till the next > > llvm release cycle (3.5?!) to upstream any more systematic > > changes. (such as adding substantially more SIMD registers to the GHC > > calling convention!) > > > > I understand your likely motivation for wanting the calling convention > > landing in the 7.8 release, namely it may eke an easy 2x perf boost in > > your stream fusion libs, i just worry that the change would ultimately > > cut off our ability to do more aggressive experimentation and > > improvements (eg more simd registers!) for ghc 7.10 over the next > > year? > > > > on an unrelated note: I will spend some time this weekend given you > > the various simd operations I want / think are valuable. the low > > hanging fruit would be figuring out a good haskell type / analogue of > > the llvm __builtin_shuffle(a,b,c) primop, because that usually should > > generate decent code. I'll work out the details of this and some other > > examples and send it your way in the next few days > > > > -Carter > > Currently, on x86-64 we pass floats, doubles, and 128-bit wide SIMD > vectors in xmm1-xmm6. I propose that we change the calling conventions > to pass 256-bit wide SIMD vectors in ymm1-ymm6 and 512-bit wide SIMD > vectors in zmm1-zmm6. I don't know why GHC doesn't use xmm0 or xmm7, as > the Linux C calling convention uses xmm0-xmm7. Simon, perhaps you know > why? I get that we only needed 6 registers originally, F1-F4, D1-D2), > but why count from one rather than zero? > > On x86-32, we pass floats, double, and all SIMD vectors on the stack. I > propose that we pass 128-bit wide SIMD vectors in xmm0-xmm2, and make > analogous arrangements for 256- and 512-bit SIMD vectors. We will still > pass floats and doubles on the stack. This matches the Linux x86 32-bit > C calling convention. > > I think these are fairly conservative changes. I also don't think we > should be afraid of revising the calling convention for GHC 7.10. Surely > the LLVM folks won't be upset if we send them one set of patches a year > instead of one set of patches every two years. > > Geoff > >

5 22

AMP (#8004) almost finished, review would be nice
by David Luposchainsky 20 Sep '13

20 Sep '13

Hey ghc-devs, I finished #8004. The patch introduces a new compiler flag, "-fwarn-amp", that is on by default. The changes are pretty local, affecting TcRnDriver.lhs, PrelNames.lhs and DynFlags.hs, and you can view it here: https://github.com/quchen/ghc/compare/amp_warnings?expand=1 (Note that that's my very dirts personal branch, It needs some squashing to make the actual patch out of this; also note that it includes coloured warning output so I could debug better. Other than that it's the real thing though.) If you have the time I'd like someone who knows his way around GHC to take a look at it so I don't break stuff unexpectedly :-) A couple of issues remain: 1. Validation does not work due to the warnings issued. Since "sh validate" uses -Werror, an AMP warning will stop the compilation before tests can even be run. Unfortunately, the build system seems to use the variables 'GhcStage1HcOpts' in the build process of phase 1, which is of course done with the current (7.6.3 or whatever is installed) compiler. When adding "-fno-warn-amp" to that variable phase 1 won't build because the parameter is unknown to the old compiler. Is there some sort of -"ignore next parameter if unknown" hack, or is there a smart solution? 2. Temporarily removing the -Werror constraint from validate-settings.mk (or by using custom-settings.mk) makes the validation build go through. However, the testsuite defines a couple of violating modules, therefore there is unexpected STDERR output, hence a handful of tests fail. Should a fix for this be included in the AMP patch, or be done as a separate instance? 3. Similarly, GHC defines around 50 offending modules, creating warnings in the standard build process. Again, should this be included, or can we push that to after the feature freeze and regard it as bugfixing? Greetings, David/quchen

3 9

Testsuite with profiling
by Jan Stolarek 19 Sep '13

19 Sep '13

I need to run testsuite with profiling turned on. When I run make WAY=normal EXTRA_HC_OPTS="-prof -fprof-auto -rtsopts" lots of tests are failing with: Dynamic linking required, but this is a non-standard build (eg. prof). You need to build the program twice: once the dynamic way, and then in the desired way using -osuf to set the object file suffix. I only added "GhcLibWays += p" to "quick" section of my build.mk. I guess this error message has something to do with DYNAMIC_BY_DEFAULT - I could probably get this right with a trial and error approach, but I'd rather know what I'm doing. Can anyone tell me what is going on and what do I need to do to make this work? Janek

2 1

primitive using PrimWrappers rather than PrimOpWrappers?
by Carter Schonwald 19 Sep '13

19 Sep '13

hey all, i just rebased my prefetch patch onto HEAD, and now i'm working on getting ghc to build looks like Data.Primitive.Types is trying to import PrimWrappers rather than PrimOpWrappers? i patched that up in my local build, but that seems to be missing, unless i'm overlooking somthing and i'm botching the submodules?

3 5

HEADS UP: New primops in HEAD
by Jan Stolarek 19 Sep '13

19 Sep '13

New comparison primops are now in HEAD. This change breaks backwards compatibility for comparison primops. What you need to do after pulling the latest HEAD is: 1) Upgrade to latest versions of Alex and Happy (cabal update && cabal install alex happy). 2) Rebase your branches on top of HEAD, if these branches were in a repository affected by primop change. Repositories that will were affected: GHC, testsuite, array, base, dph, ghc-prim, integer-gmp, integer-simple, primitive, template-haskell. 3) Blow away your build. Be sure to run ./configure to pick up new versions of Alex and Happy Janek

2 3

Overly verbose output from testsuite
by Jan Stolarek 18 Sep '13

18 Sep '13

I am getting very lengthy output at the end of testsuite run: =====> posix014(normal) 3788 of 3795 [0, 1, 0] cd ../../libraries/unix/tests/libposix && '/5playpen/t-jastol/ghc-validate-2/bindisttest/install dir/bin/ghc' -fforce-recomp -dcore-lint -dcmm-lint -dno-debug-output -no-us er-package-db -rtsopts -fno-ghci-history -o posix014 posix014.hs >posix014.comp.stderr 2>&1 cd ../../libraries/unix/tests && ./signals004 </dev/null >signals004.run.stdout 2>signals004.run.stderr cd ../../libraries/unix/tests && ./processGroup001 </dev/null >processGroup001.run.stdout 2>processGroup001.run.stderr cd ../../libraries/unix/tests && ./T3816 </dev/null >T3816.run.stdout 2>T3816.run.stderr cd ../../libraries/unix/tests && ./T1185 </dev/null >T1185.run.stdout 2>T1185.run.stderr cd ../../libraries/unix/tests && ./executeFile001 </dev/null >executeFile001.run.stdout 2>executeFile001.run.stderr cd ../../libraries/unix/tests && ./fileStatusByteString </dev/null >fileStatusByteString.run.stdout 2>fileStatusByteString.run.stderr cd ../../libraries/unix/tests && ./processGroup002 </dev/null >processGroup002.run.stdout 2>processGroup002.run.stderr cd ../../libraries/unix/tests/libposix && ./posix002 </dev/null >posix002.run.stdout 2>posix002.run.stderr cd ../../libraries/unix/tests && ./fileStatus </dev/null >fileStatus.run.stdout 2>fileStatus.run.stderr cd ../../libraries/unix/tests && ./T8108 </dev/null >T8108.run.stdout 2>T8108.run.stderr cd ../../libraries/unix/tests/libposix && ./posix003 </dev/null >posix003.run.stdout 2>posix003.run.stderr cd ../../libraries/unix/tests/libposix && ./posix005 </dev/null >posix005.run.stdout 2>posix005.run.stderr cd ../../libraries/unix/tests/libposix && ./posix004 </dev/null >posix004.run.stdout 2>posix004.run.stderr cd ../../libraries/unix/tests/libposix && ./posix009 </dev/null >posix009.run.stdout 2>posix009.run.stderr cd ../../libraries/unix/tests/libposix && ./posix006 </dev/null >posix006.run.stdout 2>posix006.run.stderr cd ../../libraries/unix/tests/libposix && ./posix010 </dev/null >posix010.run.stdout 2>posix010.run.stderr cd ../../libraries/unix/tests/libposix && ./posix014 </dev/null >posix014.run.stdout 2>posix014.run.stderr Timeout happened...killing process... Wrong exit code (expected 0 , actual 99 ) Stdout: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 6 1 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 11 4 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 20 1 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 28 .... 667 3668 3669 3670 3671 3672 3673 3674 3675 3676 3677 3678 3679 3680 3681 3682 3683 3684 3685 3686 3687 3688 3689 3690 3691 3692 3693 3694 3695 3696 3697 3698 3699 3700 3701 3702 3703 3704 3705 3706 3707 Stderr: =====> T7815(threaded1) 3795 of 3795 [0, 1, 0] cd ./rts && '/5playpen/t-jastol/ghc-validate-2/bindisttest/install dir/bin/ghc' -fforce-recomp -dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-gh ci-history -o T7815 T7815.hs -threaded -debug >T7815.comp.stderr 2>&1 cd ./rts && ./T7815 50000 +RTS -N2 -RTS </dev/null >T7815.run.stdout 2>T7815.run.stderr I don't know which test causes this, but this wasn't happening 2 or 3 days ago. Anyone knows how to silence this output? It is quite annoying. Janek

3 2

potential for controlling memory alignment?
by Nicolas Frisby 18 Sep '13

18 Sep '13

On my laptop, a program showed a consistent slowdown with -fdicts-strict I didn't find any obvious causes in the Core differences, so I turned to Intel's Performance Counter Monitor for measurements. After trying a few counters, I eventuall saw that there are about an order of magnitude more misaligned memory loads with -fdicts-strict than without, so I think that may be a significant part of the slowdown. I'm not sure if these are code or data reads. Can anyone suggest how to validate this hypothesis about misaligned reads? A subsequent commit has changed the behavior I was seeing, so I'm not interested in alternatives means to determine if -fdicts-strict is somehow at fault — I'm just asking specifically about data/code memory alignment in GHC and how to diagnose/experiment with it. Thanks.

3 3

Suggestion for resolving the Cabal/GHC dependency problems
by Duncan Coutts 18 Sep '13

18 Sep '13

All, I was discussing this with Yuri earlier and I had an idea that I think may resolve our problems. Firstly, what are the problems: 1. ghc devs and users grumble because the ghc library depends on Cabal, making it hard to use the ghc lib with a later Cabal. 2. ghc devs grumble generally that Cabal seems quite big but they only need small parts of it 3. Cabal devs complain that they cannot add useful dependencies (like a parser with error messages) because ghc depends on Cabal. Secondly, let us recall why it is that ghc does use Cabal, and where: 1. it's used by ghc-pkg to read/write the external representation of installed package files (external rep is defined by Cabal spec, and implemented in the Cabal lib) 2. it's used by ghc to read the ghc package database files/dirs. These databases use the same external representation, and ghc & ghc-pkg use the InstalledPackageInfo type internally (InstalledPackageInfo is defined in the Cabal lib). 3. it's used by the ghc build system to help with building all the libraries that ship with ghc. I believe that this part uses more of the build system part of Cabal, not just the types and external formats. 4. ghc comes with Cabal pre-installed so that users can run Setup.hs scripts to install other packages. This was part of the original Cabal design: that all compilers would use the installed package info format defined by Cabal, and all compilers would ship Cabal to users so the Setup.hs mechanism will work. Now, as far as I know, nobody is suggesting that ghc stop shipping Cabal, nor that it stop using it as part of the build system. The problems all centre around use number 2, where the ghc library package depends on Cabal. Number 1 isn't really a problem because ghc-pkg is an executable. So my suggestion is quite simple, eliminate the dependency in case 2 above, but keep it in the other three cases. Specifically: * ghc will use a new internal type to represent info coming from the ghc-pkg databases, ie not InstalledPackageInfo. This can be smaller as ghc doesn't care about the metadata. * The InstalledPackageInfo and the current need for ghc to read its external representation is the main reason the ghc lib depends on Cabal. Other dependencies should be minor and easy to remove. * ghc and ghc-pkg will agree on a new on-disk representation of the installed package info. * ghc-pkg will continue to depend on Cabal, it will continue to use the types and parsers defined by Cabal to read/write the InstalledPackageInfo. It will translate from InstalledPackageInfo into the on-disk representation that ghc & ghc-pkg share. So what might the on-disk representation for the ghc-pkg databases look like? Currently they use the external format of InstalledPackageInfo because this is convenient using Cabal. One simple option is just to store both formats for all packages. Another option would be that ghc never reads package dbs where the cache is out of date. Then it only ever reads the cache and never has to look at the other files. In principle the cache should never be out of date: there are two options for updating the db, calling ghc-pkg, or putting the file directly and calling ghc-pkg recache (distros often use the latter as it is simpler for them). In either case the db cache will be up to date. (In fact calling it a cache is not really correct.) So this is a better solution than the one previously proposed to split out some small part of Cabal, because in this proposal, ghc doesn't depend on Cabal at all, not even some smaller common lib. It's also better from the point of view of the Cabal folks because it does not involve splitting Cabal in unnatural ways. The Cabal folks do want to split the Cabal lib, but not in a way that is especially helpful to ghc. This suggestion is orthogonal to any Cabal lib splits. Further, if only ghc-pkg and the ghc build system depend on Cabal, then it is easier for Cabal to add more dependencies, since they do not have to be installed with ghc (due to the ghc lib depending on them). In particular the Cabal folks would like to use a proper parser and have suggested adding dependencies on parsec, mtl and transformers. If only ghc-pkg depends on Cabal, then these dependencies only need to be used at build time, and do not have to be installed (which also means they don't have to be kept quite so up to date). Note that this would not address SPJ's complaint that the start of building ghc involves building 60+ modules of Cabal. The ghc-cabal tool still uses Cabal and I am not suggesting changing that now. It's plausible that when the Cabal lib is split that the ghc-cabal tool could depend on just the smaller of the two (someone would need to look at how much functionality from the "Simple" build system it uses). I don't see that this is a big priority however. Duncan

4 4

AMP failure
by Simon Peyton-Jones 18 Sep '13

18 Sep '13

I think my tree is up to date but I’m getting this. Any ideas? simon "inplace/bin/ghc-stage1" -hisuf hi -osuf o -hcsuf hc -static -H32m -O -Werror -Wall -H64m -O0 -package-name xhtml-3000.2.1 -hide-all-packages -i -ilibraries/xhtml/. -ilibraries/xhtml/dist-install/build -ilibraries/xhtml/dist-install/build/autogen -Ilibraries/xhtml/dist-install/build -Ilibraries/xhtml/dist-install/build/autogen -Ilibraries/xhtml/. -optP-include -optPlibraries/xhtml/dist-install/build/autogen/cabal_macros.h -package base-4.7.0.0 -Wall -XHaskell98 -XCPP -O2 -O -dcore-lint -fno-warn-amp -fno-warn-deprecated-flags -Wwarn -ticky -no-user-package-db -rtsopts -odir libraries/xhtml/dist-install/build -hidir libraries/xhtml/dist-install/build -stubdir libraries/xhtml/dist-install/build -split-objs -dynamic-too -c libraries/xhtml/./Text/XHtml/Frameset/Elements.hs -o libraries/xhtml/dist-install/build/Text/XHtml/Frameset/Elements.o -dyno libraries/xhtml/dist-install/build/Text/XHtml/Frameset/Elements.dyn_o "inplace/bin/ghc-stage1" -hisuf dyn_hi -osuf dyn_o -hcsuf dyn_hc -fPIC -dynamic -H32m -O -Werror -Wall -H64m -O0 -hide-all-packages -i -iutils/hsc2hs/. -iutils/hsc2hs/dist-install/build -iutils/hsc2hs/dist-install/build/autogen -Iutils/hsc2hs/dist-install/build -Iutils/hsc2hs/dist-install/build/autogen -optP-include -optPutils/hsc2hs/dist-install/build/autogen/cabal_macros.h -package base-4.7.0.0 -package containers-0.5.3.1 -package directory-1.2.0.1 -package filepath-1.3.0.2 -package process-1.2.0.0 -XHaskell98 -XCPP -XForeignFunctionInterface -no-user-package-db -rtsopts -odir utils/hsc2hs/dist-install/build -hidir utils/hsc2hs/dist-install/build -stubdir utils/hsc2hs/dist-install/build -c utils/hsc2hs/./HSCParser.hs -o utils/hsc2hs/dist-install/build/HSCParser.dyn_o utils/hsc2hs/HSCParser.hs:24:10: Warning: ‛Parser’ is an instance of Monad but not Applicative - this will become an error in GHC 7.10, under the Applicative-Monad Proposal. utils/hsc2hs/HSCParser.hs:36:10: Warning: ‛Parser’ is an instance of MonadPlus but not Alternative - this will become an error in GHC 7.10, under the Applicative-Monad Proposal. <no location info>: Failing due to -Werror. make[1]: *** [utils/hsc2hs/dist-install/build/HSCParser.dyn_o] Error 1 make[1]: *** Waiting for unfinished jobs.... make: *** [all] Error 2 simonpj@cam-05-unx:~/5builds/HEAD$ Microsoft Research Limited (company number 03369488) is registered in England and Wales Registered office is at 21 Station Road, Cambridge, CB1 2FB

3 3