March 2019 - ghc-devs

Newtypes in STG
by Christopher Done 01 Apr '19

01 Apr '19

Hi all, Could you offer some insight into newtypes at the STG level? Here’s the context: 1. I’m working on this interpreter for STG (https://github.com/chrisdone/prana) and I’m trying to generate a pristine AST where all names in it are resolved to something known to me. 2. I’ve compiled ghc-prim and integer-gmp without issue, and I’m compiling base and there remains one last frontier which is newtypes. These are the culprits pointed out if I compile base: chris@precision:~/Work/chrisdone/prana/ghc-8.4/libraries/base-4.11.1.0$ PRANA_INDEX=../prana.idx stack build --exec './Setup build --ghc-options=-O0' --file-watch Preprocessing library for base-4.11.1.0.. Building library for base-4.11.1.0.. [1 of 244] Compiling GHC.Base [2 of 244] Compiling GHC.IO [3 of 244] Compiling GHC.Real [4 of 244] Compiling Data.Semigroup.Internal ... [snip] ... [242 of 244] Compiling Data.Functor.Compose [243 of 244] Compiling Data.Fixed [244 of 244] Compiling Data.Complex Errors in Data.Foldable: Variable name not found: base:Data.Semigroup.Internal.All Variable name not found: base:Data.Semigroup.Internal.Any Errors in Foreign.Marshal.Pool: Variable name not found: base:Foreign.Marshal.Pool.Pool Errors in GHC.ExecutionStack.Internal: Variable name not found: base:GHC.ExecutionStack.Internal.StackTrace Errors in Data.Bifoldable: Variable name not found: base:Data.Semigroup.Internal.All Variable name not found: base:Data.Semigroup.Internal.Any Errors in System.Timeout: Variable name not found: base:System.Timeout.Timeout Errors in Data.Data: Variable name not found: base:Foreign.Ptr.WordPtr Variable name not found: base:Foreign.Ptr.IntPtr Variable name not found: base:Data.Semigroup.Internal.Any Variable name not found: base:Data.Semigroup.Internal.All I looked these up, and they all appear to be *uses* of a newtype. For example, in the Timeout function: http://hackage.haskell.org/package/base-4.12.0.0/docs/src/System.Timeout.ht… I printed out the full [StgTopBinding] list and got: let { sat_s27NC [Occ=Once] :: IO Timeout [LclId] = [] \u [] fmap $fFunctorIO Timeout newUnique; } in >>= $fMonadIO sat_s27NC sat_s27NT; Oddly (or not?), they’re used as values, not constructors. This error comes from this part of my code: StgSyn.StgApp occ arguments -> AppExpr <$> lookupSomeVarId occ <*> traverse fromStgGenArg arguments If it was used as a constructor, it’d appear in this position, where looking up the ID would produce a “Data constructor name not found” error: StgSyn.StgConApp dataCon arguments types -> ConAppExpr <$> lookupDataConId dataCon <*> traverse fromStgGenArg arguments <*> pure (map (const Type) types) My understanding of newtypes at this stage is hazy. It seems like: - They ought to be erased and replaced with coercions by now. If they’re not replaced, it’s because they’re in a not-quite-id position like fmap Timeout .... (Arguably these could be fixed in base with a Data.Coerce.coerce?) - However, the CoreTidy/PrepPgm processing modules don’t seem to have removed or replaced these, or introduced a binding that would do something. At this stage what would you recommend? At this point type-checking is done, and I only care about interpreting the code. So I suppose they could be id for all it matters to the interpreter? I imagine they aren’t actually supposed to allocate something here. And I’m certain that any pattern matching on a newtype is also erased away by this point. Cheers! Chris

3 4

GitHub Mirror is broken
by Artem Pelenitsyn 31 Mar '19

31 Mar '19

Hello devs, This is just to let you know that the latestes commit on GitHub ghc/ghc repo dates back to 22th of January. Personally, I find GitHub mirror quite useful for ocasional searches over the code base. Therefore, I'd appreciated repairing the mirror. -- Best of luck, Artem Pelenitsyn

5 5

gitlab.haskell.org is down
by Shayne Fletcher 29 Mar '19

29 Mar '19

Hi, It seems gitlab.haskell.org has been down since around 17:30 GMT. Outage? -- Shayne Fletcher Language Engineer c: +1 917 699 7763 e: shayne.fletcher(a)daml.com Digital Asset Holdings, LLC 4 World Trade Center 150 Greenwich Street, 47th Floor <https://maps.google.com/?q=150+Greenwich+Street,+47th+Floor%C2%A0+%C2%A0+%C…> New York, NY 10007, USA <https://maps.google.com/?q=150+Greenwich+Street,+47th+Floor%C2%A0+%C2%A0+%C…> digitalasset.com <http://www.digitalasset.com/> -- This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html <http://www.digitalasset.com/emaildisclaimer.html>. If you are not the intended recipient, please delete this message.

5 5

GitLab down
by Ben Gamari 28 Mar '19

28 Mar '19

Hi everyone, Due to unplanned maintenance gitlab.haskell.org is currently down. Sorry for the inconvenience. I'll post updates as appropriate. Cheers, - Ben

1 0

RE: Hadrian Transitive Dependencies
by Andrey Mokhov 28 Mar '19

28 Mar '19

Simon: > Perhaps the remaining un-tracked dependencies will be fewer and easier to nail? There are a few other cases where we `need` only a subset of direct dependencies that "cover" all others. For example, `setup-config` files produced when configuring a package are used to signal the completion of package configuration, which might produce a few other hard-to-predict files, such as “build/include/HsBaseConfig.h”. I just realised that if one wants to have “hermetic builds”, where build commands are executed in a sandbox, then it is absolutely necessary to declare all direct dependencies without exception, because they will need to be made available in the sandbox for the build command to succeed. David: > Perhaps the best step forward is to simply silence the linting > using `trackAllow ["//*.hi"]` for haskell object rules. Then > I can continue tracking down other missing dependencies in > Hadrian with fsatrace linting. Yes, this looks like the best approach for now. Cheers, Andrey

2 1

RE: Hadrian Transitive Dependencies
by Andrey Mokhov 27 Mar '19

27 Mar '19

Simon's insight is great: if deep dependencies are captured by shallow dependencies then the cloud build system is correct even if only direct shallow inputs are tracked. That's a very non-trivial invariant, and I guess this means we can't rely on fsatrace linting for GHC compilation rules, because all deep dependencies will be reported as untracked. Cheers, Andrey On 27 Mar 2019 18:27, Simon Peyton Jones <simonpj(a)microsoft.com> wrote: With that in mind, and considering a cloud build system where "all direct inputs and direct outputs must be declared" But I question that assumption. As I mentioned, with GHC at least, the if a deep dependency changes then one of the shallow dependencies will change. So I claim that even for cloud build it should be enough to depend only on shallow dependencies. This is only true because GHC offers this guarantee. We’d need to be sure that every deep dependency was either ‘needed’ or was reflected in the contents (perhaps via a fingerprint) another ‘needed’ thing. Simon From: ghc-devs <ghc-devs-bounces(a)haskell.org> On Behalf Of David Eichmann Sent: 27 March 2019 17:12 To: Andrey Mokhov <andrey.mokhov(a)newcastle.ac.uk>; Neil Mitchell <ndmitchell(a)gmail.com> Cc: GHC developers <ghc-devs(a)haskell.org> Subject: Re: Hadrian Transitive Dependencies Hello, To reiterate some definitions consider this scenario: * A.hs imports B.hs and B.hs imports C.hs * `ghc -M A.hs` reports that A.o depends on: A.hs, B.hi * `ghc -c A.hs` produces A.o and accesses A.hs, B.hi, and C.hi There seems to be some confusion about the term "Direct Dependency" I'll use these definitions: "Shallow Dependency": With respect to a haskell object file X.o, the shallow dependencies are the source file X.hs and interface files Y.hi for all modules Y imported by X. * These are the dependencies of X.o as reported by `ghc -M X.hs` * In the above scenario: * * A.o depends on: A.hs, B.hi "Deep Dependency": With respect to a haskell object file X.o, the deep dependencies are all hi files required by ghc to build X.o excluding direct dependencies: * This is a subset of modules transitively imported by X * These dependencies are NOT reported by `ghc -M X.hs` "Direct Dependency": if the command to create file X accesses file Y, then X directly depends on Y (= Y is a direct dependency of X). * In the above scenario: * * A.o directly depends on: A.hs, B.hi, and C.hi * SPJ noted that .hi files list direct dependencies. * The direct dependencies of a haskell object file is the union of its shallow and deep dependencies. "Direct Output": All files created by a rule. With that in mind, and considering a cloud build system where "all direct inputs and direct outputs must be declared" (where this agrees with the definitions above) can we do the following for the build rule of a haskell object file X.o? 1. `need` the shallow dependencies as reported by `ghc -M`. This guarantees that all shallow and deep dependencies (i.e. all direct dependencies) are built. 2. build X.o and X.hi 3. Inspect X.hi to derive the direct dependencies (and hence deep dependencies) 4. `needed` the deep dependencies Is there already an easy way to inspect *.hi files in this way? Is this use of `needed` valid? - David E On 3/27/19 3:05 PM, Andrey Mokhov wrote: Hi David, We had a discussion about this with Neil some time ago, and I think we had the following list of progressively more complex invariants for different types of build systems: * Non-cloud build systems: *all direct inputs must be declared*. If you miss a direct input dependency then a build may complete successfully but with an incorrect result. * Cloud build systems: *all direct inputs and direct outputs must be declared*. If you miss a direct output then a build may fail because the cloud will not be able to restore the corresponding output. * Cloud build systems with shallow (deferred) materialisation of build artefacts: *all transitive inputs and direct outputs must be declared*. Let’s say you’d like to download the resulting GHC binary directly, without materialising any intermediate artefacts. Then you’ll need to know GHC’s ultimate transitive inputs. I think for now we are really keen to make Hadrian a cloud build system, but whether shallow builds are valuable enough is not clear. Maybe not. Therefore, I’d say we don’t need to track transitive inputs right now. Furthermore, if we were to track all transitive inputs, we would lose the desirable early cutoff property, which prevents rebuilding after adding a comment in a file on which a lot of other files transitively depend on. Having said that, if we really access a file during compilation, then I think it is *not* a transitive dependency by definition! Any file which is accessed during a build rule is a direct dependency. > GHC is reading *.hi files that are not reported as dependencies by > `ghc -M -include-pkg-deps`. This is because they are not direct, but transitive > dependencies! So, here I’m confused. If we read a file A when compiling a file B, then it’s by definition a direct dependency. Perhaps we just read too much? Maybe the solution is to switch to fine-grained `ghc -M` mode, to analyse import dependencies for a single module instead of doing it transitively, which I believe was discussed in a ticket some time ago? I can’t find this ticket, but I think Alp was looking into it at some point. Alp: do you remember it? Thank you for all your work on Hadrian! Cheers, Andrey From: David Eichmann [mailto:davide@well-typed.com] Sent: 27 March 2019 12:54 To: Neil Mitchell <ndmitchell(a)gmail.com><mailto:ndmitchell@gmail.com>; Andrey Mokhov <andrey.mokhov(a)newcastle.ac.uk><mailto:andrey.mokhov@newcastle.ac.uk>; GHC developers <ghc-devs(a)haskell.org><mailto:ghc-devs@haskell.org> Subject: Hadrian Transitive Dependencies Hello Shake/Hadrian contributors and the like, Recently I've been putting Hadrian's fsatrace linting feature to good use, tracking down missing dependencies in Hadrian. Ultimately, we want to use shake's cloud build / shared cache feature and ensure it works across CI builds. Unfortunately the feature isn't working smoothly with Hadrian: see #16295<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.ha…>. This is very desirable to improve CI build times. It is my understanding that in order to get caching to work: 1. All accessed files must declared with `need` AND 2. All created files must be declared with `produces` (or be the target of the build rule) Is my understanding correct? Or is there a weaker condition (perhaps only 2 is necessary)? If I'm correct, this amounts to fixing all fsatrace lint errors. See here<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.ha…> for a breakdown of lint errors / missing dependencies. A large portion of these are Haskell interface files (i.e. *.hi files). Before building a Haskell object file, dependencies are discovered via `ghc` using the `-M -include-pkg-deps` options. Unfortunately, shake's fsatrace linting complains about other *.hi files being accessed! For example when building `stage1/libraries/mtl/build/Control/Monad/RWS/Class.o` we get the following dependencies from ghc: _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : libraries/mtl/Control/Monad/RWS/Class.hs _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Prelude.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Data/Monoid.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/RWS/Strict.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/RWS/Lazy.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Identity.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Maybe.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Except.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Error.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Class.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/libraries/mtl/build/Control/Monad/Writer/Class.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/libraries/mtl/build/Control/Monad/State/Class.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/libraries/mtl/build/Control/Monad/Reader/Class.hi And shake complains of the following missing deps: _build/stage0/bin/ghc -Wall -hisuf hi -osuf o -hcsuf hc -static -hide-all-packages -no-user-package-db '-package-db _build/stage1/lib/package.conf.d' '-this-unit-id mtl-2.2.2' '-package-id base-4.13.0.0' '-package-id transformers-0.5.5.0' -i -i_build/stage1/libraries/mtl/build -i_build/stage1/libraries/mtl/build/autogen -ilibraries/mtl/. -Iincludes -I_build/generated -I_build/stage1/libraries/mtl/build -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/include -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/integer-gmp-1.0.2.0/include -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/rts-1.0/include -I_build/generated -optc-I_build/generated -optP-include -optP_build/stage1/libraries/mtl/build/autogen/cabal_macros.h -outputdir _build/stage1/libraries/mtl/build -Wnoncanonical-monad-instances -optc-Werror=unused-but-set-variable -optc-Wno-error=inline -c libraries/mtl/Control/Monad/RWS/Class.hs -o _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o -O2 -H32m -Wall -fno-warn-unused-imports -fno-warn-warnings-deprecations -Wcompat -Wnoncanonical-monad-instances -Wnoncanonical-monadfail-instances -XHaskell2010 -XSafe -ghcversion-file=/home/david/MEGA/File_Dump/Well-Typed/GHC/_nosync_git/ghc/_build/generated/ghcversion.h -Wno-deprecated-flags Lint checking error - _build/HEAD_default/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o - 22 values were used but not depended upon: Used: _build/HEAD_default/stage0/lib/settings Used: _build/HEAD_default/stage0/lib/platformConstants Used: _build/HEAD_default/stage0/lib/llvm-targets Used: _build/HEAD_default/stage0/lib/llvm-passes Used: _build/HEAD_default/stage0/lib/package.conf.d/package.cache Used: _build/HEAD_default/stage1/lib/package.conf.d/package.cache Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Float.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Base.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/ghc-prim-0.5.3/GHC/Types.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Maybe.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Writer/Lazy.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Writer/Strict.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/State/Lazy.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/State/Strict.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Reader.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/List.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Cont.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/ghc-prim-0.5.3/GHC/Tuple.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/IO/Exception.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/integer-gmp-1.0.2.0/GHC/Integer/Type.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Data/Either.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Natural.hi GHC is reading *.hi files that are not reported as dependencies by `ghc -M -include-pkg-deps`. This is because they are not direct, but transitive dependencies! How do we fix these lint errors (again with the goal of using shakes shared cache feature)? Some ideas: * Wildly over approximate dependencies. This may be easier to implement but cause unneeded recompilation (when a false dependency changes). Either: * `need` all dependent packages' interface files recursively as well as transitive dependencies reported by `ghc -M -include-pkg-deps` within the current package. OR * OR `need` all transitive dependencies reported by `ghc -M -include-pkg-deps`. This will likely result in fewer dependencies but requires a bit more work in recovering dependent packages' dependency graphs. * Perhaps transitive dependencies are not important for shared caching to work. Change shakes linting feature to allow (untracked?) transitive dependencies to be accessed. Feed back would be greatly appreciated. David Eichmann -- David Eichmann, Haskell Consultant Well-Typed LLP, http://www.well-typed.com<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.well-t…> Registered in England & Wales, OC335890 118 Wymering Mansions, Wymering Road, London W9 2NF, England -- David Eichmann, Haskell Consultant Well-Typed LLP, http://www.well-typed.com<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.well-t…> Registered in England & Wales, OC335890 118 Wymering Mansions, Wymering Road, London W9 2NF, England

3 2

RE: Hadrian Transitive Dependencies
by Andrey Mokhov 27 Mar '19

27 Mar '19

Hi David, We had a discussion about this with Neil some time ago, and I think we had the following list of progressively more complex invariants for different types of build systems: · Non-cloud build systems: *all direct inputs must be declared*. If you miss a direct input dependency then a build may complete successfully but with an incorrect result. · Cloud build systems: *all direct inputs and direct outputs must be declared*. If you miss a direct output then a build may fail because the cloud will not be able to restore the corresponding output. · Cloud build systems with shallow (deferred) materialisation of build artefacts: *all transitive inputs and direct outputs must be declared*. Let’s say you’d like to download the resulting GHC binary directly, without materialising any intermediate artefacts. Then you’ll need to know GHC’s ultimate transitive inputs. I think for now we are really keen to make Hadrian a cloud build system, but whether shallow builds are valuable enough is not clear. Maybe not. Therefore, I’d say we don’t need to track transitive inputs right now. Furthermore, if we were to track all transitive inputs, we would lose the desirable early cutoff property, which prevents rebuilding after adding a comment in a file on which a lot of other files transitively depend on. Having said that, if we really access a file during compilation, then I think it is *not* a transitive dependency by definition! Any file which is accessed during a build rule is a direct dependency. > GHC is reading *.hi files that are not reported as dependencies by > `ghc -M -include-pkg-deps`. This is because they are not direct, but transitive > dependencies! So, here I’m confused. If we read a file A when compiling a file B, then it’s by definition a direct dependency. Perhaps we just read too much? Maybe the solution is to switch to fine-grained `ghc -M` mode, to analyse import dependencies for a single module instead of doing it transitively, which I believe was discussed in a ticket some time ago? I can’t find this ticket, but I think Alp was looking into it at some point. Alp: do you remember it? Thank you for all your work on Hadrian! Cheers, Andrey From: David Eichmann [mailto:davide@well-typed.com] Sent: 27 March 2019 12:54 To: Neil Mitchell <ndmitchell(a)gmail.com>; Andrey Mokhov <andrey.mokhov(a)newcastle.ac.uk>; GHC developers <ghc-devs(a)haskell.org> Subject: Hadrian Transitive Dependencies Hello Shake/Hadrian contributors and the like, Recently I've been putting Hadrian's fsatrace linting feature to good use, tracking down missing dependencies in Hadrian. Ultimately, we want to use shake's cloud build / shared cache feature and ensure it works across CI builds. Unfortunately the feature isn't working smoothly with Hadrian: see #16295<https://gitlab.haskell.org/ghc/ghc/issues/16295>. This is very desirable to improve CI build times. It is my understanding that in order to get caching to work: 1. All accessed files must declared with `need` AND 2. All created files must be declared with `produces` (or be the target of the build rule) Is my understanding correct? Or is there a weaker condition (perhaps only 2 is necessary)? If I'm correct, this amounts to fixing all fsatrace lint errors. See here<https://gitlab.haskell.org/ghc/ghc/issues/16400#note_188901> for a breakdown of lint errors / missing dependencies. A large portion of these are Haskell interface files (i.e. *.hi files). Before building a Haskell object file, dependencies are discovered via `ghc` using the `-M -include-pkg-deps` options. Unfortunately, shake's fsatrace linting complains about other *.hi files being accessed! For example when building `stage1/libraries/mtl/build/Control/Monad/RWS/Class.o` we get the following dependencies from ghc: _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : libraries/mtl/Control/Monad/RWS/Class.hs _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Prelude.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Data/Monoid.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/RWS/Strict.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/RWS/Lazy.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Identity.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Maybe.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Except.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Error.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Class.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/libraries/mtl/build/Control/Monad/Writer/Class.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/libraries/mtl/build/Control/Monad/State/Class.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/libraries/mtl/build/Control/Monad/Reader/Class.hi And shake complains of the following missing deps: _build/stage0/bin/ghc -Wall -hisuf hi -osuf o -hcsuf hc -static -hide-all-packages -no-user-package-db '-package-db _build/stage1/lib/package.conf.d' '-this-unit-id mtl-2.2.2' '-package-id base-4.13.0.0' '-package-id transformers-0.5.5.0' -i -i_build/stage1/libraries/mtl/build -i_build/stage1/libraries/mtl/build/autogen -ilibraries/mtl/. -Iincludes -I_build/generated -I_build/stage1/libraries/mtl/build -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/include -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/integer-gmp-1.0.2.0/include -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/rts-1.0/include -I_build/generated -optc-I_build/generated -optP-include -optP_build/stage1/libraries/mtl/build/autogen/cabal_macros.h -outputdir _build/stage1/libraries/mtl/build -Wnoncanonical-monad-instances -optc-Werror=unused-but-set-variable -optc-Wno-error=inline -c libraries/mtl/Control/Monad/RWS/Class.hs -o _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o -O2 -H32m -Wall -fno-warn-unused-imports -fno-warn-warnings-deprecations -Wcompat -Wnoncanonical-monad-instances -Wnoncanonical-monadfail-instances -XHaskell2010 -XSafe -ghcversion-file=/home/david/MEGA/File_Dump/Well-Typed/GHC/_nosync_git/ghc/_build/generated/ghcversion.h -Wno-deprecated-flags Lint checking error - _build/HEAD_default/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o - 22 values were used but not depended upon: Used: _build/HEAD_default/stage0/lib/settings Used: _build/HEAD_default/stage0/lib/platformConstants Used: _build/HEAD_default/stage0/lib/llvm-targets Used: _build/HEAD_default/stage0/lib/llvm-passes Used: _build/HEAD_default/stage0/lib/package.conf.d/package.cache Used: _build/HEAD_default/stage1/lib/package.conf.d/package.cache Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Float.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Base.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/ghc-prim-0.5.3/GHC/Types.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Maybe.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Writer/Lazy.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Writer/Strict.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/State/Lazy.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/State/Strict.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Reader.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/List.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Cont.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/ghc-prim-0.5.3/GHC/Tuple.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/IO/Exception.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/integer-gmp-1.0.2.0/GHC/Integer/Type.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Data/Either.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Natural.hi GHC is reading *.hi files that are not reported as dependencies by `ghc -M -include-pkg-deps`. This is because they are not direct, but transitive dependencies! How do we fix these lint errors (again with the goal of using shakes shared cache feature)? Some ideas: * Wildly over approximate dependencies. This may be easier to implement but cause unneeded recompilation (when a false dependency changes). Either: * `need` all dependent packages' interface files recursively as well as transitive dependencies reported by `ghc -M -include-pkg-deps` within the current package. OR * OR `need` all transitive dependencies reported by `ghc -M -include-pkg-deps`. This will likely result in fewer dependencies but requires a bit more work in recovering dependent packages' dependency graphs. * Perhaps transitive dependencies are not important for shared caching to work. Change shakes linting feature to allow (untracked?) transitive dependencies to be accessed. Feed back would be greatly appreciated. David Eichmann -- David Eichmann, Haskell Consultant Well-Typed LLP, http://www.well-typed.com Registered in England & Wales, OC335890 118 Wymering Mansions, Wymering Road, London W9 2NF, England

5 5

Hadrian Transitive Dependencies
by David Eichmann 27 Mar '19

27 Mar '19

Hello Shake/Hadrian contributors and the like, Recently I've been putting Hadrian's fsatrace linting feature to good use, tracking down missing dependencies in Hadrian. Ultimately, we want to use shake's cloud build / shared cache feature and ensure it works across CI builds. Unfortunately the feature isn't working smoothly with Hadrian: see #16295 <https://gitlab.haskell.org/ghc/ghc/issues/16295>. This is very desirable to improve CI build times. It is my understanding that in order to get caching to work: 1. All accessed files must declared with `need` AND 2. All created files must be declared with `produces` (or be the target of the build rule) Is my understanding correct? Or is there a weaker condition (perhaps only 2 is necessary)? If I'm correct, this amounts to fixing all fsatrace lint errors. See here <https://gitlab.haskell.org/ghc/ghc/issues/16400#note_188901> for a breakdown of lint errors / missing dependencies. A large portion of these are Haskell interface files (i.e. *.hi files). Before building a Haskell object file, dependencies are discovered via `ghc` using the `-M -include-pkg-deps` options. Unfortunately, shake's fsatrace linting complains about other *.hi files being accessed! For example when building `stage1/libraries/mtl/build/Control/Monad/RWS/Class.o` we get the following dependencies from ghc: _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : libraries/mtl/Control/Monad/RWS/Class.hs _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Prelude.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Data/Monoid.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/RWS/Strict.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/RWS/Lazy.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Identity.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Maybe.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Except.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Error.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Class.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/libraries/mtl/build/Control/Monad/Writer/Class.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/libraries/mtl/build/Control/Monad/State/Class.hi _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/libraries/mtl/build/Control/Monad/Reader/Class.hi And shake complains of the following missing deps: _build/stage0/bin/ghc -Wall -hisuf hi -osuf o -hcsuf hc -static -hide-all-packages -no-user-package-db '-package-db _build/stage1/lib/package.conf.d' '-this-unit-id mtl-2.2.2' '-package-id base-4.13.0.0' '-package-id transformers-0.5.5.0' -i -i_build/stage1/libraries/mtl/build -i_build/stage1/libraries/mtl/build/autogen -ilibraries/mtl/. -Iincludes -I_build/generated -I_build/stage1/libraries/mtl/build -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/include -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/integer-gmp-1.0.2.0/include -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/rts-1.0/include -I_build/generated -optc-I_build/generated -optP-include -optP_build/stage1/libraries/mtl/build/autogen/cabal_macros.h -outputdir _build/stage1/libraries/mtl/build -Wnoncanonical-monad-instances -optc-Werror=unused-but-set-variable -optc-Wno-error=inline -c libraries/mtl/Control/Monad/RWS/Class.hs -o _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o -O2 -H32m -Wall -fno-warn-unused-imports -fno-warn-warnings-deprecations -Wcompat -Wnoncanonical-monad-instances -Wnoncanonical-monadfail-instances -XHaskell2010 -XSafe -ghcversion-file=/home/david/MEGA/File_Dump/Well-Typed/GHC/_nosync_git/ghc/_build/generated/ghcversion.h -Wno-deprecated-flags Lint checking error - _build/HEAD_default/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o - 22 values were used but not depended upon: Used: _build/HEAD_default/stage0/lib/settings Used: _build/HEAD_default/stage0/lib/platformConstants Used: _build/HEAD_default/stage0/lib/llvm-targets Used: _build/HEAD_default/stage0/lib/llvm-passes Used: _build/HEAD_default/stage0/lib/package.conf.d/package.cache Used: _build/HEAD_default/stage1/lib/package.conf.d/package.cache Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Float.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Base.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/ghc-prim-0.5.3/GHC/Types.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Maybe.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Writer/Lazy.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Writer/Strict.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/State/Lazy.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/State/Strict.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Reader.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/List.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Cont.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/ghc-prim-0.5.3/GHC/Tuple.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/IO/Exception.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/integer-gmp-1.0.2.0/GHC/Integer/Type.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Data/Either.hi Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Natural.hi GHC is reading *.hi files that are not reported as dependencies by `ghc -M -include-pkg-deps`. This is because they are not direct, but /transitive/ dependencies! How do we fix these lint errors (again with the goal of using shakes shared cache feature)? Some ideas: * Wildly over approximate dependencies. This may be easier to implement but cause unneeded recompilation (when a false dependency changes). Either: * `need` all dependent packages' interface files recursively as well as transitive dependencies reported by `ghc -M -include-pkg-deps` within the current package. OR * OR `need` all transitive dependencies reported by `ghc -M -include-pkg-deps`. This will likely result in fewer dependencies but requires a bit more work in recovering dependent packages' dependency graphs. * Perhaps transitive dependencies are not important for shared caching to work. Change shakes linting feature to allow (untracked?) transitive dependencies to be accessed. Feed back would be greatly appreciated. David Eichmann -- David Eichmann, Haskell Consultant Well-Typed LLP, http://www.well-typed.com Registered in England & Wales, OC335890 118 Wymering Mansions, Wymering Road, London W9 2NF, England

2 1

Re: GitLab seems quite poorly
by Ryan Scott 25 Mar '19

25 Mar '19

Things still appear not to work whenever I try to push a new branch: fatal: unable to access 'https://gitlab.haskell.org/RyanGlScott/ghc.git/': The requested URL returned error: 500 Ryan S.

2 1

GitLab seems quite poorly
by Matthew Pickering 25 Mar '19

25 Mar '19

I don't know what has changed since yesterday but pushing new branches to GitLab is broken. The two most recent branches pushed to origin 404 when you click on the link. https://gitlab.haskell.org/ghc/ghc/branches https://gitlab.haskell.org/ghc/ghc/tree/wip/mp/15646 I also pushed some commits to a fork and the MR which they appear on didn't update at all. Perhaps a symptom of the same problem. Also a recent MR by Ben complains of the source branch for the MR not existing. https://gitlab.haskell.org/ghc/ghc/merge_requests/625 The summary is that GitLab seems unusable at the moment. Did you fiddle with anything recently Ben? Cheers, Matt

3 5