- ghc-devs - Haskell.org

Can strict ST break referential transparency?
by Yuras Shumovich 23 Nov '17

23 Nov '17

Hello, I was evaluating a possibility that linear types can break referential transparency [1], exactly like lazy ST [2]. But on the way I realized that even strict ST may suffer from the same issue. If ST computation is interrupted by e.g. async exception, runtime will "freeze" it at the point where it was interrupted [3]. So the question: is the "freezed" computation just a normal thunk? Note that the runtime doesn't guarantee that a thunk will be evaluated only once [4]. If the "freezed" thunk captures e.g. STRef, and will be evaluated twice, its effect could become observable from outside, just like in case of lazy ST. I tried to check the theory by stress testing RTS. Unfortunately I immediately discovered a runtime crash [5], which is probably not related to my question. Hope someone will be able to clarify things for me. Thanks, Yuras. [1] https://github.com/ghc-proposals/ghc-proposals/pull/91#issuecomment -345553071 [2] https://ghc.haskell.org/trac/ghc/ticket/14497 [3] See section 8 there: https://www.microsoft.com/en-us/research/wp-co ntent/uploads/2016/07/asynch-exns.pdf [4] https://www.microsoft.com/en-us/research/wp-content/uploads/2005/09 /2005-haskell.pdf [5] https://ghc.haskell.org/trac/ghc/ticket/14497

3 4

Re: [GHC] #14490: TTG Snags
by Alan & Kim Zimmerman 23 Nov '17

23 Nov '17

Yes, but it is not feasible before GHC 8.6, due to needing GHC >= 8.2 for bootstrapping. On 22 November 2017 at 15:26, GHC <ghc-devs(a)haskell.org> wrote: > #14490: TTG Snags > -------------------------------------+---------------------- > --------------- > Reporter: alanz | Owner: (none) > Type: bug | Status: new > Priority: normal | Milestone: > Component: Compiler | Version: 8.3 > Resolution: | Keywords: > Operating System: Unknown/Multiple | Architecture: > | Unknown/Multiple > Type of failure: None/Unknown | Test Case: > Blocked By: | Blocking: > Related Tickets: #14482 | Differential Rev(s): > Wiki Page: | > ImplementingTreesThatGrow | > -------------------------------------+---------------------- > --------------- > > Comment (by Shayan-Najd): > > > So making a Data instance per concrete version may be the best way to > go. Plan B. > > Regardless of the GHC version we use, the question is whether Plan B > really fixes the build-time problem. Can we test it? > > -- > Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14490#comment:31> > GHC <http://www.haskell.org/ghc/> > The Glasgow Haskell Compiler >

2 1

Unstuck old messages
by John Wiegley 23 Nov '17

23 Nov '17

Hello everyone, I was doing some maintenance on the mailman server that hosts this list, and several messages that were stuck in the queue for what appears to be a rather long time have been released. The queue is now empty, so if these messages are no longer relevant, please ignore them. -- John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2

1 0

Re: Can strict ST break referential transparency?
by Yuras Shumovich 23 Nov '17

23 Nov '17

22-11-2017, Срд а 15:37 -0500, David Feuer напісаў: > If there is indeed a problem, I suspect the right way to fix it is to > make sure that no partially evaluated thunk is ever resumed twice. > Inter-thread exceptions are presumably rare enough that we don't have > to worry *too* much about their cost. Sounds reasonable. But probably it already works that way? Though it doesn't seem to be covered in the papers, and looks like nobody knows an answer to my question. It makes me worry a bit. On the other hand, right now resuming frozen thunk usually crashes/hangs as discovered in #14497. Nobody noticed that so far in the wild, so probably the subject is just a dark corner case, not worse the efforts. > > > David FeuerWell-Typed, LLP > -------- Original message --------From: Yuras Shumovich <shumovichy@g > mail.com> Date: 11/21/17 12:43 PM (GMT-05:00) To: ghc-devs <ghc-dev > s(a)haskell.org> Subject: Can strict ST break referential > transparency? > > Hello, > > I was evaluating a possibility that linear types can break > referential > transparency [1], exactly like lazy ST [2]. > > But on the way I realized that even strict ST may suffer from the > same > issue. If ST computation is interrupted by e.g. async exception, > runtime will "freeze" it at the point where it was interrupted [3]. > > So the question: is the "freezed" computation just a normal thunk? > Note > that the runtime doesn't guarantee that a thunk will be evaluated > only > once [4]. If the "freezed" thunk captures e.g. STRef, and will be > evaluated twice, its effect could become observable from outside, > just > like in case of lazy ST. > > I tried to check the theory by stress testing RTS. Unfortunately I > immediately discovered a runtime crash [5], which is probably not > related to my question. > > Hope someone will be able to clarify things for me. > > Thanks, > Yuras. > > [1] https://github.com/ghc-proposals/ghc-proposals/pull/91#issuecomme > nt > -345553071 > [2] https://ghc.haskell.org/trac/ghc/ticket/14497 > [3] See section 8 there: https://www.microsoft.com/en-us/research/wp- > co > ntent/uploads/2016/07/asynch-exns.pdf > [4] https://www.microsoft.com/en-us/research/wp-content/uploads/2005/ > 09 > /2005-haskell.pdf > [5] https://ghc.haskell.org/trac/ghc/ticket/14497 > _______________________________________________ > ghc-devs mailing list > ghc-devs(a)haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

1 0

Re: Can strict ST break referential transparency?
by David Feuer 22 Nov '17

22 Nov '17

If there is indeed a problem, I suspect the right way to fix it is to make sure that no partially evaluated thunk is ever resumed twice. Inter-thread exceptions are presumably rare enough that we don't have to worry *too* much about their cost. David FeuerWell-Typed, LLP -------- Original message --------From: Yuras Shumovich <shumovichy(a)gmail.com> Date: 11/21/17 12:43 PM (GMT-05:00) To: ghc-devs <ghc-devs(a)haskell.org> Subject: Can strict ST break referential transparency? Hello, I was evaluating a possibility that linear types can break referential transparency [1], exactly like lazy ST [2]. But on the way I realized that even strict ST may suffer from the same issue. If ST computation is interrupted by e.g. async exception, runtime will "freeze" it at the point where it was interrupted [3]. So the question: is the "freezed" computation just a normal thunk? Note that the runtime doesn't guarantee that a thunk will be evaluated only once [4]. If the "freezed" thunk captures e.g. STRef, and will be evaluated twice, its effect could become observable from outside, just like in case of lazy ST. I tried to check the theory by stress testing RTS. Unfortunately I immediately discovered a runtime crash [5], which is probably not related to my question. Hope someone will be able to clarify things for me. Thanks, Yuras. [1] https://github.com/ghc-proposals/ghc-proposals/pull/91#issuecomment -345553071 [2] https://ghc.haskell.org/trac/ghc/ticket/14497 [3] See section 8 there: https://www.microsoft.com/en-us/research/wp-co ntent/uploads/2016/07/asynch-exns.pdf [4] https://www.microsoft.com/en-us/research/wp-content/uploads/2005/09 /2005-haskell.pdf [5] https://ghc.haskell.org/trac/ghc/ticket/14497 _______________________________________________ ghc-devs mailing list ghc-devs(a)haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

1 0

[ANNOUNCE] GHC 8.2.2 released
by Ben Gamari 21 Nov '17

21 Nov '17

=============================================== The Glasgow Haskell Compiler -- version 8.2.2 =============================================== The GHC Team is pleased to announce a new minor release of GHC. This release builds on the performance and stability improvements of 8.2.1, fixing a variety of correctness bugs, improving error messages, and making the compiler more portable. Notable bug-fixes include * A correctness issue resulting in segmentation faults in some FFI-users (#13707, #14346) * A correctness issue resulting in undefined behavior in some programs using STM (#14171) * A bug which may have manifested in segmentation faults in out-of-memory condition (#14329) * clearBit of Natural no longer bottoms (#13203) * A specialisation bug resulting in exponential blowup of compilation time in some specialisation-intensive programs (#14379) * ghc-pkg now works even in environments with misconfigured NFS mounts (#13945) * GHC again supports production of position-independent executables (#13702) * Better error messages around kind mismatches (#11198, #12373, #13530, #13610) A thorough list of the changes in the release can be found in the release notes, https://haskell.org/ghc/docs/8.2.2/html/users_guide/release-8-2-2.html How to get it ~~~~~~~~~~~~~ This release can be downloaded from https://www.haskell.org/ghc/download_ghc_8_2_2.html For older versions see https://www.haskell.org/ghc/ We supply binary builds in the native package format for many platforms, and the source distribution is available from the same place. Background ~~~~~~~~~~ Haskell is a standardized lazy functional programming language. GHC is a state-of-the-art programming suite for Haskell. Included is an optimising compiler generating efficient code for a variety of platforms, together with an interactive system for convenient, quick development. The distribution includes space and time profiling facilities, a large collection of libraries, and support for various language extensions, including concurrency, exceptions, and foreign language interfaces. GHC is distributed under a BSD-style open source license. A wide variety of Haskell related resources (tutorials, libraries, specifications, documentation, compilers, interpreters, references, contact information, links to research groups) are available from the Haskell home page (see below). On-line GHC-related resources ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Relevant URLs on the World-Wide Web: GHC home page https://www.haskell.org/ghc/ GHC developers' home page https://ghc.haskell.org/trac/ghc/ Haskell home page https://www.haskell.org/ Supported Platforms ~~~~~~~~~~~~~~~~~~~ The list of platforms we support, and the people responsible for them, is here: https://ghc.haskell.org/trac/ghc/wiki/Contributors Ports to other platforms are possible with varying degrees of difficulty. The Building Guide describes how to go about porting to a new platform: https://ghc.haskell.org/trac/ghc/wiki/Building Developers ~~~~~~~~~~ We welcome new contributors. Instructions on accessing our source code repository, and getting started with hacking on GHC, are available from the GHC's developer's site: https://ghc.haskell.org/trac/ghc/ Mailing lists ~~~~~~~~~~~~~ We run mailing lists for GHC users and bug reports; to subscribe, use the web interfaces at https://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets There are several other haskell and ghc-related mailing lists on www.haskell.org; for the full list, see https://mail.haskell.org/cgi-bin/mailman/listinfo Many GHC developers hang out on #haskell on IRC: https://www.haskell.org/haskellwiki/IRC_channel Please report bugs using our bug tracking system. Instructions on reporting bugs can be found here: https://www.haskell.org/ghc/reportabug

1 1

Commit Notifications Flood
by Ara Adkins 21 Nov '17

21 Nov '17

Is anyone else getting an absolute flood of commit emails related to haddock? I remember this happening a little while back with the hadrian merge into the GHC tree. Best, _ara

3 4

Re: Linux failure
by David Feuer 20 Nov '17

20 Nov '17

Digging around online, I suspect that you may not have the GNU C++ compiler installed at all. Take this with a shaker of salt. David FeuerWell-Typed, LLP -------- Original message --------From: Simon Peyton Jones via ghc-devs <ghc-devs(a)haskell.org> Date: 11/20/17 8:14 AM (GMT-05:00) To: ghc-devs(a)haskell.org Subject: Linux failure Hi devs I've got my Linux box upgraded to a recent version of Ubuntu, so I'm no longer stuck on gcc version. But I can't build either. Something strange with /lib/cpp fails sanity check. The log is below. Can anyone help? I'm a bit stalled. This call to "make" follows the same failure when I typed "sh validate -fast". But by carrying on from where it got stuck the log is way shorter. Thanks Simon simonpj@cam-05-unx:~/5builds/HEAD$ make ===--- building phase 0 make --no-print-directory -f ghc.mk phase=0 phase_0_builds make[1]: Nothing to be done for 'phase_0_builds'. ===--- building phase 1 make --no-print-directory -f ghc.mk phase=1 phase_1_builds make[1]: Nothing to be done for 'phase_1_builds'. ===--- building final phase make --no-print-directory -f ghc.mk phase=final all rts/ghc.mk:547: rts/dist/build/.depend-v-dyn-l-debug-thr-thr_debug-thr_l-debug_dyn-thr_dyn-thr_debug_dyn-l_dyn-thr_l_dyn.c_asm: No such file or directory libraries/ghci/ghc.mk:4: libraries/ghci/dist-install/build/.depend-v-dyn.haskell: No such file or directory "rm" -f libffi/stamp.ffi.static.configure "rm" -f libffi/stamp.ffi.static.build "rm" -f libffi/stamp.ffi.static.install "rm" -f libffi/stamp.ffi.static-shared.configure "rm" -f libffi/stamp.ffi.static-shared.build "rm" -f libffi/stamp.ffi.static-shared.install "rm" -rf libffi/build cat libffi-tarballs/libffi*.tar.gz | /bin/gzip -d | { cd libffi && /bin/tar -xf - ; } mv libffi/libffi-* libffi/build cp "/home/simonpj/5builds/HEAD/config.guess" libffi/build/config.guess cp "/home/simonpj/5builds/HEAD/config.sub" libffi/build/config.sub chmod +x libffi/ln # We need to use -MMD rather than -MD, as otherwise we get paths # like c:/... in the dependency files on Windows, and the extra # colons break make mv libffi/build/Makefile.in libffi/build/Makefile.in.orig sed "s/-MD/-MMD/" < libffi/build/Makefile.in.orig > libffi/build/Makefile.in # We attempt to specify the installation directory below with --libdir, # but libffi installs into 'toolexeclibdir' instead, which may differ # on systems where gcc has multilib support. Force libffi to use libdir. # (https://sourceware.org/ml/libffi-discuss/2014/msg00016.html) mv libffi/build/Makefile.in libffi/build/Makefile.in.orig sed 's:@toolexeclibdir@:$(libdir):g' < libffi/build/Makefile.in.orig > libffi/build/Makefile.in # install-sh is used when /usr/bin/install is missing; ensure its # path in libffi's Makefile is correct. See GHC #11109. mv libffi/build/Makefile.in libffi/build/Makefile.in.orig sed 's|@INSTALL@|$(subst ../install-sh,/home/simonpj/5builds/HEAD/install-sh,@INSTALL@)|g' < libffi/build/Makefile.in.orig > libffi/build/Makefile.in cd libffi && \ cd build && \ CC=gcc \ CXX=gcc \ LD=ld.gold \ AR=ar \ NM=nm \ RANLIB=ranlib \ CFLAGS="-Wall -Werror -fno-stack-protector -w" \ LDFLAGS=" -w" \ "/bin/bash" ./configure \ --prefix=/home/simonpj/5builds/HEAD/libffi/build/inst \ --libdir=/home/simonpj/5builds/HEAD/libffi/build/inst/lib \ --disable-docs \ --enable-static=yes \ --enable-shared=yes \ --host=x86_64-unknown-linux checking build system type... x86_64-pc-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking target system type... x86_64-unknown-linux-gnu continue configure in default builddir "./x86_64-unknown-linux-gnu" ....exec /bin/sh .././configure "--srcdir=.." "--enable-builddir=x86_64-unknown-linux-gnu" "linux gnu" checking build system type... x86_64-pc-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking target system type... x86_64-unknown-linux-gnu checking for gsed... sed checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for x86_64-unknown-linux-strip... no checking for strip... strip checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking for x86_64-unknown-linux-gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking whether gcc understands -c and -o together... yes checking for style of include used by make... GNU checking dependency style of gcc... gcc3 checking whether we are using the GNU C++ compiler... no checking whether gcc accepts -g... no checking dependency style of gcc... gcc3 checking dependency style of gcc... gcc3 checking how to print strings... printf checking for a sed that does not truncate output... /bin/sed checking for grep that handles long lines and -e... /bin/grep checking for egrep... /bin/grep -E checking for fgrep... /bin/grep -F checking for ld used by gcc... ld.gold checking if the linker (ld.gold) is GNU ld... yes checking for BSD- or MS-compatible name lister (nm)... nm checking the name lister (nm) interface... BSD nm checking whether ln -s works... yes checking the maximum length of command line arguments... 1635000 checking how to convert x86_64-pc-linux-gnu file names to x86_64-unknown-linux-gnu format... func_convert_file_noop checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop checking for ld.gold option to reload object files... -r checking for x86_64-unknown-linux-objdump... no checking for objdump... objdump checking how to recognize dependent libraries... pass_all checking for x86_64-unknown-linux-dlltool... no checking for dlltool... no checking how to associate runtime and link libraries... printf %s\n checking for x86_64-unknown-linux-ar... ar checking for archiver @FILE support... @ checking for x86_64-unknown-linux-strip... strip checking for x86_64-unknown-linux-ranlib... ranlib checking command to parse nm output from gcc object... ok checking for sysroot... no checking for a working dd... /bin/dd checking how to truncate binary pipes... /bin/dd bs=4096 count=1 checking for x86_64-unknown-linux-mt... no checking for mt... mt checking if mt is a manifest tool... no checking how to run the C preprocessor... gcc -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking for dlfcn.h... yes checking for objdir... .libs checking if gcc supports -fno-rtti -fno-exceptions... yes checking for gcc option to produce PIC... -fPIC -DPIC checking if gcc PIC flag -fPIC -DPIC works... yes checking if gcc static flag -static works... yes checking if gcc supports -c -o file.o... yes checking if gcc supports -c -o file.o... (cached) yes checking whether the gcc linker (ld.gold -m elf_x86_64) supports shared libraries... yes checking whether -lc should be explicitly linked in... no checking dynamic linker characteristics... GNU/Linux ld.so checking how to hardcode library paths into programs... immediate checking whether stripping libraries is possible... yes checking if libtool supports shared libraries... yes checking whether to build shared libraries... yes checking whether to build static libraries... yes checking how to run the C++ preprocessor... /lib/cpp configure: error: in `/home/simonpj/5builds/HEAD/libffi/build/x86_64-unknown-linux-gnu': configure: error: C++ preprocessor "/lib/cpp" fails sanity check See `config.log' for more details libffi/ghc.mk:47: recipe for target 'libffi/stamp.ffi.static-shared.configure' failed make[1]: *** [libffi/stamp.ffi.static-shared.configure] Error 1 Makefile:122: recipe for target 'all' failed make: *** [all] Error 2

1 0

Linux failure
by Simon Peyton Jones 20 Nov '17

20 Nov '17

Hi devs I've got my Linux box upgraded to a recent version of Ubuntu, so I'm no longer stuck on gcc version. But I can't build either. Something strange with /lib/cpp fails sanity check. The log is below. Can anyone help? I'm a bit stalled. This call to "make" follows the same failure when I typed "sh validate -fast". But by carrying on from where it got stuck the log is way shorter. Thanks Simon simonpj@cam-05-unx:~/5builds/HEAD$ make ===--- building phase 0 make --no-print-directory -f ghc.mk phase=0 phase_0_builds make[1]: Nothing to be done for 'phase_0_builds'. ===--- building phase 1 make --no-print-directory -f ghc.mk phase=1 phase_1_builds make[1]: Nothing to be done for 'phase_1_builds'. ===--- building final phase make --no-print-directory -f ghc.mk phase=final all rts/ghc.mk:547: rts/dist/build/.depend-v-dyn-l-debug-thr-thr_debug-thr_l-debug_dyn-thr_dyn-thr_debug_dyn-l_dyn-thr_l_dyn.c_asm: No such file or directory libraries/ghci/ghc.mk:4: libraries/ghci/dist-install/build/.depend-v-dyn.haskell: No such file or directory "rm" -f libffi/stamp.ffi.static.configure "rm" -f libffi/stamp.ffi.static.build "rm" -f libffi/stamp.ffi.static.install "rm" -f libffi/stamp.ffi.static-shared.configure "rm" -f libffi/stamp.ffi.static-shared.build "rm" -f libffi/stamp.ffi.static-shared.install "rm" -rf libffi/build cat libffi-tarballs/libffi*.tar.gz | /bin/gzip -d | { cd libffi && /bin/tar -xf - ; } mv libffi/libffi-* libffi/build cp "/home/simonpj/5builds/HEAD/config.guess" libffi/build/config.guess cp "/home/simonpj/5builds/HEAD/config.sub" libffi/build/config.sub chmod +x libffi/ln # We need to use -MMD rather than -MD, as otherwise we get paths # like c:/... in the dependency files on Windows, and the extra # colons break make mv libffi/build/Makefile.in libffi/build/Makefile.in.orig sed "s/-MD/-MMD/" < libffi/build/Makefile.in.orig > libffi/build/Makefile.in # We attempt to specify the installation directory below with --libdir, # but libffi installs into 'toolexeclibdir' instead, which may differ # on systems where gcc has multilib support. Force libffi to use libdir. # (https://sourceware.org/ml/libffi-discuss/2014/msg00016.html) mv libffi/build/Makefile.in libffi/build/Makefile.in.orig sed 's:@toolexeclibdir@:$(libdir):g' < libffi/build/Makefile.in.orig > libffi/build/Makefile.in # install-sh is used when /usr/bin/install is missing; ensure its # path in libffi's Makefile is correct. See GHC #11109. mv libffi/build/Makefile.in libffi/build/Makefile.in.orig sed 's|@INSTALL@|$(subst ../install-sh,/home/simonpj/5builds/HEAD/install-sh,@INSTALL@)|g' < libffi/build/Makefile.in.orig > libffi/build/Makefile.in cd libffi && \ cd build && \ CC=gcc \ CXX=gcc \ LD=ld.gold \ AR=ar \ NM=nm \ RANLIB=ranlib \ CFLAGS="-Wall -Werror -fno-stack-protector -w" \ LDFLAGS=" -w" \ "/bin/bash" ./configure \ --prefix=/home/simonpj/5builds/HEAD/libffi/build/inst \ --libdir=/home/simonpj/5builds/HEAD/libffi/build/inst/lib \ --disable-docs \ --enable-static=yes \ --enable-shared=yes \ --host=x86_64-unknown-linux checking build system type... x86_64-pc-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking target system type... x86_64-unknown-linux-gnu continue configure in default builddir "./x86_64-unknown-linux-gnu" ....exec /bin/sh .././configure "--srcdir=.." "--enable-builddir=x86_64-unknown-linux-gnu" "linux gnu" checking build system type... x86_64-pc-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking target system type... x86_64-unknown-linux-gnu checking for gsed... sed checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for x86_64-unknown-linux-strip... no checking for strip... strip checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking for x86_64-unknown-linux-gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking whether gcc understands -c and -o together... yes checking for style of include used by make... GNU checking dependency style of gcc... gcc3 checking whether we are using the GNU C++ compiler... no checking whether gcc accepts -g... no checking dependency style of gcc... gcc3 checking dependency style of gcc... gcc3 checking how to print strings... printf checking for a sed that does not truncate output... /bin/sed checking for grep that handles long lines and -e... /bin/grep checking for egrep... /bin/grep -E checking for fgrep... /bin/grep -F checking for ld used by gcc... ld.gold checking if the linker (ld.gold) is GNU ld... yes checking for BSD- or MS-compatible name lister (nm)... nm checking the name lister (nm) interface... BSD nm checking whether ln -s works... yes checking the maximum length of command line arguments... 1635000 checking how to convert x86_64-pc-linux-gnu file names to x86_64-unknown-linux-gnu format... func_convert_file_noop checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop checking for ld.gold option to reload object files... -r checking for x86_64-unknown-linux-objdump... no checking for objdump... objdump checking how to recognize dependent libraries... pass_all checking for x86_64-unknown-linux-dlltool... no checking for dlltool... no checking how to associate runtime and link libraries... printf %s\n checking for x86_64-unknown-linux-ar... ar checking for archiver @FILE support... @ checking for x86_64-unknown-linux-strip... strip checking for x86_64-unknown-linux-ranlib... ranlib checking command to parse nm output from gcc object... ok checking for sysroot... no checking for a working dd... /bin/dd checking how to truncate binary pipes... /bin/dd bs=4096 count=1 checking for x86_64-unknown-linux-mt... no checking for mt... mt checking if mt is a manifest tool... no checking how to run the C preprocessor... gcc -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking for dlfcn.h... yes checking for objdir... .libs checking if gcc supports -fno-rtti -fno-exceptions... yes checking for gcc option to produce PIC... -fPIC -DPIC checking if gcc PIC flag -fPIC -DPIC works... yes checking if gcc static flag -static works... yes checking if gcc supports -c -o file.o... yes checking if gcc supports -c -o file.o... (cached) yes checking whether the gcc linker (ld.gold -m elf_x86_64) supports shared libraries... yes checking whether -lc should be explicitly linked in... no checking dynamic linker characteristics... GNU/Linux ld.so checking how to hardcode library paths into programs... immediate checking whether stripping libraries is possible... yes checking if libtool supports shared libraries... yes checking whether to build shared libraries... yes checking whether to build static libraries... yes checking how to run the C++ preprocessor... /lib/cpp configure: error: in `/home/simonpj/5builds/HEAD/libffi/build/x86_64-unknown-linux-gnu': configure: error: C++ preprocessor "/lib/cpp" fails sanity check See `config.log' for more details libffi/ghc.mk:47: recipe for target 'libffi/stamp.ffi.static-shared.configure' failed make[1]: *** [libffi/stamp.ffi.static-shared.configure] Error 1 Makefile:122: recipe for target 'all' failed make: *** [all] Error 2

1 0

Observation on Hadrian's relative performance re current buildsystem
by Herbert Valerio Riedel 18 Nov '17

18 Nov '17

Hello GHC devs, I took the opportunity to give Hadrian a test-run to see whether it could live up to the big promise of delivering a "more scalable, faster" system than the current GNU Make based system. Unfortunately, my preliminary results don't back this claim, and actually make Hadrian appear to be significantly slower. ---- Here's the summary of the results: | Hadrian | GNU Make | +------------------------+-----------+-----------+ | Compiling `hadrian` | 5m25s | 0 | | (one-time setup) | | | +------------------------+-----------+-----------+ | build "all" at -j8 | 38m | 33m | +------------------------+-----------+-----------+ | no-op build at -j8 | 10.977s | 3.258s | +------------------------+-----------+-----------+ | "clean" | 21s | 51s | +------------------------+-----------+-----------+ So, Hadrian is ~5 minutes than GNU Make (or even ~10 minutes if you also count the one-time setup cost). And what I personally consider a bit annoying is that it's ~3 times slower detecting; i.e. you have to wait 11s for Hadrian to detect there's nothing to be done which compared to GNU Make (which currently needlessly re-runs Sphinx; so it could be even faster!) is very noticeable to me. There's a silver-lining though, deleting files is the part which is a lot more costly in the GNU Make system currently since artifacts are spread over several (scroll to the end of this email) subfolders there. Whereas Hadrian did something we should have done for the GNU Make system as well (and probably would have done sooner or later anyway in order to support the srcdir!=buildir scheme that people are used to from GNU Autotools projects); Hadrian places build-artifacts into a few top-level folders, and so cleaning up is trivial and requires to unlink only a few folders from the filesystem. ---- At the very least, I'd expect Hadrian to be as fast as the GNU Make system (and ideally beat it, not the least as this was besides maintainability its big promise), but so far it doesn't seem to deliver that promise for me. It could easily be that I'm comparing apples to oranges here or that I've otherwise overlooked something, so let me describe how I came to this conclusion: I tried this on an reasonably idle Linux workstation with an Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz CPU, and with 32GiB RAM (i.e. the filesystem content was well-cached into memory; NB: a ghc source tree + compiled artifacts takes up about 4GiB on the filesystem). I've started from a fresh Git clone, i.e. git clone --recursive git://git.haskell.org/ghc.git followed by ./boot ./configure At this point, we're at the common point from which both the Hadrian and the GNU Make build-system would start diverging: For the Hadrian build-system, we need to pay for a one-time setup, since we need to build the `hadrian` executable (which requires to build the in-tree lib:Cabal as an in-place library): $ time ./hadrian/build.sh --help ... real 5m25.992s user 6m19.196s sys 0m6.079s I'm not too worried about this part, as there's a few tricks by which we could likely bring that down to about 2 minutes or so, and we mostly pay this setup-cost, when lib:Cabal and/or `hadrian` changes and requires to be recompiled. Now, after having made sure that the `hadrian` executable is fresh, I started the actual build: $ time ./hadrian/build.sh -j8 ... shakeArgsWith 0.000s 0% Function shake 0.178s 0% Database read 0.000s 0% With database 0.000s 0% Running rules 2338.398s 99% ========================= Pool finished (5261 threads, 8 max) 0.002s 0% Lint checking 0.111s 0% Total 2338.690s 100% Build completed in 38:59m real 38m59.626s user 219m7.421s sys 11m7.584s then I immediately re-issued the same command to test how long it takes to perform a no-op build: $ time ./hadrian/build.sh -j8 Up to date Up to date shakeArgsWith 0.000s 0% Function shake 0.183s 1% Database read 0.144s 1% With database 0.241s 2% Running rules 9.379s 93% ========================= Pool finished (4165 threads, 8 max) 0.004s 0% Lint checking 0.101s 1% Total 10.051s 100% Build completed in 0:11m real 0m10.977s user 0m19.399s sys 0m2.443s Same without -j8: $ time ./hadrian/build.sh Up to date Up to date shakeArgsWith 0.000s 0% Function shake 0.175s 2% Database read 0.123s 1% With database 0.197s 2% Running rules 7.710s 92% ========================= Pool finished (1 threads, 1 max) 0.003s 0% Lint checking 0.084s 1% Total 8.293s 100% Build completed in 0:09m real 0m9.196s user 0m8.656s sys 0m0.724s And finally clean it again: $ sync $ time ./hadrian/build.sh clean Up to date Up to date | Remove directory _build/stage0 | Remove directory _build/stage1 | Remove directory _build/stage2 | Remove directory _build/stage3 | Remove directory inplace/bin | Remove directory inplace/lib | Remove directory sdistprep | Remove Hadrian files... | Remove directory _build/generated | Done. shakeArgsWith 0.000s 0% Function shake 0.175s 0% Database read 0.114s 0% With database 0.205s 0% Running rules 20.037s 97% ======================== Pool finished (1 threads, 1 max) 0.002s 0% Lint checking 0.006s 0% Total 20.540s 100% Build completed in 0:21m real 0m21.426s user 0m1.415s sys 0m1.045s ---- Running the full build via our rusty GNU Make system: $ time make V=0 -j8 real 33m30.801s user 157m49.520s sys 6m49.289s A no-op build: $ time make V=0 -j8 ===--- building phase 0 make --no-print-directory -f ghc.mk phase=0 phase_0_builds make[1]: Nothing to be done for 'phase_0_builds'. ===--- building phase 1 make --no-print-directory -f ghc.mk phase=1 phase_1_builds make[1]: Nothing to be done for 'phase_1_builds'. ===--- building final phase make --no-print-directory -f ghc.mk phase=final all make -C utils/haddock/doc html SPHINX_BUILD=/usr/bin/sphinx-build /usr/bin/sphinx-build -b html . .build-html Running Sphinx v1.3.6 loading translations [en]... done loading pickled environment... done building [mo]: targets for 0 po files that are out of date building [html]: targets for 0 source files that are out of date updating environment: 0 added, 0 changed, 0 removed looking for now-outdated files... none found no targets are out of date. build succeeded. cp -R utils/haddock/doc/.build-html utils/haddock/doc/haddock real 0m3.258s user 0m3.134s sys 0m0.283s And finally `make clean`: $ sync $ time make clean make --no-print-directory -f ghc.mk clean CLEANING=YES "rm" -rf inplace/bin inplace/lib "rm" -rf utils/touchy/dist "rm" -rf inplace/lib/bin/touchy "rm" -rf utils/unlit/dist "rm" -rf inplace/lib/bin/unlit "rm" -rf utils/unlit/dist-install "rm" -rf utils/hp2ps/dist "rm" -rf inplace/bin/hp2ps "rm" -rf inplace/lib/bin/hp2ps "rm" -rf utils/hp2ps/dist-install "rm" -rf driver/split/dist inplace/lib/bin/ghc-split "rm" -rf utils/genprimopcode/dist "rm" -rf inplace/bin/genprimopcode "rm" -rf libffi/build libffi/stamp.ffi.static-shared.configure libffi/stamp.ffi.static-shared.build libffi/stamp.ffi.static-shared.install libffi/dist-install "rm" -rf utils/deriveConstants/dist "rm" -rf inplace/bin/deriveConstants "rm" -rf includes/ghcautoconf.h includes/ghcplatform.h includes/ghcversion.h "rm" -rf rts/dist "rm" -rf bindisttest/"install dir" bindisttest/HelloWorld bindisttest/HelloWorld.o bindisttest/HelloWorld.hi bindisttest/output "rm" -rf utils/genapply/dist "rm" -rf inplace/bin/genapply "rm" -rf libraries/integer-gmp/include/ghc-gmp.h libraries/integer-gmp/gmp/config.mk libraries/integer-gmp/gmp/libgmp.a libraries/integer-gmp/gmp/gmp.h libraries/integer-gmp/gmp/gmpbuild libraries/integer-gmp/gmp/gmp-6.1.2 "rm" -rf libraries/integer-gmp/gmp/objs "rm" -rf libraries/integer-gmp/gmp/gmpbuild "rm" -rf utils/haddock/dist "rm" -rf inplace/bin/haddock "rm" -rf inplace/lib/bin/haddock "rm" -rf compiler/stage1 "rm" -rf compiler/stage2 "rm" -rf compiler/stage3 "rm" -rf utils/hsc2hs/dist "rm" -rf inplace/bin/hsc2hs "rm" -rf inplace/lib/bin/hsc2hs "rm" -rf utils/hsc2hs/dist-install "rm" -rf utils/ghc-pkg/dist "rm" -rf inplace/bin/ghc-pkg "rm" -rf inplace/lib/bin/ghc-pkg "rm" -rf utils/ghc-pkg/dist-install "rm" -rf utils/ghctags/dist-install "rm" -rf inplace/bin/ghctags "rm" -rf utils/check-api-annotations/dist-install "rm" -rf inplace/bin/check-api-annotations "rm" -rf utils/check-ppr/dist-install "rm" -rf inplace/bin/check-ppr "rm" -rf utils/ghc-cabal/dist bootstrapping "rm" -rf utils/ghc-cabal/dist-install "rm" -rf utils/hpc/dist-install "rm" -rf inplace/bin/hpc "rm" -rf inplace/lib/bin/hpc "rm" -rf utils/runghc/dist-install "rm" -rf inplace/bin/runghc "rm" -rf inplace/lib/bin/runghc "rm" -rf ghc/stage1 "rm" -rf inplace/bin/ghc-stage1 "rm" -rf inplace/lib/bin/ghc-stage1 "rm" -rf ghc/stage2 "rm" -rf inplace/bin/ghc-stage2 "rm" -rf inplace/lib/bin/ghc-stage2 "rm" -rf ghc/stage3 "rm" -rf docs/users_guide/.doctrees-html/ docs/users_guide/.doctrees-pdf/ docs/users_guide/build-html/ docs/users_guide/build-pdf/ docs/users_guide/users_guide.pdf "rm" -rf docs/users_guide/.doctrees-man/ docs/users_guide/build-man/ "rm" -rf utils/count_lines/dist inplace/bin/count_lines "rm" -rf utils/compare_sizes/dist-install "rm" -rf iserv/stage2 "rm" -rf inplace/lib/bin/ghc-iserv "rm" -rf iserv/stage2_p "rm" -rf inplace/lib/bin/ghc-iserv-prof "rm" -rf iserv/stage2_dyn "rm" -rf inplace/lib/bin/ghc-iserv-dyn "rm" -f libraries/integer-gmp/include/HsIntegerGmp.h libraries/base/include/EventConfig.h mk/config.mk.old mk/project.mk.old compiler/ghc.cabal.old includes/GHCConstants.h includes/DerivedConstants.h includes/ghcautoconf.h includes/ghcplatform.h includes/ghcversion.h utils/ghc-pkg/Version.hs compiler/prelude/primops.txt "rm" -rf includes/dist-derivedconstants "rm" -rf inplace/bin "rm" -rf inplace/lib "rm" -rf libraries/bootstrapping.conf "rm" -f mk/are-validating.mk "rm" -rf libraries/ghc-boot-th/dist-install "rm" -rf libraries/ghc-boot/dist-install "rm" -rf libraries/ghci/dist-install "rm" -rf libraries/base/dist-install "rm" -rf libraries/ghc-prim/dist-install "rm" -rf libraries/integer-gmp/dist-install "rm" -rf libraries/integer-simple/dist-install "rm" -rf libraries/template-haskell/dist-install "rm" -rf libraries/array/dist-install "rm" -rf libraries/binary/dist-install "rm" -rf libraries/bytestring/dist-install "rm" -rf libraries/Cabal/Cabal/dist-install "rm" -rf libraries/ghc-compact/dist-install "rm" -rf libraries/containers/dist-install "rm" -rf libraries/deepseq/dist-install "rm" -rf libraries/directory/dist-install "rm" -rf libraries/filepath/dist-install "rm" -rf libraries/haskeline/dist-install "rm" -rf libraries/hpc/dist-install "rm" -rf libraries/mtl/dist-install "rm" -rf libraries/parsec/dist-install "rm" -rf libraries/pretty/dist-install "rm" -rf libraries/process/dist-install "rm" -rf libraries/terminfo/dist-install "rm" -rf libraries/text/dist-install "rm" -rf libraries/time/dist-install "rm" -rf libraries/transformers/dist-install "rm" -rf libraries/unix/dist-install "rm" -rf libraries/Win32/dist-install "rm" -rf libraries/xhtml/dist-install "rm" -rf libraries/parallel/dist-install "rm" -rf libraries/stm/dist-install "rm" -rf libraries/random/dist-install "rm" -rf libraries/primitive/dist-install "rm" -rf libraries/vector/dist-install "rm" -rf libraries/dph/dph-base/dist-install "rm" -rf libraries/dph/dph-prim-interface/dist-install "rm" -rf libraries/dph/dph-prim-seq/dist-install "rm" -rf libraries/dph/dph-prim-par/dist-install "rm" -rf libraries/dph/dph-lifted-base/dist-install "rm" -rf libraries/dph/dph-lifted-boxed/dist-install "rm" -rf libraries/dph/dph-lifted-copy/dist-install "rm" -rf libraries/dph/dph-lifted-vseg/dist-install "rm" -rf libraries/ghc-boot-th/dist-boot "rm" -rf libraries/ghc-boot/dist-boot "rm" -rf libraries/ghci/dist-boot "rm" -rf libraries/base/dist-boot "rm" -rf libraries/ghc-prim/dist-boot "rm" -rf libraries/integer-gmp/dist-boot "rm" -rf libraries/integer-simple/dist-boot "rm" -rf libraries/template-haskell/dist-boot "rm" -rf libraries/array/dist-boot "rm" -rf libraries/binary/dist-boot "rm" -rf libraries/bytestring/dist-boot "rm" -rf libraries/Cabal/Cabal/dist-boot "rm" -rf libraries/ghc-compact/dist-boot "rm" -rf libraries/containers/dist-boot "rm" -rf libraries/deepseq/dist-boot "rm" -rf libraries/directory/dist-boot "rm" -rf libraries/filepath/dist-boot "rm" -rf libraries/haskeline/dist-boot "rm" -rf libraries/hpc/dist-boot "rm" -rf libraries/mtl/dist-boot "rm" -rf libraries/parsec/dist-boot "rm" -rf libraries/pretty/dist-boot "rm" -rf libraries/process/dist-boot "rm" -rf libraries/terminfo/dist-boot "rm" -rf libraries/text/dist-boot "rm" -rf libraries/time/dist-boot "rm" -rf libraries/transformers/dist-boot "rm" -rf libraries/unix/dist-boot "rm" -rf libraries/Win32/dist-boot "rm" -rf libraries/xhtml/dist-boot "rm" -rf libraries/parallel/dist-boot "rm" -rf libraries/stm/dist-boot "rm" -rf libraries/random/dist-boot "rm" -rf libraries/primitive/dist-boot "rm" -rf libraries/vector/dist-boot "rm" -rf libraries/dph/dph-base/dist-boot "rm" -rf libraries/dph/dph-prim-interface/dist-boot "rm" -rf libraries/dph/dph-prim-seq/dist-boot "rm" -rf libraries/dph/dph-prim-par/dist-boot "rm" -rf libraries/dph/dph-lifted-base/dist-boot "rm" -rf libraries/dph/dph-lifted-boxed/dist-boot "rm" -rf libraries/dph/dph-lifted-copy/dist-boot "rm" -rf libraries/dph/dph-lifted-vseg/dist-boot "rm" -rf libraries/ghc-boot-th/dist libraries/ghc-boot/dist libraries/ghci/dist libraries/base/dist libraries/ghc-prim/dist libraries/integer-gmp/dist libraries/integer-simple/dist libraries/template-haskell/dist libraries/array/dist libraries/binary/dist libraries/bytestring/dist libraries/Cabal/Cabal/dist libraries/ghc-compact/dist libraries/containers/dist libraries/deepseq/dist libraries/directory/dist libraries/filepath/dist libraries/haskeline/dist libraries/hpc/dist libraries/mtl/dist libraries/parsec/dist libraries/pretty/dist libraries/process/dist libraries/terminfo/dist libraries/text/dist libraries/time/dist libraries/transformers/dist libraries/unix/dist libraries/Win32/dist libraries/xhtml/dist libraries/parallel/dist libraries/stm/dist libraries/random/dist libraries/primitive/dist libraries/vector/dist libraries/dph/dph-base/dist libraries/dph/dph-prim-interface/dist libraries/dph/dph-prim-seq/dist libraries/dph/dph-prim-par/dist libraries/dph/dph-lifted-base/dist libraries/dph/dph-lifted-boxed/dist libraries/dph/dph-lifted-copy/dist libraries/dph/dph-lifted-vseg/dist "rm" -f libraries/base/base.buildinfo libraries/integer-gmp/integer-gmp.buildinfo libraries/terminfo/terminfo.buildinfo libraries/unix/unix.buildinfo "rm" -f libraries/ghc-boot-th/config.log libraries/ghc-boot/config.log libraries/ghci/config.log libraries/base/config.log libraries/ghc-prim/config.log libraries/integer-gmp/config.log libraries/integer-simple/config.log libraries/template-haskell/config.log libraries/array/config.log libraries/binary/config.log libraries/bytestring/config.log libraries/Cabal/Cabal/config.log libraries/ghc-compact/config.log libraries/containers/config.log libraries/deepseq/config.log libraries/directory/config.log libraries/filepath/config.log libraries/haskeline/config.log libraries/hpc/config.log libraries/mtl/config.log libraries/parsec/config.log libraries/pretty/config.log libraries/process/config.log libraries/terminfo/config.log libraries/text/config.log libraries/time/config.log libraries/transformers/config.log libraries/unix/config.log libraries/Win32/config.log libraries/xhtml/config.log libraries/parallel/config.log libraries/stm/config.log libraries/random/config.log libraries/primitive/config.log libraries/vector/config.log libraries/dph/dph-base/config.log libraries/dph/dph-prim-interface/config.log libraries/dph/dph-prim-seq/config.log libraries/dph/dph-prim-par/config.log libraries/dph/dph-lifted-base/config.log libraries/dph/dph-lifted-boxed/config.log libraries/dph/dph-lifted-copy/config.log libraries/dph/dph-lifted-vseg/config.log "rm" -f libraries/ghc-boot-th/config.status libraries/ghc-boot/config.status libraries/ghci/config.status libraries/base/config.status libraries/ghc-prim/config.status libraries/integer-gmp/config.status libraries/integer-simple/config.status libraries/template-haskell/config.status libraries/array/config.status libraries/binary/config.status libraries/bytestring/config.status libraries/Cabal/Cabal/config.status libraries/ghc-compact/config.status libraries/containers/config.status libraries/deepseq/config.status libraries/directory/config.status libraries/filepath/config.status libraries/haskeline/config.status libraries/hpc/config.status libraries/mtl/config.status libraries/parsec/config.status libraries/pretty/config.status libraries/process/config.status libraries/terminfo/config.status libraries/text/config.status libraries/time/config.status libraries/transformers/config.status libraries/unix/config.status libraries/Win32/config.status libraries/xhtml/config.status libraries/parallel/config.status libraries/stm/config.status libraries/random/config.status libraries/primitive/config.status libraries/vector/config.status libraries/dph/dph-base/config.status libraries/dph/dph-prim-interface/config.status libraries/dph/dph-prim-seq/config.status libraries/dph/dph-prim-par/config.status libraries/dph/dph-lifted-base/config.status libraries/dph/dph-lifted-boxed/config.status libraries/dph/dph-lifted-copy/config.status libraries/dph/dph-lifted-vseg/config.status "rm" -f libraries/base/include/HsBaseConfig.h libraries/process/include/HsProcessConfig.h libraries/unix/include/HsUnixConfig.h "rm" -rf libraries/dist-haddock "rm" -rf bindistprep/ test ! -d testsuite || make -C testsuite clean make[1]: Entering directory '/stuff3/work/GHC2/ghc/testsuite' make -C ./timeout clean make[2]: Entering directory '/stuff3/work/GHC2/ghc/testsuite/timeout' test ! -f Setup || ./Setup clean rm -f -rf install-inplace rm -f -f calibrate.out rm -f -f Setup Setup.exe Setup.hi Setup.o make[2]: Leaving directory '/stuff3/work/GHC2/ghc/testsuite/timeout' rm -f -f mk/*.o rm -f -f mk/*.hi rm -f -f mk/ghcconfig*.mk rm -f -f mk/ghc-config mk/ghc-config.exe rm -f -f driver/*.pyc make[1]: Leaving directory '/stuff3/work/GHC2/ghc/testsuite' real 0m50.990s user 0m0.496s sys 0m1.582s --

5 5