build ghc with make (ghc-boot-th broken deps)

Hi List, I pulled ghc repo (568d7279a) and trying to build with following env: Ghc from stack (8.8.3). I made links to ghc and ghc-pkg in /usr/bin ./boot is passing well, but ./configure breaks on ghc-boot step. I have no idea how could I fix this issue. I manually modified dep versions in to what I have for ghc-8.8.3 and issue with ghc-boot-th disappeared and appeared parsec libraries/bootstrapping.conf/binary-0.8.7.0.conf depends: array-0.5.4.0 base-4.13.0.0 bytestring-0.10.10.0 containers-0.6.2.1 but there was depends: array-0.5.3.0 base-4.12.0.0 bytestring-0.10.8.2 containers-0.6.0.1 Where is actual problem? Old cabal? 10:45$ cabal --version cabal-install version 1.24.0.2 compiled using version 1.24.2.0 of the Cabal library Configuring ghc-boot-th-8.11.0.20200520... "/usr/bin/ghc-pkg" update -v0 --force --package-db=libraries/bootstrapping.conf libraries/ghc-boot-th/dist-boot/inplace-pkg-config "inplace/bin/ghc-cabal" configure libraries/ghc-boot dist-boot --with-ghc="/usr/bin/ghc" --with-ghc-pkg="/usr/bin/ghc-pkg" --package-db=/home/dan/demo/haskell/compiler/ghc/libraries/bootstrapping.conf --disable-library-for-ghci --enable-library-vanilla --enable-library-for-ghci --disable-library-profiling --disable-shared --with-hscolour="/home/dan/.cabal/bin/HsColour" --configure-option=CFLAGS="-Wall -Werror=unused-but-set-variable -Wno-error=inline -iquote /home/dan/demo/haskell/compiler/ghc/libraries/ghc-boot" --configure-option=LDFLAGS=" " --configure-option=CPPFLAGS=" " --gcc-options="-Wall -Werror=unused-but-set-variable -Wno-error=inline -iquote /home/dan/demo/haskell/compiler/ghc/libraries/ghc-boot " --constraint "binary == 0.8.7.0" --constraint "transformers == 0.5.6.2" --constraint "mtl == 2.2.2" --constraint "hpc == 0.6.1.0" --constraint "ghc-boot-th == 8.11.0.20200520" --constraint "ghc-boot == 8.11.0.20200520" --constraint "template-haskell == 2.17.0.0" --constraint "text == 1.2.4.0" --constraint "parsec == 3.1.14.0" --constraint "Cabal == 3.3.0.0" --constraint "ghc-heap == 8.11.0.20200520" --constraint "exceptions == 0.10.4" --constraint "ghci == 8.11.0.20200520" --constraint "terminfo == 0.4.1.4" --with-gcc="gcc" --with-ld="ld.gold" --with-ar="ar" --with-alex="/home/dan/.cabal/bin/alex" --with-happy="/home/dan/.cabal/bin/happy" Configuring ghc-boot-8.11.0.20200520... Error: The following packages are broken because other packages they depend on are missing. These broken packages must be rebuilt before they can be used. installed package binary-0.8.7.0 is broken due to missing package array-0.5.3.0, base-4.12.0.0, bytestring-0.10.8.2, containers-0.6.0.1 libraries/ghc-boot/ghc.mk:3: recipe for target 'libraries/ghc-boot/dist-boot/package-data.mk' failed make[1]: *** [libraries/ghc-boot/dist-boot/package-data.mk] Error 1 Makefile:123: recipe for target 'all' failed make: *** [all] Error 2 -- Best regards, Daniil Iaitskov

Daneel Yaitskov
Hi List,
I pulled ghc repo (568d7279a) and trying to build with following env: Ghc from stack (8.8.3). I made links to ghc and ghc-pkg in /usr/bin
Hi Daneel,
./boot is passing well, but ./configure breaks on ghc-boot step. I have no idea how could I fix this issue.
Can you offer more detail here? How precisely does it break? Can you paste the full output from ./configure? Cheers, - Ben

Hi Ben,
Thanks for bothering.
I solved issue.
That was ancient cabal.
I build ghc successfully once I installed Cabal-3.
I thought that configure script will warn me about incompatible cabal :)
Btw could you shed some light on RTS locking?
I am interested in:
ccall traceUserBinaryMsg(MyCapability() "ptr", msg "ptr", len);
because it looks slower on smaller chunks.
Writing data in 64 bytes chunks vs 4k (ghc 8.11) is twice slower.
https://github.com/yaitskov/eventslog-benchmark
benchmark [ LoadWithBinary-b1048576-t001-c0064] lasted for cpu
4.23 ms real 4.24 ms
benchmark [ LoadWithBinary-b1048576-t001-c0128] lasted for cpu
3.97 ms real 3.91 ms
benchmark [ LoadWithBinary-b1048576-t001-c1024] lasted for cpu
2.73 ms real 2.61 ms
benchmark [ LoadWithBinary-b1048576-t001-c2048] lasted for cpu
2.34 ms real 2.23 ms
benchmark [ LoadWithBinary-b1048576-t001-c4096] lasted for cpu
2.16 ms real 2.20 ms
Eventlog messages tend to be small.
So application developer has to think about bufferization to speed up.
OpenTelemetry gains 6x speed improvement with following hack.
mm :: IO (IORef (Int, Builder))
mm = newIORef (0, mempty)
traceBuilder :: MonadIO m => (Int, Builder) -> m ()
traceBuilder !sizedBuilder@(size, builder) = do
liftIO $! do
tls <- G.mkTLS mm
ref <- G.getTLS tls
(!bufferedSize, !bufferedBuilders) <- readIORef ref
if bufferedSize + size > 1024
then do
traceBinaryEventIO . LBS.toStrict $! toLazyByteString bufferedBuilders
writeIORef ref sizedBuilder
else do
writeIORef ref (bufferedSize + size, builder <> bufferedBuilders)
I see that some EventLog methods use mutexes, but user message function don't.
It just appends to buffer and can flush if it is full.
So I don't see where locking happening.
MyCapability is just arithmetic
#define MyCapability() (BaseReg - OFFSET_Capability_r)
I guess capability locking happens somewhere upper - before C.
How to register RTS function, which would don't acquire capability lock?
I think locking for this case could be replaced with atomic variable.
C11 standard provides such IPC abstraction.
Thanks,
Daniil
On 5/20/20, Ben Gamari
Daneel Yaitskov
writes: Hi List,
I pulled ghc repo (568d7279a) and trying to build with following env: Ghc from stack (8.8.3). I made links to ghc and ghc-pkg in /usr/bin
Hi Daneel,
./boot is passing well, but ./configure breaks on ghc-boot step. I have no idea how could I fix this issue.
Can you offer more detail here? How precisely does it break? Can you paste the full output from ./configure?
Cheers,
- Ben
-- Best regards, Daniil Iaitskov

Daneel Yaitskov
Hi Ben,
Thanks for bothering. I solved issue. That was ancient cabal. I build ghc successfully once I installed Cabal-3. I thought that configure script will warn me about incompatible cabal :)
Hmm, interesting. I'm actually rather surprised that the problem was cabal-install. Afterall, GHC's build system doesn't use cabal-install apart from building Hadrian.
Btw could you shed some light on RTS locking?
I am interested in:
ccall traceUserBinaryMsg(MyCapability() "ptr", msg "ptr", len);
because it looks slower on smaller chunks. Writing data in 64 bytes chunks vs 4k (ghc 8.11) is twice slower.
https://github.com/yaitskov/eventslog-benchmark benchmark [ LoadWithBinary-b1048576-t001-c0064] lasted for cpu 4.23 ms real 4.24 ms benchmark [ LoadWithBinary-b1048576-t001-c0128] lasted for cpu 3.97 ms real 3.91 ms benchmark [ LoadWithBinary-b1048576-t001-c1024] lasted for cpu 2.73 ms real 2.61 ms benchmark [ LoadWithBinary-b1048576-t001-c2048] lasted for cpu 2.34 ms real 2.23 ms benchmark [ LoadWithBinary-b1048576-t001-c4096] lasted for cpu 2.16 ms real 2.20 ms
Eventlog messages tend to be small.
So application developer has to think about bufferization to speed up. OpenTelemetry gains 6x speed improvement with following hack.
... Interesting. Please do open a GitLab ticket to track this. You might also be interested in #17949, which is covers the case of traceEvent being expensive when tracing is disabled.
I see that some EventLog methods use mutexes, but user message function don't. It just appends to buffer and can flush if it is full. So I don't see where locking happening. MyCapability is just arithmetic
#define MyCapability() (BaseReg - OFFSET_Capability_r)
I guess capability locking happens somewhere upper - before C.
There is no locking necessary. Capabilities are owned by a particular operating system thread. The capability is held until the program yields it (which generally only happens during a GC or foreign function call) and during this time the program has full access to all of the state in the Capability structure without the need for a lock. This is why each Capability has its own eventlog buffer; it allows lock-free tracing. Cheers, - Ben
participants (2)
-
Ben Gamari
-
Daneel Yaitskov