cabal keeps relinking
Dear Haskell-Cafe, cabal seems to be re-linking every time I run cabal build Wait was not an issue for small projects but for big ones it is starting to get in the way… How could I persuade cabal to re-link only if there are changes in the source tree and only these exes that require it? Many thanks, Semen PS I googled around and found this https://www.haskell.org/pipermail/libraries/2005-August/004306.html which seems to imply relink is (was?) conditional on multiple exes being present in the project. However, for me it was always relinking regardless. -- Семен Тригубенко http://trygub.com
Which cabal version is it? If older than 1.20, this is the fix that went into 1.20: https://github.com/haskell/cabal/issues/1177 On 07/11/14 16:35, Semen Trygubenko / Семен Тригубенко wrote:
cabal seems to be re-linking every time I run cabal build
On Fri, Nov 07, 2014 at 07:23:16PM +0100, Niklas Hambüchen wrote:
Which cabal version is it?
If older than 1.20, this is the fix that went into 1.20: https://github.com/haskell/cabal/issues/1177
$ cabal --version cabal-install version 1.20.0.3 using version 1.20.0.2 of the Cabal library -- Семен Тригубенко http://trygub.com
Could you detail how cabal relinks for you, e.g. with some cabal -v2 / -v3 output? On 07/11/14 21:09, Semen Trygubenko / Семен Тригубенко wrote:
On Fri, Nov 07, 2014 at 07:23:16PM +0100, Niklas Hambüchen wrote:
Which cabal version is it?
If older than 1.20, this is the fix that went into 1.20: https://github.com/haskell/cabal/issues/1177
$ cabal --version cabal-install version 1.20.0.3 using version 1.20.0.2 of the Cabal library
On Fri, Nov 07, 2014 at 10:16:31PM +0100, Niklas Hambüchen wrote:
Could you detail how cabal relinks for you, e.g. with some cabal -v2 / -v3 output?
The -vN outputs are verbose, so I ended up creating a minimalistic cabal package config that illustrates the problem (attached for reference — see cabalTest.tar.gz). This is how the non-verbose outputs look like for builds on FreeBSD: $ cabal build Package has never been configured. Configuring with default flags. If this fails, please run configure manually. Resolving dependencies... Configuring cabalTest-0.1.0.0... Building cabalTest-0.1.0.0... Preprocessing library cabalTest-0.1.0.0... [1 of 1] Compiling CabalTest ( src/CabalTest.hs, dist/build/CabalTest.o ) In-place registering cabalTest-0.1.0.0... Preprocessing executable 'cabalTest' for cabalTest-0.1.0.0... [1 of 2] Compiling CabalTest ( src/CabalTest.hs, dist/build/cabalTest/cabalTest-tmp/CabalTest.o ) [2 of 2] Compiling Main ( src/cabalTest.hs, dist/build/cabalTest/cabalTest-tmp/Main.o ) Linking dist/build/cabalTest/cabalTest ... $ cabal clean cleaning... $ cabal test Package has never been configured. Configuring with default flags. If this fails, please run configure manually. Resolving dependencies... Configuring cabalTest-0.1.0.0... Building cabalTest-0.1.0.0... Preprocessing library cabalTest-0.1.0.0... [1 of 1] Compiling CabalTest ( src/CabalTest.hs, dist/build/CabalTest.o ) In-place registering cabalTest-0.1.0.0... Preprocessing executable 'cabalTest' for cabalTest-0.1.0.0... [1 of 2] Compiling CabalTest ( src/CabalTest.hs, dist/build/cabalTest/cabalTest-tmp/CabalTest.o ) [2 of 2] Compiling Main ( src/cabalTest.hs, dist/build/cabalTest/cabalTest-tmp/Main.o ) Linking dist/build/cabalTest/cabalTest ... Preprocessing test suite 'CabalTestTest' for cabalTest-0.1.0.0... [1 of 1] Compiling Main ( test/cabalTest.hs, dist/build/CabalTestTest/CabalTestTest-tmp/Main.o ) Linking dist/build/CabalTestTest/CabalTestTest ... Running 1 test suites... Test suite CabalTestTest: RUNNING... Test suite CabalTestTest: PASS Test suite logged to: dist/test/cabalTest-0.1.0.0-CabalTestTest.log 1 of 1 test suites (1 of 1 test cases) passed. and then for rebuilds: $ cabal build Building cabalTest-0.1.0.0... Preprocessing library cabalTest-0.1.0.0... In-place registering cabalTest-0.1.0.0... Preprocessing executable 'cabalTest' for cabalTest-0.1.0.0... Linking dist/build/cabalTest/cabalTest ... Preprocessing test suite 'CabalTestTest' for cabalTest-0.1.0.0... Linking dist/build/CabalTestTest/CabalTestTest ... $ cabal test Building cabalTest-0.1.0.0... Preprocessing library cabalTest-0.1.0.0... In-place registering cabalTest-0.1.0.0... Preprocessing executable 'cabalTest' for cabalTest-0.1.0.0... Linking dist/build/cabalTest/cabalTest ... Preprocessing test suite 'CabalTestTest' for cabalTest-0.1.0.0... Linking dist/build/CabalTestTest/CabalTestTest ... Running 1 test suites... Test suite CabalTestTest: RUNNING... Test suite CabalTestTest: PASS Test suite logged to: dist/test/cabalTest-0.1.0.0-CabalTestTest.log 1 of 1 test suites (1 of 1 test cases) passed. Then I ran the following five commands: cabal clean cabal build cabal test cabal build cabal test three times --- first without, then with -v2 and -v3, on two platforms. The outputs are here: http://trygub.com/download/cabalTest-FreeBSD.out.gz http://trygub.com/download/cabalTest-Linux.out.gz The behaviour is different on Linux and FreeBSD. This is the diff of the non-verbose outputs: 10a11
[1 of 1] Compiling CabalTest ( src/CabalTest.hs, dist/build/CabalTest.p_o ) 14a16,17 [1 of 2] Compiling CabalTest ( src/CabalTest.hs, dist/build/cabalTest/cabalTest-tmp/CabalTest.p_o ) [2 of 2] Compiling Main ( src/cabalTest.hs, dist/build/cabalTest/cabalTest-tmp/Main.p_o ) 25d27 < Linking dist/build/cabalTest/cabalTest ... 27c29 < [1 of 1] Compiling Main ( test/cabalTest.hs, dist/build/CabalTestTest/CabalTestTest-tmp/Main.o )
[1 of 1] Compiling Main ( test/cabalTest.hs, dist/build/CabalTestTest/CabalTestTest-tmp/Main.p_o ) 39d40 < Linking dist/build/cabalTest/cabalTest ... 41d41 < Linking dist/build/CabalTestTest/CabalTestTest ... 47d46 < Linking dist/build/cabalTest/cabalTest ... 49d47 < Linking dist/build/CabalTestTest/CabalTestTest ...
So it seems that what I've observed is FreeBSD-specific — FreeBSD rebuild outputs have extra "Linking ..." messages in them… Many thanks for your help, Semen PS Linux $ cabal --version cabal-install version 1.18.0.2 using version 1.18.1.2 of the Cabal library FreeBSD $ cabal --version cabal-install version 1.20.0.3 using version 1.20.0.2 of the Cabal library -- Семен Тригубенко http://trygub.com
Ok, so it seems like it works on Linux, but not FreeBSD. To explain the situation a little bit: The compiled object files (.o) are linked together in an archive file (.a) by the `ar` program (similar to `tar`). The file entries in the .a file by default contain timestamps of the files, which usually means the mtime (last modification time). The build only looks at the .a file as a whole, and if its contents changed from the last build, it has to relink. Of course if a timestamp inside the .a changed, that's treated as "the .a file is different". Newer versions of `ar` have a -D option to avoid putting the mtimes of the .o files into the archive. But not all platforms supported by GHC/cabal support this flag. That's why we implemented a manual method to wipe the timestamps (set them to 0), so that they can no longer have an effect on the build. Seems like that doesn't work for FreeBSD. That's a shame, because we did take special care to make sure that it works cross-platform. You could help debug it: First, create an issue on cabal's Github so that we can track the bug, and explain the problem there and paste the output. Then try to find the craeated .a file (the -v outputs should help you where it is, something like libHSyourlibrary.a or similar), and observe how its contents change across different builds (you could use `cmp` to check if the files are byte-identical or something like `diff -u <(xxd firstbuild.a) <(xxd secondbuild.a)` to view the contents as a hex diff). Then try to see if this program I wrote: http://hackage.haskell.org/package/ar-timestamp-wiper makes the two .a files byte-identical. That will probably help us find the issue and get relinking-avoidance also on the BSDs. On 11/11/14 13:01, Semen Trygubenko / Семен Тригубенко wrote:
So it seems that what I've observed is FreeBSD-specific — FreeBSD rebuild outputs have extra "Linking ..." messages in them…
On Tue, Nov 11, 2014 at 08:28:26PM +0100, Niklas Hambüchen wrote:
Ok, so it seems like it works on Linux, but not FreeBSD.
To explain the situation a little bit:
The compiled object files (.o) are linked together in an archive file (.a) by the `ar` program (similar to `tar`). The file entries in the .a file by default contain timestamps of the files, which usually means the mtime (last modification time). The build only looks at the .a file as a whole, and if its contents changed from the last build, it has to relink. Of course if a timestamp inside the .a changed, that's treated as "the .a file is different".
Newer versions of `ar` have a -D option to avoid putting the mtimes of the .o files into the archive. But not all platforms supported by GHC/cabal support this flag. That's why we implemented a manual method to wipe the timestamps (set them to 0), so that they can no longer have an effect on the build.
Seems like that doesn't work for FreeBSD. That's a shame, because we did take special care to make sure that it works cross-platform.
Thanks for your reply!
You could help debug it:
First, create an issue on cabal's Github so that we can track the bug, and explain the problem there and paste the output.
Done — see https://github.com/haskell/cabal/issues/2384 . Also, I've re-run it with the latest cabal cabal-install version 1.22.0.0 using version 1.22.0.0 of the Cabal library and noticed a bit of a difference in behaviour this time. Namely, for the sequence I've ran (as before): cabal clean cabal build cabal test cabal build cabal test There's an improvement in both "cabal test" steps when moving to 1.22 in the sense that the following lines have disappeared from the output: Building cabalTest-0.1.0.0... Preprocessing executable 'cabalTest' for cabalTest-0.1.0.0... Linking dist/build/cabalTest/cabalTest ... (i.e., when running test target the exe itself (named cabalTest) is no longer relinked unnecessarily, only the test for it (named cabalTestTest). However, when compared with old output on Linux there are still some extra relinks present: For the second run of "cabal bulid" I still see Linking dist/build/cabalTest/cabalTest … Linking dist/build/CabalTestTest/CabalTestTest … For the second run of "cabal test" I still see Linking dist/build/CabalTestTest/CabalTestTest ... So it seems that the problem is still present in the latest version, it's just that test target no longer relinks the exes, only test exes.
Then try to find the created .a file (the -v outputs should help you where it is, something like libHSyourlibrary.a or similar), and observe how its contents change across different builds (you could use `cmp` to check if the files are byte-identical or something like `diff -u <(xxd firstbuild.a) <(xxd secondbuild.a)` to view the contents as a hex diff).
I ran this: cabal clean cabal build cp ./dist/build/libHScabalTest-0.1.0.0.a build.a cabal test cp ./dist/build/libHScabalTest-0.1.0.0.a test.a cabal build cp ./dist/build/libHScabalTest-0.1.0.0.a rebuild.a cabal test cp ./dist/build/libHScabalTest-0.1.0.0.a retest.a All .a files are identical.
Then try to see if this program I wrote: http://hackage.haskell.org/package/ar-timestamp-wiper makes the two .a files byte-identical.
Installed ar-timestamp-wiper-0.1.0 . Running e.g. ar-timestamp-wiper build.a does not change its md5.
That will probably help us find the issue and get relinking-avoidance also on the BSDs.
On 11/11/14 13:01, Semen Trygubenko / Семен Тригубенко wrote:
So it seems that what I've observed is FreeBSD-specific — FreeBSD rebuild outputs have extra "Linking ..." messages in them…
Let me know if I could be of further help! Thanks again, S. -- Семен Тригубенко http://trygub.com
participants (2)
-
Niklas Hambüchen -
Semen Trygubenko / Семен Тригубен ко