Rebuilding ghc changes interface hashes?
Dear ghc developers, I am currently rebuilding ghc-7.0.2 to upload some packaging changes. Unfortunately, this had the effect of changing the package id of "base", which would require rebuilding every Haskell library. In the times of ghc-6.12, rebuilding the compiler left the hashes intact. Is there anything new in ghc-7 that interferes here? Also, the initial upload was built using ghc-6.12.1, while the upload now is build with ghc-7.0.2. Given that it is a two-stage compiler, I assume that the output should be identical, but it seems not to be the case. Any idea what could be causing this? Greetings, Joachim -- Joachim "nomeata" Breitner Debian Developer nomeata@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C JID: nomeata@joachim-breitner.de | http://people.debian.org/~nomeata
On Tue, Apr 05, 2011 at 09:59:23PM +0530, Joachim Breitner wrote:
Also, the initial upload was built using ghc-6.12.1, while the upload now is build with ghc-7.0.2. Given that it is a two-stage compiler, I assume that the output should be identical, but it seems not to be the case.
Changing the bootstrapping compiler changes the ABIs. The same can happen if you interrupt and restart the build. http://hackage.haskell.org/trac/ghc/ticket/4012 Ciao, Kili
Hi, Am Dienstag, den 05.04.2011, 18:51 +0200 schrieb Matthias Kilian:
On Tue, Apr 05, 2011 at 09:59:23PM +0530, Joachim Breitner wrote:
Also, the initial upload was built using ghc-6.12.1, while the upload now is build with ghc-7.0.2. Given that it is a two-stage compiler, I assume that the output should be identical, but it seems not to be the case.
Changing the bootstrapping compiler changes the ABIs. The same can happen if you interrupt and restart the build.
thanks for the pointer. I wish I had known earlier (no offence!) and made an extra round of ghc uploads before uploading several dozend of libraries, which need to be rebuild now. Greetings, Joachim -- Joachim "nomeata" Breitner Debian Developer nomeata@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C JID: nomeata@joachim-breitner.de | http://people.debian.org/~nomeata
On 05/04/11 17:51, Matthias Kilian wrote:
On Tue, Apr 05, 2011 at 09:59:23PM +0530, Joachim Breitner wrote:
Also, the initial upload was built using ghc-6.12.1, while the upload now is build with ghc-7.0.2. Given that it is a two-stage compiler, I assume that the output should be identical, but it seems not to be the case.
Changing the bootstrapping compiler changes the ABIs. The same can happen if you interrupt and restart the build.
In general changes can creep in randomly, there's no direct link between changing the bootstrap compiler and getting different compilation results (as far as I know), but there are some small elements of randomness. I did manage to squash most causes of this, but a couple remain, see http://hackage.haskell.org/trac/ghc/wiki/Commentary/Compiler/RecompilationAv... there may be others that I don't know about. I would really like to fix this, I'll think about whether we can make a deterministic sort-order for bindings in Core, that would fix the CSE issue. Cheers, Simon
http://hackage.haskell.org/trac/ghc/ticket/4012
Ciao, Kili
_______________________________________________ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
On Tue, Apr 05, 2011 at 08:51:52PM +0100, Simon Marlow wrote:
On Tue, Apr 05, 2011 at 09:59:23PM +0530, Joachim Breitner wrote:
Also, the initial upload was built using ghc-6.12.1, while the upload now is build with ghc-7.0.2. Given that it is a two-stage compiler, I assume that the output should be identical, but it seems not to be the case.
Changing the bootstrapping compiler changes the ABIs. The same can happen if you interrupt and restart the build.
In general changes can creep in randomly, there's no direct link between changing the bootstrap compiler and getting different compilation results (as far as I know), but there are some small elements of randomness.
Oh, I confused those two problems again :-( But memories come back, and IIRC the problem with the bootstrapping compiler is caused by the "Booter version" (as printed by ghc --info) passed up to the stage2 compiler. Wasn't the plan (about a year ago) to remove this? If time permits, I'll send a patch in a few days, but I first have to follow the switch to git, and I also want to update our (OpenBSDs) ports to ghc-7.0.3 and hp-2011.2. Ciao, Kili
On 05/04/2011 23:14, Matthias Kilian wrote:
On Tue, Apr 05, 2011 at 08:51:52PM +0100, Simon Marlow wrote:
On Tue, Apr 05, 2011 at 09:59:23PM +0530, Joachim Breitner wrote:
Also, the initial upload was built using ghc-6.12.1, while the upload now is build with ghc-7.0.2. Given that it is a two-stage compiler, I assume that the output should be identical, but it seems not to be the case.
Changing the bootstrapping compiler changes the ABIs. The same can happen if you interrupt and restart the build.
In general changes can creep in randomly, there's no direct link between changing the bootstrap compiler and getting different compilation results (as far as I know), but there are some small elements of randomness.
Oh, I confused those two problems again :-( But memories come back, and IIRC the problem with the bootstrapping compiler is caused by the "Booter version" (as printed by ghc --info) passed up to the stage2 compiler. Wasn't the plan (about a year ago) to remove this?
I don't think that is actually causing any problems. It just means that stage2 claims to have been bootstrapped by whatever you used to build stage1. Maybe that was intentional (Ian?). Cheers, Simon
If time permits, I'll send a patch in a few days, but I first have to follow the switch to git, and I also want to update our (OpenBSDs) ports to ghc-7.0.3 and hp-2011.2.
Ciao, Kili
On Tue, Apr 05, 2011 at 08:51:52PM +0100, Simon Marlow wrote:
On 05/04/11 17:51, Matthias Kilian wrote:
On Tue, Apr 05, 2011 at 09:59:23PM +0530, Joachim Breitner wrote:
Also, the initial upload was built using ghc-6.12.1, while the upload now is build with ghc-7.0.2. Given that it is a two-stage compiler, I assume that the output should be identical, but it seems not to be the case.
Changing the bootstrapping compiler changes the ABIs. The same can happen if you interrupt and restart the build.
In general changes can creep in randomly, there's no direct link between changing the bootstrap compiler and getting different compilation results (as far as I know)
compiler/stage2/build/Config.hs includes cBooterVersion, so if you boot with a different version then you get different code => different ABI. We could just remove this. In theory, stage2 won't be affected by the bootstrapping compiler at all anyway. Alternatively, if we make a config file (as we were discussing for putting the paths to gcc and ar in) then it could go in there, and then wouldn't be part of the ABI. Thanks Ian
On 06/04/2011 12:40, Ian Lynagh wrote:
On Tue, Apr 05, 2011 at 08:51:52PM +0100, Simon Marlow wrote:
On 05/04/11 17:51, Matthias Kilian wrote:
On Tue, Apr 05, 2011 at 09:59:23PM +0530, Joachim Breitner wrote:
Also, the initial upload was built using ghc-6.12.1, while the upload now is build with ghc-7.0.2. Given that it is a two-stage compiler, I assume that the output should be identical, but it seems not to be the case.
Changing the bootstrapping compiler changes the ABIs. The same can happen if you interrupt and restart the build.
In general changes can creep in randomly, there's no direct link between changing the bootstrap compiler and getting different compilation results (as far as I know)
compiler/stage2/build/Config.hs includes cBooterVersion, so if you boot with a different version then you get different code => different ABI.
Oh, for the GHC package, yes. The complaint was that the hash for base changed though.
We could just remove this. In theory, stage2 won't be affected by the bootstrapping compiler at all anyway.
Right.
Alternatively, if we make a config file (as we were discussing for putting the paths to gcc and ar in) then it could go in there, and then wouldn't be part of the ABI.
Ok, I suppose we could do that. I lean towards avoiding it if we can though (global state is bad, one more thing to go wrong, etc. etc.). Cheers, Simon
On Wed, Apr 06, 2011 at 12:40:45PM +0100, Ian Lynagh wrote:
compiler/stage2/build/Config.hs includes cBooterVersion, so if you boot with a different version then you get different code => different ABI.
We could just remove this. In theory, stage2 won't be affected by the bootstrapping compiler at all anyway.
Alternatively, if we make a config file (as we were discussing for putting the paths to gcc and ar in) then it could go in there, and then wouldn't be part of the ABI.
Even if Simon prefers the config file approach (and there's still the question wether the booter version is useful at all), attached is an old patch in darcs format i found on my disk (dated april 24th 2010). Just in case anyone wants to play with it. I still have to find my noticeses about wether cBooterVersion affects more than only the ghc lib. Ciao, Kili
On Wed, Apr 06, 2011 at 08:44:33PM +0200, Matthias Kilian wrote:
I still have to find my noticeses about wether cBooterVersion affects more than only the ghc lib.
Did a quick test the other day; bootstrapping ghc-7.0.3 from ghc-6.12.3 with two different VERSION_DATE files. The only visible change was in the ghc library, so this part indeed isn't worth any work. Ciao, Kili
Hi, Am Freitag, den 08.04.2011, 22:00 +0200 schrieb Matthias Kilian:
On Wed, Apr 06, 2011 at 08:44:33PM +0200, Matthias Kilian wrote:
I still have to find my noticeses about wether cBooterVersion affects more than only the ghc lib.
Did a quick test the other day; bootstrapping ghc-7.0.3 from ghc-6.12.3 with two different VERSION_DATE files. The only visible change was in the ghc library, so this part indeed isn't worth any work.
so, in summary, there is no easy and obvious way to avoid the other interface changes (e.g. in base) that occur when building with 7.0.2 instead of 6.12, and therefore a double-round of uploads of ghc in Debian cannot be avoided? Greetings, Joachim -- Joachim "nomeata" Breitner Debian Developer nomeata@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C JID: nomeata@joachim-breitner.de | http://people.debian.org/~nomeata
Hi, Am Freitag, den 08.04.2011, 22:00 +0200 schrieb Matthias Kilian:
On Wed, Apr 06, 2011 at 08:44:33PM +0200, Matthias Kilian wrote:
I still have to find my noticeses about wether cBooterVersion affects more than only the ghc lib.
Did a quick test the other day; bootstrapping ghc-7.0.3 from ghc-6.12.3 with two different VERSION_DATE files. The only visible change was in the ghc library, so this part indeed isn't worth any work.
This is one of the changes causing a break in the base library. "-" is bootstrapped with 7.0.2, "+" was bootstrapped with 6.12. # diff -u <(ghc --show-iface /usr/lib/ghc-7.0.2/base-4.3.1.0/GHC/List.hi) <(ghc --show-iface /other/usr/lib/ghc-7.0.2/base-4.3.1.0/GHC/List.hi) --- /dev/fd/63 2011-04-09 05:14:12.454269915 +0000 +++ /dev/fd/62 2011-04-09 05:14:12.454269915 +0000 @@ -5,8 +5,8 @@ Way: Wanted [], got [] interface base:GHC.List 7002 - interface hash: 87193984226122c2dd5964fd86876196 - ABI hash: 8d7faf049b24c406583f0286cc44d7db + interface hash: d8ee4b841dafe12780c3b173e214b5c7 + ABI hash: 180a90cd1355a939180d28fcca979825 export-list hash: 0c27906a7932ade6018f397c78520a54 orphan hash: 693e9af84d3dfcc71e640e005bdc5e2e where @@ -151,7 +151,7 @@ case @ a ds of wild { [] -> z : x xs -> lgo (f z x) xs } } in lgo z0 xs0) -} -4314014978c443774e83654e714a6768 +a018d005d4132f400d13b973d8352563 foldr1 :: forall a. (a -> a -> a) -> [a] -> a {- Arity: 2, Strictness: LS, Unfolding: (\ @ a ds :: a -> a -> a ds1 :: [a] -> @@ -161,9 +161,9 @@ -> case @ a ds2 of wild1 { [] -> x : ipv ipv1 - -> ds x (GHC.List.foldr1_$sfoldr1 @ a ipv ipv1 ds) } }) -} -7e3de2c9e0866b189d8f94b9801e6793 - foldr1_$sfoldr1 :: forall a. a -> [a] -> (a -> a -> a) -> a + -> ds x (GHC.List.foldr1_$sfoldr1 @ a ds ipv ipv1) } }) -} +68a72d0de3128a684265d8108a56fd7b + foldr1_$sfoldr1 :: forall a. (a -> a -> a) -> a -> [a] -> a {- Arity: 3, HasNoCafRefs, Strictness: LLL -} 328ef6e4bc937a0ad2498240a9327c37 foldr2 :: forall a b c. (a -> b -> c -> c) -> c -> [a] -> [b] -> c @@ -489,10 +489,10 @@ r :: b -> c (f x y) r) -} "SC:GHC.List.foldr10" [ALWAYS] forall @ a - sc :: a - sc1 :: [a] - sc2 :: a -> a -> a - GHC.List.foldr1 @ a sc2 (GHC.Types.: @ a sc sc1) + sc :: a -> a -> a + sc1 :: a + sc2 :: [a] + GHC.List.foldr1 @ a sc (GHC.Types.: @ a sc1 sc2) = GHC.List.foldr1_$sfoldr1 @ a sc sc1 sc2 "all/build" [ALWAYS] forall @ a p :: a -> GHC.Bool.Bool So something about the order of arguments changed. I don’t see how that should be influenced by the bootstrapping compiler, but I hope it is – if it were not any rebuild of ghc would risk changing the ABI hashes... Greetings, Joachim -- Joachim "nomeata" Breitner Debian Developer nomeata@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C JID: nomeata@joachim-breitner.de | http://people.debian.org/~nomeata
Hi again, Am Samstag, den 09.04.2011, 10:46 +0530 schrieb Joachim Breitner:
Am Freitag, den 08.04.2011, 22:00 +0200 schrieb Matthias Kilian:
On Wed, Apr 06, 2011 at 08:44:33PM +0200, Matthias Kilian wrote:
I still have to find my noticeses about wether cBooterVersion affects more than only the ghc lib.
Did a quick test the other day; bootstrapping ghc-7.0.3 from ghc-6.12.3 with two different VERSION_DATE files. The only visible change was in the ghc library, so this part indeed isn't worth any work.
So something about the order of arguments changed. I don’t see how that should be influenced by the bootstrapping compiler, but I hope it is – if it were not any rebuild of ghc would risk changing the ABI hashes...
I just redid the experiment with 7.0.3 vs. 7.0.2, and now only the hash of the ghc package changed when building 7.0.3 with 7.0.3 instead of 7.0.2, but the other hashes stayed stable. Given that only few libraries use the ghc package, this is good enough for Debian. I’m uploading ghc-7.0.3 to Debian very shortly, but will not rebuild it before triggering rebuilds of all libraries. Greetings, Joachim -- Joachim "nomeata" Breitner Debian Developer nomeata@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C JID: nomeata@joachim-breitner.de | http://people.debian.org/~nomeata
On 09/04/11 06:16, Joachim Breitner wrote:
Hi,
Am Freitag, den 08.04.2011, 22:00 +0200 schrieb Matthias Kilian:
On Wed, Apr 06, 2011 at 08:44:33PM +0200, Matthias Kilian wrote:
I still have to find my noticeses about wether cBooterVersion affects more than only the ghc lib.
Did a quick test the other day; bootstrapping ghc-7.0.3 from ghc-6.12.3 with two different VERSION_DATE files. The only visible change was in the ghc library, so this part indeed isn't worth any work.
This is one of the changes causing a break in the base library. "-" is bootstrapped with 7.0.2, "+" was bootstrapped with 6.12.
# diff -u<(ghc --show-iface /usr/lib/ghc-7.0.2/base-4.3.1.0/GHC/List.hi)<(ghc --show-iface /other/usr/lib/ghc-7.0.2/base-4.3.1.0/GHC/List.hi) --- /dev/fd/63 2011-04-09 05:14:12.454269915 +0000 +++ /dev/fd/62 2011-04-09 05:14:12.454269915 +0000 @@ -5,8 +5,8 @@ Way: Wanted [], got [] interface base:GHC.List 7002 - interface hash: 87193984226122c2dd5964fd86876196 - ABI hash: 8d7faf049b24c406583f0286cc44d7db + interface hash: d8ee4b841dafe12780c3b173e214b5c7 + ABI hash: 180a90cd1355a939180d28fcca979825 export-list hash: 0c27906a7932ade6018f397c78520a54 orphan hash: 693e9af84d3dfcc71e640e005bdc5e2e where @@ -151,7 +151,7 @@ case @ a ds of wild { [] -> z : x xs -> lgo (f z x) xs } } in lgo z0 xs0) -} -4314014978c443774e83654e714a6768 +a018d005d4132f400d13b973d8352563 foldr1 :: forall a. (a -> a -> a) -> [a] -> a {- Arity: 2, Strictness: LS, Unfolding: (\ @ a ds :: a -> a -> a ds1 :: [a] -> @@ -161,9 +161,9 @@ -> case @ a ds2 of wild1 { [] -> x : ipv ipv1 - -> ds x (GHC.List.foldr1_$sfoldr1 @ a ipv ipv1 ds) } }) -} -7e3de2c9e0866b189d8f94b9801e6793 - foldr1_$sfoldr1 :: forall a. a -> [a] -> (a -> a -> a) -> a + -> ds x (GHC.List.foldr1_$sfoldr1 @ a ds ipv ipv1) } }) -} +68a72d0de3128a684265d8108a56fd7b + foldr1_$sfoldr1 :: forall a. (a -> a -> a) -> a -> [a] -> a {- Arity: 3, HasNoCafRefs, Strictness: LLL -} 328ef6e4bc937a0ad2498240a9327c37 foldr2 :: forall a b c. (a -> b -> c -> c) -> c -> [a] -> [b] -> c @@ -489,10 +489,10 @@ r :: b -> c (f x y) r) -} "SC:GHC.List.foldr10" [ALWAYS] forall @ a - sc :: a - sc1 :: [a] - sc2 :: a -> a -> a - GHC.List.foldr1 @ a sc2 (GHC.Types.: @ a sc sc1) + sc :: a -> a -> a + sc1 :: a + sc2 :: [a] + GHC.List.foldr1 @ a sc (GHC.Types.: @ a sc1 sc2) = GHC.List.foldr1_$sfoldr1 @ a sc sc1 sc2 "all/build" [ALWAYS] forall @ a p :: a -> GHC.Bool.Bool
So something about the order of arguments changed. I don’t see how that should be influenced by the bootstrapping compiler, but I hope it is – if it were not any rebuild of ghc would risk changing the ABI hashes...
That's right, the ABI hashes are not guaranteed to be stable, and in practice do change unpredictably. While this doesn't affect correctness, I realise it's a problem for you, and it's something we'd like to fix. The case above looks like a new one I didn't know about, so I've added it to the wiki. In fact, I'm pretty impressed that we manage to get the same hashes even sometimes. Do you have any idea how many bits have to be in the right order for that to happen :-) Cheers, Simon
Hi, Am Samstag, den 09.04.2011, 20:18 +0100 schrieb Simon Marlow:
That's right, the ABI hashes are not guaranteed to be stable, and in practice do change unpredictably. While this doesn't affect correctness, I realise it's a problem for you, and it's something we'd like to fix. The case above looks like a new one I didn't know about, so I've added it to the wiki.
In fact, I'm pretty impressed that we manage to get the same hashes even sometimes. Do you have any idea how many bits have to be in the right order for that to happen :-)
anyways, rather changing hashes than installed packages breaking... so we are still happy about the hashes. Nevertheless, any efforts to make them more stable are appreciated. After all, it helps to safe the climate, as any unforseen change can cause rebuilds of >200 packages on
10 architectures, burning a lot of Watts ;-)
Greetings, Joachim -- Joachim "nomeata" Breitner Debian Developer nomeata@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C JID: nomeata@joachim-breitner.de | http://people.debian.org/~nomeata
participants (4)
-
Ian Lynagh -
Joachim Breitner -
Matthias Kilian -
Simon Marlow