Package abi hash and interface file versions
Hi, in Debian, we use the ghc-pkg ABI hashes to find out what packages we need to rebuild when. So far, the system has worked great. But now something unfortunate happened: Yesterday, I upgraded ghc from 7.0.3 to 7.0.4. The ABI hash of base and other libraries changed only on amd64 and kfreebsd-amd64, while they stays unmodified on the other architectures. That by itself is interesting, but nothing to worry about, as I assumed that the hashes ensure that packages are rebuild where required. But now I see this build log: https://buildd.debian.org/status/fetch.php?pkg=haskell-mtlparse&ver=0.1.2-1&arch=powerpc&stamp=1310414417 with the relevant line: Text/ParserCombinators/MTLParse/MTLParseCore.hs:69:1: Bad interface file: /usr/lib/haskell-packages/ghc/lib/mtl-2.0.1.0/ghc-7.0.3/Control/Monad/Trans.hi mismatched interface file versions (wanted "7004", got "7003") Obviously, the ghc version is encoded into the .hi file and checked before using it, including the minor patch level. But it seems that this bit of information is not included in the calculation of the interface hash! So although the ABI hash did not change between 7.0.3 and 7.0.4, the package is broken. I see two solutions for this problem: a) (preferred) The ghc version number is not encoded in the .hi file any more. Instead, a version counter is used that is manually incremented if there has been an incompatible change to the interface – after all, not every minor release of ghc changes this format. b) The interface file version should be included in the hash generation, so that a package built against the base package of 7.0.3 will clearly state that it does not work with the base package of 7.0.4 and needs to be rebuild. (Actually, b) should be the case even if a) is done). Thanks, Joachim -- Joachim "nomeata" Breitner Debian Developer nomeata@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C JID: nomeata@joachim-breitner.de | http://people.debian.org/~nomeata
On 11/07/2011 21:56, Joachim Breitner wrote:
Hi,
in Debian, we use the ghc-pkg ABI hashes to find out what packages we need to rebuild when. So far, the system has worked great. But now something unfortunate happened: Yesterday, I upgraded ghc from 7.0.3 to 7.0.4. The ABI hash of base and other libraries changed only on amd64 and kfreebsd-amd64, while they stays unmodified on the other architectures. That by itself is interesting, but nothing to worry about, as I assumed that the hashes ensure that packages are rebuild where required.
But now I see this build log: https://buildd.debian.org/status/fetch.php?pkg=haskell-mtlparse&ver=0.1.2-1&arch=powerpc&stamp=1310414417 with the relevant line: Text/ParserCombinators/MTLParse/MTLParseCore.hs:69:1: Bad interface file: /usr/lib/haskell-packages/ghc/lib/mtl-2.0.1.0/ghc-7.0.3/Control/Monad/Trans.hi mismatched interface file versions (wanted "7004", got "7003")
I don't quite understand - how did 7.0.4 come to be using packages from 7.0.3? The packages go into different directories don't they?
Obviously, the ghc version is encoded into the .hi file and checked before using it, including the minor patch level. But it seems that this bit of information is not included in the calculation of the interface hash! So although the ABI hash did not change between 7.0.3 and 7.0.4, the package is broken.
I see two solutions for this problem: a) (preferred) The ghc version number is not encoded in the .hi file any more. Instead, a version counter is used that is manually incremented if there has been an incompatible change to the interface – after all, not every minor release of ghc changes this format.
b) The interface file version should be included in the hash generation, so that a package built against the base package of 7.0.3 will clearly state that it does not work with the base package of 7.0.4 and needs to be rebuild.
(Actually, b) should be the case even if a) is done).
Perhaps we should factor the version into the ABI hash. But I'd like to understand more about how things are going wrong for you - I think your use case is one we hadn't considered. Cheers, Simon
Dear Simon, Am Mittwoch, den 13.07.2011, 09:11 +0100 schrieb Simon Marlow:
On 11/07/2011 21:56, Joachim Breitner wrote:
in Debian, we use the ghc-pkg ABI hashes to find out what packages we need to rebuild when. So far, the system has worked great. But now something unfortunate happened: Yesterday, I upgraded ghc from 7.0.3 to 7.0.4. The ABI hash of base and other libraries changed only on amd64 and kfreebsd-amd64, while they stays unmodified on the other architectures. That by itself is interesting, but nothing to worry about, as I assumed that the hashes ensure that packages are rebuild where required.
But now I see this build log: https://buildd.debian.org/status/fetch.php?pkg=haskell-mtlparse&ver=0.1.2-1&arch=powerpc&stamp=1310414417 with the relevant line: Text/ParserCombinators/MTLParse/MTLParseCore.hs:69:1: Bad interface file: /usr/lib/haskell-packages/ghc/lib/mtl-2.0.1.0/ghc-7.0.3/Control/Monad/Trans.hi mismatched interface file versions (wanted "7004", got "7003")
I don't quite understand - how did 7.0.4 come to be using packages from 7.0.3? The packages go into different directories don't they?
not here, and not any more. I was assuming that I can use the same packages across the two versions and changed the path (ghclibdir) to make that happen, hoping to avoid needless recompilations. You are right that this cannot happen with default settings.
Perhaps we should factor the version into the ABI hash. But I'd like to understand more about how things are going wrong for you - I think your use case is one we hadn't considered.
Even if you do not plan to support modules across minor versions of ghc, I’d like to get that into the hash. This would save us the trouble of tracking which package was built with what version of ghc and the same rebuild logic would apply that already handles the case of updated package dependencies. Thanks, Joachim -- Joachim "nomeata" Breitner Debian Developer nomeata@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C JID: nomeata@joachim-breitner.de | http://people.debian.org/~nomeata
Hi, Am Mittwoch, den 13.07.2011, 14:09 +0200 schrieb Joachim Breitner:
Even if you do not plan to support modules across minor versions of ghc, I’d like to get that into the hash. This would save us the trouble of tracking which package was built with what version of ghc and the same rebuild logic would apply that already handles the case of updated package dependencies.
looking at the code, all that is needed would be to modify this file in ghc/Main.hs, and add something like the marked line to it (untested, as I do not have a partial build lying around, and my machine sometimes doesn’t take the heat from building ghc without crashing :-( ): abiHash :: [(String, Maybe Phase)] -> Ghc () abiHash strs = do hsc_env <- getSession let dflags = hsc_dflags hsc_env liftIO $ do let find_it str = do let modname = mkModuleName str r <- findImportedModule hsc_env modname Nothing case r of Found _ m -> return m _error -> ghcError $ CmdLineError $ showSDoc $ cannotFindInterface dflags modname r mods <- mapM find_it (map fst strs) let get_iface modl = loadUserInterface False (text "abiHash") modl ifaces <- initIfaceCheck hsc_env $ mapM get_iface mods bh <- openBinMem (3*1024) -- just less than a block put_ bh opt_HiVersion -- would adding this be sufficient mapM_ (put_ bh . mi_mod_hash) ifaces f <- fingerprintBinMem bh putStrLn (showSDoc (ppr f)) Haskell is currently broken in Debian on i386 and other arches and I’d like to get this fix in quickly, but of course not without upstream review. Greetings, Joachim -- Joachim "nomeata" Breitner Debian Developer nomeata@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C JID: nomeata@joachim-breitner.de | http://people.debian.org/~nomeata
On 14/07/2011 09:28, Joachim Breitner wrote:
Am Mittwoch, den 13.07.2011, 14:09 +0200 schrieb Joachim Breitner:
Even if you do not plan to support modules across minor versions of ghc, I’d like to get that into the hash. This would save us the trouble of tracking which package was built with what version of ghc and the same rebuild logic would apply that already handles the case of updated package dependencies.
looking at the code, all that is needed would be to modify this file in ghc/Main.hs, and add something like the marked line to it (untested, as I do not have a partial build lying around, and my machine sometimes doesn’t take the heat from building ghc without crashing :-( ):
abiHash :: [(String, Maybe Phase)] -> Ghc () abiHash strs = do hsc_env<- getSession let dflags = hsc_dflags hsc_env
liftIO $ do
let find_it str = do let modname = mkModuleName str r<- findImportedModule hsc_env modname Nothing case r of Found _ m -> return m _error -> ghcError $ CmdLineError $ showSDoc $ cannotFindInterface dflags modname r
mods<- mapM find_it (map fst strs)
let get_iface modl = loadUserInterface False (text "abiHash") modl ifaces<- initIfaceCheck hsc_env $ mapM get_iface mods
bh<- openBinMem (3*1024) -- just less than a block put_ bh opt_HiVersion -- would adding this be sufficient mapM_ (put_ bh . mi_mod_hash) ifaces f<- fingerprintBinMem bh
putStrLn (showSDoc (ppr f))
Haskell is currently broken in Debian on i386 and other arches and I’d like to get this fix in quickly, but of course not without upstream review.
So that will only affect the hash on the package, not the ABI hashes on individual modules. Maybe that's ok, but I have to think it through. Cheers, Simon
participants (2)
-
Joachim Breitner -
Simon Marlow