Re: Non-deterministic package IDs are really bad in 7.10

newer
Re: Interactions between...

older
Re: [commit: ghc] master: Refactor...

Mateusz Kowalczyk

18 May 2015 18 May '15

10:13 p.m.

On 05/14/2015 05:43 PM, Joachim Breitner wrote:

...

Hi,

Am Sonntag, den 10.05.2015, 19:39 +0100 schrieb Mateusz Kowalczyk:

...
I'd like to bring some attention to ticket #4012 about non-determinism. As many of you may know, the nix package manager distributes binaries throughout its binary caches. The binaries are shared as long as the hash of some of their inputs matches: this means that we can end up with two of the same hashes of inputs but thanks to #4012 means that the actual contents can differ. You end up with machines with some packages built locally and some elsewhere and due to non-determinism, the GHC package IDs don't line up and everything is broken.

is NixOS sensitive to changes in the build directory. Debian is, and since 7.8 the build path creeps into the interface files and affects the hash: https://bugs.debian.org/785282

But you probably are not, otherwise you’d have complained when 7.8 was out, and not now :-)

Greetings, Joachim

Not a problem for nix, the paths are are calculated based on the derivation inputs. I don't want to go into details but if same expression goes in, same path comes out. If the path that comes out is different, you simply don't get to use it. More or less nix does ‘calculate the store path from this expression, check if it exists in binary caches and fetch it if it does, build if it does not’. So for us the store paths in the interface files would always be the same for the same inputs hence no problem there. -- Mateusz K.

Show replies by date

Daniel Peebles

19 May 19 May

1:46 a.m.

New subject: Non-deterministic package IDs are really bad in 7.10

Don't Nix builds all happen in a randomly generated temporary directory by default? Then you just copy to $out in installPhase. Perhaps stabilizing the build directory would help alleviate some of the immediate problems, if what Joachim is describing affects us too. On Mon, May 18, 2015 at 10:13 PM, Mateusz Kowalczyk

...

wrote:

...

On 05/14/2015 05:43 PM, Joachim Breitner wrote:

...
Hi,

Am Sonntag, den 10.05.2015, 19:39 +0100 schrieb Mateusz Kowalczyk:

...
I'd like to bring some attention to ticket #4012 about non-determinism. As many of you may know, the nix package manager distributes binaries throughout its binary caches. The binaries are shared as long as the hash of some of their inputs matches: this means that we can end up with two of the same hashes of inputs but thanks to #4012 means that the actual contents can differ. You end up with machines with some packages built locally and some elsewhere and due to non-determinism, the GHC package IDs don't line up and everything is broken.

is NixOS sensitive to changes in the build directory. Debian is, and since 7.8 the build path creeps into the interface files and affects the hash: https://bugs.debian.org/785282

But you probably are not, otherwise you’d have complained when 7.8 was out, and not now :-)

Greetings, Joachim

Not a problem for nix, the paths are are calculated based on the derivation inputs. I don't want to go into details but if same expression goes in, same path comes out. If the path that comes out is different, you simply don't get to use it. More or less nix does ‘calculate the store path from this expression, check if it exists in binary caches and fetch it if it does, build if it does not’.

So for us the store paths in the interface files would always be the same for the same inputs hence no problem there.

-- Mateusz K. _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Mateusz Kowalczyk

2:27 a.m.

New subject: Non-deterministic package IDs are really bad in 7.10

On 05/19/2015 06:46 AM, Daniel Peebles wrote:

...

Don't Nix builds all happen in a randomly generated temporary directory by default? Then you just copy to $out in installPhase. Perhaps stabilizing the build directory would help alleviate some of the immediate problems, if what Joachim is describing affects us too.

Oh, I was far too hasty in reading the bug on the Debian tracker. Yes, I suppose this could be a problem for us though I'd like to see it first before trying to work around it.

...

On Mon, May 18, 2015 at 10:13 PM, Mateusz Kowalczyk
...
wrote:

...
On 05/14/2015 05:43 PM, Joachim Breitner wrote:

...
Hi,

Am Sonntag, den 10.05.2015, 19:39 +0100 schrieb Mateusz Kowalczyk:

...
I'd like to bring some attention to ticket #4012 about non-determinism. As many of you may know, the nix package manager distributes binaries throughout its binary caches. The binaries are shared as long as the hash of some of their inputs matches: this means that we can end up with two of the same hashes of inputs but thanks to #4012 means that the actual contents can differ. You end up with machines with some packages built locally and some elsewhere and due to non-determinism, the GHC package IDs don't line up and everything is broken.

is NixOS sensitive to changes in the build directory. Debian is, and since 7.8 the build path creeps into the interface files and affects the hash: https://bugs.debian.org/785282

But you probably are not, otherwise you’d have complained when 7.8 was out, and not now :-)

Greetings, Joachim

Not a problem for nix, the paths are are calculated based on the derivation inputs. I don't want to go into details but if same expression goes in, same path comes out. If the path that comes out is different, you simply don't get to use it. More or less nix does ‘calculate the store path from this expression, check if it exists in binary caches and fetch it if it does, build if it does not’.

So for us the store paths in the interface files would always be the same for the same inputs hence no problem there.

--

-- Mateusz K.

Joachim Breitner

3:33 a.m.

New subject: Non-deterministic package IDs are really bad in 7.10

Hi, Am Dienstag, den 19.05.2015, 07:27 +0100 schrieb Mateusz Kowalczyk:

...

On 05/19/2015 06:46 AM, Daniel Peebles wrote:

...
Don't Nix builds all happen in a randomly generated temporary directory by default? Then you just copy to $out in installPhase. Perhaps stabilizing the build directory would help alleviate some of the immediate problems, if what Joachim is describing affects us too.

Oh, I was far too hasty in reading the bug on the Debian tracker. Yes, I suppose this could be a problem for us though I'd like to see it first before trying to work around it.

it’s easy to check: Just run $ ghc --show-iface /usr/lib/ghc/base-4.7.0.2/System/Info.hi|grep Dep addDependentFile "/build/ghc-LIQtDg/ghc-7.8.4/includes/ghcplatform.h" addDependentFile "libraries/base/dist-install/build/autogen/cabal_macros.h" addDependentFile "/usr/include/stdc-predef.h" If you see the build path appearing there, you are affected by the bug. Greetings, Joachim -- Joachim “nomeata” Breitner mail@joachim-breitner.de • http://www.joachim-breitner.de/ Jabber: nomeata@joachim-breitner.de • GPG-Key: 0xF0FBF51F Debian Developer: nomeata@debian.org

Mateusz Kowalczyk

20 May 20 May

2:01 p.m.

New subject: Non-deterministic package IDs are really bad in 7.10

On 05/19/2015 08:33 AM, Joachim Breitner wrote:

...

Hi,

Am Dienstag, den 19.05.2015, 07:27 +0100 schrieb Mateusz Kowalczyk:

...
On 05/19/2015 06:46 AM, Daniel Peebles wrote:

...
Don't Nix builds all happen in a randomly generated temporary directory by default? Then you just copy to $out in installPhase. Perhaps stabilizing the build directory would help alleviate some of the immediate problems, if what Joachim is describing affects us too.

Oh, I was far too hasty in reading the bug on the Debian tracker. Yes, I suppose this could be a problem for us though I'd like to see it first before trying to work around it.

it’s easy to check: Just run $ ghc --show-iface /usr/lib/ghc/base-4.7.0.2/System/Info.hi|grep Dep addDependentFile "/build/ghc-LIQtDg/ghc-7.8.4/includes/ghcplatform.h" addDependentFile "libraries/base/dist-install/build/autogen/cabal_macros.h" addDependentFile "/usr/include/stdc-predef.h"

Seems we're good?: [nix-shell:~]$ ghc --show-iface /nix/store/rnxk1a1bz4rgy1rw4973blbfp9f0ic89-ghc-7.8.4/lib/ghc-7.8.4/base-4.7.0.2/System/Info.hi | grep Dep addDependentFile "libraries/base/dist-install/build/autogen/cabal_macros.h" addDependentFile "/nix/store/6k9z1sfl7kghmagwd205k3i81pbcw57s-glibc-2.21/include/stdc-predef.h" With our new architecture using 7.10.1: [nix-shell:~]$ ghc --show-iface /nix/store/j50pk13r5jysqk20gsyjdi89qcpfii57-ghc-7.10.1/lib/ghc-7.10.1/base_I5BErHzyOm07EBNpKBEeUv/System/Info.hi | grep Dep addDependentFile "libraries/base/dist-install/build/autogen/cabal_macros.h" addDependentFile "/nix/store/c2zjp7bqmp5wvpkrpl7p7sfhxbdyfryy-glibc-2.21/include/stdc-predef.h"

...

If you see the build path appearing there, you are affected by the bug.

In light of above, I don't know if I should be happy because we don't seem to be affected or scared that it's just subtle and will bite us regardless.

...

Greetings, Joachim

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

-- Mateusz K.

3653

Age (days ago)

3654

Last active (days ago)

List overview

Download

4 comments

3 participants

participants (3)

Daniel Peebles
Joachim Breitner
Mateusz Kowalczyk