[GHC] #12485: -package-db flags now need to be sorted by dependency order

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Package | Version: 8.0.1 system | Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- After 39b71e81ec1044518f065d0055676d713521e483 putting the -package-db flags in the wrong order will result in a compile error: {{{ <command line>: cannot satisfy -package-id b-1-XXX: b-1-XXX is unusable due to missing or recursive dependencies: a-1-XXX (use -v for more information) }}} See attached file for a repro. It's reproduces in 8.0.1 and HEAD. Doesn't reproduce in GHC 7.10.3 or after reverting the above mentioned commit. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by niteria): * Attachment "repro.sh" added. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by niteria): I'm working on a fix, but I'll be of the grid next week. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): D2450 Wiki Page: | -------------------------------------+------------------------------------- Changes (by niteria): * differential: => D2450 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450 Wiki Page: | -------------------------------------+------------------------------------- Changes (by niteria): * differential: D2450 => phab:D2450 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.2 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450 Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * milestone: => 8.0.2 Comment: Any word on a fix, niteria? It would be nice to get the fix in to 8.0.2. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.2 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450 Wiki Page: | -------------------------------------+------------------------------------- Comment (by niteria): Sorry, I wanted to get in on this, but I ended up locally reverting the patch that broke it. I can't make any promises about fixing it this week, as I have some responsibilities that take priority over this this week. I want to get this fixed, but with the 8.0.2 coming soon I want to set the expectations straight. Is revert an option here for 8.0.2? It looks like the patch caused a bigger problem than it fixed. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.2 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450 Wiki Page: | -------------------------------------+------------------------------------- Comment (by ezyang): Sorry, this fell off my radar after niteria said that he was going to fix it. If you revert this patch, bootstrapping will break, so that's probably not a good idea. Let me see if I can fix it. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.2 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450 Wiki Page: | -------------------------------------+------------------------------------- Comment (by ezyang): A few things: First, niteria, I'm curious why you need top be able to pass the package databases out of order. If I understand correctly, neither Stack nor Cabal use the databases this way. Indeed, when I wrote this patch, I assumed the databases would be in the right order because that's how shadowing works: later databases shadow earlier ones. Second, it's impossible to do ABI sanity checking in the way it is done today if I can't assume things are in the right order. When the db order was assumed to be specified correctly, I could assume all references to p-0.1-xxx refer to the topmost package with this IPID in the Stack; if the ABIs mismatch then I should remove everyone below me. But if there might be a reference to that particular package in a later package database, this assumption doesn't hold. Assuming that we DO want databases to be specified in any order (do we?!) here's what I think we should do: We NEVER remove packages because something got shadowed. Either the shadower is ABI compatible (in which case we never need to eliminate anything), OR we immediately error (because we have no idea which ones to eliminate.) I don't know if this brings back enough shadowing to fix bootstrapping though; will have to test. #12518 seems related; it also requests that we error on shadowing. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.2 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450 Wiki Page: | -------------------------------------+------------------------------------- Comment (by niteria):
First, niteria, I'm curious why you need top be able to pass the package databases out of order. If I understand correctly, neither Stack nor Cabal use the databases this way.
I don't use Stack or Cabal to build the packages. I can't do that because I'm building packages needed to build `cabal-install`. The way the packages are built is with `Setup.hs` and a separate `package.db` for each package. The scripts I have would need some additional logic to pass the `-package-db` flags in dependency order. Once I get past that obstacle I'm not sure if Buck or Bazel do any dependency sorting on `-package-db` flags. Putting this constraint on every build system sounds suboptimal to me.
Indeed, when I wrote this patch, I assumed the databases would be in the right order because that's how shadowing works: later databases shadow earlier ones.
Second, it's impossible to do ABI sanity checking in the way it is done today if I can't assume things are in the right order. When the db order was assumed to be specified correctly, I could assume all references to
This is an undocumented change in behaviour in the very least. The manual didn't state that they can be in any order, but also didn't put any constraints on order. People trying to integrate this into their build systems will have no idea that this is the requirement. p-0.1-xxx refer to the topmost package with this IPID in the Stack; if the ABIs mismatch then I should remove everyone below me. But if there might be a reference to that particular package in a later package database, this assumption doesn't hold. This sounds like an implementation detail informing the specification. Is there a fundamental reason why the flags have to be ordered?
Assuming that we DO want databases to be specified in any order (do we?!) here's what I think we should do: We NEVER remove packages because something got shadowed. Either the shadower is ABI compatible (in which case we never need to eliminate anything), OR we immediately error (because we have no idea which ones to eliminate.) I don't know if this brings back enough shadowing to fix bootstrapping though; will have to test.
I'm not familiar with package shadowing and also don't know how it fixes bootstrapping. Your patch doesn't explain it either. If you could explain maybe it would move this issue forward. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

The way the packages are built is with Setup.hs and a separate
#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.2 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450 Wiki Page: | -------------------------------------+------------------------------------- Comment (by ezyang): OK. So the way I'll structure this is first describe some workarounds to work with the current behavior, and then assuming those workarounds don't work / are undesirable I'll try to comment on how we can make this work. package.db for each package. One thing you can do in this situation is use ghc-pkg recache to create a merged package database and pass that off to GHC. So you'd swizzle all the text files into a directory, make the db, and you'd be off to the races.
This is an undocumented change in behaviour in the very least. The manual didn't state that they can be in any order, but also didn't put any constraints on order.
To be fair, the manual does state that package databases are ordered, and that packages closer to the top will shadow those below them (https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/packages.htm... #package-databases). It doesn't explicitly state that every substack should be well-formed, but this is a constraint that `ghc-pkg` checks (if you're not forcing it to register). (FWIW, Harendrar posted a Diff https://phabricator.haskell.org/D2464 which should improve the docs here further.)
This sounds like an implementation detail informing the specification. Is there a fundamental reason why the flags have to be ordered?
OK, let me explain the shadowing situation in more detail, and also how the package database handling has changed in the recent few releases. There is a very important correctness constraint GHC enforces on the package databases that it reads in, which is there should not be two distinct packages with the same "key" (what constitutes a key has changed over time). This is pretty important because if two unit ids are equal, GHC assumes they are type equal: if there are two distinct packages (which could define totally types and functions) with the same key, GHC will mix them up and generate code that almost certainly will segfault. In GHC 7.8 and earlier, the key was just the package name plus version. So it was not that uncommon to have two package databases which defined the same package name and version. To keep things safe, GHC shadowed packages, throwing out packages with any conflicting source package IDs. Every package was also associated with an installed package id, which was derived directly off of the ABI of a package. When two installed package ids coincided, it was always safe to pick one (the later one) because the coinciding installed package id meant that the ABIs matched, so there'd be no problem reusing it. OK, so this business where you can't have two packages with the same source package ID was the source of Cabal hell (cabal-install was not clever enough to put every package in a separate db) so in 7.10 we introduced "package keys", which were a bit more fine-grained than source package ids and what we used for type equality, linker symbols, etc. IPIDs continued to be derived off ABI hashes. Package keys didn't really necessitate any changes in how shadowing worked, since there still was a separate notion for IPIDs. At some point in the GHC 8.0 release cycle, SPJ was wondering why we need package keys and IPIDs. At the same time, work on cabal new-build was afoot, which eschewed the use of ABI hashes for IPIDs (since they couldn't be computed before we actually built the package; new-build needs to compute the IDs ahead of time so that it can determine if the particular build it needs is already built.) So in GHC 8.0 we unified IPIDs and package keys. OK, and now we get to the set of commits which broke database for you. So, when IPIDs don't track ABI hashes anymore, it's a bit more difficult to say what ABI a package depends on: after all, we record dependencies as IPIDs, not ABIs (maybe we should have recorded ABIs of the deps!) So, I needed to find a new algorithm which: 1. Maintained the safety invariant, that we never tried to load two distinct packages with the same IPID (previously package key, previously source package id), and never used the wrong copy of the package with a package that was compiled with a different package 2. Preserved the old shadowing behavior when the IPID conflicted when two package databases merged together--we need to prefer the latest one (I didn't want to implement this but bootstrapping stopped working without this. I believe the issue is that the distributed boot libraries with GHC don't come with hashes, so when we rebuild those libraries to boot, we end up picking the same name. Better use the new one!) 3. Preserved the old behavior where if you were ABI-compatible, you could override a package from the earlier database as long as the ABIs matched, without breaking a pile of packages. So... I sinned, and assumed that as we added databases to our stack, the database would continue to be well-formed. Which has ruined your day! Having written this, I don't think my suggested fix will keep GHC bootstrapping as it is today. It's a bit unavoidable: if the build system picks a deterministic id for a boot library, if you then immediately bootstrap with that GHC, it will pick the *same* id. So you *need* some form of shadowing. Maybe what you just want is another mode for GHC to make it process package databases differently? (This is why #12518 seems relevant.) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order
-------------------------------------+-------------------------------------
Reporter: niteria | Owner:
Type: bug | Status: new
Priority: normal | Milestone: 8.0.2
Component: Package system | Version: 8.0.1
Resolution: | Keywords:
Operating System: Unknown/Multiple | Architecture:
| Unknown/Multiple
Type of failure: None/Unknown | Test Case:
Blocked By: | Blocking:
Related Tickets: | Differential Rev(s): phab:D2450
Wiki Page: |
-------------------------------------+-------------------------------------
Comment (by Ben Gamari

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.2 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450 Wiki Page: | -------------------------------------+------------------------------------- Comment (by niteria): Thank you for your detailed response! I can't think of a better way of handling things, so I'm fine with just making it a bit more clear in the documentation. I didn't make the leap from understanding how shadowing is handled to how it impacts the expected order of `-package-db` flags. I shall make the test case not `expect_broken`, but expect the current behaviour. I believe it's useful to keep it. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.2 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450, Wiki Page: | phab:D2514 -------------------------------------+------------------------------------- Changes (by ezyang): * differential: phab:D2450 => phab:D2450, phab:D2514 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:12 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: patch Priority: normal | Milestone: 8.0.2 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450, Wiki Page: | phab:D2514 -------------------------------------+------------------------------------- Changes (by bgamari): * status: new => patch -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:13 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order
-------------------------------------+-------------------------------------
Reporter: niteria | Owner:
Type: bug | Status: patch
Priority: normal | Milestone: 8.0.2
Component: Package system | Version: 8.0.1
Resolution: | Keywords:
Operating System: Unknown/Multiple | Architecture:
| Unknown/Multiple
Type of failure: None/Unknown | Test Case:
Blocked By: | Blocking:
Related Tickets: | Differential Rev(s): phab:D2450,
Wiki Page: | phab:D2514
-------------------------------------+-------------------------------------
Comment (by Ben Gamari

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: patch Priority: normal | Milestone: 8.0.2 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450, Wiki Page: | phab:D2514 -------------------------------------+------------------------------------- Comment (by simonmar): @ezyang, I'd like to find a solution to this, because otherwise we need some pretty horrible hacks at our end to keep our package DBs ordered. I read your long description in comment:9 (twice!) but I'm afraid I still don't understand what change required this new ordering constraint on package DBs. The original commit (39b71e81ec1044518f065d0055676d713521e483) didn't explain the motivation either. My recollection of the algorithm before was that we throw all the packages together, resolve any clashes by using the package DB ordering, and then throw away packages whose deps are not satisfied, recursively. Could you explain what went wrong that caused GHC to not bootstrap? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:15 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: patch Priority: normal | Milestone: 8.0.2 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450, Wiki Page: | phab:D2514 -------------------------------------+------------------------------------- Comment (by ezyang): In the old world, it was OK to resolve clashes in the end because we assumed that if two packages had the same IPID, it would be OK to pick one arbitrarily. In other words, we assumed that equal IPIDs implies equal ABIs, making them interconvertible. We always computed the IPID by running `ghc --abi-hash`. Now, we wanted to stop computing IPIDs based on `--abi-hash`. The reason is that cabal-install wants to assign a unique IPID prior to compilation, based on all of the inputs to the compilation (in Nix-style local builds). ABI hashes get in the way because we find them out too late. In an ideal world, we could assume that if two packages have the same IPID, their ABI hashes are the same. But basically any package manager that is not cabal- install with Nix-style local builds can't really give a guarantee like this. So I decided we should just not assume that ABI hashes are the same, even if the IPIDs are the same. Now, the crux of the issue: it is no longer true that if two packages had the same IPID, it would be OK to pick one arbitrarily. That is only true if the ABIs are the same. If the ABIs are not the same, one needs to shadow the other. But how do we know which packages were compiled for package 1, as opposed to package 2? Without package database ordering, we don't know. (Remember with shadowing in GHC 7.8 was just making sure we didn't have conflicting package id derived symbol names; but since dependencies were always recorded as IPID, it was a pretty simple matter to figure out what to remove. Nothing ever got removed if there was an IPID conflict.) As I mentioned earlier, one way we can resolve this is by recording both the IPID and the ABI of the dependencies of each package. Then we can resolve conflicts at the very end. I suppose there are less invasive ideas where we play more fast and loose about how we combine package databases. But does this explain the situation? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:16 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.2 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450, Wiki Page: | phab:D2514 -------------------------------------+------------------------------------- Changes (by bgamari): * status: patch => new Comment: comment:14 merged to `ghc-8.0` as 44e823d39195f8cddd2ae4c587292a3639502050. Reopening due to last few comments. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:17 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.2 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450, Wiki Page: | phab:D2514 -------------------------------------+------------------------------------- Comment (by simonmar): Ok, I understand that IPID does not determine ABI any more, and dependencies only record IPID. This situation bothers me for a couple of reasons. * Now we are relying on the package-db ordering on the command line for dependency safety, whereas previously we could check because we had the ABIs of the dependencies. * We've lost the property that you can just union all the package DBs, which is a bit sad. The idea of a package DB "stack" was really just a temporary situation to deal with the fact that we couldn't link multiple package instances into the same binary so we had to have some way to resolve conflicts; my intention was that eventually the idea of a "stack" would just go away when it wasn't necessary any more. The problem is really that a dependency doesn't uniquely specify its target. We don't want dependencies to be GUIDs because something something determinism, but having a dependency be a pair of (ABI,IPID) as you suggest would be good enough. In the short term, would anything go wrong if we allowed an IPID anywhere in the stack to satisfy a dependency provided it was unique? And if a dependency is not unique, we resolve in the current way, using the DB ordering. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:18 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.3 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450, Wiki Page: | phab:D2514 -------------------------------------+------------------------------------- Changes (by bgamari): * milestone: 8.0.2 => 8.0.3 Comment: Pushing off to 8.0.3. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:19 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.3 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450, Wiki Page: | phab:D2514 -------------------------------------+------------------------------------- Comment (by ezyang):
Now we are relying on the package-db ordering on the command line for dependency safety, whereas previously we could check because we had the ABIs of the dependencies.
Yes, you are right. :(
The idea of a package DB "stack" was really just a temporary situation to deal with the fact that we couldn't link multiple package instances into the same binary so we had to have some way to resolve conflicts; my intention was that eventually the idea of a "stack" would just go away when it wasn't necessary any more.
And this is what I get for not being involved in the initial discussion ;) But it is hard for me to see how you could actually do this, because you can always end up in a situation where you have two separate packages with the same symbol name, and now you have to pick one.
In the short term, would anything go wrong if we allowed an IPID anywhere in the stack to satisfy a dependency provided it was unique? And if a dependency is not unique, we resolve in the current way, using the DB ordering.
Nothing wrong! It just sounds really annoying to implement. ;) (I guess... we do a pre-pass to figure out conflicts, and then run the shadowing algorithm?) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:20 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.3 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450, Wiki Page: | phab:D2514 -------------------------------------+------------------------------------- Comment (by simonmar):
And this is what I get for not being involved in the initial discussion ;) But it is hard for me to see how you could actually do this, because you can always end up in a situation where you have two separate packages with the same symbol name, and now you have to pick one.
That's true. In that case, either * The ABIs are the same, and we can just pick one, or * The ABIs are different, in which case we can either * report an error, or * Pick one to keep, and throw away everything that depends on the dropped one Reporting an error is bad - when we had something similar before (7.2? I forget exactly) you could get into a state where GHC fails to start because you have some conflict in your package database. That's why the GHC tries really hard to find a consistent set of packages by throwing things away. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:21 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.3 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450, Wiki Page: | phab:D2514 -------------------------------------+------------------------------------- Comment (by ezyang): FWIW, in #12518 Harendra wants to bring back erroring, but guarded by a flag ;) Note about timeline: I'm unlikely to be able to patchfix the quick workaround before the end of ICFP (on vacation). I can handle adding ABI hashes to depends though for 8.2. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:22 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.3 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450, Wiki Page: | phab:D2514 -------------------------------------+------------------------------------- Comment (by simonmar): If you're not working on it right now, I might hack on it on the plane to ICFP. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:23 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.3 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450, Wiki Page: | phab:D2514 -------------------------------------+------------------------------------- Comment (by ezyang): Feel free! -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:24 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.3 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450, Wiki Page: | phab:D2514 -------------------------------------+------------------------------------- Changes (by harendra): * cc: harendra (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:25 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.3 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450, Wiki Page: | phab:D2514 -------------------------------------+------------------------------------- Comment (by ezyang): Given that it didn't happen during this plane ride, I'm on it! -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:26 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.3 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450, Wiki Page: | phab:D2514 -------------------------------------+------------------------------------- Comment (by ezyang):
In the short term, would anything go wrong if we allowed an IPID anywhere in the stack to satisfy a dependency provided it was unique? And if a dependency is not unique, we resolve in the current way, using the DB ordering.
OK, while I was implementing this, I discovered a fairly nasty edge case. Consider: * db1: p (depends on q) * db2: q (depends on r from db3) * db3: r * db4: r (shadows r from db3) A "proper" order to specify these databases is db3 db2 db1 db4, which would have the net effect of only bringing r into scope (since everything else gets killed with db3's r gets shadowed.) Alternately, db4 db3 db2 db1 would see just db4 be killed. What do you expect to happen if the databases are specified: `db1 db3 db4 db2`? Under the "non-duplicated package can be specified anywhere" rule, `db1` is OK because it refers to `q`, which indeed is only mentioned once. OK, now the nastiness begins. Remember that we still need to process shadowing in order. So we accept `db1`, process `db3` (fine), and then process `db4`, at which point there is a problem. The r from db4 needs to shadow the db3 r, and invalidate anything that depends on it. This *somehow* has to include db1 and db4, but we haven't even "gotten" to db4 to process it yet. A giant mess! I don't really care about trying hard to salvage the situation, if a user comes into it. But what I do care about is giving an easily interpretable explanation about whatever behavior we do. But it doesn't seem like there is reasonable "non-magical" semantics I can give for this case. Here's an alternative: what if we allow you to order your `-package-db` flags however you want, but ONLY if there is NO shadowing anywhere in the stack? In that case, matters are simple. I am only a little less enthusiastic about this because you only need to have one case of irrelevant shadowing and boom your command line stops working. Guidance requested! What do you care about? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:27 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.3 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450, Wiki Page: | phab:D2514 -------------------------------------+------------------------------------- Comment (by simonmar): Ok, I'm still trying to get my head around this problem. Here's the algorithm I want to use: * for each package DB * add packages one by one to an accumulating `Map PackageId (Maybe Package)` * if we encounter a `PackageKey` that we already have: * If the ABIs are the same, replace the old one by the new one * If the ABIs are different, insert a `Nothing` entry in the map (and report an error with -v2) * If we already have a `Nothing` for this `PackageKey`, leave it as `Nothing` (and report an error with -v2) * Afterwards, remove any package from the DB with one or more missing dependencies, recursively until you can remove no more packages. (report what we're doing with -v2) Does anything go wrong with this? It's a slight variant of the algorithm that we used before `PackageKey`s I think. When we have dependencies that specify ABIs we can refine this to be more intelligent about what to throw away, but in the meantime I think this will be fine. It's very similar to what we used to do. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:28 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.3 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450, Wiki Page: | phab:D2514 -------------------------------------+------------------------------------- Comment (by ezyang): Under your scheme, if a unit id (package key in your comment) ever occurs twice in the database with different ABI, it is permanently "killed" and we never ever use it again. Other backwards incompatibility notwithstanding, I think this will break GHC bootstrapping itself. Here's the situation: 1. GHC builds and ships a pile of boot libraries and GHC with deterministically chosen unit ids; e.g., `transformers-0.5.2.0`. The primary reason for this is when you are compiling a development version of the compiler, you really want the symbol names of your libraries to stay fixed; otherwise, you'd have to recompile them every time you made a small edit. 2. When we start a GHC build, we build a few libraries and GHC using the bootstrapping compiler. When we compile these, we have to register these using the bootstrapping compiler. Now we have a shadowing situation! Because the unit ids are deterministically chosen, the boot libraries from the bootstrapping compiler will always have the same unit id as the boot libraries; the only way to get GHC to agree to it is to shadow (but the algorithm you gave above would just mark them as unusable.) But maybe this is the tail wagging the dog; what we should do is have the stage1 build have "stage1" appended to their unit ids so they don't shadow. The last time I looked at this (#11025) it seemed difficult to do. Maybe we should try again though. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:29 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.3 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450, Wiki Page: | phab:D2514 -------------------------------------+------------------------------------- Comment (by simonmar): * db1: p (depends on q) * db2: q (depends on r from db3) * db3: r * db4: r (shadows r from db3) I presume you mean that db3:r and db4:r have different ABIs, and q was built against db3:r. We are given these in the order db1, db3, db4, db2. Now, since q's dependency doesn't say which r it requires, we have to pick one, and the only sensible strategy is to pick the rightmost (db4:r). This is the wrong r, so we can get segfaults. Given that we can get segfaults even trying to do clever shadowing, I'm going to suggest that we don't do clever shadowing at all. * Just merge databases from left to right. When there are multiple packages with the same key, we get the rightmost one. We would accept the small possibility that if someone has somehow managed to create two packages with the same key but different ABIs (which is difficult with determinism) then GHC can produce a segfaulting binary. Too bad. It's unlikely, and as a user you can fix it by deleting or rebuilding packages. Furthermore it was always possible to screw up like this because the ABI hash is not a complete specification of a package - there are semantic differences not captured by the ABI hash. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:30 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.3 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2450, Wiki Page: | phab:D2514 -------------------------------------+------------------------------------- Comment (by ezyang): OK, I think this is a plan that we can actually do for 8.0.2 :) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:31 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: patch Priority: normal | Milestone: 8.0.3 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2613 Wiki Page: | -------------------------------------+------------------------------------- Changes (by ezyang): * status: new => patch * differential: phab:D2450, phab:D2514 => phab:D2613 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:32 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: patch Priority: normal | Milestone: 8.0.3 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2613 Wiki Page: | -------------------------------------+------------------------------------- Comment (by duncan): I'm struggling to get this. Seems to me that anything based on ABI is fishy since ABI is not a reliable indicator of identity. The 'id' field is for identity and the proper solution is to make that strong enough to be a proper identity. I'm sympathetic to @simonmar's point that it's nice to be able to just union all dbs. And that should work just fine provided that ids are strong enough. If your use case could in principle be handled by coping all the package.db/*.conf files from all the DBs into one DB then you're in a situation where there is only one instance of any pkgid already, and unordered unioning is fine. If we're using weak package ids, as in Simon's example in comment:30 then we're in deep trouble already. There's no way to know which r instance the q package needs. So if I understand correctly, @ezyang is proposing a kludge for this situation where we additionally record and stash the ABI use it also on all edges. But the ABI isn't enough. The proper solution imho is to distinguish the two 'r' instances by giving them different ids. But it doesn't sound like the situation @niteria has looks like this, right? I assume that you've got the ids actually unique, and you're not relying on shadowing. Your example would work just fine with unioning presumably. I don't mind ABI tracking as an additional sanity check (and it tells you when you could plausibly deliberately try switching deps without re- compiling), but I don't think we should be using it for anything important like disambiguating as part of db merging. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:33 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

But it doesn't sound like the situation @niteria has looks like this, right? I assume that you've got the ids actually unique, and you're not relying on shadowing. Your example would work just fine with unioning
#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: patch Priority: normal | Milestone: 8.0.3 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2613 Wiki Page: | -------------------------------------+------------------------------------- Comment (by niteria): presumably. Yes, I'm not relying on shadowing. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:34 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: patch Priority: normal | Milestone: 8.0.3 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2613 Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonmar): @duncan in practice the keys will often be strong enough. However, there are a couple of reasons why we can end up with package builds that have the same key but are incompatible. * Prior to GHC 8.0.2 non-determinism meant that the key did not determine the ABI * We don't want the key to be a function of the source code, because that would entail too much recompilation during development. For this reason, the GHC build system picks deterministic package keys, and I believe Cabal doesn't hash the source code to compute the package key (but I just gave up trying to confirm that after trawling through the Cabal source code for a few minutes which seems to have got terrifyingly complex since I last looked at it!) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:35 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: patch Priority: normal | Milestone: 8.0.3 Component: Package system | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2613 Wiki Page: | -------------------------------------+------------------------------------- Comment (by ezyang): bgamari notes that the cabal08 test will need to be adjusted once we fix this properly. We dropped the test for GHC 8.0 since it wasn't really testing something meaningful. The Cabal change is here https://github.com/haskell/cabal/pull/4008 but Duncan objected to computing ABI hashes on every build, so the patch is in limbo right now. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:36 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12485: -package-db flags now need to be sorted by dependency order
-------------------------------------+-------------------------------------
Reporter: niteria | Owner:
Type: bug | Status: patch
Priority: normal | Milestone: 8.0.3
Component: Package system | Version: 8.0.1
Resolution: | Keywords:
Operating System: Unknown/Multiple | Architecture:
| Unknown/Multiple
Type of failure: None/Unknown | Test Case:
Blocked By: | Blocking:
Related Tickets: | Differential Rev(s): phab:D2613
Wiki Page: |
-------------------------------------+-------------------------------------
Comment (by Edward Z. Yang

#12485: -package-db flags now need to be sorted by dependency order -------------------------------------+------------------------------------- Reporter: niteria | Owner: Type: bug | Status: closed Priority: normal | Milestone: 8.0.3 Component: Package system | Version: 8.0.1 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D2613 Wiki Page: | -------------------------------------+------------------------------------- Changes (by ezyang): * status: patch => closed * resolution: => fixed Comment: Now fixed in HEAD! -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12485#comment:38 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
participants (1)
-
GHC