[GHC] #10622: Rename Backpack packages to units
#10622: Rename Backpack packages to units -------------------------------------+------------------------------------- Reporter: ezyang | Owner: ezyang Type: task | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.11 Keywords: backpack | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Blocked By: Test Case: | Related Tickets: Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- After today's weekly Backpack call, we have come to the conclusion that we have two different types of "packages" in the Backpack world: 1. Cabal packages, which have a single `.cabal` file and are a unit of distribution which get uploaded to Hackage, and 2. Backpack packages, of which there may be multiple defined in a Backpack file shipped with a Cabal package; and are the building blocks for modular development in the small. It's really confusing to have both of these called packages: thus, we propose to rename all occurrences of Backpack package to unit. A Cabal package may contain MULTIPLE Cabal units, although old-style Cabal files will only define one unit. Here are some of the consequences: 1. We rename `PackageKey` to `UnitId`, as it identifies a unit rather than a Cabal package. (I think this actually makes the function of these identifiers clearer.) We'll also distinguish Cabal-file level `PackageName`s from Backpack-file `UnitName`s. Finally, any given unit will be uniquely identified by an `InstalledUnitId`. 2. The source-level syntax of Backpack files will use `unit` in place of where `package` was used before. 3. For backwards compatibility reasons, we'll sometimes arrange for `PackageName`/`UnitName` and `InstalledUnitId`/`InstalledPackageId` to coincide. Specifically, the unit of a package which has the same `UnitName` as the `PackageName` is treated specially: its `InstalledUnitId` is guaranteed to be the same as the `InstalledPackageId` and it is what is "visible" when a user uses old concepts such as `-package foo-0.1` to select a package. 4. The installed package database is extended to record some number of units per an installed package: the public facing API is that you can register a package CONTAINING some number of units. For old-style packages, there will be only one unit with a matching `UnitName`, so this will be isomorphic to an old style package; however, GHC will grow some new APIs for selecting specific units to bring into scope. GHC really mostly only cares about units, but if a user asks for a package using, e.g. `-package` it will translate this request into a request for the appropriate unit. 5. For old-style packages, Cabal will continue to write and register a package configuration file which implicitly defines a single unit. However, the plan is to give GHC the capacity to generate unit description files (like package description files, but per unit), which Cabal can interpret and use to register packages in the global database using a `ghc-pkg` which can register units and "unitless" package description files which don't implicitly define a unit. (NB: we must create unit description files, because `ghc-pkg recache` is expected to be able to regenerate the database.) (NB: we want to be able to add units for a package separately, because this is how units for indefinite packages are created as they are instantiated with new implementations.) 6. Cabal could also finally grow the "multiple libraries per single Cabal file" support people have wanted; it's just a stylized use of the Backpack facilities. The work plan: 1. Modify `bin-package-db` to reflect the unit/package split, but otherwise keep ghc-pkg and GHC the same (so old-style package description still supported, and interpreted as a package containing one unit.) Source modifications to GHC assume that a package only has one unit. 2. Generalize GHC to work with packages with multiple units 3. Add capability to Cabal/ghc-pkg to register just units. For compilation of Backpack files, GHC will write out units which Cabal will then install to the real registry 4. Backpack! -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10622 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10622: Rename Backpack packages to units -------------------------------------+------------------------------------- Reporter: ezyang | Owner: ezyang Type: task | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.11 Resolution: | Keywords: backpack Operating System: Unknown/Multiple | Architecture: Type of failure: None/Unknown | Unknown/Multiple Blocked By: | Test Case: Related Tickets: | Blocking: | Differential Revisions: -------------------------------------+------------------------------------- Description changed by ezyang: Old description:
After today's weekly Backpack call, we have come to the conclusion that we have two different types of "packages" in the Backpack world:
1. Cabal packages, which have a single `.cabal` file and are a unit of distribution which get uploaded to Hackage, and
2. Backpack packages, of which there may be multiple defined in a Backpack file shipped with a Cabal package; and are the building blocks for modular development in the small.
It's really confusing to have both of these called packages: thus, we propose to rename all occurrences of Backpack package to unit. A Cabal package may contain MULTIPLE Cabal units, although old-style Cabal files will only define one unit. Here are some of the consequences:
1. We rename `PackageKey` to `UnitId`, as it identifies a unit rather than a Cabal package. (I think this actually makes the function of these identifiers clearer.) We'll also distinguish Cabal-file level `PackageName`s from Backpack-file `UnitName`s. Finally, any given unit will be uniquely identified by an `InstalledUnitId`.
2. The source-level syntax of Backpack files will use `unit` in place of where `package` was used before.
3. For backwards compatibility reasons, we'll sometimes arrange for `PackageName`/`UnitName` and `InstalledUnitId`/`InstalledPackageId` to coincide. Specifically, the unit of a package which has the same `UnitName` as the `PackageName` is treated specially: its `InstalledUnitId` is guaranteed to be the same as the `InstalledPackageId` and it is what is "visible" when a user uses old concepts such as `-package foo-0.1` to select a package.
4. The installed package database is extended to record some number of units per an installed package: the public facing API is that you can register a package CONTAINING some number of units. For old-style packages, there will be only one unit with a matching `UnitName`, so this will be isomorphic to an old style package; however, GHC will grow some new APIs for selecting specific units to bring into scope. GHC really mostly only cares about units, but if a user asks for a package using, e.g. `-package` it will translate this request into a request for the appropriate unit.
5. For old-style packages, Cabal will continue to write and register a package configuration file which implicitly defines a single unit. However, the plan is to give GHC the capacity to generate unit description files (like package description files, but per unit), which Cabal can interpret and use to register packages in the global database using a `ghc-pkg` which can register units and "unitless" package description files which don't implicitly define a unit. (NB: we must create unit description files, because `ghc-pkg recache` is expected to be able to regenerate the database.) (NB: we want to be able to add units for a package separately, because this is how units for indefinite packages are created as they are instantiated with new implementations.)
6. Cabal could also finally grow the "multiple libraries per single Cabal file" support people have wanted; it's just a stylized use of the Backpack facilities.
The work plan:
1. Modify `bin-package-db` to reflect the unit/package split, but otherwise keep ghc-pkg and GHC the same (so old-style package description still supported, and interpreted as a package containing one unit.) Source modifications to GHC assume that a package only has one unit.
2. Generalize GHC to work with packages with multiple units
3. Add capability to Cabal/ghc-pkg to register just units. For compilation of Backpack files, GHC will write out units which Cabal will then install to the real registry
4. Backpack!
New description: After today's weekly Backpack call, we have come to the conclusion that we have two different types of "packages" in the Backpack world: 1. Cabal packages, which have a single `.cabal` file and are a unit of distribution which get uploaded to Hackage, and 2. Backpack packages, of which there may be multiple defined in a Backpack file shipped with a Cabal package; and are the building blocks for modular development in the small. It's really confusing to have both of these called packages: thus, we propose to rename all occurrences of Backpack package to unit. A Cabal package may contain MULTIPLE Cabal units, although old-style Cabal files will only define one unit. Here are some of the consequences: 1. We rename `PackageKey` to `UnitKey`, as it identifies a unit rather than a Cabal package. (I think this actually makes the function of these identifiers clearer.) We'll also distinguish Cabal-file level `PackageName`s from Backpack-file `UnitName`s. Finally, any given unit will be uniquely identified by an `InstalledUnitId`. 2. The source-level syntax of Backpack files will use `unit` in place of where `package` was used before. 3. For backwards compatibility reasons, we'll sometimes arrange for `PackageName`/`UnitName` and `InstalledUnitId`/`InstalledPackageId` to coincide. Specifically, the unit of a package which has the same `UnitName` as the `PackageName` is treated specially: its `InstalledUnitId` is guaranteed to be the same as the `InstalledPackageId` and it is what is "visible" when a user uses old concepts such as `-package foo-0.1` to select a package. 4. The installed package database is extended to record some number of units per an installed package: the public facing API is that you can register a package CONTAINING some number of units. For old-style packages, there will be only one unit with a matching `UnitName`, so this will be isomorphic to an old style package; however, GHC will grow some new APIs for selecting specific units to bring into scope. GHC really mostly only cares about units, but if a user asks for a package using, e.g. `-package` it will translate this request into a request for the appropriate unit. 5. For old-style packages, Cabal will continue to write and register a package configuration file which implicitly defines a single unit. However, the plan is to give GHC the capacity to generate unit description files (like package description files, but per unit), which Cabal can interpret and use to register packages in the global database using a `ghc-pkg` which can register units and "unitless" package description files which don't implicitly define a unit. (NB: we must create unit description files, because `ghc-pkg recache` is expected to be able to regenerate the database.) (NB: we want to be able to add units for a package separately, because this is how units for indefinite packages are created as they are instantiated with new implementations.) 6. Cabal could also finally grow the "multiple libraries per single Cabal file" support people have wanted; it's just a stylized use of the Backpack facilities. The work plan: 1. Modify `bin-package-db` to reflect the unit/package split, but otherwise keep ghc-pkg and GHC the same (so old-style package description still supported, and interpreted as a package containing one unit.) Source modifications to GHC assume that a package only has one unit. 2. Generalize GHC to work with packages with multiple units 3. Add capability to Cabal/ghc-pkg to register just units. For compilation of Backpack files, GHC will write out units which Cabal will then install to the real registry 4. Backpack! -- -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10622#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10622: Rename Backpack packages to units -------------------------------------+------------------------------------- Reporter: ezyang | Owner: ezyang Type: task | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.11 Resolution: | Keywords: backpack Operating System: Unknown/Multiple | Architecture: Type of failure: None/Unknown | Unknown/Multiple Blocked By: | Test Case: Related Tickets: | Blocking: | Differential Revisions: Phab:D1057 -------------------------------------+------------------------------------- Changes (by ezyang): * differential: => Phab:D1057 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10622#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10622: Rename Backpack packages to units -------------------------------------+------------------------------------- Reporter: ezyang | Owner: ezyang Type: task | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.11 Resolution: | Keywords: backpack Operating System: Unknown/Multiple | Architecture: Type of failure: None/Unknown | Unknown/Multiple Blocked By: | Test Case: Related Tickets: | Blocking: | Differential Revisions: Phab:D1057 -------------------------------------+------------------------------------- Description changed by simonpj: Old description:
After today's weekly Backpack call, we have come to the conclusion that we have two different types of "packages" in the Backpack world:
1. Cabal packages, which have a single `.cabal` file and are a unit of distribution which get uploaded to Hackage, and
2. Backpack packages, of which there may be multiple defined in a Backpack file shipped with a Cabal package; and are the building blocks for modular development in the small.
It's really confusing to have both of these called packages: thus, we propose to rename all occurrences of Backpack package to unit. A Cabal package may contain MULTIPLE Cabal units, although old-style Cabal files will only define one unit. Here are some of the consequences:
1. We rename `PackageKey` to `UnitKey`, as it identifies a unit rather than a Cabal package. (I think this actually makes the function of these identifiers clearer.) We'll also distinguish Cabal-file level `PackageName`s from Backpack-file `UnitName`s. Finally, any given unit will be uniquely identified by an `InstalledUnitId`.
2. The source-level syntax of Backpack files will use `unit` in place of where `package` was used before.
3. For backwards compatibility reasons, we'll sometimes arrange for `PackageName`/`UnitName` and `InstalledUnitId`/`InstalledPackageId` to coincide. Specifically, the unit of a package which has the same `UnitName` as the `PackageName` is treated specially: its `InstalledUnitId` is guaranteed to be the same as the `InstalledPackageId` and it is what is "visible" when a user uses old concepts such as `-package foo-0.1` to select a package.
4. The installed package database is extended to record some number of units per an installed package: the public facing API is that you can register a package CONTAINING some number of units. For old-style packages, there will be only one unit with a matching `UnitName`, so this will be isomorphic to an old style package; however, GHC will grow some new APIs for selecting specific units to bring into scope. GHC really mostly only cares about units, but if a user asks for a package using, e.g. `-package` it will translate this request into a request for the appropriate unit.
5. For old-style packages, Cabal will continue to write and register a package configuration file which implicitly defines a single unit. However, the plan is to give GHC the capacity to generate unit description files (like package description files, but per unit), which Cabal can interpret and use to register packages in the global database using a `ghc-pkg` which can register units and "unitless" package description files which don't implicitly define a unit. (NB: we must create unit description files, because `ghc-pkg recache` is expected to be able to regenerate the database.) (NB: we want to be able to add units for a package separately, because this is how units for indefinite packages are created as they are instantiated with new implementations.)
6. Cabal could also finally grow the "multiple libraries per single Cabal file" support people have wanted; it's just a stylized use of the Backpack facilities.
The work plan:
1. Modify `bin-package-db` to reflect the unit/package split, but otherwise keep ghc-pkg and GHC the same (so old-style package description still supported, and interpreted as a package containing one unit.) Source modifications to GHC assume that a package only has one unit.
2. Generalize GHC to work with packages with multiple units
3. Add capability to Cabal/ghc-pkg to register just units. For compilation of Backpack files, GHC will write out units which Cabal will then install to the real registry
4. Backpack!
New description: After today's weekly Backpack call, we have come to the conclusion that we have two different types of "packages" in the Backpack world: 1. Cabal packages, which have a single `.cabal` file and are a unit of distribution which get uploaded to Hackage, and 2. Backpack packages, of which there may be multiple defined in a Backpack file shipped with a Cabal package; and are the building blocks for modular development in the small. It's really confusing to have both of these called packages: thus, we propose to rename all occurrences of Backpack package to unit. A Cabal ''package'' may contain MULTIPLE Backpack ''units'', although old-style Cabal files will only define one unit. A Cabal package remains * The unit of distribution * The unit that Hackage handles * The unit of versioning * The unit of ownership (who maintains it etc) Here are some of the consequences: 1. We rename `PackageKey` to `UnitKey`, as it identifies a unit rather than a Cabal package. (I think this actually makes the function of these identifiers clearer.) We'll also distinguish Cabal-file level `PackageName`s from Backpack-file `UnitName`s. Finally, any given unit will be uniquely identified by an `InstalledUnitId`. 2. The source-level syntax of Backpack files will use `unit` in place of where `package` was used before. 3. For backwards compatibility reasons, we'll sometimes arrange for `PackageName`/`UnitName` and `InstalledUnitId`/`InstalledPackageId` to coincide. Specifically, the unit of a package which has the same `UnitName` as the `PackageName` is treated specially: its `InstalledUnitId` is guaranteed to be the same as the `InstalledPackageId` and it is what is "visible" when a user uses old concepts such as `-package foo-0.1` to select a package. 4. The installed package database is extended to record some number of units per an installed package: the public facing API is that you can register a package CONTAINING some number of units. For old-style packages, there will be only one unit with a matching `UnitName`, so this will be isomorphic to an old style package; however, GHC will grow some new APIs for selecting specific units to bring into scope. GHC really mostly only cares about units, but if a user asks for a package using, e.g. `-package` it will translate this request into a request for the appropriate unit. 5. For old-style packages, Cabal will continue to write and register a package configuration file which implicitly defines a single unit. However, the plan is to give GHC the capacity to generate unit description files (like package description files, but per unit), which Cabal can interpret and use to register packages in the global database using a `ghc-pkg` which can register units and "unitless" package description files which don't implicitly define a unit. (NB: we must create unit description files, because `ghc-pkg recache` is expected to be able to regenerate the database.) (NB: we want to be able to add units for a package separately, because this is how units for indefinite packages are created as they are instantiated with new implementations.) 6. Cabal could also finally grow the "multiple libraries per single Cabal file" support people have wanted; it's just a stylized use of the Backpack facilities. The work plan: 1. Modify `bin-package-db` to reflect the unit/package split, but otherwise keep ghc-pkg and GHC the same (so old-style package description still supported, and interpreted as a package containing one unit.) Source modifications to GHC assume that a package only has one unit. 2. Generalize GHC to work with packages with multiple units 3. Add capability to Cabal/ghc-pkg to register just units. For compilation of Backpack files, GHC will write out units which Cabal will then install to the real registry 4. Backpack! -- -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10622#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10622: Rename Backpack packages to units -------------------------------------+------------------------------------- Reporter: ezyang | Owner: ezyang Type: task | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.11 Resolution: | Keywords: backpack Operating System: Unknown/Multiple | Architecture: Type of failure: None/Unknown | Unknown/Multiple Blocked By: | Test Case: Related Tickets: | Blocking: | Differential Revisions: Phab:D1057 -------------------------------------+------------------------------------- Description changed by ezyang: Old description:
After today's weekly Backpack call, we have come to the conclusion that we have two different types of "packages" in the Backpack world:
1. Cabal packages, which have a single `.cabal` file and are a unit of distribution which get uploaded to Hackage, and
2. Backpack packages, of which there may be multiple defined in a Backpack file shipped with a Cabal package; and are the building blocks for modular development in the small.
It's really confusing to have both of these called packages: thus, we propose to rename all occurrences of Backpack package to unit. A Cabal ''package'' may contain MULTIPLE Backpack ''units'', although old-style Cabal files will only define one unit.
A Cabal package remains * The unit of distribution * The unit that Hackage handles * The unit of versioning * The unit of ownership (who maintains it etc)
Here are some of the consequences:
1. We rename `PackageKey` to `UnitKey`, as it identifies a unit rather than a Cabal package. (I think this actually makes the function of these identifiers clearer.) We'll also distinguish Cabal-file level `PackageName`s from Backpack-file `UnitName`s. Finally, any given unit will be uniquely identified by an `InstalledUnitId`.
2. The source-level syntax of Backpack files will use `unit` in place of where `package` was used before.
3. For backwards compatibility reasons, we'll sometimes arrange for `PackageName`/`UnitName` and `InstalledUnitId`/`InstalledPackageId` to coincide. Specifically, the unit of a package which has the same `UnitName` as the `PackageName` is treated specially: its `InstalledUnitId` is guaranteed to be the same as the `InstalledPackageId` and it is what is "visible" when a user uses old concepts such as `-package foo-0.1` to select a package.
4. The installed package database is extended to record some number of units per an installed package: the public facing API is that you can register a package CONTAINING some number of units. For old-style packages, there will be only one unit with a matching `UnitName`, so this will be isomorphic to an old style package; however, GHC will grow some new APIs for selecting specific units to bring into scope. GHC really mostly only cares about units, but if a user asks for a package using, e.g. `-package` it will translate this request into a request for the appropriate unit.
5. For old-style packages, Cabal will continue to write and register a package configuration file which implicitly defines a single unit. However, the plan is to give GHC the capacity to generate unit description files (like package description files, but per unit), which Cabal can interpret and use to register packages in the global database using a `ghc-pkg` which can register units and "unitless" package description files which don't implicitly define a unit. (NB: we must create unit description files, because `ghc-pkg recache` is expected to be able to regenerate the database.) (NB: we want to be able to add units for a package separately, because this is how units for indefinite packages are created as they are instantiated with new implementations.)
6. Cabal could also finally grow the "multiple libraries per single Cabal file" support people have wanted; it's just a stylized use of the Backpack facilities.
The work plan:
1. Modify `bin-package-db` to reflect the unit/package split, but otherwise keep ghc-pkg and GHC the same (so old-style package description still supported, and interpreted as a package containing one unit.) Source modifications to GHC assume that a package only has one unit.
2. Generalize GHC to work with packages with multiple units
3. Add capability to Cabal/ghc-pkg to register just units. For compilation of Backpack files, GHC will write out units which Cabal will then install to the real registry
4. Backpack!
New description: After today's weekly Backpack call, we have come to the conclusion that we have two different types of "packages" in the Backpack world: 1. Cabal packages, which have a single `.cabal` file and are a unit of distribution which get uploaded to Hackage, and 2. Backpack packages, of which there may be multiple defined in a Backpack file shipped with a Cabal package; and are the building blocks for modular development in the small. It's really confusing to have both of these called packages: thus, we propose to rename all occurrences of Backpack package to unit. A Cabal ''package'' may contain MULTIPLE Backpack ''units'', although old-style Cabal files will only define one unit. A Cabal package remains * The unit of distribution * The unit that Hackage handles * The unit of versioning * The unit of ownership (who maintains it etc) Here are some of the consequences: 1. We rename `PackageKey` to `UnitKey`, as it identifies a unit rather than a Cabal package. (I think this actually makes the function of these identifiers clearer.) We'll also distinguish Cabal-file level `PackageName`s from Backpack-file `UnitName`s. Finally, any given unit will be uniquely identified by an `InstalledUnitId`. 2. The source-level syntax of Backpack files will use `unit` in place of where `package` was used before. 3. For backwards compatibility reasons, we'll sometimes arrange for `PackageName`/`UnitName` and `InstalledUnitId`/`InstalledPackageId` to coincide. Specifically, the unit of a package which has the same `UnitName` as the `PackageName` is treated specially: its `InstalledUnitId` is guaranteed to be the same as the `InstalledPackageId` and it is what is "visible" when a user uses old concepts such as `-package foo-0.1` to select a package. 4. The installed package database is extended to record some number of units per an installed package: the public facing API is that you can register a package CONTAINING some number of units. For old-style packages, there will be only one unit with a matching `UnitName`, so this will be isomorphic to an old style package; however, GHC will grow some new APIs for selecting specific units to bring into scope. GHC really mostly only cares about units, but if a user asks for a package using, e.g. `-package` it will translate this request into a request for the appropriate unit. 5. For old-style packages, Cabal will continue to write and register a package configuration file which implicitly defines a single unit. However, the plan is to give GHC the capacity to generate unit description files (like package description files, but per unit), which Cabal can interpret and use to register packages in the global database using a `ghc-pkg` which can register units and "unitless" package description files which don't implicitly define a unit. (NB: we must create unit description files, because `ghc-pkg recache` is expected to be able to regenerate the database.) (NB: we want to be able to add units for a package separately, because this is how units for indefinite packages are created as they are instantiated with new implementations.) (NB: For BC purposes, since Cabal has to write old-style packages for old GHC, this is pretty irritating.) 6. Cabal could also finally grow the "multiple libraries per single Cabal file" support people have wanted; it's just a stylized use of the Backpack facilities. (Note: Actually, not QUITE: multiple libraries are likely to want separate sets of dependencies, whereas our units proposal has external dependencies shared over all units. If we want the multiple libraries, cabal-install has to learn about units.) The work plan: 1. Modify `bin-package-db` to reflect the unit/package split, but otherwise keep ghc-pkg and GHC the same (so old-style package description still supported, and interpreted as a package containing one unit.) Source modifications to GHC assume that a package only has one unit. 2. Generalize GHC to work with packages with multiple units 3. Add capability to Cabal/ghc-pkg to register just units. For compilation of Backpack files, GHC will write out units which Cabal will then install to the real registry 4. Backpack! -- -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10622#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10622: Rename Backpack packages to units -------------------------------------+------------------------------------- Reporter: ezyang | Owner: ezyang Type: task | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.11 Resolution: | Keywords: backpack Operating System: Unknown/Multiple | Architecture: Type of failure: None/Unknown | Unknown/Multiple Blocked By: | Test Case: Related Tickets: | Blocking: | Differential Revisions: Phab:D1057 -------------------------------------+------------------------------------- Comment (by ezyang): Here is a counter-proposal, organized around avoiding changing the package database format: Cabal already has limited support for multiple "packages" in a distribution unit, namely with its support for testing libraries. These libraries are never installed for other users to use, but internally, they can be installed and used like extra libraries. In this model, the database has a separate entry for each unit. The downside is that the package database will no longer be organized by distribution units. But this was already the case: if I install a package multiple times with different dependencies, it will occur multiple times in the database. The big upside is that the changes we have to make are now much smaller. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10622#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10622: Rename Backpack packages to units -------------------------------------+------------------------------------- Reporter: ezyang | Owner: ezyang Type: task | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.11 Resolution: | Keywords: backpack Operating System: Unknown/Multiple | Architecture: Type of failure: None/Unknown | Unknown/Multiple Blocked By: | Test Case: Related Tickets: | Blocking: | Differential Revisions: Phab:D1057 -------------------------------------+------------------------------------- Comment (by ezyang): And here is a counter-counter-proposal, where simply REDUCE sharing: 1. The basic idea is to defer making libraries/bundles of hi files until we have a complete, definite package that has been Cabalized. So if we have something like: {{{ package p where signature H module P package q where module H include p }}} distributed with a `q.cabal`, `p` NEVER SHOWS UP in the installed package database; not even the version of it instantiated with `q`. 2. This obviously breaks type-checking, since when we build `p` will still be compiled to a specific package key `p(Q -> q:H)`, but this package key won't be anywhere in our installed package database. So libraries like `q` will get a new type of entry in the installed package database: they are **fat installed packages** which can contain files for more than one package key. These keys are enumerated in the entry installed package database, and you just look in `import-dirs/key` to find the relevant interface files. So `p`'s interface files will live in something like `q -install-dir/p_KEY`. Cabal also records the ABI hash of each of the sub- packages in a fat package. 3. Suppose `p` is an indefinite package with a `p.cabal` of its own. Neither the generic `p` nor the instantiated versions of `p` have direct entries in the package database: you will only file `hi` files/libraries under the fat installs of other definite packages with used `p`. 4. What happens in this situation, when `q1` and `q2` are built in parallel? (Suppose each package has its own Cabal file) {{{ package p where signature H module P package h where module H package q1 where include h include p module Q1 package q2 where include h include p module Q2 }}} `h` is a normal package and can get installed as usual. `q1` and `q2` are FAT installed packages, they get installed with hi files and libraries for `p(H -> h:H)`. In particular, this means that means that `p` instantiated with `h` is DUPLICATED in these two fat installed packages. To avoid disaster from incompatible duplicate packages, we verify that for every duplicated package key in the package database, the ABI hashes are the same. This will work great if we have deterministic builds, and not so great if they are nondeterministic. 5. Let's say we instantiate a package, and we discover that a fat package which we don't directly depend on instantiated it already. What do we do? It should be OK to reuse it, but when Cabal goes and installs, it must copy the interface files and libraries from the other fat package into our new fat package. 6. This makes the story great for distribution packagers: they don't have to worry about two (morally separate) packages depending on common files/libraries which need to be installed in the same location. This would require a subpackage, but Debian is unlikely to want to create lots and lots of little packages to get the sharing we're aiming for here. I actually kind of suspect this is what Simon wanted to do from the beginning; apologies for not figuring it out sooner. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10622#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10622: Rename Backpack packages to units -------------------------------------+------------------------------------- Reporter: ezyang | Owner: ezyang Type: task | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.11 Resolution: | Keywords: backpack Operating System: Unknown/Multiple | Architecture: Type of failure: None/Unknown | Unknown/Multiple Blocked By: | Test Case: Related Tickets: | Blocking: | Differential Revisions: Phab:D1057 -------------------------------------+------------------------------------- Comment (by simonpj): This sounds plausible to me, though as usual with cabal/backpack I am not 100% sure of my ground. I think of it like this: * Cabal concerns itself with Cabal packages (units of distribution and versioning), including choosing version numbers, downloading them, and figuring out if that particular package (instantiated with its transitive dependencies) is already installed. * GHC concerns itself with Backpack units. Both share a single "installed package database". But they use it in a different way. For GHC at least, it's just a cache: a place to record the result of previous work (including typechecking indefinite packages), so that we don't need to repeat it. There is no harm in repeating it, but it's a waste of time. So Cabal need never see the previously-compiled indefinite packages; they are just a way for GHC to save time. Maybe that is what you are saying. Another way to attack this is to ask "what questions does Cabal ask the installed package database?" and "what questions does GHC ask?". I think the two are different. Simon -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10622#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10622: Rename Backpack packages to units -------------------------------------+------------------------------------- Reporter: ezyang | Owner: ezyang Type: task | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.11 Resolution: | Keywords: backpack Operating System: Unknown/Multiple | Architecture: Type of failure: None/Unknown | Unknown/Multiple Blocked By: | Test Case: Related Tickets: | Blocking: | Differential Revisions: Phab:D1057 -------------------------------------+------------------------------------- Comment (by ezyang): Duncan and I had a chat about this, which in particular clarified "What questions Cabal asks the installed package database." cabal-install currently queries the installed package database in order so determine what the installed packages are when it is making an install plan. However, this is going to soon not be the case with Nix style versioning: Cabal will make an install plan using purely source package info, and then query the database to find out what products it already has installed. So the key idea is: **the installed package database is a database of units**! There may be more junk in the database than you would expect from the packages you have installed, but in a Nix world where there will be many many installs of the same package, ghc-pkq is already going to not be so useful for finding info about packages. I'll update the top level description with the final plan. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10622#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10622: Rename Backpack packages to units -------------------------------------+------------------------------------- Reporter: ezyang | Owner: ezyang Type: task | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.11 Resolution: | Keywords: backpack Operating System: Unknown/Multiple | Architecture: Type of failure: None/Unknown | Unknown/Multiple Blocked By: | Test Case: Related Tickets: | Blocking: | Differential Revisions: Phab:D1057 -------------------------------------+------------------------------------- Description changed by ezyang: Old description:
After today's weekly Backpack call, we have come to the conclusion that we have two different types of "packages" in the Backpack world:
1. Cabal packages, which have a single `.cabal` file and are a unit of distribution which get uploaded to Hackage, and
2. Backpack packages, of which there may be multiple defined in a Backpack file shipped with a Cabal package; and are the building blocks for modular development in the small.
It's really confusing to have both of these called packages: thus, we propose to rename all occurrences of Backpack package to unit. A Cabal ''package'' may contain MULTIPLE Backpack ''units'', although old-style Cabal files will only define one unit.
A Cabal package remains * The unit of distribution * The unit that Hackage handles * The unit of versioning * The unit of ownership (who maintains it etc)
Here are some of the consequences:
1. We rename `PackageKey` to `UnitKey`, as it identifies a unit rather than a Cabal package. (I think this actually makes the function of these identifiers clearer.) We'll also distinguish Cabal-file level `PackageName`s from Backpack-file `UnitName`s. Finally, any given unit will be uniquely identified by an `InstalledUnitId`.
2. The source-level syntax of Backpack files will use `unit` in place of where `package` was used before.
3. For backwards compatibility reasons, we'll sometimes arrange for `PackageName`/`UnitName` and `InstalledUnitId`/`InstalledPackageId` to coincide. Specifically, the unit of a package which has the same `UnitName` as the `PackageName` is treated specially: its `InstalledUnitId` is guaranteed to be the same as the `InstalledPackageId` and it is what is "visible" when a user uses old concepts such as `-package foo-0.1` to select a package.
4. The installed package database is extended to record some number of units per an installed package: the public facing API is that you can register a package CONTAINING some number of units. For old-style packages, there will be only one unit with a matching `UnitName`, so this will be isomorphic to an old style package; however, GHC will grow some new APIs for selecting specific units to bring into scope. GHC really mostly only cares about units, but if a user asks for a package using, e.g. `-package` it will translate this request into a request for the appropriate unit.
5. For old-style packages, Cabal will continue to write and register a package configuration file which implicitly defines a single unit. However, the plan is to give GHC the capacity to generate unit description files (like package description files, but per unit), which Cabal can interpret and use to register packages in the global database using a `ghc-pkg` which can register units and "unitless" package description files which don't implicitly define a unit. (NB: we must create unit description files, because `ghc-pkg recache` is expected to be able to regenerate the database.) (NB: we want to be able to add units for a package separately, because this is how units for indefinite packages are created as they are instantiated with new implementations.) (NB: For BC purposes, since Cabal has to write old-style packages for old GHC, this is pretty irritating.)
6. Cabal could also finally grow the "multiple libraries per single Cabal file" support people have wanted; it's just a stylized use of the Backpack facilities. (Note: Actually, not QUITE: multiple libraries are likely to want separate sets of dependencies, whereas our units proposal has external dependencies shared over all units. If we want the multiple libraries, cabal-install has to learn about units.)
The work plan:
1. Modify `bin-package-db` to reflect the unit/package split, but otherwise keep ghc-pkg and GHC the same (so old-style package description still supported, and interpreted as a package containing one unit.) Source modifications to GHC assume that a package only has one unit.
2. Generalize GHC to work with packages with multiple units
3. Add capability to Cabal/ghc-pkg to register just units. For compilation of Backpack files, GHC will write out units which Cabal will then install to the real registry
4. Backpack!
New description: After today's weekly Backpack call, we have come to the conclusion that we have two different types of "packages" in the Backpack world: 1. Cabal packages, which have a single `.cabal` file and are a unit of distribution which get uploaded to Hackage, and 2. Backpack packages, of which there may be multiple defined in a Backpack file shipped with a Cabal package; and are the building blocks for modular development in the small. It's really confusing to have both of these called packages: thus, we propose to rename all occurrences of Backpack package to unit. A Cabal ''package'' may contain MULTIPLE Backpack ''units'', and old-style Cabal files will only define one unit. Every Cabal package has a distinguished unit (with the same name as the package) that serves as the publically visible unit. A Cabal package remains * The unit of distribution * The unit that Hackage handles * The unit of versioning * The unit of ownership (who maintains it etc) Here are some of the consequences: 1. The "installed package database" no longer maintains a one-to-one mapping between Cabal packages and entries in the database. This invariant is being dropped for two reasons: (1) With a Nix-style database, a package `foo-0.1` may be installed many times with different dependencies / source code, all of which live in the installed package database. (2) With Backpack, a package containing a Backpack file may install multiple units. To avoid having to rename *everything*, we'll keep calling this the installed package database, but really it's more like an installed *unit* database. 2. We rename `PackageKey` to `UnitKey`, as it identifies a unit rather than a Cabal package. (I think this actually makes the function of these identifiers clearer.) We'll also distinguish Cabal-file level `PackageName`s from Backpack-file `UnitName`s. Installed units continue to be identified by `InstalledPackageId`. 3. The source-level syntax of Backpack files will use `unit` in place of where `package` was used before. 4. For old-style packages, Cabal will continue to write and register a single entry in the installed package database. For Backpack packages, Cabal will register as many entries as is necessary to install a package. The entry with the same `UnitName` as `PackageName` is publically visible to other packages. If a Backpack file defines other packages, those packages are registered with different `UnitName`s (giving them different `InstalledPackageId`s) which are not publically visible. The non- publically visible packages will have their description/URL/etc fields blank, and have a pointer to the "real" package. 5. If when installing a unit, we discover that it is already present in the database, we check if the ABI hashes are the same. If they are, we simply skip installing the unit but otherwise proceed. If the ABI hashes are not the same, we error: the units we are installing need to be recompiled against the unit present in the database. -- -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10622#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10622: Rename Backpack packages to units -------------------------------------+------------------------------------- Reporter: ezyang | Owner: ezyang Type: task | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.11 Resolution: | Keywords: backpack Operating System: Unknown/Multiple | Architecture: Type of failure: None/Unknown | Unknown/Multiple Blocked By: | Test Case: Related Tickets: | Blocking: | Differential Revisions: Phab:D1057 -------------------------------------+------------------------------------- Comment (by simonpj):
Installed units continue to be identified by `InstalledPackageId`
Wouldn't `InstalledUnitId` be clearer? Indeed wouldn't `InstalledPackageId` be positively misleading? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10622#comment:10 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10622: Rename Backpack packages to units -------------------------------------+------------------------------------- Reporter: ezyang | Owner: ezyang Type: task | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.11 Resolution: | Keywords: backpack Operating System: Unknown/Multiple | Architecture: Type of failure: None/Unknown | Unknown/Multiple Blocked By: | Test Case: Related Tickets: | Blocking: | Differential Revisions: Phab:D1057 -------------------------------------+------------------------------------- Old description:
After today's weekly Backpack call, we have come to the conclusion that we have two different types of "packages" in the Backpack world:
1. Cabal packages, which have a single `.cabal` file and are a unit of distribution which get uploaded to Hackage, and
2. Backpack packages, of which there may be multiple defined in a Backpack file shipped with a Cabal package; and are the building blocks for modular development in the small.
It's really confusing to have both of these called packages: thus, we propose to rename all occurrences of Backpack package to unit. A Cabal ''package'' may contain MULTIPLE Backpack ''units'', and old-style Cabal files will only define one unit. Every Cabal package has a distinguished unit (with the same name as the package) that serves as the publically visible unit.
A Cabal package remains * The unit of distribution * The unit that Hackage handles * The unit of versioning * The unit of ownership (who maintains it etc)
Here are some of the consequences:
1. The "installed package database" no longer maintains a one-to-one mapping between Cabal packages and entries in the database. This invariant is being dropped for two reasons: (1) With a Nix-style database, a package `foo-0.1` may be installed many times with different dependencies / source code, all of which live in the installed package database. (2) With Backpack, a package containing a Backpack file may install multiple units. To avoid having to rename *everything*, we'll keep calling this the installed package database, but really it's more like an installed *unit* database.
2. We rename `PackageKey` to `UnitKey`, as it identifies a unit rather than a Cabal package. (I think this actually makes the function of these identifiers clearer.) We'll also distinguish Cabal-file level `PackageName`s from Backpack-file `UnitName`s. Installed units continue to be identified by `InstalledPackageId`.
3. The source-level syntax of Backpack files will use `unit` in place of where `package` was used before.
4. For old-style packages, Cabal will continue to write and register a single entry in the installed package database. For Backpack packages, Cabal will register as many entries as is necessary to install a package. The entry with the same `UnitName` as `PackageName` is publically visible to other packages. If a Backpack file defines other packages, those packages are registered with different `UnitName`s (giving them different `InstalledPackageId`s) which are not publically visible. The non- publically visible packages will have their description/URL/etc fields blank, and have a pointer to the "real" package.
5. If when installing a unit, we discover that it is already present in the database, we check if the ABI hashes are the same. If they are, we simply skip installing the unit but otherwise proceed. If the ABI hashes are not the same, we error: the units we are installing need to be recompiled against the unit present in the database.
New description: After today's weekly Backpack call, we have come to the conclusion that we have two different types of "packages" in the Backpack world: 1. Cabal packages, which have a single `.cabal` file and are a unit of distribution which get uploaded to Hackage, and 2. Backpack packages, of which there may be multiple defined in a Backpack file shipped with a Cabal package; and are the building blocks for modular development in the small. It's really confusing to have both of these called packages: thus, we propose to rename all occurrences of Backpack package to unit. A Cabal ''package'' may contain MULTIPLE Backpack ''units'', and old-style Cabal files will only define one unit. Every Cabal package has a distinguished unit (with the same name as the package) that serves as the publically visible unit. A Cabal package remains * The unit of distribution * The unit that Hackage handles * The unit of versioning * The unit of ownership (who maintains it etc) Here are some of the consequences: 1. The "installed package database" no longer maintains a one-to-one mapping between Cabal packages and entries in the database. This invariant is being dropped for two reasons: (1) With a Nix-style database, a package `foo-0.1` may be installed many times with different dependencies / source code, all of which live in the installed package database. (2) With Backpack, a package containing a Backpack file may install multiple units. To avoid having to rename *everything*, we'll keep calling this the installed package database, but really it's more like an installed *unit* database. 2. We rename `PackageKey` to `UnitKey`, as it identifies a unit rather than a Cabal package. (I think this actually makes the function of these identifiers clearer.) We'll also distinguish Cabal-file level `PackageName`s from Backpack-file `UnitName`s. Installed units are identified by an `InstalledUnitId` instead of an `InstalledPackageId`. 3. The source-level syntax of Backpack files will use `unit` in place of where `package` was used before. 4. For old-style packages, Cabal will continue to write and register a single entry in the installed package database. For Backpack packages, Cabal will register as many entries as is necessary to install a package. The entry with the same `UnitName` as `PackageName` is publically visible to other packages. If a Backpack file defines other packages, those packages are registered with different `UnitName`s (giving them different `InstalledPackageId`s) which are not publically visible. The non- publically visible packages will have their description/URL/etc fields blank, and have a pointer to the "real" package. 5. If when installing a unit, we discover that it is already present in the database, we check if the ABI hashes are the same. If they are, we simply skip installing the unit but otherwise proceed. If the ABI hashes are not the same, we error: the units we are installing need to be recompiled against the unit present in the database. -- Comment (by ezyang): OK. (I was thinking about BC for Cabal users but I guess it should not be too bad. We'll see.) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10622#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10622: Rename Backpack packages to units -------------------------------------+------------------------------------- Reporter: ezyang | Owner: ezyang Type: task | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.11 Resolution: | Keywords: backpack Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: Phab:D1057 -------------------------------------+------------------------------------- Old description:
After today's weekly Backpack call, we have come to the conclusion that we have two different types of "packages" in the Backpack world:
1. Cabal packages, which have a single `.cabal` file and are a unit of distribution which get uploaded to Hackage, and
2. Backpack packages, of which there may be multiple defined in a Backpack file shipped with a Cabal package; and are the building blocks for modular development in the small.
It's really confusing to have both of these called packages: thus, we propose to rename all occurrences of Backpack package to unit. A Cabal ''package'' may contain MULTIPLE Backpack ''units'', and old-style Cabal files will only define one unit. Every Cabal package has a distinguished unit (with the same name as the package) that serves as the publically visible unit.
A Cabal package remains * The unit of distribution * The unit that Hackage handles * The unit of versioning * The unit of ownership (who maintains it etc)
Here are some of the consequences:
1. The "installed package database" no longer maintains a one-to-one mapping between Cabal packages and entries in the database. This invariant is being dropped for two reasons: (1) With a Nix-style database, a package `foo-0.1` may be installed many times with different dependencies / source code, all of which live in the installed package database. (2) With Backpack, a package containing a Backpack file may install multiple units. To avoid having to rename *everything*, we'll keep calling this the installed package database, but really it's more like an installed *unit* database.
2. We rename `PackageKey` to `UnitKey`, as it identifies a unit rather than a Cabal package. (I think this actually makes the function of these identifiers clearer.) We'll also distinguish Cabal-file level `PackageName`s from Backpack-file `UnitName`s. Installed units are identified by an `InstalledUnitId` instead of an `InstalledPackageId`.
3. The source-level syntax of Backpack files will use `unit` in place of where `package` was used before.
4. For old-style packages, Cabal will continue to write and register a single entry in the installed package database. For Backpack packages, Cabal will register as many entries as is necessary to install a package. The entry with the same `UnitName` as `PackageName` is publically visible to other packages. If a Backpack file defines other packages, those packages are registered with different `UnitName`s (giving them different `InstalledPackageId`s) which are not publically visible. The non- publically visible packages will have their description/URL/etc fields blank, and have a pointer to the "real" package.
5. If when installing a unit, we discover that it is already present in the database, we check if the ABI hashes are the same. If they are, we simply skip installing the unit but otherwise proceed. If the ABI hashes are not the same, we error: the units we are installing need to be recompiled against the unit present in the database.
New description: After today's weekly Backpack call, we have come to the conclusion that we have two different types of "packages" in the Backpack world: 1. Cabal packages, which have a single `.cabal` file and are a unit of distribution which get uploaded to Hackage, and 2. Backpack packages, of which there may be multiple defined in a Backpack file shipped with a Cabal package; and are the building blocks for modular development in the small. It's really confusing to have both of these called packages: thus, we propose to rename all occurrences of Backpack package to unit. A Cabal ''package'' may contain MULTIPLE Backpack ''units'', and old-style Cabal files will only define one unit. Every Cabal package has a distinguished unit (with the same name as the package) that serves as the publically visible unit. A Cabal package remains * The unit of distribution * The unit that Hackage handles * The unit of versioning * The unit of ownership (who maintains it etc) Here are some of the consequences: 1. The "installed package database" no longer maintains a one-to-one mapping between Cabal packages and entries in the database. This invariant is being dropped for two reasons: (1) With a Nix-style database, a package `foo-0.1` may be installed many times with different dependencies / source code, all of which live in the installed package database. (2) With Backpack, a package containing a Backpack file may install multiple units. To avoid having to rename *everything*, we'll keep calling this the installed package database, but really it's more like an installed *unit* database. 2. We rename `PackageKey` to `UnitKey`, as it identifies a unit rather than a Cabal package. (I think this actually makes the function of these identifiers clearer.) We'll also distinguish Cabal-file level `PackageName`s from Backpack-file `UnitName`s. Installed units are identified by an `InstalledUnitId` instead of an `InstalledPackageId`. 3. The source-level syntax of Backpack files will use `unit` in place of where `package` was used before. 4. For old-style packages, Cabal will continue to write and register a single entry in the installed package database. For Backpack packages, Cabal will register as many entries as is necessary to install a package. The entry with the same `UnitName` as `PackageName` is publically visible to other packages. If a Backpack file defines other packages, those packages are registered with different `UnitName`s (giving them different `InstalledPackageId`s) which are not publically visible. The non- publically visible packages will have their description/URL/etc fields blank, and have a pointer to the "real" package. 5. If when installing a unit, we discover that it is already present in the database, we check if the ABI hashes are the same. If they are, we simply skip installing the unit but otherwise proceed. If the ABI hashes are not the same, we error: the units we are installing need to be recompiled against the unit present in the database. 6. Dependency tracking should be fine-grained within a PACKAGE, and coarse-grained outside. So we need to let interface files track module dependencies for files which are not in the same unit, but are in the same package. -- Comment (by ezyang): Add a remark about how dependency tracking should change between units in the same package. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10622#comment:12 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10622: Rename Backpack packages to units -------------------------------------+------------------------------------- Reporter: ezyang | Owner: ezyang Type: task | Status: closed Priority: normal | Milestone: Component: Compiler | Version: 7.11 Resolution: fixed | Keywords: backpack Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D1057 Wiki Page: | -------------------------------------+------------------------------------- Changes (by ezyang): * status: new => closed * resolution: => fixed Comment: This was fixed in b92a51f5e235fc22a9ae0d76735b50f095665fb7 and 5b0191f74ab05b187f81ea037623338a615b1619 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10622#comment:13 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
participants (1)
-
GHC