#10871: Implement "fat" interface files which can be directly compiled without source -------------------------------------+------------------------------------- Reporter: ezyang | Owner: ezyang Type: feature request | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.11 Resolution: | Keywords: backpack Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by ezyang): SM, I think you have convinced me that eventually, we will need a version of cabal-install that is Backpack-aware, and has a work item in its install plan for each instantiated Backpack package. I am also going to claim this implementation of `cabal-install` is not going to be "simple". Let me first state an principle which we have both agreed is desirable: **It should be possible to build a Backpack unit without Cabal** (...assuming that all external dependencies have been built already.) This principle forces a few choices: **We need a separate file format for Backpack.** Suppose that we don't have a separate file format for Backpack, e.g. that all the Backpack information is recorded in a Cabal file (as it was in our older design from a year ago.) GHC can't parse a Cabal file, but it needs to know specifically what unit provides any module it wants to import. How can you get this info to GHC? 1. In our old design, you could pass a pile of `-package` flags with renamings to make sure every module name points to the correct module from the correct instantiated unit. Obviously this is completely unusable by hand; you'd need a tool which knows about Backpack to generate these flags. So much for using Backpack without Cabal! My advisor complained a lot about this, and that was the impetus that lead us down the route to fat interface files. 2. You could write any Backpack-specific information to its own file and pass that to GHC. This is a lot better! Now the only extra information you might need to pass is how the holes are to be instantiated (in case the unit is indefinite but you want to compile it). **It is GHC, not Cabal, who knows how to instantiate units.** Simple corollary of the above. And I also assume that we don't want to duplicate code, so this code for instantiation should only live in one place. Notice that in the old design, it was Cabal that had the Backpack smarts! But I think this was the wrong choice: GHC should have the Backpack smarts. **Cabal needs to ask GHC for information about how it instantiated units.** Since the code for unit instantiation lives in GHC, but each instantiated unit needs to get installed to the database (a Cabal concern), it follows that Cabal needs to call GHC in order to get information about how units are instantiated. This is directly relevant to cabal-install. Traditionally, cabal-install takes the package to be built, resolves the dependency graph, and then builds the dependency graph (now an install plan) in topological order. But if GHC, not Cabal, does instantiation, we don't know what the full set of instantiated units a Backpack package needs until after we have configured it. Essentially, as we process packages in the install plan, after we configure, we have to ask GHC to tell us what other dependencies are needed, and then splice those dependencies into the install plan as extra targets that must be built. This is further complicated by the fact that the Cabal library may be embedded in a custom Setup script; in which case we have to define an interchange format between Cabal and cabal-install to communicate this information. I don't think this is the wrong way to do it, but it will take work to implement. Let me argue for fat interface files one more time. One of my top priorities at this stage is to get Backpack working "just enough" so that we can roll it out and let users (including me) play with it and use it for real applications. So if there is a way to get thing working today (maybe not entirely the Right(TM) way) in a way that doesn't impair our ability to make it work right later, I would quite like to take it. Fat interfaces are a way to get Backpack integrated with the Cabal ecosystem without having to rewrite cabal-install's install plan runner, make it easier to experiment with Backpack, have the property, "If you don't care about it, you don't have to", and most importantly, is mostly implemented at this point. This discussion has also reminded me that there is still an elephant in the room with regards to recompilation avoidance in Backpack. In a package model, GHC will not recompile if the package key of the depended upon package has not changed. Since we don't generally expect package keys to change on an edit/recompile cycle, it's unsound to keep around build files from an old compilation. This means that you end up doing a lot of recompiling with Cabal today when you change a package that is depended upon by other packages you are building. In fact, we want something like this: we don't *care* about recompilation avoidance when we are installing, but if we are edit/rebuilding, we want accurate recompilation avoidance for the entire set of packages which we may be editing and building. This sounds exactly like the "local package database" that we end up creating when compiling Backpack units using fat interfaces. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10871#comment:22 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler