At Thu, 13 Jul 2006 12:26:22 +0100, Simon Marlow wrote:

Jeremy Shaw wrote:

...

(1) Flag sanity checking

Not all combinations of flags are sensible. For example, we can already specify:

There's another restriction I know of: if the code uses Template Haskell, then --enable-library-vanilla will always be required. Since the extensions used are all listed in the .cabal file, checking this will be easy.

Hrm, that puts me back in the same situation I am in now -- not having a good way to split the profiling libraries into a separate package. I have written up some of my thoughts below. I would love to hear opinions from any interested parties, such as compiler writers, package maintainers, end users, etc. Goals ===== Overall Goal ------------ My overall goal is to add some flags to Cabal that will make it easier for package maintainers to provide several variations of the same library in separate packages. For example, separate packages for the library compiled: vanilla, profiling, smp, bytecode, etc. If a package (such as the profiling) needs files that are already in another package (such as vanilla), it will simply depend on the other package. (i.e. it will *not* contain a second copy of the files) Currently, in order to split the profiling and vanilla libraries into separate .debs, I have to do some fancy hacking in the Debian specific dh_haskell script. If someone wants to build .rpms, they will probably come up with their own hacks to achieve the same goal. Clearly, this should be fixed at the Cabal level, instead of having each package maintainer come up with a duplicate set of hacks. Proper Level Of Abstraction --------------------------- Cabal is nice because it hides all the nasty compiler and platform specific build issues from the user. The user puts *what* they want in the .cabal file, and they let Cabal take care of the *how*. I think it is very desirable to retain this abstraction in the configure flags as much as possible. To me, this means that the flags: --disable-library-vanilla --enable-library-profiling Should have a specified behavior that is compiler and platform independent. It is the job of the compiler specific Cabal code to meet the spec. (e.g. Distribution.Simple.GHC, etc) Appropriate Level of Error Reporting ------------------------------------ Some options never make sense, for example: --enable-library-vanilla --disable-library-vanilla We can check for that nonsense all the time. But the are other combinations of flags that only make sense from a maintainers point of view. The typical end user would just end up with a broken library install. So I propose we enable strict flag checking by default, but provide a flag, such as --maintainer-mode, that loosen the restrictions. Difficulties ------------ It is difficult to make a specification that all compilers can meet. For example, most compilers do not support profiling, so they can not support the --enable-*-profiling flag at the moment. In this case, we could just extend the spec to say that it is an error to enable profiling for a compiler that does not support profiling. On the other hand, someone might make a compiler that does not allow you to install the vanilla and profiling libraries at the same time. Now you have the problem that you do want to have a -prof .deb, but it is going to work different from the 'standard'. Of course, the specifications are supposedly reasonable specifications that are dictated by the actual needs of the users and the package maintainers. So, I think in most cases, having a specification to meet will make it easier for compiler writers, since they will not have to learn the hard way that certain features are important. There will be exceptions. That is why, for example, the .cabal file has compiler specific fields like ghc-options even though it has the Extensions field. I am not quite sure how that translates to configure flags. How to Handle GHC + Profiling + TemplateHaskell =============================================== Profiling + TemplateHaskell Question ------------------------------------ I currently only have 6.4.1 installed -- as far as I can tell, profiling + TH does not work at all in 6.4.1. In ghc head, if I do, ghc --make -prof Test.hs -o test Does the compiler automatically build the vanilla objects and then build the profiling objects? Or do I need to first build without -prof, and then build with -prof? I think I can implement a solution in three incremental steps: (1) Add finer granularity configure flags (2) Add sanity checking to the argument processing (3) Add some 'speed' hacks How It Would Work In Cabal -------------------------- The spec for the flags: --maintainer --disable-library-vanilla --enable-library-profiling is something like, + build the libraries with profiling enabled + copy out just the files needed add profiling support to a system that already has the vanilla libraries available + this assumes that the profiling and vanilla libraries can co-exist (i.e. do not have conflicting files, etc) Here is how the Cabal code for GHC would meet that spec: (1) In the 'build' phase: (i) if ghc requires you to explicitly build the vanilla libraries before building the profiling libraries, then do that. (ii) build the profiling libraries (2) In the 'install/copy' phase, *only* copy the profiling libraries out. (NOTE: This is actually quite easy -- I have already implemented it in Cabal). Downside Of This Solution ------------------------- The downside of the above implementation is that the vanilla libraries will be built twice. Once for installation in the -vanilla package, and a second time when trying to create the -prof package. Unsafe Speed Hacks: =================== In most cases, the above solution will be fine. But, if you are dealing with a package where building the vanilla libraries twice is a huge overhead, then it might be worthwhile to try to use one the hacks presented below. The downside is they require you to understand the situations in which they can be safely used. These hacks are a perhaps a bit like unsafePerformIO, unsafeCoerce, etc ;) change copy/install to allow selective copy ------------------------------------------- Add the flags --copy-only-profiling --copy-only-vanilla. This solution works for vanilla and profiling, because they contain uniquely named output files. But, if you want to build with and with out the -O2 flag, then you really do have to run the build phase twice. --no-clean ---------- If we did not delete the vanilla libraries before building the profiling libraries, then we would only have to build them once. On the other hand, if we look at our -O2 example, it *would* require a 'clean' between builds. Your Comments Here ================== If you have any suggestions, see any pitfalls, or have a completely different idea about how to solve this problem, please let me know. I think the approach is pretty conservative -- so if it does not work out, we should still be in a good position to try a different approach. But, anything we can learn now will certainly help :) j.

At Tue, 18 Jul 2006 02:07:01 +0100, Duncan Coutts wrote:

What's wrong with the simple solution of you just building with --enable-library-profiling and then splitting the ${pkg}_p.a and the *.p_hi files into your profiling .deb? The names are totally regular so it could be automated.

A good question! I think my previous email made things sound more complicated than they really are because I covered three separate, but related, additions to the current Cabal code. Let's focus on just this part: 1) Should we add --enable/disable-library-vanilla flags 2) Should we require that specifying: --disable-library-vanilla --enable-library-profiling Results in just the additional files needed to support profiling being copied in the copy/install phase. I have already implemented both methods, aka: 1) modify dh_haskell to use the existing cabal behavior 2) modify both cabal and dh_haskell to implement the new behavior so I have a bit of real world data. I think the cabal+dh_haskell solution actually took less time, is easier to understand, is easier to extend, and is less error prone :) Additionally, solving the problem in dh_haskell only solves it for Debian users. Packaging maintainers for other systems will have to come up with their own hacks, resulting in duplicate work. If jhc or yhc add profiling, then I have to *hope* they also name things in a totally regular way so that it can be automated.

The ghc devs are trying to keep the number of variants down so I don't foresee n different versions (smp, bytecode etc).

Right -- but the GHC devs are not the only consideration. YHC, for example, does generate bytecode.

Do we need anything more complex?

I think there are really three questions to ask: 1. Should we consider the current Cabal behavior to be broken? If I am not mistaken, one of the driving forces behind Cabal was to make it significantly easier for package maintainers to package libraries by providing a OS and compiler independent interface for building and installing libraries. If I have to start knowing the specifics of how various compilers name object files, then I think that might mean Cabal is falling short of its goals in some area. 2. Are there advantages to fixing it at the Cabal level instead of the package maintainer level? With the flags I proposed, I can write compiler agnostic code to divide the library variations into separate packages, such as -dev, -prof, etc. This makes life easier in the long run. If adding support for a new compiler is work -- it is much less likely to be supported. By putting the requirements in Cabal, it also serves to ensure that compilers will actually support the features that packagers need. Note that Cabal is supposed to help compiler writers by giving them guidelines they should adhere to. From the cabal homepage: "...and what Haskell implementations must do to support packages" Simon Marlow suggested that the features could also be useful for building GHC itself -- allowing them to use Cabal instead of a home-brew system. And finally, it means I get to write some of the code in Haskell instead of perl+bash ;) 3. Is there a less 'complex' solution? This is the big unknown question in my mind. I think that fixing the problem at the Cabal level is the right thing to do. The tricky part is knowing what solution will hold up for the long run. I think my proposal is good because: 1. it is easy to implement now (only a few lines of code) 2. it solves the problems we currently know about 3. if it turns out to be the wrong solution -- we will not be any worse off than we would be if we did nothing. Does this help? Thanks! j.

"Jeremy Shaw" writes:

At Thu, 13 Jul 2006 12:26:22 +0100, Simon Marlow wrote:

...

Jeremy Shaw wrote:

...

(1) Flag sanity checking

Not all combinations of flags are sensible. For example, we can already specify:

...

There's another restriction I know of: if the code uses Template Haskell, then --enable-library-vanilla will always be required. Since the extensions used are all listed in the .cabal file, checking this will be easy.

Hrm, that puts me back in the same situation I am in now -- not having a good way to split the profiling libraries into a separate package.

I have written up some of my thoughts below. I would love to hear opinions from any interested parties, such as compiler writers, package maintainers, end users, etc.

Goals =====

Overall Goal ------------

My overall goal is to add some flags to Cabal that will make it easier for package maintainers to provide several variations of the same library in separate packages. For example, separate packages for the library compiled: vanilla, profiling, smp, bytecode, etc. If a package (such as the profiling) needs files that are already in another package (such as vanilla), it will simply depend on the other package. (i.e. it will *not* contain a second copy of the files)

I like the overall goal, and totally understand the need for it. (snip)

Cabal is nice because it hides all the nasty compiler and platform specific build issues from the user. The user puts *what* they want in the .cabal file, and they let Cabal take care of the *how*.

I think it is very desirable to retain this abstraction in the configure flags as much as possible. To me, this means that the flags:

--disable-library-vanilla --enable-library-profiling

Maybe we should think about having a concept of 'ways' instead of these more one-off flags. --ways=prof,novanilla etc? (snip)

Difficulties ------------

It is difficult to make a specification that all compilers can meet. For example, most compilers do not support profiling, so they can not support the --enable-*-profiling flag at the moment. In this case, we could just extend the spec to say that it is an error to enable profiling for a compiler that does not support profiling.

Building a concept of 'ways' can let the compiler authors say "We support the profiling 'way' but not the bytecode 'way'." Instead of "we support flags xyz".

On the other hand, someone might make a compiler that does not allow you to install the vanilla and profiling libraries at the same time. Now you have the problem that you do want to have a -prof .deb, but it is going to work different from the 'standard'.

Of course, the specifications are supposedly reasonable specifications that are dictated by the actual needs of the users and the package maintainers. So, I think in most cases, having a specification to meet will make it easier for compiler writers, since they will not have to learn the hard way that certain features are important.

Can you document the requirements and assumptions that we make of the compilers? For instance, as cabal hackers who have actually spent time thinking about this, we can tell the yhc and ghc people, "it would be way better if you please do things this way" and they can conform. That is, we can ask the yhc authors to please make sure that when they implement profiling, they make sure that profiling libraries can be installed alongside non-prof libraries. While cabal does its best to change its behavior based on implementations, there's a lot of give-and-take. For instance, GHC added the --hide-all-packages flag for us. So I'm agreeing with you ;) (snip)

How It Would Work In Cabal --------------------------

The spec for the flags:

--maintainer --disable-library-vanilla --enable-library-profiling

is something like,

+ build the libraries with profiling enabled

+ copy out just the files needed add profiling support to a system that already has the vanilla libraries available

+ this assumes that the profiling and vanilla libraries can co-exist (i.e. do not have conflicting files, etc)

Here is how the Cabal code for GHC would meet that spec:

(1) In the 'build' phase:

(i) if ghc requires you to explicitly build the vanilla libraries before building the profiling libraries, then do that.

(ii) build the profiling libraries

(2) In the 'install/copy' phase, *only* copy the profiling libraries out.

(NOTE: This is actually quite easy -- I have already implemented it in Cabal).

This makes sense, but what does the 'maintainer' flag do in this case?

Downside Of This Solution -------------------------

The downside of the above implementation is that the vanilla libraries will be built twice. Once for installation in the -vanilla package, and a second time when trying to create the -prof package.

Couldn't cabal notice that they're already built?

Your Comments Here ==================

This is very good and thoughtful work! Thanks for hacking on this and working to gather input. Folks like you & Duncan, who package Haskell libraries, are indispensable to the Cabal effort. peace, isaac

Oops! I replied to the wrong message, so most of you just received a very mysterious sounding message :) Although out of context, the message is relevant -- namely, I plan to answers Isaac's questions in the parent message sometime soon, but I am a bit swamped at the moment. j. At Wed, 16 Aug 2006 15:06:25 -0700, Jeremy Shaw wrote:

At Thu, 20 Jul 2006 16:21:20 -0700, Isaac Jones wrote:

I should add, I *do* plan to get back to this sometime -- but it is not high on my list this week.

j.