cabal for ghc 6.8.2, package upgrade warnings, please?
assuming that ghc 6.8.2 is not too far away, is there any chance of cabal becoming a bit friendlier wrt old .cabal and Setup files, before that release? you've heard me say before that i'd have preferred a backwards compatibility module, but if that isn't possible, couldn't cabal at least detect that a .cabal file is probably outdated (==not obviously up to date), and issue a warning with a helpful url? something like: $ runhaskell Setup configure *** warning: this .cabal/Setup file might be out of date; *** if you run into issues, please contact the *** package maintainer, and have a look at *** http://haskell.org/haskellwiki/Upgrading_packages .. (either it works or you get errors here, but you already know something is not quite right, and what to do about it) btw, i knew that page existed, but had quite a bit of trouble finding it. shouldn't there be prominent links to it from cabal and hackage home pages? as in: concerning recent cabal updates (look here first if you have trouble with old .cabal packages). in the longer term, that could become the start of a cabal/hackage faq section.. claus
On Fri, 2007-11-30 at 23:17 +0000, Claus Reinke wrote:
assuming that ghc 6.8.2 is not too far away, is there any chance of cabal becoming a bit friendlier wrt old .cabal and Setup files, before that release?
We're pretty much fully backwards compatible with old .cabal files, the only problem is Setup.hs files.
you've heard me say before that i'd have preferred a backwards compatibility module, but if that isn't possible, couldn't cabal at least detect that a .cabal file is probably outdated (==not obviously up to date), and issue a warning with a helpful url? something like:
This would be great, unfortunately I don't think it's technically feasible with the current design. At the moment, the Setup.hs is the first entry point and if that doesn't compile then we're stuffed. We have no opportunity to offer helpful messages, the user just gets a deeply unfriendly type error message. Setup.hs files are allowed to use the entire Cabal api, which includes a lot of useful stuff but it completely exposes the inner workings. Since we keep improving the inner workings that api keeps changing. Even the more public parts place constraints on the implementation that we would like to lift in future (esp the UserHooks api). I suppose we could have kept exactly the old api and had it redirect the defaultMain and related entry points to the new implementation. The downside is that since the Cabal api is more or less the complete cabal implementation, keeping the old api and adding a new implementation would involve keeping the complete old implementation and adding a new copy. Then repeat for each api revision. Perhaps we should be using Eternal Compatibility in Theory. However I don't much relish the prospect of trying to make old apis work in terms of new implementations or explaining to package builders why they cannot use certain features with certain packages that use the old library api. The current approach works about 90%, in that 10% of packages broke in the transition from Cabal 1.1.6 to 1.2 (according to my hackage survey). We are certainly going to change the Cabal api again in incompatible ways, if only to make an api that looks like it was designed as an api that has a chance of being stable. So we have the chance to plan for a better transition in future. So if anyone has any feasible technical suggestions...
btw, i knew that page existed, but had quite a bit of trouble finding it. shouldn't there be prominent links to it from cabal and hackage home pages?
Yes, you're quite right. I've added a link from the cabal website and I've reorganised the page so it's hopefully rather clearer: http://haskell.org/haskellwiki/Upgrading_packages
as in: concerning recent cabal updates (look here first if you have trouble with old .cabal packages). in the longer term, that could become the start of a cabal/hackage faq section..
Duncan
We're pretty much fully backwards compatible with old .cabal files, the only problem is Setup.hs files.
compatible, as in "can process them", but some things, like the base split, will lead to plain failures, even though cabal could put 2 and 2 together when it sees an old .cabal file and missing dependencies in packages split from base. the package maintainer hasn't updated, the package builder has no idea what goes wrong, the whole thing breaks because of outside influences (libraries updates), and cabal could make a good guess, it just doesn't tell.
This would be great, unfortunately I don't think it's technically feasible with the current design. At the moment, the Setup.hs is the first entry point and if that doesn't compile then we're stuffed. We have no opportunity to offer helpful messages, the user just gets a deeply unfriendly type error message.
ah, yes, that is a problem. but even if non-standard Setup.hs is now discouraged, it is still part of cabal's design, so the same issue will come up the next time you improve the api. what about the old idea that there should be a 'cabal' script? in the past, that seemed somewhat superfluous, as it wouldn't have done anything but call "runhaskell Setup", but now it seems that extra level of indirection would allow for helpful checks and messages before or after compilation of Setup.hs fails. that could include format and age checks on .cabal (if it doesn't mention any new dependencies or isn't precise in versions, there are some likely issues it will encounter with a new ghc/cabal version; if it depends on base, but needs containers, it is pre-base-split, etc.), import/type checks on Setup.hs (does it expect the old or the new api), or rough postprocessing of error messages from compiling Setup.hs (pattern matching on the kind of type errors or missing import errors that unambiguously imply old api).
Setup.hs files are allowed to use the entire Cabal api, which includes a lot of useful stuff but it completely exposes the inner workings. Since we keep improving the inner workings that api keeps changing. Even the more public parts place constraints on the implementation that we would like to lift in future (esp the UserHooks api). ..
ok, i think i've accepted before that cabal's design would make it difficult to have a compatibility module. and you say that this design gives you liberties that you wouldn't want to trade for a more restrictive, more maintenance- friendly design. okay, but that isn't the only way to help with maintenance: - prefixing 'Compat.' to all cabal imports in Setup.hs seems out - too much duplication of implementation. but perhaps that duplication of implementation would have been worth it for a few months, if only to be able to issue warnings and give package authors or third parties some time to upgrade their packages. - if the new api can do everything the old api can do, is it possible to transform an old Setup.hs into one that would very nearly work with the new api? using Language.Haskell's parser/ast/pretty-printer and one of the no-boilerplater libraries, one could search for uses of the old api and replace with uses of the new api wherever drop-in replacements are available. the bulk of the code would be rules encoding old->new replacements. when you changed cabal's api, you must have had some ideas about how the new api would replace uses of the old api? - even if the transformation cannot be automated completely, can the old code be analysed to issue warnings about features it uses which are no longer available, together with a link to a wiki page that explains how to replace those uses of old cabal api features with uses of new cabal api features? again, give the package builder more info than "it doesn't compile/build". - is there a wiki page that explains the cabal api changes in more detail, with rationale, with examples of how to replace old uses with new? something that cabal and hackage home pages, as well as warning issued by cabal, could point to.
The current approach works about 90%, in that 10% of packages broke in the transition from Cabal 1.1.6 to 1.2 (according to my hackage survey).
that survey was useful, but it only covered hackage, and even there it found 10% breakage not caused by package authors or builders. 10% breakage just by time passing and by cabal not wanting to provide the old implementation side-by-side with the new, not by package providers making any changes.. as you say, the cabal is going to move on, so this is an opportunity to see what issues come up and how things might be changed to handle such issues better next time. even simple things help, such as an additional indirection (a 'cabal' script calling 'runhaskell Setup', now able to give warnings instead of fail), wiki pages documenting what people have to do to follow cabal or library changes, warning messages that direct package builders to such wiki pages. is there a hackage mailing list offering to keep all package maintainers informed about build results, breakage, issues, api changes, discussion? the email addresses could be extracted from the .cabal files being uploaded, but hackage would need to ask for permission before adding anyone to such list.
I've added a link from the cabal website and I've reorganised the page so it's hopefully rather clearer:
thanks. the part that still needs expansion is the part 1.3 on cabal api changes: any feature that the old cabal made available which is not unchanged in the new api should be listed, side-by-side with intended replacements. that is, a user-level equivalent to the page that would explain and summarize the cabal design changes. claus
On Sat, 2007-12-01 at 14:05 +0000, Claus Reinke wrote:
We're pretty much fully backwards compatible with old .cabal files, the only problem is Setup.hs files.
compatible, as in "can process them", but some things, like the base split, will lead to plain failures, even though cabal could put 2 and 2 together when it sees an old .cabal file and missing dependencies in packages split from base.
the package maintainer hasn't updated, the package builder has no idea what goes wrong, the whole thing breaks because of outside influences (libraries updates), and cabal could make a good guess, it just doesn't tell.
I'm not sure we actually can guess if a package is broken by the base split or not. build-depends: base so does it really only depend on base or does it depend on base, pretty, directory etc etc. No idea. We can take secondary things into account and note the lack of newer fields and syntax as an indication of an older package.
This would be great, unfortunately I don't think it's technically feasible with the current design. At the moment, the Setup.hs is the first entry point and if that doesn't compile then we're stuffed. We have no opportunity to offer helpful messages, the user just gets a deeply unfriendly type error message.
ah, yes, that is a problem. but even if non-standard Setup.hs is now discouraged, it is still part of cabal's design, so the same issue will come up the next time you improve the api.
what about the old idea that there should be a 'cabal' script? in the past, that seemed somewhat superfluous, as it wouldn't have done anything but call "runhaskell Setup", but now it seems that extra level of indirection would allow for helpful checks and messages before or after compilation of Setup.hs fails.
We do have this now and we'll be promoting it more in future. We currently have the cabal-setup program and the cabal-install program. We're currently working on merging the two into a more general cabal front end / cli UI program. cabal-setup does some of these things, it reads the .cabal file first and then if necessary it compiles the Setup.hs script against some hopefully appropriate version of the Cabal library. Unfortunately that doesn't help much since packages generally only specify lower not upper bounds on cabal version numbers: cabal-version: >=1.2 and the like. This is the same problem as we have with package dependencies. The package version policy is one approach to solving this.
that could include format and age checks on .cabal (if it doesn't mention any new dependencies or isn't precise in versions, there are some likely issues it will encounter with a new ghc/cabal version; if it depends on base, but needs containers, it is pre-base-split, etc.), import/type checks on Setup.hs (does it expect the old or the new api), or rough postprocessing of error messages from compiling Setup.hs (pattern matching on the kind of type errors or missing import errors that unambiguously imply old api).
We currently cannot check if a package depends on containers without declaring it because Cabal does not do any of it's own module chasing. When it does we'll be able to do a bit better here.
Setup.hs files are allowed to use the entire Cabal api, which includes a lot of useful stuff but it completely exposes the inner workings. Since we keep improving the inner workings that api keeps changing. Even the more public parts place constraints on the implementation that we would like to lift in future (esp the UserHooks api). ..
ok, i think i've accepted before that cabal's design would make it difficult to have a compatibility module. and you say that this design gives you liberties that you wouldn't want to trade for a more restrictive, more maintenance- friendly design. okay, but that isn't the only way to help with maintenance:
- prefixing 'Compat.' to all cabal imports in Setup.hs seems out - too much duplication of implementation. but perhaps that duplication of implementation would have been worth it for a few months, if only to be able to issue warnings and give package authors or third parties some time to upgrade their packages.
It's not the biggest problem. As I said it only broke 10% of packages, the vast majority of packages were broken by the base split.
- if the new api can do everything the old api can do, is it possible to transform an old Setup.hs into one that would very nearly work with the new api?
using Language.Haskell's parser/ast/pretty-printer and one of the no-boilerplater libraries, one could search for uses of the old api and replace with uses of the new api wherever drop-in replacements are available. the bulk of the code would be rules encoding old->new replacements.
Sounds like a lot of work for relatively little benefit.
when you changed cabal's api, you must have had some ideas about how the new api would replace uses of the old api?
Some, but as I've said, there's no clear line about what is a public Cabal api and what is just internal stuff. We currently export pretty much everything. Most of the api bits we change we do not expect many packages are using. We do try rather harder not to change the bits we expect more packages use. Generally we consider the internal/external line to be Distribution.Simple.* vs Distribution.* but there are certainly various useful things that some Setup.hs scripts use from within Distribution.Simple. We don't have a very good feel for what those bits are exactly.
- even if the transformation cannot be automated completely, can the old code be analysed to issue warnings about features it uses which are no longer available, together with a link to a wiki page that explains how to replace those uses of old cabal api features with uses of new cabal api features? again, give the package builder more info than "it doesn't compile/build".
- is there a wiki page that explains the cabal api changes in more detail, with rationale, with examples of how to replace old uses with new?
something that cabal and hackage home pages, as well as warning issued by cabal, could point to.
This is all stuff we should have of course. My immediate reaction I admit is that it sounds like a lot of work that might be better invested in just improving the design of cabal generally.
The current approach works about 90%, in that 10% of packages broke in the transition from Cabal 1.1.6 to 1.2 (according to my hackage survey).
that survey was useful, but it only covered hackage, and even there it found 10% breakage not caused by package authors or builders. 10% breakage just by time passing and by cabal not wanting to provide the old implementation side-by-side with the new, not by package providers making any changes..
We can and do provide the old implementation side-by-side. You can install multiple versions of the Cabal lib.
as you say, the cabal is going to move on, so this is an opportunity to see what issues come up and how things might be changed to handle such issues better next time.
even simple things help, such as an additional indirection (a 'cabal' script calling 'runhaskell Setup', now able to give warnings instead of fail), wiki pages documenting what people have to do to follow cabal or library changes, warning messages that direct package builders to such wiki pages.
Or designing the Cabal API with external users in mind and making everything else private.
is there a hackage mailing list offering to keep all package maintainers informed about build results, breakage, issues, api changes, discussion?
There is this list. The hackage build results to not get emailed to anyone.
the email addresses could be extracted from the .cabal files being uploaded, but hackage would need to ask for permission before adding anyone to such list.
I've added a link from the cabal website and I've reorganised the page so it's hopefully rather clearer:
thanks. the part that still needs expansion is the part 1.3 on cabal api changes: any feature that the old cabal made available which is not unchanged in the new api should be listed, side-by-side with intended replacements.
There are a lot of api changes that are pretty much irrelevant for most Setup.hs scripts. We could list the top few most likely changes summarising from the hackage survey email I sent round.
that is, a user-level equivalent to the page that would explain and summarize the cabal design changes.
What you're proposing generally really is a lot of work and I think requires rather more active Cabal hackers, or a much slower rate of change. Duncan
On Sun, 2007-12-02 at 22:47 +0000, Duncan Coutts wrote:
What you're proposing generally really is a lot of work and I think requires rather more active Cabal hackers, or a much slower rate of change.
Sorry Claus, I don't want to put you off making suggestions. It would be helpful to record the ones we agree are technically feasible in the trac as feature requests / tasks. That way we don't forget and it provides a list for interested hackers to pick up without having to follow the mailing list discussions too closely. Duncan
I'm not sure we actually can guess if a package is broken by the base split or not.
build-depends: base
so does it really only depend on base or does it depend on base, pretty, directory etc etc. No idea.
not directly. but you could then filter the error messages and provide helpful higher-level hints. example filter rules: - build-depends: base error message: "member of package _, which is hidden" (where _ is any of the packages split from base) emit hint: the base package has been split up, please check <url> for package upgrade information - error message: Couldn't match expected type `(Either GenericPackageDescription PackageDescription, HookedBuildInfo)' emit hint: Setup.[l]hs uses part of the internal cabal api that has changed. see <url> for upgrade tips or ask on cabal-devel@. - there is no build-type field emit warning: please update package description. see <url> - there is a non-standard Setup.[l]hs emit warning: non-standard Setup.[l]hs, uses internal cabal api. this api is subject to development, and need not be used for most packages. see <url> (the url should invite package authors to report gaps that force them to write their own Setup.[l]hs to cabal-devel@) such things could work by simple pattern matching over the compiler error messages, in the cabal commandline interface. a lot cheaper than the other alternatives, can be improved incrementally, and provides users with high-level hints and further information on what to do. anything to prevent them from running away blogging noisily about how haskell packages do not even install.
We do have this now and we'll be promoting it more in future. We currently have the cabal-setup program and the cabal-install program. We're currently working on merging the two into a more general cabal front end / cli UI program.
similar to darcs, ghc-pkg, etc? that would be great. Setup.hs should neither be mentioned ("cabal configure" instead of "runhaskell Setup.hs configure") nor exist in the package directory (a library of useful defaults, selected by .cabal build-types, should be part of the cabal installation). explicit use of Setup.hs should be reserved for exceptions, where cabal does not (yet) cover the functionality needed.
cabal-version: >=1.2 .. and the like. This is the same problem as we have with package dependencies. The package version policy is one approach to solving this.
if that is likely to cause problems, how about a warning, with link to policy url?-) and how do you determine the cabal api version a Setup.hs was written against? if the Setup.hs is non-standard, any use of ranges in cabal-version: (let alone absence of that field) should be an error.
We currently cannot check if a package depends on containers without declaring it because Cabal does not do any of it's own module chasing. When it does we'll be able to do a bit better here.
okay, but you could help the package user to interpret the compiler error messages, as outlined above.
This is all stuff we should have of course. My immediate reaction I admit is that it sounds like a lot of work that might be better invested in just improving the design of cabal generally.
that would be true if cabal wasn't already out in the field!-)
What you're proposing generally really is a lot of work and I think requires rather more active Cabal hackers, or a much slower rate of change.
hm, darcs has the same problem. how about using the upcoming communities report to ask for some essential community tool hacking?-) the cabal and darcs developers could set up a page with well-defined tasks in priority order, and invite all those shootout/euler/xmonad hackers to spend January, say, on tackling as many of those tasks as possible while discussing the issues arising on the usual mailing lists and collecting karma;-) it should be restricted to darcs+cabal, to avoid losing hacking power to other distracting projects, it should be limited in time, to get people to focus their efforts, and the core developers would have to prepare the ground, select the tasks, and provide assistance. you could coordinate with the darcs developers, and haskell-cafe people do tend to flock toward projects where they can be seen and can chat with others working on the same issues:-) just another thought, claus
On Mon, 2007-12-03 at 17:08 +0000, Claus Reinke wrote:
I'm not sure we actually can guess if a package is broken by the base split or not.
build-depends: base
so does it really only depend on base or does it depend on base, pretty, directory etc etc. No idea.
not directly. but you could then filter the error messages and provide helpful higher-level hints. example filter rules:
- build-depends: base error message: "member of package _, which is hidden" (where _ is any of the packages split from base) emit hint: the base package has been split up, please check <url> for package upgrade information
I really don't like the idea of trying to parse ghc's error messages and trying to divine some info out of them. I think the development time needed to do that could be better spent making Cabal do dependency analysis, in which case we could do our own error for when a package is needed but it has not been declared as a dependency (or allow it to be just a warning in some cases).
- error message: Couldn't match expected type `(Either GenericPackageDescription PackageDescription, HookedBuildInfo)' emit hint: Setup.[l]hs uses part of the internal cabal api that has changed. see <url> for upgrade tips or ask on cabal-devel@.
- there is no build-type field emit warning: please update package description. see <url>
Yes, I think that's a good idea.
- there is a non-standard Setup.[l]hs emit warning: non-standard Setup.[l]hs, uses internal cabal api. this api is subject to development, and need not be used for most packages. see <url> (the url should invite package authors to report gaps that force them to write their own Setup.[l]hs to cabal-devel@)
such things could work by simple pattern matching over the compiler error messages, in the cabal commandline interface. a lot cheaper than the other alternatives, can be improved incrementally, and provides users with high-level hints and further information on what to do.
But it is legitimate to use a custom Setup.hs. I don't think we should cast aspersions over them unnecessarily. Now if the cabal script compiles the custom Setup.hs and it fails to compile then sure, we can provide our best guesses and pointers to further information.
anything to prevent them from running away blogging noisily about how haskell packages do not even install.
We do have this now and we'll be promoting it more in future. We currently have the cabal-setup program and the cabal-install program. We're currently working on merging the two into a more general cabal front end / cli UI program.
similar to darcs, ghc-pkg, etc? that would be great.
You can try both now. They're available from hackage and the development versions are on darcs.haskell.org.
Setup.hs should neither be mentioned ("cabal configure" instead of "runhaskell Setup.hs configure")
Right, cabal-install and cabal-setup do not mention Setup.hs in their help messages.
nor exist in the package directory (a library of useful defaults, selected by .cabal build-types, should be part of the cabal installation).
The Setup.hs is only required if you use build-type: Custom.
explicit use of Setup.hs should be reserved for exceptions, where cabal does not (yet) cover the functionality needed.
That is indeed current policy. So we should consider promoting the use of build-type by warning whenever it is missing.
cabal-version: >=1.2 .. and the like. This is the same problem as we have with package dependencies. The package version policy is one approach to solving this.
if that is likely to cause problems, how about a warning, with link to policy url?-)
and how do you determine the cabal api version a Setup.hs was written against? if the Setup.hs is non-standard, any use of ranges in cabal-version: (let alone absence of that field) should be an error.
I'm included to agree. Perhaps if build-type is Custom then we should require both a lower and upper bound in the cabal-version field. We can assume that Cabal itself is following the package version policy.
We currently cannot check if a package depends on containers without declaring it because Cabal does not do any of it's own module chasing. When it does we'll be able to do a bit better here.
okay, but you could help the package user to interpret the compiler error messages, as outlined above.
That does seem pretty hard.
This is all stuff we should have of course. My immediate reaction I admit is that it sounds like a lot of work that might be better invested in just improving the design of cabal generally.
that would be true if cabal wasn't already out in the field!-)
What you're proposing generally really is a lot of work and I think requires rather more active Cabal hackers, or a much slower rate of change.
hm, darcs has the same problem. how about using the upcoming communities report to ask for some essential community tool hacking?-)
Yes.
the cabal and darcs developers could set up a page with well-defined tasks in priority order, and invite all those shootout/euler/xmonad hackers to spend January, say, on tackling as many of those tasks as possible while discussing the issues arising on the usual mailing lists and collecting karma;-)
We have the trac. We can use the priority and field that indicates the amount of time we think the task will take.
it should be restricted to darcs+cabal, to avoid losing hacking power to other distracting projects, it should be limited in time, to get people to focus their efforts, and the core developers would have to prepare the ground, select the tasks, and provide assistance. you could coordinate with the darcs developers, and haskell-cafe people do tend to flock toward projects where they can be seen and can chat with others working on the same issues:-)
just another thought,
Thanks. Duncan
okay, but you could help the package user to interpret the compiler error messages, as outlined above.
That does seem pretty hard.
it isn't nice, and it will never cover all cases. but the
issues are out there now, and offering the innocent
package builder some help shouldn't be that hard.
a quick hack with a couple of examples is attached -
needs work, but doesn't look "hard" in principle.
it is probably easy to get wrong, like all such things,
but as long as the current partially-broken state of
doing things is still available, offering a helpful command
wrapper could help to reduce the impact of those
package issues. and the second release of a new
ghc version tends to live a lot longer than the first..
you could probably scrape the most relevant
message patterns right from your hackage survey,
hackage build logs and recent haskell-cafe build
failure emails.
claus
begin 666 cabal.hs
M:6UP;W)T(%-Y
participants (2)
-
Claus Reinke -
Duncan Coutts