On Sat, 2007-12-01 at 14:05 +0000, Claus Reinke wrote:
We're pretty much fully backwards compatible with old .cabal files, the only problem is Setup.hs files.
compatible, as in "can process them", but some things, like the base split, will lead to plain failures, even though cabal could put 2 and 2 together when it sees an old .cabal file and missing dependencies in packages split from base.
the package maintainer hasn't updated, the package builder has no idea what goes wrong, the whole thing breaks because of outside influences (libraries updates), and cabal could make a good guess, it just doesn't tell.
I'm not sure we actually can guess if a package is broken by the base split or not. build-depends: base so does it really only depend on base or does it depend on base, pretty, directory etc etc. No idea. We can take secondary things into account and note the lack of newer fields and syntax as an indication of an older package.
This would be great, unfortunately I don't think it's technically feasible with the current design. At the moment, the Setup.hs is the first entry point and if that doesn't compile then we're stuffed. We have no opportunity to offer helpful messages, the user just gets a deeply unfriendly type error message.
ah, yes, that is a problem. but even if non-standard Setup.hs is now discouraged, it is still part of cabal's design, so the same issue will come up the next time you improve the api.
what about the old idea that there should be a 'cabal' script? in the past, that seemed somewhat superfluous, as it wouldn't have done anything but call "runhaskell Setup", but now it seems that extra level of indirection would allow for helpful checks and messages before or after compilation of Setup.hs fails.
We do have this now and we'll be promoting it more in future. We currently have the cabal-setup program and the cabal-install program. We're currently working on merging the two into a more general cabal front end / cli UI program. cabal-setup does some of these things, it reads the .cabal file first and then if necessary it compiles the Setup.hs script against some hopefully appropriate version of the Cabal library. Unfortunately that doesn't help much since packages generally only specify lower not upper bounds on cabal version numbers: cabal-version: >=1.2 and the like. This is the same problem as we have with package dependencies. The package version policy is one approach to solving this.
that could include format and age checks on .cabal (if it doesn't mention any new dependencies or isn't precise in versions, there are some likely issues it will encounter with a new ghc/cabal version; if it depends on base, but needs containers, it is pre-base-split, etc.), import/type checks on Setup.hs (does it expect the old or the new api), or rough postprocessing of error messages from compiling Setup.hs (pattern matching on the kind of type errors or missing import errors that unambiguously imply old api).
We currently cannot check if a package depends on containers without declaring it because Cabal does not do any of it's own module chasing. When it does we'll be able to do a bit better here.
Setup.hs files are allowed to use the entire Cabal api, which includes a lot of useful stuff but it completely exposes the inner workings. Since we keep improving the inner workings that api keeps changing. Even the more public parts place constraints on the implementation that we would like to lift in future (esp the UserHooks api). ..
ok, i think i've accepted before that cabal's design would make it difficult to have a compatibility module. and you say that this design gives you liberties that you wouldn't want to trade for a more restrictive, more maintenance- friendly design. okay, but that isn't the only way to help with maintenance:
- prefixing 'Compat.' to all cabal imports in Setup.hs seems out - too much duplication of implementation. but perhaps that duplication of implementation would have been worth it for a few months, if only to be able to issue warnings and give package authors or third parties some time to upgrade their packages.
It's not the biggest problem. As I said it only broke 10% of packages, the vast majority of packages were broken by the base split.
- if the new api can do everything the old api can do, is it possible to transform an old Setup.hs into one that would very nearly work with the new api?
using Language.Haskell's parser/ast/pretty-printer and one of the no-boilerplater libraries, one could search for uses of the old api and replace with uses of the new api wherever drop-in replacements are available. the bulk of the code would be rules encoding old->new replacements.
Sounds like a lot of work for relatively little benefit.
when you changed cabal's api, you must have had some ideas about how the new api would replace uses of the old api?
Some, but as I've said, there's no clear line about what is a public Cabal api and what is just internal stuff. We currently export pretty much everything. Most of the api bits we change we do not expect many packages are using. We do try rather harder not to change the bits we expect more packages use. Generally we consider the internal/external line to be Distribution.Simple.* vs Distribution.* but there are certainly various useful things that some Setup.hs scripts use from within Distribution.Simple. We don't have a very good feel for what those bits are exactly.
- even if the transformation cannot be automated completely, can the old code be analysed to issue warnings about features it uses which are no longer available, together with a link to a wiki page that explains how to replace those uses of old cabal api features with uses of new cabal api features? again, give the package builder more info than "it doesn't compile/build".
- is there a wiki page that explains the cabal api changes in more detail, with rationale, with examples of how to replace old uses with new?
something that cabal and hackage home pages, as well as warning issued by cabal, could point to.
This is all stuff we should have of course. My immediate reaction I admit is that it sounds like a lot of work that might be better invested in just improving the design of cabal generally.
The current approach works about 90%, in that 10% of packages broke in the transition from Cabal 1.1.6 to 1.2 (according to my hackage survey).
that survey was useful, but it only covered hackage, and even there it found 10% breakage not caused by package authors or builders. 10% breakage just by time passing and by cabal not wanting to provide the old implementation side-by-side with the new, not by package providers making any changes..
We can and do provide the old implementation side-by-side. You can install multiple versions of the Cabal lib.
as you say, the cabal is going to move on, so this is an opportunity to see what issues come up and how things might be changed to handle such issues better next time.
even simple things help, such as an additional indirection (a 'cabal' script calling 'runhaskell Setup', now able to give warnings instead of fail), wiki pages documenting what people have to do to follow cabal or library changes, warning messages that direct package builders to such wiki pages.
Or designing the Cabal API with external users in mind and making everything else private.
is there a hackage mailing list offering to keep all package maintainers informed about build results, breakage, issues, api changes, discussion?
There is this list. The hackage build results to not get emailed to anyone.
the email addresses could be extracted from the .cabal files being uploaded, but hackage would need to ask for permission before adding anyone to such list.
I've added a link from the cabal website and I've reorganised the page so it's hopefully rather clearer:
thanks. the part that still needs expansion is the part 1.3 on cabal api changes: any feature that the old cabal made available which is not unchanged in the new api should be listed, side-by-side with intended replacements.
There are a lot of api changes that are pretty much irrelevant for most Setup.hs scripts. We could list the top few most likely changes summarising from the hackage survey email I sent round.
that is, a user-level equivalent to the page that would explain and summarize the cabal design changes.
What you're proposing generally really is a lot of work and I think requires rather more active Cabal hackers, or a much slower rate of change. Duncan