On Fri, 2008-04-18 at 13:59 +0400, Bulat Ziganshin wrote:

Hello Duncan,

Friday, April 18, 2008, 1:43:24 PM, you wrote:

...

older Cabal versions). We can make it stricter again in the future when Cabal-1.4+ is much more widely deployed.

the problem, imho, is that such tools as Cabal, GHC, Hackage should be built with forward and backward compatibility in mind. otherwise, Haskell will still remain mostly a hacker tool

Yes! Yes you're absolutely right, that's why I've been fixing it :-) Duncan

In message <20080418121351.GJ17560@sliver.repetae.net> haskell-cafe@haskell.org writes:

On Fri, Apr 18, 2008 at 10:43:24AM +0100, Duncan Coutts wrote:

...

I have now fixed that by eliminating the use of Read in the .cabal parser and basically adding an Other/Unknown constructor to several of the enumeration types, including Extension. So as of Cabal-1.4 it will be possible to add new extensions in later Cabal versions that are not in Cabal-1.4 without Cabal-1.4 falling over with a parse error. Indeed, if the compiler knows about the extension then it will actually work. The only restriction is that unknown extensions cannot be used in packages uploaded to hackage, which is pretty reasonable I think. If an extension is going to be used in widely distributed packages then that extension should be registered in Language.Haskell.Extension. It's trivial to add and update hackage to recognise it.

And then have everyone have to upgrade their cabal?

No, that's the nice thing about the changes I've made already. It already works (in Cabal-1.4) to use the PArr extension that ghc-6.8.2 supports but is not finalised yet and is therefore not listed in Language.Haskell.Extension. So only the list that the hackage server itself knows about has to be updated so that it can

It should just be

...

newtype Extension = Extension String

Perhaps but the main point I think this that Cabal/Hackage needs to know both what the global list of known extensions and what extensions are supported by particular versions of compilers. It's obviously legitimate for compilers to add new extensions that are not globally registered (and that works now in Cabal-1.4) but I don't think it's legitimate to have such packages be uploaded to hackage. If they're going to be publicly distributed then the extensions they use should be known too. Then there's the simple matter of reporting to users that they've mis-spelled an extension. We need enough information about known extensions to be able to do that.

it would simplify a lot of code, be more forwards and backwards proof, and remove oddness like do 'Other "PatternGuards"' and 'PatternGuards' mean the same thing?

In order to be both backwards and forwards compatable, everyone writing haskell code that links against cabal and cares about extensions will have to have special code treating both the same, and in fact, conditionally compile part of it, since 'PatternGuards' might not even be valid in some old version of cabal.

(replace PatternGuards with some other soon to be standardized extension)

Normalized data types are a good thing. A centralized registry of things hackage recognizes is fine, but it shouldn't be cluttering up the source code.

Yeah maybe. I don't really care about the representation so long as we can all agree about what users/packagers/hackage/compilers want from extensions. Duncan

In message <200804180805.12793.jgoerzen@complete.org> John Goerzen writes:

On Fri April 18 2008 4:43:24 am Duncan Coutts wrote:

...

...

It seems arbitrary that Hackage would suddenly reject this valid usage.

Yes it is valid though I hope you can see the general intention of the suggestion. If it were not for the compatibility problem it would be preferable to use:

Sure, I do. It's a good point.

But I think there are a couple of issues here:

1) Hackage is rejecting things that Cabal accepts without trouble

This is by design. There are dubious things you can do locally that are not really acceptable in publicly distributed packages. As a simple example you can't put 'AllRightsReserved' packages on the public hackage. Hackage upload is an excellent point to apply stricter QA for distributed packages than what you'd use locally for quick hacks. Remember that Cabal is aimed as a haskell build system as well as a haskell package distribution system. So it does not make sense in every context to apply the strictest levels of checking. For example in the longer term I'd like to see 'cabal build Foo.hs' as an upgraded 'ghc --make Foo.hs' ie just the ordinary build stuff - but with parallel build and understanding preprocessors and without having to supply a .cabal file. So the Cabal checking code has a few levels of severity and depending on the context we make those fatal errors, warnings or ignore them completely. The strictest levels of checks are reserved for distributing packages. It's easy to adjust the levels for individual checks.

2) The behavior of Hackage is unpredictable in what it will accept and what it will reject

Actually with the latest Cabal it's quite predictable. 'cabal sdist' now reports exactly the same errors and warnings as hackage upload. You see the difference at the moment because you're using an older version of Cabal compared to the one on hackage. There is also a new 'cabal check' command for running these additional QA checks (which we hope to extend with more expensive checks along the lines of autotools's 'make distcheck' feature).

3) The behavior of Hackage changes rapidly

They will remain synchronised because Hackage just uses the Cabal library to do its checks. It changed very rapidly recently because we added this checking infrastructure and added dozens of new checks based on looking at the dubious things people have put in existing .cabal files in hackage packages.

It's been quite frustrating lately as many of my packages that used to upload fine still build fine but get rejected at upload time.

If there are any that you think are rejecting legitimate packages then do complain (as in this thread). They're not set in stone. Probably better to complain to cabal-devel or the libraries list, or file bug repots so we spot them. I do realise there is the danger of going too far and having pointless finicky rules. We rely on feedback on this.

I think that Hackage is the wrong place for these checks.

I disagree. I think it's absolutely the best place for these checks. Of course they need to be in the client too, that's coming soon with Cabal-1.4 (or now if you use the development version of Cabal). As an ex-linux-distro maintainer I think this is absolutely the right thing to do - to automate and distribute QA as much as possible. Maintaining and improving the quality of the hackage collection is really important.

This stuff should go in Cabal, and ./setup configure should print a big DEPRECATION WARNING for a major release before the stuff gets yanked.

Yes, that's what will happen when you use the new Cabal (the same version that hackage is using). Though it will not print all warnings on configure because we think that'd probably be too annoying for people working on quick hacks. But it does run the full set of checks with the 'check' and 'sdist' commands.

...

So that obviously does not solve the problem that Cabal-1.2 and older are not very good with forwards compat in the parser. The solution is probably just to downgrade that check to a warning rather than outright rejection (or possibly limit the check to extensions that existed in older Cabal versions). We can make it stricter again in the future when Cabal-1.4+ is much more widely deployed.

Sound ok?

Yes, that makes a lot of sense, too. Can cabal-put be tweaked to make sure to output that warning by default?

There's a bug open on that and Ross and Bjorn are working on getting the cgi upload script to reporte errors/warnings to http clients that only accept text not html output (ie cabal upload). Does that help explain what's going on and what we're up to with this checking stuff? I should also note that there will be a Cabal-1.4 and cabal-install release in the near future but you can grab the pre-releases or the darcs versions now and try things out and report any problems. The first pre-release was announced to the libraries list a bit over a week ago. I'll probably announce the next pre-release on this list so we get wider testing. Duncan

On Apr 21, 2008, at 7:38 , Hans van Thiel wrote:

On Fri, 2008-04-18 at 16:19 +0100, Duncan Coutts wrote:

...

If there are any that you think are rejecting legitimate packages then do complain (as in this thread). Configuring Emping-0.5.1... cabal-setup: At least the following dependencies are missing: gtk -any

cabal-install doesn't handle the case of ghc packages that aren't named the same as hackage packages? (gtk is part of the gtk2hs hackage) -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH

On Mon, 2008-04-21 at 13:38 +0200, Hans van Thiel wrote:

Configuring Emping-0.5.1... cabal-setup: At least the following dependencies are missing: gtk -any

I'll have to leave it at that, since my local Cabal version does configure and build, and I obviously can't use Hackage as a development tool, uploading new versions until I get it right. Or, on second thought, does the above message just mean there's no gtk on Hackage?

That's right.

Some general questions, suggestions:

To improve the usability of Hackage, wouldn't it be better to hold new uploads in limbo until they pass all the checks, instead of publishing them anyway?

Yes. Or the way we've been thinking about it is to define various subsets of hackage of packages that pass various automatic and manual tests. Of course at the moment the package build is not sufficiently accurate to determine if a package builds eg the gtk issue you found. The plan there is to use the cabal-install client to gather build reports.

The current a priori check, which said there were no fatal errors, while the a posteriori check failed, is misleading. Wouldn't it be better to warn potential uploaders that this first check is not complete?

I'm not sure I see what you're getting at. We can't do a full build check synchronously at upload time.

I'd also like the warning about the -O2 optimization flag to go away...I'd guess this is a leftover from an earlier time in GHC.

It's not a leftover, it was added recently. The intention is that user gets to set the optimisation level (cabal configure -O or -O2 or -O0). It's not really great when packages override that choice. From experience packaging haskell libraries for gentoo we found that most packages that do set -O2 do so out of habit without any particular justification and it significantly increases compile times and memory use. It's not banned though as there are some cases where it's perfectly justified.

More importantly, if Cabal is sort of an rpm for Haskell programs, shouldn't the uploader use version numbers of the libraries in the Cabal file? For example, I believe in my case at least gtk2hs 0.9.12 is needed, while Ubuntu seems to have only 0.9.11 (until a few months ago, at least). So, if an uploader knows such dependencies, how to express them?

build-depends: gtk >= 0.9.12 && < 0.10 Duncan