On Saturday 01 September 2007 16:45, Thomas Schilling wrote:

On Sat, 2007-09-01 at 14:39 +0100, Claus Reinke wrote:

...

[subject taken from cvs-libraries@; discussion directed to libraries@]

There are even more patches sitting on my disk regarding this area, stay tuned... :-P

[...] So, could you please exclude Cabal (and stay with me when I say we hope to have things stable by the next ghc release), re-evaluate the situation excluding anything related to Cabal issues (ie, setup.lhs errors), and see if your issue still exists?

The main problem is that I've been hearing the sentence "Cabal is unstable at the moment, but with the next GHC release everything will be fixed and rock-solid, never changing again, ..." at least for a year now. In my experience, Cabal is *the* #1 reason for breaking build for aeons, and this is really getting frustrating. When trying to build a GHC RPM at an arbitrary point in time, you have an almost 99% chance that it won't work. The GHC project has really made a step backwards in this respect, and I hope that this will improve again. I really wish back the good old times when "make" was king... (Does anybody remember changes in "make"'s basic syntax? I don't...) And while we are at regressions: Although darcs concepts are OK (although personally I would have been happier with Subversion's model, you can always easily do a 3-way merge for personal development), the performance for getting repositories is ridiculous: Due to various obscure things I've experienced, partial repositories are not an option for developers, but getting complete repositories for GHC + extra libs takes about half a day, even when you have a "fat" line to the Internet. The tarball snapshots of the repositories are not really an option in the long run IMHO and defeat the purpose of a versioning tool. To be usable, a speedup of at least factor 10 would be required. Is there any hope for this? The aging CVS at least scaled... Cheers, S. (going back to the next build failure...)

On Sunday 02 September 2007 22:59, Ian Lynagh wrote:

On Sat, Sep 01, 2007 at 05:24:51PM +0200, Sven Panne wrote: [...]

...

To be usable, a speedup of at least factor 10 would be required. Is there any hope for this?

Patch application can be sped up with a planned change of hunk format. If downloading lots of small patch files individually is the problem then darcs could tar them up when checkpointed tags are made.

Will this improve the time for a "./darcs-all pull" for the GHC repository, too? With extra-libraries it takes over 20 minutes for me, even though only 3-4 small patches have actually been pulled and my 2Mbit DSL line sits basically idle during the whole time. This is fairly disappointing... :-( Cheers, S.

On Tue, Sep 04, 2007 at 06:47:06PM +0200, Sven Panne wrote:

too? With extra-libraries it takes over 20 minutes for me, even though only 3-4 small patches have actually been pulled and my 2Mbit DSL line sits basically idle during the whole time.

By the way, if you're currently pulling over SSH then you'll probably find it much faster to pull over HTTP. With no patches to pull I get this for ghc + testsuite + corelibs: ./darcs-all pull -a 2.21s user 0.42s system 1% cpu 2:49.42 total and this for extralibs: ./darcs-all pull -a 0.82s user 0.33s system 1% cpu 1:40.47 total Thanks Ian

On Thursday 06 September 2007 12:14, Simon Marlow wrote:

Ian Lynagh wrote:

...

By the way, if you're currently pulling over SSH then you'll probably find it much faster to pull over HTTP. [...]

And it's dead easy to have a local repo tree that is kept up to date by a cron job. A 'darcs-all pull -a' from the local tree then takes only a few seconds.

Ummm... Now I'm completely confused, perhaps I'm using darcs in a suboptimal way. Anyway, let's not forget the small minority of people at home behind something slower than T1, which like to switch off their boxes from time to time... (including me ;-) My usage scenario is still the same as it has been for lots of years: * I work on GHC from home. * My computer is not running 24/7, therefore everyhing cron-lilke is a no-go. * I'm behind something reasonably fast for home purposes (currently 2Mbit/s DSL). * I need all of GHC + extralibs. * Non-partial darcs repositories seem to be necessary for development to keep oneself in a sane state of mind. * I need to be able to push my changes to darcs.haskell.org (possible via SSH only, I guess). To work like this, I did the obvious (at least obvious for me): "darcs get" full repositories via SSH (a Herculean effort, lasting more than half a day), synching everything via "darcs-all pull" (i.e. effectively via SSH, taking about 20min), pushing via plain "darcs push" (i.e. SSH again, not fast, but bearable). Is there anything wrong with that? Hints? Cheers, S.

On Saturday 08 September 2007 13:50, Ian Lynagh wrote:

I recommend getting over HTTP (starting with http://darcs.haskell.org/ghc-HEAD-2007-08-29-ghc-corelibs-testsuite.tar.bz2 is even quicker) and then using one of: * http://www.haskell.org/pipermail/cvs-ghc/2007-April/034792.html * http://www.haskell.org/pipermail/cvs-ghc/2007-April/034813.html * % cat push-monk #!/bin/sh ./push-all darcs.haskell.org:/home/darcs ${1+"$@"}

I use a mixture of the first and third, depending on whether I can remember which repos need to be pushed or not.

Push is still over SSH, but pull is over HTTP.

Hmmm, I think all this stuff should be at some prominent place in the GHC developer Wiki. This seems all very counterintuitive to me and basically a collection of random workarounds. :-( While we are at workarounds: darcs annotate takes ages for even a small file like mk/config.mk.in (around 10min, needing +RTS -K20M to complete), resulting in a hardly human-readable format. Is there a better way to find out which lines where added by whom/when/why? This is no fun, either, and some kind of annotation facility is a crucial part of a VC system... Cheers, S.

On Sat, Sep 01, 2007 at 05:24:51PM +0200, Sven Panne wrote:

The main problem is that I've been hearing the sentence "Cabal is unstable at the moment, but with the next GHC release everything will be fixed and rock-solid, never changing again, ..." at least for a year now.

I don't recall that myself, but I certainly hope no-one's given you the impression that Cabal will be fixed and rock-solid, never changing again, after the upcoming GHC release.

-----Original Message----- From: libraries-bounces@haskell.org [mailto:libraries-bounces@haskell.org] On Behalf Of Peter Gavin Sent: Monday, September 03, 2007 9:16 PM Cc: libraries@haskell.org; Claus Reinke Subject: Re: patch applied (packages/regex-base): Make setup script compileagain after recent Cabal changes On 9/1/07, Sven Panne wrote:

The main problem is that I've been hearing the sentence "Cabal is unstable at the moment, but with the next GHC release everything will be fixed and rock-solid, never changing again, ..." at least for a year now. In my experience, Cabal is *the* #1 reason for breaking build for aeons, and this is really getting frustrating.

It doesn't sound like you'd disagree with me, I just want to make a couple comments about this... Autoconf had similar problems during its infancy. Packages were constantly having problems. It takes time to work out the kinks, and bugs should be expected for the time being. And if you consider that Cabal is trying to be autoconf, automake, and libtool, and make, in a single tool, there will be that many more kinks to work out :) I completely agree. The level of instability is actually quite good given the location of Cabal in the software life cycle. Obviously, more stable is better, but IMO the developers have delivered a level of stability that, given the context, is above reasonable expectations Seth Kurtzberg Software Engineer Specializing in Security, Reliability, and the Hardware/Software Interface Pete _______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries

On Tue, 2007-09-04 at 18:25 +0200, Sven Panne wrote:

On Tuesday 04 September 2007 03:16, Peter Gavin wrote:

...

[...] And if you consider that Cabal is trying to be autoconf, automake, and libtool, and make, in a single tool, there will be that many more kinks to work out :)

Huh? Cabal tries to be autoconf? Unless I've missed something, there's no way Cabal can help me to figure out the paths to OpenGL headers, the wildly varying linker options for OpenGL apps,

Well, if the library you're binding to happens to be one of the 350+ that use pkg-config then cabal can indeed find the paths to the headers and the varying linker options (by asking pkg-config). pkgconfig-depends: gtk+-2.0 >= 2.8, cairo This is a new feature in Cabal-1.2. Of course it doesn't help for libs like GL that don't provide pkg-config files.

if the installed OpenAL's alcCloseDevice returns void or an int,

That's trickier.

the matching Haskell type for an ALuint, etc.

c2hs can do that. Sure, it's not an autoconf replacement and though you can use Setup.lhs to do some of those things, the api is not sufficiently stable to recommend using that yet. Duncan

...

it is almost as if everything feels free these days to evolve in non- backwards-compatible ways, following the motto "what do i care for my apis of yesterday?".

i picked cabal as an easily demonstrated *example*, not as the only culprit. for instance, if base gets split up into smaller parts, and the *remainder* gets called base, even though it no longer provides the same functionality as the old base package, that is not cabal's fault. i'm not interested in blame-games, and my concern is with a general trend rather than any particular tool or library. as far as _stability only_ is concerned, there is nothing wrong with *adding* features or with *cleaning up* existing code; the problems come when *externally used* features are *removed* or *new dependencies* are introduced. lets have a look at that base package example, for a change: if the remainder had been called base-core and base had become a proxy package that provides no functionality other than recording the new dependencies needed to get hold of the old functionality, by depending on the spin-off packages, then there wouldn't have been any externally visible changes to base, old code depending on base would still work as long as base was installed, and new code could bypass base, using base-core and other spin-off packages directly. same for cabal: if it learns to do new things, do old things in nicer ways, or interpret more information, *that should not affect any old code*. if old code breaks, that means that old information is no longer accepted, old dependencies are no longer sufficient, or old functionality is no longer provided.

This is the development version, so you can't really expect it to be stable at all.

i'm not concerned with the development versions of cabal (and again:none of this is specific to cabal), which i never expect to see. i'm concerned with the versions of cabal that are in active use, in ghc head, hackage or elsewhere. if a version of some centrally important software (be it library or tool) is used, then that version's functionality/api should be supported for as long as practical. compatibility packages and automatic translators are not ideal, but better than nothing; simply abandoning old clients is the worst situation. ideally, new versions of such software should still support old clients, while warning about uses of deprecated functionality (somehow, deprecation warnings have come to mean "it will be gone in the next version", when it used to mean "it will not be here indefinitely").

So, could you please exclude Cabal (and stay with me when I say we hope to have things stable by the next ghc release), re-evaluate the situation excluding anything related to Cabal issues (ie, setup.lhs errors), and see if your issue still exists?

yes, it does. it just turns up in cabal so often because (a) there's a lot of development there (which is great!-), (b) cabal is aiming to become central to all haskell development (which is rather a huge burden of responsibility). the more cabal succeeds, the more maintenance issues will be related to cabal, and the more cabal is used, the higher the standards for backwards-compatibility it should aspire to achieve. </soapbox> :-) claus

On Saturday 01 September 2007 20:13, Ross Paterson wrote:

[...] However a package that uses a boilerplate Setup.hs (as 90% of the packages on Hackage do) needs only backwards compatibility of the .cabal format, and that has been carefully preserved.

Alas, that's true: Today I had to unbreak the "time" package exactly because the .cabal format has been changed.

[...] There is a common issue, but it's the "birthpangs of a new package system". Pouring concrete over Cabal in its current state won't help.

Fair enough, but giving a painful birth for months after months is really annoying. Our very big mistake has been to use Cabal in its current immature state in the GHC (+ libs) build system. Being progressive and open to new technologies is a good thing, but *not* in the area of versioning systems, build infrastructure and the like, at least this should be avoided when the SW in question is built by a large amount of people. Experiments should be done in isolation... Cheers, S.

On Sat, Sep 01, 2007 at 08:34:43PM +0200, Sven Panne wrote:

On Saturday 01 September 2007 20:13, Ross Paterson wrote:

...

[...] However a package that uses a boilerplate Setup.hs (as 90% of the packages on Hackage do) needs only backwards compatibility of the .cabal format, and that has been carefully preserved.

Alas, that's [not] true: Today I had to unbreak the "time" package exactly because the .cabal format has been changed.

Only because the development version of time.cabal was using the new extensions while they were being developed. The backwards compatibility is between released versions. The .cabal file in the time package in Hackage should still work with the new Cabal.

On Sat, Sep 01, 2007 at 11:48:34PM +0100, Claus Reinke wrote:

i just did a little experiment, with ghc-6.6.1. consider:

-- Y.hs module Main where import Data.Time main = print =<< getCurrentTime

now, consider this cabal package, the only purpose of which will be to make Y.hs compileable:

-- P.cabal Exposed-modules: Data Build-Depends: base, time

-- Data.hs (yes..) module Data(module Time) where import Data.Time as Time

I'm a bit confused; is this Data module necessary? If so, is its name important? Are you proposing a new extension?

configure, build, install, and 'ghc -package P Y.hs' seems to work.

$ ghc -package P Y.hs

Why is -package P better than -package time? Incidentally, one thing that might help is to make an empty "time" package for people with old GHCs to install (or even to make the time package appear empty to such people, with configurations. Except they'd have to upgrade Cabal to support configurations first). Thanks Ian

| - Ross pointed out that packages can't simply re-export modules, | so the straightforward solution of a package without sources, | just with a .cabal file, seems barred for the moment (though i | don't understand why this restriction is there, so perhaps i am | asking for an extension?) Yes this is undoubtedly a bug, in Cabal or GHC or (probably) both. Of *course* a package should be able to re-export a module from another package, just as a module can re-export a function imported from another module. I don't think there is any difficulty in principle; it just needs to be implemented. Simon

...

...

...

-- Data.hs (yes..) module Data(module Time) where import Data.Time as Time

- taking time as a small example, i was looking for a way around this limitation, to reexport time's modules via a different package:

- it seems that cabal needs some sources for exported modules - module Data.Time where import Data.Time, then exposing Data.Time, does not work, because of cycle - module Data(module Time) import Data.Time as Time, then exposing Data, does work, as demonstrated

But the above is making a module called Data which exports everything that Data.Time exports. The Data.Time user isn't importing Data, so I don't see how that can help. As far as I can see, if this does work then there is at least one misfeature involved.

yes, i continue to be rather surprised by all of this, that simple module re-export in packages is not implemented, that import has a "letrec semantics" (the only reason for having the current module in scope for import seems that, before packages, there was no nesting; but with packages, there is the question of how to refer to a module of the same name in the dependencies, isn't there? perhaps there should be an optional "from <package>" qualifier for imports?), and that the variant i found works (if import Data.Time refers to Data's Time module in the client, why isn't the same import cyclic in Data.hs? just splitting the name breaks the cycle?).. but then, the draft hierarchical module spec didn't say much beyond "the names may now have dots". perhaps module Main where import Data.Time main = print =<< getCurrentTime and module Main where import Data(module Time) main = print =<< Time.getCurrentTime are really meant to be equivalent? claus

The second example isn't syntactically correct. What are you getting at here?

is that really as completely unobvious to you as your question implies? -- A module Main where import Data.Time main = print =<< getCurrentTime -- B module Main where import Data main = print =<< Time.getCurrentTime given that package P only exposes a module Data (which, in turn, exports a module Time), it appears that A behaves as if it was B. is that intended? or is something else going on? which is what Ian was asking about, wasn't it?

FWIW, there was a proposal to add a package qualifier to import declarations, but it wasn't unanimously viewed as the right thing at the time (check the libraries@haskell.org archives). Same goes for the various "grafting"/"mounting" proposals - we just haven't seen a proposal that has the right power to weight ratio. Meanwhile, the current package system seems to be scaling quite nicely, thank you.

no offense intended;-)

We do intend, I think, to allow packages to re-export modules, it just needs to be implemented. (I see it as a missing feature, rather than a bug or a "design flaw", though).

as long as we agree on the features we want, we don't need to agree on whether their lack is a flaw or a miss. claus

Claus Reinke wrote:

...

-- A module Main where import Data.Time main = print =<< getCurrentTime

-- B module Main where import Data main = print =<< Time.getCurrentTime

that last line should be

...

main = print =<< getCurrentTime

i probably thought for a moment that i was exporting a module, rather than its contents. was that the source of confusion?

actually I was confused because you had import Data(module Time) and that isn't Haskell. But never mind, I think I now get the gist of what you're asking about... see below.

so why did the experiment reported in

http://www.haskell.org/pipermail/libraries/2007-September/008062.html

work, when Y.hs imports Data.Time, but the package P only exposes Data? is Y importing time's Data.Time directly, even though P doesn't expose it?

Interesting. To recap, you had -- Y.hs module Main where import Data.Time main = print =<< getCurrentTime and you compiled it like this; $ ghc -package P Y.hs without failure. Package P simply depends on package time, that's all. You have to understand that the above command does two things: it compiles Y.hs to Y.o, and then it links Y.o with libraries to form a binary. In the first stage, all exposed packages are available (because you didn't say -hide-all-packages), so Data.Time from package time is in scope and you can successfully import it. At link time, all you have to do is make sure the required packages are linked in, and you did that by explicitly linking something (P) that depends on package time, so package time was linked in too. I don't think there's anything really "wrong" here, that's just the way it works when you don't use --make. Cheers, Simon

On Sat, 2007-09-01 at 19:13 +0100, Ross Paterson wrote:

Cabal really is a special case. The API is still much too immature to stabilize, which is why it's handled differently from the other libraries. It was never claimed to be stable. However a package that uses a boilerplate Setup.hs (as 90% of the packages on Hackage do) needs only backwards compatibility of the .cabal format, and that has been carefully preserved. With the recent improvements to Cabal, fewer packages will need the tailored setup scripts that are so fragile. To impose extra constraints on the Cabal API at this point would make it harder to make the wholesale cleanups and enhancements needed to benefit all the other packages, such as Thomas's recent implementation of configurations.

There is a common issue, but it's the "birthpangs of a new package system". Pouring concrete over Cabal in its current state won't help.

I fully agree. We've done a lot of work on Cabal recently fixing a huge long list of bugs, complaints and feature requests. None of this would have been possible without doing lots of refactoring at the same time. Anyone is welcome to look at the internal code before and after too see what I mean. We still need to make significant further internal API changes to keep the code sane, to get features to be implemented in one place rather than spread throughout the code base. So the internal APIs are not stable and we can't promise to make them stable without halting all significant future development. On the other hand we have been extremely careful with not breaking the Cabal file format (and most of the command line interface). It's backwards compatible and in many ways future compatible too (ie adding new fields does not break old versions of Cabal, though it does generate warnings). GHC adopting Cabal for it's library build system has driven many improvements in Cabal but yes has caused quite a bit of breakage in GHC HEAD and the associated libraries. This isn't too bad since we have Ian on hand to keep things in sync. For other libraries using trivial Setup.hs files it's fine. I really don't know what to do to make the base split up easier. The main problem is that there really is nothing that identifies a piece of code as the package or module it lives in changes. The modules in packages can overlap so the module name isn't sufficient, though it's probably good enough in most cases. I've also been guilty of changing the internal api of ByteString, though I do note that it is only the internal api which we never promised to keep stable. The only changes in the public part of the api were as a result of requests and discussion on this mailing list (see unsafe CString bits and renaming .Base to .Internal and .Unsafe to reduce confusion). Duncan