On Mon, 2005-08-15 at 08:52 -0700, Isaac Jones wrote:

Frederik Eaton writes:

...

4) Cabal's inability to let users specify package.conf-s other than the user or global package.conf

I think the gentoo guys were going to implement this for us; in any case, it's a subset of 1). This should be pretty easy; the only question is whether we should have a --in-place or a --package-conf=... Where in-place would choose a "standard" one (./packages.conf or something). The reason I'd prefer the former is that not all systems _have_ an idea of a package.conf file, so --package-conf= would make no sense for most compilers. I try to avoid that. The reason for the later would be if there's some use-case that --in-place does not work for.

So ./setup register --in-place suggests that the library will be registered in it's current location, ie the build tree. So this would allow the library to be used in building other libraries or programs in the same "distributable". Or it would allow the library to be used without installing to its final location which might be useful in some circumstances (which I must admit are not obvious to me). So this is a useful feature and we're happy to help out in implementing it. However what we the gentoo folk want is subutly different. We want to register into a package database other than the global or user one. However we want to register it for it's final installed location, not the current location in the build tree. Actually, it's not that we really want this but it would be sufficient for our needs. For the sake of innocent bystanders let me recap on a discussion the gentoo packaging people have been having with Isaac over the past few weeks: The way packages are built for a distribution (like gentoo, debian, fedora etc) is slightly different from the way you normally build and install a package. We configure and build and install the package in a "sandbox", that is in a temporary directory and with permissions set up such that there is no write access to the "live" filesystem. In particular the instllation phase is also done to a temporary directory. This gives us an "image" of the package. Now in a binary packaging system that image gets compressed into a binary package file (eg a .deb or a .rpm). With gentoo we skip that step and just merge the image of the package into the filesystem. And note that this marging phase is totally under control of the package manager and the package's build system get's no control. This is partly a security measure; the package's build system only gets to run in the sandbox and so never has write access to the live filesystem. So this is why people who do packaging like packages that can be installed just by copying files and without having to modify anything else on the target system - like registering. For example, many systems which need to reigister clients (eg cron, pam, apache modules etc) have been converted to use directories of config files rather that having to modify a single global file since the former allows one file to be owned per package. Packages can have post install actions. So for cabal packages, this is the obvious place to put the registering phase. However we can't just run ./setup register. There are a couple reasons for this, one is that the build directory doesn't actually exist anymore since it gets cleaned before merging (this is not unreasonable since we can also build binary packages and move them to another machine in which case the build directory is not available either). Also it would be against the packaging policy to run an untrusted program (the cabal setup program) outside of the sandbox. So this is why cabal provides the ./setup register --gen-script feature. It produces a shell script that performs the registration. It means all you need is the script rather than the setup program and any other bits of build tree that the setup program uses. However this is still a violation of the packaging policy since we're running a script produced by an untrusted program outside of the sandbox. From a security point of view it's preferable to have the data and run trusted actions on that data rather than be given the actions that know what to do with the data. The downside from the cabal point of view is that just giving out the data is against the spirit of the abstraction that cabal is trying to provide. However the package manager needs to do more things than the cabal abstraction currently provides hence the current pressure to "just give me the data!". The --gen-script feature allows us to register but we also need to be able to unregister and re-register. And we need to be able to do more complicated things like unregistering an old version *after* registering a new version without accidentally unregistering that new version. Now these are not features that we expect cabal to provide (since they are peculuiar to package management systems and probably vary from one to another - especially the idea of installing a new version of a package before uninstalling an old version of a package). However if we are to be able to implement these features (and we must otherwise ghc's package database will be out of sync with the installed packages) then we need some way of getting at the collection of packages that each cabal distributable provides. Then we can take responsability for registering, unregstering and re-registering. [In case anyone's even remotely interested in the problem of "uninstall after install", here it is: suppose the gentoo package "foo-1.0" provides a cabal package "Foo-1.0". Now we fix a bug or something and release a new revision "foo-1.0-r1" which still provides the cabal package "Foo-1.0". Then if we were to unregister the packages provided by foo-1.0 after registering the ones provided by foo-1.0-r1 then we would end up unregistering Foo-1.0 completely. Instead we need to take the set difference of the packages provided by the new and old versions and only unregister those. If we could register/unregister packages just by dropping files in a package database directory then this would all "Just Work"tm but that's sadly not the case. So we need to do more complicated things.] So back to the original topic... we wanted to implement a feature that would enable us to get at the packages that need to be registered. So we were hoping to subvert the ./setup register --in-place idea. However as I said at the beginning it doesn't quite do what we want. We're happy to help with implementing this useful feature but we're also still looking for a nice comprimise that will allow use to implement the features we need for proper package management while at the same time respecting the need to maintain a sane interface to cabal. Duncan

On Mon, Aug 15, 2005 at 08:52:18AM -0700, Isaac Jones wrote:

Frederik Eaton writes:

...

1) Cabal files can have different names, yet only one can exist in a directory. I've been told that the reason for the former is that Isaac wants to support building multiple packages in a directory, but currently the feature isn't implemented, so only one cabal file is allowed.

Certainly, it would be a nice feature, but more importantly it seems that it would require some API changes or additions which should be made sooner rather than later. Is there a document describing how it would work? Is there a roadmap, a schedule for implementation?

I'm not so sure that the change is very big at all... it should just be a matter of adding a feature to allow libraries to register themselves "in-place" or something... then compute the dependencies between the packages, build them in order, install the libraries you'll need in-place, then compile the programs.

The problems come up where a package might need to actually be in the install location in order to work. Hopefully that's not too common.

This sounds rather vague. For instance, how does it interact with configure/make based packages? What do you think about putting a TODO list linking to designish documents on the wiki, so that people can flesh these plans out and comment on them there?

...

3) The way that a 'cabal' package with a 'configure' script should be built is not easy to determine. There are some very useful programs (http://toastball.net/toast/) which build and manage packages of all kinds automatically, by running 'configure' with a local prefix, etc. How should these tools be extended to work with 'cabal' packages? With some 'cabal' packages, the user is supposed to run 'configure' first, and then 'runhaskell Setup.hs'. With other 'cabal' packages, the user is supposed to just run 'runhaskell Setup.hs', and this will run 'configure' automatically. Is there a right way? Is there a way to distinguish the two types of packages automatically?

What packages require you to run configure by hand? That is indeed incorrect in the sense that it doesn't conform to the cabal interface. If the packager is doing that in order to alter the .cabal file, then that's a problem... the correct way to do this is covered in the manual, and configure has built-in support in cabal.

I don't know of any packages off-hand. I think my friend who wrote "toast" may have found one, but he may also have just liked 'configure' better. Anyway, it's good to know that there is a standard. It might be good for the manual to mention explicitly that configure/make are always run by Setup.hs and not vice-versa.

...

4) Cabal's inability to let users specify package.conf-s other than the user or global package.conf

I think the gentoo guys were going to implement this for us; in any case, it's a subset of 1). This should be pretty easy; the only question is whether we should have a --in-place or a --package-conf=... Where in-place would choose a "standard" one (./packages.conf or something). The reason I'd prefer the former is that not all systems _have_ an idea of a package.conf file, so --package-conf= would make no sense for most compilers. I try to avoid that. The reason for the later would be if there's some use-case that --in-place does not work for.

Is it really a subset of (1)? I don't think it is. I think a --package-conf= would be very nice. If it is not supported by the compiler, then an error should be generated.

...

, and ghc's refusal (and that of other compilers) to let users specify paths for things via environment variables, (snip) To lag behind 'gcc' and 'autoconf' is not acceptable.

GCC lets you specify things via environment variables? I've never seen that... if ghc were to adopt this kind of behavior, I hope it wouldn't be the default; as in 'make', it should require an extra flag like -e.

Mostly CPATH and LIBRARY_PATH. Perhaps, variants for other languages.

...

5) It seems that there should be a way to build multiple packages which depend on each other, in such a way as that the first package doesn't have to be installed for the second package to be built. In other words, perhaps something like a dummy installation as in #4 is called for. I have seen mention that support for building multiple packages is planned, but how? When?

This is the same as 1.

Well, not if they aren't in the same directory. Again, we need to have more explicit details about how things are planned to work.

...

What stage of implementation are we in?

We're in the stage of "Isaac has been too busy writing papers and book reviews and needs to get working." I'm always happy to get patches, though.

OK, correct me if I'm wrong but I think we need to tell people what kind of patches to write, and ask them to write them, to get things moving.

...

6) I think it should be easy to use 'cabal' for development; however, when I am building a package with multiple executables, every 'build' seems to re-link each executable, regardless of whether it needs re-linking or not, which is quite slow.

I'm not sure why ghc --make re-links executables every time.

For libraries, I think we could use support from ghc to tell whether we need to re-link the library; ghc goes through and skips stuff that doesn't need to get built, and then we link everything in the library; if ghc could somehow let us know that nothing needed to get built, that would be very helpful; otherwise, someone has to write the logic to go through and check it all just like ghc does. This code should be out there somewhere.

Isn't there a way to get ghc to emit Makefile fragments which solves this problem? Not that solving it in ghc wouldn't be good as well.

...

I'm not sure how much of this Isaac has planned out on his own and is busy implementing, vs. how much is waiting for someone to write a design document for.

You're always welcome to submit patches and suggestions on how to do stuff, and I and other folks here will be more than happy to engage you. I've been a little too distracted with other stuff, and keep hoping that GHC 6.4.1 will appear so I can release cabal 6.2 and get hacking on other stuff. I should just make a 'stable' branch or something.

Finally, thanks for all of your great suggestions. We need feedback from as many different users as possible. We've spent a long time guessing about what folks might need, and now that cabal is in the wild, we're getting good feedback about what they actually need :)

You're welcome. Frederik

On Fri, Aug 19, 2005 at 10:11:52AM -0700, Isaac Jones wrote:

Frederik Eaton writes:

...

On Mon, Aug 15, 2005 at 08:52:18AM -0700, Isaac Jones wrote:

...

Frederik Eaton writes:

...

1) Cabal files can have different names, yet only one can exist in a directory. I've been told that the reason for the former is that Isaac wants to support building multiple packages in a directory, but currently the feature isn't implemented, so only one cabal file is allowed.

Certainly, it would be a nice feature, but more importantly it seems that it would require some API changes or additions which should be made sooner rather than later. Is there a document describing how it would work? Is there a roadmap, a schedule for implementation?

I'm not so sure that the change is very big at all... it should just be a matter of adding a feature to allow libraries to register themselves "in-place" or something... then compute the dependencies between the packages, build them in order, install the libraries you'll need in-place, then compile the programs.

The problems come up where a package might need to actually be in the install location in order to work. Hopefully that's not too common.

This sounds rather vague. For instance, how does it interact with configure/make based packages?

If they conform to the cabal standard, providing configure, build, and install steps, then they are built exactly the same. Can you be more specific about your question?

What I meant is, if a 'make'-based build depends on a cabal package being installed in a temporary location, how do we communicate to the 'make'-based build what that temporary location is? There needs to be some standard mechanism and it is important to facilitate some sort of discussion about it so that the right standard mechanism is agreed upon. It's not trivial, there is a package database, a base for library files and executables, and these have to be communicated to different kinds of compilers... I expect that even how things will work with standard cabal packages will not be trivial.

...

What do you think about putting a TODO list linking to designish documents on the wiki, so that people can flesh these plans out and comment on them there?

Sure. Feel free to add a wiki page for this. There's a TODO list in the darcs repository, but it may not be understandable by anyone but me; it still might be a good place to start.

Maybe we'll need your help on the TODO list part. I'm willing to contribute ideas once the structure

I caution you, though, on spending too much time making plans and designs and not much time writing code; the problem is usually that we have plenty of ideas about what needs to get done and not enough coders. The code base is pretty manageable, so if you want to help out, just pick something you think needs to get done, or ask me what is most important and start working on it :) I usually ask for help on this list when something comes up.

I caution you on the opposite. Cabal is very format and protocol-heavy. I imagine that much of the work has gone into documentation and mindshare, in which case it becomes very important to keep people informed and working together. Suggesting that individuals need to contribute patches unilaterally before anything will happen seems counterproductive to me, at least when it comes to some of the larger parts. On a meta-tangent, if we could be discussing things on a wiki, rather than a mailing list, then I think discussions could go a lot faster. For instance: person A says something, person B asks for clarification, person A amends what he says and deletes person B's request. On a list you have 3 messages, on a wiki you have 1. So asking someone to "go read the thread" doesn't sound so ridiculous anymore. If it were a wiki with email notification capabilities, then it would be even better since it could evolve in real time. This is the sort of thing I'm advocating. Partly because I think it would be good for Cabal, partly because I want to do the experiment. Only if other people think it's a good idea though.

Something that you could spend a few hours on is getting the test suite to run conveniently on other people's computers, and figuring out how to get Hugs to pick up changes directly from the current directory during testing.

...

...

...

3) The way that a 'cabal' package with a 'configure' script should be built is not easy to determine. There are some very useful programs (http://toastball.net/toast/) which build and manage packages of all kinds automatically, by running 'configure' with a local prefix, etc. How should these tools be extended to work with 'cabal' packages? With some 'cabal' packages, the user is supposed to run 'configure' first, and then 'runhaskell Setup.hs'. With other 'cabal' packages, the user is supposed to just run 'runhaskell Setup.hs', and this will run 'configure' automatically. Is there a right way? Is there a way to distinguish the two types of packages automatically?

What packages require you to run configure by hand? (snip) I don't know of any packages off-hand.

Whew

...

I think my friend who wrote "toast" may have found one, but he may also have just liked 'configure' better. Anyway, it's good to know that there is a standard. It might be good for the manual to mention explicitly that configure/make are always run by Setup.hs and not vice-versa.

Here's a good example; the manual is in the darcs repository, doc/cabal.xml, I think. Just go ahead and add that and send me a darcs patch :)

I've attached a patch.

...

...

...

4) Cabal's inability to let users specify package.conf-s other than the user or global package.conf

I think the gentoo guys were going to implement this for us; in any case, it's a subset of 1). This should be pretty easy; the only question is whether we should have a --in-place or a --package-conf=... Where in-place would choose a "standard" one (./packages.conf or something). The reason I'd prefer the former is that not all systems _have_ an idea of a package.conf file, so --package-conf= would make no sense for most compilers. I try to avoid that. The reason for the later would be if there's some use-case that --in-place does not work for.

Is it really a subset of (1)? I don't think it is.

I think a --package-conf= would be very nice. If it is not supported by the compiler, then an error should be generated.

In a way, the problem isn't "lack of support" but a different model of finding packages... It's not like a compiler extension that one system supports and one doesn't; this flag breaks abstraction between compilers in a way that --in-place does not. I haven't heard any use cases where --in-place won't work.

Well, we're talking about two different features here. I want to be able to specify an arbitrary location. You want to be able to specify the current working directory. Not the same thing, is it? I think it should be possible to specify an arbitrary location, at least for ghc.

(snip)

...

...

...

5) It seems that there should be a way to build multiple packages which depend on each other, in such a way as that the first package doesn't have to be installed for the second package to be built. In other words, perhaps something like a dummy installation as in #4 is called for. I have seen mention that support for building multiple packages is planned, but how? When?

This is the same as 1.

Well, not if they aren't in the same directory. Again, we need to have more explicit details about how things are planned to work.

Hackage is the tool here; it's still almost ready. It has 19 module dependencies and is a pain to bootstrap. We've written about it here & there, but if you can't dig up concrete information, come onto #haskell and ask Lemmih what's up.

I was thinking something like WASH, where the packages are in different directories, but they aren't registered in some sort of central database. BTW, I find that Haskell modules are simple enough to maintain and merge that sticking these 19 hackage modules into 1 might not be too bad of an idea.

...

...

...

6) I think it should be easy to use 'cabal' for development; however, when I am building a package with multiple executables, every 'build' seems to re-link each executable, regardless of whether it needs re-linking or not, which is quite slow.

I'm not sure why ghc --make re-links executables every time.

For libraries, I think we could use support from ghc to tell whether we need to re-link the library; ghc goes through and skips stuff that doesn't need to get built, and then we link everything in the library; if ghc could somehow let us know that nothing needed to get built, that would be very helpful; otherwise, someone has to write the logic to go through and check it all just like ghc does. This code should be out there somewhere.

Isn't there a way to get ghc to emit Makefile fragments which solves this problem? Not that solving it in ghc wouldn't be good as well.

Relying on Make isn't any good for us for the simple build infrastructure; it needs to be more portable than that.

Really? Surely every platform has some basic 'make' installed. And just parsing out the rules can't be that hard. Frederik -- http://ofb.net/~frederik/

On Fri, 2005-08-19 at 22:49 -0700, Frederik Eaton wrote:

On Fri, Aug 19, 2005 at 10:11:52AM -0700, Isaac Jones wrote:

...

Relying on Make isn't any good for us for the simple build infrastructure; it needs to be more portable than that.

Really? Surely every platform has some basic 'make' installed.

Sadly Windows does not have make installed by default. And it's not really reasonable to ask windows users to install MSYS or cygwin first. Duncan

(snip)

...

...

I caution you, though, on spending too much time making plans and designs and not much time writing code; the problem is usually that we have plenty of ideas about what needs to get done and not enough coders. The code base is pretty manageable, so if you want to help out, just pick something you think needs to get done, or ask me what is most important and start working on it :) I usually ask for help on this list when something comes up.

I caution you on the opposite. Cabal is very format and protocol-heavy. I imagine that much of the work has gone into documentation and mindshare, in which case it becomes very important to keep people informed and working together. Suggesting that individuals need to contribute patches unilaterally before anything will happen seems counterproductive to me, at least when it comes to some of the larger parts.

I'd be happy to try the wiki idea. I haven't done something like this recently because I haven't identified such a need; if I had a wiki page, I think I'd be the only one who reads it and the only one who modifies it. That's the case with the TODO list, which is under version control, so theoretically anyone could modify it, but no one does.

Well, a wiki page is a bit more accessible. For one example, it could serve to alert potential users of Cabal, who have not downloaded the package yet, to open bugs. Besides, is there a way for an arbitrary person to change the TODO list without checking with you first? You have to remember that when people come up with publicly useful ideas, they want to publish them, get them out of the way and move on, without adding several layers of communication latency overhead. Even if it was just a matter of sending you a patch which you would apply within the next few hours, it's not the same as a wiki. They can't fire-and-forget. Human context switching is expensive.

I usually avoid adding "process" until a need arises. If we were ever in a situation where we need more analysis than code, then I would spend more time writing documents and wikis (which we used to do much more; you'll find old pages of discussion on the wiki, but not many people besides me contributed; it was mostly copying and pasting from mailing lists). We spent a lot of time on discussion and documentation early on in the project (probably a year), and the original proposal was the product of that work. I think that at some point, we will want to commit to a "cabal version two" where we will go into an analysis phase again. I just don't think we're at that point yet. I still think we just need to get code out there for people to try and see what works, what doesn't, and what the needs are that we hadn't anticipated. Cabal is still a very young project, believe it or not, and we have spent a great deal of its lifetime writing documentation and having discussions. I hope you won't begrudge me a few more months of writing code ;)

If there have been a lot of discussions and decisions, I don't think that mailing list archives, or wherever the analysis is located, are a good repository for design documents. I believe (and I'm not saying you disagree at this point) that things which are planned and which we want people to potentially help out with should go on the wiki, along with their rationale.

...

On a meta-tangent, if we could be discussing things on a wiki, rather than a mailing list, then I think discussions could go a lot faster. (snip for instance)

I am willing to try doing more on the wiki, but in the spirit of wiki, I think you should just go ahead and start it :)

Later this week, if everything goes well.

...

I've attached a patch.

Thanks.

(snip)

...

...

In a way, the problem isn't "lack of support" but a different model of finding packages... It's not like a compiler extension that one system supports and one doesn't; this flag breaks abstraction between compilers in a way that --in-place does not. I haven't heard any use cases where --in-place won't work.

Well, we're talking about two different features here.

I'm talking about use cases, not features :)

We're talking about two different use cases.

...

I want to be able to specify an arbitrary location. You want to be able to specify the current working directory. Not the same thing, is it? I think it should be possible to specify an arbitrary location, at least for ghc.

Since this breaks abstraction, I would prefer to avoid this until I see a convincing use case.

I thought I already gave one: http://toastball.net/toast/ Here's another: http://www.wigwam-framework.org/doc/overview.html But let's think about what you're saying. You don't want to break some abstractions in your implementation, so you're causing the abstraction that you are supposed to be providing your users with to break. You are breaking it by removing the ability to virtualize the most basic aspect of the package management process, the package database. A bit myopic, methinks. We need to be able to deal with packages getting installed anywhere and everywhere and every which way. Everything about the installation of a package needs to be virtualizable. The package database should not be tied to the user or the system or the compiler version. If some compilers are not capable of providing Cabal with the interface it needs to implement a suitably virtualizable installation environment, then Cabal should simply refuse to continue in certain situations. Those compilers can catch up when they do. Seriously, why is it that functional programmers profess to care so much about modularity on a small scale, yet are so quick to defenestrate it when confronted by larger systems? :)

(snip)

...

...

...

...

...

6) I think it should be easy to use 'cabal' for development; however, when I am building a package with multiple executables, every 'build' seems to re-link each executable, regardless of whether it needs re-linking or not, which is quite slow.

I'm not sure why ghc --make re-links executables every time.

For libraries, I think we could use support from ghc to tell whether we need to re-link the library; ghc goes through and skips stuff that doesn't need to get built, and then we link everything in the library; if ghc could somehow let us know that nothing needed to get built, that would be very helpful; otherwise, someone has to write the logic to go through and check it all just like ghc does. This code should be out there somewhere.

Isn't there a way to get ghc to emit Makefile fragments which solves this problem? Not that solving it in ghc wouldn't be good as well.

Relying on Make isn't any good for us for the simple build infrastructure; it needs to be more portable than that.

Really? Surely every platform has some basic 'make' installed.

I don't think it's the case that every platform has make installed. Windows, for instance. In fact, one convincing argument for cabal is that it doesn't require make.

...

And just parsing out the rules can't be that hard.

Interpreting a makefile is probably pretty hard. I don't want to do it, personally. But the rules that ghc produces are probably a subset of the entire make language... still, I'd think that it would be far more productive to write a dependency analizer that doesn't rely on ghc than to write a makefile interpreter that does.

Makefiles are a pretty standard way for compilers to output dependency information. Dependency analysis would be nice, but I imagine that there are all sorts of compiler options you'd have to know about to properly locate things. Frederik