Hello, during AngloHaskell, me, Duncan, Shae and David had the occasion to think about cabal-install (formerly known as cabal-get). The current situation: There is a server-program called cabal-server-install which checks that a given package is valid and updates the index accordingly. Its code can be found here <http://hackage.haskell.org/darcs/cabal-server- install/>. The current index file is a list of {name, description, dependencies, url} informations for each package. The hierarchy of packages and the corresponding index file can be found here <http:// hackage.haskell.org/darcs/cabal-server-install/pub/unstable/>. The problems: The current hierarchy is full of outdate packages, this is easy solvable after we've made some design decisions about the format of the hierarchy and the index file. The current index file format is a made-up format consisting of a certain number of fields, this is a problem because: * Cabal is going to support more dependencies information, like optional dependencies, which will require a field per-se. * For example GUI package managers need to know more than the little information now available after an update command. * The index file contains URL information for each package, which we don't know at the moment of producing that file on the server. A solution to all these problems is having all the `.cabal' files in a tarball as the index file. This is good for a number of reasons, some of those are: * No code is needed to generate the index. One can simple tar the directory hierarchy skipping the tarballs. Now, it will only pack the `.cabal' files; in future, it easily packs all the other files we will eventually need, like `.sig' files for signatures and whatnot. This is a big advantage. * The tarball format is very simple, it is all ASCII, and an Haskell un-tar implementation is very few lines of code, which will be needed for Windows. * Using the `.cabal' files for all the information means that cabal- get is easily {backward,forward}-compatible with changes in the `.cabal' files format, because it just relies on the Cabal parser. (e.g. optional dependencies in the near(?) future.) * We have all the informations from the beginning, after the first `cabal-install update' command. We are quite happy with this solution, but some other decisions need to be made: Now cabal-install saves the server list file in configuration-dir/ serv.lst; the index file in conf-dir/pkg.lst; and downloads all the tarballs in conf-dir/packages/. I'd like to discuss what should the local hierarchy of files look like with this change in the index file format. One problem is that we need to support multiple repositories, and still map the packages to the right URLs. The proposed solution would look like this: We have a conf-dir/repositories/ directory which contains a subdirectories. Those subdirectories are the string-encoded URLs of each repository. They contain the local mirror of the remote hierarchy produced by simply unpacking the index file; it generates a perfect local mirror missing the tarballs. This looks like a sensible nice thing. `cabal-install' will then fetch the informations it needs by looking order-sensitively at the server-list-file, then recurse through the hierarchy finding all the .cabal files. This looks like a pratical solution which should work well until we get thousands of packages. If we ever reach that point and efficiency becomes an issue we could simply load the tarball in memory and play with it lazily, mapping in- memory-cabal-files to records structures with the informations we need. What do you think? Does it look like a good plan? More questions: I am personally not sure about the OutputGen type which is defined in Cabal/Network/Hackage/CabalInstall/Types.hs. It seem to be evil and should be removed. Anybody has a precise idea about it? I have some initial code that makes cabal-get work on the tarball index, but I am not sure about how to integrate with the OutputGen actions, so it is not working now. I need some explanation about it. I hope the contents of this email are understandable, please ask for whatever I have explained badly or I have assumed for some reason. Peace, Paolo. --
On 8/9/06, Paolo Martini <paolo@nemail.it> wrote:
Hello,
during AngloHaskell, me, Duncan, Shae and David had the occasion to think about cabal-install (formerly known as cabal-get).
The current situation:
There is a server-program called cabal-server-install which checks that a given package is valid and updates the index accordingly. Its code can be found here <http://hackage.haskell.org/darcs/cabal-server- install/>.
The current index file is a list of {name, description, dependencies, url} informations for each package. The hierarchy of packages and the corresponding index file can be found here <http:// hackage.haskell.org/darcs/cabal-server-install/pub/unstable/>.
The problems:
The current hierarchy is full of outdate packages, this is easy solvable after we've made some design decisions about the format of the hierarchy and the index file.
The current index file format is a made-up format consisting of a certain number of fields, this is a problem because: * Cabal is going to support more dependencies information, like optional dependencies, which will require a field per-se. * For example GUI package managers need to know more than the little information now available after an update command.
To elaborate a little: Most package managers would appreciate extra information such as license, authors, description, etc.
* The index file contains URL information for each package, which we don't know at the moment of producing that file on the server.
A solution to all these problems is having all the `.cabal' files in a tarball as the index file.
This is good for a number of reasons, some of those are: * No code is needed to generate the index. One can simple tar the directory hierarchy skipping the tarballs. Now, it will only pack the `.cabal' files; in future, it easily packs all the other files we will eventually need, like `.sig' files for signatures and whatnot. This is a big advantage. * The tarball format is very simple, it is all ASCII, and an Haskell un-tar implementation is very few lines of code, which will be needed for Windows. * Using the `.cabal' files for all the information means that cabal- get is easily {backward,forward}-compatible with changes in the `.cabal' files format, because it just relies on the Cabal parser. (e.g. optional dependencies in the near(?) future.) * We have all the informations from the beginning, after the first `cabal-install update' command.
We are quite happy with this solution, but some other decisions need to be made:
Now cabal-install saves the server list file in configuration-dir/ serv.lst; the index file in conf-dir/pkg.lst; and downloads all the tarballs in conf-dir/packages/.
I'd like to discuss what should the local hierarchy of files look like with this change in the index file format.
One problem is that we need to support multiple repositories, and still map the packages to the right URLs. The proposed solution would look like this:
We have a conf-dir/repositories/ directory which contains a subdirectories. Those subdirectories are the string-encoded URLs of each repository. They contain the local mirror of the remote hierarchy produced by simply unpacking the index file; it generates a perfect local mirror missing the tarballs. This looks like a sensible nice thing.
`cabal-install' will then fetch the informations it needs by looking order-sensitively at the server-list-file, then recurse through the hierarchy finding all the .cabal files. This looks like a pratical solution which should work well until we get thousands of packages. If we ever reach that point and efficiency becomes an issue we could simply load the tarball in memory and play with it lazily, mapping in- memory-cabal-files to records structures with the informations we need.
What do you think? Does it look like a good plan?
More questions:
I am personally not sure about the OutputGen type which is defined in Cabal/Network/Hackage/CabalInstall/Types.hs. It seem to be evil and should be removed. Anybody has a precise idea about it?
OutputGen /is/ evil and should be replaced by something like 'log_action :: Severity -> Message -> IO ()'.
I have some initial code that makes cabal-get work on the tarball index, but I am not sure about how to integrate with the OutputGen actions, so it is not working now. I need some explanation about it.
I hope the contents of this email are understandable, please ask for whatever I have explained badly or I have assumed for some reason.
Peace, Paolo. --
_______________________________________________ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel
-- Cheers, Lemmih
I'm still thinking about how to manage splitting a big Cabal package into an easily portable core package and sub-packages for special features, while still presenting everything as a single project. People have proposed using the CPP preprocessor, but I want to keep this out of my Haskell modules. The radical way is splitting a big Cabal package into smaller ones. But then the Haddock documentation is also split (namely the contents and index file), the sub-packages must be downloaded and compiled separately, in the right order (I know that cabal-get will simplify that). Further on the project will be shipped by Cabal in separate archives, and I guess I have to duplicate the directory structure for my project (module A.B.C goes to sub-package X and module A.B.D goes to sub-package Y) and I have complicated recompilations after changes in the core package. Even more I always have to install the recompiled core before I can access it from a sub-package. I tried to solve the problem by composing a user dependent Cabal file from small Cabal files in the configure phase. That is I divide the big Cabal file into small ones for each sub-package. Then 'Setup.lhs configure' is implemented that way, that it finds out the dependencies of the sub-packages and configures them in the right order. If one configuration fails, the sub-package and all its dependents are excluded. I merge the successfully configured sub-packages into a big Cabal file. This method let me still handle the project as one unit, the Haddock documentation is merged and no intermediate installations of packages are necessary on recompilation. However this technique is not optimal because foreign packages may depend on special features provided by sub-packages which are not installed on a particular machine. To sum it up, what I need is: Things that shall remain together: Haddock documentation Files for distributed source archive Things that shall be split: package identifiers for special features Don't bother: compiled files Any new ideas?
Henning Thielemann <lemming@henning-thielemann.de> writes:
I'm still thinking about how to manage splitting a big Cabal package into an easily portable core package and sub-packages for special features, while still presenting everything as a single project.
People have proposed using the CPP preprocessor, but I want to keep this out of my Haskell modules. The radical way is splitting a big Cabal package into smaller ones. But then the Haddock documentation is also split (namely the contents and index file),
Isn't there a way for the haddock documents to be generated together using certain haddock flags? Wouldn't it be nicer to just add this feature to cabal somehow? Are there any existing proposals that solve this question?
the sub-packages must be downloaded and compiled separately, in the right order (I know that cabal-get will simplify that).
It'll be cabal-install, and if you want to help hack on that, you're more than welcome :)
Further on the project will be shipped by Cabal in separate archives, and I guess I have to duplicate the directory structure for my project (module A.B.C goes to sub-package X and module A.B.D goes to sub-package Y) and I have complicated recompilations after changes in the core package. Even more I always have to install the recompiled core before I can access it from a sub-package.
Sholdn't module A and B go into package Z, and C (from package X) depend on Z, etc?
I tried to solve the problem by composing a user dependent Cabal file from small Cabal files in the configure phase. That is I divide the big Cabal file into small ones for each sub-package.
This approach will probably break cabal-install. Generating .cabal files is not really approved :)
Then 'Setup.lhs configure' is implemented that way, that it finds out the dependencies of the sub-packages and configures them in the right order. If one configuration fails, the sub-package and all its dependents are excluded. I merge the successfully configured sub-packages into a big Cabal file. This method let me still handle the project as one unit, the Haddock documentation is merged and no intermediate installations of packages are necessary on recompilation. However this technique is not optimal because foreign packages may depend on special features provided by sub-packages which are not installed on a particular machine.
I'm curious, why do folks frequently implement complex external tools like this for features that probably belong in Cabal, rather than contributing to cabal itself? I'm not criticizing, I'm just wondering what we develpers can do to make Cabal hacking more approachable. peace, isaac
Hi
I'm curious, why do folks frequently implement complex external tools like this for features that probably belong in Cabal, rather than contributing to cabal itself? I'm not criticizing, I'm just wondering what we develpers can do to make Cabal hacking more approachable.
Hacking Cabal was a bit frightening - in particular if you don't precisely understand what happens during configure, preInstall, postInstall and all the other stages and how they relate to each other in quite a bit of detail, you can't really figure out where to even start hacking. It's also a large code base, with lots of conventions and lots of code - which gives people a big shock when starting. That said, I did manage to get some Cabal patches in, so its not impossible by any means. Thanks Neil
On Thu, 24 Aug 2006, Isaac Jones wrote:
Isn't there a way for the haddock documents to be generated together using certain haddock flags? Wouldn't it be nicer to just add this feature to cabal somehow? Are there any existing proposals that solve this question?
Somehow haddock can do this. We would need some agreement where to install a global index file. This would depend on whether the current package is global or local. I could even add that feature by myself, since I already successfully improved the Haddock support. :-)
Further on the project will be shipped by Cabal in separate archives, and I guess I have to duplicate the directory structure for my project (module A.B.C goes to sub-package X and module A.B.D goes to sub-package Y) and I have complicated recompilations after changes in the core package. Even more I always have to install the recompiled core before I can access it from a sub-package.
Sholdn't module A and B go into package Z, and C (from package X) depend on Z, etc?
Is this a question concerning the splitting of the package without splitting the development location?
I tried to solve the problem by composing a user dependent Cabal file from small Cabal files in the configure phase. That is I divide the big Cabal file into small ones for each sub-package.
This approach will probably break cabal-install. Generating .cabal files is not really approved :)
I suspected this. :-(
Then 'Setup.lhs configure' is implemented that way, that it finds out the dependencies of the sub-packages and configures them in the right order. If one configuration fails, the sub-package and all its dependents are excluded. I merge the successfully configured sub-packages into a big Cabal file. This method let me still handle the project as one unit, the Haddock documentation is merged and no intermediate installations of packages are necessary on recompilation. However this technique is not optimal because foreign packages may depend on special features provided by sub-packages which are not installed on a particular machine.
I'm curious, why do folks frequently implement complex external tools like this for features that probably belong in Cabal, rather than contributing to cabal itself? I'm not criticizing, I'm just wondering what we develpers can do to make Cabal hacking more approachable.
The reason is simply that I have to experiment with it before adding it to Cabal. It wouldn't be sensible to add ill-designed, non-working, or even non-compiling code to the Cabal package. Even more development in a local Setup file is simpler than compiling and installing Cabal after each change. As it looks now, my experiments are not successful and I will not bother anyone with my code. Unless someone asks for it.
participants (5)
-
Henning Thielemann -
Isaac Jones -
Lemmih -
Neil Mitchell -
Paolo Martini