Hello libraries, current situation when Base lib includes everything, from 'const' to starship control, creates serious problems for upgrading to next GHC version. this problem arise because every GHC version arrives with its own, non-upgradable version of Base library and even minor API changes here results in total loss of ability to use new ghc version without #ifdef workarounds it seems strange, but even in this situation some developers prefer to include their modules in base lib instead of providing them as separate libs that can be easily upgraded using Cabal so that any library version can be used with any version of ghc after analyzing the situation, i concluded that source of problem, at least partially, is GHC HQ's policy of packaging libraries with GHC. The list of such libraries is *closed* and odd at first look. why, for example, it includes 2 of 4 regexp libs? at last end, we can learn that this is the list of libs required to build ghc itself! unfortunately, ghc building process don't require FilePath or ByteString support. this results in situation when authors of these libs, looking for a way to include their libs in every ghc distro, don't find other solution except for including them in Base lib. moreover, while Simon & Ian argued against inclusion of these very useful libs in base libs set, they don't have anything against inclusion of the same code in Base lib i'm all for docracy, but not in this case. having the experience of patching the program after each ghc upgrade, and having a bunch of conditionally compiled code in my libs, i can easily imagine ghc version hell that these new modules will provide for us so, i think, it is time to revise ghc libraries policy. i propose to split libs into the 3 "rings": core, base and other * core libs includes ghc version dependent libs. currently it's base, stm and th * base libs are libs, which is guaranteed to be shipped with *every* haskell distribution. on windows, it should be a part of monolithic installer while on unixes these libs may be a separate packages, but porters should ensure that they ported all these libs too this set should include only small and widely used libs, what is mainly datastructure processing ones. my current list includes regexps, bytestring, filepath, time, mtl, fgl, newbinary and so on * other libs are either too specific or too large, such as gui, database and xml ones. these may be included in some distros or installed separately these rules don't say anything about ghc self-building process. it will be great to maintain the list of libraries, required to do it, separately and while we are here - splitting of Base library should be very easy task. its ghc version specific part is in GHC.* modules. these modules should be moved into the new Core library. plus a few modules from Data.* or System.* hierarchy they imports. plus a few modules imported by Hugs/NHC Prelude. the rest of Base library should be pretty portable, at least between various GHC versions. and independent on this work, we can always look into moving pieces of Base into independent libs. as a rule of thumb, everything not used in Base lib may be moved outside of it. of course, these new libs should be included in base libs set in particular, my hottest hope is that ghc 6.6.1 will be shipped with fps 0.8 as separate library that will provide both backward compatibility with 6.6 and will allow to upgrade fps without recompiling ghc itself :D -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com
This is a small reply to clarify the regular expression situation: Bulat Ziganshin wrote:
Hello libraries,
after analyzing the situation, i concluded that source of problem, at least partially, is GHC HQ's policy of packaging libraries with GHC. The list of such libraries is *closed* and odd at first look. why, for example, it includes 2 of 4 regexp libs? at last end, we can learn that this is the list of libs required to build ghc itself!
Some of the other regex-* libraries are interfaces to c-library backends (pcre and tre) and it is not GHC's place to come with copies of those libraries. But GHC used regex internally, so it must come with at least a single backend.
[...snip...]
and while we are here - splitting of Base library should be very easy task. its ghc version specific part is in GHC.* modules. these modules should be moved into the new Core library. plus a few modules from Data.* or System.* hierarchy they imports. plus a few modules imported by Hugs/NHC Prelude. the rest of Base library should be pretty portable, at least between various GHC versions.
and independent on this work, we can always look into moving pieces of Base into independent libs. as a rule of thumb, everything not used in Base lib may be moved outside of it. of course, these new libs should be included in base libs set
The regex-* modules that are installed with GHC are not in the "base" package. They are in "regex-base" and "regex-posix" and "regex-compat" because I took excellent design advice and finely split the modules. I see no reason one could not by able to upgrade these on top of the GHC versions.
in particular, my hottest hope is that ghc 6.6.1 will be shipped with fps 0.8 as separate library that will provide both backward compatibility with 6.6 and will allow to upgrade fps without recompiling ghc itself :D
By separate library you really mean using a separate package name instead of "base". The Text.XHtml is already separate into the "xhtml" package. You are correct: as far as I can see the only module that is in "base" which might need to be split out is Data.ByteString. The same may or may not apply to Data.Sequence. Cheers, Chris
Simon PJ and I spent some time thinking about this today, and wrote a wiki page. There are several interconnected issues, which makes this hard to discuss on a mailing list. I hope this can evolve into a concrete design for a reorganisation of the packages. Please feel free to edit the wiki, or continue to discuss (individual points one at a time, preferably!) on the list. http://hackage.haskell.org/trac/ghc/wiki/PackageReorg Cheers, Simon
Hello Simon, Friday, November 24, 2006, 1:58:36 PM, you wrote:
There are several interconnected issues, which makes this hard to discuss on a mailing list.
i hardly imagine how it can be discussed on wiki :)
thanks, it's much closer to my POV. still some problems exists. first, this proposal don't realize problems with upgrading libraries used by Cabal itself. is it possible to have, say, base 2.0 and 2.1 installed side-by-side and make Cabal import just base 2.0 against which it was written? second, rules for ghc-specific libs pops up as some modification of initial rules, it will be better to include them in general rules themselves. third, this don't split core libs into the pre-installed and installed by package manager. so, my updated proposal: Now Haskell standard libraries (as specified in H98 Report) are very small and old-fashioned, it's impossible to develop real programs using this API. In order to facilitate Haskell-based development, we need a much wider, not-fixed set of libs which are guaranteed to be available with every Haskell compiler. libraries should be split into 4 rings: frozen, core, base and the rest * frozen libs are installed with haskell compiler and cannot be upgraded using Cabal. it includes Cabal itself and libraries required by Cabal, currently it's the Base library. compiler may provide its own ways to upgrade these libs * core libs are tied closely to specific compiler and version. therefore, these are also preinstalled but can be upgraded with Cabal. in order to make upgrading not-a-headache, version of such library must include compiler version, i.e. TH 6.6.0. for ghc these currently includes TH and STM libs * base libs are *guaranteed* to be shipped with any haskell compiler (compliant with this proposal). these libs are installed using Cabal or OS package manager, only on poor Windows-like systems these should be included in monolithic compiler distro. the reason is to allow user to got latest, bug-fixed versions of these libraries for the installation moment these libs should include rich set of functionality. criteria of inclusion library in this set are: widely used, small and portable. Portability: these libs should work on windows and major unix branches, on ghc/hugs/nhc and other proposal-compliant compilers, otherwise we can't guarantee availability everywhere. It should be easy for porters to prepare such lib for their systems Widely used: this should include data structures, general data processing, other things that facilitates development in general as opposite to solve some particular problem. plus basic OS interfaces. haskell98, readline, unix/Win32, QuickCheck, fgl, haskell-src, html, mtl, network, parsec, time, xhtml, ByteString, regex-*, Edison, Filepath, MissingH, NewBinary, arrows, HUnit, QuickCheck, monads it should be great also to strip down size of HaXml lib and include it here. now it don't conform only to size criterion * there are also other great libs that are either too fat or os/compiler specific, such as gui and database ones. these can be included in some distros or downloaded and installed separately using Cabal infrastructure there are also a number of Cabal-specific problems we should solve: 1) libraries required by setup scripts (Setup.hs) should be specified as dependencies in .cabal file: Build-Depends: FilePath this way, the library will be installed just before running Setup script. of course, all the Build-Depends libs should be made available for setup script 2) cabal, cabal-get and other package management tools should use only libraries from frozen set. any extra functionality required (networking, file management...) should be *copied* into these packages. this just means handmade instead of automatic dependencies tracking, but we can't rely on automatic management for tools that implement this management! (this proposal don't include some topics, like testing and Hugs, where i can't say anything new compared to Simon's page) -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com
Am Freitag, 24. November 2006 15:42 schrieb Bulat Ziganshin:
[...] libraries should be split into 4 rings: frozen, core, base and the rest
That's one possibility, but not the only one. Especially I don't see the need to distinguish between "frozen" and "core".
[...] these libs should include rich set of functionality. criteria of inclusion library in this set are: widely used, small and portable.
While "portable" (in the sense of: "runs on a gives set of Haskell implementations/platforms") is a good criterion, the other ones might need some discussion: How can somebody determine if a package is "widely used"? If there was a reliable way to figure this out, I would probably keep it for myself and make millions of dollars in the marketing industry... And "small" is not a criterion at all. Is Boost small? No. Is java.io.* or java.util.* small? No. Etc. etc.
[...] haskell98, readline, unix/Win32, QuickCheck, fgl, haskell-src, html, mtl, network, parsec, time, xhtml, ByteString, regex-*, Edison, Filepath, MissingH, NewBinary, arrows, HUnit, QuickCheck, monads
This is a completely random selection of packages based on personal preferences. I could give my personal favourites, which would not have a large intersection with the set above, but this would lead us nowhere. So the meta-question is: How will we determine which packages are worty enough to be "base" packages?
[...] 2) cabal, cabal-get and other package management tools should use only libraries from frozen set. any extra functionality required (networking, file management...) should be *copied* into these packages. this just means handmade instead of automatic dependencies tracking, but we can't rely on automatic management for tools that implement this management!
Uh, oh, copying... :-P Redundancy is the source of all evil! If we step back a little bit, we can see that there is no intrinsic need that those tools are even written in Haskell, even less need that libraries needed to build those tools are available on non-developer installations. If e.g. cabal-get is a statically linked application, nobody is forced to install any networking/crypto/... packages. The situation here is quite similar to the (mostly) statically linked commands in /bin on most *nices. Cheers, S.
On Fri, 2006-11-24 at 17:42 +0300, Bulat Ziganshin wrote:
libraries should be split into 4 rings: frozen, core, base and the rest
* frozen libs are installed with haskell compiler and cannot be upgraded using Cabal. it includes Cabal itself and libraries required by Cabal, currently it's the Base library. compiler may provide its own ways to upgrade these libs
Cabal can be upgraded using Cabal. This is important. Unlike well specified libraries like ByteString, Cabal tackles an ill-specified problem and we're still learning how best to solve it, so the api and features change much more frequently. Duncan
participants (5)
-
Bulat Ziganshin -
Chris Kuklewicz -
Duncan Coutts -
Simon Marlow -
Sven Panne