Hello Neil, Thursday, November 23, 2006, 4:17:19 AM, you wrote:

...

* base libs are libs, which is guaranteed to be shipped with *every* haskell distribution.

That sounds fine, and when this work has been done, bytestring and filepath both deserve to live in this second ring. Only modules which are deeply tied to the compiler should live in that core ring.

However, for the moment, base (as we know it) is the first two layers of your proposal. As such, both bytestring and filepath belong in the base library.

so, the only reason why you want to include FilePath in Base is to make it available for any haskeller? i consider this as one more argument for GHC HQ to accept my proposal ;)

There is also the question of Cabal, your second layer is not guaranteed to be installed on all systems,

reread first line of message, please :)

so Cabal setup programs cannot use them. This means that all Cabal setup programs will have to keep randomly string hacking at FilePath's.

This definitely sounds like the way to go for the future - it will make library porting to other compilers much easier. It is important to realise that this is the future, not the present, so to reject things from the base library on the basis they need moving to a place that doesn't currently exist is probably the wrong solution.

I know Bulat is investigating this, are the GHC team/Igloo aiming to split up base with any particular target timeframe?

actually i investigate(d) another problem - splitting ghc.* into the simple ghc-specific and complex compiler-independent parts, i.e. reorganizing the future-coming Core library in a way that facilitate adding support of new compilers. this really needs rather complex rewriting of existing code what i proposed in this letter is an *independent* task. even two tasks - first, make a decision that filepath and fps modules may become a part of base libs if peoples want to have them with any ghc installation and this don't need much time ;) second - split base into GHC.* and remaining parts and see how this construction will fly :) sorry, i can't try it myself without recompiling GHC -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Hello Neil, Thursday, November 23, 2006, 1:23:19 PM, you wrote:

...

so, the only reason why you want to include FilePath in Base is to make it available for any haskeller? i consider this as one more argument for GHC HQ to accept my proposal ;)

"on windows, it should be a part of monolithic installer while on unixes these libs may be a separate packages, but porters should ensure that they ported all these libs too"

To me that implies that on unixes they will all be available to install, but not all be installed by default. That means they can't be used in cabal.

i'm not a unix guru so i missed this point :) but this seems to me as an argument to include filepath in a set of core libs, not a Base lib. i should reformulate this paragraph of my proposal: * core libs are installed as a aprt of ghc installation process. these includes libs that are either ghc-version dependent (base/stm/th) or required in process of installation of other libs (cabal, filepath, filesystem?) such decision will kill both hares - it will allow to use filepath lib in setup scripts and cabal itself and at the same time it will allow to install additional versions of filepath lib to use in user projects

For FilePath there is also the specific issue that its actually required by base (see System.Directory.Internal),

and you want to replace this module with your own, raising incompatibility problem again? how about going in opposite direction - moving filesystem-related modules from Base into your library? say, current version of Base is 2.0. you can release Base 2.1 + FilePath 2.1 which provides the same functionality as Base 2.0, only with modules rearranged. and then you can go to release FilePath 3.0, 4.0 and so on which may have backward-incompatible changes. anyone requiring just the old functionality then can install FilePath 2.10 and be happy, irrespective of Base version he uses -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Hello Neil, Thursday, November 23, 2006, 4:33:57 PM, you wrote:

...

* core libs are installed as a aprt of ghc installation process. these includes libs that are either ghc-version dependent (base/stm/th) or required in process of installation of other libs (cabal, filepath, filesystem?)

Yep, that seems a more sensible definition of core. All core libraries MUST be available to all Haskell implementations. If/when that direction is taken, things can be split out, but for the moment this is a future thing.

it's no more future than inclusion of module in Base lib. we can discuss this just now and if GHC HQ will decline such proposal, i will agree that it may be better to include FilePath module in Cabal lib

...

...

For FilePath there is also the specific issue that its actually required by base (see System.Directory.Internal),

and you want to replace this module with your own, raising incompatibility problem again?

I don't want to replace it now, but this might be something for the future, if the code is examined in detail, and seems to the same thing.

so your argument for inclusion it into the Base is to have *two* modules here with overlapping functionality? :) someone who will does this optimization task may follow the same recommendations and move existing System.Directory modules into the FilePath lib. anyway, it will be better because it guarantees to not introduce backward-incompatible changes in non-upgradeable Base lib

...

backward-incompatible changes. anyone requiring just the old functionality then can install FilePath 2.10 and be happy, irrespective of Base version he uses

I don't want to evolve or enhance FilePath, I want it to reach a stage where it provides a useful set of operations, and freeze it. I also don't want to pick up the responsibility for maintaining the File related chunks of base :)

i always suspect that base inflated so much just because lack of responsibilitiness in some haskellers :) of course, it's much easier to put anything you can write here and don't think how it can be used and, while you can't imagine progress of FilePath module, i can do. what you propose to do in this case - have two filepath modules, one in the base and one in other place? or make incompatible changes in Base? to be concrete, i want to make these functions polymorphic and be able to work on various string-alikes experience shows that even such well-established modules as Data.List are sometimes changed. so, while your work is great, this don't mean that it will be frozen and can't be replaced with something even better -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Hello Duncan, Thursday, November 23, 2006, 7:56:35 PM, you wrote:

...

it's no more future than inclusion of module in Base lib. we can discuss this just now and if GHC HQ will decline such proposal, i will agree that it may be better to include FilePath module in Cabal lib

No, putting FilePath into the Cabal lib is a very bad idea. Then loads of libs/progs will depend on the Cabal lib. As we know, the Cabal library api is not terribly stable.

this looks strange but it is current state-of-the-art: it's better now to put anything into Cabal package rather than Base. why? because both packages are installed with ghc and cabal can be upgraded while base cannot :) well, i'm not serious. actually we need some package that can be upgraded via Cabal itself -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Hello Neil, Thursday, November 23, 2006, 7:57:53 PM, you wrote:

...

and, while you can't imagine progress of FilePath module, i can do. what you propose to do in this case - have two filepath modules, one in the base and one in other place? or make incompatible changes in Base? to be concrete, i want to make these functions polymorphic and be able to work on various string-alikes

type FilePath = String

I would hope that when/if we ever replace FilePath with a string-a-like or ADT, we might want to rename the type at the same time.

what i mean here: now we have, for example splitToBaseExt :: FilePath -> (FilePath,FilePath) the same function, in polymorphic way: splitToBaseExt :: FilePathClass fp => fp -> (fp,fp) and instance FilePathClass FilePath instance FilePathClass UTF8ByteString instance FilePathClass UTF16ByteString this change in interface would be indistinguishable for 99% of us. for the rest, old version of library should still be available. and only way to do it - versioning, that is impossible with Base

...

experience shows that even such well-established modules as Data.List are sometimes changed. so, while your work is great, this don't mean that it will be frozen and can't be replaced with something even better

It can, but I suspect the better thing wouldn't manipulate FilePath's, but general URL's or something with the file:// protocol as just a special case.

We will never achieve perfection, but at some point we have to commit to providing something with a fixed API that people can use.

YES! it's my whole argument - with Base, you can't provide fixed API. the only way to do it is to request library with specific version number that is installed via Cabal

Having things in a set of libraries which are tightly controlled is a good idea - currently as my libraries stand, I can change the functionality to reverse all the strings on the way out as I choose - hopefully if I did that to a base library someone would stop me.

sorry, i don't known that you are agent of KGB :) -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

On Sat, Nov 25, 2006 at 02:34:21AM +0100, Udo Stenzel wrote:

Pathnames are short strings. No need to optimize for performance, instead optimize for clarity.

There have been times in darcs where the performance of path names was an issue. For example, running "darcs whatsnew" in a large repository with no changes was taking a few seconds, I think (which doesn't sound very long, but feels like an age when you are waiting for the prompt to appear). I was able to cut it down to something almost unnoticable by optimising (and thus losing clarity), but I think the code would have been both faster and clearer had a suitable bytestring-like file path representation been available. Thanks Ian

Hello Udo, Saturday, November 25, 2006, 4:34:21 AM, you wrote:

...

the same function, in polymorphic way:

splitToBaseExt :: FilePathClass fp => fp -> (fp,fp)

...

this change in interface would be indistinguishable for 99% of us.

Err, no. 99% of us _don't_ write flawless programs. With every overloaded symbol you get worse error messages. Have you ever used C++? Ever got a screenful of completely incomprehensible gobbledygook because you forgot a namespace qualification somewhere? Go ahead with all these FooLike and BarAble classes and you get the same from Haskell (and then some, including exponential runtime of the type checker).

you shouldn't mix C++ template system with Haskell classes. C++ can't check templates at the time where template defined that leads to too late error messages. just imagine that improper definition of 'sum' will barks on each call site with a message that you can't use (&&) on Ints. and complex types in such messages is just because templates *don't* use C++ class system with its abstraction ability and the whole type structure shows up in the message if you want to argument against my idea, it's better to find situation where Haskell classes bite you (and stop rely on false assumption that something that is like C++ is automatically bad)

Pathnames are short strings. No need to optimize for performance, instead optimize for clarity.

let's say that 99% of programs don't need any optimization at all. so there is no need in ByteString, Int and other stupid types. stop polymorphism and use BigDecimal for any numeric data! :) functional programming is widely recognized as strong but slow approach. while now anyone who need to write efficient program will probably select C++, things changes and we wrote more and more efficient libraries - be it for integers, strings, arrays or pathnames. I've broken the rule that archivers (like zip) should be written in C++ and developed one with Haskell. in order to make this program efficient, i've written several general-purpose libs that don't existed before - such as serialization or packed string ones, or I/O using unicode filenames so, now i have a code that allows to write efficient programs dealing with a very large number of files. next, Unix/Win works with utf8/utf16 encoded filenames. so, efficient program should use natural filename representation instead of converting to/from String while we can develop a series of libraries that works with filenames represented as Strings, UTF8 ByteStrings and so on, i think the right way is to have one FilePath library that uses operations of some typeclass. as long as you don't argue for restricting monad libraries to IO monad, array libraries - to boxed arrays and numeric libraries - to Integer, you should agree that functionality required only by *some* developers, makes error messages more complex for the rest moreover, i think that Haskell now is the best language for system programming, such as developing mailer daemon, gaming server, or transaction system. but lack of efficient networking, I/O and data structure libraries makes development of system software sort of do-it-yourself task

An algebraic datatype reflecting the algebra of filenames would likely be even more clear, but defining a type class just to support different encodings is definitely overengineering and very likely a bad idea. Remember, this ends with std::basic_ios< char, std::char_traits<char> > or worse.

yes, types are complex nowadays :) you should look into my Streams library which includes such type of beasts. it uses class-based, multi-layered design in order to provide wide set of operations on wide set of stream types with wide set of optional modifiers (such as Char encoding or inter-thread locking). type classes allows here to split functionality into smaller manageable pieces, that can be easily replaced with 3rd-party libs. can you propose alternative design that allow the same flexibility with a less cluttered type names? i think no. the good news for you is that Haskell can infer types and constraints so using very complex types don't need a tons of type constructors as in C++ complex type names reflects *factorizing* of functionality and that is the C++ templates idea that i reused in my Streams lib. instead of defining a lot of independent getByte operations for each possible stream type, i use a class here, and this allows the next, serialization level, to work with any stream. i think that type classes are underweighted in Haskell community because it's not classical FP technique, and while we can't live without standard classes, we develop too small number of our own ones. or at least, not in all areas. besides of Monad or IArray classes, we should think about collection classes (for which Stringable is just a very partial case) -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Hello Udo, Sunday, November 26, 2006, 5:21:29 PM, you wrote:

...

you shouldn't mix C++ template system with Haskell classes. C++ can't check templates at the time where template defined that leads to too late error messages. just imagine that improper definition of 'sum' will barks on each call site with a message that you can't use (&&) on Ints.

Instead it will tell you that there is no instance for (Boolean Int), and it will do so at every call site, assuming (&&) was indeed overloaded. Where's the difference? And conversely, a properly defined 'sum' will tell you that there is no instance for (Num String) at every wrong call site, which again is just the same, of course assuming that nobody made String into an instance of Num. Sounds like we're actually in violent agreement here.

seems that you never used type signatures. just try: sum :: Num a => [a] -> a sum = foldr1 (&&) -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Hello Sven, Friday, November 24, 2006, 11:32:35 AM, you wrote:

Hmmm, this whole base library reoganization discussion gets a little bit long a confusing, because quite a few things are mixed up IMHO, so I don't even know to which mail I should reply...

Just a few miscellaneous thoughts about this area:

* Splitting up libraries in tiny little packages does not only have advantages. Even if every small package is well-tested, there is no guarantee at all that it will work well with the rest of the packages. If we split up things more and more, this problem will get more severe, because probably nobody will have the same set of versioned packages as somebody else => testing nightmare!

this problem may be solved by versioning policy (or police :), i will write a separate letter about it

* There are other quite successful libraries out there, which are very monolithic: Java's standard libraries, Boost, etc. Their advantage is: If I download version XY of them, I get a consistent, well-tested library.

i think that you just not enough involved to see complaints about impossibility of using new features in old projects

* Bulat wants to do several things at once, if I understand it correctly: a)

no. i discuss several things at once, because Neil thinks that these things must be done at once. But as i said in each letter, we don't need to wait major splitting of Base library to start development in new way. problems he mentioned may be solved just by inclusion FilePath into the set of base libs, and it's very easy change

Split the compiler-dependent parts of the libraries from the portable ones b) Introduce a finer package granularity in general. While I think that a) is a worthy goal, I think that this will be very, very tricky due to the fact that different Haskell implementations have quite different notions of builtins, extensions, etc. Furthermore, I am not yet fully convinced that the resulting structure will be vastly superior regarding readability and maintainability, sometimes some simple CPP tricks are the lesser evil.

i'm not purist and don't propose to do something supernatural. my plan is just to split base at the natural bound (ghc.*) and look how this can be made more portable by moving parts of code between two splitted parts

* It is silly to split a data type like FilePath and the only portable operations on it into different packages. Let's put the file path handling operations whereever FilePath lives, and this is "base" in the foreseeable future. Discussing if FilePath should belong into "base" is a different topic.

of course, it will be much better to move all I/O stuff from Base into separate package. that's impossible, though, because: 1) Prelude needs readFile and other I/O operations that can be imported only from Base (afaiu) 2) "instance Monad IO" defines 'fail' for I/O monad which uses Exception type whose definition involves Handle i think that these two problems shows "monolithic design" problem of Haskell itself, but while we should live with it, it should not force us to put everything Prelude-related into Base. naturally, FilePath type should belong to Files package, not amorphous and non-upgradeable Base. while we can't do it, we should at least put any other functionality here, again, while "put the type and its functions together" is a nice theory, practice, in this case, shows that 1) FilePath type never mind to change itself :) 2) Base is non-upgradeable which creates real problems for using functionality included here -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

On Thu, Nov 23, 2006 at 01:17:19AM +0000, Neil Mitchell wrote:

I know Bulat is investigating this, are the GHC team/Igloo aiming to split up base with any particular target timeframe?

It would be possible to split off bytestring, containers, prettyprinting and printf now (though some of them would still be in GHC's core, as they're used by Cabal). Splitting the rest is more complicated than picking GHC.*, as there are many interdependencies. But we could do the easy part now. On Thu, Nov 23, 2006 at 03:35:49AM +0300, Bulat Ziganshin wrote:

in particular, my hottest hope is that ghc 6.6.1 will be shipped with fps 0.8 as separate library that will provide both backward compatibility with 6.6 and will allow to upgrade fps without recompiling ghc itself :D

This would be a violation of the "no interface changes in minor releases" rule (though at the package level rather than the module level, assuming the split fps was core), but I think there's a strong argument for it, since development of this package is continuing outside base.

ross:

On Thu, Nov 23, 2006 at 01:17:19AM +0000, Neil Mitchell wrote:

...

I know Bulat is investigating this, are the GHC team/Igloo aiming to split up base with any particular target timeframe?

It would be possible to split off bytestring, containers, prettyprinting and printf now (though some of them would still be in GHC's core, as they're used by Cabal). Splitting the rest is more complicated than picking GHC.*, as there are many interdependencies. But we could do the easy part now.

On Thu, Nov 23, 2006 at 03:35:49AM +0300, Bulat Ziganshin wrote:

...

in particular, my hottest hope is that ghc 6.6.1 will be shipped with fps 0.8 as separate library that will provide both backward compatibility with 6.6 and will allow to upgrade fps without recompiling ghc itself :D

This would be a violation of the "no interface changes in minor releases" rule (though at the package level rather than the module level, assuming the split fps was core), but I think there's a strong argument for it, since development of this package is continuing outside base.

I'd be ok with this. I'm interested in SimonM's opinion, as he was interested in adding it to base originally (to replace PackedString, and to have things in base use ByteStrings). -- Don

Donald Bruce Stewart wrote:

ross:

...

On Thu, Nov 23, 2006 at 01:17:19AM +0000, Neil Mitchell wrote:

...

I know Bulat is investigating this, are the GHC team/Igloo aiming to split up base with any particular target timeframe?

It would be possible to split off bytestring, containers, prettyprinting and printf now (though some of them would still be in GHC's core, as they're used by Cabal). Splitting the rest is more complicated than picking GHC.*, as there are many interdependencies. But we could do the easy part now.

On Thu, Nov 23, 2006 at 03:35:49AM +0300, Bulat Ziganshin wrote:

...

in particular, my hottest hope is that ghc 6.6.1 will be shipped with fps 0.8 as separate library that will provide both backward compatibility with 6.6 and will allow to upgrade fps without recompiling ghc itself :D

This would be a violation of the "no interface changes in minor releases" rule (though at the package level rather than the module level, assuming the split fps was core), but I think there's a strong argument for it, since development of this package is continuing outside base.

I'd be ok with this.

I'm interested in SimonM's opinion, as he was interested in adding it to base originally (to replace PackedString, and to have things in base use ByteStrings).

We shouldn't change anything for 6.6.1, but we can reconsider for 6.8. If we think it's unlikely that anything in base will use ByteString, or that we can extract anything that does require ByteString into separate packages, then it might make sense to separate it again. Cheers, Simon

Hello Sven, Friday, November 24, 2006, 11:01:55 AM, you wrote:

...

...

in particular, my hottest hope is that ghc 6.6.1 will be shipped with fps 0.8 as separate library that will provide both backward compatibility with 6.6 and will allow to upgrade fps without recompiling ghc itself :D

This would be a violation of the "no interface changes in minor releases" rule (though at the package level rather than the module level, assuming the split fps was core), but I think there's a strong argument for it, since development of this package is continuing outside base.

I strongly object against the violation of this rule: A lot people on this list seem to have a programmer's view only, but forget about the people packaging and distributing SW. Breaking the rule makes the life for the latter people much, much harder and would lead our numbering scheme ad absurdum.

we want to break this rule just to make packaging and distribution easier. inclusion of fps into Bse was mistake, imho, which should be fixed asap. 6.6.0 probably is not in wide use, and we propose the most compatible plan which breaks only .cabal files. these files are already broken on 6.4->6.6 switch and we just want to restore 6.4 behavior - usage of ByteString require import of FPS library. so, we will end up with two sorts of Cabal files - one for 6.6.0 and one for all other ghc versions alternative solution - continue to include fps 0.8 in Base lib, and upgrade it to 0.9 in 6.8, will mean a much more serious incompatibility when switching to 6.8. plus, it will make impossible using of fps 0.9 in about year, until 6.8 arrives Don's solution - include fps 0.9 as part of 6.6.1 Base is even more terrible - it's 100% violation of these rules so, now we need to select most gracious plan of escaping from problem created by inclusion of FPS in base lib -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Am Freitag, 24. November 2006 12:23 schrieb Duncan Coutts:

[...] Um, no it isn't. fps 0.9 is completely API compatible with the current version. It's a performance upgrade only. We are not breaking API.

We knew this when we had it included in base, so we trimmed the API down to make it supportable. We have no problem following the rules.

If this is the case, there is no reason to *not* include fps 0.9 in "base" for 6.6.1. If we all agree that it would be better to have it somewhere else, we can do this for 6.8. Cheers, S.

Hello Sven, Friday, November 24, 2006, 3:00:35 PM, you wrote:

...

Um, no it isn't. fps 0.9 is completely API compatible with the current version. It's a performance upgrade only. We are not breaking API.

We knew this when we had it included in base, so we trimmed the API down to make it supportable. We have no problem following the rules.

If this is the case, there is no reason to *not* include fps 0.9 in "base" for 6.6.1. If we all agree that it would be better to have it somewhere else, we can do this for 6.8.

oh no! if you mean Base library here, this will continue the same problem - when something new will be added to FPS, it will be impossible to use it before 6.8 arrives. and with 6.8, all old programs will stop working because FPS changes i just try to solve our *future* problems, which you don't foresee -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Am Freitag, 24. November 2006 10:18 schrieb Bulat Ziganshin:

[...] we want to break this rule just to make packaging and distribution easier. [...]

Apart from being a contradiction in itself, it is quite funny that you argue against your own version numbering proposal. Of course everybody thinks that he has a very special case, and general rules of course don't apply then, but then we could immediately abandon any numbering scheme because its semantics would be void...

inclusion of fps into Bse was mistake, imho, which should be fixed asap. 6.6.0 probably is not in wide use, and we propose the most compatible plan which breaks only .cabal files. these files are already broken on 6.4->6.6 switch and we just want to restore 6.4 behavior - usage of ByteString require import of FPS library. so, we will end up with two sorts of Cabal files - one for 6.6.0 and one for all other ghc versions [...]

Making assumptions who will use what are a bit dangerous. And it doesn't really matter if "only" .cabal files are broken or an import has to be changed or an identifier has to replaced all over the project: A working project will be broken, and some manual intervention will be needed. Cheers, S.

Bulat Ziganshin wrote:

what is the base library now? is it about data structures, I/O, concurrency or monads? there is no answer - this lib contains everything developed by peoples having accounts at darcs.haskell.org. why it's impossible to use Applicative class with ghc 6.4? because Ross Paterson was so kind to include this in a base lib. why peoples get a problems with their mutable arrays in 6.6? because arrays interface was changed and there is no way to use old lib in 6.6. why ForeignPtrs in 6.4 are slow? because it's a part of base lib and it's impossible to use faster ForeignPtr implementation without upgrading entire compiler. and so on, so on

inflation of base library and inability to upgrade it have cumulative effect. i think that our ultimate goal is a set of ghc-independent concrete libs: arrays, control, files, concurrency... and any step in this direction will simplify upgrading to next ghc version

Chris:

...

...

example, it includes 2 of 4 regexp libs? at last end, we can learn that this is the list of libs required to build ghc itself!

...

Some of the other regex-* libraries are interfaces to c-library backends (pcre and tre) and it is not GHC's place to come with copies of those

I think Bulat has a made very convincing argument here. These problems with upgrading (or downgrading, for compatibility) of base library have bitten me too sometimes. libraries.  But

...

GHC used regex internally, so it must come with at least a single backend.

what means "used internally"? you mean compiler sources? i don't see reasons why each copy of ghc should be able to compile ghc itself

In the improbable case that Haskell compiler has been installed without some basic libraries, I think it would be okay if ghc build would complain that it needs library X or Y. It should say from where these can be downloaded and the installation should be straight forward. Ben

Neil Mitchell wrote:

...

In the improbable case that Haskell compiler has been installed without some basic libraries, I think it would be okay if ghc build would complain that it needs library X or Y. It should say from where these can be downloaded and the installation should be straight forward.

That's not the job of GHC, that's the job of Cabal. And if the libraries to run Cabal are not present, then what happens?

This would be bad. Of course there must be some core libraries that have to be shipped with the compiler. Cabal and anything that cabal needs to run is definitely a must-have. What I am arguing for (and Bulat, if I understand him correctly) is that it's ok if building the _compiler itself_ (or profiler or whatever else (apart from some exceptions like cabal) is shipped with the compiler) needs libraries from outside the core library set. Cheers, Ben

Duncan Coutts wrote:

...

This would be bad. Of course there must be some core libraries that have to be shipped with the compiler. Cabal and anything that cabal needs to run is definitely a must-have.

Sadly this may be rather expansive. Building software is not a simple task. If we include cabal-get too then that pulls in networking, http, and if we want to do security properly a bit of crypto too.

While I'm happy to see this discussion (the base inflation has been a hobby horse of mine for some time now), I think that the important issue is the definition of the packages, not bundling and unbundling with the compiler. As I see it, this comes down to whether you want cathedral or bazaar-style development: When a library comes as a separate package, it is easy to download the current development version with darcs, install it in place of the default compiler version with cabal, and tinker away. If it is part of base (or any other huge, catch-all package), things get more complicated and it is simpler to wait for the next release ex cathedra. For instance, I used to follow FPS regularly, and played with it a bit, (e.g., I implemented support for non-latin 8-bit charsets). After it got incorporated in base, I instead just whine and wait, even for simple stuff like interface issues that I could easily fix myself. This is not to say that the cathedral model is necessarily a bad one, but it seems a shame that with tools that seem so ideal for bazaar-style development, users are reduced from participants to consumers. -k

Hi I'm proud to present yet another tremendously important contribution to the ever-growing collection of Haskell libraries. FPS was included in 'base' for recent GHCs, which has broken backwards compatiblity of .cabal files (since they typically Build-Depends on base and fps). My solution is a new fps package which contains no useful functionality at all, and which consequently, should go in the officially sanctioned and sprinkled-with-holy-water core libraries ASAP. :-) Anyway: Version: 0.8 Availability: darcs get http://www.ii.uib.no/~ketil/fps Functionality: none Bugs: none known Todo: QuickCheck properties, Haddock documentation Enjoy! -k

Hello Sven, Friday, November 24, 2006, 12:24:34 PM, you wrote:

...

This is not to say that the cathedral model is necessarily a bad one, but it seems a shame that with tools that seem so ideal for bazaar-style development, users are reduced from participants to consumers.

Even if my last emails on this list might give a different impression, I really like bazaar-style development. But currently all the necessary tools (cabal-get etc.) and APIs (e.g. stuff needed in the Setup.hs for Cabal) are far from being stable and widespread (Does your latest and greatest openSUSE DVD or Ubuntu download have all this? And if yes, which API version? Does your Setup.hs still work?). This is no criticism of the projects, quite the opposite, it is just the current state of affairs. I fear that if we move from the cathedral to the bazaar too fast without having the necessary infrastructure already on everybody's hard disk, we will hurt Haskell development more than we help it.

i'm pretty happy with just darcs + cabal (and i also participated in fps development a bit). while for the users who can't use it, the things will remain the same - they will got a large number of preinstalled and/or easily installed libs which they can consider as unbreakable thing. and at the same time, any new libs they install may silently require other versions of the same libs, install them and users will not even see that something changed so, i think that both models now supported, we just don't have tools to simplify working in bazaar model -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Duncan Coutts wrote:

On Thu, 2006-11-23 at 19:15 +0100, Benjamin Franksen wrote:

...

What I am arguing for (and Bulat, if I understand him correctly) is that it's ok if building the _compiler itself_ (or profiler or whatever else (apart from some exceptions like cabal) is shipped with the compiler) needs libraries from outside the core library set.

Yes. I agree.

Since GHC is currently statically linked, the set of libs that are required to build GHC need not be related to the set of libs that come with GHC by default.

Just one comment here: the set of packages required to build GHC is actually very small: just unix (if you're on Unix) and regex-compat. The set of packages required to build a stage 2 GHC is larger, but we include all those in the source tree. Yes there's no reason that we actually have to *install* these packages too, but it seems silly not to, since we just built them. It would be more trouble to arrange that we didn't install them, I guess. Cheers, Simon

Bulat Ziganshin wrote:

Hello Simon,

Friday, November 24, 2006, 2:09:05 PM, you wrote:

...

Yes there's no reason that we actually have to *install* these packages too, but it seems silly not to, since we just built them. It would be more trouble to arrange that we didn't install them, I guess.

for me the problem is not that you include these packages - i like them all (except for ghc-library). the problem is that you don;t want to make guarantees that anything else will be available with *every* haskell installation.

What makes you think we don't want that? We (the GHC developers) are very supportive of the idea of having a consistent set of packages provided by default (whatever that means) across different compilers and platforms. Cheers, Simon

Hello Ian, Friday, November 24, 2006, 8:17:59 PM, you wrote:

...

...

Since GHC is currently statically linked, the set of libs that are required to build GHC need not be related to the set of libs that come with GHC by default.

Also, once we have dynamic libraries working we'll have to go back to installing them again (or jump through some hoops), so it seems shortsighted to go this route.

i think about standardization (i.e. inclusion in set of Core libs) of library as 4-step process: * library author propose to do it * user wants this library * compiler developers and porters agree to support it * deal! in general, users want to include as much libs as possible while developers/porters want to have to support as few libs as possible. so main problem is balance between these two requirements if developers want to include more - as in this case - it is no problem at all (with exception of large libs that will significantly increase size of downloading/installation) so, i don't see any reason to discontinue including in GHC bundles these small libs which you support anyway. my request was about inclusion in this set additional libs, which is not required to compile ghc but nevertheless widely used and should be easily available, say, for windows users -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Hello Ian, Monday, November 27, 2006, 5:47:01 PM, you wrote:

It means library writers are unhappy that no-one uses their libraries, users are unhappy that it is hard to get libraries they need and increases the pressure on us to put more libraries into core libs.

All you need is to be in Cabal! Cabal, Cabal, Cabal! It's all you need!!! :) i will write more serious answer in another letter :) -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

On Mon, Nov 27, 2006 at 02:47:01PM +0000, Ian Lynagh wrote:

For GHC at least, the set of libraries included isn't going to change before 6.8, so we have until then to bring Hackage up to speed so that this is not true.

So, I would like to through down the gauntlet to the community to make this happen! We have ambitions to make large steps in this direction at Hac 07 http://haskell.org/haskellwiki/Hac_2007 (but that shouldn't stop people who can't be there from working on it too, or people working on it before then!).

A noble quest. Does anyone know if the following are the most recent versions of the various components of Hackage? cabal-setup: darcs get http://darcs.haskell.org/cabal cabal-install: darcs get http://darcs.haskell.org/cabal-with-install hackagedb: darcs get http://darcs.haskell.org/~lemmih/hackagedb package upload tool: none, though there is a Perl script /usr/local/bin/cabal-put on darcs.haskell.org Anything else?

On Tue, Nov 28, 2006 at 01:19:21PM +0000, Duncan Coutts wrote:

On Tue, 2006-11-28 at 12:23 +0000, Ross Paterson wrote:

...

Does anyone know if the following are the most recent versions of the various components of Hackage?

cabal-setup: darcs get http://darcs.haskell.org/cabal

Yes.

...

cabal-install: darcs get http://darcs.haskell.org/cabal-with-install

I think this is now the latest. The -with-install branch is slated to be merged with the main cabal branch some time (Paolo: any word on that?).

How about hackageDB? What's the situation there?

On Tue, Nov 28, 2006 at 05:13:24PM +0100, Sven Panne wrote:

I am sure that this has already been considered somehow, but could somebody be so kind to give a short explanation of the plans in this area?

There are a number of things going on here. If SuSE has an RPM for hnop then yes, using yast is definitely the way to go. If it doesn't then you can either tell the magic rpm-maker to convert hnop-0.1.tar.gz into an rpm and install that as above, or you can tell cabal to install it into /usr/local. This last option has the downside that yast/rpm won't know it exists, and so can't use it to know the deps of something else are satisfied (although presumably nothing will depend on it as it wasn't packaged), and it could stop working at a later date if a package it depends on that was provided by an rpm is removed. What hackage provides is a place to keep all the tarballs, a way to view and search for them and a way to upload new packages and new versions. Then people making SuSE rpms, or people looking for things that aren't packaged for SuSE, know where to go. It also provides a set of packages that can be used by cabal-get, for SuSE users to get packages that have not been made into SuSE RPMs, as well as for people who do not use a package manager (e.g. students on uni machines that they don't have root on). In the future we can add things like the ability to rate packages, mark packages as bitrotted, automated testing that packages compile and their testsuites pass, mailing lists, darcs repos, bug trackers etc for packages in hackage. Thanks Ian

Am Donnerstag, 30. November 2006 18:10 schrieb Malcolm Wallace:

[...] Right. And one of the explicit goals of Cabal was to provide enough infrastructure on the Haskell side to make it easy for rpm/apt/smart builders to do their job. [...] Cabal-get and hackage do go a bit beyond that, but they are intended for those who do not live in an OS-package managed universe (or who don't want to). [...]

OK, that makes sense. Just to test if I understood things correctly: * For a package building RPMs, only Cabal is relevant and it is used from .spec files to (un-)register Haskell packages. * Resolving dependencies, upgrading, etc. is done as usual in the RPM world, the .cabal files are just a hint for writing the .spec files. * For a package builder, Hackage will make it easier to find the latest and greatest packages in source, but apart from that it is of no use/only a convenient "one-stop-shop". * Joe User on an RPM-based system will never invoke Cabal and will never use Hackage. The same holds for other packaging systems, of course. Cheers, S.

Simon Marlow wrote:

In 6.6 you can, in theory, upgrade the base package.

This, I think, is the heart of Bulat's complaint: namely, that until recently it was not possible to upgrade the 'base' package in ghc. All the other implementations of Haskell do permit the base package to be upgraded (or downgraded) separately from upgrading the compiler itself. All of the questions about whether such-and-such a library belongs in the 'base' package or in some separate package, are questions of policy. The answers really don't matter quite so much, provided the mechanism exists to choose to install whatever version of whatever packages you like. So can we take all this discussion about contents of the 'base' package as a bug-report about (older versions of) ghc, rather than a bug-report about the 'base' package itself?

However, it's highly unlikely that you'll ever be able to separate ForeignPtr from the base library, because of the dependencies: ForeignPtr is part of the FFI, the FFI is used by other libraries, etc. etc. The dependencies in base are very intertwined, as you well know.

Ah, so how do the other compilers manage it? Answer: Hugs has a separate package 'hugs-base' with essential but implementation-dependent things like ForeignPtr in it. (Likewise nhc98/yhc.) The 'base' package just imports and uses it. If we could separate out a 'ghc-base' package from 'base', that would be great. (And I think that is one of Bulat's proposals.) Regards, Malcolm

| as i many times said, the problem is two-sided: first, Base can't be | upgraded without upgrading ghc. I don't think that is any longer true, though we have admittedly not run a full-scale experiment. Please try. | second, Base contains many different | functionality which is artificially tied together. That remains a good point. S

Bulat Ziganshin wrote:

Hello Simon,

Friday, November 24, 2006, 7:59:13 PM, you wrote:

...

| as i many times said, the problem is two-sided: first, Base can't be | upgraded without upgrading ghc.

...

I don't think that is any longer true, though we have admittedly not run a full-scale experiment. Please try.

where can i download Base 2.0 for ghc 6.2 ? :)

Nobody has implemented it. That doesn't mean it cannot be done. Cheers, Simon

Simon Peyton-Jones wrote:

http://hackage.haskell.org/trac/ghc/wiki/PackageReorg

I'd be interested to know whether the proposal has general support;

Yes, it has my general support. The proposed terminology is good.

what goals are omitted;

Possibly the following are just more detailed expansions of the existing two goals: * Forwards compatibility. Users would like their programs written against the 'core' packages to continue to work, without modification to source text or build system, after upgrading the compiler, or its packages, or switching to a different compiler. * Backwards compatibility. Users would like to be able to take a program written against some version of the 'core' packages, and build it with an older compiler, accepting that they may have to install newer versions of the 'core' packages in order to do so. It may not be possible to fully achieve these goals (in particular the latter), but that does not mean we should not aim for them. Regards, Malcolm

On Fri, Nov 24, 2006 at 05:35:31PM +0000, Simon Peyton-Jones wrote:

http://hackage.haskell.org/trac/ghc/wiki/PackageReorg

I'd be interested to know whether the proposal has general support; what goals are omitted; and what other details are omitted. I'm sure there are many, because this is a complex issue.

I'd like to see some rationale for why each of the core packages have been chosen. I'd have base, haskell98, Cabal, filepath with the rationale that it is likely that people will want to write Setup.hs with them (haskell98 is debatable here, but could probably be argued in on the grounds that it's needed for lots of books/tutorials/... to work). I can't really think of any rationale for any others. For userfriendliness we can have a cabal package called usefulstuff that depends on mtl, regex, ..., but I don't see any need to bundle them all with the compiler. The less we force the implementations to come with, the quicker compilation will be when developing, the smaller Debian packages (for example) can be, the lower the disk space requirements to build GHC, the lower the time wasted when a Debian package (for example) build fails and the fewer packages we are tangling up with compiler release schedules (OK, they can be upgraded independently, but it causes headaches for package-management systems). On quickcheck, the testsuite and GHC: we can have GHC compile any libraries that are put in libraries/ and make ./darcs-all --get-dev grab quickcheck too. On the other hand, you might want to run the testsuite automatically when building debs/rpms/... but personally I'd prefer to use a copy of QC compiled and installed under the testsuite tree for that. This way we wouldn't suffer from the disadvantages above. We don't need the testsuite programs to work after GHC has been installed so we don't have to worry about providing the QC dynamic libraries in the future. Thanks Ian

Malcolm Wallace wrote:

Simon Marlow wrote:

...

However, it's highly unlikely that you'll ever be able to separate ForeignPtr from the base library, because of the dependencies: ForeignPtr is part of the FFI, the FFI is used by other libraries, etc. etc. The dependencies in base are very intertwined, as you well know.

Ah, so how do the other compilers manage it? Answer: Hugs has a separate package 'hugs-base' with essential but implementation-dependent things like ForeignPtr in it. (Likewise nhc98/yhc.) The 'base' package just imports and uses it. If we could separate out a 'ghc-base' package from 'base', that would be great. (And I think that is one of Bulat's proposals.)

As I understand it, Hugs and nhc98 do this by having their own copies of a lot of the library code (up to and including the Prelude?). We could do this for GHC, but it would mean extracting a lot of code that isn't really GHC-specific into ghc-base, and that doesn't seem like the right way to go. So there are two approaches to reducing the size of the base package: 1. removing modules from the top into separate packages 2. try to extract a compiler-dependent portion from the bottom (1) is way, way easier than (2), because of the heavily intertwined dependencies between the code underneath Prelude. That's unfortunate. Bulat did start to try to separate things this way, but he didn't finish (Bulat - did you give up because it was too hard, or just stop?). In the wiki page, we describe how to get part of the benefit of (2) as far as the user is concerned, by having a separate "view" of the base package with the compiler-dependent modules removed. Cheers, Simon

On Mon, Nov 27, 2006 at 02:42:18PM +0000, Malcolm Wallace wrote:

Simon Marlow wrote:

...

...

Answer: Hugs has a separate package 'hugs-base' with essential but implementation-dependent things like ForeignPtr in it. (Likewise nhc98/yhc.) The 'base' package just imports and uses it.

As I understand it, Hugs and nhc98 do this by having their own copies of a lot of the library code (up to and including the Prelude?).

Yes. Essentially, we took the view that the Haskell'98 libraries (+ primitives + FFI) can be assumed, and then built the 'base' package on top of that. Whereas GHC took the opposite approach, making 'base' the bottom of the hierarchy and building 'haskell98' on top of that. Hence why lots of GHC-only primitives ended up in 'base', which is not good for a so-called portable package.

Hugs takes an intermediate approach, basically as described by Simon: There's a copy of the Prelude (and parts of other modules that are mutually recursive with it, and some other primitives); base is built on that, and haskell98 on top of that. So we share most of base (and all of haskell98) with GHC. The Hugs module system would allow the Prelude to import other modules like Data.Maybe and System.IO, getting more sharing, but since the whole thing will be interpreted each time Hugs starts, we preferred to minimize the part under the Prelude. There is some other duplication, notably large chunks of Numeric, due to constraints on module relationships imposed by GHC's specialization pragmas.

On Thu, Nov 23, 2006 at 10:20:51AM +0000, Chris Kuklewicz wrote:

Some of the other regex-* libraries are interfaces to c-library backends (pcre and tre) and it is not GHC's place to come with copies of those libraries. But GHC used regex internally, so it must come with at least a single backend.

A problem with these packages is that they're a good deal less portable than the Text.Regex they replaced. They use multi-parameter type classes, but they also rely on late overlapping resolution, which is a GHC-only feature. That is, instances overlap, but they're only used outside of the overlap, e.g.: instance (RegexLike a b) => RegexContext a b (Array Int b) instance (RegexLike a b) => RegexContext a b MatchArray type MatchArray = Array Int (MatchOffset, MatchLength) type MatchOffset = Int type MatchLength = Int These overlap at RegexContext a (Int,Int) (Array Int (Int,Int)), but as long as you don't use that instance GHC doesn't complain. (And you won't, because there's no instance RegexLike a (Int,Int), but overlaps are decided by comparing instance heads only.) Similarly instance (RegexLike a b) => RegexContext a b [Array Int b] instance (RegexLike a b) => RegexContext a b [MatchArray] and also instance (RegexLike a b) => RegexContext a b [b] instance (RegexLike a b) => RegexContext a b [MatchArray] instance (RegexLike a b) => RegexContext a b [(MatchOffset,MatchLength)] Can anything be done about that?