(i crosspost my letter into main haskell list because i hope that proposed solution is of great interest for (discussion with) many developers) Hello Ian, Friday, September 8, 2006, 5:52:57 AM, you wrote:

...

and last question - i don't like inclusion of unix and win32 in a list of core libs. why they are here? may be it's possible to include small modules with functionality required for compiler itself in GHC.* hierarchy and then move the rest into extra libraries?

This doesn't work well for any parts shared with the other compilers. "core libs" is possibly the wrong name for it - "bootstrapping libs" might be more appropriate.

Note also that unix and win32 haven't been included in anything they weren't already, it's just that other libraries (those that are now called "extra libs") have been taken out. We can always take more out for 6.8 if we want.

thank you - you have cleaned up the situation for me. thanks to Cabal, now ghc is much more modular than 6.4. but it is not yet the ultimate solution and i propose to discuss what we can do in future, possibly even in 6.6.1. i will become devil advocate for a little :) i am, John de MacLee programmer, never planned to build ghc itself and i don't need any bootstrapping libs in my download. please sell it as separate ghc-for-ghc package :) the core-ghc package should then contain only libraries that are dependent on GHC compiler internals (GHC.* part of base, stm and th - and nothing more!) moreover, i want to be able to upgrade even these libraries without upgrading compiler proper. or, to be exact, i may need to install newer versions of these libraries which contains new features and therefore not 100% compatible with libraries shipped at the moment when GHC 6.6.1 was released why this may be impossible? first, because existing libs and programs may rely on older interfaces. i think that Cabal should eventually solve this problem so that multiple version of any lib can be installed on computer (to be exact, on concrete GHC installation) and proper version of library selected for any project. but that's another discussion.. what is a 'base' library now? it is the library that implements common set of operations for latest versions of ghc, hugs and nhc. it contains low-level implementation for ghc, but relies on separate hugsbase package for hugs (the same for nhc, afaiu). so, first step is obvious - separate ghc-base library from the rest. hugsbase, ghc-base and nhc-base packages should provide common set of low-level operations, hiding from other libraries implementation details, differences between compilers, and differences between compiler versions. they should provide _incremental_ interfaces so that old code will continue to work with newer compilers. eventually compiler-specific code for stm and th should also go into these libraries but that is not the immediate goal then, a base library may be written against "virtual Haskell compiler", which provides uniform set of low-level features while 'base' decorates these features with user-friendly interfaces even more interesting variant is to allow ghc-base and other compiler-specific base packages to export non-incremental interfaces and use 'base' solely to "equalize" all compilers to some common interface, providing emulation of all missing features (i think that such emulation will be compiler-independent that means that it's better to put it into compiler-independent package). then, _all_ other libs should rely on version of base package version instead of version of compiler they are use. so: ghc 6.2 ghc 6.2.2 ghc 6.4.3 hugs 2003 .... all supported in base 1.0 package which expose stable interface independent on compiler used. all other libraries relies on this interface and therefore works with any compiler whose support included in base 1.0. as ghc 6.6 rolls out, we add its support to 'base' library, rolling out base-1.0.1. and all libraries written against base 1.0, now will work with ghc 6.6, although they can ignore some features what was not included in base 1.0 API. at the same time, base-2.0 rolled out which includes new APIs (but don't omit old ones!), supporting new features of ghc 6.6. but base 2.0 continues to support existing compilers, providing emulation of new features for old compilers. those developers that need these new features upgrade their cabal files to require base 2.0. those users of old compilers that go to compile these apps download (automatically) and install base-2.0 lib so, i propose: - ghc-base/hugsbase/.. libs to implement _subset_ of common low-level API - base lib to "equalize" several compilers and compiler versions, providing _full_ common low-level API. when we need to include new API, we roll out new major version of base and work hard to support old compilers by providing some emulation of new feature. base lib versions should be independent on compiler versions and as much backward-compatible as possible - all other libs to just specify version of base lib they utilize and be happy and of course we should split base lib into ~10 independent libraries to simplify development and deployment of new features. if i need, say, new ByteString features, i would be very unhappy if i need at the same time to upgrade interface to arrays last line: i have some experience of writing compiler-independent code with Haskell and C++ and believe that this plan is realistic -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Hello Neil, Thursday, September 14, 2006, 6:14:30 PM, you wrote:

...

then, a base library may be written against "virtual Haskell compiler", which provides uniform set of low-level features while 'base' decorates these features with user-friendly interfaces

Nice idea. There are a few practical issues - for example does this virtual Haskell copmiler support higher rank types? Multi-parameter type classes? Bang patterns? It quickly gets a lot more complicated.

i mean by "virtual Haskell compiler" set of _library_ functions, not the language features, pragmas and other differences. problem of availability of language features for standard libraries typically solved in conservative way - i.e. we try to use as less language features as possible and separate modules that use non-standard features in separate packages. for example, "modern array libraries" should go into separate package because they use MPTC and therefore not available for many compilers. shebang patters, of course, should be no used - it's a feature for applications, not for writers of standard packages

...

- ghc-base/hugsbase/.. libs to implement _subset_ of common low-level API Sounds like a very good idea.

...

- base lib to "equalize" several compilers and compiler versions, Yes, some operations might be implemented in the base library, but have more efficient versions in the ghc-base library. For example, map for base should be defined the obvious way, for ghc it should be defined with foldr. How can you accomodate this?

i'm pragmatic and don't think that base package should hide _all_ the compiler differences and especially all compiler-specific optimizations. going this way, we should put to ghc-base byte strings and many other things my proposal is to extract from 'base' package all high-level, written in pure Haskell algorithms and put it in ~10 'application' packages, say ByteString, Array, DataStructures, FFI... 'map' definitely should be defined here, even if its definition will have ghc-specific version 'base' package should provide common API to compiler libraries. for example, it should provide functions integerMult, arrayCreate, type IO and so on *hc-base packages should provide implementations of these functions, specific for concrete compiler, i.e. arrayCreate = arrayCreate# integerMult (S# a) (S# b) = intMult# a è ... newtype IO = IO (...) 'base' package should then analyze compiler name and version and import appropriate modules or define operations itself if there is no implementation: module ArrayOps where #if GHC import GHC.Arr #elseif Hugs import Hugs.Arr #else type Array a b = [(a,b)] arrayCreate = [] ... #endif all functions/types implemented in pure Haskell, all complex algorithms, all class definitions should go away from this package! it just provides common set of low-level operations and of no interest for end users. it's just a tool which provides "virtual Haskell compiler API", which allows to write all other libraries in rather portable way

...

last line: i have some experience of writing compiler-independent code with Haskell and C++ and believe that this plan is realistic The differences between Haskell compilers may well be bigger than those between C++ compilers!

I wish you the best of luck, and think this would be really nice to hvae - unfortunately I think its unobtainable - but if we could just get some of this goodness that would be fantastic!

you've missed one point - we _already_ have working solution, the 'base' library itself. all that we need is just to split it carefully to modules which may be independently upgraded, plus add compatibility with previous compiler version that 'base' currently lacks. so, it's more moving code around and careful planning task than a real technological challenge :) i think you just misunderstood me - i don't plan to make ultimate solution, just to solve some current meaningless problems - say, that we can't use old MArray interface with ghc 6.6 or new implementation of HashTable with ghc 6.2. cabal provided us with all the instruments required - all that is remain is to refactor base library to make it compiler-version-independent -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Hello Ian, Friday, September 15, 2006, 8:20:36 PM, you wrote:

...

what is a 'base' library now? it is the library that implements common set of operations for latest versions of ghc, hugs and nhc. it contains low-level implementation for ghc, but relies on separate hugsbase package for hugs (the same for nhc, afaiu). so, first step is obvious - separate ghc-base library from the rest. hugsbase, ghc-base and nhc-base packages should provide common set of low-level operations,

As it happens I was working on getting GHC to use cabal to build base et al on the plane the other day, and I had a brief look at this. Unfortunately there is a tangled web of dependencies, e.g. you need the low level Int# stuff in ghc-base, then Int in base, but then any other GHC-specific stuff can't use Int because it's in base. We could put everything into ghc-base and just re-export the common stuff in base, but then we can't share any code between ghc, hugs etc. I haven't looked in detail to see just how bad the problem is, but I agree it would be really good if we could split things up somehow so that base (or whatever base gets split into) is the same everywhere.

yes, it is one of problems that i was overlooked (and i expect that discussing my plan will show other problems i skipped by ignorance) first, let's specify that i propose (my today letter in haskell list contains more detailed plan): ghc-base should export Int operations. why? because it can't export Int# operations, they are not supported by other compilers (as the whole "unboxed type" concept), so they are useless to export. 'core' library should provide some common API. *hc-core libs should provide _subset_ of this API with hope that 'core' will emulate missing features but problem your mentioned still remains - while ghc-base defines operations on Int, it don't contains class Num definition, so that (*) or (+) operations can't be used. so that can we do? we should use intMul, intAdd and other operations directly. we can even define (*) and (+) operations for _internal_ use inside our ghc-base package, but not export them. while this seems a little Draconic, it will allow us to share Num defining code with other compilers and even introduce libraries with alternative Num/(*) definitions while idea of using some internal (*), (+) ... definitions may seem like work duplication, my experience says that it's much better to define duplicate operations for internal use only rather than try to implement whole Num class inside each compiler-specific library - because this definition should be a high-quality code and we don't want to copy such code over and over again and i hope that *hc-base libraries will not use Num operations too much because their main purpose is to give standard interface to compiler-specific functions, not to implement any algorithms. for example, looking to GHC.* modules in my own ArrayRef lib (which implements boxed and unboxed arrays), i don't see any arithmetic in _process_ of rewriting base lib, we should not have problems with GHC, because we can use recursive imports. but in order to retain compatibility with Hugs we may need to move Hugs.* modules inside 'base' package (the same for nhc). well, i don't know the best plan for intermediate versions. one possible but slow variant is to introduce intAdd/... operations, then rewrite ghc.*/hugs.*/... using these operations, then move out non-core stuff and then rewrite it back... -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Hello Ian, Friday, September 8, 2006, 5:52:57 AM, you wrote:

Both fixed, thanks.

Thanks to you - owing to your work this version Release Notes is much more useful for ghc users which don't track development process. now it contains exhaustive information about all ghc improvements together with links to "further reading" parts of manual i have several more suggestions: in GHC library paragraph - add a link to the API documentation. btw, my download (http://www.haskell.org/ghc/dist/current/dist/ghc-6.5.20060901-i386-unknown-m...) don't includes any library docs nor $(GHC)/doc/html/libraries/index.html can you please fix this? next paragraph - should point to the 7.4.10 section of user's guide instead of "Simon's letter" :) new RTS -V option is not mentioned here RTS -I option - i propose to document the fact that it controls _new_ GHC facility. smth like "GHC 6.6 added experimental feature that runs major GCs when program is idle more than 0.3 seconds, this feature is controlled by new RTS -I flag" "The -split-objs flag, which makes libraries smaller" is incorrect. It should be said "The -split-objs flag, which should be used when building libraries in order to shrink size of executables that links against these libraries". and documentation of this flag may be something like: "Tell the compiler to split the single object file that would normally be generated into multiple object files, one per top-level Haskell function or type in the module. This option may be used only when building libraries. It allows to make binaries build with these libraries smaller. On the other side, it greatly enlarges time of library compilation, so we suggest you to use this option only to finally compile your library and in .cabal files" back to Release Notes, libraries section - can you please include GHC library too, with some trivial comment like "First time included, version 1.0". The meaning of this is to have here _full_ list of all libraries included in 6.6. it will also appear in Table of contents. also, it will be great to use each library section heading as URL to the docs of corresponding library: 1.4.4.1 [[url://libraries/base/index.html base]] - version number 2.0 .... (i wrote it in Wiki style to demonstrate the idea) It will be great to include in each paragraph here links to the according module documentation (and, for comparison, documentation of 6.4 module). this should help users, making easier migration from 6.4 "GHC.Exts now provides a function lazy which forces GHC to think that its argument is lazy in its first argument. " - a copy-paste typo? lazy/inline function information can also give links to appropriate manual sections, 7.11.1/7.11.2 section 7.5 GADT of manual - can you please change section heading to "Generalised Algebraic Data Types (GADT)" - this simplifies search for GADT in table of contents; and include the following instead of "These and many other examples are given in papers by Hongwei Xi, and Tim Sheard":

You can find more information about GADTs at the http://haskell.org/haskellwiki/GADT In particular, article "Fun with phantom types" (http://www.informatik.uni-bonn.de/~ralf/publications/With.pdf) contains a number of examples of using GADTs (please note that it uses slightly different syntax comparing to actually implemented extension)

actually, i don't seen other papers which discuss GADTs _usage_ as opposite to implementation :( -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

The flags under section 4.17.26. Compiler debugging options seem to be out of date. They seem to be wrong for both 6.42 and 6.6 For example --ddump-cmm is not listed, but works. But --ddump-absC is listed, and gives the error unknown flag. I can't get -ddump-stix to work either. Gone, or renamed, or only works in combination with other flags?