November 2006 - Libraries

Standard (core) libraries initiative: rationale
by Bulat Ziganshin 28 Nov '06

28 Nov '06

Hello libraries, like "computer is a net", nowadays language is a library. there is nothing exceptional in C++ and Java languages except for their huge library codebase that makes them so widely appreciated while it's impossible for Haskell to have the same level of libraries maturity, we can try to do our best. Libraries was considered so important, that in H98 report libs required more pages than language itself. But, really, all libraries described there together is appropriate only for learning and small programs - to do real work, we need even much, much more fortunately, now we have large enough set of libs. moreover, this set grows each year. but these libs don't have official/recommended status. now we have two languages - H98 as reported with its bare libs, which is appropriate only for teaching, and real Haskell language with many extensions and rich set of libs, used to develop real programs with a language itself, now we go to standardize current practice and include into language definition all popular extensions. this will close the gap between standard and practice. Haskell' committee also plan to define new version of standard Haskell library. but what a library can be defined in this way? slightly extended version of standard Haskell98 lib? or, if it will be significantly extended - how much time this work will require and isn't that a duplication of work done at libraries list? i propose not to try to define reality, but accept existing one and join committee's work on new library definition with a current discussion of core libraries, which should define a set of libs available on any Haskell compiler on any platform - aren't goals the same? instead of providing rather small and meaningless standard Haskell library, now we can just include in Report docs existing and widely used libs, such as Network, mtl and so on. This will mean that language, defined in Haskell standard, can be used to write real programs, which will be guaranteed to run in any Haskell environment. of course, this mind game can't change anything in one moment. but it will change *accents* first, Haskell with its libraries will become language for a real work. such extended language isn't small nor easy to master in full, but it is normal for any mature programming environment. people learning Haskell should select in which area they need to specialize - be it gaming or web service development, and study appropriate subset of libs. people teaching Haskell now can show how *standard* Haskell may be used to solve real world problems, and this should change treatment of Haskell as academic language. also, we may expect that books teaching Haskell will start to teach on using standard libs, while their authors now don't consider teaching for non-standard libs second, by declaring these libs as standard ones we create sort of lingua franca, common language spoken by all Haskell users. for example, now there are about 10 serialization libs. by declaring one of them as standard, we will make choice simpler for most of users (who don't need very specific features) and allow them to speak in common language. in other words, number of Haskell libs is so large now that we should define some core subset in order to escape syndrome of Babel tower. defining core libraries set is just sharing knowledge that some libraries are more portable, easier to use, faster and so on, so they become more popular than alternatives in this area third. now we have Cabal that automates installation of any lib. next year we will got Hackage that automates downloading and checking dependencies. but these tools still can't replace a rich set of standard libs shipped with compiler. there are still many places and social situations where Internet downloading isn't available. Compiler can be sold on CD, transferred on USB stick. and separate Haskell libs probably will be not included here. Standard libraries bundled with compiler will ensure that at least this set of libs will be available for any haskell installation. Internet access shouldn't be a precondition for Haskell usage! :) fourth. now there is tendency to write ghc-specific libs. by defining requirements to the standard libs we may facilitate development of more portable, well documented and quick-checked ones. or may be some good enough libraries will be passed to society which will "polish" them in order to include in the set. anyway, i hope that *extensible* set of standard libraries with a published requirements to such libs would facilitate "polishing" of all Haskell libs just because ;) and this leads us to other question - whether this set and API of each library should be fixed in language standard or it can evolve during the time?... -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

4 3

Re: base libraries
by Simon Marlow 27 Nov '06

27 Nov '06

Bulat Ziganshin wrote: > libraries should be split into 4 rings: frozen, core, base and the rest > > * frozen libs are installed with haskell compiler and cannot be > upgraded using Cabal. it includes Cabal itself and libraries required > by Cabal, currently it's the Base library. compiler may provide its > own ways to upgrade these libs > > * core libs are tied closely to specific compiler and version. > therefore, these are also preinstalled but can be upgraded with Cabal. > in order to make upgrading not-a-headache, version of such library must > include compiler version, i.e. TH 6.6.0. for ghc these currently > includes TH and STM libs > > * base libs are *guaranteed* to be shipped with any haskell compiler > (compliant with this proposal). these libs are installed using Cabal > or OS package manager, only on poor Windows-like systems these should be > included in monolithic compiler distro. the reason is to allow user to > got latest, bug-fixed versions of these libraries for the installation > moment Please let's not change the terminology again. On the wiki page we use the term "Core libraries" for the set of libraries shipped or available on all the compilers. I don't see a good reason for naming any more sets of libraries, so can we just keep things simple? > haskell98, readline, unix/Win32, QuickCheck, fgl, haskell-src, html, > mtl, network, parsec, time, xhtml, > ByteString, regex-*, Edison, Filepath, MissingH, NewBinary, arrows, > HUnit, QuickCheck, monads So you want to have two variants of monad infrastructure, mtl and monads? How does the user decide which one to use? Two variants of HTML infrastructure? Why Edison rather than any other data structure framework? One particular non-portable line-editing library? Some of these choices seem entirely arbitrary. When there's an overlap and no clear choice, usually the right thing to do is let the user decide. At least until a concensus emerges. We have an effective mechanism for proposing library changes now, so I suggest that the Core libraries should start small, and then expand via proposals. That way we get to discuss additions one at a time, rather than throwing the kitchen sink in to begin with and having to argue about what to remove later (and upsetting users when we do remove things). This way We end up with a more consistent, well-design, whole. Cheers, Simon

2 1

Re: [Haskell-cafe] deepSeq vs rnf
by Cale Gibbard 27 Nov '06

27 Nov '06

On 22/10/06, Chad Scherrer <chad.scherrer(a)gmail.com> wrote: > Hi, > > I had posted this question a while back, but I think it was in the > middle of another discussion, and I never did get a reply. Do we > really need both Control.Parallel.Strategies.rnf and deepSeq? Should > we not always have > > x `deepSeq` y == rnf x `seq` y > ? > > Maybe there's a distinction I'm missing, but it seems to me they're > basically the same. I agree, they are the same. The Strategies library also gives much more general operations for working with strictness and parallelisation. That library seems to need more love, I think it's a great idea, but it doesn't really get noticed all that much. The Hierarchical libraries documentation for it is a little lacking -- it doesn't even provide a reference or link to the paper, and many of the combinators, as well as the general idea of how to use it are undocumented from there. It also spuriously contains an Assoc datatype, which if I recall correctly, was an example from the paper, but doesn't really belong in the library as far as I can tell. It would also be really nice to see the list of instances for the NFData class expanded to include other datatypes in the libraries, possibly also with compiler support for deriving, since it's mostly boilerplate. Speaking of boilerplate and the scrapping thereof, Data.Generics could theoretically also be used to write a relatively generic rnf/deepSeq, but in my attempts, it seems to be much much slower than using a specific normal form class. Here's my code from quite a while back. As I recall, it's semantically correct, but ran about an order of magnitude slower. There might be a much better way to do it, I don't really know Data.Generics all that well. rnf :: (Data a) => a -> () rnf x = everything (\x y -> x `seq` y) (\x -> x `seq` ()) x deepSeq x y = rnf x `seq` y f $!! x = rnf x `seq` f x - Cale

6 9

Plan for file processing libs
by Bulat Ziganshin 26 Nov '06

26 Nov '06

Hello libraries, i think that ultimate structure for file-related libs should be the following, in the dependence order: * ByteString support * UnicodeByteString support * Stringable class that supports the common interface to String, UTF8ByteString, UTF16ByteString and so on * FilePath operations implemented via Stringable class * FileSystem operations (readdir, stat, copy, delete, mkdir...) implemented for Stringable arguments using FilePath module * FD operations which supports manipulation of file contents via unix-style file descriptors (read/write/seek) where 'open' operation should support any Stringable parameter * Streams-like layer which provides rich set of file manipulation operations in native Haskell way and use FD operations for its file support * NewBinary-like layer which provides binary I/O and serialization on top of Streams Neil, as one variant of placing FilePath module we may join filepath, filesystem and FD levels into the one library, say named Files. currently, we can include here: * FilePath module * copies of System.Directory, System.Directory.Internals, System.Posix.Internals modules which is now *unsupported* part of Base lib * System.FD and System.MMapFile modules from my Streams lib so, this lib will provide a rich set of file-related operations - from parsing filenames to low-level access to file contents, allowing to put more sophisticated I/O libs on top of it the reason to put high-level I/O lib in separate package is that there are plenty of different design choices possible (and exists). on the other hand, all operations provided by such Files library are low-level ones, there are no design decisions here -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

4 5

Re: [Haskell] Re[2]: base libraries
by Ross Paterson 26 Nov '06

26 Nov '06

On Fri, Nov 24, 2006 at 05:42:00PM +0300, Bulat Ziganshin wrote: > * base libs are *guaranteed* to be shipped with any haskell compiler > (compliant with this proposal). [...] > these libs should include rich set of functionality. criteria of > inclusion library in this set are: widely used, small and portable. > > Portability: these libs should work on windows and major unix > branches, on ghc/hugs/nhc and other proposal-compliant compilers, > otherwise we can't guarantee availability everywhere. > [...] > haskell98, readline, unix/Win32, QuickCheck, fgl, haskell-src, html, > mtl, network, parsec, time, xhtml, > ByteString, regex-*, Edison, Filepath, MissingH, NewBinary, arrows, > HUnit, QuickCheck, monads Your portability criterion would rule out fgl, mtl, regex-*, Edison, NewBinary, arrows and monads.

3 3

Proposal: Add System.FilePath to base
by Neil Mitchell 24 Nov '06

24 Nov '06

Hi, PROPOSAL I guess most people saw this coming, but here is the proposal to add System.FilePath to base. The System.FilePath module is located at http://www-users.cs.york.ac.uk/~ndm/projects/libraries.php#filepath In particular I propose the addition of System.FilePath (a cross-platform FilePath manipulation library), System.FilePath.Windows (Windows paths on all OS's) and System.FilePath.Posix (Posix paths on all OS's). The Version_* stuff is not being proposed for addition. Following the library submission guidelines: * Currency, I do not include a darcs patch, that can be done later - this is a stand alone module so currency is not a necessary criteria. The current module does not use CPP in a particularly nice way (so as to leave the primary module selecting its OS at runtime). The code that would actually be committed will be functionally identical, but slightly faster. If people agree to everything else I'm happy to wave a patch past people before the final committing. * Portability, Windows+Linux, GHC+Hugs+Yhc * Style, not applicable to adding a new module. The module is clean and well implemented, by most style guidelines. * Documentation, detailed - see the haddock link above * Tests, very thorough, over 200 tests/properties DISCUSSION Since the people who hang out on the libraries list may not include all the people who have previously commented on the FilePath module in previous versions, I have CC'd some of these people (particularly those with a strong opinion), so as not to exclude anyone from the discussion. If these people want to continue to discuss System.FilePath they should probably join the libraries list for the time being. Since this module has had extensive comments on it previously, I have attempted to summarise some of the reasons that people have given against the inclusion of these modules, along with the basic counter arguments. This is just an attempt to give some background to people new to the discussion - feel free to expand on any point, or raise new points etc - discussion welcome. 1 type FilePath = String is a bad design choice, which should not be encouraged * This is the way it is, Haskell 98 says so * This library allows better structuring of FilePath's, so makes the interface more abstract * This library makes a good attempt to be correct in it's transformations * We now have explicit properties for FilePath interaction - this isn't string hackery * Paves the way for an ADT, in the future (if desired) * Reverse compatible 2 The base libraries are not the place for a FilePath library * Setup.hs scripts will need it, and they only depend on base and Cabal * Could remove the version from Cabal * Encourages programmers to use it * FilePath functions are (very) easy to write wrongly * FilePath handling should have been present from the begining * It's already there internally, see System.Directory.Internals 3 The base libraries are being split up * This is something happening in the future * Put FilePath in, where it currently belongs, move it later * The FilePath library belongs with type FilePath, which belongs in base * If we ever move to an ADT, this library will have to be in base 4 The interface in this library is poorly designed * Repeated iterations in public have lead to a reasonable concensus * Clear and concrete properties given for all operations * If particular problems still remain there is some flexibility 5 Why not type FilePath = ByteString * If you argue for this, and for 1, you're not internally consistent! * It's not reverse compatible * This should be tackled as a whole thing, i.e. String defaulting * Not lazy I'll open a trac bug shortly. I suspect that there will be plenty of discussion, so I'd rather not set any time limit until everyone has had a chance to express their full opinion. However if we could try and wrap this up before Christmas that would be nice :) Thanks Neil

8 22

Library versioning policy
by Bulat Ziganshin 24 Nov '06

24 Nov '06

Hello haskell, We all know what is a "DLL hell" - if some program written against version 2.0 of library, then using version 1.0 or 3.0 of the same library when program compiled may call all sorts of devil there is a great solution of this problem - Eternal Compatibility Theory. unfortunately, it's so great that seldom used on practice because it requires significant efforts from library developer i propose another solution for this problem which don't require to write anything and just impose some policies both for library developer and user library developer should just explicitly declare and obey some policy of incrementing library versions which will allow library user to select proper range of versions which are guaranteed to work with his code. i propose the following as standard policy: * major version change (2.2->3.0) means total incompatibility. all sorts of changes may come * middle version change (2.1.3->2.2) means addition of new facilities (functions, constructors, classes...). in most cases, such semi-compatible version can be used if that's done with caution * minor change (2.1.1->2.1.2) means only internal changes and bugfixes other policies may be used as well, library developer should just declare his policy at the library birth and use it consistently this will allow library users to declare in their cabal files range of library versions they are accepted and be sure that this range will include latest bugfixes but don't include any incompatible changes. then Cabal (which now can handle many versions of the same lib) should select latest acceptable version of library requested and make it available for compilation say, if i wrote my program using Streams 1.2.1, i can specify Build-Depends: Streams 1.2.* or, if i brave/accurate enough: Build-Depends: Streams 1.* and Cabal should select the best library available at build time -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

3 3

Re: [Hugs] #49: LDFLAGS and hugs packages on OS X 10.4
by Ross Paterson 24 Nov '06

24 Nov '06

On Fri, Nov 24, 2006 at 10:30:16AM -0000, Hugs wrote: > #49: LDFLAGS and hugs packages on OS X 10.4 > The value of the environnment variable LDFLAGS is not used to compile > hug's packages. Consequently, make fails to build X11 package (on OS X > 10.4.8) > [...] > My LDFLAGS variable whose value is "-L/usr/X11R6/lib -L/Users/julien/local/lib" has not been used. > Simply add -L/usr/X11R6/lib to the gcc command and it compile fine. This is really an issue between Cabal and the packages. I presume it's expected that LDFLAGS will be used because configure uses it. So should Cabal use LDFLAGS, or should packages that use autoconf add LDFLAGS to ld-options?

1 0

portability of Data.ByteString.Lazy
by Ross Paterson 24 Nov '06

24 Nov '06

Data.ByteString.Lazy defines ByteString as a type synonym, and then uses that in instances, which isn't permitted by Haskell 98. How about defining a newtype ByteString in Data.ByteString.Lazy.Base instead?

5 11

ANNOUNCE: PQC: QuickCheck in the Age of Concurrency
by dons＠cse.unsw.edu.au 24 Nov '06

24 Nov '06

------------------------------------------------------------------------ PQC: QuickCheck in the Age of Concurrency An SMP parallel QuickCheck driver ------------------------------------------------------------------------ Do you: * Have (or want) lots of QuickCheck properties? * Run them often (maybe on every darcs commit)? * Tired of waiting for the testsuite to finish? * Got a multi-core box with cpus sitting idle...? Yes? You need Parallel QuickCheck! PQC provides a single module: Test.QuickCheck.Parallel. This is a QuickCheck driver that runs property lists as jobs in parallel, and will utilise as many cores as you wish, with the SMP parallel GHC 6.6 runtime. It is simple, scalable replacement for Test.QuickCheck.Batch. An example, on a 4 cpu linux server, running 20 quickcheck properties. With 1 thread only: $ time ./a.out 1 1: sort1 : OK, 1000 tests. 1: sort2 : OK, 1000 tests. 1: sort3 : OK, 1000 tests. 1: sort4 : OK, 1000 tests. ... ./a.out 1 > x 18.94s user 0.01s system 99% cpu 18.963 total 18 seconds, 99% cpu. But I've got another 3 2.80GHz processors sitting idle! Let's use them, to run the testsuite faster. No recompilation required. 4 OS threads, 4 Haskell threads: $ time ./a.out 4 +RTS -N4 > /dev/null ./a.out 4 +RTS -N4 > /dev/null 20.65s user 0.22s system 283% cpu 7.349 total 283% cpu, not bad. We're getting close to being limited by the length of the longest running test. Or on a dual core macbook, thanks to Spencer Janssen for macbook data and testing: 1 thread: ./Example 1 17.256s 2 threads: ./Example 2 +RTS -N2 10.402s Get it! Homepage: http://www.cse.unsw.edu.au/~dons/pqc.html Haddocks: http://www.cse.unsw.edu.au/~dons/pqc/ Example : http://www.cse.unsw.edu.au/~dons/code/pqc/examples/Example.hs darcs get http://www.cse.unsw.edu.au/~dons/code/pqc Happy, quick checking, Don

1 0