July 2006 - Libraries

Tracking Changes
by Dominic Steinitz 16 Jul '06

16 Jul '06

I seem to be getting an increasing number of patches for the crypto library - so thanks to all the contributors. However, I have no way of tracking whether I've actually actioned things. This is probably frustrating to contributors if I accidentally forget their patch and they are too polite to remind me. I've noticed that ghc and cabal both use trac. Is this somehow available for other libraries? What would I have to do to set something up for the crypto library (I'm assuming I've correctly understood that trac will track change requests / patch contributions). Thanks, Dominic.

1 0

Re[2]: FPS again
by Bulat Ziganshin 16 Jul '06

16 Jul '06

Hello Donald, can you test that this implementation lines = split 0x0a is as fast as existing (long) ones both for Lazy and Strict ByteString? also, is not it faster to use the following implementation: isSpaceWord8 = (spacesFlagsArray!) ? also, i propose to move getLine/getContents/putStr/interact/readFile-type functions into .Char8 modules (both for strict and lazy bytestrings), because these functions are encoding-dependent and work with texts (as opposite to hGet/hPut which works with raw binary data blocks). in particular, i tried to implement Lazy.hGetLines as 'hGetContents >>= lines' but it was impossible because 'lines' function is defined only in Lazy.Char8 module i send you a bunch of small patches that fixes I/O part of library, providing the same set of operations for lazy and strict bytestrings, for ghc and non-ghc platforms also, i run into small problems using FPS repository to development (seems that i'm first windows developer of the lib). First, i propose to change darcs 'prefs' file to the following: test cd tests && make fast - it should work both on unix and windows second, i've changed 'time' calls in tests/Makefile to use my own 't' utility instead of 'time'. but of course it's not universal solution. at least, 'time' in windows shell (cmd.exe) is _built-in_ utility that don't have anything common with unix 'time' :) -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

2 9

Building Data.Time on WinXP GHC6.4.1
by Alistair Bayley 14 Jul '06

14 Jul '06

Has anyone built Ashley's Data.Time package on Windows? I ask because timestuff.c fails to compile for me because it refers to fields (tm_zone, tm_gmtoff) in struct tm which don't seem to exist on Windows (at least, mingw). Alistair

2 1

FPS: useAsCString variants
by Bulat Ziganshin 14 Jul '06

14 Jul '06

Hello Donald, Friday, July 14, 2006, 8:11:22 AM, you wrote: >> afaik, CStringLen don't require zero byte at the end of string. So the >> following implementation will be much faster and still correct (not >> tested, it's just sketch): > Well spotted! I've applied this patch, with a tiny tweak (you missed a > withForeignPtr in useAsCStringLen). thanks. after writing this letter i spotted unsafeUseAsCStringLen and used it. now i'm curious about having these 4 functions: useAsCString useAsCStringLen unsafeUseAsCString unsafeUseAsCStringLen and differences between their behavior. may be, it will be better to keep only first two functions, with semantics already implemented (with my patch) - `useAsCString` add zero byte while useAsCStringLen doesn't ? i copy this letter to libraries list in hope that other FPS developers may clear the question -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

2 3

ANN: Takusen 0.5
by Alistair Bayley 14 Jul '06

14 Jul '06

Oleg and I are pleased to announce the release of a new version of Takusen (it's been a while; so long that we don't remember the last version number we used). The most significant code change is a new internal design (courtesy of Oleg) which gives better separation of concerns like statement preparation, binding, and result-set processing. Also of note is a conversion to darcs, and hosting at haskell.org (thanks to Simon Marlow): darcs get http://darcs.haskell.org/takusen ... which also has the nice property that the Haddock documentation is browseable online: http://darcs.haskell.org/takusen/doc/html/index.html A comprehensive description of API usage can be found in the documentation for module Database.Enumerator: http://darcs.haskell.org/takusen/doc/html/Database-Enumerator.html We hope that the use of darcs will encourage contributions... Other changes: - support for multiple result sets returned from functions (PostgreSQL only). Oracle support for this is in the pipeline, which will allow use of nested cursors, and processing RefCursors returned as output parameters from procedure calls. Future plans: - Cabalisation - use of Data.Time instead of System.Time - ODBC and MS Sql Server backends For those of you unfamiliar with Takusen, here is our HCAR blurb: Takusen is a library for accessing DBMS's. It is a low-level library like HSQL, in the sense that it is used to issue SQL statements. Takusen's "unique-selling-point" is a design for processing query results using a left-fold enumerator. For queries the user creates an iteratee function, which is fed rows one-at-a-time from the result-set. We also support processing query results using a cursor interface, if you require finer-grained control. Currently we fully support Oracle, Sqlite, and PostgreSQL. Alistair

1 0

Streams 0.2.1 beta released
by Bulat Ziganshin 12 Jul '06

12 Jul '06

Hello libraries, I've updated http://www.haskell.org/library/StreamsBeta.tar.gz with the following: * fixed problem with Unix compilation * fixed bug with writing after vSeek * added support for memory-mapped files on Unix * added support for files >4GB on Windows * added/improved test programs MMFile.hs, wcMMFile.hs, FD5gb.hs, DeriveBinary.hs Look at http://article.gmane.org/gmane.comp.lang.haskell.libraries/4820 for installation/usage instructions Using memory-mapped files provides slight speed improvement because it excludes the need to move data between OS buffer and application buffer. Moreover, this may scale better in multi-threading environments without need to implement special async I/O, although i don't checked this assumption. I thanks Adam Langley who's provided initial mmaped files implementation and tested all the changes. Moreover, he tested the lib on 1.66Ghz Intel Core Duo box and got the results stating that Strings I/O speed is at range of 30-100 mb/sec, while with Chars/ByteStrings/buffers I/O speed is at range of 100-400 mb/sec. It's the full test results. Each line shows the time of reading or writing 100 mb of data using various Stream types, I/O routines and line/buffer sizes (20 or 80 bytes): Unchecked MemBuf: vPutBuf: 0.093 secs (user: 0.090 secs) vGetBuf: 0.035 secs (user: 0.030 secs) vPutChar: 0.708 secs (user: 0.700 secs) vGetChar: 0.346 secs (user: 0.340 secs) vPutBuf20: 0.191 secs (user: 0.190 secs) vGetBuf20: 0.191 secs (user: 0.190 secs) vPutByteStrLn20: 0.448 secs (user: 0.450 secs) vGetByteLine20 : 0.632 secs (user: 0.630 secs) vPutStrLn20: 1.229 secs (user: 1.230 secs) vGetLine20 : 1.387 secs (user: 1.380 secs) vGetByteLines20: 0.725 secs (user: 0.720 secs) vPutByteStrLn80: 0.139 secs (user: 0.130 secs) vGetByteLine80 : 0.253 secs (user: 0.250 secs) vPutStrLn80: 1.066 secs (user: 1.070 secs) vGetLine80 : 1.251 secs (user: 1.250 secs) vGetByteLines80: 0.273 secs (user: 0.270 secs) vGetContents: 2.767 secs (user: 2.770 secs) MemBuf: vPutBuf: 0.094 secs (user: 0.090 secs) vGetBuf: 0.036 secs (user: 0.030 secs) vPutChar: 0.567 secs (user: 0.560 secs) vGetChar: 0.406 secs (user: 0.410 secs) vPutBuf20: 0.221 secs (user: 0.220 secs) vGetBuf20: 0.227 secs (user: 0.230 secs) vPutByteStrLn20: 0.460 secs (user: 0.460 secs) vGetByteLine20 : 0.667 secs (user: 0.670 secs) vPutStrLn20: 1.249 secs (user: 1.250 secs) vGetLine20 : 1.589 secs (user: 1.590 secs) vGetByteLines20: 0.759 secs (user: 0.750 secs) vPutByteStrLn80: 0.142 secs (user: 0.150 secs) vGetByteLine80 : 0.253 secs (user: 0.260 secs) vPutStrLn80: 1.066 secs (user: 1.070 secs) vGetLine80 : 1.348 secs (user: 1.350 secs) vGetByteLines80: 0.284 secs (user: 0.280 secs) vGetContents: 2.841 secs (user: 2.840 secs) FD: vPutBuf: 0.338 secs (user: 0.250 secs) vGetBuf: 0.091 secs (user: 0.100 secs) vPutChar: 0.680 secs (user: 0.680 secs) vGetChar: 0.499 secs (user: 0.500 secs) vPutBuf20: 0.354 secs (user: 0.340 secs) vGetBuf20: 0.317 secs (user: 0.310 secs) vPutByteStrLn20: 3.482 secs (user: 0.560 secs) vGetByteLine20 : 0.788 secs (user: 0.790 secs) vPutStrLn20: 1.360 secs (user: 1.340 secs) vGetLine20 : 1.677 secs (user: 1.670 secs) vGetByteLines20: 0.866 secs (user: 0.860 secs) vPutByteStrLn80: 0.251 secs (user: 0.250 secs) vGetByteLine80 : 0.347 secs (user: 0.350 secs) vPutStrLn80: 1.176 secs (user: 1.180 secs) vGetLine80 : 1.435 secs (user: 1.440 secs) vGetByteLines80: 0.374 secs (user: 0.370 secs) vGetContents: 2.939 secs (user: 2.940 secs) FD utf8: vPutBuf: 0.118 secs (user: 0.100 secs) vGetBuf: 0.087 secs (user: 0.090 secs) vPutChar: 3.931 secs (user: 3.930 secs) vGetChar: 3.282 secs (user: 3.280 secs) vPutBuf20: 0.330 secs (user: 0.330 secs) vGetBuf20: 0.308 secs (user: 0.300 secs) vPutByteStrLn20: 0.564 secs (user: 0.560 secs) vGetByteLine20 : 0.771 secs (user: 0.770 secs) vPutStrLn20: 6.104 secs (user: 3.950 secs) vGetLine20 : 4.285 secs (user: 4.280 secs) vGetByteLines20: 0.861 secs (user: 0.860 secs) vPutByteStrLn80: 0.255 secs (user: 0.250 secs) vGetByteLine80 : 0.351 secs (user: 0.360 secs) vPutStrLn80: 3.700 secs (user: 3.690 secs) vGetLine80 : 4.037 secs (user: 4.040 secs) vGetByteLines80: 0.373 secs (user: 0.370 secs) vGetContents: 5.337 secs (user: 5.340 secs) FD locked: vPutBuf: 0.114 secs (user: 0.110 secs) vGetBuf: 0.087 secs (user: 0.080 secs) vPutChar: 8.292 secs (user: 8.300 secs) vGetChar: 8.626 secs (user: 8.630 secs) vPutBuf20: 0.545 secs (user: 0.550 secs) vGetBuf20: 0.519 secs (user: 0.520 secs) vPutByteStrLn20: 0.877 secs (user: 0.870 secs) vGetByteLine20 : 1.022 secs (user: 1.020 secs) vPutStrLn20: 3.127 secs (user: 1.540 secs) vGetLine20 : 1.945 secs (user: 1.940 secs) vGetByteLines20: 0.868 secs (user: 0.870 secs) vPutByteStrLn80: 0.333 secs (user: 0.300 secs) vGetByteLine80 : 0.398 secs (user: 0.400 secs) vPutStrLn80: 1.201 secs (user: 1.200 secs) vGetLine80 : 1.490 secs (user: 1.490 secs) vGetByteLines80: 0.379 secs (user: 0.370 secs) vGetContents: 2.940 secs (user: 2.940 secs) Handle: vPutBuf: 0.117 secs (user: 0.120 secs) vGetBuf: 0.107 secs (user: 0.110 secs) vPutChar: 32.172 secs (user: 32.170 secs) vGetChar: 19.427 secs (user: 19.420 secs) vPutBuf20: 1.283 secs (user: 1.280 secs) vGetBuf20: 1.236 secs (user: 1.240 secs) vPutByteStrLn20: 3.283 secs (user: 1.900 secs) vGetByteLine20 : 1.867 secs (user: 1.860 secs) vPutStrLn20: 6.252 secs (user: 5.580 secs) vGetLine20 : 2.501 secs (user: 2.500 secs) vGetByteLines20: 2.064 secs (user: 2.060 secs) vPutByteStrLn80: 0.598 secs (user: 0.600 secs) vGetByteLine80 : 0.950 secs (user: 0.950 secs) vPutStrLn80: 2.348 secs (user: 2.350 secs) vGetLine80 : 1.741 secs (user: 1.740 secs) vGetByteLines80: 0.995 secs (user: 0.990 secs) vGetContents: 12.351 secs (user: 12.350 secs) Handle buffered: vPutBuf: 0.117 secs (user: 0.110 secs) vGetBuf: 0.087 secs (user: 0.090 secs) vPutChar: 0.668 secs (user: 0.670 secs) vGetChar: 0.488 secs (user: 0.490 secs) vPutBuf20: 0.330 secs (user: 0.330 secs) vGetBuf20: 0.308 secs (user: 0.300 secs) vPutByteStrLn20: 0.561 secs (user: 0.560 secs) vGetByteLine20 : 0.772 secs (user: 0.780 secs) vPutStrLn20: 3.982 secs (user: 1.370 secs) vGetLine20 : 1.722 secs (user: 1.720 secs) vGetByteLines20: 0.863 secs (user: 0.860 secs) vPutByteStrLn80: 0.255 secs (user: 0.260 secs) vGetByteLine80 : 0.352 secs (user: 0.350 secs) vPutStrLn80: 1.177 secs (user: 1.180 secs) vGetLine80 : 1.502 secs (user: 1.500 secs) vGetByteLines80: 0.378 secs (user: 0.380 secs) vGetContents: 2.929 secs (user: 2.910 secs) Handle buffered+locked: vPutBuf: 0.116 secs (user: 0.110 secs) vGetBuf: 0.087 secs (user: 0.090 secs) vPutChar: 8.393 secs (user: 8.390 secs) vGetChar: 8.118 secs (user: 8.120 secs) vPutBuf20: 0.552 secs (user: 0.550 secs) vGetBuf20: 0.519 secs (user: 0.520 secs) vPutByteStrLn20: 0.782 secs (user: 0.780 secs) vGetByteLine20 : 0.975 secs (user: 0.980 secs) vPutStrLn20: 3.392 secs (user: 1.600 secs) vGetLine20 : 1.933 secs (user: 1.920 secs) vGetByteLines20: 0.875 secs (user: 0.870 secs) vPutByteStrLn80: 0.328 secs (user: 0.310 secs) vGetByteLine80 : 0.401 secs (user: 0.400 secs) vPutStrLn80: 1.204 secs (user: 1.210 secs) vGetLine80 : 1.546 secs (user: 1.530 secs) vGetByteLines80: 0.379 secs (user: 0.380 secs) vGetContents: 2.911 secs (user: 2.910 secs) MMFile: vPutBuf: 0.108 secs (user: 0.110 secs) vGetBuf: 0.107 secs (user: 0.100 secs) vPutChar: 0.592 secs (user: 0.590 secs) vGetChar: 0.427 secs (user: 0.430 secs) vPutBuf20: 0.247 secs (user: 0.250 secs) vGetBuf20: 0.248 secs (user: 0.250 secs) vPutByteStrLn20: 0.496 secs (user: 0.500 secs) vGetByteLine20 : 0.687 secs (user: 0.690 secs) vPutStrLn20: 1.282 secs (user: 1.280 secs) vGetLine20 : 1.578 secs (user: 1.570 secs) vGetByteLines20: 0.785 secs (user: 0.790 secs) vPutByteStrLn80: 0.170 secs (user: 0.170 secs) vGetByteLine80 : 0.279 secs (user: 0.280 secs) vPutStrLn80: 1.097 secs (user: 1.090 secs) vGetLine80 : 1.351 secs (user: 1.350 secs) vGetByteLines80: 0.310 secs (user: 0.310 secs) vGetContents: 2.843 secs (user: 2.840 secs) -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

2 3

Packages and .NET assemblies
by Simon Marlow 12 Jul '06

12 Jul '06

I just took a deeper look at how .NET handles packages, since they have thought through many of the same issues and have a scheme that scales well. There's a great deal of similarity with the schemes we're thinking about for Haskell. The executive summary: - A .NET assembly fulfills the same function as a Haskell package: an assembly is the unit of versioning and distribution, and carries with it a bunch of metadata (author, copyright, etc.). - An assembly is uniquely identified by a triple (name,version,culture). - An assembly travels in a single file (.DLL) - An assembly may define multiple namespaces (like A.B.C), and entities within namespaces. Much like modules in Haskell. - Assemblies are not named directly in source code. C# code (for example) refers to namespaces and entities only. - When compiling C# code, you name the DLLs that contain assemblies to bring into scope (in fact, the transitive closure) using command-line options. - The C# compiler brings a set of assemblies into scope by default (== exposed packages in GHC). - The binary has baked into it the exact identity of the assemblies it was compiled against. - At runtime, the binary might be rebound to different versions of the assemblies it depends on, via a complicated structure of configuration files. So there's no grafting, or even assembly-qualified imports. How do they get away with this? Well, one reason is that namespaces are generally globally unique, because the convention is to use names beginning with an organisation (eg. Microsoft.System). So when you bring a bunch of assemblies into scope, their namespaces generally don't conflict. Interestingly, the assembly story is almost identical to what GHC supports right now, and when we change module identities to include the package name (which I'm working on now), it will be even closer. Not that I necessarily think we can get away with doing nothing to the language, but at lesat this seems to argue for being conservative. Cheers, Simon

2 1

updates to HaXml (1.13.1 and 1.16)
by Malcolm Wallace 12 Jul '06

12 Jul '06

For those who use the current stable version of HaXml, I'd like to announce a new patch-level release, 1.13.1, which contains the following bugfixes: * permit percent character in attribute values * parse unquoted attribute values starting '+' or '#' in HTML * keep the original DTD in the output of 'processXmlWith' See http://www.haskell.org/HaXml/ for downloads. For those living on the development edge, I'd like to report that the current darcs version darcs get http://www.cs.york.ac.uk/fp/darcs/HaXml contains a new set of parser combinators (with the same API as before) that is lazier, whilst still allowing backtracking. By lazy, I mean it can start to return partial values as soon as it has consumed e.g. the start tag of an element, without waiting to check that the close tag matches. This has two good effects: * your program will run faster * it will consume less memory and two bad effects: * if there are errors in the document, they will throw an exception in the middle of your processing * the error message in the exception may be rather less accurate about the cause and location than previously. The older XML parser has also been retained, since the lazy version is still experimental. To use the new one, import Text.XML.HaXml.ParseLazy There are also lazy versions of the usual demo programs CanonicaliseLazy XtractLazy As an example of the improved speed, a query to extract all the <key> tags from a 3.7Mb XML document: Xtract "//key" file.xml did not give any results after more than ten minutes on my machine, but XtractLazy "//key" file.xml started producing results immediately, and completed the task in 25 seconds (returning 52584 tags). Separate website and downloads at http://www.cs.york.ac.uk/fp/HaXml-devel Regards, Malcolm

3 4

Re: Packages and modules
by Malcolm Wallace 11 Jul '06

11 Jul '06

John Meacham <john(a)repetae.net> wrote: > Package names should never appear in source files IMHO. if a package > name is in the source file, then you might as well make it part of the > module name. Packages exist for 'meta-organization' of code. A way to > deal with mapping code _outside_ of the language itself, putting > packages inside the code will force the creation of yet another level, > meta-packages, or something. packages should not be a language issue, > they are an environment one. I tend to the opposite view. The meaning of the code should be expressed in the code itself. If a module M imports A.B.C, and I can see two such modules called A.B.C, then the meaning of the code is ambiguous and ill-defined. I would rather not have to look elsewhere (in the build system? Makefile? scons? Cabal file? DOS batch file? where?) to find out how to resolve the ambiguity. Surely the programmer knew which import was intended. Is it so difficult to communicate that information somewhere close to the import itself? This is the same situation as with compiler flags for language extensions, foreign imports, and so on. If a module uses a non-standard language feature that changes the interpretation of the syntax (e.g. template haskell), it should say so in the module itself {-# OPTIONS_GHC -fth #-}, so that I as a human reader know what is going on. Likewise, if a foreign import requires a C header file, that should also be referred to in the FFI decl, not solely as a build option at some arbitrary distance in the filesystem. Regards, Malcolm

4 18

Re: Why is there no splitBy in the list module?
by Christian Maeder 11 Jul '06

11 Jul '06

Simon Marlow schrieb: > Donald Bruce Stewart wrote: > >> Hacking up your own custom split (or a tokens/splitOnGlue) must be one >> of the most common questions from beginners on the irc channel. >> >> Anyone rememeber what the result of the "let's get split into the base >> library" movement's work was? > > ISTR there wasn't a concensus, so nothing happened. Which is silly, > really - I agree we should definitely have a Data.List.split. > Maybe someone can extract a result from the old discussion about "Prelude function suggestions" http://thread.gmane.org/gmane.comp.lang.haskell.libraries/1684/focus=1684 Cheers Christian Our current (special) version is: {- | A function inspired by the perl function split. A list is splitted on a seperator element in smaller non-empty lists. The seperator element is dropped from the resulting list. -} splitOn :: Eq a => a -- ^ seperator -> [a] -- ^ list to split -> [[a]] splitOn x xs = let (l, r) = break (==x) xs in (if null l then [] else [l]) ++ (if null r then [] else splitOn x $ tail r)

3 3