- Libraries - Haskell.org

FPS: useAsCString variants
by Bulat Ziganshin 14 Jul '06

14 Jul '06

Hello Donald, Friday, July 14, 2006, 8:11:22 AM, you wrote: >> afaik, CStringLen don't require zero byte at the end of string. So the >> following implementation will be much faster and still correct (not >> tested, it's just sketch): > Well spotted! I've applied this patch, with a tiny tweak (you missed a > withForeignPtr in useAsCStringLen). thanks. after writing this letter i spotted unsafeUseAsCStringLen and used it. now i'm curious about having these 4 functions: useAsCString useAsCStringLen unsafeUseAsCString unsafeUseAsCStringLen and differences between their behavior. may be, it will be better to keep only first two functions, with semantics already implemented (with my patch) - `useAsCString` add zero byte while useAsCStringLen doesn't ? i copy this letter to libraries list in hope that other FPS developers may clear the question -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

2 3

ANN: Takusen 0.5
by Alistair Bayley 14 Jul '06

14 Jul '06

Oleg and I are pleased to announce the release of a new version of Takusen (it's been a while; so long that we don't remember the last version number we used). The most significant code change is a new internal design (courtesy of Oleg) which gives better separation of concerns like statement preparation, binding, and result-set processing. Also of note is a conversion to darcs, and hosting at haskell.org (thanks to Simon Marlow): darcs get http://darcs.haskell.org/takusen ... which also has the nice property that the Haddock documentation is browseable online: http://darcs.haskell.org/takusen/doc/html/index.html A comprehensive description of API usage can be found in the documentation for module Database.Enumerator: http://darcs.haskell.org/takusen/doc/html/Database-Enumerator.html We hope that the use of darcs will encourage contributions... Other changes: - support for multiple result sets returned from functions (PostgreSQL only). Oracle support for this is in the pipeline, which will allow use of nested cursors, and processing RefCursors returned as output parameters from procedure calls. Future plans: - Cabalisation - use of Data.Time instead of System.Time - ODBC and MS Sql Server backends For those of you unfamiliar with Takusen, here is our HCAR blurb: Takusen is a library for accessing DBMS's. It is a low-level library like HSQL, in the sense that it is used to issue SQL statements. Takusen's "unique-selling-point" is a design for processing query results using a left-fold enumerator. For queries the user creates an iteratee function, which is fed rows one-at-a-time from the result-set. We also support processing query results using a cursor interface, if you require finer-grained control. Currently we fully support Oracle, Sqlite, and PostgreSQL. Alistair

1 0

Streams 0.2.1 beta released
by Bulat Ziganshin 12 Jul '06

12 Jul '06

Hello libraries, I've updated http://www.haskell.org/library/StreamsBeta.tar.gz with the following: * fixed problem with Unix compilation * fixed bug with writing after vSeek * added support for memory-mapped files on Unix * added support for files >4GB on Windows * added/improved test programs MMFile.hs, wcMMFile.hs, FD5gb.hs, DeriveBinary.hs Look at http://article.gmane.org/gmane.comp.lang.haskell.libraries/4820 for installation/usage instructions Using memory-mapped files provides slight speed improvement because it excludes the need to move data between OS buffer and application buffer. Moreover, this may scale better in multi-threading environments without need to implement special async I/O, although i don't checked this assumption. I thanks Adam Langley who's provided initial mmaped files implementation and tested all the changes. Moreover, he tested the lib on 1.66Ghz Intel Core Duo box and got the results stating that Strings I/O speed is at range of 30-100 mb/sec, while with Chars/ByteStrings/buffers I/O speed is at range of 100-400 mb/sec. It's the full test results. Each line shows the time of reading or writing 100 mb of data using various Stream types, I/O routines and line/buffer sizes (20 or 80 bytes): Unchecked MemBuf: vPutBuf: 0.093 secs (user: 0.090 secs) vGetBuf: 0.035 secs (user: 0.030 secs) vPutChar: 0.708 secs (user: 0.700 secs) vGetChar: 0.346 secs (user: 0.340 secs) vPutBuf20: 0.191 secs (user: 0.190 secs) vGetBuf20: 0.191 secs (user: 0.190 secs) vPutByteStrLn20: 0.448 secs (user: 0.450 secs) vGetByteLine20 : 0.632 secs (user: 0.630 secs) vPutStrLn20: 1.229 secs (user: 1.230 secs) vGetLine20 : 1.387 secs (user: 1.380 secs) vGetByteLines20: 0.725 secs (user: 0.720 secs) vPutByteStrLn80: 0.139 secs (user: 0.130 secs) vGetByteLine80 : 0.253 secs (user: 0.250 secs) vPutStrLn80: 1.066 secs (user: 1.070 secs) vGetLine80 : 1.251 secs (user: 1.250 secs) vGetByteLines80: 0.273 secs (user: 0.270 secs) vGetContents: 2.767 secs (user: 2.770 secs) MemBuf: vPutBuf: 0.094 secs (user: 0.090 secs) vGetBuf: 0.036 secs (user: 0.030 secs) vPutChar: 0.567 secs (user: 0.560 secs) vGetChar: 0.406 secs (user: 0.410 secs) vPutBuf20: 0.221 secs (user: 0.220 secs) vGetBuf20: 0.227 secs (user: 0.230 secs) vPutByteStrLn20: 0.460 secs (user: 0.460 secs) vGetByteLine20 : 0.667 secs (user: 0.670 secs) vPutStrLn20: 1.249 secs (user: 1.250 secs) vGetLine20 : 1.589 secs (user: 1.590 secs) vGetByteLines20: 0.759 secs (user: 0.750 secs) vPutByteStrLn80: 0.142 secs (user: 0.150 secs) vGetByteLine80 : 0.253 secs (user: 0.260 secs) vPutStrLn80: 1.066 secs (user: 1.070 secs) vGetLine80 : 1.348 secs (user: 1.350 secs) vGetByteLines80: 0.284 secs (user: 0.280 secs) vGetContents: 2.841 secs (user: 2.840 secs) FD: vPutBuf: 0.338 secs (user: 0.250 secs) vGetBuf: 0.091 secs (user: 0.100 secs) vPutChar: 0.680 secs (user: 0.680 secs) vGetChar: 0.499 secs (user: 0.500 secs) vPutBuf20: 0.354 secs (user: 0.340 secs) vGetBuf20: 0.317 secs (user: 0.310 secs) vPutByteStrLn20: 3.482 secs (user: 0.560 secs) vGetByteLine20 : 0.788 secs (user: 0.790 secs) vPutStrLn20: 1.360 secs (user: 1.340 secs) vGetLine20 : 1.677 secs (user: 1.670 secs) vGetByteLines20: 0.866 secs (user: 0.860 secs) vPutByteStrLn80: 0.251 secs (user: 0.250 secs) vGetByteLine80 : 0.347 secs (user: 0.350 secs) vPutStrLn80: 1.176 secs (user: 1.180 secs) vGetLine80 : 1.435 secs (user: 1.440 secs) vGetByteLines80: 0.374 secs (user: 0.370 secs) vGetContents: 2.939 secs (user: 2.940 secs) FD utf8: vPutBuf: 0.118 secs (user: 0.100 secs) vGetBuf: 0.087 secs (user: 0.090 secs) vPutChar: 3.931 secs (user: 3.930 secs) vGetChar: 3.282 secs (user: 3.280 secs) vPutBuf20: 0.330 secs (user: 0.330 secs) vGetBuf20: 0.308 secs (user: 0.300 secs) vPutByteStrLn20: 0.564 secs (user: 0.560 secs) vGetByteLine20 : 0.771 secs (user: 0.770 secs) vPutStrLn20: 6.104 secs (user: 3.950 secs) vGetLine20 : 4.285 secs (user: 4.280 secs) vGetByteLines20: 0.861 secs (user: 0.860 secs) vPutByteStrLn80: 0.255 secs (user: 0.250 secs) vGetByteLine80 : 0.351 secs (user: 0.360 secs) vPutStrLn80: 3.700 secs (user: 3.690 secs) vGetLine80 : 4.037 secs (user: 4.040 secs) vGetByteLines80: 0.373 secs (user: 0.370 secs) vGetContents: 5.337 secs (user: 5.340 secs) FD locked: vPutBuf: 0.114 secs (user: 0.110 secs) vGetBuf: 0.087 secs (user: 0.080 secs) vPutChar: 8.292 secs (user: 8.300 secs) vGetChar: 8.626 secs (user: 8.630 secs) vPutBuf20: 0.545 secs (user: 0.550 secs) vGetBuf20: 0.519 secs (user: 0.520 secs) vPutByteStrLn20: 0.877 secs (user: 0.870 secs) vGetByteLine20 : 1.022 secs (user: 1.020 secs) vPutStrLn20: 3.127 secs (user: 1.540 secs) vGetLine20 : 1.945 secs (user: 1.940 secs) vGetByteLines20: 0.868 secs (user: 0.870 secs) vPutByteStrLn80: 0.333 secs (user: 0.300 secs) vGetByteLine80 : 0.398 secs (user: 0.400 secs) vPutStrLn80: 1.201 secs (user: 1.200 secs) vGetLine80 : 1.490 secs (user: 1.490 secs) vGetByteLines80: 0.379 secs (user: 0.370 secs) vGetContents: 2.940 secs (user: 2.940 secs) Handle: vPutBuf: 0.117 secs (user: 0.120 secs) vGetBuf: 0.107 secs (user: 0.110 secs) vPutChar: 32.172 secs (user: 32.170 secs) vGetChar: 19.427 secs (user: 19.420 secs) vPutBuf20: 1.283 secs (user: 1.280 secs) vGetBuf20: 1.236 secs (user: 1.240 secs) vPutByteStrLn20: 3.283 secs (user: 1.900 secs) vGetByteLine20 : 1.867 secs (user: 1.860 secs) vPutStrLn20: 6.252 secs (user: 5.580 secs) vGetLine20 : 2.501 secs (user: 2.500 secs) vGetByteLines20: 2.064 secs (user: 2.060 secs) vPutByteStrLn80: 0.598 secs (user: 0.600 secs) vGetByteLine80 : 0.950 secs (user: 0.950 secs) vPutStrLn80: 2.348 secs (user: 2.350 secs) vGetLine80 : 1.741 secs (user: 1.740 secs) vGetByteLines80: 0.995 secs (user: 0.990 secs) vGetContents: 12.351 secs (user: 12.350 secs) Handle buffered: vPutBuf: 0.117 secs (user: 0.110 secs) vGetBuf: 0.087 secs (user: 0.090 secs) vPutChar: 0.668 secs (user: 0.670 secs) vGetChar: 0.488 secs (user: 0.490 secs) vPutBuf20: 0.330 secs (user: 0.330 secs) vGetBuf20: 0.308 secs (user: 0.300 secs) vPutByteStrLn20: 0.561 secs (user: 0.560 secs) vGetByteLine20 : 0.772 secs (user: 0.780 secs) vPutStrLn20: 3.982 secs (user: 1.370 secs) vGetLine20 : 1.722 secs (user: 1.720 secs) vGetByteLines20: 0.863 secs (user: 0.860 secs) vPutByteStrLn80: 0.255 secs (user: 0.260 secs) vGetByteLine80 : 0.352 secs (user: 0.350 secs) vPutStrLn80: 1.177 secs (user: 1.180 secs) vGetLine80 : 1.502 secs (user: 1.500 secs) vGetByteLines80: 0.378 secs (user: 0.380 secs) vGetContents: 2.929 secs (user: 2.910 secs) Handle buffered+locked: vPutBuf: 0.116 secs (user: 0.110 secs) vGetBuf: 0.087 secs (user: 0.090 secs) vPutChar: 8.393 secs (user: 8.390 secs) vGetChar: 8.118 secs (user: 8.120 secs) vPutBuf20: 0.552 secs (user: 0.550 secs) vGetBuf20: 0.519 secs (user: 0.520 secs) vPutByteStrLn20: 0.782 secs (user: 0.780 secs) vGetByteLine20 : 0.975 secs (user: 0.980 secs) vPutStrLn20: 3.392 secs (user: 1.600 secs) vGetLine20 : 1.933 secs (user: 1.920 secs) vGetByteLines20: 0.875 secs (user: 0.870 secs) vPutByteStrLn80: 0.328 secs (user: 0.310 secs) vGetByteLine80 : 0.401 secs (user: 0.400 secs) vPutStrLn80: 1.204 secs (user: 1.210 secs) vGetLine80 : 1.546 secs (user: 1.530 secs) vGetByteLines80: 0.379 secs (user: 0.380 secs) vGetContents: 2.911 secs (user: 2.910 secs) MMFile: vPutBuf: 0.108 secs (user: 0.110 secs) vGetBuf: 0.107 secs (user: 0.100 secs) vPutChar: 0.592 secs (user: 0.590 secs) vGetChar: 0.427 secs (user: 0.430 secs) vPutBuf20: 0.247 secs (user: 0.250 secs) vGetBuf20: 0.248 secs (user: 0.250 secs) vPutByteStrLn20: 0.496 secs (user: 0.500 secs) vGetByteLine20 : 0.687 secs (user: 0.690 secs) vPutStrLn20: 1.282 secs (user: 1.280 secs) vGetLine20 : 1.578 secs (user: 1.570 secs) vGetByteLines20: 0.785 secs (user: 0.790 secs) vPutByteStrLn80: 0.170 secs (user: 0.170 secs) vGetByteLine80 : 0.279 secs (user: 0.280 secs) vPutStrLn80: 1.097 secs (user: 1.090 secs) vGetLine80 : 1.351 secs (user: 1.350 secs) vGetByteLines80: 0.310 secs (user: 0.310 secs) vGetContents: 2.843 secs (user: 2.840 secs) -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

2 3

Packages and .NET assemblies
by Simon Marlow 12 Jul '06

12 Jul '06

I just took a deeper look at how .NET handles packages, since they have thought through many of the same issues and have a scheme that scales well. There's a great deal of similarity with the schemes we're thinking about for Haskell. The executive summary: - A .NET assembly fulfills the same function as a Haskell package: an assembly is the unit of versioning and distribution, and carries with it a bunch of metadata (author, copyright, etc.). - An assembly is uniquely identified by a triple (name,version,culture). - An assembly travels in a single file (.DLL) - An assembly may define multiple namespaces (like A.B.C), and entities within namespaces. Much like modules in Haskell. - Assemblies are not named directly in source code. C# code (for example) refers to namespaces and entities only. - When compiling C# code, you name the DLLs that contain assemblies to bring into scope (in fact, the transitive closure) using command-line options. - The C# compiler brings a set of assemblies into scope by default (== exposed packages in GHC). - The binary has baked into it the exact identity of the assemblies it was compiled against. - At runtime, the binary might be rebound to different versions of the assemblies it depends on, via a complicated structure of configuration files. So there's no grafting, or even assembly-qualified imports. How do they get away with this? Well, one reason is that namespaces are generally globally unique, because the convention is to use names beginning with an organisation (eg. Microsoft.System). So when you bring a bunch of assemblies into scope, their namespaces generally don't conflict. Interestingly, the assembly story is almost identical to what GHC supports right now, and when we change module identities to include the package name (which I'm working on now), it will be even closer. Not that I necessarily think we can get away with doing nothing to the language, but at lesat this seems to argue for being conservative. Cheers, Simon

2 1

updates to HaXml (1.13.1 and 1.16)
by Malcolm Wallace 12 Jul '06

12 Jul '06

For those who use the current stable version of HaXml, I'd like to announce a new patch-level release, 1.13.1, which contains the following bugfixes: * permit percent character in attribute values * parse unquoted attribute values starting '+' or '#' in HTML * keep the original DTD in the output of 'processXmlWith' See http://www.haskell.org/HaXml/ for downloads. For those living on the development edge, I'd like to report that the current darcs version darcs get http://www.cs.york.ac.uk/fp/darcs/HaXml contains a new set of parser combinators (with the same API as before) that is lazier, whilst still allowing backtracking. By lazy, I mean it can start to return partial values as soon as it has consumed e.g. the start tag of an element, without waiting to check that the close tag matches. This has two good effects: * your program will run faster * it will consume less memory and two bad effects: * if there are errors in the document, they will throw an exception in the middle of your processing * the error message in the exception may be rather less accurate about the cause and location than previously. The older XML parser has also been retained, since the lazy version is still experimental. To use the new one, import Text.XML.HaXml.ParseLazy There are also lazy versions of the usual demo programs CanonicaliseLazy XtractLazy As an example of the improved speed, a query to extract all the <key> tags from a 3.7Mb XML document: Xtract "//key" file.xml did not give any results after more than ten minutes on my machine, but XtractLazy "//key" file.xml started producing results immediately, and completed the task in 25 seconds (returning 52584 tags). Separate website and downloads at http://www.cs.york.ac.uk/fp/HaXml-devel Regards, Malcolm

3 4

Re: Packages and modules
by Malcolm Wallace 11 Jul '06

11 Jul '06

John Meacham <john(a)repetae.net> wrote: > Package names should never appear in source files IMHO. if a package > name is in the source file, then you might as well make it part of the > module name. Packages exist for 'meta-organization' of code. A way to > deal with mapping code _outside_ of the language itself, putting > packages inside the code will force the creation of yet another level, > meta-packages, or something. packages should not be a language issue, > they are an environment one. I tend to the opposite view. The meaning of the code should be expressed in the code itself. If a module M imports A.B.C, and I can see two such modules called A.B.C, then the meaning of the code is ambiguous and ill-defined. I would rather not have to look elsewhere (in the build system? Makefile? scons? Cabal file? DOS batch file? where?) to find out how to resolve the ambiguity. Surely the programmer knew which import was intended. Is it so difficult to communicate that information somewhere close to the import itself? This is the same situation as with compiler flags for language extensions, foreign imports, and so on. If a module uses a non-standard language feature that changes the interpretation of the syntax (e.g. template haskell), it should say so in the module itself {-# OPTIONS_GHC -fth #-}, so that I as a human reader know what is going on. Likewise, if a foreign import requires a C header file, that should also be referred to in the FFI decl, not solely as a build option at some arbitrary distance in the filesystem. Regards, Malcolm

4 18

Re: Why is there no splitBy in the list module?
by Christian Maeder 11 Jul '06

11 Jul '06

Simon Marlow schrieb: > Donald Bruce Stewart wrote: > >> Hacking up your own custom split (or a tokens/splitOnGlue) must be one >> of the most common questions from beginners on the irc channel. >> >> Anyone rememeber what the result of the "let's get split into the base >> library" movement's work was? > > ISTR there wasn't a concensus, so nothing happened. Which is silly, > really - I agree we should definitely have a Data.List.split. > Maybe someone can extract a result from the old discussion about "Prelude function suggestions" http://thread.gmane.org/gmane.comp.lang.haskell.libraries/1684/focus=1684 Cheers Christian Our current (special) version is: {- | A function inspired by the perl function split. A list is splitted on a seperator element in smaller non-empty lists. The seperator element is dropped from the resulting list. -} splitOn :: Eq a => a -- ^ seperator -> [a] -- ^ list to split -> [[a]] splitOn x xs = let (l, r) = break (==x) xs in (if null l then [] else [l]) ++ (if null r then [] else splitOn x $ tail r)

3 3

RE: Packages and modules
by Simon Peyton-Jones 05 Jul '06

05 Jul '06

| So instead of just taking this simple solution, the wiki proposal is instead | destroying the beauty of the per-package namespace idea by incorporating | into it the existing shared namespaces with their attendant problems, | instead of just letting the existing messy system die a natural death | through the syntactic isolation I proposed. Brian, I think your proposal may be clearer to you than to everyone else. It's always hard to reconstruct a detailed proposal by reading long email threads. Suggestion: if you feel strongly about this, why not start a Wiki page (you can link to it from the current one) to describe the design you propose, at a comparable level of detail? Incidentally, compatibility with Cabal is a significant goal. Simon

1 0

RE: Packages and modules
by Simon Peyton-Jones 05 Jul '06

05 Jul '06

In response to Brian and Ian's helpful comments, I've added a bunch more stuff to our proposal about packages. If I have missed anything, let me know. http://hackage.haskell.org/trac/ghc/wiki/GhcPackages If you or anyone else thinks the choices made there are poor ones, continue to say so. It's a helpful process. One particular point: | 2b) Exporting the contents of an external module qualified | | module Foo | ( qualified module P | ) where While this might be a good idea, it's an orthogonal one, and the proposal doesn't tackle it. One thing at a time! Simon

1 0

QDBM / Hyper Estraier bindings for haskell
by Jun Mukai 28 Jun '06

28 Jun '06

Hello all, I pleased to announce the QDBM and Hyper Estraier bindings for haskell. If you are interested in, they can be obtained from http://www.city5.org/haskellprog/qdbm/ http://www.city5.org/haskellprog/hstraier/ These are darcs repository, so you can obtain them via darcs. These bindings are already cabalized, you have to type only as follows to install them: % runghc ./Setup.hs configure % runghc ./Setup.hs build # runghc ./Setup.hs install the online API documentation generated by haddock is also shown at http://www.city5.org/doc/qdbm/ http://www.city5.org/doc/hstraier/ * What is QDBM? QDBM, stands for Quick DBM, is a database module similar to GDBM or NDBM or Berkeley-DB, but the specific feature is performance and simple API. By now, my QDBM binding supports only Depot, Curia, Villa, and a part of Cabin. QDBM can handle several kinds of DBM, and Depot and Curia and Villa is a name of each DBM. Depot is a basic DBM. Curia is a `large' DBM and bundles many Depot files for its internal implementation. Curia also can store `large objects' as a file. Villa is B+Tree DBM and uses a function to compare keys. Villa also has faculty of cursors and transactions. Vista, `large' version of Villa, is not supported. Odeum, inverted index, is also not supported. Cabin is auxiliary functions collection module and contains a kind of utility functions of string operation such as Base64 encode/decode, URL encode/decode, quoted printable encode/decode, mime analysis, csv reading and so on. These functions will be useful for many Haskellers, and run faster. If you are interested in QDBM itself, see http://qdbm.sourceforge.net/ * What is Hyper Estraier Hyper Estraier is a full-text search system using QDBM. You can search documents according to the keywords specified and so on. My Hyper Estraier binding supports only `core' API. Core API can handle document object, conditionals, and full-text search DB. Hyper Estraier itself has `node' API, which invokes search queries to remote servers using HTTP. Node API is not supported yet in my binding now. If you are interested in Hyper Estraier itself, see http://hyperestraier.sourceforge.net/ Any comments are welcome. Thanks, Jun Mukai mukai(a)jmuk.org

2 2