ANNOUNCE: fps -- fast packed strings
Hey. I've cabalised the FastPackedString module from darcs, used in h4sh, as the `fps' library. fps provides mmapped and malloc'd packed strings (byte arrays held by a ForeignPtr), along with a list interface to these strings. It lets you do extremely fast IO in Haskell. For example, the shell identity function written in Haskell (i): $ time cat 200M > /dev/null 0.00s user 0.19s system 2% cpu 6.481 total $ time i 200M > /dev/null 0.00s user 0.00s system 0% cpu 0.172 total $ time cat 200M > /tmp/f 0.00s user 0.38s system 1% cpu 27.932 total $ time i 200M > /tmp/f 0.00s user 0.48s system 7% cpu 6.454 total The FastPackedString module was originally written by Bryan O'Sullivan, it was rewritten to use UArray by Simon Marlow, then rewritten and extended to support slices, mmapping, ForeignPtrs and lots of other stuff by David Roundy and the darcs team. This version fixes a couple of bugs related to conformance to List, adds a testsuite, and makes the code conveniently available as a cabalised library. Get it: http://www.cse.unsw.edu.au/~dons/fps.html darcs get http://www.cse.unsw.edu.au/~dons/code/fps -- Don Stewart
Hello Donald, Thursday, August 25, 2005, 8:10:44 AM, you wrote: DBS> I've cabalised the FastPackedString module from darcs, used in h4sh, as the DBS> `fps' library. if you plan to support/improve this library, can you please: 1) add psniceq/pscmp to export list (with better names, smtg like eqPS/cmpPS) 2) add case-ignoring-versions of these operations (i want to use packed strings to represent filenames, and under Windows filenames are case-insensetive, while under Unix is not) about details of realization - is not it better to use just memcmp to implement pscmp and also "psniceq a b = (pscmp a b) == EQ" ? may be it is better to open representation for library users? i think that number of operations which can be added for packed strings by external libraries is unlimited - GhcBinary support, alternative I/O schemes imho, such library must be included in GHC itself - sooner or later. using 8-times memory representation for ordinary strings is just disastrous for some projects -- Best regards, Bulat mailto:bulatz@HotPOP.com
bulatz:
Hello Donald,
Thursday, August 25, 2005, 8:10:44 AM, you wrote:
DBS> I've cabalised the FastPackedString module from darcs, used in h4sh, as the DBS> `fps' library.
if you plan to support/improve this library, can you please:
I intend to further clean up this code, and also some profiling and quichchecking -- at the moment it is fresh from the darcs-unstable branch. However, before doing more than just clean ups, I'd be inclined to see SimonM's code (rumour has it that he has some fast IO module in the works).
1) add psniceq/pscmp to export list (with better names, smtg like eqPS/cmpPS)
Ok.
2) add case-ignoring-versions of these operations (i want to use packed strings to represent filenames, and under Windows filenames are case-insensetive, while under Unix is not)
Hmm. Couldn't you just mapPS toLower ? We don't have case-ignoring versions of any List functions, so it seems strange to have them here.
about details of realization - is not it better to use just memcmp to implement pscmp and also "psniceq a b = (pscmp a b) == EQ" ?
I would think so, yes. I'll have to work out why it has been written the way has been.
imho, such library must be included in GHC itself - sooner or later. using 8-times memory representation for ordinary strings is just disastrous for some projects
Agreed. -- Don
1) add psniceq/pscmp to export list (with better names, smtg like eqPS/cmpPS)
is it possible to avoid module/type identifiers ("ps") hardwired into the names? We have hierarchical namespaces for that (PS.eq, PS.cmp etc.) This has many advantages, e. g. you could then do "import PackedString.Fast as PS" - otherwise the names were wrong (they should be eqFPS etc.) Also, why the abbreviations ("cmp"). See discussion in ttp://c2.com/cgi/wiki?MeaningfulName (and a lot of other places and books). Sure, a lot of the problems that the (OO) refactoring hype is trying to solve do not arise in the first place if we do functional programming, but quite some are independent of the paradigm (e. g. those on naming) - so we should consider carefully the problem indicators and solutions that have been suggested. Just my opinion (not specific to the packed string library). -- -- Johannes Waldmann -- Tel/Fax (0341) 3076 6479/80 -- ---- http://www.imn.htwk-leipzig.de/~waldmann/ -------
waldmann:
1) add psniceq/pscmp to export list (with better names, smtg like eqPS/cmpPS)
is it possible to avoid module/type identifiers ("ps") hardwired into the names? We have hierarchical namespaces for that (PS.eq, PS.cmp etc.) This has many advantages, e. g. you could then do "import PackedString.Fast as PS" - otherwise the names were wrong
Yes. Good idea. We can follow Data.Map/Set on this.
(they should be eqFPS etc.) Also, why the abbreviations ("cmp").
*shrug* Like I say, this has come straight out of darcs, and wasn't being developed as an fptools-style library, so it needs some polishing. However, enough interest was shown when it got used in h4sh, that I figured it best to cabalise and release as is -- which is definitely fast if not always pretty :) -- Don
After some weeks of hacking, and after taking into consideration the suggestions on this list, from darcs-devel and on #haskell, I've tagged and released v0.1 of the FastPackedString library, FPS. FPS provides mmapped and malloc'd packed strings, along with a list interface to these strings. It lets you do extremely fast IO in Haskell; in some cases, even faster than typical C implementations. Some of the changes since the port from darcs include: * Added a full List interface * Full haddocks * QuickCheck and HUnit tests * Improved performance for many functions, after extensive profiling * Now uses qualified names, e.g. P.length instead of lengthPS * Type is now 'FastString', so as not to clash with the old 'PackedString' * Some new constructors, particularly for Addr# and CStrings Home: http://www.cse.unsw.edu.au/~dons/fps.html Docs: http://www.cse.unsw.edu.au/~dons/fps/Data.FastPackedString.html Src: ftp://ftp.cse.unsw.edu.au/pub/users/dons/fps/fps-0.1.tar.gz Darcs: darcs get --partial http://www.cse.unsw.edu.au/~dons/code/fps Feedback is very welcome, and encouraged. -- Don Stewart
participants (3)
-
Bulat Ziganshin -
dons＠cse.unsw.edu.au -
Johannes Waldmann