Exactly. Someone on the list gave me this example awhile back for reading CSV files. I can process a gigabyte (simple unpack and print to dev/null for IO testing purposes) in about two and a half seconds using this code. import Data.ByteString.Lazy.Char8 as C -- | Read a datafile and turn it into lists of columns readDatafileAndTranspose name = do sheet <- (transpose . map (C.split '\t') . C.lines) `fmap` C.readFile name return $ foldl' go M.empty sheet where go m (x:xs) = M.insert (C.unpack x) xs m 2008/5/29 Adrian Neumann :

Isn't fast IO what ByteStrings where invented for?

Adrian

Tom Harper schrieb:

...

I'm trying to implement some file I/O where I can read in a file to an array, but do so without having to know how much space I will need. (Before you suggest it, lists are too slow/space consuming.) I was thinking that one possible solution is to use a strategy used in dynamic arrays, where everytime you run out of space you double the capacity of the array. Trouble is, most arrays in Haskell would require you to traverse the entire array and copy each individual cell, which would make it worthless.

Are there any low-level array types (MutableByteArray#, for example) that support a memcpy-like function where I could copy the entire array into the new one instead of copying it one value at a time? Is there another solution that I'm missing?

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

-- I try to take things like a crow; war and chaos don't always ruin a picnic, they just mean you have to be careful what you swallow. -- Jessica Edwards

Why not just read it into a lazy ByteString? Are you looking to use an array with elements of a different type? You could then convert it to a strict ByteString.

Because I'm writing the Unicode-friendly ByteString =p -- Tom Harper MSc Computer Science '08 University of Oxford Mobile: +44 (0)7533 998 591 Skype: +1 949 273 4627 (harpertom)

On Thu, 2008-05-29 at 11:25 -0700, Bryan O'Sullivan wrote:

Tom Harper wrote:

...

Because I'm writing the Unicode-friendly ByteString =p

Perhaps I'm not understanding. Why wouldn't you use ByteString for I/O, even if you're writing a different library? After all, ByteString's own internals use memcpy.

Just to clarify Tom's question... He's designing a proper Unicode type along the lines of ByteString. However it does not use a ByteString underneath so we cannot re-use the IO operations from ByteString, though obviously we can steal ideas. The reason we do not want to re-use ByteString as the underlying representation is because they're not good for short strings and we expect that for Unicode text (more than arbitrary blobs of binary data) people will want efficient short strings. So instead of using a ForeignPtr to pinned heap memory (read: slow allocation, lots of fragmentation) we're looking at using unpinned heap arrays. So that means MutableByteArr# (or STUArray for the first prototype). We cannot use memcpy because it operates on raw pointers but that's no good for a movable heap object like a ByteArr#. So that's the background to the question. I think the answer is either that there's a primitive to do this copying in the GHC.* libs or we'll need to add one. Duncan

duncan.coutts:

On Thu, 2008-05-29 at 11:25 -0700, Bryan O'Sullivan wrote:

...

Tom Harper wrote:

...

Because I'm writing the Unicode-friendly ByteString =p

Perhaps I'm not understanding. Why wouldn't you use ByteString for I/O, even if you're writing a different library? After all, ByteString's own internals use memcpy.

Just to clarify Tom's question...

He's designing a proper Unicode type along the lines of ByteString. However it does not use a ByteString underneath so we cannot re-use the IO operations from ByteString, though obviously we can steal ideas.

The reason we do not want to re-use ByteString as the underlying representation is because they're not good for short strings and we expect that for Unicode text (more than arbitrary blobs of binary data) people will want efficient short strings.

So instead of using a ForeignPtr to pinned heap memory (read: slow allocation, lots of fragmentation) we're looking at using unpinned heap arrays. So that means MutableByteArr# (or STUArray for the first prototype).

We cannot use memcpy because it operates on raw pointers but that's no good for a movable heap object like a ByteArr#.

So that's the background to the question.

I think the answer is either that there's a primitive to do this copying in the GHC.* libs or we'll need to add one.

There are these guys, foreign import ccall unsafe "__hscore_memcpy_dst_off" memcpy_baoff_ba :: RawBuffer -> CInt -> RawBuffer -> CSize -> IO (Ptr ()) foreign import ccall unsafe "__hscore_memcpy_dst_off" memcpy_baoff_ptr :: RawBuffer -> CInt -> Ptr a -> CSize -> IO (Ptr ()) type RawBuffer = MutableByteArray# RealWorld

Hello Ketil, Friday, May 30, 2008, 11:00:10 AM, you wrote:

I guess this is where I don't follow: why would you need more short strings for Unicode text than for ASCII or 8-bit latin text?

BS package require too much memory for storing short strings. alternative package that minimizes memory would be useful -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

"Johan Tibell" writes:

...

I guess this is where I don't follow: why would you need more short strings for Unicode text than for ASCII or 8-bit latin text?

But ByteStrings are neither ASCII nor 8-bit Latin text! [...] The intent of the not-yet-existing Unicode string is to represent text not bytes.

Right, so this will replace the .Char8 modules as well? What confused me was my misunderstanding Duncan to mean that Unicode text would somehow imply shorter strings than non-Unicode (i.e. 8-bit) text.

To give just one example, short (Unicode) strings are common as keys in associative data structures like maps

I guess typically, you'd break things down to words, so strings of lenght 4-10 or so. BS uses three words and LBS four (IIRC), so the cost of sharing typically outweighs the benefit.

Can I also here insert a plea for keeping lazy I/O out of the new Unicode module?

I use ByteString.Lazy almost exclusively. I realize it there's a penalty in time and space, but the ability to write applications that stream over multi-Gb files is essential. Of course, these applications couldn't care less about Unicode, so perhaps the usage is different. -k -- If I haven't seen further, it is by standing in the footprints of giants

"Johan Tibell" writes:

Lazy I/O comes with a penalty in terms of correctness!

Is there a page describing this in more detail? I believe my programs to be correct, but would like to know if there are additional pitfalls, beyond hClosing a handle with outstanding (unevaluated) reads.

Lazy I/O is kinda, maybe usable for small scripts that reads a file or two and spits out a result but for servers it doesn't work at all.

So don't use it for servers, then?

Here are two possible interfaces for safe I/O. One isstream based one with explicit close and the other fold based one (i.e. inversion of control):

Say I have two files with variable-size records, that need to be merged. In other words, I can do something like: do xs <- readFile "X" ys <- readFile "Y" return $ zipWith combine (parseX xs) (parseY ys) Clearly in the "small script" category and not "servers", but is it still unsafe?

...

-- Stream based I/O. -- Left fold/callback based I/O.

Perhaps I don't quite understand the concepts, but I don't see how to implement the example as elegantly and simply with these approaches? -k -- If I haven't seen further, it is by standing in the footprints of giants

On Fri, May 30, 2008 at 12:11 PM, Duncan Coutts wrote:

On Fri, 2008-05-30 at 10:38 +0200, Ketil Malde wrote:

...

"Johan Tibell" writes:

...

But ByteStrings are neither ASCII nor 8-bit Latin text! [...] The intent of the not-yet-existing Unicode string is to represent text not bytes.

Right, so this will replace the .Char8 modules as well?

No, there is still a use-case for that, namely all these network protocols and file formats that mix binary and ASCII data.

After implementing an HTTP parser using ByteString in my web server I can say that the only thing you need from the .Char8 module is `pack' to be able to write byte literals. Matching bytes is better done using something like a `ByteSet' rather than the .Char8 functions as it gives better performance. -- Johan

Ketil Malde wrote:

I guess this is where I don't follow: why would you need more short strings for Unicode text than for ASCII or 8-bit latin text?

Because ByteString is optimised for dealing with big blobs of binary data, its performance when you split a big ByteString into a pile of words is quite poor. Also, people use ByteStrings for dealing with text mostly because their native character sets let them get away with it, not because it's an intrinsically good idea.

Jed Brown writes:

Uh, ByteString is Unicode-agnostic. ByteString.Char8 is not. So why not do IO with lazy ByteString and parse into your own representation (which might look a lot like StorableVector)?

One problem you might run into doing it this way is if a wide character is split between two different arrays. In that case you have to do some post-porcessing to put the pieces back together. More efficient, I think, if you could force a given alignment when reading in the lazy bytestring. But there's not a way to do that, is there? I hope this makes sense. It's the problem I ran into when I tried once to use lazy bytestrings instead of a storable vector, reasoning that the more recent fusion work in bytestring would give a speed boost. But then I was doing numerical stuff, and I don't know much about unicode. Chad