On 23 March 2010 00:10, Johan Tibell wrote:

A sequence of bytes is not the same thing as a sequence of Unicode code points. If you want to replace String by something more efficient have a look at Data.Text.

Though Data.Text still has the disadvantage of not being as nice to deal with as String, since you can't pattern match on it, etc. Whilst it may degrade performance, treating String as a list of characters rather than an array provides you with greater flexibility of how to deal with it. -- Ivan Lazar Miljenovic Ivan.Miljenovic@gmail.com IvanMiljenovic.wordpress.com

On 22.03.2010 14:15, Ivan Miljenovic wrote:

On 23 March 2010 00:10, Johan Tibell wrote:

...

A sequence of bytes is not the same thing as a sequence of Unicode code points. If you want to replace String by something more efficient have a look at Data.Text.

Though Data.Text still has the disadvantage of not being as nice to deal with as String, since you can't pattern match on it, etc.

Whilst it may degrade performance, treating String as a list of characters rather than an array provides you with greater flexibility of how to deal with it.

But doesn't this basically mean sacrifice performance / applicability for algorithmic beauty? Sure, pattern match comes in handy, but in the case of strings at a high price

On Mon, Mar 22, 2010 at 6:10 AM, Johan Tibell wrote:

On Mon, Mar 22, 2010 at 1:16 PM, Johann Höchtl wrote:

...

My question or discussion point: Why not depreciate [Char] altogether and favour of lazy Bytestrings?

A sequence of bytes is not the same thing as a sequence of Unicode code points. If you want to replace String by something more efficient have a look at Data.Text.

Slight correction. A sequence of bytes is exactly the same thing as a sequence of Unicode bytes when you use UTF8.

-- Johan _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Hi David Leimbach wrote:

On Mon, Mar 22, 2010 at 6:10 AM, Johan Tibell wrote: On Mon, Mar 22, 2010 at 1:16 PM, Johann Höchtl wrote: > My question or discussion point: Why not depreciate [Char] altogether > and favour of lazy Bytestrings?

A sequence of bytes is not the same thing as a sequence of Unicode code points. If you want to replace String by something more efficient have a look at Data.Text.

Slight correction.

A sequence of bytes is exactly the same thing as a sequence of Unicode bytes when you use UTF8.

And a sequence of bytes is exactly the same thing as a UTF-32 string, when the sequence is encoded as UTF-32. But that is not the point. The point is that when some function is handed a ByteString, the ByteString can be encoded in many ways. It do not even have to be text. So you need a UTF-8 String type. /Mads

-- Johan

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

On Tue, Mar 23, 2010 at 00:27, Johann Höchtl wrote:

How are ByteStrings (Lazy, UTF8) and Data.Text meant to co-exist? When I read bytestrings over a socket which happens to be UTF16-LE encoded and identify a fitting function in Data.Text, I guess I have to transcode them with Data.Text.Encoding to make the type System happy?

There's no such thing as a UTF8 or UTF16 bytestring -- a bytestring is just a more efficient encoding of [Word8], just as Text is a more efficient encoding of [Char]. If the file format you're parsing specifies that some series of bytes is text encoded as UTF16-LE, then you can use the Text decoders to convert to Text. Poor separation between bytes and characters has caused problems in many major languages (C, C++, PHP, Ruby, Python) -- lets not abandon the advantages of correctness to chase a few percentage points of performance.

On Tue, Mar 23, 2010 at 08:51:16AM -0700, John Millikin wrote:

On Tue, Mar 23, 2010 at 00:27, Johann Höchtl wrote:

...

How are ByteStrings (Lazy, UTF8) and Data.Text meant to co-exist? When I read bytestrings over a socket which happens to be UTF16-LE encoded and identify a fitting function in Data.Text, I guess I have to transcode them with Data.Text.Encoding to make the type System happy?

There's no such thing as a UTF8 or UTF16 bytestring -- a bytestring is just a more efficient encoding of [Word8], just as Text is a more efficient encoding of [Char]. If the file format you're parsing specifies that some series of bytes is text encoded as UTF16-LE, then you can use the Text decoders to convert to Text.

Poor separation between bytes and characters has caused problems in many major languages (C, C++, PHP, Ruby, Python) -- lets not abandon the advantages of correctness to chase a few percentage points of performance.

I agree with the principle of correctness, but let's be honest - it's (many) orders of magnitude between ByteString and String and Text, not just a few percentage points… I've been struggling with this problem too and it's not nice. Every time one uses the system readFile & friends (anything that doesn't read via ByteStrings), it hell slow. Test: read a file and compute its size in chars. Input text file is ~40MB in size, has one non-ASCII char. The test might seem stupid but it is a simple one. ghc 6.12.1. Data.ByteString.Lazy (bytestring readFile + length) - < 10 miliseconds, incorrect length (as expected). Data.ByteString.Lazy.UTF8 (system readFile + fromString + length) - 11 seconds, correct length. Data.Text.Lazy (system readFile + pack + length) - 26s, correct length. String (system readfile + length) - ~1 second, correct length. For the record: python2.6 (str type) - ~60ms, incorrect length. python3.1 (unicode) - ~310ms, correct length. If anyone has a solution on how to work on fast text (unicode) transformations (but not a 1:1 pipeline where fusion can work nicely), I'd be glad to hear. iustin

On Tue, Mar 23, 2010 at 01:21:49PM -0400, Nick Bowler wrote:

On 18:11 Tue 23 Mar , Iustin Pop wrote:

...

I agree with the principle of correctness, but let's be honest - it's (many) orders of magnitude between ByteString and String and Text, not just a few percentage points…

I've been struggling with this problem too and it's not nice. Every time one uses the system readFile & friends (anything that doesn't read via ByteStrings), it hell slow.

Test: read a file and compute its size in chars. Input text file is ~40MB in size, has one non-ASCII char. The test might seem stupid but it is a simple one. ghc 6.12.1.

Data.ByteString.Lazy (bytestring readFile + length) - < 10 miliseconds, incorrect length (as expected).

Data.ByteString.Lazy.UTF8 (system readFile + fromString + length) - 11 seconds, correct length.

Data.Text.Lazy (system readFile + pack + length) - 26s, correct length.

String (system readfile + length) - ~1 second, correct length.

Is this a mistake? Your own report shows String & readFile being an order of magnitude faster than everything else, contrary to your earlier claim.

No, it's not a mistake. String is faster than pack to Text and length, but it's 100 times slower than ByteString. My whole point is that difference between byte processing and char processing in Haskell is not a few percentage points, but order of magnitude. I would really like to have only the 6x penalty that Python shows, for example. regards, iustin

On Tue, Mar 23, 2010 at 03:31:33PM -0400, Nick Bowler wrote:

On 18:25 Tue 23 Mar , Iustin Pop wrote:

...

On Tue, Mar 23, 2010 at 01:21:49PM -0400, Nick Bowler wrote:

...

On 18:11 Tue 23 Mar , Iustin Pop wrote:

...

I agree with the principle of correctness, but let's be honest - it's (many) orders of magnitude between ByteString and String and Text, not just a few percentage points…

I've been struggling with this problem too and it's not nice. Every time one uses the system readFile & friends (anything that doesn't read via ByteStrings), it hell slow.

Test: read a file and compute its size in chars. Input text file is ~40MB in size, has one non-ASCII char. The test might seem stupid but it is a simple one. ghc 6.12.1.

Data.ByteString.Lazy (bytestring readFile + length) - < 10 miliseconds, incorrect length (as expected).

Data.ByteString.Lazy.UTF8 (system readFile + fromString + length) - 11 seconds, correct length.

Data.Text.Lazy (system readFile + pack + length) - 26s, correct length.

String (system readfile + length) - ~1 second, correct length.

Is this a mistake? Your own report shows String & readFile being an order of magnitude faster than everything else, contrary to your earlier claim.

No, it's not a mistake. String is faster than pack to Text and length, but it's 100 times slower than ByteString.

Only if you don't care about obtaining the correct answer, in which case you may as well just say const 42 or somesuch, which is even faster.

...

My whole point is that difference between byte processing and char processing in Haskell is not a few percentage points, but order of magnitude. I would really like to have only the 6x penalty that Python shows, for example.

Hang on a second... less than 10 milliseconds to read 40 megabytes from disk? Something's fishy here.

Of course I don't want to benchmark the disk, and therefore the source file is on tmpfs.

I ran my own tests with a 400M file (419430400 bytes) consisting almost exclusively of the letter 'a' with two Japanese characters placed at every multiple of 40 megabytes (UTF-8 encoded).

With Prelude.readFile/length and 5 runs, I see

10145ms, 10087ms, 10223ms, 10321ms, 10216ms.

with approximately 10% of that time spent performing GC each run.

With Data.Bytestring.Lazy.readFile/length and 5 runs, I see

8223ms, 8192ms, 8077ms, 8091ms, 8174ms.

with approximately 20% of that time spent performing GC each run. Maybe there's some magic command line options to tune the GC for our purposes, but I only managed to make things slower. Thus, I'll handwave a bit and just shave off the GC time from each result.

Prelude: 9178ms mean with a standard deviation of 159ms. Data.ByteString.Lazy: 6521ms mean with a standard deviation of 103ms.

Therefore, we managed a throughput of 43 MB/s with the Prelude (and got the right answer), while we managed 61 MB/s with lazy ByteStrings (and got the wrong answer). My disk won't go much, if at all, faster than the second result, so that's good.

I'll bet that for a 400MB file, if you have more than two 2GB of ram, most of it will be cached. If you want to check Haskell performance, just copy it to a tmpfs filesytem so that the disk is out of the equation.

So that's a 30% reduction in throughput. I'd say that's a lot worse than a few percentage points, but certainly not orders of magnitude.

Because you're possibly benchmarking the disk also. With a 400MB file on tmpfs, lazy bytestring readfile + length takes on my machine ~150ms, which is way faster than 8 seconds…

On the other hand, using Data.ByteString.Lazy.readFile and Data.ByteString.Lazy.UTF8.length, we get results of around 12000ms with approximately 5% of that time spent in GC, which is rather worse than the Prelude. Data.Text.Lazy.IO.readFile and Data.Text.Lazy.length are even worse, with results of around 25 *seconds* (!!) and 2% of that time spent in GC.

GNU wc computes the correct answer as quickly as lazy bytestrings compute the wrong answer. With perl 5.8, slurping the entire file as UTF-8 computes the correct answer just as slowly as Prelude. In my first ever Python program (with python 2.6), I tried to read the entire file as a unicode string and it quickly crashes due to running out of memory (yikes!), so it earns a DNF.

So, for computing the right answer with this simple test, it looks like the Prelude is the best option. We tie with Perl and lose only to GNU wc (which is written in C). Really, though, it would be nice to close that gap.

Totally agreed :) iustin

If you read the source code, length do not read the data, that's why it is so fast. It cannot be done for UTF-8 strings.

I think at this point most the amazement is directed at Data.Text being slower than good old [Char] (at least for this operation - we should probably expand our view to more than one operation).

Hey, normal string way faster than GNU wc!

No - you need to perform a fair comparison. Try "wc -c" to only count characters (not lines and words too). I'd provide numbers but my wc doesn't seem to support UTF-8 and not sure what package contains a unicode aware wc.

readChar :: L.ByteString -> Maybe Int64 readChar bs = do (c,_) <- L.uncons bs                 return (choose (fromEnum c))  where  choose :: Int -> Int64  choose c    | c < 0xc0  = 1    | c < 0xe0  = 2    | c < 0xf0  = 3    | c < 0xf8  = 4    | otherwise = 1

inspired by Data.ByteString.Lazy.UTF8, same performances as GNU wc (it is cheating because it do not check the validity of the multibyte char).

Ah, interesting and a worth-while cheat. Thomas

BOS:

Well, your benchmarks are highly suspect.

Attached is another benchmark with similar results. This is no criterion benchmark but I did try to adjust a wee bit for cache issues. Suffice to say I am not yet impressed with Data.Text performance wise. In the broader scope I feel there is a deeper point here: Data.ByteString is how most data is available in a program (be it from a file, network, or other library/device). With all this data we really should have some sort of zero-copy "safeCoerce" that can expose the bytestring as an O(1) unboxed array of some safe length. I know this would have been nice for pureMD5, which did use an ugly unsafePerformIO hack just to get Word16s but even those got boxed up at horrible cost (it now uses 'cereal' to get the Word16 - even worse performance but it lets me ignore endianess of the architecture). ---- OT ---- Beating the dead horse: I once wrote and deleted a blog post ranting about how to get Word16 in C vs in Haskell: word = ((uint16_t *)p)[index]; or word = unsafePerformIO $ withForeignPtr ptr $ \ptr' -> let p = castPtr (plusPtr ptr' off) in peekElemOff p index That Haskell snippet takes significantly longer to both write and run. ---- END OT ---- Code and benchmark numbers included below, all complaints about the accuracy of the benchmark are welcome. NOTE: tLog is a daily #haskell log file repeated many times ~ 59MB. ------- [tommd@Mavlo Test]$ ./read tLog Normal String + System.IO "61443120": 1.467924s Data.ByteString.Lazy "61450365": 0.023285s Data.ByteString.Lazy.UTF8 "61443120": 3.305154s Data.Text.Lazy "61443120": 3.99178s ----- CODE ----- import Data.Time import qualified Data.ByteString.Lazy as L import qualified Data.ByteString.Lazy.UTF8 as U import qualified Data.Text.Lazy as T import qualified Data.Text.Lazy.IO as TO import System.IO import System.Environment (getArgs) import Control.Monad (sequence_, when) main = do [file] <- getArgs let time (f, desc) = do s <- getCurrentTime r <- f let !r' = r t <- getCurrentTime let d = diffUTCTime t s when (length desc > 0) (putStrLn $ desc ++ " " ++ show r' ++ ": " ++ show d) ops = [ (readFile file >>= return . show . length, "") , (readFile file >>= return . show . length, "Normal String + System.IO") , (L.readFile file >>= return . show . L.length, "") , (L.readFile file >>= return . show . L.length, "Data.ByteString.Lazy") , (L.readFile file >>= return . show . U.length, "") , (L.readFile file >>= return . show . U.length, "Data.ByteString.Lazy.UTF8") , (TO.readFile file >>= return . show . T.length, "") , (TO.readFile file >>= return . show . T.length, "Data.Text.Lazy") ] mapM_ time ops

Bryan O'Sullivan wrote:

2010/3/23 Iustin Pop mailto:iusty@k1024.org>

I agree with the principle of correctness, but let's be honest - it's (many) orders of magnitude between ByteString and String and Text, not just a few percentage points…

Well, your benchmarks are highly suspect. See below.

Data.ByteString.Lazy.UTF8 (system readFile + fromString + length) - 11 seconds, correct length.

You should be using bytestring I/O for this.

Data.Text.Lazy (system readFile + pack + length) - 26s, correct length.

You should be using text I/O for this. This is certainly true but adds some unfortunate bloat and has the taste of Java libraries, which define a plethora of similar, but not interchangeable methods due to legacy.