On Thu, May 27, 2010 at 11:16 AM, Ivan Miljenovic

wrote:

On 27 May 2010 17:55, Michael Snoyman wrote:

...

Two comments: * The exclamation point seems good enough for attributes. I copied that for Hamlet as well. * If you're standardizing on UTF-8, why not support bytestrings? I'm aware that a user could shoot him/herself in the foot by passing in non-UTF8 data, but I would imagine the performance gains would outweigh this. My recent benchmarks on the BigTable benchmark[1] imply a huge performance gap between ByteStrings and other contenders.

Wow, I find it rather surprising that String out-performs Text; any idea why that is? I wonder if you're just using it wrong...

Could be, I'd be very happy if that were the case. All of the benchmarks are available on Github, and the bytestring[1], text[2] and string[3] versions are all rather short.

Michael [1] http://github.com/snoyberg/benchmarks/blob/master/bigtable/cgi/bytestring.hs [2] http://github.com/snoyberg/benchmarks/blob/master/bigtable/cgi/text.hs [3] http://github.com/snoyberg/benchmarks/blob/master/bigtable/cgi/string.hs

On Thu, May 27, 2010 at 11:28 AM, Ivan Miljenovic

wrote:

On 27 May 2010 18:23, Michael Snoyman wrote:

...

On Thu, May 27, 2010 at 11:16 AM, Ivan Miljenovic wrote:

...

On 27 May 2010 17:55, Michael Snoyman wrote:

...

Two comments: * The exclamation point seems good enough for attributes. I copied

that

...

...

...

for Hamlet as well. * If you're standardizing on UTF-8, why not support bytestrings? I'm aware that a user could shoot him/herself in the foot by passing in non-UTF8 data, but I would imagine the performance gains would outweigh this. My recent benchmarks on the BigTable benchmark[1] imply a huge performance gap between ByteStrings and other contenders.

Wow, I find it rather surprising that String out-performs Text; any idea why that is? I wonder if you're just using it wrong...

Could be, I'd be very happy if that were the case. All of the benchmarks are available on Github, and the bytestring[1], text[2] and string[3] versions are all rather short.

Does using lazy Text values improve this? I find it a little strange that you concatenate so many individual Strings that much. Also, how about explicitly using Text values rather than OverloadedStrings?

I don't do any string concatenation (look closely), I was very careful to avoid it. I tried with lazy text as well: it was slower. This isn't surprising, since lazy text- under the surface- is just a list of strict text. And the benchmark itself already has a lazy list of strict text. Using lazy text would just be adding a layer of wrapping.

I don't know what you mean by "explicitly using Text values"; you mean calling pack manually? That's really all that OverloadedStrings does. You can try out lots of different variants on that benchmark. I did that already, and found this to be the fastest version. Michael

On Thu, May 27, 2010 at 11:40 AM, Ivan Miljenovic

wrote:

On 27 May 2010 18:33, Michael Snoyman wrote:

...

I don't do any string concatenation (look closely), I was very careful to avoid it. I tried with lazy text as well: it was slower. This isn't surprising, since lazy text- under the surface- is just a list of strict text. And the benchmark itself already has a lazy list of strict text. Using lazy text would just be adding a layer of wrapping. I don't know what you mean by "explicitly using Text values"; you mean calling pack manually? That's really all that OverloadedStrings does. You can try out lots of different variants on that benchmark. I did that already, and found this to be the fastest version.

Fair enough. Now that I think about it, I recall once trying to have pretty generate Text values rather than String for graphviz (by using fullRender, so it was still using String under the hood until it came time to render) and it too was much slower than String (unfortunately, I didn't record a patch with these changes so I can't just go back and play with it anymore as I reverted them all :s).

Maybe Bryan can chime in with some best-practices for using Text?

Here's my guess at an explanation for what's happening in my benchmark:

text will clearly beat String in memory usage, that's what it's designed for. However, the compiler is still generating String values which are being encoded to Text as runtime. Now, this is the same process for bytestrings. However, bytestrings never have to be decoded: the IO routines simply read the character buffer. In the case of text, however, the encoded data must be decoded again to a bytestring. In other words, here's what I think the three different benchmarks are really doing: * String: generates a list of Strings, passes each String to a relatively inefficient IO routine. * ByteString: encodes Strings one by one into ByteStrings, generates a list of these ByteStrings, and passes each ByteString to a very efficient IO routine. : Text: encodes Strings one by one into Texts, generates a list of these Texts, calls a UTF-8 decoding function to decode each Text into a ByteString, and passes each resulting ByteString to a very efficient IO routine. In the case of ASCII data to be output as UTF-8, uses the Data.ByteString.Char8.pack function will most likely always be the most efficient choice, and thus it seems like something BlazeHtml should support. I'm considering releasing a Hamlet 0.3 based entirely on UTF-8 encoded ByteStrings, but I'd also like to hear from Bryan about this. Michael

On Thu, May 27, 2010 at 10:23 AM, Michael Snoyman wrote:

On Thu, May 27, 2010 at 11:16 AM, Ivan Miljenovic < ivan.miljenovic@gmail.com> wrote:

...

Wow, I find it rather surprising that String out-performs Text; any idea why that is? I wonder if you're just using it wrong...

Could be, I'd be very happy if that were the case. All of the benchmarks are available on Github, and the bytestring[1], text[2] and string[3] versions are all rather short.

Do you include the cost of encoding the result as e.g. UTF-8? The hope would be that the more compact Text would be faster to traverse, and thus encode, than the list based String.

** Advertisement ** Have you tried the library I have written, Data.Rope ? ** End of advertisement ** The algorithmic complexity of most operations on ropes is way better than on bytestrings : log n for all operations, except traversals, of course. Cheers, PE El 27/05/2010, a las 06:01, Michael Snoyman escribió:

On Thu, May 27, 2010 at 12:57 PM, Johan Tibell wrote: On Thu, May 27, 2010 at 10:23 AM, Michael Snoyman wrote: On Thu, May 27, 2010 at 11:16 AM, Ivan Miljenovic wrote: Wow, I find it rather surprising that String out-performs Text; any idea why that is? I wonder if you're just using it wrong...

Could be, I'd be very happy if that were the case. All of the benchmarks are available on Github, and the bytestring[1], text[2] and string[3] versions are all rather short.

Do you include the cost of encoding the result as e.g. UTF-8? The hope would be that the more compact Text would be faster to traverse, and thus encode, than the list based String.

No, but this is done on purpose. One of my goals in this benchmark was to determine whether I should consider switching Hamlet to ByteStrings. If I were to do so, then the UTF-8 encoding would be done at compile-time instead of run-time.

You're correct that a fair comparison would be to UTF-8 encode the Strings as well. However, that's not what most users are going to do most of the time: when dealing with ASCII data, a straight Char8.pack encoding will do the same as UTF-8. I'm simply pointing out that I think Blaze should support this style.

Michael _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

About as much different as a Data.ByteString from an Array Int Char : there are several optimizations over a Data.Sequence.Seq that are specific to characters, for instance file IO using mmap, and the use of blocks (which would have been possible with about any constant size "Storable" type, of course). Moreover, there are tricks in a Data.Rope to hide the mutability of the blocks, which I do not believe appear in Data.Sequence : you can modify a rope once without copying anything (in incomplete blocks for instance). Of course referential transparency is preserved : if you remodify it afterwards, something will have to get copied, but I found it most useful when building large strings before writing them to files. And last but not least, a data.sequence.seq char is called rope in the litterature ;-) Anyway, good point, cause now I want to benchmark it against Data.Sequence. PE El 28/05/2010, a las 14:22, Max Rabkin escribió:

On Thu, May 27, 2010 at 2:44 PM, Pierre-Etienne Meunier wrote:

...

** Advertisement ** Have you tried the library I have written, Data.Rope ? ** End of advertisement ** The algorithmic complexity of most operations on ropes is way better than on bytestrings : log n for all operations, except traversals, of course. Cheers, PE

How is a Data.Rope.Rope different from a Data.Sequence.Seq Char?

--Max

On Thu, May 27, 2010 at 10:48 AM, Jasper Van der Jeugt wrote:

...

How about also providing an enumerator back-end? http://hackage.haskell.org/packages/archive/iteratee/0.3.5/doc/html/Data-Ite...

Then your library can integrate more easily with the snap framework: http://snapframework.com

Sure, I can do that. But I already tested integration with the snap framework, the best path here seems to call the `writeLBS` function from the snap framework on the `L.ByteString` that BlazeHtml produces (`writeLBS` internally uses an enumerator).

I think it's worth analyzing if using enumerators directly gives a significant performance improvement over converting from lazy ByteStrings. Looking a the conversion functions in the snap framework it looks like we can avoid some intermediate lists for one: writeLBS :: L.ByteString -> Snap () writeLBS s = addToOutput $ enumLBS s addToOutput :: (forall a . Enumerator a) -> Snap () addToOutput enum = modifyResponse $ modifyResponseBody (>. enum) enumLBS :: (Monad m) => L.ByteString -> Enumerator m a enumLBS lbs iter = foldM k iter enums where enums = map (enumPure1Chunk . WrapBS) $ L.toChunks lbs k i e = e i -- from iteratee: enumPure1Chunk :: (SC.StreamChunk s el, Monad m) => s el -> EnumeratorGM s el m a enumPure1Chunk str iter = runIter iter (Chunk str) >>= checkIfDone return Regards, Bas