Re: [web-devel] [WAI] Switching to WAI/Warp...

On Sun, Feb 20, 2011 at 12:00 PM, Bardur Arantsson wrote:

On 2011-02-20 10:29, Michael Snoyman wrote:

...

On Sun, Feb 20, 2011 at 11:04 AM, Bardur Arantsson  wrote:

[--snip--]

...

...

Is there any way to avoid this extra "fromByteString"?

blaze-builder is usually pretty intelligent about this. If I remember correctly, Simon Meier said that it won't do a memory copy for ByteStrings larger than 8k. In any event, if you want to force insertion instead of copying, replace fromByteString with insertByteString. It *might* be that the CPU overhead is warranted, however: you may end up seeing increased system call overhead with this switch, since the average chunk size will be smaller.

Good point; I just tried upping the chunk size to 32K and explicitly using insertByteString. Even with those changes I'm still seeing a lot of CPU usage (5-10%).

I should say that I'm using a version of the enumFile enumerator from Data.Enumerator.Binary that I've adapted to support byte ranges. It may be the case that I've just done something horribly stupid or inefficient. I've attached the code for the enumerator.

I don't see any problems with your implementation, but John Millikin is definitely the guy to speak to about that. He will be able to give you a more definitive answer than I.

I guess I'll have to try to get some profiling data to see where the time is actually being spent. I suppose it's about time I learned a bit about profiling my Haskell code :).

It's entirely possible that WAI/enumerators/builder/Warp is adding some overhead. But what exactly are you comparing against? Warp is *definitely* doing some extra stuff that a simple Data.ByteString.hPut is not, such as timeout handling. I'd be interested in any numbers that you come up with from profiling, please do share.

[--snip bits about enumerators--]

Thanks for the explanation. For some reason I'm having a little trouble "connecting" all the type signatures for enumerator/iterator/iteratee. Hopefully it'll get better with practice.

There's no question that it's a difficult concept to get started with, but it many ways it's like learning monads: the concept looks ridiculously complicated at first, then you think that it's just like (spaceships/nuclear waste/burritos), and then it finally *really* clicks and becomes second nature. The only real solution is what you've already said: practice. If you haven't seen it already, I wrote a three-part series on the enumerator package. I've moved that package into a chapter of the Yesod book[1] so you can read it all on a single page if you like. Hope it helps! Michael [1] http://docs.yesodweb.com/book/enumerator