Hello, I have a question about WAI. ResponseEnumerator is defined as: type ResponseEnumerator a = (Status -> ResponseHeaders -> Iteratee Builder IO a) -> IO a Are there any reasons to use Builder instead of ByteString. I could not find code which makes use of Iteratee Builder IO a in Yesod. Michael posted the following code before. This code converts ByteString to Builder. But Builder is converted to ByteString in Warp. It seems to me that "Iteratee ByteString IO a" is more reasonable than "Iteratee ByteString IO a". Am I missing something? --Kazu ---- data Proxy = Proxy mkYesod "Proxy" [parseRoutes| / RootR GET |] instance Yesod Proxy where approot _ = "" getRootR :: GHandler Proxy Proxy () getRootR = do req <- liftIO $ parseUrl "http://www.yesodweb.com/" sendWaiResponse $ ResponseEnumerator $ \f -> withManager $ \m -> run_ (http req (blaze f) m) blaze :: (Status -> ResponseHeaders -> Iteratee Builder IO a) -> Status -> ResponseHeaders -> Iteratee ByteString IO a blaze f s h = joinI $ EL.map fromByteString $$ f s h' where h' = filter go h go ("Content-Encoding", _) = False go _ = True main :: IO () main = warpDebug 3000 Proxy ----
On Tue, Dec 13, 2011 at 9:37 AM, Kazu Yamamoto
Hello,
I have a question about WAI. ResponseEnumerator is defined as:
type ResponseEnumerator a = (Status -> ResponseHeaders -> Iteratee Builder IO a) -> IO a
Are there any reasons to use Builder instead of ByteString. I could not find code which makes use of Iteratee Builder IO a in Yesod.
Michael posted the following code before. This code converts ByteString to Builder. But Builder is converted to ByteString in Warp.
It seems to me that "Iteratee ByteString IO a" is more reasonable than "Iteratee ByteString IO a". Am I missing something?
--Kazu
---- data Proxy = Proxy
mkYesod "Proxy" [parseRoutes| / RootR GET |]
instance Yesod Proxy where approot _ = ""
getRootR :: GHandler Proxy Proxy () getRootR = do req <- liftIO $ parseUrl "http://www.yesodweb.com/" sendWaiResponse $ ResponseEnumerator $ \f -> withManager $ \m -> run_ (http req (blaze f) m)
blaze :: (Status -> ResponseHeaders -> Iteratee Builder IO a) -> Status -> ResponseHeaders -> Iteratee ByteString IO a blaze f s h = joinI $ EL.map fromByteString $$ f s h' where h' = filter go h go ("Content-Encoding", _) = False go _ = True
main :: IO () main = warpDebug 3000 Proxy ----
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel
It's a good question. The advantage of a Builder is that we get cheap concatenation, so you can create a stream of smaller ByteStrings and then Warp will automatically concatenate them together into larger chunks. There are a few claims against this: Q1. Shouldn't it be at the user's discretion to use Builders internally and then create a stream of ByteStrings? A1. That would be less efficient, as we wouldn't get cheap concatenation with the response headers. Q2. Isn't it really inefficient to convert from ByteString to Builder, and then right back to ByteString? A2. No. If the ByteStrings are small, then they will be copied into a larger buffer, which should be a performance gain overall (less system calls). If they are already large, then blaze-builder uses an InsertByteString instruction to avoid copying. Q3. Doesn't this prevent us from creating comet-style servers, since data will be cached? A3. You can force blaze-builder to output a ByteString before it is an optimal size by sending a flush command. If these answers make sense, I'll add them to the WAI docs. Michael
Michael Snoyman wrote:
It's a good question. The advantage of a Builder is that we get cheap concatenation, so you can create a stream of smaller ByteStrings and then Warp will automatically concatenate them together into larger chunks. There are a few claims against this:
Q1. Shouldn't it be at the user's discretion to use Builders internally and then create a stream of ByteStrings? A1. That would be less efficient, as we wouldn't get cheap concatenation with the response headers.
Q2. Isn't it really inefficient to convert from ByteString to Builder, and then right back to ByteString? A2. No. If the ByteStrings are small, then they will be copied into a larger buffer, which should be a performance gain overall (less system calls). If they are already large, then blaze-builder uses an InsertByteString instruction to avoid copying.
Q3. Doesn't this prevent us from creating comet-style servers, since data will be cached? A3. You can force blaze-builder to output a ByteString before it is an optimal size by sending a flush command.
If these answers make sense, I'll add them to the WAI docs.
+1 Cheers, Erik -- ---------------------------------------------------------------------- Erik de Castro Lopo http://www.mega-nerd.com/
Hello Michael, Thank you for your answer.
Q1. Shouldn't it be at the user's discretion to use Builders internally and then create a stream of ByteStrings? A1. That would be less efficient, as we wouldn't get cheap concatenation with the response headers.
But I guess response headers can be sent by sendMany (i.e. writev()) without concatenation. And if a response body is created, for example, by enumHandle, the ByteString is large enough. (ByteStrings for a response body are not concatenated at all, I believe.) So, I'm not convinced with your answer. I'm not objecting your design. Rather, I would learn reasons of your design choice. Suppose that Builder is more efficient than ByteString in some cases. Even in this case, it seems to me that the iteratee can be defined as "Iteratee ByteString IO a" and the iteratee translates ByteString to Builder (and to ByteString) inside. This design lets programms to omit "map fromByteString" (e.g. in your Proxy code). --Kazu
On Tue, Dec 13, 2011 at 10:12 AM, Kazu Yamamoto
Hello Michael,
Thank you for your answer.
Q1. Shouldn't it be at the user's discretion to use Builders internally and then create a stream of ByteStrings? A1. That would be less efficient, as we wouldn't get cheap concatenation with the response headers.
But I guess response headers can be sent by sendMany (i.e. writev()) without concatenation. And if a response body is created, for example, by enumHandle, the ByteString is large enough. (ByteStrings for a response body are not concatenated at all, I believe.) So, I'm not convinced with your answer.
writev might make the different here, I'm not sure. I haven't done any performance testing on the difference between writev with a number of smaller ByteStrings versus a single larger ByteString. That would be an interesting distinction. But I suppose there's also a A1b answer. The only downside it seems of using Builder versus ByteString is the extra typing a user needs in order to convert to Builders. If we switch to ByteString, it may be true that we can achieve the same performance of Builders (based on the question of writev performance above), but it would require extra thought from a user to ensure that they properly concatenate their Builders. In this case, I would not expect writev to have the same performance, as the ByteStrings are likely coming through in different chunks.
I'm not objecting your design. Rather, I would learn reasons of your design choice.
Suppose that Builder is more efficient than ByteString in some cases. Even in this case, it seems to me that the iteratee can be defined as "Iteratee ByteString IO a" and the iteratee translates ByteString to Builder (and to ByteString) inside. This design lets programms to omit "map fromByteString" (e.g. in your Proxy code).
Michael
writev might make the different here, I'm not sure. I haven't done any performance testing on the difference between writev with a number of smaller ByteStrings versus a single larger ByteString. That would be an interesting distinction.
But I suppose there's also a A1b answer. The only downside it seems of using Builder versus ByteString is the extra typing a user needs in order to convert to Builders. If we switch to ByteString, it may be true that we can achieve the same performance of Builders (based on the question of writev performance above), but it would require extra thought from a user to ensure that they properly concatenate their Builders. In this case, I would not expect writev to have the same performance, as the ByteStrings are likely coming through in different chunks.
OK. Let's add these to doc. Thank you for your time. --Kazu
On Tue, Dec 13, 2011 at 10:56 AM, Kazu Yamamoto
writev might make the different here, I'm not sure. I haven't done any performance testing on the difference between writev with a number of smaller ByteStrings versus a single larger ByteString. That would be an interesting distinction.
But I suppose there's also a A1b answer. The only downside it seems of using Builder versus ByteString is the extra typing a user needs in order to convert to Builders. If we switch to ByteString, it may be true that we can achieve the same performance of Builders (based on the question of writev performance above), but it would require extra thought from a user to ensure that they properly concatenate their Builders. In this case, I would not expect writev to have the same performance, as the ByteStrings are likely coming through in different chunks.
OK. Let's add these to doc. Thank you for your time.
--Kazu
Added on Git, thanks for the input. Michael
participants (3)
-
Erik de Castro Lopo -
Kazu Yamamoto -
Michael Snoyman