
Hi guys, Just wanted to give you a heads up that yesod's sendFile is not setting the Content-Length header, which 'breaks' some proxies and load balancers. (in our case, amazon elastic load balancer). I don't know if that's by design, I'm not 100% sure but not setting the header would be only useful for 'long pooling' of binary files for doing some sort of hackish http streaming, which I don't think is yesod's purpose anyways :) A workaround is setting the Content-Length header before calling sendFile, I guess it may be something easy to add, I was going to propose a patch that just wraps the current implementation in a do block that sets the header, but then I noticed a number of optimizations being made that hint there should be a 'better' way. cheers! ----nubis :)

I think the reason we didn't include the header is because, when I
wrote it, I didn't know of a cross-platform way to get file sizes. Now
that I know about unix-compat, this seems like a reasonable thing to
add. Anyone know a reason we *shouldn't* do this?
Michael
On Tue, Jun 14, 2011 at 8:16 PM, Nubis
Hi guys, Just wanted to give you a heads up that yesod's sendFile is not setting the Content-Length header, which 'breaks' some proxies and load balancers. (in our case, amazon elastic load balancer). I don't know if that's by design, I'm not 100% sure but not setting the header would be only useful for 'long pooling' of binary files for doing some sort of hackish http streaming, which I don't think is yesod's purpose anyways :) A workaround is setting the Content-Length header before calling sendFile, I guess it may be something easy to add, I was going to propose a patch that just wraps the current implementation in a do block that sets the header, but then I noticed a number of optimizations being made that hint there should be a 'better' way.
cheers! ----nubis :)
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

On Tue, Jun 14, 2011 at 11:50 AM, Michael Snoyman
I think the reason we didn't include the header is because, when I wrote it, I didn't know of a cross-platform way to get file sizes. Now that I know about unix-compat, this seems like a reasonable thing to add. Anyone know a reason we *shouldn't* do this?
In Lift, we do not automatically populate the Content-Length for generic BLOB responses because if the response is streaming, then we have to strictly evaluate the stream to determine the length. For returning large items (e.g., a 2 GB AVI), this can cause issues. We break the Response types into "In Memory" which are strict and we include the Content-Length header, but for responses that are not strict, it's up to the coder. My 2 cents.
Michael
Hi guys, Just wanted to give you a heads up that yesod's sendFile is not setting
Content-Length header, which 'breaks' some proxies and load balancers. (in our case, amazon elastic load balancer). I don't know if that's by design, I'm not 100% sure but not setting the header would be only useful for 'long pooling' of binary files for doing some sort of hackish http streaming, which I don't think is yesod's
anyways :) A workaround is setting the Content-Length header before calling sendFile, I guess it may be something easy to add, I was going to propose a patch
On Tue, Jun 14, 2011 at 8:16 PM, Nubis
wrote: the purpose that just wraps the current implementation in a do block that sets the header, but then I noticed a number of optimizations being made that hint there should be a 'better' way.
cheers! ----nubis :)
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel
-- Lift, the simply functional web framework http://liftweb.net Simply Lift http://simply.liftweb.net Follow me: http://twitter.com/dpp Blog: http://goodstuff.im

On Tue, Jun 14, 2011 at 9:55 PM, David Pollak
On Tue, Jun 14, 2011 at 11:50 AM, Michael Snoyman
wrote: I think the reason we didn't include the header is because, when I wrote it, I didn't know of a cross-platform way to get file sizes. Now that I know about unix-compat, this seems like a reasonable thing to add. Anyone know a reason we *shouldn't* do this?
In Lift, we do not automatically populate the Content-Length for generic BLOB responses because if the response is streaming, then we have to strictly evaluate the stream to determine the length. For returning large items (e.g., a 2 GB AVI), this can cause issues. We break the Response types into "In Memory" which are strict and we include the Content-Length header, but for responses that are not strict, it's up to the coder.
My 2 cents.
Nubis was referring specifically to sending files. Yesod has a similar breakdown where some datatypes (strict ByteStrings for instance) have a content-length attached, while others (lazy ByteStrings) do not. We achieve this via the optional Int argument in the ContentBuilder constructor[1]. Various instances of ToContent simply handle this differently. In the case of files, we would either use the length specified by the FilePart parameter, or simply get the file size from the file system. Michael [1] http://hackage.haskell.org/packages/archive/yesod-core/0.8.2/doc/html/Yesod-...
Michael
On Tue, Jun 14, 2011 at 8:16 PM, Nubis
wrote: Hi guys, Just wanted to give you a heads up that yesod's sendFile is not setting the Content-Length header, which 'breaks' some proxies and load balancers. (in our case, amazon elastic load balancer). I don't know if that's by design, I'm not 100% sure but not setting the header would be only useful for 'long pooling' of binary files for doing some sort of hackish http streaming, which I don't think is yesod's purpose anyways :) A workaround is setting the Content-Length header before calling sendFile, I guess it may be something easy to add, I was going to propose a patch that just wraps the current implementation in a do block that sets the header, but then I noticed a number of optimizations being made that hint there should be a 'better' way.
cheers! ----nubis :)
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel
-- Lift, the simply functional web framework http://liftweb.net Simply Lift http://simply.liftweb.net Follow me: http://twitter.com/dpp Blog: http://goodstuff.im

On Tue, Jun 14, 2011 at 12:06 PM, Michael Snoyman
On Tue, Jun 14, 2011 at 9:55 PM, David Pollak
wrote: On Tue, Jun 14, 2011 at 11:50 AM, Michael Snoyman
wrote: I think the reason we didn't include the header is because, when I wrote it, I didn't know of a cross-platform way to get file sizes. Now that I know about unix-compat, this seems like a reasonable thing to add. Anyone know a reason we *shouldn't* do this?
In Lift, we do not automatically populate the Content-Length for generic BLOB responses because if the response is streaming, then we have to strictly evaluate the stream to determine the length. For returning
large
items (e.g., a 2 GB AVI), this can cause issues. We break the Response types into "In Memory" which are strict and we include the Content-Length header, but for responses that are not strict, it's up to the coder.
My 2 cents.
Nubis was referring specifically to sending files. Yesod has a similar breakdown where some datatypes (strict ByteStrings for instance) have a content-length attached, while others (lazy ByteStrings) do not. We achieve this via the optional Int argument in the ContentBuilder constructor[1]. Various instances of ToContent simply handle this differently.
In the case of files, we would either use the length specified by the FilePart parameter, or simply get the file size from the file system.
Sorry for the newbie mistake and thanks for correction me! Rock on.
Michael
[1] http://hackage.haskell.org/packages/archive/yesod-core/0.8.2/doc/html/Yesod-...
Michael
On Tue, Jun 14, 2011 at 8:16 PM, Nubis
wrote: Hi guys, Just wanted to give you a heads up that yesod's sendFile is not
the Content-Length header, which 'breaks' some proxies and load balancers. (in our case, amazon elastic load balancer). I don't know if that's by design, I'm not 100% sure but not setting
header would be only useful for 'long pooling' of binary files for doing some sort of hackish http streaming, which I don't think is yesod's purpose anyways :) A workaround is setting the Content-Length header before calling sendFile, I guess it may be something easy to add, I was going to propose a patch that just wraps the current implementation in a do block that sets the header, but then I noticed a number of optimizations being made that hint
setting the there
should be a 'better' way.
cheers! ----nubis :)
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel
-- Lift, the simply functional web framework http://liftweb.net Simply Lift http://simply.liftweb.net Follow me: http://twitter.com/dpp Blog: http://goodstuff.im
-- Lift, the simply functional web framework http://liftweb.net Simply Lift http://simply.liftweb.net Follow me: http://twitter.com/dpp Blog: http://goodstuff.im

A simple, can be done immediately would be to put a note in the Haddocks for
sendFile: "warning, does not set the content-length http header which will
break some proxies such as amazons ELB. if you are using a proxy, please
set the header manually.
Does hFileSize in System.IO not work on windows? I'm not at the office now,
but IIRC, this is how we fixed it:
sendFileWithLength ct fp = do
hSize <- liftIO $ withFile fp ReadMode hFileSize -- All System.IO
setHeader "Content-Length" (fromString $ show hSize)
sendFile ct fp
i think this could be combined into one line if you wanna be slick, but dont
have GHC here to check..
sendFileWithLength ct fp =
(liftIO $ withFile fp ReadMode hFileSize) >>= setHeader "Content-Length"
. (fromString . show) >> sendFile ct fp
I know i'm kind of abusing fromString here, but it was 2 am and we were
tired..
On Wed, Jun 15, 2011 at 3:10 AM, David Pollak wrote: On Tue, Jun 14, 2011 at 12:06 PM, Michael Snoyman On Tue, Jun 14, 2011 at 9:55 PM, David Pollak
On Tue, Jun 14, 2011 at 11:50 AM, Michael Snoyman I think the reason we didn't include the header is because, when I
wrote it, I didn't know of a cross-platform way to get file sizes. Now
that I know about unix-compat, this seems like a reasonable thing to
add. Anyone know a reason we *shouldn't* do this? In Lift, we do not automatically populate the Content-Length for generic
BLOB responses because if the response is streaming, then we have to
strictly evaluate the stream to determine the length. For returning large items (e.g., a 2 GB AVI), this can cause issues. We break the Response
types into "In Memory" which are strict and we include the
Content-Length
header, but for responses that are not strict, it's up to the coder. My 2 cents. Nubis was referring specifically to sending files. Yesod has a similar
breakdown where some datatypes (strict ByteStrings for instance) have
a content-length attached, while others (lazy ByteStrings) do not. We
achieve this via the optional Int argument in the ContentBuilder
constructor[1]. Various instances of ToContent simply handle this
differently. In the case of files, we would either use the length specified by the
FilePart parameter, or simply get the file size from the file system. Sorry for the newbie mistake and thanks for correction me! Rock on. Michael [1]
http://hackage.haskell.org/packages/archive/yesod-core/0.8.2/doc/html/Yesod-... Michael On Tue, Jun 14, 2011 at 8:16 PM, Nubis Hi guys,
Just wanted to give you a heads up that yesod's sendFile is not the
Content-Length header, which 'breaks' some proxies and load
balancers.
(in
our case, amazon elastic load balancer).
I don't know if that's by design, I'm not 100% sure but not setting header would be only useful for 'long pooling' of binary files for
doing
some sort of hackish http streaming, which I don't think is yesod's
purpose
anyways :)
A workaround is setting the Content-Length header before calling
sendFile, I
guess it may be something easy to add, I was going to propose a patch
that
just wraps the current implementation in a do block that sets the
header,
but then I noticed a number of optimizations being made that hint setting
the
there should be a 'better' way. cheers!
----nubis :) _______________________________________________
web-devel mailing list
web-devel@haskell.org
http://www.haskell.org/mailman/listinfo/web-devel _______________________________________________
web-devel mailing list
web-devel@haskell.org
http://www.haskell.org/mailman/listinfo/web-devel --
Lift, the simply functional web framework http://liftweb.net
Simply Lift http://simply.liftweb.net
Follow me: http://twitter.com/dpp
Blog: http://goodstuff.im --
Lift, the simply functional web framework http://liftweb.net
Simply Lift http://simply.liftweb.net
Follow me: http://twitter.com/dpp
Blog: http://goodstuff.im _______________________________________________
web-devel mailing list
web-devel@haskell.org
http://www.haskell.org/mailman/listinfo/web-devel

Hello,
I think the reason we didn't include the header is because, when I wrote it, I didn't know of a cross-platform way to get file sizes. Now that I know about unix-compat, this seems like a reasonable thing to add. Anyone know a reason we *shouldn't* do this?
For high performance web servers, system calls are the bottleneck. So, Warp or its libraries should avoid unnecessary system calls. Wai applications HAVE TO call a system call (stat()) to get file information (for instance, modification time) and can get a file size at the same time. So, they can add Content-Length: with one system call. If Warp or its libraries add Content-Length:, they need to call as system call again. It gives really bad performance impact. So, I support the current way of Warp. --Kazu

On Tue, Jun 14, 2011 at 6:26 PM, Kazu Yamamoto
Hello,
I think the reason we didn't include the header is because, when I wrote it, I didn't know of a cross-platform way to get file sizes. Now that I know about unix-compat, this seems like a reasonable thing to add. Anyone know a reason we *shouldn't* do this?
For high performance web servers, system calls are the bottleneck. So, Warp or its libraries should avoid unnecessary system calls.
Wai applications HAVE TO call a system call (stat()) to get file information (for instance, modification time) and can get a file size at the same time. So, they can add Content-Length: with one system call.
There are 2 basic kinds of WAI apps- those for serving files that might change (a file server), and those that know exactly what files exist in advance (a web application). In the case of a web application, we can get all this information just once at compile time. For a file server, we could allow a configuration parameter for whether or not to do the system call. Of course, it is possible to mix these 2 use cases in one application, but that can be handled with multiple instances of wai-app-static.
If Warp or its libraries add Content-Length:, they need to call as system call again. It gives really bad performance impact.
So, I support the current way of Warp.
--Kazu
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

Hello Greg,
Wai applications HAVE TO call a system call (stat()) to get file information (for instance, modification time) and can get a file size at the same time. So, they can add Content-Length: with one system call.
There are 2 basic kinds of WAI apps- those for serving files that might change (a file server), and those that know exactly what files exist in advance (a web application). In the case of a web application, we can get all this information just once at compile time. For a file server, we could allow a configuration parameter for whether or not to do the system call. Of course, it is possible to mix these 2 use cases in one application, but that can be handled with multiple instances of wai-app-static.
Yes, I know. The former uses ResponseFile and the latter typically uses ResponseBuilder/ResponseEnumerator. I discussed in the context of ResponseFile which uses sendfile. Sorry but I don't understand what you try to explain here. --Kazu

Hi Kazu, I apologize for the confusing terminology. I am not differentiating between sending a static file with sendfile and a streaming response. I am differentiating between 2 different use cases for sending static files (with sendfile). For all of my web applications, I know what all the static files are and they will never change until I deploy another web application. That means I can stat the files once when the application is deployed and keep that information in memory. So I already have the file length information to include in the header, even though I don't do a file stat when the file is requested. wai-app-static and yesod-static supports these techniques. I hope that is clearer. Thanks, Greg
Yes, I know. The former uses ResponseFile and the latter typically uses ResponseBuilder/ResponseEnumerator. I discussed in the context of ResponseFile which uses sendfile.
Sorry but I don't understand what you try to explain here.
--Kazu
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

Greg,
I apologize for the confusing terminology. I am not differentiating between sending a static file with sendfile and a streaming response. I am differentiating between 2 different use cases for sending static files (with sendfile). For all of my web applications, I know what all the static files are and they will never change until I deploy another web application. That means I can stat the files once when the application is deployed and keep that information in memory. So I already have the file length information to include in the header, even though I don't do a file stat when the file is requested. wai-app-static and yesod-static supports these techniques.
Thanks. I think I understand. :) So, do you support to *not* change the API (apps should add CL: by themselves)? --Kazu

Let me point out one other distinction: sendFile versus yesod-static.
The former is a function you would call from a normal handler, while
yesod-static is the "magical" package which would actually know all of
the stuff about your files at compile time. For the sendFile case, the
only options for getting the file size are (1) the programmer manually
adding the header and (2) Yesod automatically doing a system call to
get it.
As for the behavior of Warp... while I agree Kazu that we should
reduce system call overhead, it might make sense for Warp to perform
the system call to get file size *if* no content-length header is
present.
And I'm still very uncomfortable setting the content-length header for
static files based on compile-time information. In this case, having
the wrong value (e.g., someone modified a CSS file after compiling)
will completely break things and corrupt an open HTTP connection.
Michael
On Wed, Jun 15, 2011 at 5:22 AM, Kazu Yamamoto
Greg,
I apologize for the confusing terminology. I am not differentiating between sending a static file with sendfile and a streaming response. I am differentiating between 2 different use cases for sending static files (with sendfile). For all of my web applications, I know what all the static files are and they will never change until I deploy another web application. That means I can stat the files once when the application is deployed and keep that information in memory. So I already have the file length information to include in the header, even though I don't do a file stat when the file is requested. wai-app-static and yesod-static supports these techniques.
Thanks. I think I understand. :)
So, do you support to *not* change the API (apps should add CL: by themselves)?
--Kazu
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

Hi Michael,
As for the behavior of Warp... while I agree Kazu that we should reduce system call overhead, it might make sense for Warp to perform the system call to get file size *if* no content-length header is present.
This might be a good idea. I'm fine if there is a way that applications can explicitly control it and the behavior is clearly documented. :) --Kazu

For the record, the original post was about dynamic files, not the kind of static files served up by Yesod.Static. I put some comments in the haddocks on github. I think thats a fair place to start. max On Jun 15, 2011, at 10:29 AM, Michael Snoyman wrote:
Let me point out one other distinction: sendFile versus yesod-static. The former is a function you would call from a normal handler, while yesod-static is the "magical" package which would actually know all of the stuff about your files at compile time. For the sendFile case, the only options for getting the file size are (1) the programmer manually adding the header and (2) Yesod automatically doing a system call to get it.
As for the behavior of Warp... while I agree Kazu that we should reduce system call overhead, it might make sense for Warp to perform the system call to get file size *if* no content-length header is present.
And I'm still very uncomfortable setting the content-length header for static files based on compile-time information. In this case, having the wrong value (e.g., someone modified a CSS file after compiling) will completely break things and corrupt an open HTTP connection.
Michael
On Wed, Jun 15, 2011 at 5:22 AM, Kazu Yamamoto
wrote: Greg,
I apologize for the confusing terminology. I am not differentiating between sending a static file with sendfile and a streaming response. I am differentiating between 2 different use cases for sending static files (with sendfile). For all of my web applications, I know what all the static files are and they will never change until I deploy another web application. That means I can stat the files once when the application is deployed and keep that information in memory. So I already have the file length information to include in the header, even though I don't do a file stat when the file is requested. wai-app-static and yesod-static supports these techniques.
Thanks. I think I understand. :)
So, do you support to *not* change the API (apps should add CL: by themselves)?
--Kazu
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

On Tue, Jun 14, 2011 at 7:29 PM, Michael Snoyman
Let me point out one other distinction: sendFile versus yesod-static. The former is a function you would call from a normal handler, while yesod-static is the "magical" package which would actually know all of the stuff about your files at compile time. For the sendFile case, the only options for getting the file size are (1) the programmer manually adding the header and (2) Yesod automatically doing a system call to get it.
As for the behavior of Warp... while I agree Kazu that we should reduce system call overhead, it might make sense for Warp to perform the system call to get file size *if* no content-length header is present.
And I'm still very uncomfortable setting the content-length header for static files based on compile-time information. In this case, having the wrong value (e.g., someone modified a CSS file after compiling) will completely break things and corrupt an open HTTP connection.
Then it should probably not be the default. Lets add a big scary warning to such a setting and tell users the files should be set to read-only permission. The most efficient technique for file serving static files that could be changed would actually be to setup a file notifier (that uses an efficient OS listener, like inotify on linux) that listens to static assets and knows when they are changed or a new one is added and would stat them just when changed. But there would be a race condition if you had to wait for the notification, so you would actually have to have the old file revision on hand- perhaps having a symlink convention for adding new files, but meaning the system could still get screwed up by someone careless.
Michael
On Wed, Jun 15, 2011 at 5:22 AM, Kazu Yamamoto
wrote: Greg,
I apologize for the confusing terminology. I am not differentiating between sending a static file with sendfile and a streaming response. I am differentiating between 2 different use cases for sending static files (with sendfile). For all of my web applications, I know what all the static files are and they will never change until I deploy another web application. That means I can stat the files once when the application is deployed and keep that information in memory. So I already have the file length information to include in the header, even though I don't do a file stat when the file is requested. wai-app-static and yesod-static supports these techniques.
Thanks. I think I understand. :)
So, do you support to *not* change the API (apps should add CL: by themselves)?
--Kazu
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

.Then it should probably not be the default. Lets add a big scary warning to such a setting and tell users the files should be set to read-only permission.
Agreed. seems to be completely counter to the intended use of static..
The most efficient technique for file serving static files that could be changed would actually be to setup a file notifier (that uses an efficient OS listener, like inotify on linux) that listens to static assets and knows when they are changed or a new one is added and would stat them just when changed. But there would be a race condition if you had to wait for the notification, so you would actually have to have the old file revision on hand- perhaps having a symlink convention for adding new files, but meaning the system could still get screwed up by someone careless.
Michael
On Wed, Jun 15, 2011 at 5:22 AM, Kazu Yamamoto
wrote: Greg,
I apologize for the confusing terminology. I am not differentiating between sending a static file with sendfile and a streaming response. I am differentiating between 2 different use cases for sending static files (with sendfile). For all of my web applications, I know what all the static files are and they will never change until I deploy another web application. That means I can stat the files once when the application is deployed and keep that information in memory. So I already have the file length information to include in the header, even though I don't do a file stat when the file is requested. wai-app-static and yesod-static supports these techniques.
Thanks. I think I understand. :)
So, do you support to *not* change the API (apps should add CL: by themselves)?
--Kazu
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

On Wed, Jun 15, 2011 at 5:57 AM, Greg Weber
On Tue, Jun 14, 2011 at 7:29 PM, Michael Snoyman
wrote: Let me point out one other distinction: sendFile versus yesod-static. The former is a function you would call from a normal handler, while yesod-static is the "magical" package which would actually know all of the stuff about your files at compile time. For the sendFile case, the only options for getting the file size are (1) the programmer manually adding the header and (2) Yesod automatically doing a system call to get it.
As for the behavior of Warp... while I agree Kazu that we should reduce system call overhead, it might make sense for Warp to perform the system call to get file size *if* no content-length header is present.
And I'm still very uncomfortable setting the content-length header for static files based on compile-time information. In this case, having the wrong value (e.g., someone modified a CSS file after compiling) will completely break things and corrupt an open HTTP connection.
Then it should probably not be the default. Lets add a big scary warning to such a setting and tell users the files should be set to read-only permission. The most efficient technique for file serving static files that could be changed would actually be to setup a file notifier (that uses an efficient OS listener, like inotify on linux) that listens to static assets and knows when they are changed or a new one is added and would stat them just when changed. But there would be a race condition if you had to wait for the notification, so you would actually have to have the old file revision on hand- perhaps having a symlink convention for adding new files, but meaning the system could still get screwed up by someone careless.
I'm really starting to think that the correct solution for this use case really is embedded files[1]. It avoids any kind of system call overhead, doesn't touch the hard drive at all, and doesn't allow for modification of the files after compile time. It would be interesting to see if it ends up faster than sendfile. Michael [1] Using Template Haskell to store the entire static file inside the executable during compilation.
Michael
On Wed, Jun 15, 2011 at 5:22 AM, Kazu Yamamoto
wrote: Greg,
I apologize for the confusing terminology. I am not differentiating between sending a static file with sendfile and a streaming response. I am differentiating between 2 different use cases for sending static files (with sendfile). For all of my web applications, I know what all the static files are and they will never change until I deploy another web application. That means I can stat the files once when the application is deployed and keep that information in memory. So I already have the file length information to include in the header, even though I don't do a file stat when the file is requested. wai-app-static and yesod-static supports these techniques.
Thanks. I think I understand. :)
So, do you support to *not* change the API (apps should add CL: by themselves)?
--Kazu
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

Changed the subject since we're pretty far from the OP. I think embedding files is a fantastic idea. Also, this would make deployment a bit easier since an entire app can be packaged into a single executable. max On Jun 15, 2011, at 11:20 AM, Michael Snoyman wrote:
On Wed, Jun 15, 2011 at 5:57 AM, Greg Weber
wrote: On Tue, Jun 14, 2011 at 7:29 PM, Michael Snoyman
wrote: Let me point out one other distinction: sendFile versus yesod-static. The former is a function you would call from a normal handler, while yesod-static is the "magical" package which would actually know all of the stuff about your files at compile time. For the sendFile case, the only options for getting the file size are (1) the programmer manually adding the header and (2) Yesod automatically doing a system call to get it.
As for the behavior of Warp... while I agree Kazu that we should reduce system call overhead, it might make sense for Warp to perform the system call to get file size *if* no content-length header is present.
And I'm still very uncomfortable setting the content-length header for static files based on compile-time information. In this case, having the wrong value (e.g., someone modified a CSS file after compiling) will completely break things and corrupt an open HTTP connection.
Then it should probably not be the default. Lets add a big scary warning to such a setting and tell users the files should be set to read-only permission. The most efficient technique for file serving static files that could be changed would actually be to setup a file notifier (that uses an efficient OS listener, like inotify on linux) that listens to static assets and knows when they are changed or a new one is added and would stat them just when changed. But there would be a race condition if you had to wait for the notification, so you would actually have to have the old file revision on hand- perhaps having a symlink convention for adding new files, but meaning the system could still get screwed up by someone careless.
I'm really starting to think that the correct solution for this use case really is embedded files[1]. It avoids any kind of system call overhead, doesn't touch the hard drive at all, and doesn't allow for modification of the files after compile time. It would be interesting to see if it ends up faster than sendfile.
Michael
[1] Using Template Haskell to store the entire static file inside the executable during compilation.
Michael
On Wed, Jun 15, 2011 at 5:22 AM, Kazu Yamamoto
wrote: Greg,
I apologize for the confusing terminology. I am not differentiating between sending a static file with sendfile and a streaming response. I am differentiating between 2 different use cases for sending static files (with sendfile). For all of my web applications, I know what all the static files are and they will never change until I deploy another web application. That means I can stat the files once when the application is deployed and keep that information in memory. So I already have the file length information to include in the header, even though I don't do a file stat when the file is requested. wai-app-static and yesod-static supports these techniques.
Thanks. I think I understand. :)
So, do you support to *not* change the API (apps should add CL: by themselves)?
--Kazu
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

I'm really starting to think that the correct solution for this use case really is embedded files[1]. It avoids any kind of system call overhead, doesn't touch the hard drive at all, and doesn't allow for modification of the files after compile time. It would be interesting to see if it ends up faster than sendfile.
This is just my idea. Not confirmed. The server saves all embedded files to the file sytem on boot time and obtains file descriptors for them. And it *unlink* files so that nobody but the server can touch. Then the server can send the file with sendfile. (sendfile API should take FD not FilePath) --Kazu

On Wed, Jun 15, 2011 at 6:51 AM, Kazu Yamamoto
I'm really starting to think that the correct solution for this use case really is embedded files[1]. It avoids any kind of system call overhead, doesn't touch the hard drive at all, and doesn't allow for modification of the files after compile time. It would be interesting to see if it ends up faster than sendfile.
This is just my idea. Not confirmed.
The server saves all embedded files to the file sytem on boot time and obtains file descriptors for them. And it *unlink* files so that nobody but the server can touch. Then the server can send the file with sendfile. (sendfile API should take FD not FilePath)
Is this to get the increased performance of the sendfile system call? I very much doubt it would work out faster. With embedded files, the entire static file is stored as a contiguous chunk of memory. It's worth an analysis before recommending this approach, of coruse. Michael
--Kazu
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

Hello,
The server saves all embedded files to the file sytem on boot time and obtains file descriptors for them. And it *unlink* files so that nobody but the server can touch. Then the server can send the file with sendfile. (sendfile API should take FD not FilePath)
Is this to get the increased performance of the sendfile system call? I very much doubt it would work out faster. With embedded files, the entire static file is stored as a contiguous chunk of memory. It's worth an analysis before recommending this approach, of coruse.
If we use send() to send a chunk of memory, the data is copied from the user space to the kernel space. If we use sendfile(), the file is mapped to the kernel space and no copy happens and the pages are probably cached. But this approach cosumes double memory. But I'm not sure this guess is true. If we can map the user space to the kernel space and can feed the kernel space to a socket, it's fastest. But I don't know such system calls. --Kazu

wouldn't the overhead of reading from the drive dominate the overhead of copying the contents from userspace to kernel space? max On Jun 15, 2011, at 3:10 PM, Kazu Yamamoto (山本和彦) wrote:
Hello,
The server saves all embedded files to the file sytem on boot time and obtains file descriptors for them. And it *unlink* files so that nobody but the server can touch. Then the server can send the file with sendfile. (sendfile API should take FD not FilePath)
Is this to get the increased performance of the sendfile system call? I very much doubt it would work out faster. With embedded files, the entire static file is stored as a contiguous chunk of memory. It's worth an analysis before recommending this approach, of coruse.
If we use send() to send a chunk of memory, the data is copied from the user space to the kernel space.
If we use sendfile(), the file is mapped to the kernel space and no copy happens and the pages are probably cached. But this approach cosumes double memory.
But I'm not sure this guess is true.
If we can map the user space to the kernel space and can feed the kernel space to a socket, it's fastest. But I don't know such system calls.
--Kazu
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

On Wed, Jun 15, 2011 at 9:16 AM, Kazu Yamamoto
wouldn't the overhead of reading from the drive dominate the overhead of copying the contents from userspace to kernel space?
Probably yes at the first time. But I'm not sure after the file is cached.
If the file is in buffer cache, sendfile should be faster, it's zero-copy.
Re: sendfile, we solve this issue by making it impossible to issue a
sendfile() without sending either a file length or a byte range. Our
file serving code only calls stat() once, of course -- I agree with
Kazu, more than once I would probably consider a bug.
G
--
Gregory Collins

Hello, I was following the "Yesod in Five Minutes Instructions" and I had a couple of bumps in the road. The first one was when trying to do step 6, I get the following terminal output: unrecognized option `--only-dependencies' After ignoring that for the moment, I proceeded to trying to run the devel server and I get this terminal output: yesod: user error (At least the following dependencies are missing: persistent-postgresql ==0.5.*) I'm guessing the second error may be related to the first, maybe persistent-postgresql was one of those dependencies that was supposed to be installed? I did create the Postgres databases and user specified in the Yesod app config file. I'm using this PPA on Ubuntu 11.04 64-bit, as the repo version of the Haskell Platform seems to be broken: https://launchpad.net/~brcha/+archive/ppa Thanks for any insight.

On Thu, Jun 16, 2011 at 12:46 AM, anthropornis
Hello,
I was following the "Yesod in Five Minutes Instructions" and I had a couple of bumps in the road.
The first one was when trying to do step 6, I get the following terminal output: unrecognized option `--only-dependencies'
After ignoring that for the moment, I proceeded to trying to run the devel server and I get this terminal output: yesod: user error (At least the following dependencies are missing: persistent-postgresql ==0.5.*)
I'm guessing the second error may be related to the first, maybe persistent-postgresql was one of those dependencies that was supposed to be installed?
I did create the Postgres databases and user specified in the Yesod app config file.
I'm using this PPA on Ubuntu 11.04 64-bit, as the repo version of the Haskell Platform seems to be broken: https://launchpad.net/~brcha/+archive/ppa
Thanks for any insight.
Try just plain "cabal install", the "--only-dependencies" is available in newer versions of the "cabal" tool. I'm going to remove it from the docs. Michael

(I forgot to mention earlier, I'm also new to Haskell, sorta starting with Yesod first to just get something running, then going on to learning Haskell, so I can get rid of a PHP CRUD interface for an existing db.) I tried cabal install without any flags and it eventually worked. Initially cabal yielded this error: cabal: Error: some packages failed to install: HDBC-postgresql-2.2.3.3 failed during the configure step. The exception was: ExitFailure 1 persistent-postgresql-0.5.0 depends on HDBC-postgresql-2.2.3.3 which failed to install. sample-0.0.0 depends on HDBC-postgresql-2.2.3.3 which failed to install. Reading further up in the terminal output I saw something mentioned about one package needing to be installed for server side extensions, or libpq-dev needing to be installed for client side applications. I took a chance that libpq-dev would be the right choice and installed it, then ran cabal install again, and this time it worked, and I was able to launch the dev server. Viewing the page in my web browser I attempted to add an email login, but nothing happened. Switching back to the terminal it said sendmail needed to be installed. After installing sendmail I was able to create a login. I had one question about sendmail. I noted somewhere else in your docs that I could possibly do something like Adobe AIR (?) and make an executable with Webkit in it as an offline version of the web app (which would be really cool, by the way). I wasn't sure, if I put that executable on someone's Windows machine, is sendmail available for Windows or can the email pieces be switched with something other than sendmail if needed? Thanks again On 06/15/2011 10:52 PM, Michael Snoyman wrote:
On Thu, Jun 16, 2011 at 12:46 AM, anthropornis
wrote: Hello,
I was following the "Yesod in Five Minutes Instructions" and I had a couple of bumps in the road.
The first one was when trying to do step 6, I get the following terminal output: unrecognized option `--only-dependencies'
After ignoring that for the moment, I proceeded to trying to run the devel server and I get this terminal output: yesod: user error (At least the following dependencies are missing: persistent-postgresql ==0.5.*)
I'm guessing the second error may be related to the first, maybe persistent-postgresql was one of those dependencies that was supposed to be installed?
I did create the Postgres databases and user specified in the Yesod app config file.
I'm using this PPA on Ubuntu 11.04 64-bit, as the repo version of the Haskell Platform seems to be broken: https://launchpad.net/~brcha/+archive/ppa
Thanks for any insight.
Try just plain "cabal install", the "--only-dependencies" is available in newer versions of the "cabal" tool. I'm going to remove it from the docs.
Michael

Can you add a Wiki page[1] with your install experiences? I'll link to
it from the "five minutes" page.
As for the sendmail issue: I'm actually going to need a backend for
Amazon SES for sending email, so I'd like to get that written. (Any
volunteers? Aristid already has a great aws package that should do
most of the heavy lifting.[2]) There's also haskellnet which uses
SMTP, but I haven't used it.
Michael
[1] http://www.yesodweb.com/page/wiki
[2] http://hackage.haskell.org/package/aws
On Thu, Jun 16, 2011 at 5:23 PM, anthropornis
(I forgot to mention earlier, I'm also new to Haskell, sorta starting with Yesod first to just get something running, then going on to learning Haskell, so I can get rid of a PHP CRUD interface for an existing db.)
I tried cabal install without any flags and it eventually worked.
Initially cabal yielded this error: cabal: Error: some packages failed to install: HDBC-postgresql-2.2.3.3 failed during the configure step. The exception was: ExitFailure 1 persistent-postgresql-0.5.0 depends on HDBC-postgresql-2.2.3.3 which failed to install. sample-0.0.0 depends on HDBC-postgresql-2.2.3.3 which failed to install.
Reading further up in the terminal output I saw something mentioned about one package needing to be installed for server side extensions, or libpq-dev needing to be installed for client side applications.
I took a chance that libpq-dev would be the right choice and installed it, then ran cabal install again, and this time it worked, and I was able to launch the dev server.
Viewing the page in my web browser I attempted to add an email login, but nothing happened. Switching back to the terminal it said sendmail needed to be installed. After installing sendmail I was able to create a login.
I had one question about sendmail. I noted somewhere else in your docs that I could possibly do something like Adobe AIR (?) and make an executable with Webkit in it as an offline version of the web app (which would be really cool, by the way). I wasn't sure, if I put that executable on someone's Windows machine, is sendmail available for Windows or can the email pieces be switched with something other than sendmail if needed?
Thanks again
On 06/15/2011 10:52 PM, Michael Snoyman wrote:
On Thu, Jun 16, 2011 at 12:46 AM, anthropornis
wrote: Hello,
I was following the "Yesod in Five Minutes Instructions" and I had a couple of bumps in the road.
The first one was when trying to do step 6, I get the following terminal output: unrecognized option `--only-dependencies'
After ignoring that for the moment, I proceeded to trying to run the devel server and I get this terminal output: yesod: user error (At least the following dependencies are missing: persistent-postgresql ==0.5.*)
I'm guessing the second error may be related to the first, maybe persistent-postgresql was one of those dependencies that was supposed to be installed?
I did create the Postgres databases and user specified in the Yesod app config file.
I'm using this PPA on Ubuntu 11.04 64-bit, as the repo version of the Haskell Platform seems to be broken: https://launchpad.net/~brcha/+archive/ppa
Thanks for any insight.
Try just plain "cabal install", the "--only-dependencies" is available in newer versions of the "cabal" tool. I'm going to remove it from the docs.
Michael

I updated the install page with notes for these issues:
http://www.yesodweb.com/page/five-minutes
Let us know if you think that would have solved your problems. If not, we
can link to a more in depth wiki page.
I believe Neil said the HaskellNet package wasn't compiling on ghc7 but that
it shouldn't be hard to fix and that he might do it. This package should
allow you to send email on Windows.
Greg Weber
On Thu, Jun 16, 2011 at 7:34 AM, Michael Snoyman
Can you add a Wiki page[1] with your install experiences? I'll link to it from the "five minutes" page.
As for the sendmail issue: I'm actually going to need a backend for Amazon SES for sending email, so I'd like to get that written. (Any volunteers? Aristid already has a great aws package that should do most of the heavy lifting.[2]) There's also haskellnet which uses SMTP, but I haven't used it.
Michael
[1] http://www.yesodweb.com/page/wiki [2] http://hackage.haskell.org/package/aws
(I forgot to mention earlier, I'm also new to Haskell, sorta starting with Yesod first to just get something running, then going on to learning Haskell, so I can get rid of a PHP CRUD interface for an existing db.)
I tried cabal install without any flags and it eventually worked.
Initially cabal yielded this error: cabal: Error: some packages failed to install: HDBC-postgresql-2.2.3.3 failed during the configure step. The exception was: ExitFailure 1 persistent-postgresql-0.5.0 depends on HDBC-postgresql-2.2.3.3 which failed to install. sample-0.0.0 depends on HDBC-postgresql-2.2.3.3 which failed to install.
Reading further up in the terminal output I saw something mentioned about one package needing to be installed for server side extensions, or
needing to be installed for client side applications.
I took a chance that libpq-dev would be the right choice and installed it, then ran cabal install again, and this time it worked, and I was able to launch the dev server.
Viewing the page in my web browser I attempted to add an email login, but nothing happened. Switching back to the terminal it said sendmail needed to be installed. After installing sendmail I was able to create a login.
I had one question about sendmail. I noted somewhere else in your docs
I could possibly do something like Adobe AIR (?) and make an executable with Webkit in it as an offline version of the web app (which would be really cool, by the way). I wasn't sure, if I put that executable on someone's Windows machine, is sendmail available for Windows or can the email
On Thu, Jun 16, 2011 at 5:23 PM, anthropornis
wrote: libpq-dev that pieces be switched with something other than sendmail if needed?
Thanks again
On 06/15/2011 10:52 PM, Michael Snoyman wrote:
On Thu, Jun 16, 2011 at 12:46 AM, anthropornis
wrote: Hello,
I was following the "Yesod in Five Minutes Instructions" and I had a couple of bumps in the road.
The first one was when trying to do step 6, I get the following
terminal
output: unrecognized option `--only-dependencies'
After ignoring that for the moment, I proceeded to trying to run the devel server and I get this terminal output: yesod: user error (At least the following dependencies are missing: persistent-postgresql ==0.5.*)
I'm guessing the second error may be related to the first, maybe persistent-postgresql was one of those dependencies that was supposed to be installed?
I did create the Postgres databases and user specified in the Yesod app config file.
I'm using this PPA on Ubuntu 11.04 64-bit, as the repo version of the Haskell Platform seems to be broken: https://launchpad.net/~brcha/+archive/ppa
Thanks for any insight.
Try just plain "cabal install", the "--only-dependencies" is available in newer versions of the "cabal" tool. I'm going to remove it from the docs.
Michael
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

My opinion is tainted by hindsight now, but the "five minutes" page as is now would probably be sufficient :) I guess it really depends on the target demographic (experienced programmers or total newbies, et al). I would love to see more use of Haskell as the first "learn to program" language (going on a tangent here, sorry). On 06/16/2011 10:47 AM, Greg Weber wrote:
I updated the install page with notes for these issues: http://www.yesodweb.com/page/five-minutes Let us know if you think that would have solved your problems. If not, we can link to a more in depth wiki page.
I believe Neil said the HaskellNet package wasn't compiling on ghc7 but that it shouldn't be hard to fix and that he might do it. This package should allow you to send email on Windows.
Greg Weber
On Thu, Jun 16, 2011 at 7:34 AM, Michael Snoyman
mailto:michael@snoyman.com> wrote: Can you add a Wiki page[1] with your install experiences? I'll link to it from the "five minutes" page.
As for the sendmail issue: I'm actually going to need a backend for Amazon SES for sending email, so I'd like to get that written. (Any volunteers? Aristid already has a great aws package that should do most of the heavy lifting.[2]) There's also haskellnet which uses SMTP, but I haven't used it.
Michael
[1] http://www.yesodweb.com/page/wiki [2] http://hackage.haskell.org/package/aws
On Thu, Jun 16, 2011 at 5:23 PM, anthropornis
mailto:anthropornis@gmail.com> wrote: > (I forgot to mention earlier, I'm also new to Haskell, sorta starting with > Yesod first to just get something running, then going on to learning > Haskell, so I can get rid of a PHP CRUD interface for an existing db.) > > I tried cabal install without any flags and it eventually worked. > > Initially cabal yielded this error: > cabal: Error: some packages failed to install: > HDBC-postgresql-2.2.3.3 failed during the configure step. The exception > was: > ExitFailure 1 > persistent-postgresql-0.5.0 depends on HDBC-postgresql-2.2.3.3 which > failed to install. > sample-0.0.0 depends on HDBC-postgresql-2.2.3.3 which failed to install. > > > Reading further up in the terminal output I saw something mentioned about > one package needing to be installed for server side extensions, or libpq-dev > needing to be installed for client side applications. > > I took a chance that libpq-dev would be the right choice and installed it, > then ran cabal install again, and this time it worked, and I was able to > launch the dev server. > > Viewing the page in my web browser I attempted to add an email login, but > nothing happened. Switching back to the terminal it said sendmail needed to > be installed. After installing sendmail I was able to create a login. > > I had one question about sendmail. I noted somewhere else in your docs that > I could possibly do something like Adobe AIR (?) and make an executable with > Webkit in it as an offline version of the web app (which would be really > cool, by the way). I wasn't sure, if I put that executable on someone's > Windows machine, is sendmail available for Windows or can the email pieces > be switched with something other than sendmail if needed? > > Thanks again > > > On 06/15/2011 10:52 PM, Michael Snoyman wrote: >> >> On Thu, Jun 16, 2011 at 12:46 AM, anthropornis mailto:anthropornis@gmail.com> >> wrote: >>> >>> Hello, >>> >>> I was following the "Yesod in Five Minutes Instructions" and I had a >>> couple >>> of bumps in the road. >>> >>> The first one was when trying to do step 6, I get the following terminal >>> output: >>> unrecognized option `--only-dependencies' >>> >>> After ignoring that for the moment, I proceeded to trying to run the >>> devel >>> server and I get this terminal output: >>> yesod: user error (At least the following dependencies are missing: >>> persistent-postgresql ==0.5.*) >>> >>> I'm guessing the second error may be related to the first, maybe >>> persistent-postgresql was one of those dependencies that was supposed to >>> be >>> installed? >>> >>> I did create the Postgres databases and user specified in the Yesod app >>> config file. >>> >>> I'm using this PPA on Ubuntu 11.04 64-bit, as the repo version of the >>> Haskell Platform seems to be broken: >>> https://launchpad.net/~brcha/+archive/ppa https://launchpad.net/%7Ebrcha/+archive/ppa >>> >>> Thanks for any insight. >>> >> Try just plain "cabal install", the "--only-dependencies" is available >> in newer versions of the "cabal" tool. I'm going to remove it from the >> docs. >> >> Michael > > _______________________________________________ web-devel mailing list web-devel@haskell.org mailto:web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

I think thats a very good point. Based on this, I'd agree that perhaps we shouldn't change sendFile. But, I still support a note in the haddocks and possibly adding a second function as in my previous post. max On Jun 15, 2011, at 9:26 AM, Kazu Yamamoto (山本和彦) wrote:
Hello,
I think the reason we didn't include the header is because, when I wrote it, I didn't know of a cross-platform way to get file sizes. Now that I know about unix-compat, this seems like a reasonable thing to add. Anyone know a reason we *shouldn't* do this?
For high performance web servers, system calls are the bottleneck. So, Warp or its libraries should avoid unnecessary system calls.
Wai applications HAVE TO call a system call (stat()) to get file information (for instance, modification time) and can get a file size at the same time. So, they can add Content-Length: with one system call.
If Warp or its libraries add Content-Length:, they need to call as system call again. It gives really bad performance impact.
So, I support the current way of Warp.
--Kazu
_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

I think thats a very good point. Based on this, I'd agree that perhaps we shouldn't change sendFile. But, I still support a note in the haddocks and possibly adding a second function as in my previous post.
Yes, its documentation should be improved. For my case, I understood that upper applications must add CT: from Warp's source code. It should be clearly explained in doc. --Kazu
participants (8)
-
anthropornis
-
David Pollak
-
Greg Weber
-
Gregory Collins
-
Kazu Yamamoto
-
Max Cantor
-
Michael Snoyman
-
Nubis