On 2008 Aug 13, at 15:04, Brandon S. Allbery KF8NH wrote:

On 2008 Aug 13, at 15:01, Chaddaï Fouché wrote:

...

2008/8/13 Jason Dusek :

...

I found an old lib for it:

http://www.haskell.org/ghc/docs/6.0/html/unix/System.Sendfile.html

Hoogle turns up nothing, though.

That don't sound very useful... Maybe when we only had String it was much more performant for big transfert, but now we can recode this in one short line of ByteString code and get the same performance as C.

sendfile() is actually a system call, not a library function.

I should clarify: what sendfile() is supposed to optimize isn't writing large strings, or even the user<->kernel roundtrips; it's an optimization to the kernel network stack (network buffer management, to be specific). Web servers use it to serve static content (e.g. icons, images, stylesheets) because it significantly reduces system load. -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH

On 2008 Aug 13, at 15:26, Chaddaï Fouché wrote:

2008/8/13 Brandon S. Allbery KF8NH :

...

I should clarify: what sendfile() is supposed to optimize isn't writing large strings, or even the user<->kernel roundtrips; it's an optimization to the kernel network stack (network buffer management, to be specific). Web servers use it to serve static content (e.g. icons, images, stylesheets) because it significantly reduces system load.

Ok, so it could still be useful in a restricted area (but then it should be easy to write a FFI wrapper for it anyway).

Right. I intended that to be a heads-up in both directions: it is not simply a library convenience function, so one needs to think about when to use it. In particular, it's possible that overuse of sendfile() in the wrong circumstances will create additional system load instead of reducing it. -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH

On 2008 Aug 13, at 18:25, Jason Dusek wrote:

Brandon S. Allbery KF8NH wrote:

...

Right. I intended that to be a heads-up in both directions: it is not simply a library convenience function, so one needs to think about when to use it. In particular, it's possible that overuse of sendfile() in the wrong circumstances will create additional system load instead of reducing it.

Can you say more about this? I assume that sending static images back and forth is a good fit for sendfile().

Your previously stated use case sounds like a good fit. I can easily imagine sendfile() implementations starving other network operations, though (and IIRC linux's early sendfile() implementation did so). It's essentially an optimization specific to web servers. -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH

On 2008 Aug 14, at 2:28, Ketil Malde wrote:

"Brandon S. Allbery KF8NH" writes:

...

Your previously stated use case sounds like a good fit. I can easily imagine sendfile() implementations starving other network operations, though (and IIRC linux's early sendfile() implementation did so). It's essentially an optimization specific to web servers.

Isn't it superseeded by splice(2) nowadays?

Most Unixen have some version of sendfile(); how portable is splice()? (Of course, there could be a system-dependent mapping; one is needed for Windows already.) -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH

On 2008 Aug 14, at 19:07, Richard A. O'Keefe wrote:

to me, two of them don't have it, and the third seems to be saying you cannot use it to move data from a file (which is not a pipe) to a socket (which is not a pipe), which is the use-case for sendfile().

Actually, while I'm not sure how Linux does it, on the *BSDs pipes are actually socketpairs. -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH

On 2008 Aug 15, at 2:23, Richard A. O'Keefe wrote:

On 15 Aug 2008, at 12:17 pm, Brandon S. Allbery KF8NH wrote:

...

Actually, while I'm not sure how Linux does it, on the *BSDs pipes are actually socketpairs.

This raises the question, which the documentation did not make clear to me, whether a "named pipe" is a pipe. One would hope it was, but S_IFIFO and S_IFSOCK are different...

They have to be: bound AF_UNIX sockets and fifos have very different semantics. That said, fifos could still be a different socket type, just as pipes are a distinct socket type. -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH

On 2008 Aug 15, at 9:34, Tony Finch wrote:

On Thu, 14 Aug 2008, Brandon S. Allbery KF8NH wrote:

...

Actually, while I'm not sure how Linux does it, on the *BSDs pipes are actually socketpairs.

Not any more. FreeBSD replaced the socketpair implementation with a faster one in 1996 and OpenBSD imported it soon after. NetBSD imported it in 2001 and Mac OS X in 10.4 (2005). Dragonfly BSD forked from FreeBSD after the new pipe code was introduced.

Hm. Somehow I missed that. In any case, the real point is that historically (the socketpair stuff goes back to 4.2BSD) pipes were often considered a special case of sockets, so documentation referring to pipes might well actually mean or include sockets. Someone will have to inspect the Linux splice() implementation to see if it actually works on sockets. -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH

Chaddaï Fouché wrote:

Maybe when we only had String it was much more performant for big transfert, but now we can recode this in one short line of ByteString code and get the same performance as C.

Oh? Using lazy ByteString? -- _jsn

On Wed, Aug 13, 2008 at 03:40:43PM -0700, Jason Dusek wrote:

A cross platfrom implementation would cover the case we most care about -- writing services that pass static files back to clients -- but would have to cut some functionality from the *NIX sendfile().

There isn't a standard unix sendfile, while a few different ones have functions called 'sendfile', they have different meanings/prototypes in general. If 'sendfile(2)' is going to be exposed, it should be in a low level platform specific library, however, since you are dealing with deciding to use unportable behavior, it is hard to see what this will gain over a manual FFI wrapper. John -- John Meacham - ⑆repetae.net⑆john⑈

On 14 Aug 2008, at 10:47 am, John Meacham wrote:

There isn't a standard unix sendfile, while a few different ones have functions called 'sendfile', they have different meanings/prototypes in general.

For example, I'm typing this on an Intel Mac running Mac OS 10.5.4, and 'man sendfile' shows sendfile -- send a file to a socket and claims that it is checked runtime error if the destination is anything but a socket. It looks as though file -> socket is the only moderately portable case, and of course systems without can fake it by a sequence of reads and writes.