On Mon, Feb 28, 2011 at 11:20 AM, Bryan O'Sullivan wrote:

On Sun, Feb 27, 2011 at 9:18 PM, wren ng thornton wrote:

...

I've written variants of the System.Posix.IO API for strict and lazy ByteStrings which are currently lingering in their own unpublished package[1]. It's silly to have a separate package for just four functions, so I'd like to see it combined into the unix package. I don't have access to create a new ticket with the patch, but it's available at [2].

I think that the strict bytestring version should just replace the current function in System.Posix.IO, and the lazy version should not go in at all. This is the approach taken by the network package, and it's cleanest.

'network' has a lazy variant, which calls 'writev'. Antoine

On Mon, Feb 28, 2011 at 11:28 AM, Bryan O'Sullivan wrote:

On Mon, Feb 28, 2011 at 9:27 AM, Antoine Latter wrote:

...

'network' has a lazy variant, which calls 'writev'.

That's been replaced in the git tree with 'sendMany', which takes a list of strict bytestrings.

That's likely a lot less spooky than making the caller know if it's safe to force the spine of a lazy bytestring. Antoine

On 2/28/11 12:20 PM, Bryan O'Sullivan wrote:

I think that the strict bytestring version should just replace the current function in System.Posix.IO, and the lazy version should not go in at all. This is the approach taken by the network package, and it's cleanest.

I still see Network.Socket.ByteString.Lazy in the latest version network-2.3.0.2[1]. Am I missing something? [1] http://hackage.haskell.org/package/network -- Live well, ~wren

On 3/5/11 10:03 AM, Bas van Dijk wrote:

On 5 March 2011 13:30, wren ng thornton wrote:

...

This currently consists of: * System.Posix.IO.ByteString * fdRead -- like System.Posix.IO.fdRead, the same as before.

What about changing this:

fdRead :: Fd -> ByteCount -> IO (ByteString, ByteCount)

to this:

fdRead :: Fd -> ByteCount -> IO ByteString

I agree that it's nice to be consistent with the String version but since that version might go away and since the length (O(1)) of the ByteString should always equal the ByteCount I think it's better to remove that invariant.

I even have a note in the source file to that effect :) One thing I considered was to offer both versions, one for compatibility with the old string versions (and for decluttering client code)[1] and then one that just gives the bytestring. Of course, then the issue is what to name them... So the big question is: how minimal should we be, vs how much in the way of convenience functions should we offer? Update: In v0.2.1 I added an fdReads function which lets you pass a predicate for determining whether to retry after incomplete reads. -- Live well, ~wren

On 3/6/11 12:22 PM, Bas van Dijk wrote:

On 6 March 2011 13:16, wren ng thornton wrote:

...

One thing I considered was to offer both versions, one for compatibility with the old string versions (and for decluttering client code)[1] and then one that just gives the bytestring. Of course, then the issue is what to name them...

So the big question is: how minimal should we be, vs how much in the way of convenience functions should we offer?

It would be great if you could analyse the direct reverse dependencies of unix to see how much fdRead is used and how much work it is to adapt packages to an fdRead which only returns the ByteString:

http://bifunctor.homelinux.net/~roel/cgi-bin/hackage-scripts/revdeps/unix-2....

What an excellent idea! It looks like unix has about 234 direct revdeps. Of which, some hasty grepping detects only 26 files which may call fdRead or fdWrite. Namely: ./HFuse-0.2.3/System/Fuse.hsc ./bindings-bfd-0.2.0/src/Bindings/Bfd/Disasm.hs ./bindings-bfd-0.2.0/src/Bindings/Bfd/Symbol.hsc ./cautious-file-0.1.5/src/System/Posix/ByteLevel.hsc ./epoll-0.2.2/examples/Stdin.hs ./epoll-0.2.2/src/System/Linux/Epoll/Buffer.hs ./funion-0.0.2/Funion.hs ./halfs-0.2/Binary.hs ./halfs-0.2/System/RawDevice/File.hs ./iteratee-0.8.1.2/src/Data/Iteratee/IO/Fd.hs ./iteratee-0.8.1.2/src/Data/Iteratee/IO/Posix.hs ./iteratee-mtl-0.5.0.0/src/Data/Iteratee/IO/Fd.hs ./iteratee-mtl-0.5.0.0/src/Data/Iteratee/IO/Posix.hs ./lhc-0.10/lib/base/src/Control/Concurrent.hs ./lhc-0.10/lib/base/src/GHC/IO/FD.hs ./lhc-0.10/src/Grin/Eval/Primitives.hs ./lhc-0.10/src/Grin/Stage2/Backend/C.hs ./liboleg-2010.1.10.0/System/IterateeM.hs ./liboleg-2010.1.10.0/System/LowLevelIO.hs ./liboleg-2010.1.10.0/System/RandomIO.hs ./liboleg-2010.1.10.0/System/SysOpen.hs ./miniplex-0.3.4/lib/System/Miniplex/Sink.hs ./serialport-0.3.3/System/Hardware/Serialport/Posix.hsc ./vty-4.6.0.4/src/Graphics/Vty/LLInput.hs ./ztail-1.0.1/TailHandle.hs Of these, many are false positives due to some local function called fdReady (and ignored hereafter), many are false positives due to local definitions of fdRead, myfdRead, etc. (discussed later), and only about half a dozen appear to be direct calls to the unix package version. Of these half dozen files, most of the use sites completely ignore the ByteCount, two or three use it only to check (==0) which can be efficiently detected by either String or ByteString's null predicate, and only two of them appear to use it in any nontrivial way (e.g., returning it from the current function, or printing it). Thus, I conclude, nobody wants the ByteCount. The switch from String to ByteString would involve far more work than correcting literally a couple pattern matches per affected project. And less than half a dozen use sites across all of Hackage might consider calling BS.length to get at the information. Of the local definitions there were two classes I noticed. * The HFuse project defines their own bindings to the pread(2) and pwrite(2) functions, thus answering my question about whether anyone'd want them. They use them as handling ByteString buffers too, no less! * The liboleg, iteratee, and iteratee-mtl packages all have copies of their own versions of fdRead (apparently a cut&paste job). In the documentation they level numerous complaints against the unix package's version of fdRead. The two notable ones are (1) that fdRead allocates a new buffer every time it's called, this seems to be addressed by fdReadBuf but the packages haven't been updated to use it; and (2) that fdRead throws errors (at all, let alone on EOF). They'd prefer the type: myfdRead :: Fd -> Ptr CChar -> ByteCount -> IO (Either Errno ByteCount) Considering that fdReadBuf already addresses the first complaint, it seems that it might be worthwhile to provide a safe version of fdReadBuf which captures the error in an Either rather than throwing it (should be cheaper than throwing it and having a wrapper catch it and convert it into Errno?). Similarly, it may be worthwhile to have a version of fdRead which doesn't throw an exception on EOF, but just returns the empty ByteString instead. -- Live well, ~wren

On March 5, 2011 07:30:24 wren ng thornton wrote:

P.S., what the heck is readv(2) supposed to be good for? Does anyone want to see a binding for it?

Strikes me as something you might want to use in code that is orientated towards working with chunks of data of a possibly fixed maximum size (i.e., a chunk orientated steaming environment). The readv call gives you the ability to fill a series of chunks (e.g., from a free chunk list or something) with one call into the kernel. This could be a significant improvement from being forced to fill them individually. Cheers! -Tyson