On Tue, Mar 1, 2011 at 1:29 AM, wren ng thornton wrote:

What's an "iovec"?

http://www.google.com/search?q=iovec

ganesh:

Hi,

On 01/03/2011 06:35, Bryan O'Sullivan wrote:

...

Well, wait. That paragraph portion seems to confuse things. At the very least, it confuses me, because I surely didn't suggest using concat, as that would be silly. I want to see four entry points for writing:

fdWrite :: Strict.ByteString -> IO Int fdWriteAll :: Strict.ByteString -> IO () fdWritev :: [Strict.ByteString] -> IO Int -- turn the list into an iovec, then call writev fdWritevAll :: [Strict.ByteString] -> IO ()

People would normally use the 'All' variants, but there are times when you really do want to know if you've performed a short write so that you can handle it yourself.

What's the practical difference between a (lazy) list of strict bytestrings and a lazy bytestring?

Nothing. Lazy ByteStrings are slightly more efficient (avoid one indirection from the (:) cell).

On 3/1/11 6:01 PM, Daniel Fischer wrote:

On Tuesday 01 March 2011 23:51:05, Ganesh Sittampalam wrote:

...

What's the practical difference between a (lazy) list of strict bytestrings and a lazy bytestring?

Lazy ByteStrings are head-strict. I don't know whether that's relevant here, though.

Or rather, element-strict. But I don't think that matters here, since writing requires element strictness anyways. It may be significant if I write new versions to use writev for lists/lazys instead of calling write repeatedly, since we'd have to force all elements before calling into C. But then writev would require forcing the spine anyways (to get the length in chunks) so it wouldn't affect asymptotic behavior or memory retention. -- Live well, ~wren

On 3/1/11 1:35 AM, Bryan O'Sullivan wrote:

I want to see four entry points for writing:

fdWrite :: Strict.ByteString -> IO Int fdWriteAll :: Strict.ByteString -> IO () fdWritev :: [Strict.ByteString] -> IO Int -- turn the list into an iovec, then call writev fdWritevAll :: [Strict.ByteString] -> IO ()

Using writev requires the length of the list in order to get a count of chunks, which forces us to hold the whole list/lazy-bytestring in memory at once and also adds O(n) time for traversing it. Also it'd require converting each of the ByteString structs into iovec structs (whereas using write allows this to be unpacked into the call frames for write). What's the benefit of doing this? Is writev that much more efficient than Haskell code with the same semantics[1]?

People would normally use the 'All' variants, but there are times when you really do want to know if you've performed a short write so that you can handle it yourself.

What are the desired semantics for the All variants? Should it retry, or fail? How many times should it retry? etc. For manual recovery from partial writes, it would be better to have the basic function be: fdWriteFoo :: [Strict.ByteString] -- or Lazy.ByteString, whichever -> IO -- The total count of bytes written ( ByteCount -- The remaining content, with the first chunk already -- accounting for the last partial write (by adjusting -- the ByteString's offset). , [Strict.ByteString] ) So that the head of the lazy bytestring can be garbage collected and so you don't have to traverse it again to figure out where printing left off. Or actually, we'd want: fdWriteBar :: [Strict.ByteString] -> IO -- The total count of bytes written ( ByteCount -- The count of bytes written from the first chunk of -- the remaining content. , ByteCount -- The remaining content, with the first chunk not -- accounting for the last partial write (use the second -- ByteCount to account for it). , [Strict.ByteString] ) When using a lazy bytestring for the input then this latter version would be silly. However, with a list of bytestrings, there can be semantics encoded into how the string is chopped up and we shouldn't corrupt that information by adjusting any of the chunks. This is one reason why using lazy bytestrings gives cleaner semantics. [1] fdWrite :: Fd -> BL.ByteString -> IO ByteCount fdWrite fd = go 0 where -- We want to do a left fold in order to avoid stack overflows, -- but we need to have an early exit for incomplete writes -- (which normally requires a right fold). Hence this recursion. go acc BLI.Empty = return acc go acc (BLI.Chunk c cs) = do rc <- PosixBS.fdWrite fd c let acc' = acc+rc in acc' `seq` do if rc == fromIntegral (BS.length c) then go acc' cs else return acc' -- Live well, ~wren

On 3/2/11 1:58 AM, Antoine Latter wrote:

On Tue, Mar 1, 2011 at 9:23 PM, wren ng thornton wrote:

...

On 3/1/11 1:35 AM, Bryan O'Sullivan wrote:

...

I want to see four entry points for writing:

fdWrite :: Strict.ByteString -> IO Int fdWriteAll :: Strict.ByteString -> IO () fdWritev :: [Strict.ByteString] -> IO Int -- turn the list into an iovec, then call writev fdWritevAll :: [Strict.ByteString] -> IO ()

Using writev requires the length of the list in order to get a count of chunks, which forces us to hold the whole list/lazy-bytestring in memory at once and also adds O(n) time for traversing it. Also it'd require converting each of the ByteString structs into iovec structs (whereas using write allows this to be unpacked into the call frames for write).

What's the benefit of doing this? Is writev that much more efficient than Haskell code with the same semantics[1]?

The benefit of using 'writev' over multiple calls to 'write' is the 'writev' is frequently a single kernel call - avoiding multiple context switches.

Ah, of course.

Whether or not it is worth it to hang on to the entire bytestring to get this advantage probably depends on the circumstance.

In that case I'd suggest having both writev and iterated write versions of handling lazy/lists-of bytestrings. I think the increased API size would be worth it to cover both situations. -- Live well, ~wren

On 3/2/11 8:49 AM, Simon Marlow wrote:

On 28/02/2011 22:57, wren ng thornton wrote:

...

On 2/28/11 12:28 PM, Johan Tibell wrote:

...

On Mon, Feb 28, 2011 at 9:20 AM, Bryan O'Sullivan wrote:

...

I think that the strict bytestring version should just replace the current function in System.Posix.IO, and [...]

So then what would become of the string variants? Backwards compatibility and all... not _everyone_ uses ByteStrings yet.

The unix package is tied to GHC releases, where the usual convention is to DEPRECATE for one major release (~ 12 months), then remove in the next release.

I'm fine with that. I don't particularly care what happens to the string functions, I'd just like to see the bytestring versions incorporated. But that does raise the issue: if we are to (eventually) remove the string versions and put the bytestring versions in situ, then how should the migration proceed? I'd suggest putting the functions in System.Posix.IO.ByteString for the interim; this has the benefit that I could release a compatibility library allowing people to upgrade without changing their GHC, if need be. But then what happens after the interim? We can't just replace the deprecated string versions with the bytestring versions directly, can we? So would we then deprecate System.Posix.IO.ByteString (another major release...) or just keep it around and have System.Posix.IO re-export it? This is part of why I wasn't suggesting to remove the string functions, much as we'd like people to migrate. -- Live well, ~wren

On Thu, Mar 3, 2011 at 5:55 PM, Brandon S Allbery KF8NH wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 3/2/11 21:38 , wren ng thornton wrote:

...

But that does raise the issue: if we are to (eventually) remove the string versions and put the bytestring versions in situ, then how should the migration proceed? I'd suggest putting the functions in System.Posix.IO.ByteString for the interim; this has the benefit that I could release a compatibility library allowing people to upgrade without changing their GHC, if need be.

It occurs to me that, if we're going to remove String versions, it should happen everywhere and in a coordinated way.

What is the benefit from removing String versions (as opposed to just adding ByteString (and/or Text) ones alongside them)?

- -- brandon s. allbery     [linux,solaris,freebsd,perl]    allbery.b@gmail.com system administrator  [openafs,heimdal,too many hats]                kf8nh -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1vyAYACgkQIn7hlCsL25USmACeJJn+QpMZJa3c5Nmf0TAxC/5R dVkAn0CI5LrxgjVc6ZugdXebPzZu87J9 =dtDw -----END PGP SIGNATURE-----

_______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries

-- Work is punishment for failing to procrastinate effectively.

2011/3/3 Bryan O'Sullivan :

2011/3/3 Gábor Lehel

...

What is the benefit from removing String versions (as opposed to just adding ByteString (and/or Text) ones alongside them)?

Strings are unicode. A file descriptor handles bytes.

I understand, and agree that ByteStrings are more appropriate. I'm not particularly attached to Strings. It might nonetheless make sense to keep the String versions around for the convenience of people who happen to have gotten a String from somewhere (there are, after all, quite a few such places), and wish to output it to a file descriptor. -- Work is punishment for failing to procrastinate effectively.

2011/3/3 Brandon S Allbery KF8NH

Which was my point. If you're going to deprecate Strings, do so across the board in an organized manner --- not via ad hoc deprecations to individual libraries.

The only problem with that is that ad hoc deprecations are achievable by individual library maintainers, while a "stop-the-world" approach won't, I think, work.

On Mar 3, 2011, at 8:27 PM, Brandon S Allbery KF8NH wrote:

Isn't this what the Haskell Platform process is for? Individual libraries, yes, but as part of an overarching plan so you don't have e.g. network desupporting String while some other library decides to hold onto them as default a bit longer.

I think that's a bit stronger onus on the platform process than intended. Certainly the platform discussion should have some strong advisory weight on decisions of maintainers, but package maintainers have the real final say. And in the case of strings/bytestrings, I think the issue is minor anyway, since converting between the two is so simple. What we can and should expect is that individual libraries don't just ax the string functions abruptly, but provide some measured process of deprecation. Which is, I think, precisely what's being posed in this case. Cheers, Sterl.