On Thu, May 14, 2009 at 8:40 PM, Don Stewart wrote:

...

I actually need little endian encoding... wondering if anyone else hit

leimy2k: this

...

with Data.Binary. (because I'm working with Bell Lab's 9P protocol which does encode things on the network in little-endian order).

Anyone got some "tricks" for this?

Yes! There are big, little and host-endian primitives in the Get/Put monads.

http://hackage.haskell.org/packages/archive/binary/0.5.0.1/doc/html/Data-Bin...

You can use these to build encoders directly.

Cool... I just have to write my own encoder and decoder now. As a request could we get encodeLe decodeLe for a later version of this library? :-) That'd be totally awesome.

Cheers, Don

On Thu, May 14, 2009 at 8:46 PM, Don Stewart wrote:

leimy2k:

...

On Thu, May 14, 2009 at 8:40 PM, Don Stewart wrote:

leimy2k: > I actually need little endian encoding... wondering if anyone else

hit

...

this > with Data.Binary. (because I'm working with Bell Lab's 9P protocol

which

...

does > encode things on the network in little-endian order). > > Anyone got some "tricks" for this?

Yes! There are big, little and host-endian primitives in the Get/Put

monads.

...

http://hackage.haskell.org/packages/archive/binary/0.5.0.1/doc/html/

...

Data-Binary-Put.html#v%3AputWord16le

You can use these to build encoders directly.

Cool... I just have to write my own encoder and decoder now.

As a request could we get encodeLe decodeLe for a later version of this library? :-) That'd be totally awesome.

Oh, you mean entirely different instances for all the current ones, that use LE encodings?

Well the library is leaning towards "Network Byte Order" in that it has encode/decode that only encode/decode for Big Endian. Us folks who have to do little endian all now have to write our own encoding/decoding :-) I'm speaking specifically of the encode/decode functions. I have no idea how they're implemented. Are you saying that encode is doing something really simple and the default encodings for things just happen to be big endian? If so, then I understand the pain.... but it still means I have to roll my own :-) I guess if one must choose, big endian kind of makes sense, except that the whole world is little endian now, except for networks :-) (No one *really* cares about anything but x86 anyway these days right?) I'm only half-kidding.

-- Don

On Thu, May 14, 2009 at 8:54 PM, Don Stewart wrote:

...

I'm speaking specifically of the encode/decode functions. I have no idea how they're implemented.

Are you saying that encode is doing something really simple and the default encodings for things just happen to be big endian? If so, then I understand the pain.... but it still means I have to roll my own :-) I guess if one must choose, big endian kind of makes sense, except that the whole world is little endian now, except for networks :-) (No one *really* cares about anything but x86 anyway these days right?)

Oh, 'encode' has type:

encode :: Binary a => a -> ByteString

it just encodes with the default instances, which are all network order:

http://en.wikipedia.org/wiki/Endianness#Endianness_in_networking

Yeah I understand that Big Endian == Network Byte Order... which would be true, if I wasn't talking about Plan 9's 9P protocol which specifies little endian bytes on the wire (as far as I can tell anyway from the man page). Dave

-- Don

On Thu, May 14, 2009 at 9:10 PM, David Leimbach wrote:

On Thu, May 14, 2009 at 8:57 PM, David Leimbach wrote:

...

On Thu, May 14, 2009 at 8:54 PM, Don Stewart wrote:

...

...

I'm speaking specifically of the encode/decode functions. I have no idea how they're implemented.

Are you saying that encode is doing something really simple and the default encodings for things just happen to be big endian? If so, then I understand the pain.... but it still means I have to roll my own :-) I guess if one must choose, big endian kind of makes sense, except that the whole world is little endian now, except for networks :-) (No one *really* cares about anything but x86 anyway these days right?)

Oh, 'encode' has type:

encode :: Binary a => a -> ByteString

it just encodes with the default instances, which are all network order:

http://en.wikipedia.org/wiki/Endianness#Endianness_in_networking

Yeah I understand that Big Endian == Network Byte Order... which would be true, if I wasn't talking about Plan 9's 9P protocol which specifies little endian bytes on the wire (as far as I can tell anyway from the man page).

Dave

FYI here's what I've ended up trying to write to negotiate the "version" of a 9p server:

main = withSocketsDo $ do ainfo <- getAddrInfo Nothing (Just "127.0.0.1") (Just "6872") -- hardcoded for now, it's an IRC filesystem server let a = head ainfo sock <- socket AF_INET Stream defaultProtocol connect sock (addrAddress a) sendAll sock $ (toLazyByteString (putWord32le (fromIntegral (16 ::Int32)))) sendAll sock $ (encode (100 ::Int8)) sendAll sock $ (toLazyByteString (putWord32le (fromIntegral (1024 ::Int32)))) sendAll sock $ (encode (C.pack "9P2000"))

I totally forgot the tag part of 9p.... ugh. I think I should probably just go to bed now.

I feel like I should use wireshark or something to watch the bytes :-) I'm not feeling very sure about this.

...

...

-- Don

timd:

On a related matter, I am using Data.Binary to serialise data from haskell for use from other languages. The Data.Binary encoding of a Double is a long integer for the mantissa, and an int for the exponent. This doesn't work too well for interacting with other languages as I'd need to have an arbitrary precision int type there to decode/encode. The CORBA CDR standard encodes doubles in a big ended fashion like this (excuse my possibly incorrect ascii art):

| byte | msb lsb | |------+---------------------------| | 0 | S E6 E0 | | 1 | E10 E9 E8 E7 F3 F2 F1 F0 | | 2 | F11 F4 | | 3 | F19 F12 | | 4 | F27 F20 | | 5 | F35 F28 | | 6 | F43 F36 | | 7 | F51 F44 |

Up until now, my code is pure haskell. Is it possible to get at the internal bits of a Double/CDouble in ghc? Or Should I use the FFI and write C to encode something like the above?

Yep, it's possible, just not portably so. Google for Data.Binary IEEE discussions.

Sven.Panne:

Am Freitag, 15. Mai 2009 06:37:22 schrieb Don Stewart:

...

timd:

...

On a related matter, I am using Data.Binary to serialise data from haskell for use from other languages. [...] [...] Yep, it's possible, just not portably so. Google for Data.Binary IEEE discussions.

I think this topic pops up over and over again, and the proposed "solutions" are no solutions at all, neither from a performance point of view, nor from an ease of use point of view. Proposing insane bit fiddling by hand when all one technically needs is often a "peek" or "poke" amounts to simply ignoring an API problem. ;-)

I think most problems can be fixed in a rather pragmatic way by adding a few functions to the binary package:

Add to Data.Binary.Builder:

putFloatIEEEbe :: Float -> Builder putDoubleIEEEbe :: Double -> Builder putFloatIEEEle :: Float -> Builder putDoubleIEEEle :: Double -> Builder putFloatIEEEhost :: Float -> Builder putDoubleIEEEhost :: Double -> Builder

Add to Data.Binary.Get:

getFloatIEEEbe :: Get Float getDoubleIEEEbe :: Get Double getFloatIEEEle :: Get Float getDoubleIEEEle :: Get Double getFloatIEEEhost :: Get Float getDoubleIEEEhost :: Get Double

Add to Data.Binary.Put:

putFloatIEEEbe :: Float -> Put putDoubleIEEEbe :: Double -> Put putFloatIEEEle :: Float -> Put putDoubleIEEEle :: Double -> Put putFloatIEEEhost :: Float -> Put putDoubleIEEEhost :: Double -> Put

The *host functions are basically peek/poke for most platforms. The *le/*be functions can use peek/poke if the endianess matches (compile time decision) *and* the alignment is OK for the given platform (runtime decision). Non-IEEE platforms always have to do the bit fiddling internally, but all this is hidden behind the above API.

IIRC I have proposed something similar 1-2 years ago, but I can't remember any reason why this hasn't been implemented. Any comments on the above functions?

Patches are welcome.

One final remarks: I think the low level functions of the binary package should really keep the notions of "endianess" and "alignment constraints" separate, something which isn't done currently: The *host functions have alignment restrictions, the *be/*le functions don't. There is no good reason for this non-orthogonality.

That seems reasonable. -- Don

On Thu, 2009-05-14 at 20:46 -0700, David Leimbach wrote:

On Thu, May 14, 2009 at 8:40 PM, Don Stewart wrote: leimy2k:

> I actually need little endian encoding... wondering if anyone else hit this > with Data.Binary. (because I'm working with Bell Lab's 9P protocol which does > encode things on the network in little-endian order). > > Anyone got some "tricks" for this?

Yes! There are big, little and host-endian primitives in the Get/Put monads.

http://hackage.haskell.org/packages/archive/binary/0.5.0.1/doc/html/Data-Bin...

You can use these to build encoders directly.

Cool... I just have to write my own encoder and decoder now.

As a request could we get encodeLe decodeLe for a later version of this library? :-) That'd be totally awesome.

The thing you're missing (and which admittedly is not clear) is that the binary package has two parts. One is a layer where you get full control over the binary representation. The other is a portable serialisation layer for pickling and unpickling Haskell values. That pickling layer (ie the Binary class) is not for working with externally-defined binary formats. It might seem like we could co-opt the Binary class for this purpose eg by parametrising by a dozen things like endian, padding, etc etc but I don't think it scales or is sufficiently flexible (and it'd be slow). What is missing in the binary package is a nice set of combinators for using the low level layer to easily construct parsers for externally-defined formats. That's what you'd want for your P9 protocol. To reduce confusion we should also split the Haskell picking layer from the lower layer. This has been on our TODO list for some time. It needs to be done pretty carefully however and we've not really had the time. Duncan

Sorry took so long to get back... Thank you for the response. Been really busy lately :-) On Sat, May 16, 2009 at 3:46 AM, Khudyakov Alexey

wrote:

...

I actually need little endian encoding... wondering if anyone else hit

On Friday 15 May 2009 06:52:29 David Leimbach wrote: this

...

with Data.Binary. (because I'm working with Bell Lab's 9P protocol which does encode things on the network in little-endian order). Anyone got some "tricks" for this?

Dave

You could just define data type and Binary instance for 9P messages. Something like this:

P9Message = Tversion { tag :: Word16, msize :: Word32, version :: String } | ...

instance Binary P9Message where put (Tverstion t m v) = putWord16le t >> putWord32le m >> put v -- and so on...

get = do length <- getWord32le id <- getWord16le case is of p9TMessage -> do ...

There are a lot of boilerplate code thought...

Thank you, this still looks like a useful way to proceed, combined with the BinaryLE approach perhaps, to avoid a lot of boilerplate.

BTW could you say what do you want to do with 9P? I tried to play with it using libixp library but without any success. It was mainly to understand how does it works and how can it be used.

From a services point of view, 9P gives you a way to host them, and even devices, on a network share that can be "mounted" into the filesystem's namespace. The net result is you've plugged into the standard unix utilities that do open, read, write etc, and can do a lot of interesting

things with mere shell scripts. Operating systems that can be clients of a 9P service include Linux, Inferno, Plan 9, and anything else that runs FUSE 9P (several BSDs).

From a client perspective, having a 9P implementation gives you a more fine-grained programatic interface to accessing other 9P services.

There are also a lot of 9P implementations in many languages that you can interoperate with: http://9p.cat-v.org/implementations

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

On Thu, May 28, 2009 at 5:42 AM, Khudyakov Alexey

wrote:

On Thursday 28 of May 2009 07:52:56 David Leimbach wrote:

...

Sorry took so long to get back... Thank you for the response. Been really busy lately :-)

There are also a lot of 9P implementations in many languages that you can interoperate with:

http://9p.cat-v.org/implementations

Thank you for that link. I didn't find it earlier. But actually I asked what are you trying to write?

I'm trying to implement the protocol, so that I can implement other things on top of that. I'm also trying to figure out how bad/good Haskell Binary IO really is that it's been addressed a few times differently :-)

-- Khudyakov Alexey _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

On Thursday, May 28, 2009, Don Stewart wrote:

leimy2k:

...

I'm also trying to figure out how bad/good Haskell Binary IO really is that it's been addressed a few times differently :-)

FWIW Binary IO as implemented in Data.Binary is widely used in our production systems at Galois. I'd be fairly confident in it.

Fair enough. I am just new to the interface, wondering if I should try matching responses by pulling apart via Get, or the bit syntax package.

-- Don

On Thu, May 28, 2009 at 9:17 AM, John Van Enk wrote:

...

Fair enough. I am just new to the interface, wondering if I should try matching responses by pulling apart via Get, or the bit syntax package.

I'm assming you have some 'data Foo = ...'? If this is the case, you're probably okay writing an instance of Binary for Foo and using encode/decode. That's all I've had to do and had superb performance.

encode/decode do Big Endian, and 9P does little endian.

From the man page:

"Each message consists of a sequence of bytes. Two–, four–, and eight–byte fields hold unsigned integers represented in little–endian order (least significant byte first)." encode/decode just won't work for me as a result, as they assume the not-so-aptly-named (at least in this case) "network" byte order. Since I'm not Rob Pike, or Ken Thompson or any of the Plan 9 guys, I can't tell you why they chose little endian for this. (I should note that I sometimes work with CAN hardware, and depending on what people want to put on the line, little endian creeps in there too, so this issue will have to be dealt with here as well, if I am to use Data.Binary) I've got (typing from memory, forgive syntax errors) module Nine (Message) where {-- missing imports --} notag = putWord16le -1 -- Message type field values -- from: http://plan9.bell-labs.com/sources/plan9/sys/include/fcall.h tversion = 100 rversion = 101 rerror = 107 -- only 9P2000 is being handled so some constructors are simple data Message = Tversion | Rversion {size :: Int, tag :: Int, msize :: Int, version :: String} | Rerror {message :: String} -- maybe this should be Maybe ByteString? ... instance Binary Message where put Tversion = putWord32le 16 >> tversion >> notag >> msize >> "9P2000" put Rversion = putWord32le 16 >> rversion >> notag >> msize >> "9P2000" get = do size <- getWord32le tag <- getWord8 case tag of rversion -> do msize <- getWord32le version <- getRemainingLazyByteString return (Rversion size tag msize (C.unpack version)) =========== I'm beginning to feel the Get Monad is not going to be as nice as if I had some bit-syntax for determining what type of return message I'm getting. Any "T" message can be met with the paired "R" response, or an Rerror message, with a detailed description of what went wrong. My instance of get is looking like it's going to be pretty nasty :-) The first message in any 9p session is Tversion with Rversion or Rerror response. There's optional authentication hooks via Tauth/Rauth, and then the filesystem namespace navigation operation messages. Dave /jve

Writing instances encode/decode that use either little endian or big endian (or mixed!) is trivial... see attached file. I don't see where your problem is... On Thu, May 28, 2009 at 12:35 PM, David Leimbach wrote:

On Thu, May 28, 2009 at 9:17 AM, John Van Enk wrote:

...

...

Fair enough. I am just new to the interface, wondering if I should try matching responses by pulling apart via Get, or the bit syntax package.

I'm assming you have some 'data Foo = ...'? If this is the case, you're probably okay writing an instance of Binary for Foo and using encode/decode. That's all I've had to do and had superb performance.

encode/decode do Big Endian, and 9P does little endian.

From the man page:

"Each message consists of a sequence of bytes. Two–, four–, and eight–byte fields hold unsigned integers represented in little–endian order (least significant byte first)."

encode/decode just won't work for me as a result, as they assume the not-so-aptly-named (at least in this case) "network" byte order. Since I'm not Rob Pike, or Ken Thompson or any of the Plan 9 guys, I can't tell you why they chose little endian for this.

(I should note that I sometimes work with CAN hardware, and depending on what people want to put on the line, little endian creeps in there too, so this issue will have to be dealt with here as well, if I am to use Data.Binary)

I've got (typing from memory, forgive syntax errors)

module Nine (Message) where

{-- missing imports --}

notag = putWord16le -1

-- Message type field values -- from: http://plan9.bell-labs.com/sources/plan9/sys/include/fcall.h tversion = 100 rversion = 101 rerror = 107

-- only 9P2000 is being handled so some constructors are simple data Message = Tversion | Rversion {size :: Int, tag :: Int, msize :: Int, version :: String} | Rerror {message :: String} -- maybe this should be Maybe ByteString? ...

instance Binary Message where put Tversion = putWord32le 16 >> tversion >> notag >> msize >> "9P2000" put Rversion = putWord32le 16 >> rversion >> notag >> msize >> "9P2000" get = do size <- getWord32le tag <- getWord8 case tag of rversion -> do msize <- getWord32le version <- getRemainingLazyByteString return (Rversion size tag msize (C.unpack version))

===========

I'm beginning to feel the Get Monad is not going to be as nice as if I had some bit-syntax for determining what type of return message I'm getting. Any "T" message can be met with the paired "R" response, or an Rerror message, with a detailed description of what went wrong. My instance of get is looking like it's going to be pretty nasty :-)

The first message in any 9p session is Tversion with Rversion or Rerror response. There's optional authentication hooks via Tauth/Rauth, and then the filesystem namespace navigation operation messages.

Dave

/jve

...

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

-- /jve

I'm trying to implement the protocol, so that I can implement other things on top of that.

I'm also trying to figure out how bad/good Haskell Binary IO really is that it's been addressed a few times differently :-)

FWIW, I've used Data.Binary extensively and have found it a joy to work with. I've used it to serialize/deserialize ethernet packets in real time for a VPN implementation and have never had a problem. It's quite fast and robust. /jve

On Thu, May 28, 2009 at 6:34 PM, Duncan Coutts wrote:

If it's not available publicly perhaps you might share it privately. Don and I have discussed a few times writing a paper on the design and implementation of a binary library.

I would definitely like to read such a paper. Except for Don's blog entries there's precious little written on the design of high performance Haskell libraries. -- Johan