ANNOUNCE: protocol-buffers-0.2.9 for Haskell is ready

Hello one and all, Amid much rejoicing, my Haskell version of protocol-buffer is now released (version 0.2.9). What is this for? What does it do? Why? Shorter answer: It generates Haskell data types that can be converted back and forth to lazy ByteStrings that interoperate with Google's generated code in C++/Java/python. It is a pure Haskell re-implementation of the Google code at http://code.Google.com/apis/protocolbuffers/docs/overview.html which is "...a language-neutral, platform-neutral, extensible way of serializing structured data for use in communications protocols, data storage, and more." Google's project produces C++, Java, and Python code. This one produces Haskell code. The release tarball (with 3 Haskell packages inside, see README in source) is at http://hackage.haskell.org/cgi-bin/hackage-scripts/package/protocol-buffers The darcs repository has moved to http://darcs.haskell.org/packages/protocol-buffers2/ You will also need a recent ghc compiler, the "binary" package and the "utf8-string" package from hackage.haskell.org (same site as mentioned above). The source compiles to 3 things: 1) the package "protocol-buffers" with the library API 2) the package "protocol-buffers-descriptor" with the descriptor.proto code 3) The 'hprotoc' executable which is a command line program similar to 'protoc'. The "examples" sub-directory in the code has the Haskell version of the "addressbook.proto" example and is compatible with Google's similar example code. The code generated from unittest.proto (and unittest_import.proto) includes messages TestAllTypes and TestAllExtensions which have been extensively tested by QuickCheck to ensure they can be wire encoded and decoded (see the "tests" sub-directory in the code). The user API, as exported by Text.ProtocolBuffers, allows for converting messages back and forth to the lazy ByteString type. And such messages can be merged, and the defaults accessed via the MessageAPI type class. The messages in Haskell as just regular data types and are thus immutable. Required types are simple record fields, optional types are Maybe, and repeated types are Seq (from Data.Sequence). Extensions are supported via Key data that allows access to the extension fields. Extensible messages contain an opaque ext'field entry of type ExtField that contains the map data structure to contain the extension field values. The User API allows for serializing messages as the usual series of fields. It also provides for a length prefix to be written to create delimited messages. It also provides to write a wire tag with any field number before the length and message data. This last form looks like a field on the wire, and there is a special api call to read back just the one message and its field number. This last API is similar to the one that is part of the C# API. No benchmarks have been run yet. Any suggestions? Unsupported for the moment is loading and storing "unknown" fields. It can be added sooner if someone has a use for this. Unsupported indefinitely is code generation for Services and Methods. I have yet to look into how this is presented in the other languages. The API to read a single message field, as mentioned above, might be extended to read any type instead of just messages. optional clever_quote { <autrijus> Perl: "Easy things are easy, hard things are possible" <autrijus> Haskell: "Hard things are easy, the impossible just happened" } Cheers! Chris Kuklewicz

haskell:
Hello one and all,
Amid much rejoicing, my Haskell version of protocol-buffer is now released (version 0.2.9).
Find it on Arch Linux, http://aur.archlinux.org/packages.php?ID=20080 Go Go Go! -- Don

Don Stewart wrote:
haskell:
Hello one and all,
Amid much rejoicing, my Haskell version of protocol-buffer is now released (version 0.2.9).
Find it on Arch Linux,
http://aur.archlinux.org/packages.php?ID=20080
Go Go Go!
Now also available in Gentoo Linux, along with the bundled lib protocol-buffers-descriptor and the tool hprotoc. Get it from the haskell overlay * dev-haskell/protocol-buffers * dev-haskell/protocol-buffers-descriptor * dev-haskell/hprotoc It's noticeable that we too were required the -fvia-C flag to build protocol-buffers package. I'm using GHC 6.8.3 on amd64. Without it, ghc died with the following error message: /tmp/ghc20461_0/ghc20461_0.s: Assembler messages: /tmp/ghc20461_0/ghc20461_0.s:6993:0: Error: too many memory references for `movzbl' /tmp/ghc20461_0/ghc20461_0.s:6994:0: Error: suffix or operands invalid for `shl' /tmp/ghc20461_0/ghc20461_0.s:6995:0: Error: suffix or operands invalid for `or' We also apply the following (tiny) patch to get hprotoc to compile: http://code.haskell.org/gentoo/gentoo-haskell/dev-haskell/hprotoc/files/hpro... diff -rN -u old-protocol-buffers-0.2.9/hprotoc/Text/ProtocolBuffers/ProtoCompile/Parser.hs new-protocol-buffers-0.2.9/hprotoc/Text/ProtocolBuffers/ProtoCompile/Parser.hs --- old-protocol-buffers-0.2.9/hprotoc/Text/ProtocolBuffers/ProtoCompile/Parser.hs 2008-09-21 15:16:08.000000000 +0200 +++ new-protocol-buffers-0.2.9/hprotoc/Text/ProtocolBuffers/ProtoCompile/Parser.hs 2008-09-21 15:16:08.000000000 +0200 @@ -45,7 +45,7 @@ default () -type P = GenParser Lexed +type P a = GenParser Lexed a pbParse :: String -> IO (Either ParseError D.FileDescriptorProto) pbParse filename = fmap (parseProto filename) (LC.readFile filename) Cheers, Lennart Kolmodin

Lennart Kolmodin wrote:
Don Stewart wrote:
haskell:
Hello one and all,
Amid much rejoicing, my Haskell version of protocol-buffer is now released (version 0.2.9).
Find it on Arch Linux,
http://aur.archlinux.org/packages.php?ID=20080
Go Go Go!
Now also available in Gentoo Linux, along with the bundled lib protocol-buffers-descriptor and the tool hprotoc.
Get it from the haskell overlay * dev-haskell/protocol-buffers * dev-haskell/protocol-buffers-descriptor * dev-haskell/hprotoc
Also, sharing the same tarball for different libs and applications makes automation of packaging slightly harder for us. This one I had to do partly by hand. Splitting the tarball into three packages would also allow users to install them directly by using cabal-install. I'd recommend it. Cheers, Lennart K

Lennart Kolmodin wrote:
Lennart Kolmodin wrote: > It's noticeable that we too were required the -fvia-C flag to build protocol-buffers package. I'm using GHC 6.8.3 on amd64. Without it, ghc died with the following error message: [snip]
Perhaps an actual bug report should be made. However, I don't have time to reduce the test case at the moment. Also, I tried two tricks: (D# x) <-> (W64# x) which works fine (F# x) <-> (W32# x) which produced garbage, so I had to replace it with alloca/poke/peek. Does anyone have a clue why one works and one does not?
Also, sharing the same tarball for different libs and applications makes automation of packaging slightly harder for us. This one I had to do partly by hand.
Splitting the tarball into three packages would also allow users to install them directly by using cabal-install.
I'd recommend it.
I'll probable split it for a future release. Hopefully it is possible to have cabal patches contributed to allow ghc 6.6 to work. I have no idea if Hugs could work. Cheers Chris

Hi Chris, On Sun, Sep 21, 2008 at 05:37:33PM +0100, Chris Kuklewicz wrote:
Also, I tried two tricks: (D# x) <-> (W64# x) which works fine (F# x) <-> (W32# x) which produced garbage, so I had to replace it with alloca/poke/peek.
This isn't supported, and I suspect is the cause of the -fasm problems. Please see http://hackage.haskell.org/trac/ghc/ticket/2209 for more details and suggested alternative. Thanks Ian

I am cross-posting this message to several lists. I had learned the trick before the documentation was updated. It seems I have used a very unreliable trick. And the "use castToSTUArray" suggested alternative is a really poor one since I am not using arrays at all. Who can suggest a way to cast from Float to Word32 and Double to Word64 using ghc? The actual task is that I need to write out the Float as a little endian sequence of four bytes and also be able to read it back in. The writing and reading are done in Put and Get monads to ByteString (from the "binary" package). The alloca/poke/peek work around I have looks like castWord32ToFloat :: Word32 -> Float castWord32ToFloat x = unsafePerformIO $ alloca $ \p -> poke p x >> peek (castPtr p) castFloatToWord32 :: Float -> Word32 castFloatToWord32 x = unsafePerformIO $ alloca $ \p -> poke p x >> peek (castPtr p) The unsafeCoerce trick that is no longer working looks like: castWord64ToDouble :: Word64 -> Double castWord64ToDouble (W64# w) = D# (unsafeCoerce# w) castDoubleToWord64 :: Double -> Word64 castDoubleToWord64 (D# d) = W64# (unsafeCoerce# d) Any ideas? Or is the alloca trick the only way to do this? Chris Ian Lynagh wrote:
Hi Chris,
On Sun, Sep 21, 2008 at 05:37:33PM +0100, Chris Kuklewicz wrote:
Also, I tried two tricks: (D# x) <-> (W64# x) which works fine (F# x) <-> (W32# x) which produced garbage, so I had to replace it with alloca/poke/peek.
This isn't supported, and I suspect is the cause of the -fasm problems.
Please see http://hackage.haskell.org/trac/ghc/ticket/2209 for more details and suggested alternative.
Thanks Ian

Chris Kuklewicz wrote:
Who can suggest a way to cast from Float to Word32 and Double to Word64 using ghc? The actual task is that I need to write out the Float as a little endian sequence of four bytes and also be able to read it back in. The writing and reading are done in Put and Get monads to ByteString (from the "binary" package).
I think alloca-like hacks is really the wrong direction and asking for trouble. You are trying to translate between platform-dependent native floats, and IEEE floats in a specified platform-independent binary format for Google. So use encodeFloat/decodeFloat - fast primitives in GHC - on the native side, and a hand-written Binary instance for the exact format you need on the Google side. My opinion, YMMV. Regards, Yitz

Chris Kuklewicz wrote:
I am cross-posting this message to several lists.
I had learned the trick before the documentation was updated. It seems I have used a very unreliable trick. And the "use castToSTUArray" suggested alternative is a really poor one since I am not using arrays at all.
castSTUArray is the way GHC does it - the idea is to allocate a small array, store the Float/Double in it, cast the type of the array to Word32 or whatever, and then read out the contents. It's more or less equivalent to the peek/poke solution, except that it doesn't need unsafePerformIO. GHC's code is here: (look for floatToWord): http://darcs.haskell.org/ghc/compiler/cmm/PprC.hs Cheers, Simon
participants (7)
-
Chris Kuklewicz
-
ChrisK
-
Don Stewart
-
Ian Lynagh
-
Lennart Kolmodin
-
Simon Marlow
-
Yitzchak Gale