ANNOUNCE: storable-endian

Eugene Kirpichov

22 Dec 2010 22 Dec '10

9:58 a.m.

Hi cafe, I've released "storable-endian" http://hackage.haskell.org/package/storable-endian It defines types like {{Int,Word}{16,32,64},Double,Float}{LE,BE} (for example Int32BE) etc. with a corresponding Storable instance. It is needed for binary interoperability with libraries or network protocols with fixed endianness. Hope you find it useful. -- Eugene Kirpichov Senior Software Engineer, Grid Dynamics http://www.griddynamics.com/

Show replies by date

Henning Thielemann

22 Dec 22 Dec

6:24 p.m.

On Wed, 22 Dec 2010, Eugene Kirpichov wrote:

...

Hi cafe,

I've released "storable-endian" http://hackage.haskell.org/package/storable-endian

It defines types like {{Int,Word}{16,32,64},Double,Float}{LE,BE} (for example Int32BE) etc. with a corresponding Storable instance.

It is needed for binary interoperability with libraries or network protocols with fixed endianness. Hope you find it useful.

How about type constructors LittleEndian and BigEndian? newtype LittleEndian a = LittleEndian a Maybe using some type classes you can even get rid of Template Haskell and get plain Haskell 98?

Henning Thielemann

6:58 p.m.

On Thu, 23 Dec 2010, Henning Thielemann wrote:

...

On Wed, 22 Dec 2010, Eugene Kirpichov wrote:

...
It defines types like {{Int,Word}{16,32,64},Double,Float}{LE,BE} (for example Int32BE) etc. with a corresponding Storable instance.

How about type constructors LittleEndian and BigEndian?

newtype LittleEndian a = LittleEndian a

Maybe using some type classes you can even get rid of Template Haskell and get plain Haskell 98?

Yes, I think you could have (given a module Data.Storable.LittleEndian as LE) instance LE.Storable a => Storable (LittleEndian a) where sizeOf (LittleEndian a) = sizeOf a alignment (LittleEndian a) = alignment a peek p = fmap LittleEndian $ LE.peek p poke p (LittleEndian a) = LE.poke p a class LE.Storable a where LE.peek :: Ptr a -> IO a LE.poke :: Ptr a -> a -> IO () instance LE.Storable Word16 where LE.peek p = getWord16le (castPtr p) LE.poke p = putWord16le (castPtr p) ... I find this much cleaner and simpler to extend to other types.

Eugene Kirpichov

24 Dec 24 Dec

2:49 a.m.

Hello Hennig, Thanks for the suggestions! I've released storable-endian 0.2.0, which does not use TH and bases on your suggestion (though it has a bit of boilerplate because of abandoning TH, but I don't think that's critical). Here's the new source: https://github.com/jkff/storable-endian/blob/master/Data/Storable/Endian.hs 2010/12/23 Henning Thielemann :

...

On Thu, 23 Dec 2010, Henning Thielemann wrote:

...
On Wed, 22 Dec 2010, Eugene Kirpichov wrote:

...
It defines types like {{Int,Word}{16,32,64},Double,Float}{LE,BE} (for example Int32BE) etc. with a corresponding Storable instance.

How about type constructors LittleEndian and BigEndian?

newtype LittleEndian a = LittleEndian a

Maybe using some type classes you can even get rid of Template Haskell and get plain Haskell 98?

Yes, I think you could have (given a module Data.Storable.LittleEndian as LE)

instance LE.Storable a => Storable (LittleEndian a) where sizeOf (LittleEndian a) = sizeOf a alignment (LittleEndian a) = alignment a peek p = fmap LittleEndian $ LE.peek p poke p (LittleEndian a) = LE.poke p a

class LE.Storable a where LE.peek :: Ptr a -> IO a LE.poke :: Ptr a -> a -> IO ()

instance LE.Storable Word16 where LE.peek p = getWord16le (castPtr p) LE.poke p = putWord16le (castPtr p)

...

I find this much cleaner and simpler to extend to other types.

-- Eugene Kirpichov Senior Software Engineer, Grid Dynamics http://www.griddynamics.com/

Eugene Kirpichov

2:49 a.m.

...I mean, storable-endian 0.2.1 actually - 0.2.0 had a stupid bug. 2010/12/24 Eugene Kirpichov :

...

Hello Hennig,

Thanks for the suggestions! I've released storable-endian 0.2.0, which does not use TH and bases on your suggestion (though it has a bit of boilerplate because of abandoning TH, but I don't think that's critical).

Here's the new source: https://github.com/jkff/storable-endian/blob/master/Data/Storable/Endian.hs

2010/12/23 Henning Thielemann :

...
On Thu, 23 Dec 2010, Henning Thielemann wrote:

...
On Wed, 22 Dec 2010, Eugene Kirpichov wrote:

...
It defines types like {{Int,Word}{16,32,64},Double,Float}{LE,BE} (for example Int32BE) etc. with a corresponding Storable instance.

How about type constructors LittleEndian and BigEndian?

newtype LittleEndian a = LittleEndian a

Maybe using some type classes you can even get rid of Template Haskell and get plain Haskell 98?

Yes, I think you could have (given a module Data.Storable.LittleEndian as LE)

instance LE.Storable a => Storable (LittleEndian a) where sizeOf (LittleEndian a) = sizeOf a alignment (LittleEndian a) = alignment a peek p = fmap LittleEndian $ LE.peek p poke p (LittleEndian a) = LE.poke p a

class LE.Storable a where LE.peek :: Ptr a -> IO a LE.poke :: Ptr a -> a -> IO ()

instance LE.Storable Word16 where LE.peek p = getWord16le (castPtr p) LE.poke p = putWord16le (castPtr p)

...

I find this much cleaner and simpler to extend to other types.

-- Eugene Kirpichov Senior Software Engineer, Grid Dynamics http://www.griddynamics.com/

-- Eugene Kirpichov Senior Software Engineer, Grid Dynamics http://www.griddynamics.com/

Gábor Lehel

3:44 a.m.

One further idea -- assuming HasBigEndian and HasLittleEndian are unexported because you don't want people writing new instances -- is to have something like class HasBigEndianPrivate => HasBigEndian instead, with the latter exported and the former not, and instances of each for the same types. What you gain is HasBigEndian and its instances showing up in the docs, and being able to use it in type signatures (while still not being able to write new instances). On Fri, Dec 24, 2010 at 8:49 AM, Eugene Kirpichov wrote:

...

...I mean, storable-endian 0.2.1 actually - 0.2.0 had a stupid bug.

2010/12/24 Eugene Kirpichov :

...
Hello Hennig,

Thanks for the suggestions! I've released storable-endian 0.2.0, which does not use TH and bases on your suggestion (though it has a bit of boilerplate because of abandoning TH, but I don't think that's critical).

Here's the new source: https://github.com/jkff/storable-endian/blob/master/Data/Storable/Endian.hs

2010/12/23 Henning Thielemann :

...
On Thu, 23 Dec 2010, Henning Thielemann wrote:

...
On Wed, 22 Dec 2010, Eugene Kirpichov wrote:

...
It defines types like {{Int,Word}{16,32,64},Double,Float}{LE,BE} (for example Int32BE) etc. with a corresponding Storable instance.

How about type constructors LittleEndian and BigEndian?

newtype LittleEndian a = LittleEndian a

Maybe using some type classes you can even get rid of Template Haskell and get plain Haskell 98?

Yes, I think you could have (given a module Data.Storable.LittleEndian as LE)

instance LE.Storable a => Storable (LittleEndian a) where sizeOf (LittleEndian a) = sizeOf a alignment (LittleEndian a) = alignment a peek p = fmap LittleEndian $ LE.peek p poke p (LittleEndian a) = LE.poke p a

class LE.Storable a where LE.peek :: Ptr a -> IO a LE.poke :: Ptr a -> a -> IO ()

instance LE.Storable Word16 where LE.peek p = getWord16le (castPtr p) LE.poke p = putWord16le (castPtr p)

...

I find this much cleaner and simpler to extend to other types.

-- Eugene Kirpichov Senior Software Engineer, Grid Dynamics http://www.griddynamics.com/

-- Eugene Kirpichov Senior Software Engineer, Grid Dynamics http://www.griddynamics.com/

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

-- Work is punishment for failing to procrastinate effectively.

Henning Thielemann

3:52 a.m.

On Fri, 24 Dec 2010, Gábor Lehel wrote:

...

One further idea -- assuming HasBigEndian and HasLittleEndian are unexported because you don't want people writing new instances

Why should people not write further instances? I could write a LittleEndianStorable instance for fixed point numbers, long characters etc.

Gábor Lehel

9:17 a.m.

2010/12/24 Henning Thielemann :

...

On Fri, 24 Dec 2010, Gábor Lehel wrote:

...
One further idea -- assuming HasBigEndian and HasLittleEndian are unexported because you don't want people writing new instances

Why should people not write further instances? I could write a LittleEndianStorable instance for fixed point numbers, long characters etc.

I don't know, I just saw that they weren't exported. But now I see that they are. So apparently it was just a mistake. -- Work is punishment for failing to procrastinate effectively.

Eugene Kirpichov

9:18 a.m.

They really weren't - I exported them according to your and Hennig's advice. 2010/12/24 Gábor Lehel :

...

2010/12/24 Henning Thielemann :

...
On Fri, 24 Dec 2010, Gábor Lehel wrote:

...
One further idea -- assuming HasBigEndian and HasLittleEndian are unexported because you don't want people writing new instances

Why should people not write further instances? I could write a LittleEndianStorable instance for fixed point numbers, long characters etc.

I don't know, I just saw that they weren't exported. But now I see that they are. So apparently it was just a mistake.

-- Work is punishment for failing to procrastinate effectively.

-- Eugene Kirpichov Senior Software Engineer, Grid Dynamics http://www.griddynamics.com/

Henning Thielemann

3:50 a.m.

On Fri, 24 Dec 2010, Eugene Kirpichov wrote:

...

I've released storable-endian 0.2.0, which does not use TH and bases on your suggestion (though it has a bit of boilerplate because of abandoning TH, but I don't think that's critical).

Great, this should make it also usable on JHC (I have not tested so far). However I'm still concerned about the use of so much Unsafe functions. You use unsafePeekOff and then wrap it in IO monad, again. Could you also stay in IO monad and omit unsafePerformIO? You use unsafeCoerce for mapping from (IO Word16) to (IO Int16) - how about just (fmap fromIntegral)? Maybe you succeed to find out machine endianess at compile time. Then you could use native memory access if the user requested endianess is the one of the machine. If you achieve this, you could implement peek and poke of Float and Double by just reversing the byte order into an 'alloca'd memory area and peek from and poke into this one with a native peek or poke, respectively. That is you would not need any unsafeCoerce anymore. Two suggestions on style: I find the class names with "Has" prefix not so nice. "Has" the type Double a Big Endian or a Little Endian? How about class names LittleEndianStorable or LittleEndianOrder? I think I like LittleEndianStorable more, since it makes clear, that this is a class similar to Storable but specialised to Little Endian byte order. I'd like to have GHC specific code in a separate module instead of preprocessor instructions, that become quickly confusing if you add support for more compilers (say JHC). You may put GHC specific code into a private module, say Data.Storable.Endian.Private, and create two instances of this module in two directories, say 'generic' and 'ghc'. Then in storable-endian.cabal you write: -- for Data.Storable.Endian Hs-Source-Dirs: src -- for Data.Storable.Endian.Private If impl(ghc) Hs-Source-Dirs: ghc Else Hs-Source-Dirs: generic ... and thank you very much for creating this package!

Eugene Kirpichov

5:36 a.m.

Hello, 2010/12/24 Henning Thielemann :

...

On Fri, 24 Dec 2010, Eugene Kirpichov wrote:

...
I've released storable-endian 0.2.0, which does not use TH and bases on your suggestion (though it has a bit of boilerplate because of abandoning TH, but I don't think that's critical).

Great, this should make it also usable on JHC (I have not tested so far).

However I'm still concerned about the use of so much Unsafe functions. You use unsafePeekOff and then wrap it in IO monad, again. Could you also stay in IO monad and omit unsafePerformIO?

Yes - it was for the sake of fewer syntactic litter (and fewer typing effort), but I still got rid of it, you're right it's better to avoid this stuff.

...

You use unsafeCoerce for mapping from (IO Word16) to (IO Int16) - how about just (fmap fromIntegral)? Maybe you succeed to find out machine endianess at compile time. Then you could use native memory access if the user requested endianess is the one of the machine. If you achieve this, you could implement peek and poke of Float and Double by just reversing the byte order into an 'alloca'd memory area and peek from and poke into this one with a native peek or poke, respectively. That is you would not need any unsafeCoerce anymore. I don't think I'd like to allocate memory in these functions - I expect them to have very predictable and very high performance. I'm using unsafeCoerce because it allows me to treat all these types in the same fashion (which is good for correctness and readability), and unsafeCoerce's semantics is identical to what is expected of this code - interpret a memory area as a value of a different type.

You're right however that some support could be added for host endianness - once I think something up, I'll do it; maybe this could be done with a couple unsafePerformIOs and runtime tests :)

...

Two suggestions on style: I find the class names with "Has" prefix not so nice. "Has" the type Double a Big Endian or a Little Endian?

This question could be raised if it was "HasBigEndianness", not "HasBigEndian", which means "has a big-endian representation" :)

...

How about class names LittleEndianStorable or LittleEndianOrder? I think I like LittleEndianStorable more, since it makes clear, that this is a class similar to Storable but specialised to Little Endian byte order. Yes, LittleEndianStorable is also OK.

...

I'd like to have GHC specific code in a separate module instead of preprocessor instructions, that become quickly confusing if you add support for more compilers (say JHC). You may put GHC specific code into a private module, say Data.Storable.Endian.Private, and create two instances of this module in two directories, say 'generic' and 'ghc'. Then in storable-endian.cabal you write:

-- for Data.Storable.Endian Hs-Source-Dirs: src

-- for Data.Storable.Endian.Private If impl(ghc) Hs-Source-Dirs: ghc Else Hs-Source-Dirs: generic

This makes sense, too - now I know how to provide support for several compilers, thanks :) I tried to actually do this, but it turned out somewhat of a mess - will fix it a while later.

...

... and thank you very much for creating this package!

Hope someone finds it useful - I thought I needed it (in http://github.com/jkff/greg), but turned out I didn't :) -- Eugene Kirpichov Senior Software Engineer, Grid Dynamics http://www.griddynamics.com/

Henning Thielemann

6:54 a.m.

On Fri, 24 Dec 2010, Eugene Kirpichov wrote:

...

I don't think I'd like to allocate memory in these functions - I expect them to have very predictable and very high performance.

I'm afraid on ix86 it is not possible to move a Double that is stored in (two 32 bit) general purpose registers over to the FPU or to an XMM register. That is, what happens if you read a bit pattern from memory while reversing the order and then converting to double is actually on machine level that it is written back to memory and then loaded into FPU or SSE unit. I'm not entirely sure, you better check the assembly code that your Haskell code generates.

...

I'm using unsafeCoerce because it allows me to treat all these types in the same fashion (which is good for correctness and readability), and unsafeCoerce's semantics is identical to what is expected of this code - interpret a memory area as a value of a different type.

That's also the purpose of castPtr. It is certainly also unsafe, but not as unsafe, since you only convert between "dead bit patterns" and not between "life Haskell values". :-)

Henk-Jan van Tuyl

9:08 a.m.

On Fri, 24 Dec 2010 11:36:23 +0100, Eugene Kirpichov wrote:

...

Hello,

2010/12/24 Henning Thielemann :

...
: :

...
Maybe you succeed to find out machine endianess at compile time. Then you could use native memory access if the user requested endianess is the one of the machine. If you achieve this, you could implement peek and poke of Float and Double by just reversing the byte order into an 'alloca'd memory area and peek from and poke into this one with a native peek or poke, respectively. That is you would not need any unsafeCoerce anymore. I don't think I'd like to allocate memory in these functions - I expect them to have very predictable and very high performance. I'm using unsafeCoerce because it allows me to treat all these types in the same fashion (which is good for correctness and readability), and unsafeCoerce's semantics is identical to what is expected of this code - interpret a memory area as a value of a different type.

You're right however that some support could be added for host endianness - once I think something up, I'll do it; maybe this could be done with a couple unsafePerformIOs and runtime tests :)

You could use ADNS.Endian.endian from package hsdns in your Setup.hs to define endianness at compile time. Regards, Henk-Jan van Tuyl -- http://Van.Tuyl.eu/ http://members.chello.nl/hjgtuyl/tourdemonad.html --

Henning Thielemann

9:21 a.m.

On Fri, 24 Dec 2010, Henk-Jan van Tuyl wrote:

...

You could use ADNS.Endian.endian from package hsdns in your Setup.hs to define endianness at compile time.

Cool, it's already there! However I would not recommend to let a low-level library depend on a higher-level one. I think it would be cleaner to move the ADNS.Endian module to storable-endian package or even to a separate package.

Peter Simons

9:42 a.m.

Hi guys,

...

...
You could use ADNS.Endian.endian from package hsdns in your Setup.hs to define endianness at compile time.

Cool, it's already there! However I would not recommend to let a low-level library depend on a higher-level one. I think it would be cleaner to move the ADNS.Endian module to storable-endian package or even to a separate package.

yes, I agree. hsdns should re-use storable-endian, not the other way round. The API offered by storable-endian is rather similar to the functions hsdns currently uses internally, so it should be fairly straightforward to adapt. Take care, Peter

Antoine Latter

9:22 p.m.

On Fri, Dec 24, 2010 at 8:08 AM, Henk-Jan van Tuyl wrote:

...

You could use ADNS.Endian.endian from package hsdns in your Setup.hs to define endianness at compile time.

Regards, Henk-Jan van Tuyl

It looks like I've reimplemented the same thing in its own package: http://hackage.haskell.org/package/byteorder With pretty much the same technique. Take care, Antoine

Eugene Kirpichov

25 Dec 25 Dec

2:31 a.m.

Great! Antoine, would you perhaps then update the cabal description of the package to include the word "endianness" in it? I Ctrl+F-ed "endian" through hackage before writing storable-endian and did not find your package. 2010/12/25 Antoine Latter :

...

On Fri, Dec 24, 2010 at 8:08 AM, Henk-Jan van Tuyl wrote:

...
You could use ADNS.Endian.endian from package hsdns in your Setup.hs to define endianness at compile time.

Regards, Henk-Jan van Tuyl

It looks like I've reimplemented the same thing in its own package: http://hackage.haskell.org/package/byteorder

With pretty much the same technique.

Take care, Antoine

-- Eugene Kirpichov Senior Software Engineer, Grid Dynamics http://www.griddynamics.com/

Antoine Latter

2:55 a.m.

On Sat, Dec 25, 2010 at 1:31 AM, Eugene Kirpichov wrote:

...

Great! Antoine, would you perhaps then update the cabal description of the package to include the word "endianness" in it? I Ctrl+F-ed "endian" through hackage before writing storable-endian and did not find your package.

It is done! Although I spelled it wrong on my first try. Take care, Antoine

...

2010/12/25 Antoine Latter :

...
On Fri, Dec 24, 2010 at 8:08 AM, Henk-Jan van Tuyl wrote:

...
You could use ADNS.Endian.endian from package hsdns in your Setup.hs to define endianness at compile time.

Regards, Henk-Jan van Tuyl

It looks like I've reimplemented the same thing in its own package: http://hackage.haskell.org/package/byteorder

With pretty much the same technique.

Take care, Antoine

-- Eugene Kirpichov Senior Software Engineer, Grid Dynamics http://www.griddynamics.com/

Eugene Kirpichov

3:01 a.m.

Thanks! Now I'll use it in storable-endian as soon as I get around to it :) 2010/12/25 Antoine Latter :

...

On Sat, Dec 25, 2010 at 1:31 AM, Eugene Kirpichov wrote:

...
Great! Antoine, would you perhaps then update the cabal description of the package to include the word "endianness" in it? I Ctrl+F-ed "endian" through hackage before writing storable-endian and did not find your package.

It is done! Although I spelled it wrong on my first try.

Take care, Antoine

...
2010/12/25 Antoine Latter :

...
On Fri, Dec 24, 2010 at 8:08 AM, Henk-Jan van Tuyl wrote:

...
You could use ADNS.Endian.endian from package hsdns in your Setup.hs to define endianness at compile time.

Regards, Henk-Jan van Tuyl

It looks like I've reimplemented the same thing in its own package: http://hackage.haskell.org/package/byteorder

With pretty much the same technique.

Take care, Antoine

-- Eugene Kirpichov Senior Software Engineer, Grid Dynamics http://www.griddynamics.com/

-- Eugene Kirpichov Senior Software Engineer, Grid Dynamics http://www.griddynamics.com/

Eugene Kirpichov

26 Dec 26 Dec

6:13 a.m.

So I got around to it. storable-endian 0.2.3 released, see code: https://github.com/jkff/storable-endian/blob/master/Data/Storable/Endian.hs There's some boilerplate there, but I think it's tractable. 2010/12/25 Eugene Kirpichov :

...

Thanks! Now I'll use it in storable-endian as soon as I get around to it :)

2010/12/25 Antoine Latter :

...
On Sat, Dec 25, 2010 at 1:31 AM, Eugene Kirpichov wrote:

...
Great! Antoine, would you perhaps then update the cabal description of the package to include the word "endianness" in it? I Ctrl+F-ed "endian" through hackage before writing storable-endian and did not find your package.

It is done! Although I spelled it wrong on my first try.

Take care, Antoine

...
2010/12/25 Antoine Latter :

...
On Fri, Dec 24, 2010 at 8:08 AM, Henk-Jan van Tuyl wrote:

...
You could use ADNS.Endian.endian from package hsdns in your Setup.hs to define endianness at compile time.

Regards, Henk-Jan van Tuyl

It looks like I've reimplemented the same thing in its own package: http://hackage.haskell.org/package/byteorder

With pretty much the same technique.

Take care, Antoine

-- Eugene Kirpichov Senior Software Engineer, Grid Dynamics http://www.griddynamics.com/

-- Eugene Kirpichov Senior Software Engineer, Grid Dynamics http://www.griddynamics.com/

-- Eugene Kirpichov Senior Software Engineer, Grid Dynamics http://www.griddynamics.com/

Aaron Gray

10:13 a.m.

Hi Eugene, I have been looking to do a ActionScript backend, it needs 24bit values. Would it be possible to add some 24bit peeks and pokes, please. Aaron On 26 December 2010 11:13, Eugene Kirpichov wrote:

...

So I got around to it. storable-endian 0.2.3 released, see code: https://github.com/jkff/storable-endian/blob/master/Data/Storable/Endian.hs There's some boilerplate there, but I think it's tractable.

2010/12/25 Eugene Kirpichov :

...
Thanks! Now I'll use it in storable-endian as soon as I get around to it :)

2010/12/25 Antoine Latter :

...
On Sat, Dec 25, 2010 at 1:31 AM, Eugene Kirpichov wrote:

...
Great! Antoine, would you perhaps then update the cabal description of the package to include the word "endianness" in it? I Ctrl+F-ed "endian" through hackage before writing storable-endian and did not find your package.

It is done! Although I spelled it wrong on my first try.

Take care, Antoine

...
2010/12/25 Antoine Latter :

...
On Fri, Dec 24, 2010 at 8:08 AM, Henk-Jan van Tuyl wrote:

...
You could use ADNS.Endian.endian from package hsdns in your Setup.hs

to

...
define endianness at compile time.

Regards, Henk-Jan van Tuyl

It looks like I've reimplemented the same thing in its own package: http://hackage.haskell.org/package/byteorder

With pretty much the same technique.

Take care, Antoine

-- Eugene Kirpichov Senior Software Engineer, Grid Dynamics http://www.griddynamics.com/

-- Eugene Kirpichov Senior Software Engineer, Grid Dynamics http://www.griddynamics.com/

-- Eugene Kirpichov Senior Software Engineer, Grid Dynamics http://www.griddynamics.com/

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

5260

Age (days ago)

5264

Last active (days ago)

List overview

Download

20 comments

7 participants

participants (7)

Aaron Gray
Antoine Latter
Eugene Kirpichov
Gábor Lehel
Henk-Jan van Tuyl
Henning Thielemann
Peter Simons