quoth Evan Laforge

Just from personal experience, the fact that haskell types like Bool and Char are Storable combined with hsc's low level nature makes it easy to create memory corruption bugs by writing `(#poke Struct field) flag` instead of `(#poke Struct field) (toBool flag)`. The former will compile and likely work most of the time, but will corrupt memory, which will eventually result in a nondeterministic GC crash some time after the actual corruption. In my case, it took over a year to find the bug.

Could you explain this problem a little further? I looked in my code and found lots of stuff like `(#poke termios, c_iflag) a iflag', so was alarmed for a moment, but naturally I've declared iflag Word32, via the hsc #type macro. And I'd expect the targets of poking to be like that - full of CChar and Word32 etc. - foreign types of the appropriate size. The idea that there'd be anything else in there changes the perspective somewhat, and if that's common, indeed the tutorial should account for it. (Or maybe I completely misunderstand what's going on here!) Donn

quoth Evan Laforge ...

If you've ensured that iflag is a C type, then you've avoided the problem. But if, for instance, iflag was a Char, not a CChar, and you poke it into a 'char' struct field forgetting to convert to a CChar, you'll get memory corruption. I don't know if it's a common mistake, but I sure made it (very infrequently, but once is enough, in fact once is even worse), and the compiler won't tell you if you did. When I mentioned it on the list way back when no one responded, so maybe other people don't fall into that trap.

Maybe they don't! I guess it isn't so much about exactly what you were up to, but for the sake of getting to whether there's an issue here for the tutorial, I wrote up a little example program, with CChar and Char. The commented alternatives work as well, at least it looks fine to me. Notes on this: - the C struct is { char a; char b; char c; } - the Haskell T struct uses CChar, and I assert that this is the only sane option -- no storable struct for foreign use should ever have a field type like Char. - that means the Storable instance in question is CChar, and it looks to me like poke reliably writes exactly one byte in this case, whatever value is supplied (I also tried Int.) - one might very well manage to keep all the poking to t fields in the T Storable instance - that's what I'd expect the tutorial to focus on. Not that it makes any great difference, but I'm just saying that the "ypoke" function in the example is there purely for the purpose of testing that Char/CChar thing you're talking about, and would be somewhat outside what I see as core usage. Donn ------------ {-# LANGUAGE ForeignFunctionInterface #-} module Main (main) where import Foreign import Foreign.C #include "ffipokehsc.h" data T = T { taflag :: CChar , tbflag :: CChar , tcflag :: CChar } deriving Show instance Storable T where sizeOf _ = #size struct t alignment _ = alignment (undefined::CDouble) peek p = do aflag <- (#peek struct t, a) p bflag <- (#peek struct t, b) p cflag <- (#peek struct t, c) p return (T aflag bflag cflag) poke p (T aflag bflag cflag) = do (#poke struct t, a) p aflag (#poke struct t, b) p bflag (#poke struct t, c) p cflag -- ypoke :: CChar -> CChar -> CChar -> IO T ypoke :: Char -> Char -> Char -> IO T ypoke a b c = alloca $ \ tp -> do (#poke struct t, a) tp a (#poke struct t, b) tp b (#poke struct t, c) tp c peek tp -- main = ypoke 97 98 99 >>= print -- main = ypoke 'a' 'b' 'c' >>= print tptr :: T -> IO (Ptr T) tptr t = alloca $ \ pt -> do poke pt t return pt main = do p <- tptr (T 97 98 99) t <- peek p print t

quoth Evan Laforge ...

Yes, it would be safe to say that all haskell data types which are serializable to C should have only C types. That also avoids the problem. However, you have to convert between haskell and C at some point, and that means you wind up with C and haskell duplicates of all records, so each one is actually expressed in 3 places: the C struct, the haskell "CType" record, and the haskell type record.

That could happen ... I suppose there's no one way to do it that fits every application, but often enough I can make do with the Haskell "CTYpe" record, and it gives me a Storable value that has a valid type enforced relationship to the C struct. You wouldn't always need or want a Haskell version of your C struct, per se, but if you do.

Another reason you wind up with Storable instances of non-CType records is Data.Vector.Storable. It's very tempting to simply reuse that to pass to C, and maybe it's initially fine because it has Ints or Word32s or something "safe", but then one day 2 years later someone who doesn't know about that adds a Char field and now you're in trouble.

Data.Vector.Storable is a new one on me! (I see it's in the non-portable, experimental category, so it figures.) The documentation I'm looking at seems to be saying it wouldn't be "instantiated" for Char, though?

...

ypoke :: Char -> Char -> Char -> IO T ypoke a b c = alloca $ \ tp -> do (#poke struct t, a) tp a (#poke struct t, b) tp b (#poke struct t, c) tp c peek tp

This is corrupting memory, since sizeOf 'c' == 4. Like I said, it will probably look like it works because it's usually just overwriting adjacent fields or perhaps alignment padding or maybe it's "safe" if it's on the stack, but you are likely to get mysterious crashes under load. Try changing the order of the pokes and see what happens.

Ah, you're right, a final poke to offset 0 overwrites everything. In retrospect I ... don't know what I was thinking! Donn

quoth Evan Laforge ...

Anyway, in the big picture, I think hsc2hs is just too low level. We shouldn't be having to manually poke structs at all, and it's fundamentally dangerous even if you use all C types because there's no typechecking. Nothing will help you when someone updates the struct and forgets to update the Storable instance. If I were starting again (or advising someone who was starting from scratch), I'd try really hard to find something that directly generates marshalling code from the .h file, perhaps c2hs can do that.

I'm not up for generating anything directly from a .h file, but just for fun I put a little time today into an hsc alternative with a little more type safety. It works to a certain extent with integral types, because I can determine the size of the C field and assign an appropriate Haskell foreign integral type. I used it to generate a module for struct termios, for which I've been using hsc2hs, and it did better than hsc2hs in an unexpected way - hsc2hs #type maps "unsigned long" to "Word32", but on MacOS X the field size is 8 bytes. Then I make a raft of peek and poke functions that take the native Haskell values that you specify, and convert them to the appropriate foreign types. This is based on a descriptor file where you specify the fields you want to use, your Haskell name and type for each, and whether it's integral or whatever. Integral or whatever is where the joy leaks out of the concept, though. I'd bet a quarter that at least one in every four .hsc files contains some custom peeking and poking for a struct field, stuff you'd never anticipate. Full support for all types seems like a nearly unbounded problem. hsc2hs could do essentially what I'm talking about, if it could tell you the size of a field. Then you could declare the foreign types like termiosC_lflag :: (#appendfieldsize Word struct termios, c_lflag) which would expand to termiosC_lflag :: Word64 (fieldsize would be sizeof(x->c_lflag) * 8) This would give us foreign types based on the C declaration. You'd have to do the conversions yourself where you want to use native types, in this scenario. Donn

quoth Richard A. O'Keefe, ...

(This actually strikes me as a flaw in Mac OS X: `typedef unsigned long tcflag_t;' should have been `typedef uint32_t tcflag_t;' because there's no _reason_ for the field size to change this way.)

True, it's crazy to have structures changing size like that when the effective size of each field is fixed per standard.

The fact that the sizes of things can vary between compilation environments on the same host is one of the reasons tools like hsc2hs are hard.

I have to confess that this was to some degree my error as well - I was looking at a somewhat elderly hsc2hs output that probably predated the current architecture and platform level, and when I run it now, hsc2hs does assign the appropriate size foreign type and the right offsets. Donn

Doesn't hsc2hs give you the size with #size? I feel like I'm not understanding your approach. I was thinking of using language-c to parse the .h file, and generate an hs file: structs.h: struct example { int a; char b; } Structs_generated.hs: module X.Y.Structs_generated where import X.Y.Structs (Example) poke_example_a :: (#type int) -> Ptr Example -> IO () poke_example_b :: (#type char) -> Ptr Example -> IO () example_alignment = 4 example_size = 5 You have to provide the Structs module that exports the types, this just makes sure that the pokes only work on the intended types. Then the poke functions tie the struct field name and its type and the record type. For the #type I could either then run this through hsc2hs to get the types, or just have a hardcoded mapping. #type sees through typedefs and the like, but all it can see is size and signedness. While I'm at it I could add foreign declarations for the function prototypes, and solve the problem of keeping those in sync. Another thing I like about this approach is that you import a generated file, but all the marshaling work is done in a plain .hs file. Putting code in a .hsc breaks ghci since you need to regenerate after each edit, and it breaks tools that want to parse hs source. I'm sure it's way more complicated than it seems though, things inevitably are. I read the c2hs docs but I kind of don't really get it yet. I should just download a project that uses it and see how it works. On Wed, Nov 12, 2014 at 5:03 PM, Richard A. O'Keefe wrote:

The fact that the sizes of things can vary between compilation environments on the same host is one of the reasons tools like hsc2hs are hard.

That just means you need the same compiler flags for the C side as for the haskell side, right? Presumably that's not too hard because you build it all together. Or is there some further complexity?

On Wed, Nov 12, 2014 at 10:23 PM, Donn Cave wrote:

Maybe it does, and I only need a clue to the syntax. I have seen it used only for the size of the struct; I'm looking for the size of a field - and devoid of parentheses or other adornments, so I can tack it onto something like Word or Int. You could define type aliases for other variable-size foreign types, like Float8 etc. and use #fieldsize to select them.

Oh I see. Maybe this: #let fieldsize t, f = "%lu", (unsigned long)sizeof(((t *)0)->f) c :: Word#{fieldsize example, c} Of course you will wind up with Word64 for a double which is not that great, and you lose signedness. The hsc_type macro has some ridiculous but effective hackery to figure that out. So I bet a #fieldtype macro can be defined without too much trouble, perhaps without even having to modify hsc2hs.

I don't know language-c. Lots of interesting potential there if it works well.

Just because I was curious, I wrote up an implementation last night. It converts: typedef int xy_t; struct example { xy_t a; }; to module Example_generated where import qualified Example as M #let alignment t = "%lu", (unsigned long)offsetof(struct {char x__; t (y__); }, y__) -- struct example sizeOf_example :: Int sizeOf_example = #size example alignment_example :: Int alignment_example = #{alignment example} poke_example_a :: (#type xy_t) -> Ptr M.Example -> IO () poke_example_a = (#poke example, a) peek_example_a :: Ptr M.Example -> IO (#type xy_t) peek_example_a = (#peek example, a) Of course now it occurs to me that if instead I emitted the typedef as 'type Xy_t = CInt' then I could hardcode the primitive C types and not need #type anymore. I'd want to emit type synonyms anyway for function typedefs. And naturally further complexity is in store for declarations like 'char c, *const *s'. One thing is I just ignore unnamed structs, since I couldn't figure out how to write a #poke for them. And then of course I want to filter by header so I don't always generate 50 zillion declarations from system headers, and now using #type instead of type synonyms looks better, otherwise I wind up needing hs versions of all system headers. So it seems feasible, but is naturally fraught with details that will turn it into a real project rather than a quick hack. Since I don't really plan to write any more FFIs in the near future, that's where I'll leave it :) Your #fieldtype macro certainly seems more practical to get something useful working quickly.

On Tue, Nov 11, 2014 at 11:37 AM, Niklas Hambüchen wrote:

Took me over a day to find, and it was just a 500 lines application.

More than a year, maybe two in my case, and was probably around 50k lines at the time, though only 1k in the FFI. It wouldn't show up in tests, except when it did, so I wrote all manner of special test frameworks to try to reproduce it reliably, nothing did. valgrind doesn't help with this kind of problem. I must have looked straight at the bad code a hundred times at least. Anyway it's a lesson I won't soon forget.