MarshalUtils ~~~~~~~~~~~~ The reason that I don't like MarshalUtils as it is are the following:
(1) It isn't H98 and in fact its interface cannot be implemented with H98[1]. The problem are the routines with signatures in patterns:
malloc :: Storable a => IO (Ptr a) malloc :: IO (Ptr a) = mallocBytes (sizeOf (undefined::a))
alloca :: Storable a => (Ptr a -> IO b) -> IO b alloca (act :: Ptr a -> IO b) = allocaBytes (sizeOf (undefined::a)) act
<etc>
How about this: malloc :: Storable a => IO (Ptr a) malloc = res where res = mallocBytes (sizeOf (unsafePerformIO (res>>=peek))) :-) The result type signatures are just a convenient (and much more readable) way to write these functions.
(2) pokeArray, peekArray, etc are really part of what should go into the high-level marshalling.
Ok. Actually I consider the current libraries to be on the verge of usability, modulo the missing string marshalling and error functionality.
The Marshal modules ~~~~~~~~~~~~~~~~~~~ I propose the following set of modules:
- MarshalAlloc: The allocation routines that are in MarshalUtils now, but modified as outlined above.
- MarshalArray: The array marshalling routines
- MarshalString: Most people seem to be of the opinion that we have to handle strings specially due to Unicode conversion requirements. So, this is were string marshalling goes.
I've been wondering whether we should have something like newtype CString = CString (Ptr CChar) which would allow GHC to replace the representation with a ByteArray leading to a much more efficient implementation of withCString. There are other reasons why having a special CString type might be a good idea: a C string really is more than just a pointer to char, it is an array with a zero terminator and is specified in some kind of encoding. Having a special CString type would provide more type safety. One problem with the optimisation is that GHC can't handle ByteArrays being *returned* from a foreign function.
- MarshalError: Combinators to test for common error conditions. The main open problem here is the handling of errno, I think.
- MarshalConv: The C2HS IntConv, BoolConv, etc classes. Marcin just uses `fromIntegral' here, but I think that this is too limited - eg, it doesn't handle Bool.
Could you explain the motivation for these? I can see the need for some way to convert Bool<=>CInt (which is defined by ISO C), but can't the others be just fromIntegral?
Moreover, soemthing along the C2HS ToAddr/FromAddr classes, but it probably makes to have `stdArrayPtr' etc as additional member functions (this is similar to the newXXX routines that are in MarshalUtils now, but with additional overloading). Furthermore, Marcin-syle withXXX routines would fo here.
I'm not sure how ToAddr/FromAddr would work. From looking at C2HS, they seem to provide a consistent interface to marshalling that is specific to a given Haskell type. I haven't found a need for something like this in my own code yet, so again could you motivate?
- MarshalUtils: The memcpy/memmove stuff, indexPtr (if it still makes sense), and whatever else doesn't get its own module.
I think, we should leave C2HS's marshX routines out. We discussed how to make them arity independent for quite a while and AFAIR, there wasn't a really good proposal. Finally, all the above modules should be jointly exported by a module `Marshal'.
Sounds fine. Cheers, Simon
Thu, 4 Jan 2001 02:59:05 -0800, Simon Marlow <simonmar@microsoft.com> pisze:
Actually I consider the current libraries to be on the verge of usability, modulo the missing string marshalling and error functionality.
For both subjects I have a proposal in QForeign. I have no doubts about the strings story, except the internals of the charset conversion machinery. Error handling functions are more arbitrarily chosen.
I've been wondering whether we should have something like
newtype CString = CString (Ptr CChar)
which would allow GHC to replace the representation with a ByteArray leading to a much more efficient implementation of withCString.
Do you have the overhead of malloc in mind? If so, perhaps a generic stack allocator would be better. Anyway, conversions between strings which involve charset conversions require a more sophisticated pattern of memory allocation. It is not known how much space will be needed in advance, it can only be estimated. It is known when the string has already been stored in some memory directly accessible to C functions (in the common case of using the default conversion). Doing it on ByteArrays would be very tricky. There is no reallocByteArray and ByteArrays are not convertible to Ptrs (I currently assume that a conversion implemented in C will be exposed as a function working on Ptrs). My current strategy is to use malloc directly instead of alloca. If there was a faster alloca allocator, it could make sense to make a guess about the size and use alloca, resorting to malloc if it didn't fit. It would have to be significantly faster to think about making it more complex at all. The conversion itself can be a larger overhead than malloc. -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
qrczak@knm.org.pl (Marcin 'Qrczak' Kowalczyk) wrote,
My current strategy is to use malloc directly instead of alloca. If there was a faster alloca allocator, it could make sense to make a guess about the size and use alloca, resorting to malloc if it didn't fit. It would have to be significantly faster to think about making it more complex at all. The conversion itself can be a larger overhead than malloc.
If there were a faster alloca, it would still speed up the common case where there is no conversion or the initial size estimate is correct. Cheers, Manuel
Sat, 06 Jan 2001 22:37:35 +1100, Manuel M. T. Chakravarty <chak@cse.unsw.edu.au> pisze:
If there were a faster alloca, it would still speed up the common case where there is no conversion or the initial size estimate is correct.
I'm afraid the common case includes a conversion. It would indeed speed up the case when the size estimate is correct, and slow down the case where it is not correct (wasting some memory). Most of the time the estimate will be correct, but it will have to be measured if the speedup is worth the compilcation (when we have a faster alloca at all)... -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
qrczak@knm.org.pl (Marcin 'Qrczak' Kowalczyk) wrote,
Sat, 06 Jan 2001 22:37:35 +1100, Manuel M. T. Chakravarty <chak@cse.unsw.edu.au> pisze:
If there were a faster alloca, it would still speed up the common case where there is no conversion or the initial size estimate is correct.
I'm afraid the common case includes a conversion.
Why? Manuel
Simon Marlow <simonmar@microsoft.com> wrote,
- MarshalString: Most people seem to be of the opinion that we have to handle strings specially due to Unicode conversion requirements. So, this is were string marshalling goes.
I've been wondering whether we should have something like
newtype CString = CString (Ptr CChar)
which would allow GHC to replace the representation with a ByteArray leading to a much more efficient implementation of withCString. There are other reasons why having a special CString type might be a good idea: a C string really is more than just a pointer to char, it is an array with a zero terminator and is specified in some kind of encoding. Having a special CString type would provide more type safety.
One problem with the optimisation is that GHC can't handle ByteArrays being *returned* from a foreign function.
I think that this is a good idea. Especially, because of the encoding business, Strings are special and it is better to make this explicit in the types. Actually, given this unicode mess, how are we supposed to handle individual `Char's? A Haskell `Char' may expand to a sequence of 8bit `char's. That's a problem when the C side only expects a `char' and not a `*char'.
- MarshalConv: The C2HS IntConv, BoolConv, etc classes. Marcin just uses `fromIntegral' here, but I think that this is too limited - eg, it doesn't handle Bool.
Could you explain the motivation for these? I can see the need for some way to convert Bool<=>CInt (which is defined by ISO C), but can't the others be just fromIntegral?
One reason for extra functions is that we might want to handle overflows differently. I think, the Haskell report doesn't specify what happens in case of an overflow. In a binding, I can imagine that I want to abort in case of overflows during testing and in production code truncate the values. Furthermore, I have instance IntConv Char where cToInt = ord cFromInt = chr
Moreover, soemthing along the C2HS ToAddr/FromAddr classes, but it probably makes to have `stdArrayPtr' etc as additional member functions (this is similar to the newXXX routines that are in MarshalUtils now, but with additional overloading). Furthermore, Marcin-syle withXXX routines would fo here.
I'm not sure how ToAddr/FromAddr would work. From looking at C2HS, they seem to provide a consistent interface to marshalling that is specific to a given Haskell type. I haven't found a need for something like this in my own code yet, so again could you motivate?
The member function `stdAddr' from `ToAddr' is similar to MarshalUtils' `new' and `addrStdKeep' (and the derived `addrStd') are the matching deallocation routine (which MarshalUtils doesn't have) - the withXXX routines combine the two. Now, the current routines in MarshalUtils are overloaded via `Storable', which works fine for the primitives (in fact C2HS uses a default method for the primitives). It becomes interesting for more complex types. For example, remember the case where Marcin had to store a nullFunPtr on the C side and explicitly test for it before calling freeHaskellFunPtr. I think, it was Malcolm who argued that Marcin should better use `Maybe (FunPtr a)' on the Haskell side. One reason that Marcin didn't do that might be that he didn't have ToAddr/FromAddr, but only functions overloaded via `Storable'. I use instance ToAddr a => ToAddr (Maybe a) where stdAddr Nothing = nullAddr stdAddr (Just x) = stdAddr x instance FromAddr a => FromAddr (Maybe a) where addrStdKeep addr | addr == nullAddr = return Nothing | otherwise = do x <- addrStdKeep addr return $ Just x This transparently handles the `Maybe' representation for NULL pointers, without cluttering the marshalling code with special code for `Maybe'. I also have a special case for `String's. Moreover, a binding author can add other instances for binding specific types. This happens, for example, in the GDK binding - however, there arguably overloading `Storable' would have been sufficient. It might be sufficient to have one additional class instead of two (at least, I don't see anymore why I originally used two). Cheers, Manuel
On Sat, 6 Jan 2001, Manuel M. T. Chakravarty wrote:
I think that this is a good idea. Especially, because of the encoding business, Strings are special and it is better to make this explicit in the types.
I'm not sure if this is a good idea. I haven't seen convincing arguments yet. Let's keep it simpler.
Actually, given this unicode mess, how are we supposed to handle individual `Char's?
Depending what do you want to do. You may pass it as Char and get HsChar in C. You may convert [ch] to a C string (which may be longer than one character) and get char * in C. You may need it in an ASCII context (e.g. a commandline flag) and cast it to CChar (there is castCharToCChar in QForeign); "cast" in the name suggests that it preserves the physical contents rather than the meaning.
A Haskell `Char' may expand to a sequence of 8bit `char's. That's a problem when the C side only expects a `char' and not a `*char'.
No library will be able to represent Char or wchar_t in a single char.
Could you explain the motivation for these? I can see the need for some way to convert Bool<=>CInt (which is defined by ISO C), but can't the others be just fromIntegral?
One reason for extra functions is that we might want to handle overflows differently.
We would need to use a different interface to signal the overflow. Some exception monad (Maybe or IO). These additional functions don't provide it anyway and I will stick with fromIntegral - the most natural Haskell's way of converting integers.
Furthermore, I have
instance IntConv Char where cToInt = ord cFromInt = chr
No external C library will want either HsChar or HsInt. When we are writing the C part ourselves, we may choose the type on the C side without conversion in Haskell. For working internally to Haskell ord and chr are sufficient.
Now, the current routines in MarshalUtils are overloaded via `Storable', which works fine for the primitives (in fact C2HS uses a default method for the primitives). It becomes interesting for more complex types.
When there is more than one way to pass a complex type, I would just use separate functions if a Storable instance does not fit. I would represent a pointer to a C string as Ptr CChar in Haskell and use different functions for distinguishing what we want to do with it. Passing the string to C functions is not the only thing we might want to do with it - e.g. we could wish to realloc it, or work on [CChar] in Haskell, or convert it to an aray of bytes, or pass a pointer to the middle to a C function. IMHO it's better and less constraining to be explicit, keep the type and make approprate functions as needed. withCString is not that harder than toAddr. There are too many possible operations to express them in one overloaded function on different types.
For example, remember the case where Marcin had to store a nullFunPtr on the C side and explicitly test for it before calling freeHaskellFunPtr. I think, it was Malcolm who argued that Marcin should better use `Maybe (FunPtr a)' on the Haskell side. One reason that Marcin didn't do that might be that he didn't have ToAddr/FromAddr, but only functions overloaded via `Storable'.
ToAddr/FromAddr would not help here, unless they provided some appropriately overloaded variant of peek. [...]
This transparently handles the `Maybe' representation for NULL pointers, without cluttering the marshalling code with special code for `Maybe'.
Passing Maybe as a maybe-null pointer is easy already: maybe ($nullPtr) withCString maybeString $ \ptr -> ... For the other direction a single function could be added: maybeNull :: (Ptr a -> IO b) -> Ptr a -> IO (Maybe b) maybeNull f ptr | ptr == nullPtr = return Nothing | otherwise = liftM Just (f ptr) Use it as such: do maybeString <- maybeNull peekCString ptr ... -- Marcin 'Qrczak' Kowalczyk
"Marcin 'Qrczak' Kowalczyk" <mk167280@zodiac.mimuw.edu.pl> wrote,
On Sat, 6 Jan 2001, Manuel M. T. Chakravarty wrote:
I think that this is a good idea. Especially, because of the encoding business, Strings are special and it is better to make this explicit in the types.
I'm not sure if this is a good idea. I haven't seen convincing arguments yet. Let's keep it simpler.
The opinion trend for MarshalXXX seems to be clear: Keep it simple. So, ok, let's leave it out.
A Haskell `Char' may expand to a sequence of 8bit `char's. That's a problem when the C side only expects a `char' and not a `*char'.
No library will be able to represent Char or wchar_t in a single char.
Sure. C can choose between char and wchar_t, we only have Char in Haskell. That is what is bothering me.
Could you explain the motivation for these? I can see the need for some way to convert Bool<=>CInt (which is defined by ISO C), but can't the others be just fromIntegral?
One reason for extra functions is that we might want to handle overflows differently.
We would need to use a different interface to signal the overflow. Some exception monad (Maybe or IO). These additional functions don't provide it anyway and I will stick with fromIntegral - the most natural Haskell's way of converting integers.
That's not what I meant. When an overflow is encountered, an `error' can be raised. The point is that during "testing" a binding, I want to see where overflows happen and not just get a core dump later on. Anyway, as I wrote earlier, let's drop MarshalConv.
Now, the current routines in MarshalUtils are overloaded via `Storable', which works fine for the primitives (in fact C2HS uses a default method for the primitives). It becomes interesting for more complex types.
When there is more than one way to pass a complex type, I would just use separate functions if a Storable instance does not fit.
I would represent a pointer to a C string as Ptr CChar in Haskell and use different functions for distinguishing what we want to do with it. Passing the string to C functions is not the only thing we might want to do with it - e.g. we could wish to realloc it, or work on [CChar] in Haskell, or convert it to an aray of bytes, or pass a pointer to the middle to a C function. IMHO it's better and less constraining to be explicit, keep the type and make approprate functions as needed. withCString is not that harder than toAddr. There are too many possible operations to express them in one overloaded function on different types.
I see your point, however, personally, I still prefer to have overloading for the common cases (that what we want most of the time when given a particular type) and use explicit functions only for the exceptions. That makes the resulting code more readable in my opinion - as it contains less clutter for the common case. But I guess, this is a matter of personal style.
For example, remember the case where Marcin had to store a nullFunPtr on the C side and explicitly test for it before calling freeHaskellFunPtr. I think, it was Malcolm who argued that Marcin should better use `Maybe (FunPtr a)' on the Haskell side. One reason that Marcin didn't do that might be that he didn't have ToAddr/FromAddr, but only functions overloaded via `Storable'.
ToAddr/FromAddr would not help here, unless they provided some appropriately overloaded variant of peek.
The definitions that I gave in my previous mail *do* help in this case. I use them myself in C2HS.
[...]
This transparently handles the `Maybe' representation for NULL pointers, without cluttering the marshalling code with special code for `Maybe'.
Passing Maybe as a maybe-null pointer is easy already: maybe ($nullPtr) withCString maybeString $ \ptr -> ...
For the other direction a single function could be added: maybeNull :: (Ptr a -> IO b) -> Ptr a -> IO (Maybe b) maybeNull f ptr | ptr == nullPtr = return Nothing | otherwise = liftM Just (f ptr)
Use it as such: do maybeString <- maybeNull peekCString ptr ...
Yes, let's use `maybeNull' and Sven's `maybeNothing'. Cheers, Manuel
"Manuel M. T. Chakravarty" wrote:
[...] I see your point, however, personally, I still prefer to have overloading for the common cases (that what we want most of the time when given a particular type) and use explicit functions only for the exceptions. That makes the resulting code more readable in my opinion - as it contains less clutter for the common case. But I guess, this is a matter of personal style. [...]
In the case of CString/Ptr CChar it's not only that: Storable.sizeOf should never use its argument, but it would have to in the case of variable length C strings. And what should be the semantics of {peek,poke}ElemOff here? In a nutshell: instance Storable CString makes sense only partially, so I fear there is no other option than using non-overloaded functions. Or we have to re-design Storable somehow (alignment, peek, peekByte, poke, and pokeByte *do* make sense for CString). Cheers, Sven
Sven Panne <Sven.Panne@informatik.uni-muenchen.de> wrote,
"Manuel M. T. Chakravarty" wrote:
[...] I see your point, however, personally, I still prefer to have overloading for the common cases (that what we want most of the time when given a particular type) and use explicit functions only for the exceptions. That makes the resulting code more readable in my opinion - as it contains less clutter for the common case. But I guess, this is a matter of personal style. [...]
In the case of CString/Ptr CChar it's not only that: Storable.sizeOf should never use its argument, but it would have to in the case of variable length C strings. And what should be the semantics of {peek,poke}ElemOff here? In a nutshell: instance Storable CString makes sense only partially, so I fear there is no other option than using non-overloaded functions. Or we have to re-design Storable somehow (alignment, peek, peekByte, poke, and pokeByte *do* make sense for CString).
That is why I was proposing a two-level hierachy with Storable and ToAddr/FromAddr. But it surely requires some more experimentation. Cheers, Manuel
Sat, 06 Jan 2001 02:24:00 +1100, Manuel M. T. Chakravarty <chak@cse.unsw.edu.au> pisze:
Actually, given this unicode mess, how are we supposed to handle individual `Char's? A Haskell `Char' may expand to a sequence of 8bit `char's. That's a problem when the C side only expects a `char' and not a `*char'.
And BTW: many languages have strings only and don't work on individual characters at all. Libraries usually work on strings. IMHO it's not a problem at all to not provide much of support for individual characters. Only strings are important in practice. Conversion which changes the length might be a problem only if a library works on positions within text. For example readline can show the whole line being read, and the cursor position. But conversion of the whole line to Haskell does not allow determining the right cursor position in terms of Haskell characters. Well, readline does not support multibyte encodings anyway, and it can be assumed that positions will match in practice. (I once hacked bash to work on a UTF-8 terminal but I don't think the patch is used anywhere.) Handling strings in C is so "manual" that I don't think much fun can be provided besides withCString / mallocCString / peekCString. Somebody might wrap a string in ForeignPtr CChar to avoid a conversion to a Haskell string, but it's already easy enough, and IMHO strings don't need yet another type just for interfacing with C. -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
participants (5)
-
Manuel M. T. Chakravarty -
Marcin 'Qrczak' Kowalczyk -
qrczak＠knm.org.pl -
Simon Marlow -
Sven Panne