The major piece missing from the FFI library is the high-level marshalling support. Moreover, `MarshalUtils' isn't quite there. In fact, I think, we should have a family of MarshalXXX modules, which include what currently is in MarshalUtils plus some more stuff. MarshalUtils ~~~~~~~~~~~~ The reason that I don't like MarshalUtils as it is are the following: (1) It isn't H98 and in fact its interface cannot be implemented with H98[1]. The problem are the routines with signatures in patterns: malloc :: Storable a => IO (Ptr a) malloc :: IO (Ptr a) = mallocBytes (sizeOf (undefined::a)) alloca :: Storable a => (Ptr a -> IO b) -> IO b alloca (act :: Ptr a -> IO b) = allocaBytes (sizeOf (undefined::a)) act <etc> (2) pokeArray, peekArray, etc are really part of what should go into the high-level marshalling. Point (2) is easily resolved by defining a family of modules including all the necessary functionality. However, Point (1) is a problem. First of all, I guess, we have to revert to the signature for allocation that I originally used in the C2HS library: malloc :: Storable a => a -> IO Addr malloc dummy = mallocBytes (sizeOf dummy) All the other allocation routines have to be altered in a similar way. A similar problem arises for copyArray and moveArray, where I guess the additional argument is in practice more awkward as for the allocation routines. Finally, indexPtr is affected, which is especially annoying since it can't be used infix anymore after the change. The Marshal modules ~~~~~~~~~~~~~~~~~~~ I propose the following set of modules: - MarshalAlloc: The allocation routines that are in MarshalUtils now, but modified as outlined above. - MarshalArray: The array marshalling routines - MarshalString: Most people seem to be of the opinion that we have to handle strings specially due to Unicode conversion requirements. So, this is were string marshalling goes. - MarshalError: Combinators to test for common error conditions. The main open problem here is the handling of errno, I think. - MarshalConv: The C2HS IntConv, BoolConv, etc classes. Marcin just uses `fromIntegral' here, but I think that this is too limited - eg, it doesn't handle Bool. Moreover, soemthing along the C2HS ToAddr/FromAddr classes, but it probably makes to have `stdArrayPtr' etc as additional member functions (this is similar to the newXXX routines that are in MarshalUtils now, but with additional overloading). Furthermore, Marcin-syle withXXX routines would fo here. - MarshalUtils: The memcpy/memmove stuff, indexPtr (if it still makes sense), and whatever else doesn't get its own module. I think, we should leave C2HS's marshX routines out. We discussed how to make them arity independent for quite a while and AFAIR, there wasn't a really good proposal. Finally, all the above modules should be jointly exported by a module `Marshal'. Comments? Cheers, Manuel [1] Which unfortunately escaped me when we discussed MarshalUtils for the first time.
On Thu, 4 Jan 2001, Manuel M. T. Chakravarty wrote:
(1) It isn't H98 and in fact its interface cannot be implemented with H98[1].
It can.
malloc :: Storable a => IO (Ptr a) malloc :: IO (Ptr a) = mallocBytes (sizeOf (undefined::a))
malloc :: Storable a => IO (Ptr a) malloc = malloc' undefined where malloc' :: Storable a => a -> IO (Ptr a) malloc' undef = mallocBytes (sizeOf undef)
alloca :: Storable a => (Ptr a -> IO b) -> IO b alloca (act :: Ptr a -> IO b) = allocaBytes (sizeOf (undefined::a)) act
alloca :: Storable a => (Ptr a -> IO b) -> IO b alloca = alloca' undefined where alloca' :: Storable a => a -> (Ptr a -> IO b) -> IO b alloca' undef = allocaBytes (sizeOf undef)
(2) pokeArray, peekArray, etc are really part of what should go into the high-level marshalling.
IMHO MarshalUtils exported by Foreign is the right place for it. But I don't care how functions will be split into modules. String handling will be needed. It would be silly to provide strings but not provide arrays in general.
The C2HS IntConv, BoolConv, etc classes. Marcin just uses `fromIntegral' here, but I think that this is too limited - eg, it doesn't handle Bool.
You handle Bool by converting True to -1. I think that most libraries would expect 1 as True. Anyway, IMHO it's ok to leave this out, because it's easy to write (/= 0) in one direction, and (\b -> if b then 1 else 0) in the other. Well, for the latter it could make sense to provide a function which is related to Bool as maybe is related to Maybe. It's a plain if with a different order of arguments: bool :: a -> a -> Bool -> a -- Or the name cond might be better. Then the conversion from Bool is: bool 1 0. Or bool 0 1. I'm not sure which order is better (false true: consistent with maybe and ordering on Bool, true false: consistent with plain if). -- Marcin 'Qrczak' Kowalczyk
"Marcin 'Qrczak' Kowalczyk" <mk167280@zodiac.mimuw.edu.pl> wrote,
On Thu, 4 Jan 2001, Manuel M. T. Chakravarty wrote:
(1) It isn't H98 and in fact its interface cannot be implemented with H98[1].
It can.
malloc :: Storable a => IO (Ptr a) malloc :: IO (Ptr a) = mallocBytes (sizeOf (undefined::a))
malloc :: Storable a => IO (Ptr a) malloc = malloc' undefined where malloc' :: Storable a => a -> IO (Ptr a) malloc' undef = mallocBytes (sizeOf undef)
Cool! You are right, I completely forgot about local type declarations. That's good, your signatures are definitelty nicer.
IMHO MarshalUtils exported by Foreign is the right place for it. But I don't care how functions will be split into modules.
If we put everything into MarshalUtils it will get too big. So, why not structure it a bit, as we did with the other FFI modules, too.
String handling will be needed. It would be silly to provide strings but not provide arrays in general.
Sure, but as you pointed out: Strings are special, but internally rely on the array routines. So it makes sense, to have a module with the general arrays stuff plus an extra module with the special string handling.
The C2HS IntConv, BoolConv, etc classes. Marcin just uses `fromIntegral' here, but I think that this is too limited - eg, it doesn't handle Bool.
You handle Bool by converting True to -1. I think that most libraries would expect 1 as True.
Ok, I can change this.
Anyway, IMHO it's ok to leave this out, because it's easy to write (/= 0) in one direction, and (\b -> if b then 1 else 0) in the other.
But if it occurs repeatedly, it is much nicer to have a function (mind you, the whole high-level libraries is only about convenience anyway).
Well, for the latter it could make sense to provide a function which is related to Bool as maybe is related to Maybe. It's a plain if with a different order of arguments: bool :: a -> a -> Bool -> a -- Or the name cond might be better. Then the conversion from Bool is: bool 1 0. Or bool 0 1. I'm not sure which order is better (false true: consistent with maybe and ordering on Bool, true false: consistent with plain if).
I don't much like having explicit constants all over the place - especially because everybody will put 1 and 0 in the wrong order half the time. Cheers, Manuel
(1) It isn't H98 and in fact its interface cannot be implemented with H98[1]. The problem are the routines with signatures in patterns:
malloc :: Storable a => IO (Ptr a) malloc :: IO (Ptr a) = mallocBytes (sizeOf (undefined::a))
I discovered this problem, and its fix, just before Christmas when I tried to implement MarshallUtils for nhc98. I seem to have lost my source code right now, but as I recall, we can keep this signature just by wrapping the old implementation: malloc :: Storable a => IO (Ptr a) malloc = malloc_ undefined malloc_ :: Storable a => a -> IO (Ptr a) malloc_ x = mallocBytes (sizeOf x) Regards, Malcolm
I think we all agree now that malloc & friend can be written in H98 + FFI, so the only question is how they should be implemented in hslibs. The variant which uses local definitions instead of pattern type signatures is nicer IMHO because it *is* H98 and given GHC's optimizations, it doesn't make any performance difference. In general we should make hslibs as H98-conforming as possible and remove most GHCisms. For performance there's always #ifdef (well, at least if we consider piping Haskell sources through cpp as "standard"/H98). Manuel's MarshalXXX family look nice, but I'd like to summarize what exactly should go into what module, marking places which probably need more thoughts/discussions with `!!!!!': -- MarshalAlloc ------------------------------------------------------ malloc :: Storable a => IO (Ptr a) mallocBytes :: Int -> IO (Ptr a) alloca :: Storable a => (Ptr a -> IO b) -> IO b allocaBytes :: Int -> (Ptr a -> IO b) -> IO b reallocBytes :: Ptr a -> Int -> IO (Ptr a) free :: Ptr a -> IO () -- MarshalArray ------------------------------------------------------ mallocArray :: Storable a => Int -> IO (Ptr a) mallocArray0 :: Storable a => Int -> IO (Ptr a) allocaArray :: Storable a => Int -> (Ptr a -> IO b) -> IO b allocaArray0 :: Storable a => Int -> (Ptr a -> IO b) -> IO b reallocArray :: Storable a => Ptr a -> Int -> IO (Ptr a) reallocArray0 :: Storable a => Ptr a -> Int -> IO (Ptr a) !!!!! The current MarshalUtils uses !!!!! peekArray :: Storable a => Ptr a -> Int -> IO [a] !!!!! but the signature below looks more consistent with peekArray0: peekArray :: Storable a => Int -> Ptr a -> IO [a] peekArray0 :: (Storable a, Eq a) => a -> Ptr a -> IO [a] pokeArray :: Storable a => Ptr a -> [a] -> IO () pokeArray0 :: Storable a => a -> Ptr a -> [a] -> IO () withArray :: Storable a => [a] -> (Ptr a -> IO b) -> IO b withArray0 :: Storable a => a -> [a] -> (Ptr a -> IO b) -> IO b newArray :: Storable a => [a] -> IO (Ptr a) newArray0 :: Storable a => a -> [a] -> IO (Ptr a) -- MarshalString ----------------------------------------------------- !!!!! I'm with Marcin here: Let's keep things simple and do not !!!!! provide an opaque type CString here. We can't guess all !!!!! possible uses, so a type synonym would be the right thing !!!!! here. Using a pointer/length combo is quite common, too, !!!!! so this deserves another type synonym. Note that some !!!!! functions below look a bit strange, but symmetry rulez! :-) !!!!! Charset conversions are a diff{erent,icult} story and should !!!!! be handled by a separate module, so only the well-known !!!!! ignore-the-upper-half variants are given here. type CString = Ptr CChar type CStringLen = (CString, Int) mallocCString :: String -> IO CString mallocCStringLen :: String -> IO CStringLen peekCString :: CString -> IO String peekCStringLen :: CStringLen -> IO String pokeCString :: CString -> String -> IO () pokeCStringLen :: CStringLen -> String -> IO () withCString :: String -> (CString -> IO a) -> IO a withCStringLen :: String -> (CStringLen -> IO a) -> IO a newCString :: String -> IO CString newCStringLen :: String -> IO CStringLen -- MarshalError ------------------------------------------------------ !!!!! Haven't thought very deeply about this, but Marcin's QErrors !!!!! probably looks OK after a little polishing (use strerror, !!!!! provide a more complete throwXXX family, ...). And remember: !!!!! errno *is* a thread-local thing. :-) -- MarshalConv ------------------------------------------------------- !!!!! I can't see a compelling reason for this module, neither what !!!!! exactly should belong here. ISO C defines a mapping to 0/1, !!!!! which can easily be captured by fromBool/toBool (in MarshalUtils !!!!! below). Maybe is handled there, too. -- MarshalUtils ------------------------------------------------------ fromBool :: Num a => Bool -> a fromBool = fromIntegral . fromEnum toBool :: Num a => a -> Bool toBool = (/= 0) withObject :: Storable a => a -> (Ptr a -> IO b) -> IO b new :: Storable a => a -> IO (Ptr a) !!!!! Example usages: !!!!! do ... !!!!! mbInt <- maybeNull peek ptr1 !!!!! mbString <- maybeNull peekCString ptr2 !!!!! ..... !!!!! maybeNothing withObject Nothing $ \ptr -> ... !!!!! maybeNothing withCString (Just "foo!") $ \ptr -> ... !!!!! !!!!! Does anybody have better names for these two functions? !!!!! They sound OK for me, but I'm not a native speaker... maybeNull :: (Ptr a -> IO b) -> Ptr a -> IO (Maybe b) maybeNull f ptr | ptr == nullPtr = return Nothing | otherwise = liftM Just $ f ptr maybeNothing :: (a -> (Ptr b -> IO c) -> IO c) -> (Maybe a -> (Ptr b -> IO c) -> IO c) maybeNothing = maybe ($ nullPtr) copyArray :: Storable a => Ptr a -> Ptr a -> Int -> IO () copyBytes :: Ptr a -> Ptr a -> Int -> IO () moveArray :: Storable a => Ptr a -> Ptr a -> Int -> IO () moveBytes :: Ptr a -> Ptr a -> Int -> IO () !!!!! Do we really need this? indexPtr :: Storable a => Ptr a -> Int -> Ptr a As Manuel already wrote it's a good idea to leave C2HS's marshX routines out and to re-export all the modules above, but should this be done via a new module Marshal or should Foreign do this? Should Marshal re-export Foreign or vice-versa? Hmmm... And just a final topic which really should be settled: "Marshal" or "Marshall"?? :-} Cheers, Sven
Sven Panne <Sven.Panne@informatik.uni-muenchen.de> wrote,
I think we all agree now that malloc & friend can be written in H98 + FFI, so the only question is how they should be implemented in hslibs. The variant which uses local definitions instead of pattern type signatures is nicer IMHO because it *is* H98 and given GHC's optimizations, it doesn't make any performance difference. In general we should make hslibs as H98-conforming as possible and remove most GHCisms. For performance there's always #ifdef (well, at least if we consider piping Haskell sources through cpp as "standard"/H98).
If hslibs is meant to be used with systems other than ghc, too - which I think was the idea - there is no choice but to rewrite it into H98. But I guess this is essentially up to Mr. HsLibs aka SimonM.
-- MarshalArray ------------------------------------------------------
[...]
!!!!! The current MarshalUtils uses !!!!! peekArray :: Storable a => Ptr a -> Int -> IO [a] !!!!! but the signature below looks more consistent with peekArray0: peekArray :: Storable a => Int -> Ptr a -> IO [a] peekArray0 :: (Storable a, Eq a) => a -> Ptr a -> IO [a]
Sounds reasonable to me.
-- MarshalString -----------------------------------------------------
!!!!! I'm with Marcin here: Let's keep things simple and do not !!!!! provide an opaque type CString here. We can't guess all !!!!! possible uses, so a type synonym would be the right thing !!!!! here. Using a pointer/length combo is quite common, too, !!!!! so this deserves another type synonym. Note that some !!!!! functions below look a bit strange, but symmetry rulez! :-) !!!!! Charset conversions are a diff{erent,icult} story and should !!!!! be handled by a separate module, so only the well-known !!!!! ignore-the-upper-half variants are given here.
type CString = Ptr CChar type CStringLen = (CString, Int)
Hmmm, yes, although pointer/length pairs are not very common in C and this module is a language dependent module (and so should be called `MarshalCString').
-- MarshalError ------------------------------------------------------
!!!!! Haven't thought very deeply about this, but Marcin's QErrors !!!!! probably looks OK after a little polishing (use strerror, !!!!! provide a more complete throwXXX family, ...). And remember: !!!!! errno *is* a thread-local thing. :-)
But SimonM said that he doesn't want to make errno Haskell thread local...
-- MarshalConv -------------------------------------------------------
!!!!! I can't see a compelling reason for this module, neither what !!!!! exactly should belong here. ISO C defines a mapping to 0/1, !!!!! which can easily be captured by fromBool/toBool (in MarshalUtils !!!!! below). Maybe is handled there, too.
Ok - as this module doesn't seem to enjoy much support, let's just drop it. I can do my only little weird experiments in C2HS.
-- MarshalUtils ------------------------------------------------------
fromBool :: Num a => Bool -> a fromBool = fromIntegral . fromEnum
toBool :: Num a => a -> Bool toBool = (/= 0)
Ok, let's do it like this, then.
withObject :: Storable a => a -> (Ptr a -> IO b) -> IO b
BTW, can't we just use `with' now. IIRC, the only reason Marcin used `withObject' was because of a clash with the implicit parameter extension. This has been changed now, hasn't it?
!!!!! Example usages: !!!!! do ... !!!!! mbInt <- maybeNull peek ptr1 !!!!! mbString <- maybeNull peekCString ptr2 !!!!! ..... !!!!! maybeNothing withObject Nothing $ \ptr -> ... !!!!! maybeNothing withCString (Just "foo!") $ \ptr -> ... !!!!! !!!!! Does anybody have better names for these two functions? !!!!! They sound OK for me, but I'm not a native speaker...
maybeNull :: (Ptr a -> IO b) -> Ptr a -> IO (Maybe b) maybeNull f ptr | ptr == nullPtr = return Nothing | otherwise = liftM Just $ f ptr
maybeNothing :: (a -> (Ptr b -> IO c) -> IO c) -> (Maybe a -> (Ptr b -> IO c) -> IO c) maybeNothing = maybe ($ nullPtr)
This is also along the lines of what Marcin proposed. So, let's use it.
!!!!! Do we really need this? indexPtr :: Storable a => Ptr a -> Int -> Ptr a
I didn't need it yet. Marcin, what did you use the function for?
As Manuel already wrote it's a good idea to leave C2HS's marshX routines out and to re-export all the modules above, but should this be done via a new module Marshal or should Foreign do this? Should Marshal re-export Foreign or vice-versa? Hmmm...
I was thinking of an extra module mainly because the MarshalXXX code is system independent - in contrast to the code in `Foreign'. However, we may also just let Foreign do the whole job. Thinking about it again, I doubt that anybody will ever import one module, but not the other - so, let's make Foreign export the whole lot. There is one exception: MarshalCString shouldn't be re-exported by Foreign (like CTypes).
And just a final topic which really should be settled: "Marshal" or "Marshall"?? :-}
I would have taken the latter, but I think, SimonM originally proposed the name MarshalUtils with one "l" and as he is the native speaker... Cheers, Manuel
"Manuel M. T. Chakravarty" wrote:
If hslibs is meant to be used with systems other than ghc, too - which I think was the idea - there is no choice but to rewrite it into H98. But I guess this is essentially up to Mr. HsLibs aka SimonM.
Well, doing some simple local transformations to get H98 compliance should be possible without consulting Mr. HsLibs in advance. :-)
[ peekArray argument order ] Sounds reasonable to me.
OK, so let's change this before too many programs use the old order.
[ MarshalString ] Hmmm, yes, although pointer/length pairs are not very common in C and this module is a language dependent module (and so should be called `MarshalCString').
Granted. And remember we already have a (quite ugly) module CString! So here is a new proposal: * Deprecate the old definitions in CString and add type CString = Ptr CChar mallocCString :: String -> IO CString peekCString :: CString -> IO String pokeCString :: CString -> String -> IO () withCString :: String -> (CString -> IO a) -> IO a newCString :: String -> IO CString * Add a new module CStringLen with type CStringLen = (CString, Int) mallocCStringLen :: String -> IO CStringLen peekCStringLen :: CStringLen -> IO String pokeCStringLen :: CStringLen -> String -> IO () withCStringLen :: String -> (CStringLen -> IO a) -> IO a newCStringLen :: String -> IO CStringLen
[ MarshalError ] But SimonM said that he doesn't want to make errno Haskell thread local...
Wasn't this only due to efficiency reasons? IIRC Fergus had a good suggestion (copying errno only on thread switches or something), so I'd like to hear Mr. HsLibs' opinion on this topic again.
[ MarshalUtils ] BTW, can't we just use `with' now. IIRC, the only reason Marcin used `withObject' was because of a clash with the implicit parameter extension. This has been changed now, hasn't it?
Nope, not yet. Although there was some consensus about SPJ's ?-suggestion for those bindings, it's not implemented yet. Any takers? If this has been done, `with' would be better, indeed.
[...] This is also along the lines of what Marcin proposed. So, let's use it.
Well, it *is* what Marcin proposed. :-) I only gave maybe ($nullPtr) a name and a (less general) signature.
[ module (re-)exports ] I was thinking of an extra module mainly because the MarshalXXX code is system independent - in contrast to the code in `Foreign'. However, we may also just let Foreign do the whole job. Thinking about it again, I doubt that anybody will ever import one module, but not the other - so, let's make Foreign export the whole lot.
There is one exception: MarshalCString shouldn't be re-exported by Foreign (like CTypes).
I'd opt for this too: Foreign re-exports MarshalAlloc, MarshalArray, MarshalError and MarshalUtils and *not* CString/CStringLen.
And just a final topic which really should be settled: "Marshal" or "Marshall"?? :-}
I would have taken the latter, but I think, SimonM originally proposed the name MarshalUtils with one "l" and as he is the native speaker...
*sigh* Have to change HOpenGL once again... :-P If there are no further objections I'll incorporate the MarshalXXX and CSTring{,Len} stuff into hslibs this weekend. Cheers, Sven
Sven Panne <Sven.Panne@informatik.uni-muenchen.de> wrote,
[ MarshalString ] Hmmm, yes, although pointer/length pairs are not very common in C and this module is a language dependent module (and so should be called `MarshalCString').
Granted. And remember we already have a (quite ugly) module CString!
Is anybody actually depending on the old `CString'? Or is everybody using `PackedString' instead?
So here is a new proposal:
* Deprecate the old definitions in CString and add
The old definitions also include ByteArray stuff. If they are still needed, we may have to leave them in as a GHC extension.
type CString = Ptr CChar mallocCString :: String -> IO CString peekCString :: CString -> IO String pokeCString :: CString -> String -> IO () withCString :: String -> (CString -> IO a) -> IO a newCString :: String -> IO CString
* Add a new module CStringLen with
type CStringLen = (CString, Int) mallocCStringLen :: String -> IO CStringLen peekCStringLen :: CStringLen -> IO String pokeCStringLen :: CStringLen -> String -> IO () withCStringLen :: String -> (CStringLen -> IO a) -> IO a newCStringLen :: String -> IO CStringLen
I think, we need the alloca and realloc versions, too. (Eg, Marcin needs the latter for his unicode stuff, I think.) Otherwise, this sounds fine to me.
[ MarshalError ] But SimonM said that he doesn't want to make errno Haskell thread local...
Wasn't this only due to efficiency reasons? IIRC Fergus had a good suggestion (copying errno only on thread switches or something), so I'd like to hear Mr. HsLibs' opinion on this topic again.
I also think that this should be possible. Simon?
[ MarshalUtils ] BTW, can't we just use `with' now. IIRC, the only reason Marcin used `withObject' was because of a clash with the implicit parameter extension. This has been changed now, hasn't it?
Nope, not yet. Although there was some consensus about SPJ's ?-suggestion for those bindings, it's not implemented yet. Any takers? If this has been done, `with' would be better, indeed.
So, let's make it `with' officially and implement `withObject' flagging it as a GHC bug :-)
[ module (re-)exports ] I was thinking of an extra module mainly because the MarshalXXX code is system independent - in contrast to the code in `Foreign'. However, we may also just let Foreign do the whole job. Thinking about it again, I doubt that anybody will ever import one module, but not the other - so, let's make Foreign export the whole lot.
There is one exception: MarshalCString shouldn't be re-exported by Foreign (like CTypes).
I'd opt for this too: Foreign re-exports MarshalAlloc, MarshalArray, MarshalError and MarshalUtils and *not* CString/CStringLen.
Ok.
And just a final topic which really should be settled: "Marshal" or "Marshall"?? :-}
I would have taken the latter, but I think, SimonM originally proposed the name MarshalUtils with one "l" and as he is the native speaker...
*sigh* Have to change HOpenGL once again... :-P
Oh - we can still change it. Native speakers (Simon, Malcolm, ...), shall we make it MarshallXXX? Cheers, Manuel
Sun, 07 Jan 2001 12:16:15 +1100, Manuel M. T. Chakravarty <chak@cse.unsw.edu.au> pisze:
I think, we need the alloca and realloc versions, too.
We can't have malloc, alloca, realloc or poke. Only peek, with and new. We can't have poke because in general the space needed for the converted string is not known. Well, we could have poke with a parameter saying how long the buffer is, so the function never writes more. Is that functionality needed? Then OK for it. mallocCString and allocaCString would be exactly the same functions as mallocArray0 and allocaArray0, only with a more specific type. There is a little point in providing the same function under multiple names, unless it really makes things consistent, or unless they can diverge in future. But nobody has proposed freeArray, freeArray0 and freeCString (equivalent to free). Wait, in the Sven's message I see that mallocCString with String as the parameter. Then how it is different from newCString? For handling parts of C strings manually, reallocation of CString buffers and converting them to Haskell's lists in terms of C characters there are always *Array0 functions. *CString functions work on whole strings, and do some conversion to translate between CChar and Char. -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
Sat, 06 Jan 2001 21:58:26 +1100, Manuel M. T. Chakravarty <chak@cse.unsw.edu.au> pisze:
type CString = Ptr CChar type CStringLen = (CString, Int)
Hmmm, yes, although pointer/length pairs are not very common in C and this module is a language dependent module (and so should be called `MarshalCString').
I added the explicit-length variants when I needed them for the Python interface, after I realized that it's not possible to express them with variants which always stoped on input '\0' and didn't mark the output length besides writing a '\0'. The '\0' variants are now small wrappers around the explicit-length variants.
But SimonM said that he doesn't want to make errno Haskell thread local...
It would force wrapping every C function which may set errno in another C function which returns the error in a thred-safe way. The wrapper would have to check for errno and pass it in some way to Haskell together with the proper result - either through a pointer (which needs extra alloca per C function call) or special thread-local variables. It would be really bad to not have thread-local errno.
withObject :: Storable a => a -> (Ptr a -> IO b) -> IO b
BTW, can't we just use `with' now. IIRC, the only reason Marcin used `withObject' was because of a clash with the implicit parameter extension.
Indeed.
This has been changed now, hasn't it?
Everybody agreed that "let" should be used instead of "with", but the change has not been made yet.
There is one exception: MarshalCString shouldn't be re-exported by Foreign (like CTypes).
If language separation is realistic, we would also keep errno handling outside basic Foreign. But it yields several separate C-specific modules... module CForeign? It would reexport Foreign, CTypes, CTypesISO, MarshalCString and CErrors (or whatever they will be called). BTW, why are CTypes and CTypesISO separate? -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
Marcin 'Qrczak' Kowalczyk wrote:
[...] If language separation is realistic, we would also keep errno handling outside basic Foreign.
Hmmm, indeed. `CErrors' or `CErrno' would be a good name IMHO.
But it yields several separate C-specific modules...
module CForeign?
It would reexport Foreign, CTypes, CTypesISO, MarshalCString and CErrors (or whatever they will be called).
I'd second that (with the small addition that MarshalCString is replaced by CString/CStringLen).
BTW, why are CTypes and CTypesISO separate?
CTypes handles what I consider "builtin" types of the C language itself and CTypesISO handles library-related types defined by the ISO standard. And a more pragmatic reason: Compilation times and memory resources during compilation are already large enough in the current state of affairs... Cheers, Sven
Sven Panne <Sven.Panne@informatik.uni-muenchen.de> wrote,
Marcin 'Qrczak' Kowalczyk wrote:
[...] If language separation is realistic, we would also keep errno handling outside basic Foreign.
Hmmm, indeed. `CErrors' or `CErrno' would be a good name IMHO.
`CError' by our current naming scheme.[1]
But it yields several separate C-specific modules...
module CForeign?
It would reexport Foreign, CTypes, CTypesISO, MarshalCString and CErrors (or whatever they will be called).
I'd second that (with the small addition that MarshalCString is replaced by CString/CStringLen).
BTW, shall we really seperate CString and CStringLen into two different modules? I am all for modularisation, but in this case I am not sure whether it is worth it. Cheers, Manuel
Sun, 07 Jan 2001 16:16:00 +1100, Manuel M. T. Chakravarty <chak@cse.unsw.edu.au> pisze:
BTW, shall we really seperate CString and CStringLen into two different modules? I am all for modularisation, but in this case I am not sure whether it is worth it.
I would put them in one module, especially if mallocCString and pokeCString are dropped. -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
qrczak@knm.org.pl (Marcin 'Qrczak' Kowalczyk) wrote,
Sat, 06 Jan 2001 21:58:26 +1100, Manuel M. T. Chakravarty <chak@cse.unsw.edu.au> pisze:
type CString = Ptr CChar type CStringLen = (CString, Int)
Hmmm, yes, although pointer/length pairs are not very common in C and this module is a language dependent module (and so should be called `MarshalCString').
I added the explicit-length variants when I needed them for the Python interface, after I realized that it's not possible to express them with variants which always stoped on input '\0' and didn't mark the output length besides writing a '\0'. The '\0' variants are now small wrappers around the explicit-length variants.
Yes, this is how I did it in C2HS, too. It is the most natural solution.
But SimonM said that he doesn't want to make errno Haskell thread local...
It would force wrapping every C function which may set errno in another C function which returns the error in a thred-safe way. The wrapper would have to check for errno and pass it in some way to Haskell together with the proper result - either through a pointer (which needs extra alloca per C function call) or special thread-local variables. It would be really bad to not have thread-local errno.
I agree. Let's see what Simon's current opinion regarding this topic is.
This has been changed now, hasn't it?
Everybody agreed that "let" should be used instead of "with", but the change has not been made yet.
There is one exception: MarshalCString shouldn't be re-exported by Foreign (like CTypes).
If language separation is realistic, we would also keep errno handling outside basic Foreign. But it yields several separate C-specific modules...
module CForeign?
It would reexport Foreign, CTypes, CTypesISO, MarshalCString and CErrors (or whatever they will be called).
Good idea!
BTW, why are CTypes and CTypesISO separate?
IIRC, the idea was to have the types defined by the standard C libraries in an extra module. But it is not that important, I think. Cheers, Manuel
Sat, 06 Jan 2001 00:24:14 +0100, Sven Panne <Sven.Panne@informatik.uni-muenchen.de> pisze:
For performance there's always #ifdef (well, at least if we consider piping Haskell sources through cpp as "standard"/H98).
hsc2hs used instead of cpp provides #ifdef too, and avoids lexical analysis of Haskell source as C source.
!!!!! The current MarshalUtils uses !!!!! peekArray :: Storable a => Ptr a -> Int -> IO [a] !!!!! but the signature below looks more consistent with peekArray0: peekArray :: Storable a => Int -> Ptr a -> IO [a] peekArray0 :: (Storable a, Eq a) => a -> Ptr a -> IO [a]
OK.
!!!!! Charset conversions are a diff{erent,icult} story and should !!!!! be handled by a separate module, so only the well-known !!!!! ignore-the-upper-half variants are given here.
I think the conversion story should proceed thus: - For now the implementation will not do any conversion. - At some point of time all IO and C string handling will switch to do the conversion, keeping the same interface for the case of the default encoding, and providing additional means for using different encodings. It is important do introduce the conversion in all places at once, otherwise reading a filename containing non-ASCII characters from a file and using it to open another file will fail. Unfortunately it means that parts of the FFI functionality will have to go to ghc's lib/std, to properly handle strings and I/O in standard libraries. The more of convenience functions will be kept in hslibs, the less fun will be to fix standard libraries. The interface of string handling functions must allow that move in future without incompatible changes. We can't have separate mallocCString and pokeCString after the conversion switch, because the size depends on the contents, because a conversion can change the length. So we should not provide them now either. When someone really wants to use mallocCString and pokeCString now (knowing that there is a little point of doing that in the case of conversions), he can use mallocArray0 and pokeArray0, after casting characters of the string to [CChar].
!!!!! Haven't thought very deeply about this, but Marcin's QErrors !!!!! probably looks OK after a little polishing (use strerror, !!!!! provide a more complete throwXXX family, ...). And remember: !!!!! errno *is* a thread-local thing. :-)
A little warning: strerror will want to translate strings, and even do the conversion (the result of strerror is in the default byte encoding). I had a module dependency loop when I wanted to use strerror and at the same time handle errno in iconv's wrapper.
-- MarshalUtils ------------------------------------------------------
fromBool :: Num a => Bool -> a fromBool = fromIntegral . fromEnum
I think that fromBool :: Num a => Bool -> a fromBool False = 0 fromBool True = 1 is more clear (and easier to compile efficiently) :-)
!!!!! mbInt <- maybeNull peek ptr1 !!!!! mbString <- maybeNull peekCString ptr2 !!!!! ..... !!!!! maybeNothing withObject Nothing $ \ptr -> ... !!!!! maybeNothing withCString (Just "foo!") $ \ptr -> ... !!!!! !!!!! Does anybody have better names for these two functions? !!!!! They sound OK for me, but I'm not a native speaker...
maybeNothing looks very cryptic for me, but I don't have a better proposal...
!!!!! Do we really need this? indexPtr :: Storable a => Ptr a -> Int -> Ptr a
I used it once in Libgr, 3 times in QString and 5 times in conversions. I would definitely keep it. Perhaps the name movePtr would be better. Let me repeat a proposal of two functions for MarshalUtils (better names are welcome): sequenceCont :: [(a -> res) -> res] -> ([a] -> res) -> res sequenceCont [] cont = cont [] sequenceCont (f:fs) cont = f (\x -> sequenceCont fs (\xs -> cont (x:xs))) mapCont :: (a -> (b -> res) -> res) -> [a] -> ([b] -> res) -> res mapCont f = sequenceCont . map f Example: mapCont withCString a_list_of_Haskell_strings $ \a_list_of_C_string_pointers -> Without them one has to write an explicit recursive function each time for passing an array of things-to-be-allocated-on-the-stack to a C function. I would like to write instance Monad (/\a. (a -> res) -> res) instead of them. Unfortunately it's not possible. -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
Marcin 'Qrczak' Kowalczyk wrote:
[...] I think the conversion story should proceed thus: - For now the implementation will not do any conversion.
... and simply throws away the upper half / fills it with zero.
[...] We can't have separate mallocCString and pokeCString after the conversion switch, because the size depends on the contents, because a conversion can change the length. So we should not provide them now either.
I don't get your point here completely, but I believe Mr. CharConv here... :-)
[...] Let me repeat a proposal of two functions for MarshalUtils (better names are welcome): [...]
I have something similar in HOpenGL for argc/argv handling, so this is probably a good idea (modulo some renaming?). Cheers, Sven
qrczak@knm.org.pl (Marcin 'Qrczak' Kowalczyk) wrote,
Sat, 06 Jan 2001 00:24:14 +0100, Sven Panne <Sven.Panne@informatik.uni-muenchen.de> pisze:
For performance there's always #ifdef (well, at least if we consider piping Haskell sources through cpp as "standard"/H98).
hsc2hs used instead of cpp provides #ifdef too, and avoids lexical analysis of Haskell source as C source.
!!!!! The current MarshalUtils uses !!!!! peekArray :: Storable a => Ptr a -> Int -> IO [a] !!!!! but the signature below looks more consistent with peekArray0: peekArray :: Storable a => Int -> Ptr a -> IO [a] peekArray0 :: (Storable a, Eq a) => a -> Ptr a -> IO [a]
OK.
!!!!! Charset conversions are a diff{erent,icult} story and should !!!!! be handled by a separate module, so only the well-known !!!!! ignore-the-upper-half variants are given here.
I think the conversion story should proceed thus:
- For now the implementation will not do any conversion.
- At some point of time all IO and C string handling will switch to do the conversion, keeping the same interface for the case of the default encoding, and providing additional means for using different encodings.
It is important do introduce the conversion in all places at once, otherwise reading a filename containing non-ASCII characters from a file and using it to open another file will fail.
Unfortunately it means that parts of the FFI functionality will have to go to ghc's lib/std, to properly handle strings and I/O in standard libraries. The more of convenience functions will be kept in hslibs, the less fun will be to fix standard libraries.
This is an aspect of GHC's implementation and shouldn't concern us for the design of the system-independent FFI. Moreover, the compiler already depends on -package lang anyway.
The interface of string handling functions must allow that move in future without incompatible changes. We can't have separate mallocCString and pokeCString after the conversion switch, because the size depends on the contents, because a conversion can change the length. So we should not provide them now either.
When someone really wants to use mallocCString and pokeCString now (knowing that there is a little point of doing that in the case of conversions), he can use mallocArray0 and pokeArray0, after casting characters of the string to [CChar].
To be honest, I don't like this. It is nice having the interface such that we can switch to using conversions at some point, but I still want to be able to conveniently deal with 8bit characters (because this is what many C libraries use). So, I want a fast and convenient interface to 8bit strings *in addition* to the interface that can deal with conversions. In particular this means that I don't want to deal with CChar in the Haskell interface only to circumvent conversion.
-- MarshalUtils ------------------------------------------------------
fromBool :: Num a => Bool -> a fromBool = fromIntegral . fromEnum
I think that fromBool :: Num a => Bool -> a fromBool False = 0 fromBool True = 1 is more clear (and easier to compile efficiently) :-)
:-)
!!!!! Do we really need this? indexPtr :: Storable a => Ptr a -> Int -> Ptr a
I used it once in Libgr, 3 times in QString and 5 times in conversions. I would definitely keep it. Perhaps the name movePtr would be better.
How about `advancePtr'? But I am wondering whether this shouldn't go into MarshalArray? It is used for array access, isn't it?
Let me repeat a proposal of two functions for MarshalUtils (better names are welcome):
sequenceCont :: [(a -> res) -> res] -> ([a] -> res) -> res sequenceCont [] cont = cont [] sequenceCont (f:fs) cont = f (\x -> sequenceCont fs (\xs -> cont (x:xs)))
mapCont :: (a -> (b -> res) -> res) -> [a] -> ([b] -> res) -> res mapCont f = sequenceCont . map f
Example: mapCont withCString a_list_of_Haskell_strings $ \a_list_of_C_string_pointers ->
Without them one has to write an explicit recursive function each time for passing an array of things-to-be-allocated-on-the-stack to a C function.
`mapCont' makes a lot of sense. I am less sure about `sequenceCont'. I imagine that you can use it to write something like sequenceCont [withCString aString, withObject anObject] $ \[s, o] -> foreignFoo s o instead of withCString aString $ \s -> withObject anObject $ \o -> foreignFoo s o but is that really useful? How about calling `mapCont` simply `withMany'. You would have withMany withCString a_list_of_Haskell_strings $ \a_list_of_C_string_pointers -> Or `withList'? Cheers, Manuel
Sun, 07 Jan 2001 13:15:21 +1100, Manuel M. T. Chakravarty <chak@cse.unsw.edu.au> pisze:
When someone really wants to use mallocCString and pokeCString now (knowing that there is a little point of doing that in the case of conversions), he can use mallocArray0 and pokeArray0, after casting characters of the string to [CChar].
To be honest, I don't like this. It is nice having the interface such that we can switch to using conversions at some point, but I still want to be able to conveniently deal with 8bit characters (because this is what many C libraries use). So, I want a fast and convenient interface to 8bit strings *in addition* to the interface that can deal with conversions. In particular this means that I don't want to deal with CChar in the Haskell interface only to circumvent conversion.
I understand everything except the last sentence. Why it is bad to deal with CChar in Haskell? It could be confusing if some String values represented texts in Unicode and others - in the C's encoding. (Especially if the programmer uses ISO-8859-1 for C encoding and does not care about the difference, and then somebody using ISO-8859-7 tries to run his code!) IMHO most strings on which C functions work (those ending with '\0') are either in the default local encoding (if they are texts in a natural language or filenames) or more rarely ASCII (if they are e.g. names of mail headers, identifiers in a C program, or commandline switches of some program). Sometimes the encoding is specified explicitly by the protocol or is stored in data itself. For ASCII the default local encoding can be used too, with a speed penalty; practically used encodings are ASCII-compatible. You can explicitly specify fromLatin1 or toLatin1 if you really want C characters to map to Haskell's '\0'..'\255' - it should be faster (does not call iconv or the like). You can also use CChar. So most of the time strings should be converted to native Haskell's encoding of Unicode, to be compatible with other parts of libraries which expect the text to be in Unicode, and to let "words" and "toUpper" and "length" work correctly (if by "length" you understand the number of characters, not bytes in the physical encoding). It's hard to do conversions in one place and not do it in others if you mix data between those places. What cases do you have in mind when strings should be passed to C libraries unconverted?
How about `advancePtr'? But I am wondering whether this shouldn't go into MarshalArray? It is used for array access, isn't it?
It's indeed only for arrays (where plusPtr is for structs, where the offset is in bytes).
`mapCont' makes a lot of sense. I am less sure about `sequenceCont'.
mapCont is indeed more common (I used it only on withCString yet). Uses of sequenceCont: sequenceCont (replicate n $ allocaArray m) $ \rows -> ... sequenceCont [ if s == "" then ($ nullPtr) else withCString s | s <- listOfStrings] $ \stringPtrs -> ...
How about calling `mapCont` simply `withMany'.
I like it. sequenceCont could be allocaMany. -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
qrczak@knm.org.pl (Marcin 'Qrczak' Kowalczyk) wrote,
Sun, 07 Jan 2001 13:15:21 +1100, Manuel M. T. Chakravarty <chak@cse.unsw.edu.au> pisze:
When someone really wants to use mallocCString and pokeCString now (knowing that there is a little point of doing that in the case of conversions), he can use mallocArray0 and pokeArray0, after casting characters of the string to [CChar].
To be honest, I don't like this. It is nice having the interface such that we can switch to using conversions at some point, but I still want to be able to conveniently deal with 8bit characters (because this is what many C libraries use). So, I want a fast and convenient interface to 8bit strings *in addition* to the interface that can deal with conversions. In particular this means that I don't want to deal with CChar in the Haskell interface only to circumvent conversion.
I understand everything except the last sentence. Why it is bad to deal with CChar in Haskell?
It could be confusing if some String values represented texts in Unicode and others - in the C's encoding. (Especially if the programmer uses ISO-8859-1 for C encoding and does not care about the difference, and then somebody using ISO-8859-7 tries to run his code!)
IMHO most strings on which C functions work (those ending with '\0') are either in the default local encoding (if they are texts in a natural language or filenames) or more rarely ASCII (if they are e.g. names of mail headers, identifiers in a C program, or commandline switches of some program). Sometimes the encoding is specified explicitly by the protocol or is stored in data itself.
For ASCII the default local encoding can be used too, with a speed penalty; practically used encodings are ASCII-compatible. You can explicitly specify fromLatin1 or toLatin1 if you really want C characters to map to Haskell's '\0'..'\255' - it should be faster (does not call iconv or the like). You can also use CChar.
The speed penalty is exactly what I am worried about. What you are proposing - if I understand you correctly - is to use Unicode whatever encodings on the Haskell side exclusively and each Haskell<->C conversion of a String has to go through a conversion. Then, as you say, poke and some other routines make no sense on Strings because of the varying string length in different encodings. Ok, I got that. Now what I am thinking is that this will be even slower than the whole business is already. So, an all Unicode Haskell will be even slower than it is now. Strings are used for two purposes in programs: (1) To represent natural language and (2) to represent unstructured program data. For the first case, we have to take the performance penalty if we want the benefit of handling non-ASCII languages. For the second case, however, I think we don't need it. Take for example (and it is not a very good example) the Tk binding for Haskell. It accesses the Tk widget set by constructing Tcl commands at runtime and sending them to the Tcl interpreter via a pipe. That's already pretty inefficient. Now when each of these commands has to go through a Unicode conversion, things will get even worse. Another example is configuration management in libraries like the Gnome library. A program can dump its session data into an ASCII file using these libraries, so that it doesn't have to mantain its own preferences and resource files. Do we really want all this stuff to go through the converter? Furthermore, to be honest, I am not really sure why we have to do the conversion anyway. When I am having a Haskell program like [1] main = putStrLn "今日は" then, there are two possibilities. Either I have a system configured with the locale jp_JP and I happen to run this Haskell program in kterm or an Mule/(X)Emacs subshell, or I will get mojibake[2] anyway. No amount of conversion is going to change that. So, what exactly do I get for the performance penalty that conversion incurs? How about having an interface where the String marshalling functions take an additional argument data CConv = NoCConv -- handle as 8bit chars | StdCConc -- standard conversion | CustomCConv String -- special conversions Then, it is up to the programmer to decide whether to use conversion. The idea of the last variant would be that in your conversion library, I can give conversions a name and identify them by that name. This way the CString wouldn't depend on the exact conversion interface, but still would be open to the addition of new conversions. Routines like mallocCString and pokeCString would only make sense for `NoCConv', then. Cheers, Manuel [1] I hope your mail reader can handle iso-2022-jp :-) [2] Mojibake is the Japanese term for Japanese text displayed through software that cannot handle it. Mojibake is written as "文字化け" in Japanese and if your mail reader can't handle Japanese, you'll see just that ;-)
Mon, 08 Jan 2001 14:55:31 +1100, Manuel M. T. Chakravarty <chak@cse.unsw.edu.au> pisze:
How about having an interface where the String marshalling functions take an additional argument
data CConv = NoCConv -- handle as 8bit chars | StdCConc -- standard conversion | CustomCConv String -- special conversions
There are already string marshalling variants which take the conversion as the argument. The first is equivalent to toLatin1 / fromLatin1, the second is localOut / localIn or using functions with no explicit conversion (names are of source subject to discussion), the third would either require building a database of name -> conversion mappings or just pass names to iconv, assuming iconv knows that conversion. They make no sense when there is never any conversion, so I haven't talked about them yet when discussing MarshalCString or equivalents. If you know that data is always ASCII, you can use toLatin1 as the conversion.
Then, it is up to the programmer to decide whether to use conversion.
There is always a conversion. Char and CChar are not the same type. But it can be as trivial as toLatin1.
The idea of the last variant would be that in your conversion library, I can give conversions a name and identify them by that name. This way the CString wouldn't depend on the exact conversion interface,
Textual names are not enough. Conversions can be constructed on the fly, e.g. by improving an existing conversion by substituting some strings for characters which can't be handled by it. Currently the type of the additional withCStringConv's argument is IO (Conv Char Byte). It's IO because this is how "not started yet conversions" are expressed. Conv Char Byte itself is an anstract type of a stateful conversion which is taking place.
Routines like mallocCString and pokeCString would only make sense for `NoCConv', then.
We already have mallocArray0 and pokeArray0. You only have to cast characters to [CChar].
Another example is configuration management in libraries like the Gnome library. A program can dump its session data into an ASCII file using these libraries, so that it doesn't have to mantain its own preferences and resource files. Do we really want all this stuff to go through the converter?
In what encoding are natural language texts in this dump? If it's ASCII, use fromLatin1. I have yet to benchmark conversions.
Furthermore, to be honest, I am not really sure why we have to do the conversion anyway. When I am having a Haskell program like [1]
main = putStrLn "$B:#F|$O(B"
then, there are two possibilities. Either I have a system configured with the locale jp_JP and I happen to run this Haskell program in kterm or an Mule/(X)Emacs subshell, or I will get mojibake[2] anyway.
You can have another terminal capable of displaying Japanese, e.g. UTF-8-patched xterm (I don't know how well it works in practice for Japanese, but they are using it for harder scripts like Arabic so I guess it's just OK). The the same compiled program will then display correctly (as long as the locale is set appropriately for the terminal). You can have Japanese texts in the source in any encoding, as long as the Haskell compiler is able to understand various source encodings, and the right encoding was specified in some way. UTF-8-patched xterm has an advantage over Mojibake that it displays not only Japanese, but about any charset which is possible to display in a fixed-with font (including double-width characters).
[2] Mojibake is the Japanese term for Japanese text displayed through software that cannot handle it. Mojibake is written as "^[$BJ8;z2=$1^[(B" in Japanese and if your mail reader can't handle Japanese, you'll see just that ;-)
Unfortunately I don't see that. But if I used a (nonexistant yet) newsreader and editor written in Haskell which made nontrivial use of the conversion machinery, I would probably see that :-) -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
qrczak@knm.org.pl (Marcin 'Qrczak' Kowalczyk) wrote,
Mon, 08 Jan 2001 14:55:31 +1100, Manuel M. T. Chakravarty <chak@cse.unsw.edu.au> pisze:
How about having an interface where the String marshalling functions take an additional argument
data CConv = NoCConv -- handle as 8bit chars | StdCConc -- standard conversion | CustomCConv String -- special conversions
There are already string marshalling variants which take the conversion as the argument. The first is equivalent to toLatin1 / fromLatin1, the second is localOut / localIn or using functions with no explicit conversion (names are of source subject to discussion), the third would either require building a database of name -> conversion mappings or just pass names to iconv, assuming iconv knows that conversion.
Where are these variants? In QForeign? The interface for CString that was under discussion here didn't say anything about conversions?
They make no sense when there is never any conversion, so I haven't talked about them yet when discussing MarshalCString or equivalents.
Fair enough. So, basically what I am saying, then, is that as long as it is not clear how the conversion interface looks in detail, let's talk about two different conversions that we definitely know we are going to need: the toLatin1/fromLatin1 conversion and the localOut/localIn conversion. I want the standard FFI CString interface to support these two. The rest we can add later, but I don't want to be restricted to the std conversion only. BTW, how efficient is the code for toLatin1/fromLatin1?
If you know that data is always ASCII, you can use toLatin1 as the conversion.
With the interface that we discussed so far, I can't.
Then, it is up to the programmer to decide whether to use conversion.
There is always a conversion. Char and CChar are not the same type. But it can be as trivial as toLatin1.
Ok, with "whether there is a conversion" I meant "whether there is more than castCharToCChar lifted to Strings".
The idea of the last variant would be that in your conversion library, I can give conversions a name and identify them by that name. This way the CString wouldn't depend on the exact conversion interface,
Textual names are not enough. Conversions can be constructed on the fly, e.g. by improving an existing conversion by substituting some strings for characters which can't be handled by it.
Currently the type of the additional withCStringConv's argument is IO (Conv Char Byte). It's IO because this is how "not started yet conversions" are expressed. Conv Char Byte itself is an anstract type of a stateful conversion which is taking place.
I didn't want to imply that this admittedly very simple data type CConv can actually represent all possible conversions. However, we might have a very simple conversion interface in CString, and then, the real fully fledged conversion interface in the libraries that you are designing. However, as I said, I want at least be able to distinguish to/fromLatin1 and localIn/localOut [Why not call it toLocal/fromLocal?]. If we have the distinction anyway, making it a little more flexible and adding CustomCConv seems sensible. We could have conversions like CustomCConv "Latin2" and CustomCConv "ja_JP.euc" Maybe your conversion library could, then, have a function like registerConv :: String -> IO (Conv Char Byte) -> IO () which allows me to give symbolic names to conversions. We could then predefine names for a set of commonly used conversions. This way many users might be able to use these conversions more easily.
Routines like mallocCString and pokeCString would only make sense for `NoCConv', then.
We already have mallocArray0 and pokeArray0. You only have to cast characters to [CChar].
Sure - but why not have this as predefined functions in CString? That's all I am proposing.
Another example is configuration management in libraries like the Gnome library. A program can dump its session data into an ASCII file using these libraries, so that it doesn't have to mantain its own preferences and resource files. Do we really want all this stuff to go through the converter?
In what encoding are natural language texts in this dump? If it's ASCII, use fromLatin1.
I have yet to benchmark conversions.
That's exactly where my concern is. We are designing an interface which requires all string marshalling to go through a procedure from which we have no idea how fast it is...for which, however, it seems clear - from what you said so far and briefly looking at the QForeign code - that it won't be really cheap. I don't like that. I want a backdoor to a really cheap and dirty to/fromLatin1 conversion in the standard FFI interface.
[2] Mojibake is the Japanese term for Japanese text displayed through software that cannot handle it. Mojibake is written as "^[$BJ8;z2=$1^[(B" in Japanese and if your mail reader can't handle Japanese, you'll see just that ;-)
Unfortunately I don't see that. But if I used a (nonexistant yet) newsreader and editor written in Haskell which made nontrivial use of the conversion machinery, I would probably see that :-)
Just use XEmacs with Mule support as your mail reader and you can enjoy that already today :-) It can even render different scripts in one buffer (like the UTF-8-patched xterm), eg, I can write Japanese/German vocabulary cheat sheets having Japanese characters and German umlauts. BTW, do you know Pango <http://www.pango.org/>? Cheers, Manuel
On Tue, 9 Jan 2001, Manuel M. T. Chakravarty wrote:
Where are these variants? In QForeign?
Yes. QForeign is being continuously updated in CVS: cvs -d:pserver:anonymous@cvs.qforeign.sourceforge.net:/cvsroot/qforeign \ login (empty password) cvs -d:pserver:anonymous@cvs.qforeign.sourceforge.net:/cvsroot/qforeign \ -z3 get qforeign
The interface for CString that was under discussion here didn't say anything about conversions?
Yes.
Fair enough. So, basically what I am saying, then, is that as long as it is not clear how the conversion interface looks in detail, let's talk about two different conversions that we definitely know we are going to need: the toLatin1/fromLatin1 conversion and the localOut/localIn conversion. I want the standard FFI CString interface to support these two. The rest we can add later, but I don't want to be restricted to the std conversion only.
OK, I agree that it should be introduced early. I didn't propose them because even the type of conversions was not completely clear. As long as there are no conversions all over the libraries and I/O and filenames default to ISO-8859-1, toLocal and fromLocal would probably be equivalent to toLatin1 and fromLatin1. When everything is ready to accept conversions, they will switch to system-dependent default local byte encoding. On Unices: based on locale.
BTW, how efficient is the code for toLatin1/fromLatin1?
fromLatin1's inner loop is: map (chr . fromIntegral) :: [Byte] -> String toLatin1's inner loop is worse. It must check for valid characters. My interface allows users of a conversion to see if there was an error, and also to get the converted string even if there was errors (with problematic places marged appropriately for the target encoding): conv :: String -> ([Word8], Bool) conv [] = ([], True) conv (ch:s) | ch <= '\xFF' = (fromIntegral (ord ch) : rest, good) | otherwise = (fromIntegral (ord '?') : rest, False) where (rest, good) = conv s If it's measured to be too slow, there could be unsafeToLatin1 which works correctly as long as all characters are valid...
[Why not call it toLocal/fromLocal?].
OK, these names are better.
If we have the distinction anyway, making it a little more flexible and adding CustomCConv seems sensible.
Conv is intentionally abstract. There is iconv :: (Storable from, Storable to) => String -> String -> [to] -> IO (Conv from to) on systems which have iconv (it's specified in Single Unix Spec, implemented in Linux' glibc and I guess it is not available natively on Windows; there are portable implementations for Unices and I don't know if they work on Windows). Parameters are: name of input encoding, name of output encoding, string to use to mark errors. A problem with using iconv directly is that there is no portable way to know either the name of a supported Unicode flavour ("UTF-8" is a good guess but it's inefficient, and not all iconv implementations provide UCS-4 in native endianness - glibc-2.1.3 does not, glibc-2.2 does) or the name of the default local encoding (there is nl_langinfo (CODESET) but AFAIK it's not available on BSD). In practice it often must be composed with the appropriate conversion which matches some iconv's Unicode flavour with an array of 32-bit Chars. Nevertheless it does provide its own database of charsets identified by names and is already implemented. Currently I have only iconv to implement the default local encoding, with appropriate autoconf magic to find a common language with it. I also had wcsrtombs and mbsrtowcs, but people said it's less portable because wchar_t needs not to be Unicode. Windows needs its own implementation, which I could take e.g. from Python's sources, but I don't have a place to test it.
Maybe your conversion library could, then, have a function like
registerConv :: String -> IO (Conv Char Byte) -> IO ()
which allows me to give symbolic names to conversions.
A central database can be built around conversions available as values in the program. But it would be a bad idea to take it as the basic identification and require registering to use a conversion. Since there can be various schemes of encoding names (MIME, iconv's names) and not all encodings must have names in all schemes (I doubt MIME talks about "UCS-4 in native endianness", which corresponds to the array of Chars), I would not encourage to work in terms of names. I've implemented conversions for simple ISO-8859-x, CP-xxxx, CP-xxx, KOI8-R and Mac encodings (excluding those which need special bidi or multibyte processing). They are implemented in Haskell. iconv is implemented in C. Both kinds of implementations are handled during String <-> CString conversions in a way which I believe is as efficient as possible (roughly). OTOH QForeign's experimental converting IO replacement, IOConv, is not efficient at all for conversions implemented in C.
We already have mallocArray0 and pokeArray0. You only have to cast characters to [CChar].
Sure - but why not have this as predefined functions in CString? That's all I am proposing.
I would not encourage people to skip the conversion and produce code which works only for ISO-8859-1. Latin1 is just one of many encodings. Since string handling in Haskell is already inefficient, I hope that adding conversions would not make a big relative difference. It would be a different story if strings could be passed to C functions without marshalling.
BTW, do you know Pango <http://www.pango.org/>?
Not yet. -- Marcin 'Qrczak' Kowalczyk
Mon, 8 Jan 2001 17:11:04 +0100 (CET), Marcin 'Qrczak' Kowalczyk <mk167280@zodiac.mimuw.edu.pl> pisze:
A central database can be built around conversions available as values in the program. But it would be a bad idea to take it as the basic identification and require registering to use a conversion.
An important reason is that one can never be sure that a particular encoding has been registered. Unless it takes care to import its module ald call the registration function, but in this case he could as well take the conversion itself without the extra indirection. I imagine that common charsets would be imported from modules, like import ConvIso ... hSetOutConv stdout Text toIso8859_2 and there could be specialized databases of e.g. MIME encodings. -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
qrczak@knm.org.pl (Marcin 'Qrczak' Kowalczyk) wrote,
Mon, 8 Jan 2001 17:11:04 +0100 (CET), Marcin 'Qrczak' Kowalczyk <mk167280@zodiac.mimuw.edu.pl> pisze:
A central database can be built around conversions available as values in the program. But it would be a bad idea to take it as the basic identification and require registering to use a conversion.
An important reason is that one can never be sure that a particular encoding has been registered.
I was thinking of having the library itself by default provide a set of standard encodings. Like - as you say later - we usually rely on a set of standard MIME encodings to be available. Given this, most normal users would never have to touch the more complicated Conv type, but just use the string handles to reference these standard encodings. Just like in MIME.
Unless it takes care to import its module ald call the registration function, but in this case he could as well take the conversion itself without the extra indirection.
Not necessarily. Imagine a module StdEncodings, with a function `initStdEncodings'. Then, my program could look like import StdEncodings ... main = do initStdEncondings ... <use encoding "iso-2022-jp"> ... This is simpler than using the fully fledged conversion interface. Moreover, we could require these standard encodings already to be pre-registered. Cheers, Manuel
Tue, 09 Jan 2001 13:59:10 +1100, Manuel M. T. Chakravarty <chak@cse.unsw.edu.au> pisze:
I was thinking of having the library itself by default provide a set of standard encodings. Like - as you say later - we usually rely on a set of standard MIME encodings to be available.
If an encoding is added to the database at some point of time, a program uses it and is then recompiled on a system which does not provide this encoding, you get a runtime error. If conversions are referred to as plain imported values, you get a compile error. A database is only useful if encodings are now known individually by the programmer and he wants his program to support everything the Haskell's library provides for some well known name scheme. It can be a convenience wrapper, but definitely not the basic reference for conversions. -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
qrczak@knm.org.pl (Marcin 'Qrczak' Kowalczyk) wrote,
Tue, 09 Jan 2001 13:59:10 +1100, Manuel M. T. Chakravarty <chak@cse.unsw.edu.au> pisze:
I was thinking of having the library itself by default provide a set of standard encodings. Like - as you say later - we usually rely on a set of standard MIME encodings to be available.
If an encoding is added to the database at some point of time, a program uses it and is then recompiled on a system which does not provide this encoding, you get a runtime error.
If conversions are referred to as plain imported values, you get a compile error.
But what would be the reason for an implementation not providing this conversion? That it can't do the conversion? But in this case, you have a problem anyway. Independent of the mechanism used to reference the conversion, you won't get it. This not different from trying to print Cyrillic text on a computer that doesn't have a Cyrillic font - you just won't see much :-) Other than that, your library could explicitly list a set of standard conversions that any complying implementation has to provide (at least if it is available on the system).
A database is only useful if encodings are now known individually by the programmer and he wants his program to support everything the Haskell's library provides for some well known name scheme. It can be a convenience wrapper, but definitely not the basic reference for conversions.
I did mean it as a convenience wrapper. There are certainly users who want the full interface. I am just thinking about the average user writing an average program who happens to write all error message in Mandarin. This person wants an easy way to access Big5 somehow, not more. Cheers, Manuel
"Marcin 'Qrczak' Kowalczyk" <mk167280@zodiac.mimuw.edu.pl> wrote,
On Tue, 9 Jan 2001, Manuel M. T. Chakravarty wrote:
We already have mallocArray0 and pokeArray0. You only have to cast characters to [CChar].
Sure - but why not have this as predefined functions in CString? That's all I am proposing.
I would not encourage people to skip the conversion and produce code which works only for ISO-8859-1. Latin1 is just one of many encodings.
Since string handling in Haskell is already inefficient, I hope that adding conversions would not make a big relative difference. It would be a different story if strings could be passed to C functions without marshalling.
Ok, I have thought about it again. malloc and poke aren't much used on strings anyway. So, the interface type CString = Ptr CChar type CStringLen = (CString, Int) peekCString :: CString -> IO String peekCStringLen :: CStringLen -> IO String withCString :: String -> (CString -> IO a) -> IO a withCStringLen :: String -> (CStringLen -> IO a) -> IO a newCString :: String -> IO CString newCStringLen :: String -> IO CStringLen which, I think, is what you want, should suffice for the moment. There is, however, one constraint that I would like to impose on the design of the conversion library. It must be possible to spot and optimise the application of cheap conversions like to/fromLatin1. In other words, the design of the system must be such that if I use, eg, toLatin1, it can use alloca instead of malloc, because it is clear that the length of the String won't change in a Latin1 conversion. In GHC, for example, I might have a rule as follows: {-# RULES "newCString/Latin1" forall s. newCStringConv toLatin1 s = newCStringLatin1 s #-} where the function `newCStringLatin1' makes use of `alloca' to speed up memory allocation. If I understood you correctly, we would have toLatin1 :: Conv Int16 CChar or so. But it would require fromLatin1 to be pre-defined as a symbol somewhere. Maybe a yucky toLatin1 :: Conv Int16 CChar toLatin1 = unsafePerformIO $ iconv "GHC dft encoding" "Latin1" In any case, this requires that the types and rest of the design is such that I can use a rewrite rule to spot such cases. Is that ok with you? Cheers, Manuel
On Wed, 10 Jan 2001, Manuel M. T. Chakravarty wrote:
which, I think, is what you want, should suffice for the moment.
I am happy with that.
There is, however, one constraint that I would like to impose on the design of the conversion library. It must be possible to spot and optimise the application of cheap conversions like to/fromLatin1.
It is possible now, even witohut RULES (which would not catch cases which are not recognizable statically). Conv is an abstract type. Currently it has alternatives for a conversion implemented in Haskell (working on lists), a conversion implemented in C (working on pointers to C arrays and with a different chunking strategy), and a conversion implemented in both styles, to be chosen by code using it (perhaps it's unnecessary, I have to measure). For example direct application of a conversion to a user-supplied String uses the Haskell variant if available, otherwise does appropriate array marshalling for the C variant. OTOH newCStringConv uses the C variant directly on the returned buffer if available, otherwise runs the Haskell variant and uses newArray0. It would be no problem to add variants for Latin1 and let withCStringConv not call newCStringConv in this case but treat it specially. (Currently alloca is implemented in terms of malloc so it does not matter anyway.) -- Marcin 'Qrczak' Kowalczyk
participants (5)
-
malcolm-ffi＠cs.york.ac.uk -
Manuel M. T. Chakravarty -
Marcin 'Qrczak' Kowalczyk -
qrczak＠knm.org.pl -
Sven Panne