On Tue, 9 Jan 2001, Manuel M. T. Chakravarty wrote:
Where are these variants? In QForeign?
Yes. QForeign is being continuously updated in CVS: cvs -d:pserver:anonymous@cvs.qforeign.sourceforge.net:/cvsroot/qforeign \ login (empty password) cvs -d:pserver:anonymous@cvs.qforeign.sourceforge.net:/cvsroot/qforeign \ -z3 get qforeign
The interface for CString that was under discussion here didn't say anything about conversions?
Yes.
Fair enough. So, basically what I am saying, then, is that as long as it is not clear how the conversion interface looks in detail, let's talk about two different conversions that we definitely know we are going to need: the toLatin1/fromLatin1 conversion and the localOut/localIn conversion. I want the standard FFI CString interface to support these two. The rest we can add later, but I don't want to be restricted to the std conversion only.
OK, I agree that it should be introduced early. I didn't propose them because even the type of conversions was not completely clear. As long as there are no conversions all over the libraries and I/O and filenames default to ISO-8859-1, toLocal and fromLocal would probably be equivalent to toLatin1 and fromLatin1. When everything is ready to accept conversions, they will switch to system-dependent default local byte encoding. On Unices: based on locale.
BTW, how efficient is the code for toLatin1/fromLatin1?
fromLatin1's inner loop is: map (chr . fromIntegral) :: [Byte] -> String toLatin1's inner loop is worse. It must check for valid characters. My interface allows users of a conversion to see if there was an error, and also to get the converted string even if there was errors (with problematic places marged appropriately for the target encoding): conv :: String -> ([Word8], Bool) conv [] = ([], True) conv (ch:s) | ch <= '\xFF' = (fromIntegral (ord ch) : rest, good) | otherwise = (fromIntegral (ord '?') : rest, False) where (rest, good) = conv s If it's measured to be too slow, there could be unsafeToLatin1 which works correctly as long as all characters are valid...
[Why not call it toLocal/fromLocal?].
OK, these names are better.
If we have the distinction anyway, making it a little more flexible and adding CustomCConv seems sensible.
Conv is intentionally abstract. There is iconv :: (Storable from, Storable to) => String -> String -> [to] -> IO (Conv from to) on systems which have iconv (it's specified in Single Unix Spec, implemented in Linux' glibc and I guess it is not available natively on Windows; there are portable implementations for Unices and I don't know if they work on Windows). Parameters are: name of input encoding, name of output encoding, string to use to mark errors. A problem with using iconv directly is that there is no portable way to know either the name of a supported Unicode flavour ("UTF-8" is a good guess but it's inefficient, and not all iconv implementations provide UCS-4 in native endianness - glibc-2.1.3 does not, glibc-2.2 does) or the name of the default local encoding (there is nl_langinfo (CODESET) but AFAIK it's not available on BSD). In practice it often must be composed with the appropriate conversion which matches some iconv's Unicode flavour with an array of 32-bit Chars. Nevertheless it does provide its own database of charsets identified by names and is already implemented. Currently I have only iconv to implement the default local encoding, with appropriate autoconf magic to find a common language with it. I also had wcsrtombs and mbsrtowcs, but people said it's less portable because wchar_t needs not to be Unicode. Windows needs its own implementation, which I could take e.g. from Python's sources, but I don't have a place to test it.
Maybe your conversion library could, then, have a function like
registerConv :: String -> IO (Conv Char Byte) -> IO ()
which allows me to give symbolic names to conversions.
A central database can be built around conversions available as values in the program. But it would be a bad idea to take it as the basic identification and require registering to use a conversion. Since there can be various schemes of encoding names (MIME, iconv's names) and not all encodings must have names in all schemes (I doubt MIME talks about "UCS-4 in native endianness", which corresponds to the array of Chars), I would not encourage to work in terms of names. I've implemented conversions for simple ISO-8859-x, CP-xxxx, CP-xxx, KOI8-R and Mac encodings (excluding those which need special bidi or multibyte processing). They are implemented in Haskell. iconv is implemented in C. Both kinds of implementations are handled during String <-> CString conversions in a way which I believe is as efficient as possible (roughly). OTOH QForeign's experimental converting IO replacement, IOConv, is not efficient at all for conversions implemented in C.
We already have mallocArray0 and pokeArray0. You only have to cast characters to [CChar].
Sure - but why not have this as predefined functions in CString? That's all I am proposing.
I would not encourage people to skip the conversion and produce code which works only for ISO-8859-1. Latin1 is just one of many encodings. Since string handling in Haskell is already inefficient, I hope that adding conversions would not make a big relative difference. It would be a different story if strings could be passed to C functions without marshalling.
BTW, do you know Pango <http://www.pango.org/>?
Not yet. -- Marcin 'Qrczak' Kowalczyk