Axel wrote [snip]
I guess that is a good point, but due to backwards compatibility this is propably not acceptable: The C interface of the FFI has the string functions: peekCString :: CString -> IO String newCString :: String -> IO CString
which should really be
peekCString :: CString -> IO [Word8] newCString :: [Word8] -> IO CString
Unless that changes, there is really no point to give the encode and decode functions that type. [snip] Such a change would be annoying, since I have already used peekCString and newCString quite a lot. (They are a great improvement on what we had before!)
Converting CStrings to [Word8] is probably a bad idea anyway, since there is absolutely no reason to assume a C character will be only 8 bits long, and under some implementations it isn't. A better suggestion would be to provide ALTERNATIVE functions which got from CString/CStringLen and friends to [CChar], and make your UTF8 converters go between [CChar] and String. However we should not be forced to do this every time we want to construct a CString from a String (a very common need when calling C functions) so the existing functions should remain with their existing semantics.
On Tue, Aug 06, 2002 at 06:11:04PM +0200, George Russell wrote: [snip]
Converting CStrings to [Word8] is probably a bad idea anyway, since there is absolutely no reason to assume a C character will be only 8 bits long, and under some implementations it isn't. But the interface should be practical. I do not really want to write Haskell programs for architectures where the smallest addressable memory entity (i.e. C's char) is something else than 8 bits.
A better suggestion would be to provide ALTERNATIVE functions which got from CString/CStringLen and friends to [CChar], and make your UTF8 converters go between [CChar] and String. However we should not be forced to do this every time we want to construct a CString from a String (a very common need when calling C functions) so the existing functions should remain with their existing semantics. But converting CChar to Char means you are assuming that the C String is ISO-8859-1, the lower 255 characters of Unicode. I guess this should be made explicit during conversion.
Axel.
Axel Simon wrote:
On Tue, Aug 06, 2002 at 06:11:04PM +0200, George Russell wrote: [snip]
Converting CStrings to [Word8] is probably a bad idea anyway, since there is absolutely no reason to assume a C character will be only 8 bits long, and under some implementations it isn't. But the interface should be practical. I do not really want to write Haskell programs for architectures where the smallest addressable memory entity (i.e. C's char) is something else than 8 bits.
Yes but if all we are talking about is practicality, I *do* want to convert between CString's and ordinary String's in the conventional way, and I bet lots of other people do.
A better suggestion would be to provide ALTERNATIVE functions which got from CString/CStringLen and friends to [CChar], and make your UTF8 converters go between [CChar] and String. However we should not be forced to do this every time we want to construct a CString from a String (a very common need when calling C functions) so the existing functions should remain with their existing semantics. But converting CChar to Char means you are assuming that the C String is ISO-8859-1, the lower 255 characters of Unicode. I guess this should be made explicit during conversion.
I suggest that for conversions CChar to Char all that should be required should be that it be a total function and that the printable ASCII characters map in the obvious way and that the results should be consistent during the course of a Haskell program. However I see no reason why the conversion should not take account of locale when the program starts and so translate (say) KOI-8 encoded Cyrillic characters to their Unicode equivalents. For Char to CChar it would surely be easiest to produce an IO error if the character doesn't fit.
On Tue, Aug 06, 2002 at 07:57:50PM +0200, George Russell wrote:
Axel Simon wrote:
On Tue, Aug 06, 2002 at 06:11:04PM +0200, George Russell wrote: [snip]
Converting CStrings to [Word8] is probably a bad idea anyway, since there is absolutely no reason to assume a C character will be only 8 bits long, and under some implementations it isn't. But the interface should be practical. I do not really want to write Haskell programs for architectures where the smallest addressable memory entity (i.e. C's char) is something else than 8 bits.
Yes but if all we are talking about is practicality, I *do* want to convert between CString's and ordinary String's in the conventional way, and I bet lots of other people do. Let's stick to CChar and provide conversion functions then! (See below)
I suggest that for conversions CChar to Char all that should be required should be that it be a total function and that the printable ASCII characters map in the obvious way and that the results should be consistent during the course of a Haskell program. However I see no reason why the conversion should not take account of locale when the program starts and so translate (say) KOI-8 encoded Cyrillic characters to their Unicode equivalents.
For Char to CChar it would surely be easiest to produce an IO error if the character doesn't fit. For safety reasons I think the user should be aware of what he is doing. Just using withCString doesn't make the user aware of possible problems. I guess we need:
encodeISO-8859-1 :: String -> [CChar] encodeISO-8859-1 = fromIntegral encodeUTF-8 = ... encodeDefault = case currentCodeset of ISO-8859-1 -> encodeISO-8859-1 UTF-8 -> encodeUTF-8 writeFile fname str = writeBinaryFile fname (encodeDefault str) withCString str = withArray 0 (encodeDefault str) An the other way round: decodeISO-8859-1 :: [CChar] -> String decodeISO-8859-1 = fromIntegral decodeUTF-8 = ... decodeGuess ('<magic number for UTF-8>':xs) = decodeUTF-8 xs decodeGuess ... decodeGuess = decodeDefault decodeDefault = case currentCodeset of ... readFile fname = liftM decodeDefault $ readBinaryFile fname peekCString sPtr = liftM decodeDefault $ peekArray0 0 sPtr Does that sound reasonable? In the documentation of GHC it says: data CChar = CChar Int8 Is this determined during compilation of GHC? Axel.
In the documentation of GHC it says:
data CChar = CChar Int8
Is this determined during compilation of GHC?
It's done at configure time. fptools/configure.in contains these lines: dnl ** map standard C types and ISO types to Haskell types FPTOOLS_CHECK_HTYPE(char) FPTOOLS_CHECK_HTYPE(signed char) FPTOOLS_CHECK_HTYPE(unsigned char) FPTOOLS_CHECK_HTYPE(short) FPTOOLS_CHECK_HTYPE(unsigned short) FPTOOLS_CHECK_HTYPE(int) FPTOOLS_CHECK_HTYPE(unsigned int) FPTOOLS_CHECK_HTYPE(long) FPTOOLS_CHECK_HTYPE(unsigned long) if test "$fptools_cv_have_long_long" = yes; then FPTOOLS_CHECK_HTYPE(long long) FPTOOLS_CHECK_HTYPE(unsigned long long) fi FPTOOLS_CHECK_HTYPE(float) FPTOOLS_CHECK_HTYPE(double) FPTOOLS_CHECK_HTYPE(ptrdiff_t) FPTOOLS_CHECK_HTYPE(size_t) FPTOOLS_CHECK_HTYPE(wchar_t) dnl Int32 is a HACK for non-ISO C compilers FPTOOLS_CHECK_HTYPE(sig_atomic_t, Int32) FPTOOLS_CHECK_HTYPE(clock_t) FPTOOLS_CHECK_HTYPE(time_t) FPTOOLS_CHECK_HTYPE(dev_t, Word32) FPTOOLS_CHECK_HTYPE(ino_t) FPTOOLS_CHECK_HTYPE(mode_t) FPTOOLS_CHECK_HTYPE(off_t) FPTOOLS_CHECK_HTYPE(pid_t) FPTOOLS_CHECK_HTYPE(gid_t) FPTOOLS_CHECK_HTYPE(uid_t) FPTOOLS_CHECK_HTYPE(cc_t) FPTOOLS_CHECK_HTYPE(speed_t) FPTOOLS_CHECK_HTYPE(tcflag_t) FPTOOLS_CHECK_HTYPE(nlink_t) FPTOOLS_CHECK_HTYPE(ssize_t) where CHECK_HTYPE (defined in fptools/aclocal.m4) figures out the best match out of {{Int,Word}{8,16,32,64},Float,Double} -- Alastair Reid alastair@reid-consulting-uk.ltd.uk Reid Consulting (UK) Limited http://www.reid-consulting-uk.ltd.uk/alastair/
Axel Simon wrote: [snip]
Let's stick to CChar and provide conversion functions then! (See below)
I would prefer the quick and dirty existing functions to remain, because they correspond to what people often what. It should be specified that they work for the ASCII characters with codes 0-127 if that's possible. Since the majority of programmers probably neither know nor care about ISO-8859-1/UTF8/ koi8/whatever I think you will find that if you don't do this, most of them will engineer their quick and dirty alternatives anyway. [snip]
For safety reasons I think the user should be aware of what he is doing. Just using withCString doesn't make the user aware of possible problems. I guess we need:
[snip] This is all too complicated for me. Look, we seem to spend a vast amount of time on this list discussing Unicode and internationalisation, but nothing actually gets done. May I suggest that either (a) we drop it and leave things to take their course, or (b) we don't drop it, but set up a proper working party and mailing list, to come up with a standard? Unfortunately, as we have seen, this interacts with FFI, but that can't be helped.
George Russell <ger@tzi.de> writes:
Look, we seem to spend a vast amount of time on this list discussing Unicode and internationalisation, but nothing actually gets done. May I suggest that either (a) we drop it and leave things to take their course, or (b) we don't drop it, but set up a proper working party and mailing list, to come up with a standard?
I think a proper working group is called for. Various individuals on other groups have made suggestions and had some influence but they're usually trying to persuade people who don't much care that their beautiful design should be cluttered by something they don't care about.
Unfortunately, as we have seen, this interacts with FFI, but that can't be helped.
I think the interaction is fairly small: the ffi is mostly concerned with C and C is mostly not concerned with internationalization. Where I see big changes is in libraries (network, filesystem, date, etc.) (which might happen to be implemented in C) which have internationalization issues. -- Alastair Reid alastair@reid-consulting-uk.ltd.uk Reid Consulting (UK) Limited http://www.reid-consulting-uk.ltd.uk/alastair/
On 06-Aug-2002, George Russell <ger@tzi.de> wrote:
Converting CStrings to [Word8] is probably a bad idea anyway, since there is absolutely no reason to assume a C character will be only 8 bits long, and under some implementations it isn't.
That's true in general; the C standard only guarantees that a C character will be at least 8 bits long. But Posix now guarantees that C's `char' is exactly 8 bits. Posix hasn't taken over the world yet, and doesn't look like doing so in the near future. So Haskell should not limit itself to being only implementable on Posix systems. However, systems which don't have 8-bit bytes are getting very very rare nowadays -- it might well be reasonable for Haskell, like Posix, to limit itself to only being implementable on systems where C's `char' is exactly 8 bits. -- Fergus Henderson <fjh@cs.mu.oz.au> | "I have always known that the pursuit The University of Melbourne | of excellence is a lethal habit" WWW: <http://www.cs.mu.oz.au/~fjh> | -- the last words of T. S. Garp.
participants (4)
-
Alastair Reid -
Axel Simon -
Fergus Henderson -
George Russell