In our new implementation of Data.Char.isUpper and friends, I made the simplifying assumption that Char==wchar_t==Unicode. With glibc, this appears to be valid as long as (a) you set LANG to something other than "C" or "POSIX", and (b) you call setlocale() first.
The glibc Info file says:
The wide character character set always is UCS4, at least on GNU systems. yes. with glibc, wchar_t is always unicode no matter what the locale. better yet, all ISO C implementations define a handy C symbol to test for this. if __STDC_ISO_10646__ is defined then wchar_t is always unicode no matter what.
Sure, but as I've been saying, the implementation of glibc doesn't do this. In the C or POSIX locale, the ctype macros only recognise ASCII. Try it: #include <wctype.h> #include <stdio.h> #include <locale.h> main() { setlocale(LC_ALL,""); printf("%d\n", iswupper('A')); printf("%d\n", iswupper(0x391)); // Greek capital alpha printf("%d\n", iswupper(0x3B1)); // Greek small alpha printf("%d\n", iswlower(0x391)); // Greek capital alpha printf("%d\n", iswlower(0x3B1)); // Greek small alpha } $ LANG=en_GB ./a.out 1 1 0 0 1 $ LANG=C ./a.out 1 0 0 0 0 Should this be considered a bug in glibc? Cheers, Simon
On Wed, Aug 27, 2003 at 09:58:08AM +0100, Simon Marlow wrote:
In our new implementation of Data.Char.isUpper and friends, I made the simplifying assumption that Char==wchar_t==Unicode. With glibc, this appears to be valid as long as (a) you set LANG to something other than "C" or "POSIX", and (b) you call setlocale() first.
The glibc Info file says: The wide character character set always is UCS4, at least on GNU systems. yes. with glibc, wchar_t is always unicode no matter what the locale. better yet, all ISO C implementations define a handy C symbol to test for this. if __STDC_ISO_10646__ is defined then wchar_t is always unicode no matter what.
Sure, but as I've been saying, the implementation of glibc doesn't do this. In the C or POSIX locale, the ctype macros only recognise ASCII.
Should this be considered a bug in glibc?
hmm.. how odd. I would consider it a bug, I think. I don't have a copy of the ISO spec handy but will be sure to look up whether that is conforming... It is certainly a malfeature if it is not a bug... John -- --------------------------------------------------------------------------- John Meacham - California Institute of Technology, Alum. - john@foo.net ---------------------------------------------------------------------------
John Meacham wrote:
In our new implementation of Data.Char.isUpper and friends, I made the simplifying assumption that Char==wchar_t==Unicode. With glibc, this appears to be valid as long as (a) you set LANG to something other than "C" or "POSIX", and (b) you call setlocale() first.
The glibc Info file says: The wide character character set always is UCS4, at least on GNU systems. yes. with glibc, wchar_t is always unicode no matter what the locale. better yet, all ISO C implementations define a handy C symbol to test for this. if __STDC_ISO_10646__ is defined then wchar_t is always unicode no matter what.
Sure, but as I've been saying, the implementation of glibc doesn't do this. In the C or POSIX locale, the ctype macros only recognise ASCII.
Should this be considered a bug in glibc?
hmm.. how odd. I would consider it a bug, I think. I don't have a copy of the ISO spec handy but will be sure to look up whether that is conforming... It is certainly a malfeature if it is not a bug...
It certainly isn't a violation of ANSI/ISO C; that simply states that "The behavior of these functions is affected by the LC_CTYPE category of the current locale". It's perfectly legal for the implementation to use different wide encodings depending upon the locale. OTOH, it could be considered contrary to the statement in the glibc documentation that "The wide character character set always is UCS4". glibc's wide-character encoding for the C locale may use the same codepoints as UCS4, but glibc appears to treat everything above 255 as "undefined" whereas UCS4 assigns an interpretation. On my RH6.2 (glibc 2.1.3) system, Simon's test program suggests that glibc exhibits the "ASCII" behaviour for any locale for which the file /usr/share/locale/<locale>/LC_CTYPE doesn't exist (as well as the "C" and "POSIX" locales, where it doesn't attempt to open an LC_CTYPE file). -- Glynn Clements <glynn.clements@virgin.net>
On Thu, Aug 28, 2003 at 05:34:09AM +0100, Glynn Clements wrote:
John Meacham wrote:
In our new implementation of Data.Char.isUpper and friends, I made the simplifying assumption that Char==wchar_t==Unicode. With glibc, this appears to be valid as long as (a) you set LANG to something other than "C" or "POSIX", and (b) you call setlocale() first. The glibc Info file says: The wide character character set always is UCS4, at least on GNU systems. yes. with glibc, wchar_t is always unicode no matter what the locale. better yet, all ISO C implementations define a handy C symbol to test for this. if __STDC_ISO_10646__ is defined then wchar_t is always unicode no matter what.
Sure, but as I've been saying, the implementation of glibc doesn't do this. In the C or POSIX locale, the ctype macros only recognise ASCII.
Should this be considered a bug in glibc?
hmm.. how odd. I would consider it a bug, I think. I don't have a copy of the ISO spec handy but will be sure to look up whether that is conforming... It is certainly a malfeature if it is not a bug...
It certainly isn't a violation of ANSI/ISO C; that simply states that "The behavior of these functions is affected by the LC_CTYPE category of the current locale". It's perfectly legal for the implementation to use different wide encodings depending upon the locale.
no, glibc #defines __STDC_ISO_10646__ so wchar_t's are guarenteed to hold UCS4 values always independent of locale. the LC_CTYPE only affects what multibyte encoding is used. What was curious was that the character classification routines changed behavior based on LC_CTYPE (despite the encoding still being UCS4) this might make sense for the classification routines dealing with upper and lower case actually, since I believe that that might depend on the language you are expressing. however, other character classification routines (such as wcwidth) should not depend on the current locale. it is unclear what the correct thing for an haskell implementation to do. possibilities are: 1) determine some locale independent semantics for the classification functions and implement that 2) guarentee the validity of character classification routines only when the character is representable in the current locale 3) link against another library such as libunicode which provides its own classification routines (this could be done optionally at compile time...) split the classification routines into locale dependent and independent ones, guarentee the locale independent ones will always work and one of the two above solutions for the rest... In any case, solution 2 seems to be what we have now, which is probably an okay interim solution as long as we add a isRepresentable to determine if a Char can be expressed in the current locale and whether we can trust the cclasification functions... I have an implementation of one in the CWString library I posted earlier... in any case, anything is better than the current 'ignore the locale' situation :) John -- --------------------------------------------------------------------------- John Meacham - California Institute of Technology, Alum. - john@foo.net ---------------------------------------------------------------------------
John Meacham wrote:
Sure, but as I've been saying, the implementation of glibc doesn't do this. In the C or POSIX locale, the ctype macros only recognise ASCII.
Should this be considered a bug in glibc?
hmm.. how odd. I would consider it a bug, I think. I don't have a copy of the ISO spec handy but will be sure to look up whether that is conforming... It is certainly a malfeature if it is not a bug...
It certainly isn't a violation of ANSI/ISO C; that simply states that "The behavior of these functions is affected by the LC_CTYPE category of the current locale". It's perfectly legal for the implementation to use different wide encodings depending upon the locale.
no, glibc #defines __STDC_ISO_10646__ so wchar_t's are guarenteed to hold UCS4 values always independent of locale.
OK; although the draft which I have only says: __STDC_ISO_10646__ A decimal constant of the form yyyymmL | (for example, 199712L), intended to | indicate that values of type wchar_t are | the coded representations of the | characters defined by ISO/IEC 10646, | along with all amendments and technical | corrigenda as of the specified year and | month. That's the only reference to that macro in the entire document. It doesn't explicitly contradict (or even reference) the comments about the semantics of the <wctype.h> functions.
the LC_CTYPE only affects what multibyte encoding is used. What was curious was that the character classification routines changed behavior based on LC_CTYPE (despite the encoding still being UCS4)
this might make sense for the classification routines dealing with upper and lower case actually, since I believe that that might depend on the language you are expressing. however, other character classification routines (such as wcwidth) should not depend on the current locale.
There are some variations between wcwidth() implementations; e.g. the XFree86 version of xterm includes two implementations, and the comment: * The following functions are the same as mk_wcwidth() and * mk_wcwidth_cjk(), except that spacing characters in the East Asian * Ambiguous (A) category as defined in Unicode Technical Report #11 * have a column width of 2. This variant might be useful for users of * CJK legacy encodings who want to migrate to UCS without changing * the traditional terminal character-width behaviour. It is not * otherwise recommended for general use. I suppose that it's possible that some systems might wish to make the behaviour locale-dependent. However, this is all a long way from the glibc behaviour, i.e. that for the C/POSIX locale, and for locales without an LC_CTYPE data file, everything outside of the ASCII range is undefined (not a member of any category, not translated by towupper() etc).
it is unclear what the correct thing for an haskell implementation to do. possibilities are: 1) determine some locale independent semantics for the classification functions and implement that 2) guarentee the validity of character classification routines only when the character is representable in the current locale 3) link against another library such as libunicode which provides its own classification routines (this could be done optionally at compile time...)
split the classification routines into locale dependent and independent ones, guarentee the locale independent ones will always work and one of the two above solutions for the rest...
In any case, solution 2 seems to be what we have now, which is probably an okay interim solution as long as we add a isRepresentable to determine if a Char can be expressed in the current locale and whether we can trust the cclasification functions... I have an implementation of one in the CWString library I posted earlier...
in any case, anything is better than the current 'ignore the locale' situation :)
Not necessarily. E.g. there are reasons why most programs don't just call setlocale(LC_ALL, "") to make everything behave according to the locale settings. I18N complicates many things sufficiently that I would favour forcing the programmer to explicitly ask for it (i.e. don't change the semantics of any existing functions, but provide new functions for use in internationalised code). -- Glynn Clements <glynn.clements@virgin.net>
participants (3)
-
Glynn Clements -
John Meacham -
Simon Marlow