Attached is a properly internationalized implementation of Foreign.C.String, along with some other routines which I feel would be very at home in the FFI standard. Note that I am trying to solve a simpler problem than full generic i18n. I just want the ability to work within the current locale, whatever it might be. I have tested these routines in utf8, latin1, greek, korean and a few other locales. they seem to work well. in addition to properly localeizing withCString, peek/pokeCString and friends I feel it is important to provide routines to work on wchar_t * strings. there are a number of reasons: * if __STDC_ISO_10646__ is defined (which is almost always), conversions can be incredibly optimized, in particular an array of Chars can be implemented directly as an array of wchar_t's * many c libraries nativly take wchar_t *s, converting to and from a multibyte encoding would be wasteful * wchar_t and the assosiated charset routines has been part of C since C90.1 and are quite stable. * and mainly, is is LIKELY that the current multibyte encoding is lossy compared to wchar_t. forcing people to go through char *s will break encodings for no good reason. * multiple locales are very common nowadays, not just for internationalization, but to differentiate between a utf8 based unix and a latin1 one. the routines I propose adding are: withCWString, withCWStringLen, newCWString, newCWStringLen, peekCWString, peekCWStringLen, -- same as CString counterparts charIsRepresentable :: Char -> IO Bool -- returns true if the argument can be represented in the current locale and the types CWChar, CWString, CWStringLen with CWChar == wchar_t and the others defined analogously to the CString versions. also, to a lesser extent I propose we add explicit utf8 routines: withUTF8String, withUTF8StringLen, newUTF8String, newUTF8StringLen, peekUTF8String, peekUTF8StringLen there are several libraries (X11 being a major one) which export an explicit utf8 based interface, it would be nice to be able to call their routines directly without worrying about the current locale. these might be more at home in a seperate library and not the FFI spec since they can be implemented independently, but can probably benefit from compiler dependent optimization. my implementation is pretty hairy, but can be improved. it currently only works on systems where __STDC_ISO_10646__ is defined, but fortunatly, I have never come across a system where it was not defined and it implemented wchar_t at all. the localized versions of the CString routines are named with LCString, which stands for localized C string...
Dnia wto 26. sierpnia 2003 02:04, John Meacham napisał:
my implementation is pretty hairy, but can be improved. it currently only works on systems where __STDC_ISO_10646__ is defined, but fortunatly, I have never come across a system where it was not defined and it implemented wchar_t at all.
AFAIK on FreeBSD wchar_t is not Unicode. It generally has broken locale and Unicode support in other respects as well.
the localized versions of the CString routines are named with LCString, which stands for localized C string...
In future they should replace current CString functions which can be renamed to Latin1CString. -- __("< Marcin Kowalczyk \__/ qrczak@knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/
On Tue, Aug 26, 2003 at 10:53:31AM +0200, Marcin 'Qrczak' Kowalczyk wrote:
Dnia wto 26. sierpnia 2003 02:04, John Meacham napisał:
my implementation is pretty hairy, but can be improved. it currently only works on systems where __STDC_ISO_10646__ is defined, but fortunatly, I have never come across a system where it was not defined and it implemented wchar_t at all.
AFAIK on FreeBSD wchar_t is not Unicode. It generally has broken locale and Unicode support in other respects as well. I guess we can't really provide a better interface than the system provides when it comes to locale. I am currently working on porting the code to solaris, where it appears that wchar_t is unicode 99% of the time but they are unwilling to guarentee it with __STDC_ISO_10646__ so I must do a run-time check of some sort.. perhaps something similar will work on the BSDs. At worst, we can always fall back to the 'C' locale which is guarenteed to be supported.
the localized versions of the CString routines are named with LCString, which stands for localized C string...
In future they should replace current CString functions which can be renamed to Latin1CString.
yeah, keeping these around would be good, there is no shortage of code which assumes everything is latin1 (like all of haskell until recently :) ) John -- --------------------------------------------------------------------------- John Meacham - California Institute of Technology, Alum. - john@foo.net ---------------------------------------------------------------------------
John Meacham wrote:
Attached is a properly internationalized implementation of Foreign.C.String, along with some other routines which I feel would be very at home in the FFI standard.
Note that I am trying to solve a simpler problem than full generic i18n. I just want the ability to work within the current locale, whatever it might be.
But bear in mind that other programmers may want to work in the "C" locale, regardless of the user's environment settings. This why the C library doesn't attempt to use those settings unless the program explicitly requests their use via setlocale(). Also, some libraries may fail to cope with locales other than "C"; particularly locales with multi-byte encodings.
also, to a lesser extent I propose we add explicit utf8 routines:
withUTF8String, withUTF8StringLen, newUTF8String, newUTF8StringLen, peekUTF8String, peekUTF8StringLen
there are several libraries (X11 being a major one) which export an explicit utf8 based interface,
Note that the Xutf8* functions are specific to XFree86's version of Xlib (and are only in 4.0.2 and later); they aren't in the vanilla OpenGroup version. They don't exist in vanilla X11R6, or in XFree86 3.x. Also those functions are redundant; you can always use the Xmb* functions with a UTF8-based locale instead. Simon Marlow wrote:
In our new implementation of Data.Char.isUpper and friends, I made the simplifying assumption that Char==wchar_t==Unicode. With glibc, this appears to be valid as long as (a) you set LANG to something other than "C" or "POSIX", and (b) you call setlocale() first.
The glibc Info file says: The wide character character set always is UCS4, at least on GNU systems.
We now call setlocale() in the RTS startup code.
So anyone who doesn't want to use the current locale now has to explicitly set it back to "C"? Also, this is just for LC_CTYPE, right? -- Glynn Clements <glynn.clements@virgin.net>
On Wed, Aug 27, 2003 at 04:28:27AM +0100, Glynn Clements wrote:
Attached is a properly internationalized implementation of Foreign.C.String, along with some other routines which I feel would be very at home in the FFI standard.
Note that I am trying to solve a simpler problem than full generic i18n. I just want the ability to work within the current locale, whatever it might be.
But bear in mind that other programmers may want to work in the "C" locale, regardless of the user's environment settings. This why the C library doesn't attempt to use those settings unless the program explicitly requests their use via setlocale().
Also, some libraries may fail to cope with locales other than "C"; particularly locales with multi-byte encodings.
then this is a library interface issue, not a locale one. If a library documents that an argument shall be nothing other than ascii text then call withAsciiCString, if the library accepts localized text then calal withCString (which is defined to return the localized version by the ffi spec) frankly, the ability to not call setlocale was a hack to work around migration issues in C programs, there is no need for haskell programs to inherit this complexity, if libraries need ascii strings, explicitly pass ascii strings, if they need localized strings pass localized ones and so forth. at worst, you can always do LANG=C foo to force a program to run in a specific locale. (but if people are smart about writing their C bindings, this sort of hack shouldn't ever be necisarry)
also, to a lesser extent I propose we add explicit utf8 routines:
withUTF8String, withUTF8StringLen, newUTF8String, newUTF8StringLen, peekUTF8String, peekUTF8StringLen
there are several libraries (X11 being a major one) which export an explicit utf8 based interface,
Note that the Xutf8* functions are specific to XFree86's version of Xlib (and are only in 4.0.2 and later); they aren't in the vanilla OpenGroup version. They don't exist in vanilla X11R6, or in XFree86 3.x.
Also those functions are redundant; you can always use the Xmb* functions with a UTF8-based locale instead.
yeah, I was giving them as an example of a place where they would be handy. locale hacks are bad! what if the system doesn't have a utf8 locale? in a multithreaded program temporarily changing the locale can be disastorous. Xutf8* lets us avoid this very nicely when they exist and we only need to fall back to less reliable locale tricks when necisarry. This is similar to my previous comment, interfaces either use data in the current locale or a specifified one only. haskell is of the later type, specifying unicode as it's character set, which means that we can talk to interfaces which are locale independent (like the Xutf8*) very easily with statically determined charset conversions. we should take advantage of this ability whenever possible.
Simon Marlow wrote:
In our new implementation of Data.Char.isUpper and friends, I made the simplifying assumption that Char==wchar_t==Unicode. With glibc, this appears to be valid as long as (a) you set LANG to something other than "C" or "POSIX", and (b) you call setlocale() first.
The glibc Info file says:
The wide character character set always is UCS4, at least on GNU systems. yes. with glibc, wchar_t is always unicode no matter what the locale. better yet, all ISO C implementations define a handy C symbol to test for this. if __STDC_ISO_10646__ is defined then wchar_t is always unicode no matter what.
We now call setlocale() in the RTS startup code. So anyone who doesn't want to use the current locale now has to explicitly set it back to "C"?
Also, this is just for LC_CTYPE, right?
if they want to use a different locale, they should change the enviornment prior to running the program. if they want to use code which only supports the C locale (meaning it only works with ascii) then call withAsciiCString and friends... John PS I made the implicit assumption that once *CString* was replaced by localized versions we would export the old versions under *AsciiCString* which just makes sense. -- --------------------------------------------------------------------------- John Meacham - California Institute of Technology, Alum. - john@foo.net ---------------------------------------------------------------------------
participants (3)
-
Glynn Clements -
John Meacham -
Marcin 'Qrczak' Kowalczyk