Hi Dimitry, | Anyone interested in Unicode support in Hugs (what it lacks so far) | please check out this URL: | | http://www.golubovsky.org/software/hugs-patch/article.html Thanks for posting this. I notice from your comments that you were trying to figure out how Hugs works from the C code alone. For many purposes, you and others trying to do similar things might find that the report I wrote about the implementation of Gofer (from which Hugs was derived) still contains many relevant tidbits of information that might make some things easier to understand. You can find it online at: http://www.cse.ogi.edu/~mpj/pubs/goferimp.html You might also find it interesting for its historical perspective. For example, it might explain some of the design decisions that made sense back then but seem harder to justify today (such as the lack of support for Unicode, which barely existed back then ...) All the best, Mark

[returning to the list after some discussion with Dimitry] Just to be clear, the Unicode support under discussion comprises only: - making ord(maxBound::Char) a lot bigger, say 0x10FFFD. - making the character classification and case conversion functions in the Char module work on the expanded range. Dimitry's approach is based on mapping Unicode code points to a smaller set of internal codes, both to conserve cell tags (which currently include all character codes) and to reduce the size of the mapping tables. One problem there is that there are an order of magnitude more characters than he thought: 235963 out of 1114110 possible codes are allocated in Unicode 4.0.1. But also the Haskell Char type belongs to the Bounded, Enum and Ix classes, and so should be a contiguous range of values with no gaps. So I think the Unicode code points should be used internally as well. How should these be stored on the heap? We could use a scheme similar to the representation of Int: values smaller than some threshold map to cell tags, while larger values are stored in a pair. How do we implement the conversion functions? The approach recently added to the CVS version of GHC is to use the native libraries, which requires the user to set the locale appropriately. More generally, should these functions be locale-dependent at all? I appreciate that case conversion varies between languages (e.g. i's in Turkish) but I think that requires the IO monad. The Haskell Report is vague on the issue, but I think the functions should just do the default mappings, just as Ord compares character codes rather than using locales. The alternative is to include tables in the Hugs binary. These could be quite compact: fewer than 800 Unicode characters have upper case mappings, and similarly for lower case mappings. Also, adjacent Unicode characters often belong to the same category: the full range can be divided into fewer than 2000 contiguous subranges of characters belonging to the same category, including gaps of unallocated characters. Applying binary search to these tables would take a little time, but probably not much more than the current in-Haskell implementations. Less than 30k of data would be required. Another problem is that Hugs contains an array indexed by character values (consCharArray) containing (: c) for each character c. With a much larger range of character codes, one approach would be a hash table holding a limited number of such values, built lazily and bypassed when full. This change could be confined to the consChar() function in builtin.c. Finally, most of this could be #ifdef UNICODE_CHARS, so the overhead could be turned off if required.