At 2002-08-07 02:43, Axel Simon wrote:
But the point was that C might have different sized characters and that these functions would still be portable even if the size of CChar changes.
Text encoded with ISO 8859-1 or UTF-8 is octets. If you want to use CChars, you should then subsequently convert the Word8s into CChars. -- Ashley Yakeley, Seattle WA
On Wed, Aug 07, 2002 at 02:54:47AM -0700, Ashley Yakeley wrote:
At 2002-08-07 02:43, Axel Simon wrote:
But the point was that C might have different sized characters and that these functions would still be portable even if the size of CChar changes.
Text encoded with ISO 8859-1 or UTF-8 is octets. If you want to use CChars, you should then subsequently convert the Word8s into CChars. Then I hope there is no C implementation where char is less than 8 bits long.
Axel.
On 2002-08-07T11:03:40+0100, Axel Simon wrote:
Then I hope there is no C implementation where char is less than 8 bits long.
Fortunately, standard C requires char to be at least 8 bits. I have a stake in using Haskell for international text processing: In particular, I have been writing Haskell code that typeset international text. Let me summarize what I think are the basic types of data that need to be distinguished and processed *somehow* within a Haskell program: (1) chars in C (perhaps distinguishing between unsigned, signed, and default) (2) 8-bit integers (i.e., signed) and words (i.e., unsigned) (3) Unicode code values (16-bit) (4) Unicode code points (32-bit, including surrogate characters, which are often treated as two consecutive 16-bit code values) The problem with the current situation is that Char in Haskell is supposed to mean 4, but in reality (e.g., GHC implementation) mostly means 1. The conflict in the present discussion arises from two desires: One, to use Char as 1 above, for FFI convenience and quick-and-dirty code. Two, to use Char as 4 above, for international text processing and conceptual correctness. I believe that we need library functions to: (a) Convert between 1 and 2, or more generally, convert between 1 and Integral types; (b) Convert between 2 and 4, under a specified encoding such as ISO-8859-1 or UTF-8; (c) Convert between 3 and 4, according to the Unicode standard. My proposal involves the following types: (1) Represent char in C as Char, and zero-terminated strings (char*) in C as CString. (2) Represent 8-bit integers and words as Int8 and Word8. (3) Represent Unicode code values as Word16, or a new Haskell type CodeValue. (4) Represent Unicode code points as Word32, or a new Haskell type CodePoint. String will continue to be a type synonym for [Char]. Some applications may find it useful to define a type synonym for [CodeValue] or [CodePoint], but given combining characters and other complexities of Unicode, I suspect many text processing applications will need a more sophisticated notion of text than just a sequence of characters. In C, char is a numeric type like int and long. In Haskell, we like to regulate against comparing 'A' against 65 directly. The functions ord and chr are useful in this regard. I suggest we extend ord and chr via a new type class, Character: class Eq c => Character c where fromChar :: Char -> c ord :: (Integral i) => c -> i chr :: (Integral i) => i -> c instance Character Char where ... instance Character CodePoint where ... instance Character CodeValue where ... Laws that should hold of Character include: chr (ord ch :: Integer) == ch for all ch :: c, where Character c fromChar char == char for all char :: Char The defaulting mechanism should be extended to character literals and string literals. That is, 'A' should mean something like chr 65 and "ABC" should mean something like map chr [65, 66, 67] Also, '\xABCD' should mean something like chr 0xABCD Pattern matching against character and string literals should make use of (==) from Eq. -- Edit this signature at http://www.digitas.harvard.edu/cgi-bin/ken/sig Use GPG!
On Wed, Aug 07, 2002 at 12:53:33PM -0400, Ken Shan wrote:
On 2002-08-07T11:03:40+0100, Axel Simon wrote:
Then I hope there is no C implementation where char is less than 8 bits long.
Fortunately, standard C requires char to be at least 8 bits. Does it?
I have a stake in using Haskell for international text processing: In particular, I have been writing Haskell code that typeset international text. Let me summarize what I think are the basic types of data that need to be distinguished and processed *somehow* within a Haskell program:
(1) chars in C (perhaps distinguishing between unsigned, signed, and default) (2) 8-bit integers (i.e., signed) and words (i.e., unsigned) (3) Unicode code values (16-bit) (4) Unicode code points (32-bit, including surrogate characters, which are often treated as two consecutive 16-bit code values)
The problem with the current situation is that Char in Haskell is supposed to mean 4, but in reality (e.g., GHC implementation) mostly means 1. Although other compilers might not, GHC does indeed support Unicode 32 bit characters directly. <advertisement> In gtk2hs ( gtk2hs.sourceforge.org ) I have a small demo displaying arabic text in a dialog which looks like this: arabic = map chr [0x647,0x644,32,0x62A,0x62C,0x62F,0x646,32,0x647,0x622, 0x633,0x643,0x622,0x644,32,0x644,0x63A,0x62A,32, 0x645,0x62F,0x647,0x634,0x62A,0x61F] </advertisement>
So instead the only thing we have to make sure is that we marshal strings from and to the outside correctly. I don't think anyone wants to fiddle with different representation within Haskell. Axel.
On 2002-08-07T18:26:49+0100, Axel Simon wrote:
Fortunately, standard C requires char to be at least 8 bits. Does it?
Yes. I am looking at section 5.2.4.2 ("Numerical limits") at the URL http://www.dkuug.dk/JTC1/SC22/WG14/www/docs/n843.htm , which is only a committee draft, but I don't think this part of the standard has changed.
Although other compilers might not, GHC does indeed support Unicode 32 bit characters directly. <advertisement> In gtk2hs ( gtk2hs.sourceforge.org ) I have a small demo displaying arabic text in a dialog which looks like this: arabic = map chr [0x647,0x644,32,0x62A,0x62C,0x62F,0x646,32,0x647,0x622, 0x633,0x643,0x622,0x644,32,0x644,0x63A,0x62A,32, 0x645,0x62F,0x647,0x634,0x62A,0x61F] </advertisement>
So instead the only thing we have to make sure is that we marshal strings from and to the outside correctly.
It seems that GTK deals directly with Unicode text. That is great. However, it is often insufficient to model text as a sequence of Unicode code points. In other words, it is impossible to marshal strings from and to the outside correctly with the (function types of the) current library. For example, a Haskell program should be able to - read and write multiple files and network sockets with different encodings; - to normalize Unicode strings into various normalization forms, for example as specified by the W3C working draft "Character Model for the World Wide Web" (http://www.w3.org/TR/2002/WD-charmod-20020430/); - to deal gracefully with unencodable characters, possibly with user interaction (e.g., "the encoding you have selected is insufficient for this document; please choose one of the following alternatives"); - to maintain state necessary for processing combining characters; - to distinguish between right-to-left or bidirectional text stored in "display order" versus "logical order"; - etc. Most of the functionality mentioned above are best handled by the Haskell code itself. It is an unrealistic simplification for the Haskell library to pretend that files store, sockets transmit, and foreign functions process Unicode characters (rather than say octets). Abstractions at a higher level than raw octets may be desirable in most circumstances, but different abstractions are needed by different applications, so a lower-lever interface should be provided to the Haskell programmer.
I don't think anyone wants to fiddle with different representation within Haskell.
As the examples above illustrate, I do. I would much rather fiddle with different representations within Haskell than within C. (: -- Edit this signature at http://www.digitas.harvard.edu/cgi-bin/ken/sig When there was no meat, we ate fowl. When there was no fowl, we ate crawdads. When there was no crawdads to be found, we ate sand.
On 2002-08-07T18:26:49+0100, Axel Simon wrote:
Although other compilers might not, GHC does indeed support Unicode 32 bit characters directly.
Let me clarify my understanding of this point a bit further. On the one hand, GHC uses Char to mean a 32-bit value like a Unicode code point. On the other hand, GHC uses Char to mean what files store and sockets transmit and foreign functions process under the C type "char". These two uses are inconsistent, and must be separated. In an earlier message, I suggested that Char mean "C char", and that a new type CodePoint be created to mean "Unicode code point". I would be perfectly happy -- in fact, happier personally -- if Char were to mean "Unicode code point" and a new type CChar were created to mean "C char". Either way, the (function types in the) libraries must be cleaned up to maintain the distinction between "C char" and "Unicode code point". Furthermore, Haskell programs must be able to access both notions. -- Edit this signature at http://www.digitas.harvard.edu/cgi-bin/ken/sig When there was no meat, we ate fowl. When there was no fowl, we ate crawdads. When there was no crawdads to be found, we ate sand.
On Wed, Aug 07, 2002 at 02:05:08PM -0400, Ken Shan wrote:
On 2002-08-07T18:26:49+0100, Axel Simon wrote: In an earlier message, I suggested that Char mean "C char", and that a new type CodePoint be created to mean "Unicode code point". I would be perfectly happy -- in fact, happier personally -- if Char were to mean "Unicode code point" and a new type CChar were created to mean "C char". Either way, the (function types in the) libraries must be cleaned up to maintain the distinction between "C char" and "Unicode code point". I agree. Haskell's Char is a Unicode code point, all the file and socket functions need to be augmented with encodings and decodings.
Furthermore, Haskell programs must be able to access both notions. I agree that it might be useful. But fiddeling with UTF-8 should mean using [Word8].
Axel.
Ken Shan <ken@digitas.harvard.edu> writes:
On the other hand, GHC uses Char to mean what files store and sockets transmit and foreign functions process under the C type "char".
Isn't "byte" or "octet" a better name for what files store and sockets transmit?
These two uses are inconsistent, and must be separated.
Right.
I would be perfectly happy -- in fact, happier personally -- if Char were to mean "Unicode code point" and a new type CChar were created to mean "C char".
I think this is a more likely scenario. I'd use Word8, and leave CChar for FFI purposes in case a "char" turns out to be different from eight bits.
Either way, the (function types in the) libraries must be cleaned up to maintain the distinction between "C char" and "Unicode code point". Furthermore, Haskell programs must be able to access both notions.
Would it be sufficient to have "raw" socket/file functions using [Word8], and let the "standard" functions (e.g. readFile) convert to [Char] according to current locale settings? With, perhaps, UTF-8 as a reasonable default? (And of course, en/decoding functions readily available for manual use) -kzm -- If I haven't seen further, it is by standing in the footprints of giants
On Thursday, 2002-08-08, 11:54, CEST, Ketil Z. Malde wrote:
[...] Isn't "byte" or "octet" a better name for what files store and sockets transmit?
In school, university and whatever, I always learned that a byte is a sequence of eight bits. But later I read (I think in the Internet RFC) something about the possibility of bytes being seven bits long. An octet is, as the name suggest, defined to be a sequence of eight bits. AFAIK, there is no guaranty about how many bits "file elements" have. In Ada 95, there is an implementation dependent type Stream_Element (and also an implementation dependent type Storage_Element which corresponds to memory cells). But in practice, I think, it is ok to assume a file element being an octet.
[...]
Wolfgang
ketil@ii.uib.no (Ketil Z. Malde) wrote,
Ken Shan <ken@digitas.harvard.edu> writes:
I would be perfectly happy -- in fact, happier personally -- if Char were to mean "Unicode code point" and a new type CChar were created to mean "C char".
I think this is a more likely scenario. I'd use Word8, and leave CChar for FFI purposes in case a "char" turns out to be different from eight bits.
I haven't followed this discussion in detail, but maybe the purpose of the inclusion of CChar in the FFI is being slightly misunderstood here. All the Cxyz types from CForeign are included to be able to write portable library bindings for libraries coded in C (or at least compiled to use C conventions). In particular, the idea is that CChar represents exactly what the corrsponding C compiler makes "char" to be. If you want an octet (without explicitly referring to some C library functions), you are better off using Int8 or Word8. Manuel
Axel Simon <A.Simon@ukc.ac.uk> wrote,
On Wed, Aug 07, 2002 at 02:54:47AM -0700, Ashley Yakeley wrote:
At 2002-08-07 02:43, Axel Simon wrote:
But the point was that C might have different sized characters and that these functions would still be portable even if the size of CChar changes.
Text encoded with ISO 8859-1 or UTF-8 is octets. If you want to use CChars, you should then subsequently convert the Word8s into CChars. Then I hope there is no C implementation where char is less than 8 bits long.
ANSI C guarantees that char is 1 byte (more precisely that "sizeof (char)" == 1). Manuel
participants (6)
-
Ashley Yakeley -
Axel Simon -
Ken Shan -
ketil＠ii.uib.no -
Manuel M T Chakravarty -
Wolfgang Jeltsch