At 2002-08-07 11:05, Ken Shan wrote:
Let me clarify my understanding of this point a bit further. On the one hand, GHC uses Char to mean a 32-bit value like a Unicode code point.
No, GHC uses Char to mean a Unicode codepoint. These are not 32-bit. It only allows the 17 pages i.e. values in the range '\x0' to '\x10FFFF'. This is the Right Thing as per Unicode 3.1 and later (current is 3.2.0).
On the other hand, GHC uses Char to mean what files store and sockets transmit and foreign functions process under the C type "char".
Right, and this is a very bad idea. The file IO functions should be using Word8s
These two uses are inconsistent, and must be separated.
I agree. At 2002-08-07 09:53, Ken Shan wrote:
I have a stake in using Haskell for international text processing: In particular, I have been writing Haskell code that typeset international text. Let me summarize what I think are the basic types of data that need to be distinguished and processed *somehow* within a Haskell program:
(1) chars in C (perhaps distinguishing between unsigned, signed, and default) (2) 8-bit integers (i.e., signed) and words (i.e., unsigned) (3) Unicode code values (16-bit)
I think the whole 16-bit code value thing was dropped as of 3.1. UTF-16 uses 16-bit values to represent text just as UTF-8 uses 8-bit values.
The conflict in the present discussion arises from two desires: One, to use Char as 1 above, for FFI convenience and quick-and-dirty code. Two, to use Char as 4 above, for international text processing and conceptual correctness.
I believe that we need library functions to:
(a) Convert between 1 and 2, or more generally, convert between 1 and Integral types; (b) Convert between 2 and 4, under a specified encoding such as ISO-8859-1 or UTF-8; (c) Convert between 3 and 4, according to the Unicode standard.
You mean according to UTF-16.
My proposal involves the following types:
(1) Represent char in C as Char, and zero-terminated strings (char*) in C as CString.
We already have the CChar type that means that.
(2) Represent 8-bit integers and words as Int8 and Word8.
Agreed.
(3) Represent Unicode code values as Word16, or a new Haskell type CodeValue.
I don't think it's appropriate to have a new type. UTF-16 is a way of representing codepoints as 16-bit integers. The UTF-16 functions should use Word16.
(4) Represent Unicode code points as Word32, or a new Haskell type CodePoint.
We already have the Char type that means that -- in GHC, at least. String and character literals in programs should use it. -- Ashley Yakeley, Seattle WA
Ashley Yakeley wrote:
At 2002-08-07 11:05, Ken Shan wrote:
Let me clarify my understanding of this point a bit further. On the one hand, GHC uses Char to mean a 32-bit value like a Unicode code point.
No, GHC uses Char to mean a Unicode codepoint. These are not 32-bit. It only allows the 17 pages i.e. values in the range '\x0' to '\x10FFFF'. This is the Right Thing as per Unicode 3.1 and later (current is 3.2.0).
On the other hand, GHC uses Char to mean what files store and sockets transmit and foreign functions process under the C type "char".
Right, and this is a very bad idea. The file IO functions should be using Word8s
It's often very useful to treat a file as a sequence of characters; in fact I'd say that's probably more common than treating them as a sequence of octets. But both are clearly needed. In my opinion, hPutChar :: Handle -> Char -> IO () should do what its name and type indicate -- write a character to the specified output handle. The I/O subsystem should take care of translation to UTF-8 (or whatever the system encoding is). hPutWord8 :: Handle -> Word8 -> IO () should be available _in addition to_ hPutChar, for applications that need to treat files as a sequence of octets.
These two uses are inconsistent, and must be separated. I agree.
Me too; but both character-based and octet-based operations are needed. --Joe English jenglish@flightlab.com
On Thu, 2002-08-08 at 01:34, Joe English wrote:
It's often very useful to treat a file as a sequence of characters; in fact I'd say that's probably more common than treating them as a sequence of octets. But both are clearly needed.
I agree with Ashley, a file is a sequence of Word8s. Very often we use files to store a sequence of characters. Treating the file as a sequence of characters is one level higher, though. In between lies the (de)serialization of the charactars (according to some code), just like you'd have it with every other "object". Is there a compelling reason against simply providing character en-/decoding functions?
In my opinion, hPutChar :: Handle -> Char -> IO () should do what its name and type indicate -- write a character to the specified output handle. The I/O subsystem should take care of translation to UTF-8 (or whatever the system encoding is).
Is there a sensible way of defining the "system encoding"? UTF-8 is a superset of the ASCII, correct? As in, encoding ASCII characters in UTF-8 yields valid ASCII? If so, this actually sounds like it would usually produce the desired result without much hassle to the developer. However, I think it is of great importance to make it very clear that a file in itself _is not_ a sequence of characters, but 8-bit-words and that any function like hPutChar would actually be just a shortcut for something like hPut . encodeChar along with which there would also be encodeCharUTF8 encodeCharASCII ... Hm, what about encodeCharUTF16? Would that return Word16s? Hrm. But then, how to write that to a file? Would there be a reason against encodeCharUTF16 returning Word8s? Otherwise there would have to be two separate functions or another level which would convert Word16s to Word8s.
hPutWord8 :: Handle -> Word8 -> IO () should be available _in addition to_ hPutChar, for applications that need to treat files as a sequence of octets.
I think having one hPut :: Handle -> Word8 -> IO () along with a bunch of serialization functions is preferable to seperate hPutWord8, hPutWord16, hPutChar, ... functions just because it gives a much cleaner picture of what's actually happening. Maybe there could even be a class Serializable so one could have hPut :: Serializable a => Handle -> a -> IO () . I think (under the assumption that encoding ASCII characters in UTF-8 yields valid ASCII code) this would actually make the nicest design, because it accomplishes these goals: - The programmer need not usually concern himself with en-/decoding. - If she needs control, she can use a specific function herself and pass the resulting [Word8] (which is trivially Serializable) to hPut. - The code expresses directly the need to encode a piece of data before it can be written to a file. Regards, Sven Moritz
It's often very useful to treat a file as a sequence of characters; in fact I'd say that's probably more common than treating them as a sequence of octets. But both are clearly needed.
hPutWord8 :: Handle -> Word8 -> IO () should be available _in addition to_ hPutChar, for applications that need to treat files as a sequence of octets.
These two uses are inconsistent, and must be separated. I agree.
How different is this from the idea of open a file in binary mode and in text mode? I know that Hugs distinguishes the two because I remember the case that triggered the requirement: generating MIDI files on Win32. I guess what I'm saying is do we need new read/write operations or just new open operations? -- Alastair Reid alastair@reid-consulting-uk.ltd.uk Reid Consulting (UK) Limited http://www.reid-consulting-uk.ltd.uk/alastair/
On Thu, 2002-08-08 at 02:42, Alastair Reid wrote:
How different is this from the idea of open a file in binary mode and in text mode? I know that Hugs distinguishes the two because I remember the case that triggered the requirement: generating MIDI files on Win32.
I guess what I'm saying is do we need new read/write operations or just new open operations?
Do we really want different open modes with respect to how the data in the file is presented to us? I think the whole idea about text/binary mode is wrong. Reading and interpretation of the data in a file are two entirely different operations, so sticking one (!) aspect of interpretation so deeply into the interface for _reading_ a file scatters the task of interpreting the file and results in confusion. Sven Moritz
A mailing list (for a working group) would be great. haskell-i18n perhaps? Or haskell-unicode? Who creates lists on haskell.org? Should I get one made at eecs.harvard.edu? Should I switch to haskell-cafe for now? The following can probably wait until the list is created, but: What do people think of the Character type class? On 2002-08-07T15:21:11-0700, Ashley Yakeley wrote:
No, GHC uses Char to mean a Unicode codepoint. These are not 32-bit. It only allows the 17 pages i.e. values in the range '\x0' to '\x10FFFF'. This is the Right Thing as per Unicode 3.1 and later (current is 3.2.0).
Ah, neat! I missed it before; thanks.
(c) Convert between 3 and 4, according to the Unicode standard. You mean according to UTF-16.
Yes.
(1) Represent char in C as Char, and zero-terminated strings (char*) in C as CString. We already have the CChar type that means that.
One thing I am concerned about is that getChar should be IO CChar, not IO Char. So CChar needs to be part of the language standard (in fact, the Prelude). It would require quite a bit of modification for old code to work -- old code that all along has pretty much used Char as a synonym for Word8. -- Edit this signature at http://www.digitas.harvard.edu/cgi-bin/ken/sig http://www.ethnologue.com/
Ken Shan <ken@digitas.harvard.edu> writes:
One thing I am concerned about is that getChar should be IO CChar, not IO Char.
Why? Why not a separate getWord8 that is IO Word8 instead? At least in my code, I mostly use textual IO, so when I say getChar, I usually mean get a Char, not a Word8. Other people's code probably differs, so I guess it is a question of what will break the least code. But I also think the name of the function shouldn't be ignored. -kzm -- If I haven't seen further, it is by standing in the footprints of giants
participants (6)
-
Alastair Reid -
Ashley Yakeley -
Joe English -
Ken Shan -
ketil＠ii.uib.no -
Sven Moritz Hallberg