- Haskell - Haskell.org

8 bit characters?
by George Russell 09 Aug '02

09 Aug '02

Patryk Zadarnowski wrote [snip] > > > > But there's no guarantee that char is exactly 8 bits wide, is there? So > > it's appropriate to have separate types, CChar/CSChar/CUChar and > > Word8/Int8. > > No, and neither are there any guarantees about there being *any* > specific-width integer types at all! If you're worried about char not > being 8 bit, it's likely that the C compiler won't provide an 8 bit > integer, either, so Word8/Int8 doesn't solve the problem. [snip] There is no reason why a Haskell implementation should not provide Word8 or Int8 even if that is not supported by the underlying hardware. Actually I remember doing something like this for MLj (ML for Java). 8 bit words are not provided by the Java Virtual Machine (or at least, weren't then). However they can be provided fairly cheaply. I think I stored them as chars, but for operations treated them as shorts or ints with extra high junk bits. You need to put in extra work to clear the bits when comparing, or doing right shifts, or dividing, but not for addition, subtraction, multiplication, other bit operations, or loading or storing. So it's pretty cheap. Manual wrote [snip] > As I understand it, in ANSI C, the only freedom that an > implementation has in choosing a concrete representation for > "char" is to decide whether it is signed or unsigned. In > any case, it is going to be an 8 bit entity. [snip] This is false. Look at section 5.2.4.2.1 http://std.dkuug.dk/JTC1/SC22/WG14/www/docs/n843.htm This says only "Their implementation-defined values shall be equal or greater in magnitude (absolute value) to those shown, with the same sign." Thus CHAR_BIT must be at least 8. It may be greater than 8. Also see section 5.2.1. This is the C revision, but I am very sure it was the same with ANSI C.

2 1

Re: Text in Haskell: A PROPOSAL
by Ashley Yakeley 08 Aug '02

08 Aug '02

At 2002-08-08 02:54, Ketil Z. Malde wrote: >Would it be sufficient to have "raw" socket/file functions using >[Word8], This would certainly be a good thing. >and let the "standard" functions (e.g. readFile) convert to >[Char] according to current locale settings? The notion of "current locale settings" (including newline conventions) bothers me. I'd like my Haskell program to do the same thing regardless of which machine executes it -- particularly these days when files get shared around a lot. Do we really need "text mode" anymore? > With, perhaps, UTF-8 as a reasonable default? Perhaps it should _always_ be UTF-8? Or is that too slow in some cases? It certainly raises "seek" issues as one Char codepoint may be represented by several octets. >(And of course, en/decoding functions readily available for manual >use) Yes. -- Ashley Yakeley, Seattle WA

4 3

Re: UTF-8 library
by Ashley Yakeley 08 Aug '02

08 Aug '02

At 2002-08-07 02:43, Axel Simon wrote: >But the point was that C might have different sized characters and that >these functions would still be portable even if the size of CChar changes. Text encoded with ISO 8859-1 or UTF-8 is octets. If you want to use CChars, you should then subsequently convert the Word8s into CChars. -- Ashley Yakeley, Seattle WA

6 10

Re: UTF-8 library
by Ashley Yakeley 08 Aug '02

08 Aug '02

At 2002-08-06 05:38, John Meacham wrote: >One major nit I have with this is the type signature of >decodeUTF8 and encodeUTF8 >a String should always represent a string of characters, not a byte >stream, the signatures should be > >decodeUTF8 :: String -> [Word8] >encodeUTF8 :: [Word8] -> String I think you mean encodeUTF8 :: String -> [Word8] decodeUTF8 :: [Word8] -> String ...or even decodeUTF8 :: [Word8] -> Maybe String It might also be useful to have stream functions. Decoding UTF8 octets is a kind of parsing, after all. But yes, you're right. A Char is a Unicode codepoint, nothing else, and certainly not a C 'char'. A C char is _usually_ a Word8 or an Int8, but not necessarily IIRC. I've always thought it a bit odd that the well-specified types Word8, Int8 etc. are hidden away in a package while the machine-dependent Int type, which I avoid in all my code, is in the Prelude. -- Ashley Yakeley, Seattle WA

2 1

Re: UTF-8 library
by Ashley Yakeley 08 Aug '02

08 Aug '02

At 2002-08-08 02:56, Patryk Zadarnowski wrote: >1990 ANSI/ISO C requires chars to be at least 8 bits wide in section >5.2.4.2.1. This extends to ANSI/ISO C++, which cites ISO C for its >definition of <limits.h>. Haven't got C'99 handy, but it'll be a >similar story - char is *at least* 8 bits wide. > >Hope that satisfies the pedants. But there's no guarantee that char is exactly 8 bits wide, is there? So it's appropriate to have separate types, CChar/CSChar/CUChar and Word8/Int8. I submit that interfaces to network handles should all use Word8, because the standards all speak unambiguously of "octets". For files it's debatable perhaps, but I strongly lean towards Word8 for any "binary" (i.e. direct) file access. CChar/CSChar/CUChar are for FFI to C and C++. -- Ashley Yakeley, Seattle WA

2 1

Re: UTF-8 library
by Ashley Yakeley 08 Aug '02

08 Aug '02

At 2002-08-08 03:16, Patryk Zadarnowski wrote: >Although, it's beyond me why the C standard should dictate anything >about Haskell, especially the arcane and obscure dark corners of it >such as the sizes of integer types. No Haskell compilers compile to >pure ISO C anyway. We need the types precisely for FFI to C, and IMO no other reason. Even then, FFI to functions using fixed-size types should use WordN/IntN types. The rest of Haskell shouldn't depend on it... although I suspect the "Int" type (which I avoid) corresponds to C "int" -- something far more likely to vary than "char". -- Ashley Yakeley, Seattle WA

1 0

Re: Text in Haskell: A PROPOSAL
by Ashley Yakeley 08 Aug '02

08 Aug '02

At 2002-08-07 11:05, Ken Shan wrote: >Let me clarify my understanding of this point a bit further. On the one >hand, GHC uses Char to mean a 32-bit value like a Unicode code point. No, GHC uses Char to mean a Unicode codepoint. These are not 32-bit. It only allows the 17 pages i.e. values in the range '\x0' to '\x10FFFF'. This is the Right Thing as per Unicode 3.1 and later (current is 3.2.0). >On the other hand, GHC uses Char to mean what files store and sockets >transmit and foreign functions process under the C type "char". Right, and this is a very bad idea. The file IO functions should be using Word8s >These two uses are inconsistent, and must be separated. I agree. At 2002-08-07 09:53, Ken Shan wrote: >I have a stake in using Haskell for international text processing: In >particular, I have been writing Haskell code that typeset international >text. Let me summarize what I think are the basic types of data that >need to be distinguished and processed *somehow* within a Haskell >program: > > (1) chars in C (perhaps distinguishing between unsigned, signed, and > default) > (2) 8-bit integers (i.e., signed) and words (i.e., unsigned) > (3) Unicode code values (16-bit) I think the whole 16-bit code value thing was dropped as of 3.1. UTF-16 uses 16-bit values to represent text just as UTF-8 uses 8-bit values. >The conflict in the present discussion arises from two desires: One, to >use Char as 1 above, for FFI convenience and quick-and-dirty code. Two, >to use Char as 4 above, for international text processing and conceptual >correctness. > >I believe that we need library functions to: > > (a) Convert between 1 and 2, or more generally, convert between 1 and > Integral types; > (b) Convert between 2 and 4, under a specified encoding such as > ISO-8859-1 or UTF-8; > (c) Convert between 3 and 4, according to the Unicode standard. You mean according to UTF-16. >My proposal involves the following types: > > (1) Represent char in C as Char, and zero-terminated strings (char*) > in C as CString. We already have the CChar type that means that. > (2) Represent 8-bit integers and words as Int8 and Word8. Agreed. > (3) Represent Unicode code values as Word16, or a new Haskell type > CodeValue. I don't think it's appropriate to have a new type. UTF-16 is a way of representing codepoints as 16-bit integers. The UTF-16 functions should use Word16. > (4) Represent Unicode code points as Word32, or a new Haskell type > CodePoint. We already have the Char type that means that -- in GHC, at least. String and character literals in programs should use it. -- Ashley Yakeley, Seattle WA

6 6

Re: Text in Haskell: A PROPOSAL
by Ashley Yakeley 08 Aug '02

08 Aug '02

At 2002-08-07 17:37, Sven Moritz Hallberg wrote: >Hm, what about encodeCharUTF16? Would that return Word16s? UTF-16 may represent a single Char as one or two Word16s. encodeCharUTF16 :: Char -> [Word16]; or encodeCharUTF16 :: Char -> (Word16,Maybe Word16); or encodeCharUTF16 :: Char -> Either Word16 (Word16,Word16); >Hrm. But then, how to write that to a file? Depends on what order you want the halves of each Word16. Unicode 3.0 defines four character encoding forms/schemes: UTF-8, UTF-16, UTF-16LE, and UTF-16BE. UTF-16 encodes as 16-bit units, the other three encode as 8-bit units. So you might have something like this: encodeUTF8 :: String -> [Word8]; encodeUTF16 :: String -> [Word16]; encodeUTF16LE :: String -> [Word8]; encodeUTF16BE :: String -> [Word8]; The authority here is Unicode Technical Report 17, which is part of the Unicode Standard. <http://www.unicode.org/unicode/reports/tr17/> But watch out... I've noticed a certain amount of incoherence in the Unicode standards, for instance Unicode 3.0 sec. 2.3 refers to UTF-16 one of four "Character Encoding Schemes" which is "an encoding form plus byte serialization" even though UTF-16 by itself doesn't include byte serialization. -- Ashley Yakeley, Seattle WA

1 0

Re: Text in Haskell: A PROPOSAL
by Ashley Yakeley 08 Aug '02

08 Aug '02

At 2002-08-07 21:26, Ken Shan wrote: >One thing I am concerned about is that getChar should be IO CChar, not >IO Char. No, it should be IO Word8, because strictly speaking a C char doesn't have to be an octet. CChar should be the machine-dependent type. Network handles should certainly be octets. And I think we can assume file handles are too... -- Ashley Yakeley, Seattle WA

1 0

RE: OT: haskell, .net, microsoft and vs
by Andrew Kennedy 08 Aug '02

08 Aug '02

> -----Original Message----- > From: Nigel Perry [mailto:NPerry@mac.com] > Sent: Tuesday, August 06, 2002 7:44 PM > To: Haskell Mailing List > Subject: Re: OT: haskell, .net, microsoft and vs ... > As part of Project 7, a Microsoft project with various language > developers during the development of .NET a number of languages added > VS support, ranging from simple syntax colouring and "makefile" > project support to full integration. Mondrian, the simple functional > scripting language, was one of these which I undertook, it's at the > simple end. However I did not also add support for Haskell (I did a > partial port [not speed optimised and many libraries missing] of GHC > to .NET using Mondrian as a backend). Other languages integrated > include Fortran (Salford), Cobol (Fujitsu) and Component Pascal (QUT). > Work-in-progress: VS integration of SML.NET (http://www.cl.cam.ac.uk/Research/TSG/SMLNET/) We use COM interop and a bootstrapped (subset) compiler to get syntax highlighting, incremental parsing and type inference, intellisense of SML and .NET library functions, and build support. Also some support for debugging (breakpoints, stack trace, locals). We hope to release this implementation at some point. - Andrew.

3 2