At 2002-08-08 02:28, Manuel M T Chakravarty wrote:
ANSI C guarantees that char is 1 byte (more precisely that "sizeof (char)" == 1).
That's also what the C++ ARM says (which I have to hand). Unfortunately, "a byte is undefined by the language except in terms of sizeof; sizeof(char) is 1." [sec. 5.3.2] Maybe ANSI C is different? -- Ashley Yakeley, Seattle WA
On Thu, 8 Aug 2002, Ashley Yakeley wrote:
At 2002-08-08 02:28, Manuel M T Chakravarty wrote:
ANSI C guarantees that char is 1 byte (more precisely that "sizeof (char)" == 1).
That's also what the C++ ARM says (which I have to hand). Unfortunately,
"a byte is undefined by the language except in terms of sizeof; sizeof(char) is 1." [sec. 5.3.2]
Maybe ANSI C is different?
1990 ANSI/ISO C requires chars to be at least 8 bits wide in section 5.2.4.2.1. This extends to ANSI/ISO C++, which cites ISO C for its definition of <limits.h>. Haven't got C'99 handy, but it'll be a similar story - char is *at least* 8 bits wide. Hope that satisfies the pedants. Pat.
Ashley Yakeley <ashley@semantic.org> wrote,
At 2002-08-08 02:28, Manuel M T Chakravarty wrote:
ANSI C guarantees that char is 1 byte (more precisely that "sizeof (char)" == 1).
That's also what the C++ ARM says (which I have to hand). Unfortunately,
"a byte is undefined by the language except in terms of sizeof; sizeof(char) is 1." [sec. 5.3.2]
Maybe ANSI C is different?
As I understand it, in ANSI C, the only freedom that an implementation has in choosing a concrete representation for "char" is to decide whether it is signed or unsigned. In any case, it is going to be an 8 bit entity. Manuel
tor 2002-08-08 klockan 14.18 skrev Manuel M T Chakravarty:
Ashley Yakeley <ashley@semantic.org> wrote,
At 2002-08-08 02:28, Manuel M T Chakravarty wrote:
ANSI C guarantees that char is 1 byte (more precisely that "sizeof (char)" == 1).
That's also what the C++ ARM says (which I have to hand). Unfortunately,
"a byte is undefined by the language except in terms of sizeof; sizeof(char) is 1." [sec. 5.3.2]
Maybe ANSI C is different?
As I understand it, in ANSI C, the only freedom that an implementation has in choosing a concrete representation for "char" is to decide whether it is signed or unsigned. In any case, it is going to be an 8 bit entity.
No, ANSI C just says that sizeof measures other things in chars. So sizeof(char) is always 1, but 1 could mean 8, 9, 16 or 17 bits depending on the architecture. However, I've yet to see an architecture where a c char is not 8 bits, and I doubt that there ever will be. So assuming char = 8 bits is not going to make things any worse, since it's already implicitly assumed in many places. Anyway, UTF-8 is as stated before an octet stream, and so, the natural choice would be to have UTF-8 encoded text as [Word8]. putChar (and putStr) should output UTF-8 text if the locale is UTF-8, and getChar (and getLine) should input UTF-8 text if the locale is UTF-8. This is the only implication you can make based on the fact that a Char is a unicode character (not iso-8859-1, not ASCII). There rarely should be a need to handle UTF-8 text internally in Haskell, but for FFI it would be neccessary. Using locale automatically there is wrong, since gtk2 uses UTF-8 always, and other interfaces uses iso-8859-1 always. However, having some conversion functions could never hurt, but they need not be in the FFI. Regards, Martin -- Martin Norbäck d95mback@dtek.chalmers.se Kapplandsgatan 40 +46 (0)708 26 33 60 S-414 78 GÖTEBORG http://www.dtek.chalmers.se/~d95mback/ SWEDEN OpenPGP ID: 3FA8580B
On Thu, Aug 08, 2002 at 02:58:42PM +0200, Martin Norbck wrote:
There rarely should be a need to handle UTF-8 text internally in Haskell, but for FFI it would be neccessary. Using locale automatically there is wrong, since gtk2 uses UTF-8 always, and other interfaces uses iso-8859-1 always. However, having some conversion functions could never hurt, but they need not be in the FFI.
Sure it is necessary to explicitly state what kind of conversion you want when you talk to your C library. So IMHO the only question that really remains is whether readFile, withCString and all other standard I/O function should assume UTF-8 by default or the current locale. I opt for the latter. Axel.
tor 2002-08-08 klockan 15.12 skrev Axel Simon:
On Thu, Aug 08, 2002 at 02:58:42PM +0200, Martin Norbck wrote:
There rarely should be a need to handle UTF-8 text internally in Haskell, but for FFI it would be neccessary. Using locale automatically there is wrong, since gtk2 uses UTF-8 always, and other interfaces uses iso-8859-1 always. However, having some conversion functions could never hurt, but they need not be in the FFI.
Sure it is necessary to explicitly state what kind of conversion you want when you talk to your C library. So IMHO the only question that really remains is whether readFile, withCString and all other standard I/O function should assume UTF-8 by default or the current locale. I opt for the latter.
Since readFile and writeFile handles Strings, they must be locale dependent. Should they fall back to the most common 8-bit format (iso-8859-1) or throw an excaption when encountering illegal multi-byte sequences? Regards, Martin -- Martin Norbäck d95mback@dtek.chalmers.se Kapplandsgatan 40 +46 (0)708 26 33 60 S-414 78 GÖTEBORG http://www.dtek.chalmers.se/~d95mback/ SWEDEN OpenPGP ID: 3FA8580B
Axel Simon wrote:
Sure it is necessary to explicitly state what kind of conversion you want when you talk to your C library. So IMHO the only question that really remains is whether readFile, withCString and all other standard I/O function should assume UTF-8 by default or the current locale. I opt for the latter.
I suggest ISO-8859-1, as 1. Assuming UTF-8 will result in errors when trying to read 8-bit data which isn't actually UTF-8. 2. There's a lot more ISO-8859-1 data in existence than UTF-8. Actually, there are quite a lot of encodings which are more popular in the real world than UTF-8. 3. Unicode code points 0..255 correspond to ISO-8859-1. 4. The current locale doesn't tell you anything about the actual encoding of most of the data streams (files, network connections) which you are likely to process. Note that we're not discussing solutions here, but workarounds. The only actual "solution" would be to redesign Haskell's I/O and string handling libraries from scratch without pretending that the octet/byte/character distinctions can be glossed over. -- Glynn Clements <glynn.clements@virgin.net>
--- Axel Simon <A.Simon@ukc.ac.uk> wrote:
Sure it is necessary to explicitly state what kind of conversion you want when you talk to your C library. So IMHO the only question that really remains is whether readFile, withCString and all other standard I/O function should assume UTF-8 by default or the current locale. I opt for the latter.
Dependence on the current locale is EXTREMELY inconvenient. Imagine that you're writing a Web browser. Your program should be able to read from many octet streams (=connections) each of which can use its own encoding. How would you switch locales in this situation? Having a locale associated with each individual stream is much more convenient. Incidentally, this is what C++ does. Presumably the C++ library designers have learned from mistakes of their predecessors :) -- a. __________________________________________________ Do You Yahoo!? HotJobs - Search Thousands of New Jobs http://www.hotjobs.com
On Thu, Aug 08, 2002 at 09:26:02AM -0700, anatoli wrote:
--- Axel Simon <A.Simon@ukc.ac.uk> wrote:
Sure it is necessary to explicitly state what kind of conversion you want when you talk to your C library. So IMHO the only question that really remains is whether readFile, withCString and all other standard I/O function should assume UTF-8 by default or the current locale. I opt for the latter.
Dependence on the current locale is EXTREMELY inconvenient. I am only talking about the default behaviour of readFile which should be locale dependant IMHO. If you know you are reading a Latin-9 or stream you will probably say something like do content <- liftM octetToLatin9 $ readOctetFile "foo"
Axel.
--- Axel Simon <A.Simon@ukc.ac.uk> wrote:
I am only talking about the default behaviour of readFile which should be locale dependant IMHO.
This makes sense.
If you know you are reading a Latin-9 or stream you will probably say something like do content <- liftM octetToLatin9 $ readOctetFile "foo"
I'd still rather associate locale with a handle. This way, all Char and String IO functions that exist, and those that are not written yet, can work with any encoding without relying on the abomination that is setlocale(). We do have quite a few of them besides readFile, and making "octet" versions of all of them is Not A Good Thing. -- a. __________________________________________________ Do You Yahoo!? HotJobs - Search Thousands of New Jobs http://www.hotjobs.com
anatoli wrote:
I'd still rather associate locale with a handle. This way, all Char and String IO functions that exist, and those that are not written yet, can work with any encoding without relying on the abomination that is setlocale().
Seconded; this is the best approach. The libc locale could be consulted to determine the initial or default encoding, or it could just be ignored (I'd vote to ignore it; setlocale() *is* an abomination.) BTW, this is how Tcl does it -- each file handle has an associated encoding (which may be changed on the fly) -- and it's very convenient. --Joe English jenglish@flightlab.com
anatoli <anatoli@yahoo.com> writes:
Dependence on the current locale is EXTREMELY inconvenient. Imagine that you're writing a Web browser.
Web browsers get input with MIME declarations, and shouldn't rely on *any* default setting. Instead, they should read [Word8] and decode the contents according to Content-Type/Content-Transfer-Encoding. -kzm -- If I haven't seen further, it is by standing in the footprints of giants
On Thu, 2002-08-08 at 18:26, anatoli wrote:
Having a locale associated with each individual stream is much more convenient.
I argue _strongly_ against associating some sort of locale state with handles. 1) In agreement with Ashley's statements, file IO should use octets, because that's what's in a file. 2) If you need to decode those octets to characters, or vice-versa, compose a (de)serialization function before it. 3) A "best shot" character reading(or writing, for that matter) function, will be convenient. This should probably use your current locale, because when writing a character, you'll probably want to be able to write your own language's characters correctly. 4) For decoding, we'll need some parsing functionality, as someone already mentioned. With that we can have functions like parseUTF8. "Associating a locale with a stream", as you put it, is a matter of, if f is the raw Word8 stream, g = parseUTF8 f, where g is the Char stream, parsed as UTF-8-encoded characters from f. Sven Moritz
--- Sven Moritz Hallberg <pesco@gmx.de> wrote:
I argue _strongly_ against associating some sort of locale state with handles.
1) In agreement with Ashley's statements, file IO should use octets, because that's what's in a file.
By the same token, we should handle CR/LF/CR-LF/LF-CR mess by hand. (Files don't have lines in them, they are just sequences of octets.) I prefer somewhat higher-level view of files.
2) If you need to decode those octets to characters, or vice-versa, compose a (de)serialization function before it.
I *always* need that. (Except for binary IO). Might as well have this functionality built in a handle.
3) A "best shot" character reading(or writing, for that matter) function, will be convenient. This should probably use your current locale, because when writing a character, you'll probably want to be able to write your own language's characters correctly.
I routinely read and write messages in three different languages that use three different encodings. All of them are my "own" languages.
4) For decoding, we'll need some parsing functionality, as someone already mentioned. With that we can have functions like parseUTF8. "Associating a locale with a stream", as you put it, is a matter of, if f is the raw Word8 stream, g = parseUTF8 f, where g is the Char stream, parsed as UTF-8-encoded characters from f.
A "Word8 stream" can be either Handle (Word8Handle?) or [Word8]. We can transform [Word8] to [Char], but not Word8Handle to CharHandle. I argue that the latter is needed as well. -- a. __________________________________________________ Do You Yahoo!? HotJobs - Search Thousands of New Jobs http://www.hotjobs.com
On Sat, 2002-08-10 at 12:03, anatoli wrote:
--- Sven Moritz Hallberg <pesco@gmx.de> wrote:
I argue _strongly_ against associating some sort of locale state with handles.
1) In agreement with Ashley's statements, file IO should use octets, because that's what's in a file.
By the same token, we should handle CR/LF/CR-LF/LF-CR mess by hand. (Files don't have lines in them, they are just sequences of octets.)
That's a good point, I've forgotten about this mess. I think that it's ugly, though, to do it somewhere outside, pretending the issue's not there. I value about Haskell it's clean representation of reality. Attaching all kinds of state to handles just isn't as clear as "Look here, a file: It's a sequence of octets.", "Watch out though, each file can use an entirely different encoding.", "The Char versions of the IO functions will try to deal with encoding for you.", and "If you know you need some special treatment, we have these functions blahblahblah..."
I prefer somewhat higher-level view of files.
Of course, so do I, I just want the higher-level view to be implemented in Haskell, not under the hood of some ominous "handle" type; which, btw, will then no longer be simply a handle but some sort of great big file IO "object". That's confusing for anyone who hasn't been exposed to the C way of dealing with files. I'd teach some old people clean concepts they might not be used to, rather than repeating the same old yuck to every new little programmer who's just starting.
2) If you need to decode those octets to characters, or vice-versa, compose a (de)serialization function before it.
I *always* need that. (Except for binary IO). Might as well have this functionality built in a handle.
Well, then *always* use the Char functions. I don't see the point.
3) A "best shot" character reading(or writing, for that matter) function, will be convenient. This should probably use your current locale, because when writing a character, you'll probably want to be able to write your own language's characters correctly.
I routinely read and write messages in three different languages that use three different encodings. All of them are my "own" languages.
Where is the problem? The system is not going to be able to decide which one to use either way, so you must make the encoding explicit. Now we just have to come up with a convenient way to do it. Transforming between [Word8] and [Char] seems plausible to me.
4) For decoding, we'll need some parsing functionality, as someone already mentioned. With that we can have functions like parseUTF8. "Associating a locale with a stream", as you put it, is a matter of, if f is the raw Word8 stream, g = parseUTF8 f, where g is the Char stream, parsed as UTF-8-encoded characters from f.
A "Word8 stream" can be either Handle (Word8Handle?) or [Word8]. We can transform [Word8] to [Char], but not Word8Handle to CharHandle. I argue that the latter is needed as well.
The only reason for that would be efficiency. Simon said something about that. I admit that I have no clue about it. Sven Moritz
[apologies if you see multiple copies; I forgot to Cc: the list the first time around.] --- Sven Moritz Hallberg <pesco@gmx.de> wrote:
[...] I think that it's ugly, though, to do it somewhere outside, pretending the issue's not there. I value about Haskell it's clean representation of reality. Attaching all kinds of state to handles just isn't as clear as "Look here, a file: It's a sequence of octets.", "Watch out though, each file can use an entirely different encoding.", "The Char versions of the IO functions will try to deal with encoding for you.", and "If you know you need some special treatment, we have these functions blahblahblah..."
As I view it, a Handle is always a stream of Char data. Why? Simply because Haskell threats Handles as streams of Char data *today*. There's no good reason to change that, unless you want to wheak havoc in existing programs. To make things i18n-friendly, the simplest (IMHO) approach is to declare that under each Hadle (i.e. Char stream) there is a BinaryHandle (i.e. Word8 stream) *plus* an associated encoding (and also maybe CR/LF handler while we're at it). I certainly don't want the same Handle type to be able to represent a sequence of octets and a sequence of Char at the same time.
I routinely read and write messages in three different languages that use three different encodings. All of them are my "own" languages.
Where is the problem? The system is not going to be able to decide which one to use either way, so you must make the encoding explicit. Now we just have to come up with a convenient way to do it. Transforming between [Word8] and [Char] seems plausible to me.
I want to be able to specify encoding explicitly *and* be able to use existing Char IO, because that's what my programs use *today* and I don't want to rework them. Rewriting all my IO because it's now Word8-based instead of Char-based is NOT convenient.
A "Word8 stream" can be either Handle (Word8Handle?) or [Word8]. We can transform [Word8] to [Char], but not Word8Handle to CharHandle. I argue that the latter is needed as well.
The only reason for that would be efficiency. Simon said something about that. I admit that I have no clue about it.
What about backward compatibility? With my approach, in order to make a Haskell program i18n-aware, you only need to change a few calls to openFile and make them openFileWithEncoding. Otherwise they will just use default encoding. -- a. __________________________________________________ Do You Yahoo!? HotJobs - Search Thousands of New Jobs http://www.hotjobs.com
Martin Norbäck <d95mback@dtek.chalmers.se> writes:
However, I've yet to see an architecture where a c char is not 8 bits, and I doubt that there ever will be.
I'm porting a C compiler to a machine where char is 9 bits (and int is 36 bits). There was a time when a 36-bit word size was the norm, not an exception.
Lars Brinkhoff wrote:
Martin Norbäck <d95mback@dtek.chalmers.se> writes:
However, I've yet to see an architecture where a c char is not 8 bits, and I doubt that there ever will be.
I'm porting a C compiler to a machine where char is 9 bits (and int is 36 bits). There was a time when a 36-bit word size was the norm, not an exception.
Doesn't sound like a very Haskell-related post though ;-) Maybe it's time to move this thread to another list? Cheers, Janis. -- Janis Voigtlaender http://wwwtcs.inf.tu-dresden.de/~voigt/ mailto:voigt@tcs.inf.tu-dresden.de
participants (12)
-
anatoli -
Ashley Yakeley -
Axel Simon -
Glynn Clements -
Janis Voigtlaender -
Joe English -
ketil＠ii.uib.no -
Lars Brinkhoff -
Manuel M T Chakravarty -
Martin Norbäck -
Patryk Zadarnowski -
Sven Moritz Hallberg