At 2002-08-08 02:54, Ketil Z. Malde wrote:
Would it be sufficient to have "raw" socket/file functions using [Word8],
This would certainly be a good thing.
and let the "standard" functions (e.g. readFile) convert to [Char] according to current locale settings?
The notion of "current locale settings" (including newline conventions) bothers me. I'd like my Haskell program to do the same thing regardless of which machine executes it -- particularly these days when files get shared around a lot. Do we really need "text mode" anymore?
With, perhaps, UTF-8 as a reasonable default?
Perhaps it should _always_ be UTF-8? Or is that too slow in some cases? It certainly raises "seek" issues as one Char codepoint may be represented by several octets.
(And of course, en/decoding functions readily available for manual use)
Yes. -- Ashley Yakeley, Seattle WA
On Thu, Aug 08, 2002 at 03:16:09AM -0700, Ashley Yakeley wrote:
At 2002-08-08 02:54, Ketil Z. Malde wrote:
and let the "standard" functions (e.g. readFile) convert to [Char] according to current locale settings?
The notion of "current locale settings" (including newline conventions) bothers me. I'd like my Haskell program to do the same thing regardless of which machine executes it -- particularly these days when files get shared around a lot. But you can't make a problem go away by pretending it isn't there. If you have a file in a different encoding than your current locale, at least you can change the locale and then run your Haskell program on that file. I think default encoding should depend on the current locale and some clever "guess" functionality which switches to UTF-8 or Unicode when it sees the magic characters at the beginning of a file. I think it is important that the representation _within_ Haskell is well-defined (i.e. Unicode code points with \n as newline).
Do we really need "text mode" anymore? What do you mean?
With, perhaps, UTF-8 as a reasonable default?
Perhaps it should _always_ be UTF-8? Or is that too slow in some cases? It certainly raises "seek" issues as one Char codepoint may be represented by several octets. Luckily "seek" is not a problem with readFile and most other common functions. It's only hSeek :: Handle -> SeekMode -> Integer -> IO () which then should get the remark "does only work reliably with hGetOctet but not with hGetChar".
Axel.
Ashley Yakeley <ashley@semantic.org> writes:
The notion of "current locale settings" (including newline conventions) bothers me.
Me too. But if we wish to support files stored in different formats (e.g. ISO-8859-1 is the standard here) I don't see how we can avoid it.
Do we really need "text mode" anymore?
I don't follow you?
With, perhaps, UTF-8 as a reasonable default?
Perhaps it should _always_ be UTF-8? Or is that too slow in some cases?
I don't think speed is much of an issue, IO is generally slow compared to processing, and I suspect speed critical applications might want to use Word8 anyway. The seek issues are...well, issues. I'd suggest restricting seeking to handles working on Word8, since that is where it makes sense and is easy to implement. I wonder if anybody are actually *using* non-octet based encodings (e.g. UTF-16/UCS-2) in files or in sockets (without wrapping the encoded content in a higher level protocol, like MIME)? Even if various standards support them, we might be better off with less complexity and handling the *useful* cases, if it turns out the complex cases aren't real world. IMHO, UTF-8 is a good compromise, it will break ISO-8859 encodings, but only here and there, as the 7-bit ASCII characters usually make up the bulk of the data. -kzm -- If I haven't seen further, it is by standing in the footprints of giants
On Thursday, 2002-08-08, 13:05, CEST, Ketil Z. Malde wrote:
I wonder if anybody are actually *using* non-octet based encodings (e.g. UTF-16/UCS-2) in files or in sockets (without wrapping the encoded content in a higher level protocol, like MIME)? Even if various standards support them, we might be better off with less complexity and handling the *useful* cases, if it turns out the complex cases aren't real world.
I would say, dealing with a character encoding _scheme_*) like UTF-16LE or UTF-16BE is as complex as dealing with any other encoding scheme. And since we may assume that files and sockets work with octets, it makes no sense to provide support for non-octet based encoding _forms_ like UTF-16 in this area. All one has to provide for such forms is, IMHO, some conversion functions/parsers.
[...]
Wolfgang *) The Unicode Standard (at least 3.0) makes a distinction between character encoding forms and character encoding schemes. Character encoding forms specify the representation of characters as actual data in a computer. The Unicode Standard uses two encoding forms: 16-bit and 8-bit [i.e. UTF-16 and UTF-8]. --- The Unicode Standard 3.0, section 2.3 A character encoding scheme consists of an encoding form plus byte serialization. --- The Unicode Standard 3.0, section 2.3
participants (4)
-
Ashley Yakeley -
Axel Simon -
ketil＠ii.uib.no -
Wolfgang Jeltsch