2011/4/4 Colin Adams :

Not from looking with your eyes perhaps. Does that matter? Your text editor, and the compiler, can surely figure it out for themselves. I am not aware of any algorithm that can reliably infer the character encoding used by just looking at the raw data. Why would people bother with stuff like <?xml version="1.0" encoding="UTF-8"?> if automatically figuring out the encoding was easy?

There aren't many Unicode encoding formats From casually scanning some articles about encodings I can count at least 70 character encodings [1].

and there aren't very many possibilities for the leading characters of a Haskell source file, are there? Since a Haskell program is a sequence of Unicode code points the programmer can choose from up to 1,112,064 characters. Many of these can legitimately be part of the interface of a module, as function names, operators or names of types.

[1] - http://en.wikipedia.org/wiki/Character_encoding

On 4 April 2011 12:22, Michael Snoyman wrote:

Firstly, I personally would love to insist on using UTF-8 and be done with it. I see no reason to bother with other character encodings.

This is also my preferred choice.

There *is* an algorithm for determining the encoding of an XML file based on a combination of the BOM (Byte Order Marker) and an assumption that the file will start with a XML declaration (i.e., <?xml ... ?>). But this isn't capable of determining every possible encoding on the planet, just distinguishing amongst varieties of UTF-(8|16|32)/(big|little) endian and EBCIDC. It cannot tell the difference between UTF-8, Latin-1, and Windows-1255 (Hebrew), for example.

I think I was confused between character encodings in general and Unicode encodings.

I think the implication of "Unicode encoding formats" is something in the UTF family. An encoding like Latin-1 or Windows-1255 can be losslessly translated into Unicode codepoints, but it's not exactly an encoding of Unicode, but rather a subset of Unicode.

That would validate Colin's point about there not being that many encodings.

My guess is that a large subset of Haskell modules start with one of left brace (starting with comment or language pragma), m (for starting with module), or some whitespace character. So it *might* be feasible to take a guess at things. But as I said before: I like UTF-8. Is there anyone out there who has a compelling reason for writing their Haskell source in EBCDIC?

I think I misinterpreted the word 'leading'. I thought Colin meant "most used". The set of characters with which Haskell programmes start is indeed small. But like you I prefer no guessing and just default to UTF-8.

Michael Snoyman writes:

My guess is that a large subset of Haskell modules start with one of left brace (starting with comment or language pragma), m (for starting with module), or some whitespace character. So it *might* be feasible to take a guess at things. But as I said before: I like UTF-8. Is there anyone out there who has a compelling reason for writing their Haskell source in EBCDIC?

Probably not EBCDIC. :-) Correct me if I'm wrong here, but I think nobody has compelling reasons for using any other Unicode format than UTF-8. Although some systems use UTF-16 (or some approximation thereof) internally, UTF-8 seems to be the universal choice external encoding. However, there probably exists a bit of code using Latin-1 and Windows charsets, and here leading characters aren't going to help you all that much. I think the safest thing to do is to require source to be ASCII, and provide escapes for code points >127... -k -- If I haven't seen further, it is by standing in the footprints of giants

2011/4/4 Ketil Malde :

I think the safest thing to do is to require source to be ASCII, and provide escapes for code points >127...

I do not think that that is the safest option. The safest is just writing down whatever GHC does. Escape codes for non-ASCII would break a lot of packages and make programming really painful. Consider the following, utf-8 encoded, file: http://code.haskell.org/numerals/test/Text/Numeral/Language/ZH/TestData.hs I don't want to imagine writing that with escape characters. It would also be very error prone, not being able to readily read what you write. But the overall consensus appears to be UTF-8 as the default encoding. I will write an official proposal to amend the haskell language specification. (Probably this evening, utc+1).

2011/4/4 Brandon Moore

The rest of the standard already just talks about sequences of unicode characters, so I don't see much to be gained by prohibiting other encodings.

In particular, I have read that systems set up for east asian scripts often use UTF-16 as a default encoding.

Presumably because this will use less disk space on average. I too don't see any reason to forbid other Unicode encodings. Perhaps mandate support for UTF-8, and allow others with a pragma. But unless someone adds support to a Haskell compiler for such a pragma, it will be fairly pointless putting this in the standard. -- Colin Adams Preston, Lancashire, ENGLAND () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments

+1 for UTF-8 only. Brandon Moore wrote:

...I don't see much to be gained by prohibiting other encodings.

Universal portability of Haskell source code with respect to its encoding is to be gained. We can achieve that simplicity now with almost no cost. Why miss the opportunity?

In particular, I have read that systems set up for east asian scripts often use UTF-16 as a default encoding.

Default encoding is not an issue for any normal source code editing tool. Thanks, Yitz

On 4/04/2011, at 10:24 PM, Daniel Fischer wrote:

Colin spoke of *leading* characters, for .hs files, that drastically reduces the possibilities - not for .lhs, though.

A .hs file can, amongst other things, begin with any "small" letter.

On 5 April 2011 10:35, Daniel Fischer wrote:

On Tuesday 05 April 2011 04:35:39, Richard O'Keefe wrote:

...

On 4/04/2011, at 10:24 PM, Daniel Fischer wrote:

...

Colin spoke of *leading* characters, for .hs files, that drastically reduces the possibilities - not for .lhs, though.

A .hs file can, amongst other things, begin with any "small" letter.

D'oh, yes, I always forget that a module declaration isn't required.

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

True, but we could say that UTF-8 is complusory in the absence of a module declaration. -- Colin Adams Preston, Lancashire, ENGLAND () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments

On Mon, 4 Apr 2011 13:30:08 +0100, you wrote:

Windows APIs use UTF-16...

The newer ones, at least. The older ones usually come in two flavors, UTF-16LE and 8-bit code page-based.

...but the encoding of files (which is the relevant point here) is almost uniformly UTF-8 - though of course you can find legacy apps making other choices.

If you're talking about files written and read by the operating system itself, then perhaps. But my experience is that there are a lot of applications that use UTF-16LE, especially ones that typically only work with smaller files (configuration files, etc.). As for Haskell, I would still vote for UTF-8 only, though. The only reason to favor anything else is legacy compatibility with existing Haskell source files, and that isn't really an issue here. -Steve Schafer

BOM is not part of UTF8, because UTF8 is byte-oriented.  But applications should be prepared to read and discard it, because some applications erroneously generate it. Regards, Malcolm On 04 Apr, 2011,at 02:09 PM, Antoine Latter wrote: On Mon, Apr 4, 2011 at 7:30 AM, Max Bolingbroke wrote:

On 4 April 2011 11:34, Daniel Fischer wrote:

...

If there's only a single encoding recognised, UTF-8 surely should be the one (though perhaps Windows users might disagree, iirc, Windows uses UCS2 as standard encoding).

Windows APIs use UTF-16, but the encoding of files (which is the relevant point here) is almost uniformly UTF-8 - though of course you can find legacy apps making other choices.

Would we need to specifically allow for a Windows-style leading BOM in UTF-8 documents? I can never remember if it is truly a part of UTF-8 or not.

Cheers, Max

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

malcolm.wallace wrote:

...

BOM is not part of UTF8, because UTF8 is byte-oriented.  But applications should be prepared to read and discard it, because some applications erroneously generate it.

For maximum portability, the standard should be require compilers to accept and discard an optional BOM as the first character of a source code file. Tako Schotanus wrote:

That's not what the official unicode site says in its FAQ:  http://unicode.org/faq/utf_bom.html#bom4 and http://unicode.org/faq/utf_bom.html#bom5

That FAQ clearly states that BOM is part of some "protocols". It carefully avoids stating whether it is part of the encoding. It is certainly not erroneous to include the BOM if it is part of the protocol for the applications being used. Applications can include whatever characters they'd like, and they can use whatever handshake mechanism they'd like to agree upon an encoding. The BOM mechanism is common on the Windows platform. It has since appeared in other places as well, but it is certainly not universally adopted. Python supports a pseudo-encoding called "utf8-bom" that automatically generates and discards the BOM in support of that handshake mechanism But it isn't really an encoding, it's a convenience. Part of the source of all this confusion is some documentation that appeared in the past on Microsoft's site which was unclear about the fact that the BOM handshake is a protocol adopted by Microsoft, not a part of the encoding itself. Some people claim that this was intentional, part of the "extend and embrace" tactic Microsoft allegedly employed in those days in an effort to expand its monopoly. The wording of the Unicode FAQ is obviously trying to tip-toe diplomatically around this issue without arousing the ire of either pro-Microsoft or anti-Microsoft developers. Thanks, Yitz

On Mon, Apr 4, 2011 at 3:52 PM, Roel van Dijk wrote:

I made an official proposal on the haskell-prime list:

http://www.haskell.org/pipermail/haskell-prime/2011-April/003368.html

Let's have further discussion there.

I'm not on that mailing list, so I'll comment here: My only caveat is that the encoding provision should apply when Haskell source is presented to the compiler as a bare stream of octets. Where Haskell source is interchanged as a stream of Unicode characters, then encoding is not relevant -- but may be likely governed by some outer protocol - and hence may not be UTF-8 but nonetheless invisible at the Haskell level. Two examples where this might come into play are: 1) An IDE that stores module source in some database. It would not be relevant what encoding that IDE and database choose to store the source in if the source is presented to the integrated compiler as Unicode characters. 2) If a compilation system fetches module source via HTTP (I could imagine a compiler that chased down included modules directly off of Hackage, say), then HTTP already has a mechanism (via MIME types) of transmitting the encoding clearly. As such, there should be no problem if that outer protocol (HTTP) transmits the source to the compiler via some other encoding. There is no reason (and only potential interoperability restrictions) to enforce that UTF-8 be the only legal encoding here.

On Mon, Apr 4, 2011 at 17:51, Yitzchak Gale wrote:

malcolm.wallace wrote:

...

...

BOM is not part of UTF8, because UTF8 is byte-oriented. But applications should be prepared to read and discard it, because some applications erroneously generate it.

For maximum portability, the standard should be require compilers to accept and discard an optional BOM as the first character of a source code file.

Tako Schotanus wrote:

...

That's not what the official unicode site says in its FAQ: http://unicode.org/faq/utf_bom.html#bom4 and http://unicode.org/faq/utf_bom.html#bom5

That FAQ clearly states that BOM is part of some "protocols". It carefully avoids stating whether it is part of the encoding.

It is certainly not erroneous to include the BOM if it is part of the protocol for the applications being used. Applications can include whatever characters they'd like, and they can use whatever handshake mechanism they'd like to agree upon an encoding. The BOM mechanism is common on the Windows platform. It has since appeared in other places as well, but it is certainly not universally adopted.

Python supports a pseudo-encoding called "utf8-bom" that automatically generates and discards the BOM in support of that handshake mechanism But it isn't really an encoding, it's a convenience.

Part of the source of all this confusion is some documentation that appeared in the past on Microsoft's site which was unclear about the fact that the BOM handshake is a protocol adopted by Microsoft, not a part of the encoding itself. Some people claim that this was intentional, part of the "extend and embrace" tactic Microsoft allegedly employed in those days in an effort to expand its monopoly.

The wording of the Unicode FAQ is obviously trying to tip-toe diplomatically around this issue without arousing the ire of either pro-Microsoft or anti-Microsoft developers.

Some reliable sources for all this would be entertaining (although irrelevant for the rest of this discussion). Cheers, -Tako