Hal Daume III wrote:

There's a Binary module that comes with GHC that you can get somewhere (I believe Simon M wrote it). I have hacked it up a bit and added support for bit-based writing, to bring it more in line with the NHC module. Mine, with various information, etc., is available at:

http://www.isi.edu/~hdaume/haskell/NewBinary/

Hmmm, I'm not sure if that is what Ron asked for. What I guess is needed is support for things like: "read the next 4 bytes as a low-endian unsigned integer" "read the next 8 bytes as a big-endian IEEE 754 double" "write the Int16 as a low-endian signed integer" "write the (StorableArray Int Int32) as big-endian signed integers" ... plus perhaps some String I/O with a few encodings. Alas, we do *not* have something in our standard libs, although there were a few discussions about it. I know that one can debate ages about byte orders, external representation of arrays and floats, etc. Nevertheless, I guess covering only low-/big-endian systems, IEEE 754 floats, and arrays as a simple 0-based sequence of its elements (with an explicit length stated somehow) would make at least 90% of all users happy and would be sufficient for most "real world" file formats. Currently one is bound to hGetBuf/hPutBuf, which is not really a comfortable way of doing binary I/O. Cheers, S.

Ron de Bruijn wrote:

Basically, I just want to have a function, that converts a list with zeros and ones to a binary file and the other way around.

If I write 11111111 to a file, it would take 8 bytes. But I want it to take 8 bits.

import Char (digitToInt, chr) import Word (Word8) import System.IO (openBinaryFile) import IO (IOMode(..), hPutStr, hClose) bitsToOctet :: [Char] -> Word8 bitsToOctet ds = fromIntegral $ sum $ zipWith (*) powers digits where powers = [2^n | n <- [7,6..0]] digits = map digitToInt ds octetToChar :: Word8 -> Char octetToChar = chr . fromIntegral bits :: [[Char]] bits = [ "01101000" -- 0x68 'h' , "01100101" -- 0x65 'e' , "01101100" -- 0x6c 'l' , "01101100" -- 0x6c 'l' , "01101111" -- 0x6f 'o' , "00001010" -- 0x0a '\n' ] main :: IO () main = do h <- openBinaryFile "out.dat" WriteMode hPutStr h $ map (octetToChar . bitsToOctet) bits hClose h -- Glynn Clements

Sven Panne wrote:

...

[...] main :: IO () main = do h <- openBinaryFile "out.dat" WriteMode hPutStr h $ map (octetToChar . bitsToOctet) bits hClose h

Hmmm, using string I/O when one really wants to do binary I/O gives me a bad feeling. Haskell characters are defined to be Unicode characters, so the above only works because current Haskell implementations usually get this wrong (either no Unicode support at all and/or ignoring any encodings and doing I/O only with the lower 8 bits of the characters)... hGetBuf/hPutBuf plus their non-blocking variants are the only way to *really* do binary I/O currently.

Which is unfortunate, because of the requirement to pass a Ptr. We really need "hPutBytes :: [Word8] -> IO ()" etc. Also, changing the existing functions to deal with encodings is likely to break a lot of things (i.e. anything which reads or writes data which is in neither UTF-8 nor the locale-specified encoding). -- Glynn Clements

Abraham Egnor wrote:

Passing a Ptr isn't that onerous; it's easy enough to make functions that have the signature you'd like:

import System.IO import Data.Word (Word8) import Foreign.Marshal.Array

hPutBytes :: Handle -> [Word8] -> IO () hPutBytes h ws = withArray ws $ \p -> hPutBuf h p $ length ws

hGetBytes :: Handle -> Int -> IO [Word8] hGetBytes h c = allocaArray c $ \p -> do c' <- hGetBuf h p c peekArray c' p

The problem with this approach is that the entire array has to be held in memory, which could be an issue if the amount of data involved is large. -- Glynn Clements

Abraham Egnor wrote:

Passing a Ptr isn't that onerous; it's easy enough to make functions that have the signature you'd like:

import System.IO import Data.Word (Word8) import Foreign.Marshal.Array

hPutBytes :: Handle -> [Word8] -> IO () hPutBytes h ws = withArray ws $ \p -> hPutBuf h p $ length ws

hGetBytes :: Handle -> Int -> IO [Word8] hGetBytes h c = allocaArray c $ \p -> do c' <- hGetBuf h p c peekArray c' p

If it's that simple, why don't the libraries provide it? It would save a lot of traffic on haskell and haskell-cafe alone. :-) -Jan-Willem Maessen

On Sun, 12 Sep 2004 14:24:56 +0100, Glynn Clements wrote:

...

Sven Panne wrote:

...

...

[...] main :: IO () main = do h <- openBinaryFile "out.dat" WriteMode hPutStr h $ map (octetToChar . bitsToOctet) bits hClose h

Hmmm, using string I/O when one really wants to do binary I/O gives me a bad feeling. Haskell characters are defined to be Unicode characters, so the above only works because current Haskell implementations usually get this wrong (either no Unicode support at all and/or ignoring any encodings and doing I/O only with the lower 8 bits of the characters)... hGetBuf/hPutBuf plus their non-blocking variants are the only way to *really* do binary I/O currently.

Which is unfortunate, because of the requirement to pass a Ptr. We really need "hPutBytes :: [Word8] -> IO ()" etc.

Also, changing the existing functions to deal with encodings is likely to break a lot of things (i.e. anything which reads or writes data which is in neither UTF-8 nor the locale-specified encoding).

-- Glynn Clements _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Sven Panne wrote:

...

Also, changing the existing functions to deal with encodings is likely to break a lot of things (i.e. anything which reads or writes data which is in neither UTF-8 nor the locale-specified encoding).

Hmmm, the Unicode tables start with ISO-Latin-1, so what would exactly break when we stipulate that the standard encoding for string I/O in Haskell is ISO-Latin-1?

That would essentially be formally specifying the existing behaviour, which wouldn't break anything, including the mechanism for reading/writing binary data which I suggested (and which is the only choice if your Haskell implementation doesn't have h{Get,Put}Buf). The problems would come if it was decided to change the existing behaviour, i.e. use something other than Latin1. -- Glynn Clements

Marcin 'Qrczak' Kowalczyk wrote:

But the default encoding should come from the locale instead of being ISO-8859-1.

The problem with that is that, if the locale's encoding is UTF-8, a lot of stuff is going to break (i.e. anything in ISO-8859-* which isn't limited to the 7-bit ASCII subset). The advantage of assuming ISO-8859-* is that the decoder can't fail; every possible stream of bytes is valid. This isn't the case for UTF-8. The advantage of ISO-8859-1 in particular is that it's trivial to convert the string back into the bytes which were actually read. The key problem with using the locale is that you frequently encounter files which aren't in the locale's encoding, and for which the encoding can't easily be deduced. If you assume ISO-8859-*, you can at least read them in, manipulate the contents (in any way that doesn't require interpreting any non-ASCII characters), and write out the results. OTOH, if you assume UTF-8 (e.g. because that happens to be the locale's encoding), the decoder is likely to abort shortly after the first non-ASCII character it finds (either that, or it will just silently drop characters). -- Glynn Clements

Marcin 'Qrczak' Kowalczyk wrote:

...

...

But the default encoding should come from the locale instead of being ISO-8859-1.

The problem with that is that, if the locale's encoding is UTF-8, a lot of stuff is going to break (i.e. anything in ISO-8859-* which isn't limited to the 7-bit ASCII subset).

What about this transition path:

1. API for manipulating byte sequences in I/O (without representing them in String type).

Note that this needs to include all of the core I/O functions, not just reading/writing streams. E.g. FilePath is currently an alias for String, but (on Unix, at least) filenames are strings of bytes, not characters. Ditto for argv, environment variables, possibly other cases which I've overlooked.

2. API for conversion between explicitly specified encodings and byte sequences, including attaching converters to Handles. There is also a way to obtain the locale encoding.

3. The default encoding is settable from Haskell, defaults to ISO-8859-1.

Agreed.

4. Libraries are reviewed to ensure that they work with various encoding settings.

There are limits to the extent to which this can be achieved. E.g. what happens if you set the encoding to UTF-8, then call getDirectoryContents for a directory which contains filenames which aren't valid UTF-8 strings?

5. The default encoding is settable from Haskell, defaults to the locale encoding.

I feel that the default encoding should be one whose decoder cannot fail, e.g. ISO-8859-1. You should have to explicitly request the use of the locale's encoding (analogous to calling setlocale(LC_CTYPE, "") at the start of a C program; there's a good reason why C doesn't do this without being explicitly told to). Actually, the more I think about it, the more I think that "simple, stupid programs" probably shouldn't be using Unicode at all. I.e. Char, String, string literals, and the I/O functions in Prelude, IO etc should all be using bytes, with a distinct wide-character API available for people who want to make the (substantial) effort involved in writing (genuinely) internationalised programs. Right now, the attempt at providing I18N "for free", by defining Char to mean Unicode, has essentially backfired, IMHO. Anything that isn't ISO-8859-1 just doesn't work for the most part, and anyone who wants to provide real I18N first has to work around the pseudo-I18N that's already there (e.g. convert Chars back into Word8s so that they can decode them into real Chars). Oh, and because bytes are being stored in Chars, the type system won't help if you neglect to decode a string, or if you decode it twice.

...

The advantage of assuming ISO-8859-* is that the decoder can't fail; every possible stream of bytes is valid.

Assuming. But UTF-8 is not ISO-8859-*. When someday I change most of my files and filenames from ISO-8859-2 to UTF-8, and change the locale, the assumption will be wrong. I can't change that now, because too many programs would break.

The current ISO-8859-1 assumption is also wrong. A program written in Haskell which sorts strings would break for non-ASCII letters even now that they are ISO-8859-2 unless specified otherwise.

1. In that situation, you can't avoid the encoding issues. It doesn't matter what the default is, because you're going to have to set the encoding anyhow. 2. If you assume ISO-8859-1, you can always convert back to Word8 then re-decode as UTF-8. If you assume UTF-8, anything which is neither UTF-8 nor ASCII will fail far more severely than just getting the collation order wrong.

...

The key problem with using the locale is that you frequently encounter files which aren't in the locale's encoding, and for which the encoding can't easily be deduced.

Programs should either explicitly set the encoding for I/O on these files to ISO-8859-1, or manipulate them as binary data.

Well, my view is essentially that files should be treated as containing bytes unless you explicitly choose to decode them, at which point you have to specify the encoding.

The problem is that API for that yet is not even designed, so programs can't be written such that they will work after the default encoding change.

Personally, I would take the C approach: redefine Char to mean a byte (i.e. CChar), treat string literals as bytes, keep the existing type signatures on all of the existing Haskell98 functions, and provide a completely new wide-character API for those who wish to use it. That gets the failed attempt at I18N out of everyone's way with a minimum of effort and with maximum backwards compatibility for existing code. Given the frequency with which this issue crops up, and the associated lack of action to date, I'd rather not have to wait until someone finally gets around to designing the new, improved, genuinely-I18N-ised API before we can read/write arbitrary files without too much effort. My main concern is that someone will get sick of waiting and make the wrong "fix", i.e. keep the existing API but default to the locale's encoding, so that every simple program then has to explicitly set it back to ISO-8859-1 to get reasonable worst-case behaviour. -- Glynn Clements

Marcin 'Qrczak' Kowalczyk wrote:

...

...

1. API for manipulating byte sequences in I/O (without representing them in String type).

Note that this needs to include all of the core I/O functions, not just reading/writing streams. E.g. FilePath is currently an alias for String, but (on Unix, at least) filenames are strings of bytes, not characters. Ditto for argv, environment variables, possibly other cases which I've overlooked.

They don't hold binary data; they hold data intended to be interpreted as text.

No. They frequently hold data intended to be passed to system functions which interpret them simply as bytes, without regard to encoding.

If the encoding of the text doesn't agree with the locale, the environment setup is broken and 'ls' and 'env' misbehave on an UTF-8 terminal.

ls and env just write bytes to stdout (which may or may not refer to the terminal). A particular terminal may not display them correctly, but that's a separate issue. Unless you are the sole user of a system, you have no control over what filenames may occur on it (and even if you are the sole user, you may wish to use packages which don't conform to your rules). As environment variables frequently contain pathnames, this fact may get propagated to the environment (however, system directories are usually restricted to ASCII, so this aspect is less likely to be an issue).

A program can explicitly set the default encoding to ISO-8859-1 if it wishes to do something in a broken environment.

...

...

4. Libraries are reviewed to ensure that they work with various encoding settings.

There are limits to the extent to which this can be achieved. E.g. what happens if you set the encoding to UTF-8, then call getDirectoryContents for a directory which contains filenames which aren't valid UTF-8 strings?

The library fails. Don't do that. This environment is internally inconsistent.

Call it what you like, it's a reality, and one which programs need to deal with. Most programs don't care whether any filenames which they deal with are valid in the locale's encoding (or any other encoding). They just receive lists (i.e. NUL-terminated arrays) of bytes and pass them directly to the OS or to libraries.

...

I feel that the default encoding should be one whose decoder cannot fail, e.g. ISO-8859-1.

But filenames on my filesystem and most file contents are *not* encoded in ISO-8859-1. Assuming that they are ISO-8859-1 is plainly wrong.

For the most part, assuming that they are encoded in *any* coding system is wrong. However, If you treat them as ISO-8859-* (it doesn't matter which one, so long as you're consistent), the Haskell I/O functions will at least pass them through unmodified. Consider a trivial "cp" program: main = do [src, dst] <- getArgs text <- readFile src writeFile dst text If the assumed encoding is ISO-8859-*, this program will work regardless of the filenames which it is passed or the contents of the file (modulo the EOL translation on Windows). OTOH, if it were to use UTF-8 (e.g. because that was the locale's encoding), it wouldn't work correctly if either filename or the file's contents weren't valid UTF-8.

...

You should have to explicitly request the use of the locale's encoding (analogous to calling setlocale(LC_CTYPE, "") at the start of a C program; there's a good reason why C doesn't do this without being explicitly told to).

C usually uses the paradigm of representing text in their original 8-bit encodings. This is why getting C programs to work in a UTF-8 locale is such a pain. Only some programs use wchar_t internally.

Many C programs don't care about encodings. It's only if you actually have to interpret the bytes (e.g. ctype.h, strcoll) that encodings start to matter. At which point, you have to know the encoding.

Java and C# uses the paradigm of representing text in Unicode internally, recoding it on boundaries with the external world.

The second paradigm has a cost that you must be aware what encodings are used in texts you manipulate.

And that cost can be a pretty high; e.g. gratuitously failing in the case where you have no idea which encoding is used but where you shouldn't actually need to know.

Locale gives a reasonable default for simple programs which aren't supposed to work with multiple encodings, and it specifies the encoding of texts which don't have an encoding specified elsewhere (terminal I/O, filenames, environment variables).

More accurately, it specifies which encoding to assume when you *need* to know the encoding (i.e. ctype.h etc), but you can't obtain that information from a more reliable source. My central point is that the existing API forces the encoding to be an issue when it shouldn't be.

ncurses wide character API is still broken. I reported bugs, the author acknowledged them, but hasn't fixed them. (Attributes are ignored on add_wch; get_wch is wrong for non-ASCII keys pressed if the locale is different from ISO-8859-1 and UTF-8.) It seems people don't use that API yet, because C traditionally uses the model of representing texts in byte sequences. But the narrow character API of ncurses is unusable with UTF-8 - this is not an implementation limitation but inherent limitation of the interface.

Well, to an extent it is an implementation issue. Historically, curses never cared about encodings. A character is a byte, you draw bytes on the screen, curses sends them directly to the terminal. The terminal's encoding has to match that used by the program or it displays incorrectly. The curses library neither knows nor cares about the encoding of either the terminal or the application. Furthermore, the curses model relies upon monospaced fonts, and falls down once you encounter CJK text (where a "monospaced" font means one whose glyphs are an integer multiple of the cell size, not necessarily a single cell). Extending something like curses to handle encoding issues is far from trivial; which is probably why it hasn't been finished yet.

...

I.e. Char, String, string literals, and the I/O functions in Prelude, IO etc should all be using bytes, with a distinct wide-character API available for people who want to make the (substantial) effort involved in writing (genuinely) internationalised programs.

This would cause excessive duplication of APIs. Look, Java and C# don't do that. Only file contents handling needs a byte API, because many files don't contain text.

Interfacing to (byte-oriented) OS functions also needs a byte API if you want them to work reliably.

This would imply isAlpha :: Char -> IO Bool.

Yes. You cannot determine whether a *byte* is alphabetical, only characters. Although, if you're going to have implicit String -> [Word8] converters, there's no reason why you can't do the reverse, and have isAlpha :: Word8 -> IO Bool. Although, like ctype.h, this will only work for single-byte encodings.

...

Right now, the attempt at providing I18N "for free", by defining Char to mean Unicode, has essentially backfired, IMHO.

Because it needs to be accompanied with character recoders, both invoked explicitly (also lazily) and attached to file handles, and with a way to obtain recoders for various encodings.

Assuming that the same encoding is used everywhere and programs can just copy bytes without interpreting them no longer works today.

It works in plenty of situations. And it doesn't assume that the same encoding is used everywhere; it just assumes that the encoding is irrelevant except when there's a reason to the contrary.

A mail client is expected to respect the encoding set in headers.

A client typically needs to know the encoding in order to display the text. As a counter-example, a mail *server* can do its job without paying any attention to the encodings used. It can also handle non-MIME email (which doesn't specify any encoding) regardless of the encoding.

...

Oh, and because bytes are being stored in Chars, the type system won't help if you neglect to decode a string, or if you decode it twice.

This is why I said "1. API for manipulating byte sequences in I/O (without representing them in String type)".

Yes. But that API also needs to include functions such as those in the Directory and System modules. It isn't just about reading and writing streams. Most of the Unix API (kernel, libc, and many standard libraries) is byte-oriented rather than character-oriented.

...

2. If you assume ISO-8859-1, you can always convert back to Word8 then re-decode as UTF-8. If you assume UTF-8, anything which is neither UTF-8 nor ASCII will fail far more severely than just getting the collation order wrong.

If I know the encoding, I should set the I/O handle to that encoding in the first place instead of reinterpreting characters which have been read using the default.

And if you don't know the encoding?

...

Personally, I would take the C approach: redefine Char to mean a byte (i.e. CChar), treat string literals as bytes, keep the existing type signatures on all of the existing Haskell98 functions, and provide a completely new wide-character API for those who wish to use it.

Well, this is the paradigm which has problems in different areas. It will often break in UTF-8 locale, it needs isAlpha :: Char -> IO Bool, and it's painful to support multiple encodings.

Agreed. But writing programs which support I18N, multi-byte encodings, wide character sets (>256 codepoints) and the like on an OS whose core API is byte-oriented involves work. And it can't all be hidden within a library. Some of the work falls on the application programmers, who have to deal with determining the correct encoding in each situation, converting between encodings, handling encoding and decoding failures (e.g. when you encounter a Unicode filename but the terminal only has Latin1), and so on.

Char is *the* new API. What is missing is byte API in areas which work with arbitrary binary data (mostly file contents).

And the ability to actually use any encoding except ISO-8859-1 in any meaningful way. I.e. encoders/decoders for other encodings, along with the means to specify which encoding to use for functions which need to perform encoding or decoding.

...

My main concern is that someone will get sick of waiting and make the wrong "fix", i.e. keep the existing API but default to the locale's encoding, so that every simple program then has to explicitly set it back to ISO-8859-1 to get reasonable worst-case behaviour.

Supporting byte I/O and supporting character recoding needs to be done before this.

My view is that, right now, we have the worst of both worlds, and taking a short step backwards (i.e. narrow the Char type and leave the rest alone) is a lot simpler (and more feasible) than the long journey towards real I18N. More generally, this is the most intrusive example of a common problem with too many Haskell libraries, i.e. exporting an interface which is too high-level and glosses over too many detail. But this isn't some obscure third-party libray. This is the Haskell98 standard library; some of it's in the Prelude. -- Glynn Clements

Marcin 'Qrczak' Kowalczyk wrote:

...

Unless you are the sole user of a system, you have no control over what filenames may occur on it (and even if you are the sole user, you may wish to use packages which don't conform to your rules).

For these occasions you may set the encoding to ISO-8859-1. But then you can't sensibly show them to the user in a GUI, nor in ncurses using the wide character API, nor you can't sensibly store them in a file which is to be always encoded in UTF-8 (e.g. XML file where you can't put raw bytes without knowing their encoding).

If you need to preserve the data exactly, you can use octal escapes (\337), URL encoding (%DF) or similar. If you don't, you can just approximate it (e.g. display unrepresentable characters as "?"). But this is an inevitable consequence of filenames being bytes rather than chars. [Actually, regarding on-screen display, this is also an issue for Unicode. How many people actually have all of the Unicode glyphs? I certainly don't.]

There are two paradigms: manipulate bytes not knowing their encoding, and manipulating characters explicitly encoded in various encodings (possibly UTF-8). The world is slowly migrating from the first to the second.

This migration isn't a process which will ever be complete. There will always be plenty of cases where bytes really are just bytes. And even to the extent that it can be done, it will take a long time. Outside of the Free Software ghetto, long-term backward compatibility still means a lot. [E.g.: EBCDIC has been in existence longer than I have and, in spite of the fact that it's about the only widely-used encoding in existence which doesn't have ASCII as a subset, it shows no sign of dying out any time soon.]

...

...

...

There are limits to the extent to which this can be achieved. E.g. what happens if you set the encoding to UTF-8, then call getDirectoryContents for a directory which contains filenames which aren't valid UTF-8 strings?

The library fails. Don't do that. This environment is internally inconsistent.

Call it what you like, it's a reality, and one which programs need to deal with.

The reality is that filenames are encoded in different encodings depending on the system. Sometimes it's ISO-8859-1, sometimes ISO-8859-2, sometimes UTF-8. We should not ignore the possibility of UTF-8-encoded filenames.

I'm not suggesting we do.

In CLisp it fails silently (undecodable filenames are skipped), which is bad. It should fail loudly.

No, it shouldn't fail at all.

...

Most programs don't care whether any filenames which they deal with are valid in the locale's encoding (or any other encoding). They just receive lists (i.e. NUL-terminated arrays) of bytes and pass them directly to the OS or to libraries.

And this is why I can't switch my home environment to UTF-8 yet. Too many programs are broken; almost all terminal programs which use more than stdin and stdout in default modes, i.e. which use line editing or work in full screen. How would you display a filename in a full screen text editor, such that it works in a UTF-8 environment?

So, what are you suggesting? That the whole world switches to UTF-8? Or that every program should pass everything through iconv() (and handle the failures)? Or what?

...

If the assumed encoding is ISO-8859-*, this program will work regardless of the filenames which it is passed or the contents of the file (modulo the EOL translation on Windows). OTOH, if it were to use UTF-8 (e.g. because that was the locale's encoding), it wouldn't work correctly if either filename or the file's contents weren't valid UTF-8.

A program is not supposed to encounter filenames which are not representable in the locale's encoding.

Huh? What does "supposed to" mean in this context? That everything would be simpler if reality wasn't how it is? If that's your position, then my response is essentially: Yes, but so what?

In your setting it's impossible to display a filename in a way other than printing to stdout.

Writing to stdout doesn't amount to "displaying" anything; stdout doesn't have to be a terminal.

...

More accurately, it specifies which encoding to assume when you *need* to know the encoding (i.e. ctype.h etc), but you can't obtain that information from a more reliable source.

In the case of filenames there is no more reliable source.

Sure; but that doesn't automatically mean that the locale's encoding is correct for any given filename. The point is that you often don't need to know the encoding. Converting a byte string to a character string when you're just going to be converting it back to the original byte string is pointless. And it introduces unnecessary errors. If the only difference between (decode . encode) and the identity function is that the former sometimes fails, what's the point?

...

My central point is that the existing API forces the encoding to be an issue when it shouldn't be.

It is an unavoidable issue because not every interface in a given computer system uses the same encoding. Gtk+ uses UTF-8; you must convert text to UTF-8 in order to display it, and in order to convert you must know its encoding.

It frequently *is* an avoidable issue, because not every interface uses *any* encoding. Most of the standard Unix utilities work fine without even considering encodings.

...

Extending something like curses to handle encoding issues is far from trivial; which is probably why it hasn't been finished yet.

It's almost finished. The API specification was ready in 1997. It works in ncurses modulo unfixed bugs.

Right, so it's taken 7 years to get to "almost finished, modulo unfixed bugs". Which seems to back up my "far from trivial" claim.

...

Although, if you're going to have implicit String -> [Word8] converters, there's no reason why you can't do the reverse, and have isAlpha :: Word8 -> IO Bool. Although, like ctype.h, this will only work for single-byte encodings.

We should not ignore multibyte encodings like UTF-8, which means that Haskell should have a Unicoded character type. And it's already specified in Haskell 98 that Char is such a type!

I'm not suggesting that we ignore them. I'm suggesting that we: 1. Don't provide a broken API which makes it impossible to write programs which work reliably in the real world (rather than some fantasy world where inconveniences (like filenames which don't match the locale's encoding) never happen). 2. Don't force everyone to deal with all of the the complexities involved in character encoding even when they shouldn't have to.

What is missing is API for manipulating binary files, and conversion between byte streams and character streams using particular text encodings.

...

...

A mail client is expected to respect the encoding set in headers.

A client typically needs to know the encoding in order to display the text.

This is easier to handle when String type means Unicode.

Not necessarily. There are advantages and disadvantages to both the byte-stream approach and the wide-character approach. And, given that Unicode isn't a simple "one code, one character" system (what with composing characters), it isn't actually all that much simpler than dealing with multi-byte strings. From a display perspective, it's largely multi-byte strings vs multi-wchar_t strings. The main advantage of Unicode for display is that there's only one encoding. Unfortunately, given that most of the existing Unicode fonts are a bit short on actual glyphs, you typically just end up converting the Unicode back into pseudo-ISO-2022 anyhow.

...

As a counter-example, a mail *server* can do its job without paying any attention to the encodings used. It can also handle non-MIME email (which doesn't specify any encoding) regardless of the encoding.

So it should push bytes, not characters.

And so should a lot of software. But it helps if languages and libraries doesn't go to great lengths to try and coerce everything into characters.

...

...

This is why I said "1. API for manipulating byte sequences in I/O (without representing them in String type)".

Yes. But that API also needs to include functions such as those in the Directory and System modules.

If deemed really necessary, I will not fight against them.

Oh, it's necessary. As to whether it will be *deemed* necessary might be a different issue. The "wishful thinking" approach to problem solving is no stranger around these parts.

...

It isn't just about reading and writing streams. Most of the Unix API (kernel, libc, and many standard libraries) is byte-oriented rather than character-oriented.

Because they are primarily used from C,

The Unix API is primarily used from *everything which runs on Unix*. Particularly the kernel API (i.e. system calls), which you have no choice but to use; if you want to open a file, it *will* go through the open() system call at some point.

which use the older paradigm of handling text: represent it in an unspecified external encoding rather than in Unicode.

OTOH newer Windows APIs use Unicode.

Haskell aims at being portable. It's easier to emulate the traditional C paradigm in the Unicode paradigm than vice versa,

I'm not entirely sure what you mean by that, but I think that I disagree. The C/Unix approach is more general; it isn't tied to any specific encoding.

and Haskell already tries to specify that it uses Unicode internally.

I don't have any complaints about it using Unicode for characters. My gripe is with the fact that it tries hard to coerce what is actually arbitrary binary data into being characters.

It's not that hard if you may sacrifice supporting every broken configuration. I did it myself, albeit without serious testing in real world situations and without trying to interface to too many libraries.

I take it that, by "broken", you mean any string of bytes (file, string, network stream, etc) which neither explicitly specifies its encoding(s) nor uses your locale's encoding? That's a pretty big sacrifice.

...

My view is that, right now, we have the worst of both worlds, and taking a short step backwards (i.e. narrow the Char type and leave the rest alone) is a lot simpler (and more feasible) than the long journey towards real I18N.

It would bury any hope in supporting a UTF-8 environment.

I've heard that RedHat tried to impose UTF-8 by default. It was mostly a failure because it's too early, too many programs are not ready for it. I guess the RedHat move helped to identify some of them. But UTF-8 will inevitably be usable in future.

If they tried a decade hence, it would still be too early. The single-byte encodings (ISO-8859-*, koi-8, win-12xx) aren't likely to be disappearing any time soon, nor is ISO-2022 (UTF-8 has quite spectacularly failed to make inroads in CJK-land; there are probably more UTF-8 users in the US than there).

It would be great if Haskell programs were in the group which can support it instead of being forced to be abandoned because of lack of Unicode support in the language they are written in.

Haskell should be able to support it, but it shouldn't refuse to support anything else, it shouldn't make you jump through hoops to write usable programs, and we shouldn't have to wait until all of the encoding issues have been sorted out to do things which don't even deal with encodings. Look, C has all of the functionality that we're talking about: wide characters, wide versions of string.h and ctype.h, and conversion between byte-streams and wide characters. But it did it without getting in the way of writing programs which don't care about encodings, without consigning everything which has gone before to the scrap heap, and without everyone having to wait a couple of decades to (reliably) do simple things like copying a file to a socket or enumerating a directory. -- Glynn Clements

Marcin 'Qrczak' Kowalczyk wrote:

...

[Actually, regarding on-screen display, this is also an issue for Unicode. How many people actually have all of the Unicode glyphs? I certainly don't.]

If I don't have a particular character in fonts, I will not create files with it in filenames. Actually I only use 9 Polish letters in addition to ASCII, and even them rarely. Usually it's only a subset of ASCII.

But this seems to be assuming a closed world. I.e. the only files which the program will ever see are those which were created by you, or by others who are compatible with your conventions.

Some programs use UTF-8 in filenames no matter what the locale is. For example the Evolution mail program which stores mail folders as files under names the user entered in a GUI.

This is entirely reasonable for a file which a program creates. If a filename is just a string of bytes, a program can use whatever encoding it wants.

I had to rename some of these files in order to import them to Gnus, as it choked on filenames with strange characters, never mind that it didn't display them correctly (maybe because it tried to map them to virtual newsgroup names, or maybe because they are control characters in ISO-8859-x).

If it had just treated them as bytes, rather than trying to interpret them as characters, there wouldn't have been any problems.

If all programs consistently used the locale encoding for filenames, this should have worked.

But again, for this to work in general, you have to assume a closed world.

When I switch my environment to UTF-8, which may happen in a few years, I will convert filenames to UTF-8 and set up mount options to translate vfat filenames to/from UTF-8 instead of to ISO-8859-2.

But what about files which were been created by other people, who don't use UTF-8?

I expect good programs to understand that and display them correctly no matter what technique they are using for the display.

When it comes to display, you have to have to deal with encoding issues one way or another. But not all programs deal with display.

For example the Epiphany web browser, when I open the file:/home/users/qrczak URL, displays ISO-8859-2-encoded filenames correctly. The virtual HTML file it created from the directory listing has &x105; in its <title> where the directory name had 0xB1 in ISO-8859-2. When I run Epiphany with the locale set to pl_PL.UTF-8, it displays UTF-8 filenames correctly and ISO-8859-2 filenames are not shown at all.

For many (probably most) programs, omitting such files would be an unacceptable failure.

...

And even to the extent that it can be done, it will take a long time. Outside of the Free Software ghetto, long-term backward compatibility still means a lot.

Windows has already switched most of its internals to Unicode, and it did it faster than Linux.

Microsoft is actively hostile to both backwards compatibility and cross-platform compatibility. I consider the fact that some Unix (primarily Linux) developers seem equally hostile to be a problem. Having said that, with Linux developers, the issue usually due to not being bothered. Assuming that everything is UTF-8 allows a lot of potential problems to be ignored. Fortunately, the problem is mostly consigned to the periphery, i.e. the desktop, where most programs have to deal with display issues (so you *have* to decode bytes into characters), and it isn't too critical if they have limitations. The core OS and network server applications essentially remain encoding-agnostic.

...

...

In CLisp it fails silently (undecodable filenames are skipped), which is bad. It should fail loudly.

No, it shouldn't fail at all.

Since it uses Unicode as string representation, accepting filenames not encoded in the locale encoding would imply making garbage from filenames correctly encoded in the locale encoding. In a UTF-8 environment character U+00E1 in the filename means bytes 0xC3 0xA1 on ext2 filesystem (and 0x00E1 on vfat filesystem), so it can't at the same time mean 0xE1 on ext2 filesystem.

But, as I keep pointing out, filenames are byte strings, not character strings. You shouldn't be converting them to character strings unless you have to.

...

...

And this is why I can't switch my home environment to UTF-8 yet. Too many programs are broken; almost all terminal programs which use more than stdin and stdout in default modes, i.e. which use line editing or work in full screen. How would you display a filename in a full screen text editor, such that it works in a UTF-8 environment?

So, what are you suggesting? That the whole world switches to UTF-8?

No, each computer system decides for itself, and announces it in the locale setting. I'm suggesting that programs should respect that and correctly handle all correctly encoded texts, including filenames.

1. Actually, each user decides which locale they wish to use. Nothing forces two users of a system to use the same locale. 2. Even if the locale was constant for all users on a system, there's still the (not exactly minor) issue of networking.

...

Or that every program should pass everything through iconv() (and handle the failures)?

If it uses Unicode as internal string representation, yes (because the OS API on Unix generally uses byte encodings rather than Unicode).

The problem with that is that you need to *know* the source and destination encodings. The program gets to choose one of them, but it may not even know the other one.

This should be done transparently in libraries of respective languages instead of in each program independently.

The application still has to tell the library which encoding is to be used; if it can actually determine it.

...

...

A program is not supposed to encounter filenames which are not representable in the locale's encoding.

Huh? What does "supposed to" mean in this context? That everything would be simpler if reality wasn't how it is?

It means that if it encounters a filename encoded differently, it's usually not the fault of the program but of whoever caused the mismatch in the first place.

The term "mismatch" implies that there have to be at least two things. If they don't match, which one is at fault? If I make a tar file available for you to download, and it contains non-UTF-8 filenames, is that my fault or yours? In any case, if a program refuses to deal with a file because it is cannot convert the filename to characters, even when it doesn't have to, it's the program which is at fault. Or are you suggesting that it's acceptable for e.g. "rm <filename>" to refuse to work because the the filename cannot be converted to characters? BTW, to go back to this point:

...

Huh? What does "supposed to" mean in this context? That everything would be simpler if reality wasn't how it is?

Note that I entirely agree that everything really would be simpler if reality wasn't how it is, e.g. if the entire world only ever used UTF-8. But that doesn't change anything. The reality is that multiple encodings are in widespread use, and that situation won't change for the foreseeable future. There exists a vast amount of software that is limited to certain encodings (e.g. just ASCII, or just single-byte encodings, or just ISO-2022-compatible encodings), and some of that will still be in use decades hence (for commercial software, particularly bespoke software, upgrading could cost vast amounts of money).

...

Sure; but that doesn't automatically mean that the locale's encoding is correct for any given filename. The point is that you often don't need to know the encoding.

What if I do need to know the encoding? I must assume something.

If you need to know it, then you need to know it. But that wasn't my point. My specific point is that the Haskell98 API has a very big problem due to the assumption that the encoding is always known. Existing implementations work around the problem by assuming that the encoding is always ISO-8859-1. The implementations could be changed to do something else, e.g. assume that it's always the locale's encoding, or that it's always UTF-8. Although neither of those is a significant improvement, and both are, in some regards, even worse: UTF-8 decoding can fail (unlike ISO-8859-*), while assuming the locale's encoding can result in the program's behaviour varying for a reason which may not be readily apparent. The real problem is that the API is broken, and that isn't something which the implementations can fix.

...

Converting a byte string to a character string when you're just going to be converting it back to the original byte string is pointless.

It's necessary if the channel through which the filename is transferred uses Unicode text, or bytes in some explicitly chosen encoding, rather than raw bytes in some unspecified encoding.

I did say "... when you're just going to be converting it back to the original byte string ...". I'd really appreciate it if you could address what I'm actually saying, rather than setting up straw-men. I'm not going to address all of the other examples of "... but here's a situation where you *do* need to deal with characters". That isn't my point.

...

2. Don't force everyone to deal with all of the the complexities involved in character encoding even when they shouldn't have to.

I don't see how to have this property and at the same time make writing programs which do handle various encodings reasonably easy. With my choices all Haskell APIs use Unicode, so once libraries which interface with the world are written, the program passes strings between them without recoding. With your choices the API for filenames uses a different encoding than the API for GUI, so the conversion logic must be put in each program separately.

The API for filenames doesn't use any encoding, because filenames are just bytes. And this isn't my choice, it's just how things are.

...

...

OTOH newer Windows APIs use Unicode.

Haskell aims at being portable. It's easier to emulate the traditional C paradigm in the Unicode paradigm than vice versa,

I'm not entirely sure what you mean by that, but I think that I disagree. The C/Unix approach is more general; it isn't tied to any specific encoding.

If filenames were expressed as bytes in the Haskell program, how would you map them to WinAPI? If you use the current Windows code page, the set of valid characters is limited without a good reason.

Windows filenames are arguably characters rather than bytes. However, if you want to present a common API, you can just use a fixed encoding on Windows (either UTF-8 or UTF-16).

...

If they tried a decade hence, it would still be too early. The single-byte encodings (ISO-8859-*, koi-8, win-12xx) aren't likely to be disappearing any time soon, nor is ISO-2022 (UTF-8 has quite spectacularly failed to make inroads in CJK-land; there are probably more UTF-8 users in the US than there).

Which is a pity. ISO-2022 is brain-damaged because of enormous complexity,

Or, depending upon ones perspective, Unicode is brain-damaged because, for the sake of simplicity, it over-simplifies the situation. The over-simplification is one reason for it's lack of adoption in the CJK world. Multi-lingual text consists of distinct sections written in distinct languages with distinct "alphabets". It isn't actually one big chunk in a single global language with a single massive alphabet.

and ISO-8859-x have small repertoires.

Which is one of the reasons why they are likely to persist for longer than UTF-8 "true believers" might like. E.g. languages which don't primarily use the Roman alphabet (Greek, Russian) can still be represented as one byte per character. And it's feasible to have tables which are indexed by codepoint; as a counter-example, calling XQueryFont for a Unicode font *really* sucks if either the server doesn't have the BigFont extension or, worse still, it can't use it because the client is remote.

I would not *force* UTF-8, but it should work for those who voluntarily choose to use it as their locale encoding. Including filenames.

Not forcibly decoding filenames isn't the same thing as preventing them from being decoded.

...

Look, C has all of the functionality that we're talking about: wide characters, wide versions of string.h and ctype.h, and conversion between byte-streams and wide characters.

ctype.h is useless for UTF-8.

Hello? Let's try that again, with emphasis:

...

C has ... WIDE VERSIONS OF string.h and ctype.h

They're called wchar.h and wctype.h.

There is no capability of attaching automatic recoders of explicitly chosen encodings to file handles.

At this point you starting engaging in diversionary tactics. Again.

No, the C language doesn't make these issues easy and has lots of historic baggage.

The issues aren't easy, and have lots of historic baggage. That's reality. Fortunately, C has a history of being geared to reality, rather than the comfortable fantasy where the issues don't exist. Which is why everyone uses it.

...

But it did it without getting in the way of writing programs which don't care about encodings,

It does get in the way of writing programs which do care, because they must do whole recoding themselves and remember which API has which character set limitations.

No. Not doing something for you isn't the same thing as getting in the way. Getting in the way is doing for you something which you didn't want done in the first place. Getting in the way is not letting you do something yourself. -- Glynn Clements

Marcin 'Qrczak' Kowalczyk wrote:

...

...

When I switch my environment to UTF-8, which may happen in a few years, I will convert filenames to UTF-8 and set up mount options to translate vfat filenames to/from UTF-8 instead of to ISO-8859-2.

But what about files which were been created by other people, who don't use UTF-8?

All people sharing a filesystem should use the same encoding.

Again, this is just "hand waving" the issues away.

BTW, when ftping files between Windows and Unix, a good ftp client should convert filenames to keep the same characters rather than bytes, so CP-1250 encoded names don't come as garbage in the encoding used on Unix which is definitely different (ISO-8859-2 or UTF-8) or vice versa.

Which is fine if the FTP client can figure out which encoding is used on the remote end. In practice, you have to tell it, i.e. have a list of which servers (or even which directories on which servers) use which encoding.

...

...

I expect good programs to understand that and display them correctly no matter what technique they are using for the display.

When it comes to display, you have to have to deal with encoding issues one way or another. But not all programs deal with display.

So you advocate using multiple encodings internally. This is in general more complicated than what I advocate: using only Unicode internally, limiting other encodings to I/O boundary.

How do you draw that conclusion from what I wrote here? There are cases where it's advantages to use multiple encodings, but I wasn't suggesting that in the above. What I'm suggesting in the above is to sidestep the encoding issue by keeping filenames as byte strings wherever possible.

...

The core OS and network server applications essentially remain encoding-agnostic.

Which is a problem when they generate an email, e.g. to send a non-empty output of a cron job, or report unauthorized use of sudo. If the data involved is not pure ASCII, I will often be mangled.

It only gets mangled if you feed it to a program which is making assumptions about the encoding. Non-MIME messages neither specify nor imply an encoding. MIME messages can use either "text/plain; charset=x-unknown" or application/octet-stream if they don't undertand the encoding. And program-generated email notifications frequently include text with no known encoding (i.e. binary data). Or are you going to demand that anyone who tries to hack into your system only sends it UTF-8 data so that the alert messages are displayed correctly in your mail program?

It's rarely a problem in practice because filenames, command arguments, error messages, user full names etc. are usually pure ASCII. But this is slowly changing.

To the extent that non-ASCII filenames are used, I've encountered far more filenames in both Latin1 and ISO-2022 than in UTF-8. Japanese FTP sites typically use ISO-2022 for everything; even ASCII names may have "\e(B" prepended to them.

...

But, as I keep pointing out, filenames are byte strings, not character strings. You shouldn't be converting them to character strings unless you have to.

Processing data in their original byte encodings makes supporting multiple languages harder. Filenames which are inexpressible as character strings get in the way of clean APIs. When considering only filenames, using bytes would be sufficient, but in overall it's more convenient to Unicodize them like other strings.

It also harms reliability. Depending upon the encoding, two distinct byte strings may have the same Unicode representation. E.g. if you are interfacing to a server which uses ISO-2022 for filenames, you have to get the escapes correct even when they are no-ops in terms of the string representation. If you obtain a directory listing, receive the filename "\e(Bfoo.txt", and convert it to Unicode, you get "foo.txt". If you then convert it back without the leading escape, the server is going to say "file not found".

...

The term "mismatch" implies that there have to be at least two things. If they don't match, which one is at fault? If I make a tar file available for you to download, and it contains non-UTF-8 filenames, is that my fault or yours?

Such tarballs are not portable across systems using different encodings.

Well, programs which treat filenames as byte strings to be read from argv[] and passed directly to open() won't have any problems with this. It's only a problem if you make it a problem.

If I tar a subdirectory stored on ext2 partition, and you untar it on a vfat partition, whose fault it is that files which differ only in case are conflated?

Arguably, it's Microsoft's fault for not considering the problems caused by multiple encodings when they decided that filenames were going to be case-folded.

...

In any case, if a program refuses to deal with a file because it is cannot convert the filename to characters, even when it doesn't have to, it's the program which is at fault.

Only if it's a low-level utility, to be used in an unfriendly environment.

A Haskell program in my world can do that too. Just set the encoding to Latin1.

But programs should handle this by default, IMHO. Filenames are, for the most part, just "tokens" to be passed around. You get a value from argv[], and pass it to open() or whatever. It doesn't need to have any meaning.

...

My specific point is that the Haskell98 API has a very big problem due to the assumption that the encoding is always known. Existing implementations work around the problem by assuming that the encoding is always ISO-8859-1.

The API is incomplete and needs to be enhanced. Programs written using the current API will be limited to using the locale encoding.

That just adds unnecessary failure modes.

Just as ReadFile is limited to text files because of line endings. What do you prefer: to provide a non-Haskell98 API for binary files, or to "fix" the current API by forcing programs to use "\r\n" on Windows and "\n" on Unix manually?

That's a harder case. There is a good reason for auto-converting EOL, as most programs actually process file contents. Most programs don't "process" filenames; they just pass them around.

...

...

If filenames were expressed as bytes in the Haskell program, how would you map them to WinAPI? If you use the current Windows code page, the set of valid characters is limited without a good reason.

Windows filenames are arguably characters rather than bytes. However, if you want to present a common API, you can just use a fixed encoding on Windows (either UTF-8 or UTF-16).

This encoding would be incompatible with most other texts seen by the program. In particular reading a filename from a file would not work without manual recoding.

We already have that problem; you can't read non-Latin1 strings from files. In some regards, the problem is worse on Windows, because of the prevalence of non-ASCII text (Windows 12xx and "smart" quotes), so using UTF-8 for file contents on Windows is even harder.

...

...

Which is a pity. ISO-2022 is brain-damaged because of enormous complexity,

Or, depending upon ones perspective, Unicode is brain-damaged because, for the sake of simplicity, it over-simplifies the situation. The over-simplification is one reason for it's lack of adoption in the CJK world.

It's necessary to simplify things in order to make them usable by ordinary programs. People reject overly complicated designs even if they are in some respects more general.

ISO-2022 didn't catch - about the only program I've seen which tries to fully support it is Emacs.

And X. Compound text is ISO-2022. For commercial X software, Motif (which uses compound text) is still the most widely-used toolkit. But, then, the fact that you haven't seen many ISO-2022 programs is probably because you're used to using programs developed by and for Westerners. In the far East, ISO-2022 is by far the most popular encoding. There, you could realistically ignore all other encodings. BTW, that's why Emacs (and XEmacs) support ISO-2022 much better than they do UTF-8. Because MuLE was written by Japanese developers.

...

Multi-lingual text consists of distinct sections written in distinct languages with distinct "alphabets". It isn't actually one big chunk in a single global language with a single massive alphabet.

Multi-lingual text is almost context-insensitive. You can copy a part of it into another text, even written in another language, and it will retain its alphabet - this is much harder with stateful ISO-2022.

ISO-2022 is wrong not by distinguishing alphabets but by being stateful.

Sure, the statefulness adds complexity (which is one of the reasons so many people prefer to work with UTF-8), but it has the benefit of providing distinct markers to indicate where the character set is being switched (that isn't a compelling advantage; you could reconstruct the markers if you could uniquely determine the character set for each character). OTOH, Unicode is wrong by not distinguishing character sets. This is a significant reason why it hasn't been adopt in the far East (specifically, Han unification).

...

...

and ISO-8859-x have small repertoires.

Which is one of the reasons why they are likely to persist for longer than UTF-8 "true believers" might like.

My I/O design doesn't force UTF-8, it works with ISO-8859-x as well.

But I was specifically addressing Unicode versus multiple encodings internally. The size of the Unicode "alphabet" effectively prohibits using codepoints as indices. -- Glynn Clements

Marcin 'Qrczak' Kowalczyk wrote:

...

What I'm suggesting in the above is to sidestep the encoding issue by keeping filenames as byte strings wherever possible.

Ok, but let it be in addition to, not instead treating them as character strings.

Provided that you know the encoding, nothing stops you converting them to strings, should you have a need to do so.

...

...

Processing data in their original byte encodings makes supporting multiple languages harder. Filenames which are inexpressible as character strings get in the way of clean APIs. When considering only filenames, using bytes would be sufficient, but in overall it's more convenient to Unicodize them like other strings.

It also harms reliability. Depending upon the encoding, two distinct byte strings may have the same Unicode representation.

Such encodings are not suitable for filenames.

Regardless of whether they are "suitable", they are used.

For me ISO-2022 is a brain-damaged concept and should die.

Well, it isn't likely to. I haven't addressed any of the other stuff about ISO-2022, as it isn't really relevant. Whether ISO-2022 is good or bad doesn't matter; what matters is that it is likely to remain in use for the foreseeable future.

...

...

Such tarballs are not portable across systems using different encodings.

Well, programs which treat filenames as byte strings to be read from argv[] and passed directly to open() won't have any problems with this.

The OS itself may have problems with this; only some filesystems accept arbitrary bytes apart from '\0' and '/' (and with the special meaning for '.'). Exotic characters in filenames are not very portable.

No, but most Unix programs manage to handle them without problems.

...

...

A Haskell program in my world can do that too. Just set the encoding to Latin1.

But programs should handle this by default, IMHO.

IMHO it's more important to make them compatible with the representation of strings used in other parts of the program.

Why?

...

Filenames are, for the most part, just "tokens" to be passed around.

Filenames are often stored in text files,

True.

whose bytes are interpreted as characters.

Sometimes true, sometimes not. Where filenames occur in data files, e.g. configuration files, the program which reads the configuration file typically passes the bytes directly to the OS without interpretation.

Applying QP to non-ASCII parts of filenames is suitable only if humans won't edit these files by hand.

Who said anything about QP?

...

...

...

My specific point is that the Haskell98 API has a very big problem due to the assumption that the encoding is always known. Existing implementations work around the problem by assuming that the encoding is always ISO-8859-1.

The API is incomplete and needs to be enhanced. Programs written using the current API will be limited to using the locale encoding.

That just adds unnecessary failure modes.

But otherwise programs would continuously have bugs in handling text which is not ISO-8859-1, especially with multibyte encoding where pretending that ISO-8859-2 is ISO-8859-1 too often doesn't work.

Why?

I can't switch my environment to UTF-8 yet precisely because too many programs were written with the attitude you are promoting: they don't care about the encoding, they just pass bytes around.

That's all that many programs should be doing.

Bugs range from small annoyances like tabular output which doesn't line up, through mangled characters on a graphical display, to full-screen interactive programs being unusable on a UTF-8 terminal.

IOW: 1. display doesn't work correctly, 2. display doesn't work correctly, and 3. display doesn't work correctly. You keep citing cases involving graphical display as a reason why all programs should be working with characters all of the time. I haven't suggested that programs should never deal with characters, yet you keep insinuating that is my argument, then proceed to attack it. -- Glynn Clements

On Sat, Sep 18, 2004 at 10:58:21AM +0200, Marcin 'Qrczak' Kowalczyk wrote:

Glynn Clements writes:

...

...

Ok, but let it be in addition to, not instead treating them as character strings.

Provided that you know the encoding, nothing stops you converting them to strings, should you have a need to do so.

There are already APIs which use Strings for filenames. I meant to keep them, let them use a program-settable encoding which defaults to the locale encoding - this is the only sane interpretation of this interface on Unix I can imagine. And in addition to them we may have APIs which use byte strings, for those who prefer the ability to handle all filenames to using a uniform string representation inside the program.

Keep in mind, if you make this change to the IO libraries, you also will have to simultaneously fix the Foreign.C.String module to use the same locale as is used by the IO libraries when they deal with FilePaths. Incidentally, this change will break existing darcs repositories... but that should at least be repairable. According to the FFI the CString functions do a locale-based conversion, but of course in practice that isn't the case. -- David Roundy http://www.abridgegame.org

On Mon, Sep 13, 2004 at 12:01:58PM +0100, Glynn Clements wrote:

My view is that, right now, we have the worst of both worlds, and taking a short step backwards (i.e. narrow the Char type and leave the rest alone) is a lot simpler (and more feasible) than the long journey towards real I18N.

This being Haskell, I can't imagine a consensus on a step backwards. In any case, a Char type distinct from bytes and the rest is the most valuable part of the current situation. The rest is just libraries, and the solution to that is to create other libraries. (It's true that the Prelude is harder to work around, but even that can be done, as with the new exception interface.) Indeed more than one approach can proceed concurrently, and that's probably what's going to happen: The Right Thing proceeds in stages: 1. new byte-based libraries 2. conversions sitting on top of these 3. the ultimate I18N API The Quick Fix: alter the existing implementation to use the encoding determined by the current locale at the borders. When the Right Thing is finished, the Quick Fix can be recast as a special case. The Right Thing might take a very long (possibly infinite) time, because this is the sort of thing people can argue about endlessly. Still, the first stage would deal with most of the scenarios you raised. It just needs a group of people who care about it to get together and do it. The Quick Fix is the most logical implementation of the current definition of Haskell, and entirely consistent with its general philosophy of presenting the programmer with an idealized (some might say oversimplified) model of computation. From the start, Haskell has supported only character-based I/O, with whatever translations were required to present a uniform view on all platforms. And that's not an entirely bad thing. It won't work all the time, but it will be simple, and good enough for most people. Its existence will not rule out binary I/O or more sophisticated alternatives. Those who need more may be motivated to help finish the Right Thing.

Graham Klyne wrote:

In particular, the idea of "narrowing" the Char type really seems like a bad idea to me (if I understand the intent correctly). Not so long ago, I did a whole load of work on the HaXml parser so that, among other things, it would support UTF-8 and UTF-16 Unicode (as required by the XML spec). To do this depends upon having a Char type that can represent the full repertoire of Unicode characters.

Note: I wasn't proposing doing away with wide character support altogether. Essentially, I was suggesting making Char a byte and having e.g. WideChar for wide characters. The reason being that the existing Haskell98 API uses Char for functions which are actually dealing with bytes. In an ideal world, the IO, System and Directory modules (and the Prelude I/O functions) would have used Byte, leaving Char to represent a (wide) character. However, that isn't the hand we've been dealt. -- Glynn Clements

Glynn Clements wrote:

Actually, the more I think about it, the more I think that "simple, stupid programs" probably shouldn't be using Unicode at all.

I.e. Char, String, string literals, and the I/O functions in Prelude, IO etc should all be using bytes, with a distinct wide-character API available for people who want to make the (substantial) effort involved in writing (genuinely) internationalised programs.

I have become very sympathetic to this viewpoint. In particular, it is nigh-impossible to just move bits into and out of a Haskell program. There certainly isn't a simple, portable way to do so. In the absence of such a mechanism, there are *no* *portable* *workarounds*. The absence of real internationalization is galling, but not nearly as galling as the absence of simple, stupid, reproducible I/O facilities. Forget even ISO-8859-1. Just give me bytes.

Personally, I would take the C approach: redefine Char to mean a byte (i.e. CChar), treat string literals as bytes, keep the existing type signatures on all of the existing Haskell98 functions, and provide a completely new wide-character API for those who wish to use it.

Much as we may hate to admit it, this would probably just work in 99% of all cases, and would make it a simple exercise to build working solutions.

Given the frequency with which this issue crops up, and the associated lack of action to date, I'd rather not have to wait until someone finally gets around to designing the new, improved, genuinely-I18N-ised API before we can read/write arbitrary files without too much effort.

This is simple stuff, and far more basic than the details of test representation. Why must it be so hard? And why must we drag students through the muck of allocating and managing explicit byte buffers and (God forbid) understanding the FFI just to get a few bytes on and off disk without the system monkeying with them en route? Simplify. Please. -Jan-Willem Maessen

Glynn Clements wrote:

Marcin 'Qrczak' Kowalczyk wrote:

...

[...] Note that this needs to include all of the core I/O functions, not just reading/writing streams. E.g. FilePath is currently an alias for String, but (on Unix, at least) filenames are strings of bytes, not characters. Ditto for argv, environment variables, possibly other cases which I've overlooked.

I don't think so. They all are sequences of CChars, and C isn't particularly known for keeping bytes and chars apart. I believe, Windows NT has (alternate) filename handling functions that use unicode stringsr. This would strengthen the view that a filename is a sequence of characters. Ditto for argv, env, whatnot; they are typically entered from the shell and therefore are characters in the local encoding.

...

3. The default encoding is settable from Haskell, defaults to ISO-8859-1.

Agreed.

Oh no, please don't do that. A global, settable encoding is, well, dys-functional. Hidden state makes programs hard to understand and Haskell imho shouldn't go that route. And please don't introduce the notion of a "default" encoding. I'd like to see the following: - Duplicate the IO library. The duplicate should work with [Byte] everywhere where the old library uses String. Byte is some suitable unsigned integer, on most (all?) platforms this will be Word8 - Provide an explicit conversion between encodings. A simple conversion of type [Word8] -> String would suit me, iconv would provide all that is needed. - iconv takes names of encodings as arguments. Provide some names as constants: one name for the internal encoding (probably UCS4), one name for the canonical external encoding (probably locale dependent). - Then redefine the old IO API in terms of the new API and appropriate conversions. While we're at it, do away with the annoying CR/LF problem on Windows, this should simply be part of the local encoding. This way file can always be opened as binary, hSetBinary can be dropped. (This won't wont on ancient platforms where text files and binary files are genuinely different, but these are probably not interesting anyway.) The same thoughts apply to filenames. Make them [Word8] and convert explicitly. By the way, I think a path should be a list of names (that is of type [[Word8]]) and the library would be concerned with putting in the right path separator. Add functions to read and show pathnames in the local conventions and we'll never need to worry about path separators again.

There are limits to the extent to which this can be achieved. E.g. what happens if you set the encoding to UTF-8, then call getDirectoryContents for a directory which contains filenames which aren't valid UTF-8 strings?

Well, then you did something stupid, didn't you? If you don't know the encoding you shouldn't decode anything. That's a strong point against any implicit decoding, I think. Also, if efficiency is a concern, lists probably shouldn't be passed between filesystem operations and iconv. I think, we need a better representation here (like PackedString for Word8), not a convoluted API. Regards, Udo. -- If Perl is the solution, you're solving the wrong problem. -- Erik Naggum

Udo Stenzel writes:

I'd like to see the following:

- Duplicate the IO library. The duplicate should work with [Byte] everywhere where the old library uses String. Byte is some suitable unsigned integer, on most (all?) platforms this will be Word8

- Provide an explicit conversion between encodings. A simple conversion of type [Word8] -> String would suit me, iconv would provide all that is needed.

I like this idea, but I say there should be a bit-oriented layer beneath everything. Reading a text file, for example, can be interpreted at three levels: 1. bit string 2. octet string 3. character string Most of the work involves moving between 2 and 3 but there's no reason to move from 1 to 2 automatically, either. In any case, if the functions compose well, people who only care about octets don't have to see the bit-string level. For example, this progression makes more sense for UTF-16 files: 1. bit string 2. Word16 string 3. character string -- David Menendez | "In this house, we obey the laws http://www.eyrie.org/~zednenem | of thermodynamics!"

Glynn Clements writes:

David Menendez wrote:

...

...

I'd like to see the following:

- Duplicate the IO library. The duplicate should work with [Byte] everywhere where the old library uses String. Byte is some suitable unsigned integer, on most (all?) platforms this will be Word8

- Provide an explicit conversion between encodings. A simple conversion of type [Word8] -> String would suit me, iconv would provide all that is needed.

I like this idea, but I say there should be a bit-oriented layer beneath everything.

The byte stream is inherent, as that's (usually) what the OS gives you. Everything else is synthesised.

I was unclear. I meant the bit layer would be beneath everything conceptually. On today's machines, it would be implemented in terms of a byte stream and the "conversion" to the byte stream type would get compiled away. -- David Menendez http://www.eyrie.org/~zednenem/

Henning Thielemann writes:

I even plead for an abstract data type FilePath which supports operations like 'enter a directory', 'go one level higher' and so on.

Beware of Common Lisp history: http://www.gigamonkeys.com/book/practical-a-portable-pathname-library.html "As we discussed in chapter TK, Common Lisp provides an abstraction, the pathname, that is supposed to insulate us from the details of how different OS's and file systems name files. However the gods of abstraction give and they take away. While, with a bit of care, pathnames can be used to write code that will be portable between different OS's it can, ironically, be quite tricky to write pathname code that that will be portable between different Common Lisp implementations on the *same* OS. The root of the problem is that the pathname abstraction was designed to represent file names on a wide variety of file systems. The set of file systems still in general use at the time Common Lisp was being standardized was much more variegated than the set of file systems that are commonly used now. Unfortunately, by making pathnames abstract enough to account for a wide variety of file systems, Common Lisp's designers left implementors with a fair number of choices to be made when mapping the pathname abstraction onto a particular file system. Consequently different implementors, each implementing the pathname abstraction for the same file system, by making different choices at a few key junctions, could end up with conforming implementations that nonetheless provide different behavior for several of the main pathname-related functions." -- __("< Marcin Kowalczyk \__/ qrczak@knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/

On Wed, 15 Sep 2004, Glynn Clements wrote:

Henning Thielemann wrote:

...

I even plead for an abstract data type FilePath which supports operations like 'enter a directory', 'go one level higher' and so on.

Are you referring to "pure" operations on the FilePath, e.g. appending and removing entries?

Yep, thus the signature should be like parentDirectory :: FileName -> FilePath -> FilePath

That's reasonable enough. But it needs to be borne in mind that there's a difference between:

setCurrentDirectory ".." and: dir <- getCurrentDirectory setCurrentDirectory $ parentDirectory dir

[where parentDirectory is a pure FilePath -> FilePath function.]

if the last component in the path is a symlink.

If you want to make FilePath an instance of Eq, the situation gets much more complicated.

I agree, though these are not problems introduced by the abstraction of FilePath but they already exist for FilePath = String.

Gabriel Ebner wrote:

...

...

3. The default encoding is settable from Haskell, defaults to ISO-8859-1.

Agreed.

So every haskell program that did more than just passing raw bytes From stdin to stdout should decode the appropriate environment variables, and set the encoding by itself?

This statement is too restrictive. Passing bytes isn't limited to stdin->stdout, and there's no reason why setting the encoding needs to be any more involved than e.g. "setLocaleEncoding". If you change it to:

So every haskell program that did more than just passing raw bytes ... should ... set the encoding by itself?

then the answer is yes.

IMO that's too much of redundancy, the RTS should actually do that.

The RTS doesn't know the encoding. Assuming that the data will use the locale's encoding will be wrong too often.

...

There are limits to the extent to which this can be achieved. E.g. what happens if you set the encoding to UTF-8, then call getDirectoryContents for a directory which contains filenames which aren't valid UTF-8 strings?

Then you _seriously_ messed up. Your terminal would produce garbage, Nautilus would break, ...

Like so many other people, you're making an argument based upon fiction (specifically, that you have a closed world where everything always uses the same encoding) then deeming anyone who is unable to maintain the fiction to be "wrong".

...

...

5. The default encoding is settable from Haskell, defaults to the locale encoding.

I feel that the default encoding should be one whose decoder cannot fail, e.g. ISO-8859-1. You should have to explicitly request the use of the locale's encoding (analogous to calling setlocale(LC_CTYPE, "") at the start of a C program; there's a good reason why C doesn't do this without being explicitly told to).

So that any haskell program that doesn't call setlocale and outputs anything else than US-ASCII will produce garbage on an UTF-8 system?

No. If a program just passes bytes around, everything will work so long as the inputs use the encoding which the outputs are assumed to use. And if the inputs aren't in the "correct" encoding, then you have to deal with encodings manually regardless of the default behaviour.

...

Actually, the more I think about it, the more I think that "simple, stupid programs" probably shouldn't be using Unicode at all.

Care to give any examples? Everything that has been mentioned until now would break with an UTF-8 locale: - ls (sorting would break), - env (sorting too)

Sorting according to codepoints inevitably involves decoding. However, getting the order wrong is usually considered less problematic than failing outright.

...

I.e. Char, String, string literals, and the I/O functions in Prelude, IO etc should all be using bytes,

I don't want the same mess as in C, where strings and raw data are the very same.

Tough. You already have it, and will do for the foreseeable future. Many existing APIs (including the core Unix API), protocols and file formats are defined in terms of byte strings with no encoding specified or implied.

Haskell has a nice type system and nicely defined types for binary data ([Word8]) and for Strings (String), why don't use it?

I'd like to. But many of the functions which provide or accept binary data (e.g. FilePath) insist on represent it using Strings.

...

with a distinct wide-character API available for people who want to make the (substantial) effort involved in writing (genuinely) internationalised programs.

If you introduce an entirely new "i18n-only" API, then it'll surely become difficult. :-)

I18N is inherently difficult. Lots of textual data exists in lots of different encodings, and the encoding is frequently unspecified. It would be easier if we had a closed world where only one encoding was ever used. But we don't, and pretending that we do doesn't make it so.

...

Anything that isn't ISO-8859-1 just doesn't work for the most part, and anyone who wants to provide real I18N first has to work around the pseudo-I18N that's already there (e.g. convert Chars back into Word8s so that they can decode them into real Chars).

One more reason to fix the I/O functions to handle encodings and have a seperate/underlying binary I/O API.

The problem is that we also need to fix them to handle *no encoding*. Also, binary data and text aren't disjoint. Everything is binary; some of it is *also* text.

...

Oh, and because bytes are being stored in Chars, the type system won't help if you neglect to decode a string, or if you decode it twice.

Yes, that's the problem with the current approach, i.e. that there's no easy way get a list of Word8's out of a handle.

Or out of getDirectoryContents, getArgs, getEnv etc. Or to pass a list of Word8s to a handle, or to openFile, getEnv etc.

...

...

The current ISO-8859-1 assumption is also wrong. A program written in Haskell which sorts strings would break for non-ASCII letters even now that they are ISO-8859-2 unless specified otherwise.

1. In that situation, you can't avoid the encoding issues. It doesn't matter what the default is, because you're going to have to set the encoding anyhow.

Why do you always want me to set the encoding? That should be the job of the RTS.

Because you might know the encoding, and the RTS doesn't. The locale is a fallback mechanism, for the situation where you *need* an encoding but one hasn't been specified by other means.

...

2. If you assume ISO-8859-1, you can always convert back to Word8

If I want a list of Word8's, then I should be able to get them without extracting them from a string.

The point is that, currently, you can't. Nothing in the core Haskell98 API actually uses Word8, it all uses Char/String.

...

then re-decode as UTF-8. If you assume UTF-8, anything which is neither UTF-8 nor ASCII will fail far more severely than just getting the collation order wrong.

If I use String's to handle binary data, then I should expect things to break. If I want to get text, and it's not in the expected encoding, then the user has messed up.

Or maybe the expectation is incorrect.

...

Well, my view is essentially that files should be treated as containing bytes unless you explicitly choose to decode them, at which point you have to specify the encoding.

Why do you always want me to _manually_ specify an encoding?

Because we don't have an "oracle" which will magically determine the encoding for you.

If I want bytes, I'll use the (currently being discussed, see beginning of this thread) binary I/O API, if I want String's (i.e. text), I'll use the current I/O API (which is pretty text-orientated anyway, see hPutStrLn, hGetLine, ...).

If you want text, well, tough; what comes out most system calls and core library functions (not just read()) are bytes. There isn't any magic wand which will turn them into characters without knowing the encoding.

...

completely new wide-character API for those who wish to use it.

Which would make it horrendously difficult to do even basic I18N.

Why?

...

That gets the failed attempt at I18N out of everyone's way with a minimum of effort and with maximum backwards compatibility for existing code.

If existing code, expects String's to be just a list of bytes, it's _broken_.

I know. That's what I'm saying. The problem is that the broken "code" is the Haskell98 API.

String's are a list of unicode characters, [Word8] is a list of bytes.

And what comes out of (and goes into) most core library functions is the latter. -- Glynn Clements

Gabriel Ebner wrote:

...

The RTS doesn't know the encoding. Assuming that the data will use the locale's encoding will be wrong too often.

If the program wants to get bytes, it should get bytes explicitly, not some sort of pseudo-Unicode String.

Er, that's what I've been saying. And most programs should be getting filenames as bytes.

...

Like so many other people, you're making an argument based upon fiction (specifically, that you have a closed world where everything always uses the same encoding) then deeming anyone who is unable to maintain the fiction to be "wrong".

Everything's fine here with LANG=de_AT.utf8. And I can't recall having any problems with it. But well, YMMV.

So either you never encounter Latin1 files or the programs aren't trying to decode them. Bear in mind that the standard libraries don't automatically decode everything according to the locale's encoding. A lot of programs completely ignore the locale, and many of those which use it for something don't decode strings into wide strings.

...

No. If a program just passes bytes around, everything will work so long as the inputs use the encoding which the outputs are assumed to use. And if the inputs aren't in the "correct" encoding, then you have to deal with encodings manually regardless of the default behaviour.

The only programs that just pass bytes around that come to mind are basic Unix utilities. Basically everything else will somehow process the data.

I would suggest that most programs which deal with filenames merely pass them around. I.e. read bytes from argv or the environment or files, and pass the bytes to open() etc. And when programs do process filenames, the processing is usually trivial, and not influenced by the encoding, e.g. appending or removing directories or extensions.

...

Tough. You already have it, and will do for the foreseeable future. Many existing APIs (including the core Unix API), protocols and file formats are defined in terms of byte strings with no encoding specified or implied.

Guess why I like Haskell (the language; the implementations are not up to that ideal yet).

You're missing the point. Haskell is implemented upon those existing APIs, and Haskell programs need to understand those protocols and file formats. Nothing that Haskell (or an implementation thereof) does can make the issues go away.

...

I'd like to. But many of the functions which provide or accept binary data (e.g. FilePath) insist on represent it using Strings.

Good point. Adding functions that accept bytes instead of strings would be a major undertaking.

Which is why I'm suggesting changing Char to be a byte, so that we can have the basic, robust API now and wait for the more advanced API, rather than having to wait for a usable API while people sort out all of the issues.

...

I18N is inherently difficult. Lots of textual data exists in lots of different encodings, and the encoding is frequently unspecified.

That's the problem with the current API. You can neither easily read/write bytes nor strings in a specified encoding.

No, that's the problem with reality. It has nothing to do with Haskell beyond the issue of whether Haskell is based upon reality or fiction.

...

The problem is that we also need to fix them to handle *no encoding*.

That's binary data. (assuming you didn't want to say 'unknown')

Yes. Filenames are binary data; environment strings are binary data; argv[] is binary data. They may *also* be text, but if they are, the encoding is, in general, unknown.

...

Also, binary data and text aren't disjoint. Everything is binary; some of it is *also* text.

Simon's new-io proposal does this very nicely. Stdin is by default a binary stream and you can obtain a TextInputStream for it using either the locale's encoding or a specified encoding. That's the way I'd like it to be.

Yes. From what I've seen of it, it's basically the right thing, so far as it goes, which unfortunately isn't that far. The issues go far beyond reading and writing streams. The problem is that this *isn't* the Haskell98 API; it isn't even included in any existing implentation.

...

Or out of getDirectoryContents, getArgs, getEnv etc. Or to pass a list of Word8s to a handle, or to openFile, getEnv etc.

That's a real issue. Adding new functions with a bin- is the only solution that comes to my mind.

Well, the other obvious solution is changing the existing functions to use Word8s (obviously, we want a better name, e.g. Byte or Char, or even just CChar) and make the new functions use wide characters. It isn't as if the existing functions actually deal with anything other than bytes. You never get anything other than Latin1 from a system function which returns Char (or IO Char), and passing anything which isn't Latin1 to such a function results in it being silently cast to a byte.

...

Because we don't have an "oracle" which will magically determine the encoding for you.

That "oracle" is called locale setting. If I want to read text and can't determine the encoding by other ways (protocol spec, ...), then it's what the user set his locale setting to.

No. An "oracle" would always get it right. The locale merely provides a fallback.

...

If you want text, well, tough; what comes out most system calls and core library functions (not just read()) are bytes.

Which need to be interpreted by the program depending on where these bytes come from.

They don't necessarily need to be interpreted. A lot of data simply gets "routed" from one place to another. E.g. a program reads a filename from argv[i] and passes it to open(). It doesn't matter if the filename is in Klingon.

...

There isn't any magic wand which will turn them into characters without knowing the encoding.

If I know the encoding, I should be able to set it. If I don't, it's the locale setting.

If you *need* an encoding, and don't have any better information, then the locale provides a last resort. Decoding bytes according to the locale for the sake of it just adds an unnecessary failure mode.

...

...

...

completely new wide-character API for those who wish to use it.

Which would make it horrendously difficult to do even basic I18N.

Why?

Having different types for single-byte and multi-byte strings together with seperate functions to handle them (that's what I assume you mean by a new wide-character API) with single-byte strings being the preferred one (the cause of being a seperate API) would make sorting, upper/lower case testing etc. not exactly easier.

For case testing, locale-dependent sorting and the like, you need to convert to characters. [Although possibly only temporarily; you can sort a list of byte strings based upon their corresponding character strings using sortBy. This means that a decoding failure only means that the ordering will be wrong. This is essentially what happens with "ls" if you have filenames which aren't valid in the current locale.] Note: there are still situations where sorting bytes makes sense, i.e. where you only need *an* ordering rather than a specific ordering, e.g. uniq.

...

I know. That's what I'm saying. The problem is that the broken "code" is the Haskell98 API.

No, it's not broken. It just has some missing features (i.e. I/O / env functions accepting bytes instead of strings).

It's broken. Being able to represent filenames as byte strings is fundamental. Being able to convert them to or from character strings is useful but not essential. The only reason why the existing API doesn't cause serious problems is because the translation is currently hardwired to an encoding which can't fail.

...

...

String's are a list of unicode characters, [Word8] is a list of bytes.

And what comes out of (and goes into) most core library functions is the latter.

Strictly speaking, the former comes out with the semantics of the latter. :-)

By "core library functions", I was referring primarily to libc, not the Haskell library functions which were built upon them. The Haskell developers can change Haskell, they can't change libc. -- Glynn Clements

Gabriel Ebner wrote:

...

For case testing, locale-dependent sorting and the like, you need to convert to characters. [Although possibly only temporarily; you can sort a list of byte strings based upon their corresponding character strings using sortBy. This means that a decoding failure only means that the ordering will be wrong. This is essentially what happens with "ls" if you have filenames which aren't valid in the current locale.]

sortBy could only cope with single-byte encodings. Multi-byte encodings would need something else.

I think that you may have misunderstood my point. I was referring to something like this: type ByteString = [Word8] decode :: ByteString -> String decode = ... comparator :: ByteString -> ByteString comparator s1 s2 = compare (decode s1) (decode s2) sortByteStrings :: [ByteString] -> [ByteString] sortByteStrings ss = sortBy comparator ss The byte strings which are returned from sortByteStrings are the original byte strings, but the ordering will be determined by the encoding. This produces the same results as decode->sort->encode (in the cases where the latter actually works), but is more robust.

...

It's broken. Being able to represent filenames as byte strings is fundamental. Being able to convert them to or from character strings is useful but not essential. The only reason why the existing API doesn't cause serious problems is because the translation is currently hardwired to an encoding which can't fail.

Handling binary filenames is hardly fundamental. It isn't even very portable, see the posts about filename handling under modern Windows. It might be an important feature, but there are other programs out there (mostly GUIs) that expect filenames to be encoded according to the locale settings too.

It's fundamental if you want your programs to be robust. For most programs, there is no legitimate reason to refuse to read a file because of its name. A GUI program (or for that matter, a terminal) might legitimately fail to *display* a filename correctly if it can't decode it (it has to index into the font). But that isn't a reason to reject it altogether. E.g. if I create a file whose name contains control characters, most GUI programs display it incorrectly in the file selection dialog, but they still manage to open it. -- Glynn Clements