Ivan Lazar Miljenovic wrote:

John Goerzen writes:

...

ghci GHCi, version 6.12.1: http://www.haskell.org/ghc/ :? for help Loading package ghc-prim ... linking ... done. Loading package integer-gmp ... linking ... done. Loading package base ... linking ... done. Prelude> :m System.Process Prelude System.Process> runCommand "echo привет" ?@825B

Are you arguing about IO-specific stuff like this, or for all non-ASCII Strings?

I'm not sure I understand the question. I consider the behavior in System.IO to be well-documented. The behavior in System.Process is not documented at all. As I said, I'm not certain what the proper answer is, but not documenting what happens probably isn't it. Actually, the behavior of openFile when given a String with characters > 0xFF is also completely undocumented. I am not sure what it does with that. It should probably be the same as runCommand, whatever it is. -- John

Khudyakov Alexey wrote:

...

Actually, the behavior of openFile when given a String with characters > 0xFF is also completely undocumented. I am not sure what it does with that. It should probably be the same as runCommand, whatever it is.

Under unices file names are just array of bytes. There is no notion of encoding at all. It's just matter of interpretation of that array.

Quite right. One must be able to pass binary strings, which contain anything except \0 and '/' to openFile. The same goes for runCommand. I am uncomfortable, for this reason, with saying that runCommand ought to re-encode in the system locale while openFile doesn't. It is preferable to drop characters than to drop the ability to pass arbitrary binary data. So I am not sure I agree with your stance in http://hackage.haskell.org/trac/ghc/ticket/4006 -- John

...

...

...

Actually, the behavior of openFile when given a String with characters > 0xFF is also completely undocumented.  I am not sure what it does with that.  It should probably be the same as runCommand, whatever it is.

Actually, the behaviour of openFile is known to be platform-dependent. According to Simon Marlow,

Be careful with FilePaths. On Windows they are interpreted as Unicode, on Unix they are interpreted as [Word8], by taking the low 8 bits of each Char. So if you always encode FilePaths to UTF-8, that will break on Windows. Fixing FilePaths is a high priority.

The last sentence gives me some hope. http://ghcmutterings.wordpress.com/2009/09/30/heads-up-what-you-need-to-know...

But truncation makes impossible to pass non ASCII strings portably. They should be encoded there is no easy way to do so.

Actually problem is use of strings. String is sequence of _characters_ and program talk to outside world using sequence of bytes. I think that right (but impossible) way to solve this problem is to use separate data types for file path, command line arguments.

I think that Strings should be used _only_ for characters (code points). Using the same data type for encoded/truncated data is dangerous. Most of the current problems with Unicode is due to the fact that Strings could turn out to be anything (are not strictly typed from this point of view). Hence, the runtime checks and hacks like isUTF8Encoded :: String -> Bool, encodeString :: String -> String and decodeString :: String -> String... So I absolutely support that truncating is wrong. Expecting encoded data in Strings is wrong too. So the only option (except changing the standard library and introducing a new type for FilePath), is to do all necessary conversions inside openFile and similar functions.

I think there are two alternatives. One is to encode/decode strings using current locale and provide [Word8] based variants. Main problem is that seeming innocent actions like getting directory content could crash program (exception )

Actually, any IO action is unpredictible. So trying to get directory contents can produce an error (for various reasons, e.g. permission denied). If it reports an error when there are filenames not presentable in the current locale (e.g. contain invalid UTF-8 sequences in UTF-8 locale), the problem is likely to be the wrong locale settings. What's the problem with an exception? I think [Word8] variants for those who wants to deal with such cases (guess file system encoding etc.) is enough.

Another options is to provide function to encode/decode strings. This is ugly and mix strings which hold characters and string which hold bytes and completely unhaskellish but it seems there is no good solution.

This is ugly, because it's impossible to know if a String is already encoded or not. This is ugly because application code will be polluted with conditional compilation to be cross-platform (or worse, people will forget to write cross-platform code in _some_ cases).

Also truncation could have security implications. It makes almost impossible to escape dangerous characters robustly. Consider following code. This is more matter of speculations than real threat but nevertheless:

Nice example. It shows that escaping should be the last step. S.