On Wed, Sep 12, 2007 at 05:19:22PM +0200, Andrea Rossato wrote:

And so it's my job to convert it in what I need. Luckily I've just discovered (and now I'm reading) some of John Meacham's code on locale. This is going to be very helpful (unfortunately I don't see Licenses coming with HsLocale, but if I'm reading correctly there is something like this in Riot - and this was BSD3 released).

it is BSD3. in general, pretty much everything I write is BSD3 except for large projects as a whole which get GPL>=2. Though I am more than happy to BSD3 any incidentally useful parts of my projects that others would find useful. John -- John Meacham - ⑆repetae.net⑆john⑈

On 12/09/2007, Seth Gordon wrote:

I � Unicode.

Was it intentional that the central character appears as a little '?', even though the aleph on the line above worked? Either way it would be very amusing, but for different reasons... D

On Wed, Sep 12, 2007 at 11:16:25AM -0400, Seth Gordon wrote:

It appears that in spite of the locale definition, hGetContents is treating each byte as a separate character without translating the multi-byte sequences *from* UTF-8, and then putStrLn sends each of those bytes to standard output without translating the non-ASCII characters *to* UTF-8. So the second line of your program's output is correct...but only by accident.

that's it indeed. As I said in the message I've just sent, I've read that the String/CString conversion is automatically done in ISO-8859-1, so "èèè", which are 6 bytes in utf-8, are translated into 6 iso-8859-1 characters. What puzzles me is the behavior of putStrLn. Thanks for your time. Andrea

On Wed, Sep 12, 2007 at 11:40:11AM -0400, Seth Gordon wrote:

The Unix utility "od" can be very helpful in figuring out problems like this.

Thanks for pointing me to "od", I didn't know it.

[*]At least on my computer, I get the same result *even if* I change "LANG" from "en_US.utf8" to "C".

As far as I understand it is the terminal emulator responsible for translating the bytes to characters. If I run it in a console I get abA"A"A" (sort of) no matter what my LANG is - 8 single 8 -bit characters. Cheers, Andrea

David Benbennick wrote:

On 9/12/07, Andrea Rossato wrote:

...

If I run it in a console I get abA"A"A" (sort of) no matter what my LANG is - 8 single 8 -bit characters.

It's possible to set your Linux console to grok UTF8. I don't remember the details, but I'm sure you can Google for it.

By the way, does anyone know The Right Way to deal with UTF-8 in Haskell? I.e., take that 8 byte UTF-8 string and convert it to a 5 character Unicode string (so it can be manipulated)?

There is no UTF8 decode support in the standard libraries. There are some contributed libraries which can do it. Data.CompactString is one. Jules

Hello Stefan, Thursday, September 13, 2007, 4:40:17 AM, you wrote:

I'm pretty sure Hugs does the right thing. NHC is probably broken. In any case, we already have hGetBuf / hPutBuf in the standard base libaries for raw binary IO, so code that uses getChar for bytes really has no excuse. We can and should fix the bug.

are you ever heard about backward compatibility? :/ -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

On Thu, Sep 13, 2007 at 12:06:15PM +0200, Ketil Malde wrote:

On Wed, 2007-09-12 at 17:40 -0700, Stefan O'Rear wrote:

...

On Thu, Sep 13, 2007 at 12:23:33AM +0000, Aaron Denney wrote:

...

Unfortunately, at this point it is a well entrenched bug, and changing the behaviour will undoubtedly break programs. ... There should be another system for getting the exact bytes in and out (as Word8s, say, rather than Chars),

...

I'm pretty sure Hugs does the right thing.

..which makes me wonder what the right thing actually is?

Since IO on Unix (or at least on Linux) consists of bytes, I don't see how a Unicode-only interface is ever going to do the 'right thing' for all people.

I never said it was Unicode-only. hGetBuf / hPutBuf - Raw Word8 access getChar etc - Uses locale info Stefan

On 2007-09-13, Stefan O'Rear wrote:

In any case, we already have hGetBuf / hPutBuf in the standard base libaries for raw binary IO, so code that uses getChar for bytes really has no excuse.

Except, of course, that hGetBuf and hPutBuf are (a) allocating the memory for the buffers is a pain (does it require the FFI?) (b) are something of a pain to use, requiring explicitly managing what's valid in these buffers (though a wrapper only need be written once) (c) while in the "standard base libraries" are not in the report or library report. i.e. there's no guarantee that a conforming Haskell implementation will have them. It'd be silly for an implementation to not support them, of course, but... The ByteString library at least fixes (a) and (b). -- Aaron Denney -><-

On Thu, Sep 13, 2007 at 11:07:03AM +0200, Stephane Bortzmeyer wrote:

On Thu, Sep 13, 2007 at 12:23:33AM +0000, Aaron Denney wrote a message of 76 lines which said:

...

the characters read and written should correspond to the native environment notions and encodings. These are, under Unix, determined by the locale system.

Locales, while fine for things like the language of the error messages or the format to use to display the time, are *not* a good solution for things like file names and file contents.

Even on a single Unix machine (without networking), there are *several* users. Using the locale to find out the charset used for a file name won't work if these users use different locales.

Yes indeed. And I find it a real mess. And I don't see any way out.

Same thing for file contents. The charset used must be marked in the file (XML...) or in the metadata, somehow. Otherwise, there is no way to exchange files or even to change the locale (if I switch from Latin1 to UTF-8, what do my files become?)

Ok, you are perfectly right, but we live in an imperfect world and we must come up with a solution. In my case I'm developing this prompt for xmonad and a Chinese user wants directory and file names to be correctly displayed. What else can I do but using locale technologies? This is something I don't know. The code below is not perfect but it works to some extent. Nonetheless, if you have 2 users using an iso-8859-1 locale the first and utf-8 one the second, non ascii characters in file names of the first users will produce invalid character sequences for the second users. The reverse will work, though. I'm still puzzled and still find the thread title appropriate. Thanks for your kind attention. Andrea The locale aware version of the previous code (needs hsc2hs) {-# OPTIONS -fglasgow-exts #-} import Prelude hiding (catch) import System.Process import System.IO import Control.Monad import System.Directory import Foreign import Foreign.C import Data.Char import Control.Exception runProcessWithInput cmd args input = do (pin, pout, perr, ph) <- runInteractiveProcess cmd args Nothing Nothing hPutStr pin input hClose pin output <- hGetContents pout when (output==output) $ return () hClose pout hClose perr waitForProcess ph return output main = do setupLocale l <- fmap lines $ runProcessWithInput "/bin/bash" [] "ls ab*\n" l' <- mapM fromLocale l l'' <- mapM toLocale l' putStrLn (show l') mapM_ putStrLn l'' mapM_ (putStrLn . show . length) l' -- This code comes from John Meacham's HsLocale -- http://repetae.net/john/repos/HsLocale/ toLocale :: String -> IO String toLocale s = catch (stringToBytes s >>= return . map (chr . fromIntegral)) (const $ return "invalid character sequence") fromLocale :: String -> IO String fromLocale s = bytesToString (map (fromIntegral . ord) s) `catch` \_ -> return "invalid character sequence" stringToBytes :: String -> IO [Word8] stringToBytes cs = (withIConv "" "UTF-32" $ \ic -> convertRaw ic cs) bytesToString :: [Word8] -> IO String bytesToString xs = (withIConv "UTF-32" "" $ \ic -> convertRaw ic xs) >>= return . f where f ('\65279':xs) = xs -- discard byte order marker f xs = xs newtype IConv = IConv (#type intptr_t) deriving(Num,Eq,Show) foreign import ccall unsafe "iconv.h iconv_open" iconv_open :: Ptr CChar -> Ptr CChar -> IO IConv foreign import ccall unsafe "iconv.h iconv_close" iconv_close :: IConv -> IO CInt foreign import ccall unsafe "iconv.h iconv" iconv :: IConv -> Ptr (Ptr CChar) -> Ptr CSize -> Ptr (Ptr CChar) -> Ptr CSize -> IO CInt withIConv :: String -> String -> (IConv -> IO a) -> IO a withIConv to from action = bracket open close action where close ic = throwErrnoIfMinus1_ "iconv_close" (iconv_close ic) open = throwErrnoIfMinus1 "iconv_open" iopen iopen = do withCAString to $ \to -> do withCAString from $ \from -> do iconv_open to from convertRaw :: (Storable a, Storable b) => IConv -> [a] -> IO [b] convertRaw ic xs = do with (fromIntegral $ sizeOf (head xs) * length xs) $ \inptrSz -> do withArray xs $ \arr -> do with (castPtr arr) $ \inptr -> do allocaBytes (1024) $ \outptr -> do with outptr $ \outptrptr -> do with 1024 $ \outptrSz -> do let outSz = fromIntegral $ sizeOf $ unsafePerformIO (peek outptr) let go = do ret <- iconv ic inptr inptrSz (castPtr outptrptr) outptrSz err <- getErrno case (ret,err) of (-1,_) | err == e2BIG -> do oz <- peek outptrSz x <- peekArray ((1024 - fromIntegral oz) `div` outSz) (castPtr outptr) poke outptrptr outptr poke outptrSz 1024 y <- go return $ x ++ y (-1,_) -> throwErrno "iconv" (_,_) -> do oz <- peek outptrSz peekArray ((1024 - fromIntegral oz) `div` outSz) outptr go #include foreign import ccall unsafe "locale.h setlocale" setlocale :: CInt -> CString -> IO CString setupLocale :: IO () setupLocale = withCString "" $ \s -> do setlocale (#const LC_ALL) s return ()