Issue 348 in xmonad: Encoding task force
Status: New Owner: ---- Labels: Type-Enhancement New issue 348 by SpencerJanssen: Encoding task force http://code.google.com/p/xmonad/issues/detail?id=348 xmonad is sloppy about strings and encoding. We should go through all of xmonad and make sure all code follows these guidelines: * All String ([Char]) values should be lists of Unicode code points, and should NOT be in UTF-8 or any other encoded form. * All input that creates String values should decoded from bytes in whatever encoding to a lists of code points (this means System.IO and proper locale settings in GHC 6.12, System.IO.UTF8 in older) * All output that takes String values should encode the code points to bytes in whatever encoding (again System.IO in 6.12, utf8-string otherwise) * All non-string data (lists of bytes, encoded strings, etc.) should be stored at types other than [Char] If anybody wants to spearhead this, feel free to come forward. Alternatively we could have maintainers self-certify their modules? -- You received this message because you are listed in the owner or CC fields of this issue, or because you starred this issue. You may adjust your issue notification preferences at: http://code.google.com/hosting/settings
Comment #1 on issue 348 by alexey.skladnoy: Encoding task force http://code.google.com/p/xmonad/issues/detail?id=348 Modules which import Codec.Binary.UTF8.*: * XMonad.Hooks.EwmhDesktops OK used to encode desktop names. * XMonad.Actions.Search OK used to encode URLs * XMonad.Util.XSelection Seems to be OK. Documentation is outdated and shoud be updated * XMonad.Util.Font Doesn't use function but jus rexport them under different names: `encodeOutout' and `decodeInput' encodeOutput is only used in X.Prompt.Shell and X.Hooks.DynamicLog I think it would be better to remove encodeOutput function and just import encodeString from utf8-string. decodeInput is used in X.Prompt.Shell and X.Promt modules. X.Promt deserve special investigation since it stopped correcty work with unicode some time ago. == Unicode bugs == Here is list of found bugs related to unicode: X.Hooks.DynamicLog: xmonadPropLog' Function sets UTF8_STRING atom without properly encoding it. Actually it require utf8-encoded string as input. X.Hooks.DynamicLog: Returns utf8-encoded string. X.Hooks.SetWMName: setWMName Doesn't properly encode UTF8_STRING atom. Comment from function says: "now only accepts latin1 names to eliminate dependency on utf8 encoder" Since utf8-string dependency is now madatory this is clearly outdated X.Prompt Some time ago it suddently begun to mangle non-ASCII input. Attachments: Search-unicode-clean.dpatch 40.7 KB
Comment #2 on issue 348 by alexey.skladnoy: Encoding task force http://code.google.com/p/xmonad/issues/detail?id=348 Patch above: utf8-encode string as list of bytes not list of chars
encodeOutput is only used in X.Prompt.Shell and X.Hooks.DynamicLog I
Comment #3 on issue 348 by gwern0: Encoding task force http://code.google.com/p/xmonad/issues/detail?id=348 think it would be better to remove encodeOutput function and just import encodeString from utf8-string.
decodeInput is used in X.Prompt.Shell and X.Promt modules. X.Promt deserve special investigation since it stopped correcty work with unicode some time ago.
I've removed both encodeOutput and decodeInput in darcs by pushing the attached patch. Attachments: decodeInput.dpatch 49.9 KB
Comment #4 on issue 348 by gwern0: Encoding task force http://code.google.com/p/xmonad/issues/detail?id=348
Patch above: utf8-encode string as list of bytes not list of chars
I've pushed this.
Comment #5 on issue 348 by gwern0: Encoding task force http://code.google.com/p/xmonad/issues/detail?id=348
Seems to be OK. Documentation is outdated and shoud be updated
I'm assuming this is a reference to the docs saying Unicode input is unreliable? I've pushed the attached patch. Attachments: xselection.dpatch 45.4 KB
Comment #6 on issue 348 by gwern0: Encoding task force http://code.google.com/p/xmonad/issues/detail?id=348 I should probably mention that according to my copy of the config archive, no users were using encodeOutput or decodeInput, so removing them should be perfectly safe.
Comment #7 on issue 348 by ilabath: Encoding task force http://code.google.com/p/xmonad/issues/detail?id=348 Function X.H.DynamicLog.dynamicLogString :: PP -> X String encodes the resulting string to utf8, which is sloppy according to the guidelines and produces double-encoded output with ghc 6.12.3 on darcs version of xmonad+contib when outputting to a pipe. I've removed the encodeString processing, and added a cleaner equivalent to the xmonadPropLog, where I believe the output should be utf8-encoded. It fixes encoding issues for me, but I have only tested it with pipe output on my current setup. I don't know whether it will work on older versions of ghc, where output might not be automatically encoded. While fixing the log hook, I've noticed a redundant depend in the cabal file and (hopefully) made the X.U.NamedWindows.getName function more readable. I don't know what to do with those patches, so I'm just attaching them as well. Attachments: log-fix.dpatch 48.8 KB nodepend.dpatch 47.6 KB nicer-getname.dpatch 48.5 KB
Updates: Status: Patch Comment #8 on issue 348 by vogt.adam: Encoding task force http://code.google.com/p/xmonad/issues/detail?id=348 (No comment was entered for this change.)
Comment #9 on issue 348 by vej....@gmail.com: Encoding task force http://code.google.com/p/xmonad/issues/detail?id=348 Such problems still exist, at least at two places: - In XMonad.Core, the spawn function breaks Unicode strings. For instance, if I do "spawn "notify-send \"Réveille-toi !\"" the "é" character in the notification becomes "Ã(c)". - In XMonad.Util.Dmenu, the menu arguments are similarly broken if they contain such characters. In the second case, I could solve the problem by removing all "encodeString" occurrences in the function runProcessWithInput from XMonad.Util.Run (there are other occurrences of "encodeString"in XMonad.Util.Run which may cause the same problem). I suspect (but have not tried) that removing "encodeString" in spawnPID (from XMonad.Core) would solve the first problem. -- You received this message because this project is configured to send all issue notifications to this address. You may adjust your notification preferences at: https://code.google.com/hosting/settings
Comment #10 on issue 348 by vej....@gmail.com: Encoding task force http://code.google.com/p/xmonad/issues/detail?id=348 Such problems still exist, in least at two places: - In XMonad.Core, the spawn function breaks Unicode strings. For instance, if I do "spawn "notify-send \"Réveille-toi !\""" the "é" character in the notification becomes "Ã(c)". - In XMonad.Util.Dmenu, the menu arguments are similarly broken if they contain such characters. In the second case, I could solve the problem by removing all "encodeString" occurrences in the function runProcessWithInput from XMonad.Util.Run (there are other occurrences of "encodeString"in XMonad.Util.Run which may cause the same problem). I suspect (but have not tried) that removing "encodeString" in spawnPID (from XMonad.Core) would solve the first problem. -- You received this message because this project is configured to send all issue notifications to this address. You may adjust your notification preferences at: https://code.google.com/hosting/settings
Comment #11 on issue 348 by allber...@gmail.com: Encoding task force http://code.google.com/p/xmonad/issues/detail?id=348 Actually the problem here is that we fixed encoding issues once... and then ghc was changed to deal with encoding issues, and later the core libraries were modified to use ghc's encoding support, and we were left with a backward compatibility issue. Making sure that everything works properly on everything from Debian (typically with an ancient ghc that doesn't fully handle encoding) to Arch (typically bleeding edge) without either losing encoding or double-encoding is difficult. -- You received this message because this project is configured to send all issue notifications to this address. You may adjust your notification preferences at: https://code.google.com/hosting/settings
participants (1)
-
codesite-noreply＠google.com