#10907: GHC fails to read file with byte-order mark when LANG=C -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 (Parser) | Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 Type of failure: GHC doesn't work | (amd64) at all | Test Case: Blocked By: | Blocking: Related Tickets: #6016, #6037 | Differential Revisions: -------------------------------------+------------------------------------- Changes (by RyanGlScott): * Attachment "ByteOrderMark.hs" added. Minimal example -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10907 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10907: GHC fails to read file with byte-order mark when LANG=C -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 (Parser) | Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 Type of failure: GHC doesn't work | (amd64) at all | Test Case: Blocked By: | Blocking: Related Tickets: #6016, #6037 | Differential Revisions: -------------------------------------+------------------------------------- Comment (by osa1): Just tried this and it worked fine on my x86_64 Linux, with GHC 7.10.1 and HEAD. Is there anything else I need to set other than `LANG`? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10907#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10907: GHC fails to read file with byte-order mark when LANG=C -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 (Parser) | Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 Type of failure: GHC doesn't work | (amd64) at all | Test Case: Blocked By: | Blocking: Related Tickets: #6016, #6037 | Differential Revisions: -------------------------------------+------------------------------------- Comment (by nomeata): I can confirm as well. Furthermore it worked with 7.8, and it only seems to affect files with a BOM, files with other unicode characters, but no BOM, are read successfully. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10907#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10907: GHC fails to read file with byte-order mark when LANG=C -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 (Parser) | Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 Type of failure: GHC doesn't work | (amd64) at all | Test Case: Blocked By: | Blocking: Related Tickets: #6016, #6037 | Differential Revisions: -------------------------------------+------------------------------------- Comment (by nomeata): The problem seems to be `skipBOM` in `StringUtils.hs`, which switches to text mode so that `hLookAhead` is able to consume the whole BOM, instead of just the first character. But in text mode we are locale dependent. At first I thought it would make sense to stay in binary mode, but then `hLookAhead` returns just one bytes, which is not enough to detect a bom. Using `hGetChar` twice would help, but if there is no BOM, we’d have to rewind. Are we sure we can `hSeek` on all buffers that we need to? A `Word16` encoding would help. Or maybe it works well enough to force utf8 for this single `hLookAhead`. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10907#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10907: GHC fails to read file with byte-order mark when LANG=C -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 (Parser) | Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 Type of failure: GHC doesn't work | (amd64) at all | Test Case: Blocked By: | Blocking: Related Tickets: #6016, #6037 | Differential Revisions: -------------------------------------+------------------------------------- Comment (by thomie): Please go ahead and fix it yourself if you know of a good solution. Otherwise I'll put it on my list. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10907#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10907: GHC fails to read file with byte-order mark when LANG=C -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: Type: bug | Status: patch Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 (Parser) | Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 Type of failure: GHC doesn't work | (amd64) at all | Test Case: Blocked By: | Blocking: Related Tickets: #6016, #6037 | Differential Revisions: Phab:D1274 -------------------------------------+------------------------------------- Comment (by Joachim Breitner ): In [changeset:"f7fd864ce6d41cf22d25f18a0cdc5e2e9db71304/ghc" f7fd864c/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="f7fd864ce6d41cf22d25f18a0cdc5e2e9db71304" Skip a possible BOM in utf8 encoding and not the system locale, which might be something else. This fixes bug #10907. A test is added, but less useful than it could be until task #10909 is done. Differential Revision: D1274 }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10907#comment:10 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler