#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: None/Unknown | Unknown/Multiple Blocked By: | Test Case: T8958a Related Tickets: #10298, #7695 | Blocking: | Differential Revisions: -------------------------------------+------------------------------------- Description changed by bgamari: Old description:

Phab:D898 was primarily intended to fix hangs in the event that `iconv` was unavailable (namely #10298 and #7695). In addition to this fix, it also introduced self-contained handling of ANSI terminals to allow compiled executables to run in minimal environments lacking iconv.

However, the behavior that the patch introduced is highly suspicious. Specifically, it gives the user a UTF-8 encoding even if they requested ASCII.

New description: Phab:D898 was primarily intended to fix hangs in the event that `iconv` was unavailable (namely #10298 and #7695). In addition to this fix, it also introduced self-contained handling of ANSI terminals to allow compiled executables to run in minimal environments lacking iconv. However, the behavior that the patch introduced is highly suspicious. Specifically, it gives the user a UTF-8 encoding even if they requested ASCII. This has the potential to break quite a lot of code. At very least it breaks GHC's Unicode terminal detection logic, which attempts to catch an invalid character when encoding a pair of smart-quotes. Of course, this exception will never be thrown if a UTF-8 encoder is used. -- -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10623#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: None/Unknown | Unknown/Multiple Blocked By: | Test Case: T8958a Related Tickets: #10298, #7695 | Blocking: | Differential Revisions: Phab:1059 -------------------------------------+------------------------------------- Comment (by Ben Gamari ): In [changeset:"d69dfba4e27c4ec33459906fd87c9a56a371f510/ghc"]: {{{ #!CommitTicketReference repository="ghc" revision="d69dfba4e27c4ec33459906fd87c9a56a371f510" Fix self-contained handling of ASCII encoding D898 was primarily intended to fix hangs in the event that iconv was unavailable (namely #10298 and #7695). In addition to this fix, it also introduced self-contained handling of ANSI terminals to allow compiled executables to run in minimal environments lacking iconv. However, the behavior that the patch introduced is highly suspicious. Specifically, it gives the user a UTF-8 encoding even if they requested ASCII. This has the potential to break quite a lot of code. At very least it breaks GHC's Unicode terminal detection logic, which attempts to catch an invalid character when encoding a pair of smart-quotes. Of course, this exception will never be thrown if a UTF-8 encoder is used. Here we use the `char8` encoding to handle requests for ASCII encodings in the event that we find iconv to be non-functional. Fixes #10623. Test Plan: Validate with T8959a Reviewers: rwbarton, hvr, austin, hsyl20 Subscribers: thomie Differential Revision: https://phabricator.haskell.org/D1059 GHC Trac Issues: #10623 }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10623#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.2 Component: Compiler | Version: 7.10.2-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: T8958a Blocked By: | Blocking: Related Tickets: #10298, #7695 | Differential Revisions: | Phab:D1059, Phab:D1085 -------------------------------------+------------------------------------- Changes (by rwbarton): * status: closed => new * priority: normal => high * differential: Phab:1059 => Phab:D1059, Phab:D1085 * resolution: fixed => * milestone: => 7.10.2 Comment: I realized that at least hsyl20 and perhaps others actually intend to run their programs in the (what I would consider to be broken) setting of a chroot without a working iconv installation. In that case we really shouldn't use `char8` when asked for an ASCII locale. If the programmer wants `char8` they can use it explicitly. I implemented an ASCII encoding in Phab:D1085. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10623#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: closed Priority: high | Milestone: 7.10.2 Component: Compiler | Version: 7.10.2-rc2 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: T8958a Blocked By: | Blocking: Related Tickets: #10298, #7695 | Differential Revisions: | Phab:D1059, Phab:D1085 -------------------------------------+------------------------------------- Changes (by bgamari): * status: new => closed * resolution: => fixed -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10623#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.2 Component: Compiler | Version: 7.10.2-rc2 Resolution: | Keywords: Operating System: Windows | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: T8958a Blocked By: | Blocking: Related Tickets: #10298, #7695 | Differential Revisions: | Phab:D1059, Phab:D1085 -------------------------------------+------------------------------------- Changes (by thomie): * status: closed => new * resolution: fixed => * os: Unknown/Multiple => Windows Comment: The test `encoding005` is failing on Windows with an unknown ASCII encoding error. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10623#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.3 Component: Compiler | Version: 7.10.2-rc2 Resolution: | Keywords: Operating System: Windows | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: T8958a Blocked By: | Blocking: Related Tickets: #10298, #7695 | Differential Revisions: | Phab:D1059, Phab:D1085 -------------------------------------+------------------------------------- Comment (by rwbarton): It's not a regression, just a new test (I added it in dbe6dac96543f426297a59d8d16c3f5afacf42d4). The test uses `mkTextEncoding "ASCII"` which clearly (if you read `GHC.IO.Encoding.mkTextEncoding'`) won't work on Windows. It didn't work in the past either except between e28462de700240288519a016d0fe44d4360d9ffd and d69dfba4e27c4ec33459906fd87c9a56a371f510. Should we make it work, i.e., handle encodings in our hard-coded list as ASCII on Windows too? Personally I don't see why not. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10623#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.1 Component: Test Suite | Version: 7.10.2-rc2 Resolution: | Keywords: Operating System: Windows | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: T8958a Blocked By: | Blocking: Related Tickets: #10298, #7695 | Differential Rev(s): Phab:D1059, Wiki Page: | Phab:D1085 -------------------------------------+------------------------------------- Changes (by thomie): * priority: high => normal * component: Compiler => Test Suite -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10623#comment:13 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: patch Priority: normal | Milestone: 8.0.2 Component: Test Suite | Version: 7.10.2-rc2 Resolution: | Keywords: Operating System: Windows | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: T8958a Blocked By: | Blocking: Related Tickets: #10298, #7695 | Differential Rev(s): Phab:D1059, Wiki Page: | Phab:D1085, Phab:D2262 -------------------------------------+------------------------------------- Changes (by thomie): * status: new => patch * differential: Phab:D1059, Phab:D1085 => Phab:D1059, Phab:D1085, Phab:D2262 Comment: I have another patch for this series. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10623#comment:15 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: closed Priority: normal | Milestone: 8.2.1 Component: Test Suite | Version: 7.10.2-rc2 Resolution: fixed | Keywords: Operating System: Windows | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: T8958a, | libraries/base/tests/IO/encoding005 Blocked By: | Blocking: Related Tickets: #10298, #7695 | Differential Rev(s): Phab:D1059, Wiki Page: | Phab:D1085, Phab:D2262 -------------------------------------+------------------------------------- Changes (by thomie): * testcase: T8958a => T8958a, libraries/base/tests/IO/encoding005 * status: patch => closed * resolution: => fixed * milestone: 8.0.2 => 8.2.1 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10623#comment:17 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler