[GHC] #10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection
#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2-rc2 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Blocked By: Test Case: T8958a | Related Tickets: #10298, #7695 Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Phab:D898 was primarily intended to fix hangs in the event that `iconv` was unavailable (namely #10298 and #7695). In addition to this fix, it also introduced self-contained handling of ANSI terminals to allow compiled executables to run in minimal environments lacking iconv. However, the behavior that the patch introduced is highly suspicious. Specifically, it gives the user a UTF-8 encoding even if they requested ASCII. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/10623> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: None/Unknown | Unknown/Multiple Blocked By: | Test Case: T8958a Related Tickets: #10298, #7695 | Blocking: | Differential Revisions: -------------------------------------+------------------------------------- Description changed by bgamari: Old description:
Phab:D898 was primarily intended to fix hangs in the event that `iconv` was unavailable (namely #10298 and #7695). In addition to this fix, it also introduced self-contained handling of ANSI terminals to allow compiled executables to run in minimal environments lacking iconv.
However, the behavior that the patch introduced is highly suspicious. Specifically, it gives the user a UTF-8 encoding even if they requested ASCII.
New description: Phab:D898 was primarily intended to fix hangs in the event that `iconv` was unavailable (namely #10298 and #7695). In addition to this fix, it also introduced self-contained handling of ANSI terminals to allow compiled executables to run in minimal environments lacking iconv. However, the behavior that the patch introduced is highly suspicious. Specifically, it gives the user a UTF-8 encoding even if they requested ASCII. This has the potential to break quite a lot of code. At very least it breaks GHC's Unicode terminal detection logic, which attempts to catch an invalid character when encoding a pair of smart-quotes. Of course, this exception will never be thrown if a UTF-8 encoder is used. -- -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/10623#comment:1> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: None/Unknown | Unknown/Multiple Blocked By: | Test Case: T8958a Related Tickets: #10298, #7695 | Blocking: | Differential Revisions: Phab:1059 -------------------------------------+------------------------------------- Changes (by bgamari): * differential: => Phab:1059 -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/10623#comment:2> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: None/Unknown | Unknown/Multiple Blocked By: | Test Case: T8958a Related Tickets: #10298, #7695 | Blocking: | Differential Revisions: Phab:1059 -------------------------------------+------------------------------------- Comment (by Ben Gamari <ben@…>): In [changeset:"d69dfba4e27c4ec33459906fd87c9a56a371f510/ghc"]: {{{ #!CommitTicketReference repository="ghc" revision="d69dfba4e27c4ec33459906fd87c9a56a371f510" Fix self-contained handling of ASCII encoding D898 was primarily intended to fix hangs in the event that iconv was unavailable (namely #10298 and #7695). In addition to this fix, it also introduced self-contained handling of ANSI terminals to allow compiled executables to run in minimal environments lacking iconv. However, the behavior that the patch introduced is highly suspicious. Specifically, it gives the user a UTF-8 encoding even if they requested ASCII. This has the potential to break quite a lot of code. At very least it breaks GHC's Unicode terminal detection logic, which attempts to catch an invalid character when encoding a pair of smart-quotes. Of course, this exception will never be thrown if a UTF-8 encoder is used. Here we use the `char8` encoding to handle requests for ASCII encodings in the event that we find iconv to be non-functional. Fixes #10623. Test Plan: Validate with T8959a Reviewers: rwbarton, hvr, austin, hsyl20 Subscribers: thomie Differential Revision: https://phabricator.haskell.org/D1059 GHC Trac Issues: #10623 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/10623#comment:3> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: closed Priority: normal | Milestone: Component: Compiler | Version: 7.10.2-rc2 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: None/Unknown | Unknown/Multiple Blocked By: | Test Case: T8958a Related Tickets: #10298, #7695 | Blocking: | Differential Revisions: Phab:1059 -------------------------------------+------------------------------------- Changes (by bgamari): * status: new => closed * resolution: => fixed Comment: A version of this is also present in the ghc-7.10 branch as 677552f21690761b89255d05e42976679be4d374. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/10623#comment:4> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.2 Component: Compiler | Version: 7.10.2-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: T8958a Blocked By: | Blocking: Related Tickets: #10298, #7695 | Differential Revisions: | Phab:D1059, Phab:D1085 -------------------------------------+------------------------------------- Changes (by rwbarton): * status: closed => new * priority: normal => high * differential: Phab:1059 => Phab:D1059, Phab:D1085 * resolution: fixed => * milestone: => 7.10.2 Comment: I realized that at least hsyl20 and perhaps others actually intend to run their programs in the (what I would consider to be broken) setting of a chroot without a working iconv installation. In that case we really shouldn't use `char8` when asked for an ASCII locale. If the programmer wants `char8` they can use it explicitly. I implemented an ASCII encoding in Phab:D1085. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/10623#comment:5> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.2 Component: Compiler | Version: 7.10.2-rc2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: T8958a Blocked By: | Blocking: Related Tickets: #10298, #7695 | Differential Revisions: | Phab:D1059, Phab:D1085 -------------------------------------+------------------------------------- Comment (by Ben Gamari <ben@…>): In [changeset:"dbe6dac96543f426297a59d8d16c3f5afacf42d4/ghc" dbe6dac/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="dbe6dac96543f426297a59d8d16c3f5afacf42d4" When iconv is unavailable, use an ASCII encoding to encode ASCII D898 and D1059 implemented a fallback behavior to handle the case that the end user's iconv installation is broken (typically due to running inside a chroot in which the necessary locale files and/or gconv modules have not been installed). In this case, if the program requests an ASCII locale, GHC's char8 encoding is used rather than the program failing. However, silently mangling data like char8 does when the programmer did not ask for it is poor behavior, for reasons described in D1059. This commit implements an ASCII encoding and uses it in the fallback case when iconv is unavailable and the user has requested ASCII. Test Plan: Added tests for the encodings defined in Latin1. Also, manually ran a statically-linked executable of that test in a chroot and the tests passed (up to the ones that call mkTextEncoding "LATIN1", since there is no fallback from iconv for that case yet). Reviewers: austin, hvr, hsyl20, bgamari Reviewed By: hsyl20, bgamari Subscribers: thomie Differential Revision: https://phabricator.haskell.org/D1085 GHC Trac Issues: #7695, #10623 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/10623#comment:6> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: closed Priority: high | Milestone: 7.10.2 Component: Compiler | Version: 7.10.2-rc2 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: T8958a Blocked By: | Blocking: Related Tickets: #10298, #7695 | Differential Revisions: | Phab:D1059, Phab:D1085 -------------------------------------+------------------------------------- Changes (by bgamari): * status: new => closed * resolution: => fixed -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/10623#comment:7> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: closed Priority: high | Milestone: 7.10.2 Component: Compiler | Version: 7.10.2-rc2 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: T8958a Blocked By: | Blocking: Related Tickets: #10298, #7695 | Differential Revisions: | Phab:D1059, Phab:D1085 -------------------------------------+------------------------------------- Comment (by Thomas Miedema <thomasmiedema@…>): In [changeset:"89060374ea0242bc2d38425ba63f33b5189325b4/ghc" 89060374/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="89060374ea0242bc2d38425ba63f33b5189325b4" Testsuite: mark encoding005 expect_broken(#10623) on Windows }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/10623#comment:8> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.2 Component: Compiler | Version: 7.10.2-rc2 Resolution: | Keywords: Operating System: Windows | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: T8958a Blocked By: | Blocking: Related Tickets: #10298, #7695 | Differential Revisions: | Phab:D1059, Phab:D1085 -------------------------------------+------------------------------------- Changes (by thomie): * status: closed => new * resolution: fixed => * os: Unknown/Multiple => Windows Comment: The test `encoding005` is failing on Windows with an unknown ASCII encoding error. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/10623#comment:9> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.3 Component: Compiler | Version: 7.10.2-rc2 Resolution: | Keywords: Operating System: Windows | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: T8958a Blocked By: | Blocking: Related Tickets: #10298, #7695 | Differential Revisions: | Phab:D1059, Phab:D1085 -------------------------------------+------------------------------------- Changes (by thoughtpolice): * milestone: 7.10.2 => 7.10.3 Comment: Moving out to 7.10.3. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/10623#comment:10> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.3 Component: Compiler | Version: 7.10.2-rc2 Resolution: | Keywords: Operating System: Windows | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: T8958a Blocked By: | Blocking: Related Tickets: #10298, #7695 | Differential Revisions: | Phab:D1059, Phab:D1085 -------------------------------------+------------------------------------- Comment (by rwbarton): It's not a regression, just a new test (I added it in dbe6dac96543f426297a59d8d16c3f5afacf42d4). The test uses `mkTextEncoding "ASCII"` which clearly (if you read `GHC.IO.Encoding.mkTextEncoding'`) won't work on Windows. It didn't work in the past either except between e28462de700240288519a016d0fe44d4360d9ffd and d69dfba4e27c4ec33459906fd87c9a56a371f510. Should we make it work, i.e., handle encodings in our hard-coded list as ASCII on Windows too? Personally I don't see why not. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/10623#comment:11> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: high | Milestone: 8.0.1 Component: Compiler | Version: 7.10.2-rc2 Resolution: | Keywords: Operating System: Windows | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: T8958a Blocked By: | Blocking: Related Tickets: #10298, #7695 | Differential Rev(s): Phab:D1059, Wiki Page: | Phab:D1085 -------------------------------------+------------------------------------- Changes (by bgamari): * milestone: 7.10.3 => 8.0.1 Comment: The issue mentioned in comment:11 is still outstanding. Punting to 8.0.1. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/10623#comment:12> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.1 Component: Test Suite | Version: 7.10.2-rc2 Resolution: | Keywords: Operating System: Windows | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: T8958a Blocked By: | Blocking: Related Tickets: #10298, #7695 | Differential Rev(s): Phab:D1059, Wiki Page: | Phab:D1085 -------------------------------------+------------------------------------- Changes (by thomie): * priority: high => normal * component: Compiler => Test Suite -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/10623#comment:13> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.2 Component: Test Suite | Version: 7.10.2-rc2 Resolution: | Keywords: Operating System: Windows | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: T8958a Blocked By: | Blocking: Related Tickets: #10298, #7695 | Differential Rev(s): Phab:D1059, Wiki Page: | Phab:D1085 -------------------------------------+------------------------------------- Changes (by thoughtpolice): * milestone: 8.0.1 => 8.0.2 Comment: Punting to 8.0.2, since this just affects the testsuite. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/10623#comment:14> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: patch Priority: normal | Milestone: 8.0.2 Component: Test Suite | Version: 7.10.2-rc2 Resolution: | Keywords: Operating System: Windows | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: T8958a Blocked By: | Blocking: Related Tickets: #10298, #7695 | Differential Rev(s): Phab:D1059, Wiki Page: | Phab:D1085, Phab:D2262 -------------------------------------+------------------------------------- Changes (by thomie): * status: new => patch * differential: Phab:D1059, Phab:D1085 => Phab:D1059, Phab:D1085, Phab:D2262 Comment: I have another patch for this series. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/10623#comment:15> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: patch Priority: normal | Milestone: 8.0.2 Component: Test Suite | Version: 7.10.2-rc2 Resolution: | Keywords: Operating System: Windows | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: T8958a Blocked By: | Blocking: Related Tickets: #10298, #7695 | Differential Rev(s): Phab:D1059, Wiki Page: | Phab:D1085, Phab:D2262 -------------------------------------+------------------------------------- Comment (by Thomas Miedema <thomasmiedema@…>): In [changeset:"1319363f7633c441bdb1f659616d71ecd700076d/ghc" 1319363f/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="1319363f7633c441bdb1f659616d71ecd700076d" Always use native-Haskell de/encoders for ASCII and latin1 This fixes test encoding005 on Windows (#10623). Reviewed by: austin, bgamari Differential Revision: https://phabricator.haskell.org/D2262 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/10623#comment:16> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#10623: Handling of ASCII encodings introduced in D898 breaks Unicode terminal detection -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: closed Priority: normal | Milestone: 8.2.1 Component: Test Suite | Version: 7.10.2-rc2 Resolution: fixed | Keywords: Operating System: Windows | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: T8958a, | libraries/base/tests/IO/encoding005 Blocked By: | Blocking: Related Tickets: #10298, #7695 | Differential Rev(s): Phab:D1059, Wiki Page: | Phab:D1085, Phab:D2262 -------------------------------------+------------------------------------- Changes (by thomie): * testcase: T8958a => T8958a, libraries/base/tests/IO/encoding005 * status: patch => closed * resolution: => fixed * milestone: 8.0.2 => 8.2.1 -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/10623#comment:17> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
participants (1)
-
GHC