[GHC] #12213: Setting stdout to unbuffered causes us to switch to ascii output locale
#12213: Setting stdout to unbuffered causes us to switch to ascii output locale -------------------------------------+------------------------------------- Reporter: ezyang | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Test Suite | Version: 8.0.1 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- I had a failing test case which was producing UTF-8 output, and Python was choking on it trying to print it. I traced the problem to these lines of code in `testsuite/driver/runtests.py`: {{{ sys.stdout.flush() if PYTHON3: # in Python 3, we output text, which cannot be unbuffered sys.stdout = os.fdopen(sys.__stdout__.fileno(), "w") else: # set stdout to unbuffered (is this the best way to do it?) sys.stdout = os.fdopen(sys.__stdout__.fileno(), "w", 0) }}} On Python 2.7.11 on Linux, with `LANG=en_US.UTF-8` in the environment, if you place a `print(u"\u2018")` before these lines of code, Python will successfully print it; however, if you place it after, Python will crash: {{{ Traceback (most recent call last): File "../../driver/runtests.py", line 279, in <module> print(u"\u2018") UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position 0: ordinal not in range(128) }}} So, I guess `fdopen` resets the codec to `ascii`, rather than whatever the locale detected. Drat! I did some brief googling but I couldn't tell what the right way to rewrite this line of code is. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12213> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12213: In Python testsuite, setting stdout to unbuffered causes us to switch to ascii output locale -------------------------------------+------------------------------------- Reporter: ezyang | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12213#comment:1> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12213: In Python testsuite, setting stdout to unbuffered causes us to switch to ascii output locale -------------------------------------+------------------------------------- Reporter: ezyang | Owner: thomie Type: bug | Status: new Priority: normal | Milestone: Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by thomie): * owner: => thomie Comment: I think can just delete that code. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12213#comment:2> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12213: In Python testsuite, setting stdout to unbuffered causes us to switch to ascii output locale -------------------------------------+------------------------------------- Reporter: ezyang | Owner: thomie Type: bug | Status: new Priority: normal | Milestone: Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by thomie):
I had a failing test case which was producing UTF-8 output
Can I see this test please? There are plenty of tests that produce UTF-8 output (for example the .stderr from GHC), and they don't cause any problems. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12213#comment:3> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12213: In Python testsuite, setting stdout to unbuffered causes us to switch to ascii output locale -------------------------------------+------------------------------------- Reporter: ezyang | Owner: thomie Type: bug | Status: new Priority: normal | Milestone: Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by ezyang): If it is Unicode output, but the output is what we "expected", then the test suite driver doesn't try to print it and there is no error. The second thing is that for direct invocations, the driver twiddles environment variables so that GHC does not produce Unicode, so you need to do something like a run_command test. So, one way to get the error is to do this: 1. Open `tests/typecheck/should_compile/tc170.hs` and add a type error, e.g., `x = True :: Int` 2. Run `make TEST=tc170` 3. Error! -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12213#comment:4> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12213: In Python testsuite, setting stdout to unbuffered causes us to switch to ascii output locale -------------------------------------+------------------------------------- Reporter: ezyang | Owner: thomie Type: bug | Status: new Priority: normal | Milestone: Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by Thomas Miedema <thomasmiedema@…>): In [changeset:"e8d62711e6cbc3065ee5e6f6a654667f02a0bcd1/ghc" e8d6271/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="e8d62711e6cbc3065ee5e6f6a654667f02a0bcd1" Testsuite: do not depend on sys.stdout.encoding The cause of #12213 is in dump_stdout and dump_stderr: print(read_no_crs(<filename>)) Commit 6f6f515401a29d26eaa5daae308b8e700abd4c04 changed read_no_crs to return a unicode string. Printing a unicode strings works fine as long as sys.stdout.encoding is 'UTF-8'. There are two reasons why sys.stdout.encoding might not be 'UTF-8'. * When output is going to a file, sys.stdout and sys.stdout do not respect the locale: $ LC_ALL=en_US.utf8 python -c 'import sys; print(sys.stderr.encoding)' UTF-8 $ LC_ALL=en_US.utf8 python -c 'import sys; print(sys.stderr.encoding)' 2>/dev/null None * When output is going to the terminal, explicitly reopening sys.stdout has the side-effect of changing sys.stdout.encoding from 'UTF-8' to 'None'. sys.stdout = os.fdopen(sys.__stdout__.fileno(), "w", 0) We currently do this to set a buffersize of 0 (the actual buffersize used is irrelevant for the sys.stdout.encoding problem). Solution: fix dump_stdout and dump_stderr to not use read_no_crs. }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12213#comment:5> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12213: In Python testsuite, setting stdout to unbuffered causes us to switch to ascii output locale -------------------------------------+------------------------------------- Reporter: ezyang | Owner: thomie Type: bug | Status: closed Priority: normal | Milestone: Component: Test Suite | Version: 8.0.1 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by thomie): * status: new => closed * resolution: => fixed Comment: Serious bug! I fixed it. I need to think about how to prevent this from happening in the future. I think it's possible to simplify this unicode and buffer handling, but I need to carefully test it first (on all platforms, for all supported versions of Python). -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12213#comment:6> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
participants (1)
-
GHC