[GHC] #15010: Application (warp) server crashing periodically with "TSO object entered"

#15010: Application (warp) server crashing periodically with "TSO object entered" -------------------------------------+------------------------------------- Reporter: ramirez7 | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.4.3 Component: Runtime | Version: 8.0.2 System | Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: Runtime crash Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- We are seeing one of our warp servers periodically crash with the following error: {{{ Apr 06 10:44:28 stdouterr.log: binary-name-elided: internal error: TSO object entered! Apr 06 10:44:28 stdouterr.log: (GHC version 8.0.2 for x86_64_unknown_linux) }}} We are sometimes seeing CPU/memory spikes concurrently with this error, but not always. We are running very similar servers for other applications and they are '''not''' experiencing this error. But this server is one of our highest- load ones. Other tickets I found with this error: * https://ghc.haskell.org/trac/ghc/ticket/12383 * https://ghc.haskell.org/trac/ghc/ticket/8316 But these seem to be happening in ghci and ghc itself, not in a binary compiled by ghc. Questions I have: * What exactly is this error? * Is there anything else we can do on our side to help debug? Any artifacts that should be produced upon crash? * Should we consider upgrading our GHC version from 8.0.2 to get a bugfix? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15010 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#15010: Application (warp) server crashing periodically with "TSO object entered" -------------------------------------+------------------------------------- Reporter: ramirez7 | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.4.3 Component: Runtime System | Version: 8.0.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonmar): The error is an internal error in the runtime system or compiled code, and doesn't tell us very much unfortunately. A wide variety of different kinds of problems can lead to this symptom. I'm not aware of any known bugs in this area. To narrow this down we'll need to be able to reproduce it and debug it. Anything you could do to produce a standalone reproduction for the bug would be really helpful. Failing that if you could provide a binary and a core dump, that might also let us dig around and get more information. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15010#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#15010: Application (warp) server crashing periodically with "TSO object entered" -------------------------------------+------------------------------------- Reporter: ramirez7 | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.4.3 Component: Runtime System | Version: 8.0.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by ramirez7): Just saw the error happen again. I'll investigate the server to see if I can find any more artifacts. We did find some extra logs in {{{messages}}}: {{{ Apr 12 09:43:07 stdouterr.log: binary-name-elided: internal error: TSO object entered! Apr 12 09:43:07 stdouterr.log: (GHC version 8.0.2 for x86_64_unknown_linux) Apr 12 09:43:07 stdouterr.log: Please report this as a GHC bug: http://www.haskell.org/ghc/reportabug Apr 12 09:43:07 messages: Apr 12 16:43:07 ip-10-10-28-23 kernel: [413908.905615] traps: ghc_worker[23036] general protection ip:7f5b72a24196 sp:7f5b64de2c10 error:0 Apr 12 09:43:07 messages: Apr 12 16:43:07 ip-10-10-28-23 kernel: [413908.911110] in libc-2.23.so[7f5b729ed000+1c0000] }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15010#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#15010: Application (warp) server crashing periodically with "TSO object entered" -------------------------------------+------------------------------------- Reporter: ramirez7 | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.4.3 Component: Runtime System | Version: 8.0.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): Can you offer any information about the nature of your application? Does it use concurrency? STM? Make heavy use of exceptions? Any additional information would be helpful. Of course, a core dump would also be extremely helpful. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15010#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#15010: Application (warp) server crashing periodically with "TSO object entered" -------------------------------------+------------------------------------- Reporter: ramirez7 | Owner: (none) Type: bug | Status: infoneeded Priority: normal | Milestone: 8.6.1 Component: Runtime System | Version: 8.0.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * status: new => infoneeded * milestone: 8.4.3 => 8.6.1 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15010#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#15010: Application (warp) server crashing periodically with "TSO object entered" -------------------------------------+------------------------------------- Reporter: ramirez7 | Owner: (none) Type: bug | Status: infoneeded Priority: normal | Milestone: 8.10.1 Component: Runtime System | Version: 8.0.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: Runtime crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by sseverance): I hit something similar last night. Unfortunately I don't have a way to make a repro but will provide what information I do have in case it helps eventually. Our app is a server, although not a web server. We make extensive use of the STM and there may be hundreds of threads running at a time. I would say there are a moderate amount of exceptions, mostly forced on us from their use in other libraries, as we tend not to use them. Happy to provide more specific info if I can. App was started with the following parameters: +RTS -N -qg1 -A5m -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15010#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
participants (1)
-
GHC