strange hangs with -threaded runtime
I have a rather large program that works fine with ghc's default
runtime, but now I'd like to build it with -threaded, and this
causes it to hang in a way I have not been able to pin down to anything
specific. It hangs at different points each time it's run; running
it in strace will consistently avoid the problem. If I look at
it once it's hung, I always see a pair of processes, both blocked
in futex:
joey@wren:~/src/git-annex>strace -p 21749
Process 21749 attached - interrupt to quit
futex(0x8b03930, FUTEX_WAIT_PRIVATE, 69, NULL^C
On Fri, Jul 13, 2012 at 2:26 PM, Joey Hess
Are there any common gotchas with converting to the threaded runtime that might provide a hint to what's wrong? My code does not currently use Control.Concurrent or threads, although it does use some libraries like MissingH that use forkIO or forkOS.
Off the top of my head, I'm not aware of any (non-Windows-related) gotchas with using -threaded. However, some functions in MissingH (e.g. in System.Cmd.Utils) call forkProcess. To quote the documentation of forkProcess [1]: --- forkProcess comes with a giant warning: since any other running threads are not copied into the child process, it's easy to go wrong: e.g. by accessing some shared resource that was held by another thread in the parent. --- Perhaps the forked process is inheriting an interleaved computation that tries to use resources in the parent process. [1]: http://hackage.haskell.org/packages/archive/unix/latest/doc/html/System-Posi...
Joey Adams wrote:
Off the top of my head, I'm not aware of any (non-Windows-related) gotchas with using -threaded. However, some functions in MissingH (e.g. in System.Cmd.Utils) call forkProcess. To quote the documentation of forkProcess [1]:
--- forkProcess comes with a giant warning: since any other running threads are not copied into the child process, it's easy to go wrong: e.g. by accessing some shared resource that was held by another thread in the parent. ---
Perhaps the forked process is inheriting an interleaved computation that tries to use resources in the parent process.
This does seem at least a possibility, I'll try forcing inputs and see. Are there any good general methods to debug threading mutex type problems in ghc? -- see shy jo
I've found a minimal test case that seems to demonstrate a bug in either
MissingH or ghc's threaded runtime. Or I'm doing something stupid in
fewer lines of code than usual. ;)
When built with the threaded runtime, after a short while it hangs in
hGetContents.
import System.Cmd
import System.IO
import System.Cmd.Utils
import System.Posix.Process
import Control.Monad
main :: IO ()
main = forever $ pipeRead ["hello", "world"]
pipeRead :: [String] -> IO ()
pipeRead params = do
-- removing this next line avoids the hang somehow
print $ "pipeRead in " ++ show params
(p, h) <- hPipeFrom "echo" params
print "pipeRead getcontents"
c <- hGetContents h
print $ "got: " ++ c
_ <- getProcessStatus True False $ processID p
-- removing this last line avoids the hang somehow
print "pipeRead out"
joey@wren:~>ghc --make -threaded test
[1 of 1] Compiling Main ( test.hs, test.o )
Linking test ...
joey@wren:~>./test
"pipeRead in [\"hello\",\"world\"]"
"pipeRead getcontents"
"got: hello world\n"
"pipeRead out"
"pipeRead in [\"hello\",\"world\"]"
"pipeRead getcontents"
"got: hello world\n"
"pipeRead out"
Well, hPipeFrom does indeed call forkProcess internally. I don't fully
understand when it is and is not safe to use 'forkProcess' with the
threaded runtime of GHC.
Which version of GHC are you using?
Antoine
On Sat, Jul 14, 2012 at 1:24 PM, Joey Hess
I've found a minimal test case that seems to demonstrate a bug in either MissingH or ghc's threaded runtime. Or I'm doing something stupid in fewer lines of code than usual. ;)
When built with the threaded runtime, after a short while it hangs in hGetContents.
import System.Cmd import System.IO import System.Cmd.Utils import System.Posix.Process import Control.Monad
main :: IO () main = forever $ pipeRead ["hello", "world"]
pipeRead :: [String] -> IO () pipeRead params = do -- removing this next line avoids the hang somehow print $ "pipeRead in " ++ show params (p, h) <- hPipeFrom "echo" params print "pipeRead getcontents" c <- hGetContents h print $ "got: " ++ c _ <- getProcessStatus True False $ processID p -- removing this last line avoids the hang somehow print "pipeRead out"
joey@wren:~>ghc --make -threaded test [1 of 1] Compiling Main ( test.hs, test.o ) Linking test ... joey@wren:~>./test "pipeRead in [\"hello\",\"world\"]" "pipeRead getcontents" "got: hello world\n" "pipeRead out" "pipeRead in [\"hello\",\"world\"]" "pipeRead getcontents" "got: hello world\n" "pipeRead out"
"pipeRead in [\"hello\",\"world\"]" "pipeRead getcontents" <hang> Ghc 7.4.2, Debian Linux
-- see shy jo
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Antoine Latter wrote:
Well, hPipeFrom does indeed call forkProcess internally. I don't fully understand when it is and is not safe to use 'forkProcess' with the threaded runtime of GHC.
Which version of GHC are you using?
I've reproduced the problem with 7.4.2, and 7.4.1. Just tried 6.12.1, which is interesting.. after around the same number of iterations at which it hangs with the newer ghcs, it instead does this: "pipeRead in [\"hello\",\"world\"]" "pipeRead getcontents" test: internal error: MUT_ARR_PTRS_FROZEN object entered! (GHC version 6.12.1 for i386_unknown_linux) Please report this as a GHC bug: http://www.haskell.org/ghc/reportabug "got: " "pipeRead out" -- see shy jo
I've isolated the hang further to hPipeFrom's use of System.Log.Logger. Commenting out calls to logging functions makes it reliably run ok. See the "LINE OF DEATH" in the attached updated test case. ... And it turns out that System.Log.Logger uses a MVar to hold the logger information. So I think this points to a MissingH bug, although I don't yet understand it. -- see shy jo
If the MVar was set up prior to the fork, I can imagine things going crazy
trying to use it on the ther side of the fork - especially if their were
waiting readers or writers while the fork was executing.
On Jul 14, 2012 3:01 PM, "Joey Hess"
I've isolated the hang further to hPipeFrom's use of System.Log.Logger. Commenting out calls to logging functions makes it reliably run ok. See the "LINE OF DEATH" in the attached updated test case.
... And it turns out that System.Log.Logger uses a MVar to hold the logger information. So I think this points to a MissingH bug, although I don't yet understand it.
-- see shy jo
Just following up to my problem, I was seeing lots of hangs in various places in my program when it was built with the threaded runtime. I eventually tracked every single hang back to calls to MissingH's System.Cmd.Utils, including pipeFrom, pipeTo, pipeBoth, and pOpen. I was at this point running my program in a loop 1000 times, and it'd hang between 1 and 10 times on average, since these hangs seem to be timing-related. In all cases, when it hung, it had forked a child, and the child was blocked in a futex() call. Each of these functions calls forkProcess, and then does some very simple setup before it calls executeFile -- but as far as I could see, the forked process never ran a single thing before hanging. The solution, for me, was to convert all my code to use System.Process instead of System.Cmd.Utils. It seems that System.Process does all its setup between fork and exec using C code, and so avoids this problem. I think it'd make sense to either add deprecation warnings to System.Cmd.Utils, or to rewrite it to be a wrapper around System.Process. -- see shy jo
participants (3)
-
Antoine Latter -
Joey Adams -
Joey Hess