On Mon, Oct 15, 2012 at 12:30 PM, Joey Hess wrote:

forkProcess comes with a giant warning: since any other running threads are not copied into the child process, it's easy to go wrong: e.g. by accessing some shared resource that was held by another thread in the parent.

In my experience, forkProcess often behaves incomprehensibly (to me) with -threaded, typically resulting in a forked process hanging, and quite often only some small percentage of the time, which makes it really hard to track down and try to diagnose what thunk might not be getting forced until after the fork, or whatever.

The forkProcess MissingH discussion recently left me feeling like this is going to be triggered in all cases, because it sounds like one of the problematic threads is the I/O manager's thread? -- brandon s allbery kf8nh sine nomine associates allbery.b@gmail.com ballbery@sinenomine.net unix/linux, openafs, kerberos, infrastructure http://sinenomine.net

On Mon, Oct 15, 2012 at 6:30 PM, Joey Hess wrote:

...

I think I have a misunderstanding of how forkProcess should be working. Ultimately this relates to some bugs in the development version of keter, but I've found some behavior in a simple test program which I wouldn't have expected either, which may or may not be related.

With the program at the end of this email, I would expect that, once per second, I would get a message printed from each forkIO'd green thread,

...

forked process, and the master process. And if I spawn 8 or less child

...

that's precisely what happens. However, as soon as I up that number to 9, the child process is never run. The process is, however, created, as can be confirmed by looking at the process table.

This only occurs when using the multithreaded runtime. In other words, compiling with "ghc --make test.hs" seems to always produce the expected output, whereas "ghc --make -threaded test.hs" causes the behavior described above. Having looked through the code for the process package a bit, my initial guess is that this is being caused by a signal being sent to the child

Michael Snoyman wrote: the threads process,

...

but I'm not familiar enough with the inner workings to confirm or disprove this guess.

If anyone has any ideas on this, I'd appreciate it.

While I'm not reproducing that behavior here with your test case and 7.4.1, I recently converted a large program to use -threaded (because I needed to use yesod in it, actually :), and had large quantities of pain involving forkProcess. It seemed to come down to this easily overlooked note in the docs:

forkProcess comes with a giant warning: since any other running threads are not copied into the child process, it's easy to go wrong: e.g. by accessing some shared resource that was held by another thread in the parent.

In my experience, forkProcess often behaves incomprehensibly (to me) with -threaded, typically resulting in a forked process hanging, and quite often only some small percentage of the time, which makes it really hard to track down and try to diagnose what thunk might not be getting forced until after the fork, or whatever.

I did some analysis and produced a test case for problems caused by use of forkProcess in parts of MissingH, here: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=681621

My understanding is that System.Process avoids these problems by doing all the setup around forking a command in C code. I've banished forkProcess from my code base entirely, except for a double fork I need to daemonize, and I don't even trust that call. :/

Well, I tried switching my code to forking/execing from C in a very similar manner to the process package, and it seems to work. Thanks for the input everyone! Michael

Since we're talking about forkIO here - not forkOS - is it possible to control the use of OS threads to avoid this problem? As I understand it, the problem is with real OS threads. A program running entirely in multiple `green' threads will fork to the same set of threads in the same state, creating no problem with internal state. I'm guessing this is why he sees the problem at 9 threads - maybe the runtime picks up another OS thread at that point. Of course one could link the unthreaded RTS, and that will cause everything, including runtime gc etc., to run in the parent thread, true? And there are some runtime options to control number of threads scheduled. Donn

On Tue, 16 Oct 2012 21:55:44 +0200 Alexander Kjeldaas wrote:

There are variants of this, but the meta-problem is that at the point in time when you call forkProcess, you must control all threads, ensuring that *all invariants hold*. Thus no locks held, no thread is in the C library, no foreign calls active etc. As an example, if one thread is in the C library doing some stdio, then the invariants in that library will not hold, and you cannot expect stdio to work in the child. This means that the only thing you can really do in the child process is call exec.

Further, you can only call exec if you make sure that the exec correctly reverts everything back to a state where those invariants hold. Mostly, this is automatic as resources get freed on exec and "do the right thing." Locks on file descriptors that aren't closed on exec will leave dangling locks, and locks on files that are closed on exec will unexpectedly close them in the parent. http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

Sorry to be reviving this thread so long after.... but I seem to be running into similar issues as Michael S. did at the start. In short, I'm using forkProcess with the threaded RTS, and see occasional hangs: - I see these only on Linux. On Mac OS X, I never do. - I'm using GHC 7.4.2 - I noticed the warning in the doc for forkProcess, but assumed I was safe, as I wasn't holding any shared resources at the time of the fork, and no shared resources in the program are used in the child. - WIth gdb, I've traced the hang to here in the run-time: forkProcess > discardTasksExcept > freeTask > closeMutex(&task->lock)

pthread_mutex_destroy

The discussion in this thread leaves me with these questions: - Is there reason to think the situation has gotten better in 7.6 and later? - Isn't the only reason *System.Process* is safer because it does an immediate exec in the child? Alas, I really want to just fork()sometimes. - Is it really true that even if my program has no shared resources with the child, that the IO subsystem and FFI system do anyway? Surely the RTS would take care of doing the right thing with those, no? - There should be no concern with exec w.r.t. library invariants since exec is wholesale replacement - all the libraries will reinitialize. Is there a problem here I'm missing? Alas, I've stopped using the threaded RTS until I understand this better. - Mark

On Mon, Jan 21, 2013 at 12:15 AM, Mark Lentczner wrote:

Sorry to be reviving this thread so long after.... but I seem to be running into similar issues as Michael S. did at the start.

In short, I'm using forkProcess with the threaded RTS, and see occasional hangs:

- I see these only on Linux. On Mac OS X, I never do.

Previous versions of the linux pthreads library didn't hold any shared resources in locks, so pthread_mutex_destroy could not hang. Now Linux is much improved, and thus it hangs (see pthread_mutex_destroy man page) ;-).

- I'm using GHC 7.4.2 - I noticed the warning in the doc for forkProcess, but assumed I was safe, as I wasn't holding any shared resources at the time of the fork, and no shared resources in the program are used in the child. - WIth gdb, I've traced the hang to here in the run-time: forkProcess

...

discardTasksExcept > freeTask > closeMutex(&task->lock) pthread_mutex_destroy

The discussion in this thread leaves me with these questions:

- Is there reason to think the situation has gotten better in 7.6 and later? - Isn't the only reason *System.Process* is safer because it does an immediate exec in the child? Alas, I really want to just fork()sometimes.

If you immediately do exec() in the child, you can use vfork() which blocks the parent. This serializes the actions and makes this whole mess smooth and consistent.

- Is it really true that even if my program has no shared resources with the child, that the IO subsystem and FFI system do anyway? Surely the RTS would take care of doing the right thing with those, no?

It looks like the RTS is trying to do a lot of things to control all the

Haskell threads etc. But I don't think it waits for FFI-land threads before commencing a fork(), so that's why I'm guessing that some interaction between threads using FFI and fork() could be the issue.

- There should be no concern with exec w.r.t. library invariants since exec is wholesale replacement - all the libraries will reinitialize. Is there a problem here I'm missing?

I think that's right. vfork() + exec() can be serialized and deterministic

thus is a lot easier to reason about. Alexander