Thanks Edward! Another question... deleteThread() calls throwToSingleThreaded(). I can update this so that it also calls throwToSingleThreaded() in the case of BlockedOnCCall_Interruptible (currently it explicitly excludes this case), but this doesn't solve the problem, because throwToSingleThreaded() doesn't seem to interrupt blocked calls at all. That functionality is in throwTo(), which is not called by throwToSingleThreaded(). Why are we using throwToSingleThreaded() in deleteThread() rather than throwTo()? Can I switch deleteThread() to use throwTo()? Or should I use throwTo() in deleteThread() only for the special case of BlockedOnCCall_Interruptible? Or should throwToSingleThreaded() be updated to do the same thing that throwTo does for the case of BlockedOnCCall_Interruptible? Thanks, Andi On Wed, Jul 30, 2014 at 6:57 PM, Edward Z. Yang wrote:

Recalling when I implemented this functionality, I think not interrupting threads in the exit sequence was just an oversight, and I think we could implement it. Seems reasonable to me.

Edward

Excerpts from Andreas Voellmy's message of 2014-07-30 23:49:24 +0100:

...

Hi GHCers,

I've been looking into issue #9284, which boils down to getting certain foreign calls issued by HS threads to finish (i.e. return) in the exit sequence of forkProcess.

There are several options for solving the particular problem in #9284; one option is to issue the particular foreign calls causing that issue as "interruptible" and then have the exit sequence interrupt interruptible foreign calls.

The exit sequence, starting from hs_exit(), goes through hs_exit_(), exitScheduler(), scheduleDoGC(), deleteAllThreads(), and then deleteThread(), where deleteThread is this:

static void deleteThread (Capability *cap STG_UNUSED, StgTSO *tso) { // NOTE: must only be called on a TSO that we have exclusive // access to, because we will call throwToSingleThreaded() below. // The TSO must be on the run queue of the Capability we own, or // we must own all Capabilities. if (tso->why_blocked != BlockedOnCCall && tso->why_blocked != BlockedOnCCall_Interruptible) { throwToSingleThreaded(tso->cap,tso,NULL); } }

So it looks like interruptible foreign calls are not interrupted in the exit sequence.

Is there a good reason why we have this behavior? Could we change it to interrupt TSO's with why_blocked == BlockedOnCCall_Interruptible in the exit sequence?

Thanks, Andi

P.S. It looks like this was introduced in commit 83d563cb9ede0ba792836e529b1e2929db926355.

Hello Andreas, Yes, it seems that deleteAllThreads() is too late. The problem is that you need to defer the major GC until all the logical threads (now including interruptible FFI threads) have terminated themselves. This happens automatically for Haskell threads, because they are either (1) chilling in a run queue, in which case we don't need to do anything, or (2) being processed by a capability, in which case the GC will wait until the capability is done processing the thread and gets the memo (since it needs to acquire all locks.) However, in the case of safe FFI calls, these give up the capability before going off to C-land, so trying to acquire all the capabilities won't block on the FFI calls coming back. So you'll need to do something else. It's not necessarily clear what the best course of action here is: if an interruptible thread fails to be interrupted, should we keep waiting? (Maybe we should; that's how we handle non-preemptible Haskell threads too.) In any case, you'll need some way of waiting on the FFI calls, which might need some more gyrations as well. Cheers, Edward Excerpts from Andreas Voellmy's message of 2014-08-02 21:28:31 +0100:

I tried to go ahead and call throwTo() instead of throwToSingleThreaded() for threads BlockedOnCCall_Interruptible state during the shutdown sequence. Unfortunately something goes wrong with this change. I haven't tracked it down yet, but it looks like the following happens...

hs_exit() eventually result in a call to scheduleDoGC(), which does acquireAllCapabilities() and then deleteAllThreads() interrupts interruptible foreign calls. Those foreign calls come back and call waitForReturnCapability() but get stuck here:

if (!task->wakeup) waitCondition(&task->cond, &task->lock);

I guess the scheduleDoGC is blocking the interrupted Haskell threads from finishing.

One possible fix is to have the returning foreign call see that we are in the exit sequence and avoid trying to return to the Haskell caller - I guess it can just exit. I tried adding some code in resumeThread() to exit if sched_state is SCHED_INTERRUPTING or SCHED_SHUTTING_DOWN, but this caused more trouble, so it seems that it's not a simple change.

On Sat, Aug 2, 2014 at 1:55 PM, Andreas Voellmy wrote:

...

Thanks Edward! Another question...

deleteThread() calls throwToSingleThreaded(). I can update this so that it also calls throwToSingleThreaded() in the case of BlockedOnCCall_Interruptible (currently it explicitly excludes this case), but this doesn't solve the problem, because throwToSingleThreaded() doesn't seem to interrupt blocked calls at all. That functionality is in throwTo(), which is not called by throwToSingleThreaded(). Why are we using throwToSingleThreaded() in deleteThread() rather than throwTo()? Can I switch deleteThread() to use throwTo()? Or should I use throwTo() in deleteThread() only for the special case of BlockedOnCCall_Interruptible? Or should throwToSingleThreaded() be updated to do the same thing that throwTo does for the case of BlockedOnCCall_Interruptible?

Thanks, Andi

On Wed, Jul 30, 2014 at 6:57 PM, Edward Z. Yang wrote:

...

Recalling when I implemented this functionality, I think not interrupting threads in the exit sequence was just an oversight, and I think we could implement it. Seems reasonable to me.

Edward

Excerpts from Andreas Voellmy's message of 2014-07-30 23:49:24 +0100:

...

Hi GHCers,

I've been looking into issue #9284, which boils down to getting certain foreign calls issued by HS threads to finish (i.e. return) in the exit sequence of forkProcess.

There are several options for solving the particular problem in #9284; one option is to issue the particular foreign calls causing that issue as "interruptible" and then have the exit sequence interrupt interruptible foreign calls.

The exit sequence, starting from hs_exit(), goes through hs_exit_(), exitScheduler(), scheduleDoGC(), deleteAllThreads(), and then deleteThread(), where deleteThread is this:

static void deleteThread (Capability *cap STG_UNUSED, StgTSO *tso) { // NOTE: must only be called on a TSO that we have exclusive // access to, because we will call throwToSingleThreaded() below. // The TSO must be on the run queue of the Capability we own, or // we must own all Capabilities. if (tso->why_blocked != BlockedOnCCall && tso->why_blocked != BlockedOnCCall_Interruptible) { throwToSingleThreaded(tso->cap,tso,NULL); } }

So it looks like interruptible foreign calls are not interrupted in the exit sequence.

Is there a good reason why we have this behavior? Could we change it to interrupt TSO's with why_blocked == BlockedOnCCall_Interruptible in the exit sequence?

Thanks, Andi

P.S. It looks like this was introduced in commit 83d563cb9ede0ba792836e529b1e2929db926355.