I tried to go ahead and call throwTo() instead of throwToSingleThreaded() for threads BlockedOnCCall_Interruptible state during the shutdown sequence. Unfortunately something goes wrong with this change. I haven't tracked it down yet, but it looks like the following happens...

hs_exit() eventually result in a call to scheduleDoGC(), which does acquireAllCapabilities() and then deleteAllThreads() interrupts interruptible foreign calls. Those foreign calls come back and call waitForReturnCapability() but get stuck here:

if (!task->wakeup) waitCondition(&task->cond, &task->lock);

I guess the scheduleDoGC is blocking the interrupted Haskell threads from finishing.

One possible fix is to have the returning foreign call see that we are in the exit sequence and avoid trying to return to the Haskell caller - I guess it can just exit. I tried adding some code in resumeThread() to exit if sched_state is SCHED_INTERRUPTING or SCHED_SHUTTING_DOWN, but this caused more trouble, so it seems that it's not a simple change.

On Sat, Aug 2, 2014 at 1:55 PM, Andreas Voellmy <andreas.voellmy@gmail.com> wrote:

Thanks Edward! Another question...

deleteThread() calls throwToSingleThreaded(). I can update this so that it also calls throwToSingleThreaded() in the case of BlockedOnCCall_Interruptible (currently it explicitly excludes this case), but this doesn't solve the problem, because throwToSingleThreaded() doesn't seem to interrupt blocked calls at all. That functionality is in throwTo(), which is not called by throwToSingleThreaded(). Why are we using throwToSingleThreaded() in deleteThread() rather than throwTo()? Can I switch deleteThread() to use throwTo()? Or should I use throwTo() in deleteThread() only for the special case of BlockedOnCCall_Interruptible? Or should throwToSingleThreaded() be updated to do the same thing that throwTo does for the case of BlockedOnCCall_Interruptible?

Thanks,
Andi

On Wed, Jul 30, 2014 at 6:57 PM, Edward Z. Yang <ezyang@mit.edu> wrote:

Recalling when I implemented this functionality, I think not
interrupting threads in the exit sequence was just an oversight,
and I think we could implement it. Seems reasonable to me.

Edward

Excerpts from Andreas Voellmy's message of 2014-07-30 23:49:24 +0100:

> Hi GHCers,
>
> I've been looking into issue #9284, which boils down to getting certain
> foreign calls issued by HS threads to finish (i.e. return) in the exit
> sequence of forkProcess.
>
> There are several options for solving the particular problem in #9284; one
> option is to issue the particular foreign calls causing that issue as
> "interruptible" and then have the exit sequence interrupt interruptible
> foreign calls.
>
> The exit sequence, starting from hs_exit(), goes through hs_exit_(),
> exitScheduler(), scheduleDoGC(), deleteAllThreads(), and then
> deleteThread(), where deleteThread is this:
>
> static void
> deleteThread (Capability *cap STG_UNUSED, StgTSO *tso)
> {
> // NOTE: must only be called on a TSO that we have exclusive
> // access to, because we will call throwToSingleThreaded() below.
> // The TSO must be on the run queue of the Capability we own, or
> // we must own all Capabilities.
> if (tso->why_blocked != BlockedOnCCall &&
> tso->why_blocked != BlockedOnCCall_Interruptible) {
> throwToSingleThreaded(tso->cap,tso,NULL);
> }
> }
>
> So it looks like interruptible foreign calls are not interrupted in the
> exit sequence.
>
> Is there a good reason why we have this behavior? Could we change it to
> interrupt TSO's with why_blocked == BlockedOnCCall_Interruptible in the
> exit sequence?
>
> Thanks,
> Andi
>
> P.S. It looks like this was introduced in commit
> 83d563cb9ede0ba792836e529b1e2929db926355.