GitLab

Duncan Coutts pushed to branch wip/dcoutts/capability-yield at Glasgow Haskell Compiler / GHC

Commits:

15bbf0ed

by Duncan Coutts at 2026-08-19T13:43:20+01:00

Move several Capability utils from Schedule.{c,h} to Capability.{c,h}

They probably should have been there all along. This means all the
pending_sync functionality is within Capability.{c,h}. We only expose
pending_sync for the purpose of inline header functions.

d8b84119

by Duncan Coutts at 2026-08-19T13:43:20+01:00

Shuffle the pending sync type declarations for better readability

Move them together into the section with the related functions that use
them.

Also drop the legacy use of the C 'volatile' modifier on the
pending_sync variable. We use C atomics for such access, not volatile.

adcf739e

by Duncan Coutts at 2026-08-19T13:43:20+01:00

Rename returning task queue helpers

Follows a naming convention elsewhere. It also gives us suitable names
to distinguish appending vs prepending to the queue, and we're about to
add a prepend operation.

46ec6801

by Duncan Coutts at 2026-08-19T13:43:20+01:00

Add a prepend operation for the returning task queue

With a pending sync (e.g. for GC), we really want to be able to
prioritise the task waiting on the sync over all other returning tasks.
To do that we will need to prepend to the queue rather than append.

5643d6fb

by Duncan Coutts at 2026-08-19T13:43:20+01:00

Split waitForSomeCapability out of waitForCapability and adjust callers

Previously waitForCapability had a general interface covering serveral
situations, but it's more useful and easier to understand with two more
specialised functions.

Previously waitForCapability had an in/out Capability parameter: if the
cap was non-null then it would wait to acquire that specific capability
(though in this case it had to be the capability the task was already
associated with), or if it was null then it would pick a suitable
capability, associate the task with it and wait for that capability.

So overall, this required and in/out parameter and suggested that the
capability returned could be different from the one passed in. That was
true if the task was associated with no capability, but false if it was.
This complicated the call sites where we know that the task does have an
associated capability, as it implied the capability could change when in
fact it cannot.

So we now split it: waitForSomeCapability handles this general case
where the task may or may not have an associated capability, and it
returns that new capability. And the waitForCapability now assumes that
the task is associated with a capability and does not need to return
anything. Neither function needs a Capability *cap argument any more.

This simplifies several call sites.

0a6ae704

by Duncan Coutts at 2026-08-19T13:43:20+01:00

Update stale comments related to waitFor[Some]Capability

In particular waitForCapability cannot change the capability.

Update a comment about grabbing a new Task in performGC_. This comment was
accurate in 2006, but it is no longer accurate since newBoundTask does not
in fact produce a new Task (typically). The right thing now is to talk about
InCalls, not tasks. That said, the code is still correct since
newBoundTask will push a new incall, but also deal with the general case
of an OS thread that does not yet have an associated Task.

47a69f63

by Duncan Coutts at 2026-08-19T13:43:20+01:00

Introduce waitForCapability_ with additional priority arg

Split waitForCapability into a wrapper with the existing type and a
worker with an extra argument.

The new high_priority argument controls whether the task is appended or
prepended to the returing task queue. The default, used by the
waitForCapability wrapper, is false, meaning append to the end of the
queue. This gives fairness.

The high_priority==true case will be used in the subsequent commit.

f5041fa8

by Duncan Coutts at 2026-08-19T13:43:20+01:00

Make acquireAllCapabilities use high_priority on waitForCapability_

As discussed in issue #27473, a sync of all capabilities is something
that needs to happen promptly (but often doesn't).

One source of delay is that acquireAllCapabilities using
waitForCapability would put the task trying to acquire each capability
at the _end_ of the returning task queue. This gave every other returing
task a full timeslice to run. Meanwhile, several other capabilities are
blocked waiting for the sync to complete, leading to a loss of
throughput.

We use the new high_priority arg to waitForCapability_ to ensure that
the requesting task is put on the front of the returing task queue. This
will ensure that releaseCapability_ will prioritise giving the
capability to the task requesting the sync.

7bda759f

by Duncan Coutts at 2026-08-19T13:43:20+01:00

In releaseCapability_ make the pending_sync case self-contained

Previously the pending_sync case had to be checked _after_ the returning
tasks case, since one of the possibilities (indeed the more likely
possibility) is that the task calling waitForCapability will have
enqueued itself as a returning task.

Now we make the pending_sync case self-contained. We note in a comment
the two possibilities: either the task calling waitForCapability has
enqueued itself already and is waiting, or it's not got there yet. We
can handle the first case by giving the capability to the task at the
head of the returning tasks queue, and the second case by leaving the
capability free.

Another way to look at this, is that we move a special case of handling
of the returning task case into the pending_sync case. That special case
being a returing task during a pending sync.

This makes the order of handling returning tasks vs pending sync
independent. This is good, because really they're in the wrong priority
order and we want to flip them around.

c78e52b9

by Duncan Coutts at 2026-08-19T13:43:20+01:00

In releaseCapability_ prioritise pending sync over returning tasks

Fixes issue #27460

As explained in the issue, a pending sync (e.g. for GC) should be dealt
with promptly. Returning tasks are a lower priority.

Historically however we had to check returning tasks first, because the
synchronisation mechanism mixed up the task doing a sync with the tasks
returning from safe FFI calls. The task performing the sync was very
likely to be queued on the returning task list (and historically it was
at the _end_ of this list!).

We have now arranged that the task performing the sync is at the front
of the returning task list, and in the pending sync case we now check
the returning task list and run the first task from there if it there is
one.

This is by no means perfect, but it is better. See issue #27473 for a
more general issue of cleaning up the design of the pending sync.

ff821186

by Duncan Coutts at 2026-08-19T13:43:20+01:00

Split releaseCapability_ to hide the rarely-used bool parameter

releaseCapability_ had an extra bool param: always_wakeup. This is
rather a design wart. The reasons are now documented in comments.

Reduce the number of cases where we need to use always_wakeup == true.
There is now only one: prodCapability. That is because
releaseCapability_ does not know about signals (which is an example of
the design wart).

We can thus simplify most callers by splitting releaseCapability_ into a
worker and wrapper, where the wrapper does not have to have the
always_wakeup parameter. This simplifies all call sites except one.

This will also reduce churn at call site when we add a second (rarely
used) parameter in the subsequent commit.

2eda7b95

by Duncan Coutts at 2026-08-19T13:43:20+01:00

Extend releaseCapability__ with an extra wakeup_worker modifier

Document within releaseCapability__ the basic approach of looking for a
series of conditions in priority order and acting on them. Explain the
existing modifier within that understanding.

Then add a new modifier, wakeup_worker and explain it in similar terms.
What it does is skip two of the conditions in the priority list, with
the effect that we prioritise waking up a worker task over a returning
task or bound task.

This feature is not yet used in this commit, but it will be used as
part of a scheme to allow in-RTS I/O managers in the threaded RTS. This
scheme will make use of being able to start a background worker thread,
and that will use this feature to start it promptly.

Also correct the yieldCapability docs to cover all the conditions, and
in priority order for consistency.

b3246103

by Duncan Coutts at 2026-08-19T13:43:20+01:00

Move enqueueWorker next to where it is used.

It's not general purpose at all. It's very specifically crafted to work
with it's only caller: yieldCapability. It does very suprising things
like releaseCapability_, release locks and terminate threads. This logic
would be much clearer if done within yieldCapability.

c0c6029d

by Duncan Coutts at 2026-08-19T13:43:20+01:00

Move code out of enqueueWorker and into releaseCapability_

Instead of directly releasing locks and terminating tasks, have it
return whether the enqueue was successful or not. In the latter case,
releaseCapability_ itself will release locks and terminate the task.

This makes the logic of releaseCapability_ a lot clearer. Fiddling with
tasks is what releaseCapability_ does, so it's better not to try and
encapsulate this within a helper function.

5ffc4c50

by Duncan Coutts at 2026-08-19T13:43:20+01:00

Clarify the logic and control flow in yieldCapability

yieldCapability is unfortunately a bit complicated. This change
restructures things slightly but should keep the behaviour the same.

Previously after calling releaseCapability_ we had a bunch of
alternatives, where in each branch we would use RELEASE_LOCK(cap->lock)
and do various things before/after the lock is released. This was a bit
hard to follow, or to extend (which we need to do).

So now we have unconditional acquire and release of the cap->lock, so
it's clear where that happens, with releaseCapability_ in between. Then
in between these steps we have the various other pre/post actions. Some
before releaseCapability_, some after while holing the lock, and some
after having released the lock.

We explain this structure in a longer comment, and refer back to the
structure from the code.

Changes:

rts/Capability.c

@@ -58,11 +58,25 @@ Capability **capabilities;
  // locking, so we don't do that.
  static Capability *last_free_capability[MAX_NUMA_NODES];
 +#if defined(THREADED_RTS)
  /*
   * Indicates that the RTS wants to synchronise all the Capabilities
   * for some reason.  All Capabilities should yieldCapability().
 + *
 + * This is an atomic variable, all accesses must use appropriate atomics.
 + */
 +PendingSync *pending_sync = NULL;
++
 +/*
 + * sync_finished_cond allows threads which do not own any capability (e.g. the
 + * concurrent mark thread) to participate in the sync protocol. In particular,
 + * if such a thread requests a sync while sync is already in progress it will
 + * block on sync_finished_cond, which will be signalled when the sync is
 + * finished (by releaseAllCapabilities).
   */
 -PendingSync * volatile pending_sync = 0;
 +static Condition sync_finished_cond;
 +static Mutex sync_finished_mutex;
 +#endif
  // Number of logical NUMA nodes
  uint32_t n_numa_nodes;
@@ -204,7 +218,7 @@ anySparks (void)
  #if defined(THREADED_RTS)
  STATIC_INLINE void
 -newReturningTask (Capability *cap, Task *task)
 +appendToReturningTaskQueue (Capability *cap, Task *task)
+ {
      ASSERT_LOCK_HELD(&cap->lock);
      ASSERT(task->next == NULL);
@@ -222,8 +236,25 @@ newReturningTask (Capability *cap, Task *task)
      ASSERT_RETURNING_TASKS(cap,task);
+ }
 +STATIC_INLINE void
 +prependToReturningTaskQueue (Capability *cap, Task *task)
 +{
 +    ASSERT_LOCK_HELD(&cap->lock);
 +    ASSERT(task->next == NULL);
 +    task->next = cap->returning_tasks_hd;
 +    cap->returning_tasks_hd = task;
 +    if (cap->returning_tasks_tl == NULL) {
 +        cap->returning_tasks_tl = task;
 +    }
++
 +    // See Note [Data race in shouldYieldCapability] in Schedule.c.
 +    RELAXED_ADD(&cap->n_returning_tasks, 1);
++
 +    ASSERT_RETURNING_TASKS(cap,task);
 +}
++
  STATIC_INLINE Task *
 -popReturningTask (Capability *cap)
 +popReturningTaskQueue (Capability *cap)
+ {
      ASSERT_LOCK_HELD(&cap->lock);
      Task *task;
@@ -349,6 +380,11 @@ initCapability (Capability *cap, uint32_t i)
   * ------------------------------------------------------------------------- */
  void initCapabilities (void)
+ {
 +#if defined(THREADED_RTS)
 +    initMutex(&sync_finished_mutex);
 +    initCondition(&sync_finished_cond);
 +#endif
++
      /* Declare a couple capability sets representing the process and
         clock domain. Each capability will get added to these capsets. */
      traceCapsetCreate(CAPSET_OSPROCESS_DEFAULT, CapsetTypeOsProcess);
@@ -523,7 +559,7 @@ giveCapabilityToTask (Capability *cap USED_IF_DEBUG, Task *task)
  #endif
  /* ----------------------------------------------------------------------------
 - * releaseCapability_
 + * releaseCapability and releaseCapability_
+  *
   * This serves two purposes:
+  *
@@ -533,32 +569,60 @@ giveCapabilityToTask (Capability *cap USED_IF_DEBUG, Task *task)
+  *
   * 2. There is no current task (cap->task == NULL), and thus the Capability
   *    is idle, and we want to wake up an idle Task to animate the Capability.
 - *    In this case set always_wakeup. See also prodCapability.
 - *
 - * Setting the always_wakeup parameter (almost) ensures that the capability is
 - * not left idle: even if there is no known work to do, the capability will be
 - * given to a worker task. There are two exceptions to this:
 - *  1. if there is a pending sync then the capability is left idle, but in
 - *     anticipation of whichever task initiated the sync picking it up shortly.
 - *  2. if the scheduler is shutting down and there are no threads on the run
 - *     queue and there are no spare workers then the capability is left idle.
 - *     It is not entirely clear if this corner case is intentional.
 + *    See also prodCapability.
+  *
 - * The caller must hold cap->lock and will still hold it after the call returns.
 + * Difference:
 + * - releaseCapability  the caller /must not/ hold cap->lock.
 + * - releaseCapability_ the caller /must/ hold cap->lock.
+  *
   * N.B. May need to take all_tasks_mutex, if it needs to start a new task.
+  *
   * ------------------------------------------------------------------------- */
  #if defined(THREADED_RTS)
 -void
 -releaseCapability_ (Capability* cap,
 -                    bool always_wakeup)
 +static void releaseCapability__ (Capability* cap,
 +                                 bool always_wakeup,
 +                                 bool wakeup_worker);
++
 +void releaseCapability (Capability* cap)
 +{
 +    ACQUIRE_LOCK(&cap->lock);
 +    releaseCapability__(cap, false /*always_wakeup*/,
 +                             false /*wakeup_worker*/);
 +    RELEASE_LOCK(&cap->lock);
 +}
++
 +void releaseCapability_ (Capability* cap)
 +{
 +    releaseCapability__(cap, false /*always_wakeup*/,
 +                             false /*wakeup_worker*/);
 +}
++
 +/* The fact that we need an always_wakeup parameter for releaseCapability__ is
 + * a design wart. The Capability layer knows about most but not all sources of
 + * work for a capability. The always_wakeup parameter is there to account for
 + * the ones it does /not/ know about (which is I/O manager stuff: I/O, timers
 + * & signals).
 + *
 + * There are two sane designs:
 + * 1. the Capability layer knows nothing about the sources of work that the
 + *    scheduler might want to do
 + * 2. the Capability layer knows *everything* about the sources of work.
 + *
 + * In neither sane design would we need this parameter. In the first design we
 + * would know externally if we should be waking or releasing a task and would
 + * instruct accordingly (probably by splitting releaseCapability to cover the
 + * two cases). In the second design, it would simply know about all the sources
 + * and so again there would be no need.
 + */
++
 +static void releaseCapability__ (Capability* cap,
 +                                 bool always_wakeup,
 +                                 bool wakeup_worker)
+ {
+     {
          Task *task = cap->running_task;
 -        ASSERT(task || always_wakeup);
          // To cover purpose 2 above, we allow the cap->running_task to be
          // NULL, to handle cases where a thread (that is not itself a Task)
          // needs to wake up an idle task for the capability.
@@ -572,36 +636,101 @@ releaseCapability_ (Capability* cap,
      // Remove the current Task owning the Capability (if any, see purpose 2).
      RELAXED_STORE(&cap->running_task, NULL);
 -    // Check to see whether a worker thread can be given
 -    // the go-ahead to return the result of an external call..
 -    if (cap->n_returning_tasks != 0) {
 -        giveCapabilityToTask(cap,cap->returning_tasks_hd);
 -        // The Task pops itself from the queue (see waitForCapability())
 -        return;
 -    }
 +    // We now look for a task to give the capability to, or otherwise we leave
 +    // the capability free.
 +    //
 +    // We take one of these guarded actions, in priority order:
 +    //
 +    // 1. If there's a pending synchronisation of all capabilities (e.g. GC),
 +    //    then give the capability to the task performing the sync.
 +    // 2. If there's a task returning (e.g. from safe FFI) on this capability,
 +    //    then give the capability to the first such task.
 +    // 3. If the next runnable thread on this capability is a bound thread,
 +    //    then give the capability to the corresponding bound task.
 +    // 4. If there are no spare worker tasks for this capability,
 +    //    then start one and give the capability to the new task.
 +    // 5. If there is some work to do on this capability (e.g. runnable thread),
 +    //    then give the capability to a worker task.
 +    // 6. Otherwise leave the capability free/idle.
 +    //
 +    // There are two modifiers to this priority list:
 +    //
 +    // * Setting always_wakeup modifies the case 5 predicate to be always true.
 +    //   This has the effect of giving the cap to a worker task, rather than
 +    //   leaving the cap idle, even if there is no obvious work to do.
 +    //
 +    // * Setting wakeup_worker skips cases 2 & 3. This prioritises waking a
 +    //   worker over returning tasks or bound tasks. It is also usually used in
 +    //   combination with always_wakeup.
++
 -    // If there is a pending sync, then we should just leave the Capability
 -    // free.  The thread trying to sync will be about to call
 -    // waitForCapability().
 +    // Guarded action 1:
 +    // If there's a pending synchronisation of all capabilities (e.g. GC),
 +    // then give the capability to the task performing the sync.
 +    //
 +    // If there is a pending sync, the task that requested the sync will
 +    // subsequently use acquireAllCapabilities to place itself on the (front of
 +    // the) returning_task list (of all capabilities). We will then be in one
 +    // of two cases:
 +    //
 +    // 1. the task that requested the pending sync has already put itself onto
 +    //    the returning_tasks list; or
 +    // 2. the task that requested the pending sync has not yet put itself onto
 +    //    the returning_tasks list. This is unlikely but possible depending on
 +    //    how the race is resolved.
      //
 -    // Note: this is *after* we check for a returning task above,
 -    // because the task attempting to acquire all the capabilities may
 -    // be currently in waitForCapability() waiting for this
 -    // capability, in which case simply setting it as free would not
 -    // wake up the waiting task.
 +    // The cap->lock is used by both waitForCapability and releaseCapability_
 +    // to ensure we are definitely in one of the two cases above, and not some
 +    // hideous mish-mash.
 +    //
 +    // In the first case we can give the capability to that task. It is highly
 +    // likely that the task has prepended itself to the returning task queue,
 +    // so we can give the capability to the task at the head of the returning
 +    // task queue. It is not a correctness issue however if another returning
 +    // task gets run first (indeed this was the historical behaviour).
 +    //
 +    // Note that there can be false positives for this case: if any other
 +    // returning task is queued on the returning tasks list. We will still
 +    // incur delays if this occurs, scheduling those returning tasks. It is
 +    // likely however that the task performing the sync gets to waiting before
 +    // the task running the capability responds to the interrupt signal.
 +    //
 +    // In the second case we leave the capability free since the task trying to
 +    // sync will be about to call waitForCapability().
      //
      // FIXME: this pending_sync approach is a poor design, hard to understand
      // and subject to various unnecessary delays. See issues #27460 and #27473.
      //
      PendingSync *sync = SEQ_CST_LOAD(&pending_sync);
      if (sync && (sync->type != SYNC_GC_PAR || sync->idle[cap->no])) {
 -        debugTrace(DEBUG_sched, "sync pending, freeing capability %d", cap->no);
 +        if (cap->n_returning_tasks != 0) {
 +            // The task doing the sync should have used waitForCapability_
 +            // using high_priority, so it should be at the head of the queue:
 +            debugTrace(DEBUG_sched, "sync pending, passing capability %d", cap->no);
 +            giveCapabilityToTask(cap,cap->returning_tasks_hd);
 +            // The Task pops itself from the queue (see waitForCapability())
 +        } else {
 +            debugTrace(DEBUG_sched, "sync pending, freeing capability %d", cap->no);
 +        }
          return;
+     }
 -    // If the next thread on the run queue is a bound thread,
 -    // give this Capability to the appropriate Task.
 -    if (!emptyRunQueue(cap) && peekRunQueue(cap)->bound) {
 +    // Skip guarded actions 2 & 3 if wakeup_worker. See the list of actions and
 +    // modifiers above.
++
 +    // Guarded action 2:
 +    // If there's a task returning (e.g. from safe FFI) on this capability,
 +    // then give the capability to the first such task.
 +    if (!wakeup_worker && cap->n_returning_tasks != 0) {
 +        giveCapabilityToTask(cap,cap->returning_tasks_hd);
 +        // The Task pops itself from the queue (see waitForCapability())
 +        return;
 +    }
++
 +    // Guarded action 3:
 +    // If the next runnable thread on this capability is a bound thread,
 +    // then give the capability to the bound thread's corresponding task.
 +    if (!wakeup_worker && !emptyRunQueue(cap) && peekRunQueue(cap)->bound) {
          // Make sure we're not about to try to wake ourselves up
          // ASSERT(task != cap->run_queue_hd->bound);
          // assertion is false: in schedule() we force a yield after
@@ -612,11 +741,13 @@ releaseCapability_ (Capability* cap,
          return;
+     }
 +    // Guarded action 4:
 +    // If there are no spare worker tasks for this capability,
 +    // then start one and give the capability to the new task.
      if (!cap->spare_workers) {
 -        // Create a worker thread if we don't have one.  If the system
 -        // is interrupted, we only create a worker task if there
 -        // are threads that need to be completed.  If the system is
 -        // shutting down, we never create a new worker.
 +        // If the system is interrupted, we only create a worker task if there
 +        // are threads that need to be completed.  If the system is shutting
 +        // down, we never create a new worker.
          if (getSchedState() < SCHED_SHUTTING_DOWN || !emptyRunQueue(cap)) {
              debugTrace(DEBUG_sched,
                         "starting new worker on capability %d", cap->no);
@@ -625,8 +756,9 @@ releaseCapability_ (Capability* cap,
+         }
+     }
 -    // If we have an unbound thread on the run queue, or if there's
 -    // anything else to do, give the Capability to a worker thread.
 +    // Guarded action 5:
 +    // If there is some work to do on this capability (e.g. runnable thread),
 +    // then give the capability to a worker task.
      if (always_wakeup ||
          !emptyRunQueue(cap) || !emptyInbox(cap) ||
          (!cap->disabled && !emptySparkPoolCap(cap)) || globalWorkToDo()) {
@@ -637,59 +769,14 @@ releaseCapability_ (Capability* cap,
+         }
+     }
 +    // Guarded action 6:
 +    // Otherwise leave the capability free/idle.
  #if defined(PROFILING)
      cap->r.rCCCS = CCS_IDLE;
  #endif
      RELAXED_STORE(&last_free_capability[cap->node], cap);
      debugTrace(DEBUG_sched, "freeing capability %d", cap->no);
+ }
+-
 -void
 -releaseCapability (Capability* cap)
 -{
 -    ACQUIRE_LOCK(&cap->lock);
 -    releaseCapability_(cap, false);
 -    RELEASE_LOCK(&cap->lock);
 -}
+-
 -void
 -releaseAndWakeupCapability (Capability* cap)
 -{
 -    ACQUIRE_LOCK(&cap->lock);
 -    releaseCapability_(cap, true);
 -    RELEASE_LOCK(&cap->lock);
 -}
+-
 -static void
 -enqueueWorker (Capability* cap)
 -{
 -    Task *task;
+-
 -    task = cap->running_task;
+-
 -    // If the Task is stopped, we shouldn't be yielding, we should
 -    // be just exiting.
 -    ASSERT(!task->stopped);
 -    ASSERT(task->worker);
+-
 -    if (cap->n_spare_workers < MAX_SPARE_WORKERS)
 -    {
 -        task->next = cap->spare_workers;
 -        cap->spare_workers = task;
 -        cap->n_spare_workers++;
 -    }
 -    else
 -    {
 -        debugTrace(DEBUG_sched, "%d spare workers already, exiting",
 -                   cap->n_spare_workers);
 -        releaseCapability_(cap,false);
 -        // hold the lock until after workerTaskStop; c.f. scheduleWorker()
 -        workerTaskStop(task);
 -        RELEASE_LOCK(&cap->lock);
 -        shutdownThread();
 -    }
 -}
+-
  #endif
  /*
@@ -822,7 +909,7 @@ static Capability * waitForReturnCapability (Task *task)
                  continue;
+             }
              RELAXED_STORE(&cap->running_task, task);
 -            popReturningTask(cap);
 +            popReturningTaskQueue(cap);
              RELEASE_LOCK(&cap->lock);
              break;
+         }
@@ -893,38 +980,65 @@ static Capability * find_capability_for_task(const Task * task)
  #endif /* THREADED_RTS */
  /* ----------------------------------------------------------------------------
 - * waitForCapability (Capability **pCap, Task *task)
 + * Capability *waitForCapability (Task *task)
+  *
 - * Purpose:  when an OS thread returns from an external call,
 - * it calls waitForCapability() (via Schedule.resumeThread())
 - * to wait for permission to enter the RTS & communicate the
 - * result of the external call back to the Haskell thread that
 - * made it.
 + * Purpose:  when an OS thread returns from an external call, it calls
 + * waitForCapability() (via Schedule.resumeThread()) to wait for
 + * permission to enter the RTS & communicate the result of the external
 + * call back to the Haskell thread that made it. The task must already
 + * be associated with a capability.
+  *
 - * pCap is strictly an output.
 + * Capability *waitForSomeCapability (Task *task)
+  *
 + * Like waitForCapability but the task need not already be associated
 + * with a capability. If it is not associated, an appropriate one will
 + * be chosen. Used in rts_lock(), for calling into the RTS from outside.
   * ------------------------------------------------------------------------- */
 -void waitForCapability (Capability **pCap, Task *task)
 -{
  #if !defined(THREADED_RTS)
+-
 +Capability *waitForSomeCapability (Task *task)
 +{
      MainCapability.running_task = task;
      task->cap = &MainCapability;
 -    *pCap = &MainCapability;
 +    return &MainCapability;
 +}
++
 +void waitForCapability (Task *task)
 +{
 +    waitForSomeCapability(task);
 +}
  #else
 -    Capability *cap = *pCap;
 +static void waitForCapability_ (Task *task,
 +                                bool high_priority);
++
 +Capability *waitForSomeCapability (Task *task)
 +{
 +    Capability *cap = task->cap;
      if (cap == NULL) {
          cap = find_capability_for_task(task);
          // record the Capability as the one this Task is now associated with.
          task->cap = cap;
 -    } else {
 -        ASSERT(task->cap == cap);
+     }
 +    waitForCapability_(task, false /*high_priority*/);
++
 +    return task->cap;
 +}
++
 +void waitForCapability (Task *task)
 +{
 +    waitForCapability_(task, false /*high_priority*/);
 +}
++
 +static void waitForCapability_ (Task *task,
 +                                bool high_priority)
 +{
 +    Capability *cap = task->cap;
 +    ASSERT(task->cap);
++
      debugTrace(DEBUG_sched, "returning; I want capability %d", cap->no);
      ACQUIRE_LOCK(&cap->lock);
@@ -933,7 +1047,11 @@ void waitForCapability (Capability **pCap, Task *task)
          RELAXED_STORE(&cap->running_task, task);
          RELEASE_LOCK(&cap->lock);
      } else {
 -        newReturningTask(cap,task);
 +        if (high_priority) {
 +            prependToReturningTaskQueue(cap,task);
 +        } else {
 +            appendToReturningTaskQueue(cap,task);
 +        }
          RELEASE_LOCK(&cap->lock);
          cap = waitForReturnCapability(task);
+     }
@@ -945,10 +1063,8 @@ void waitForCapability (Capability **pCap, Task *task)
      ASSERT_FULL_CAPABILITY_INVARIANTS(cap, task);
      debugTrace(DEBUG_sched, "resuming capability %d", cap->no);
+-
 -    *pCap = cap;
 -#endif
+ }
 +#endif
  /* ----------------------------------------------------------------------------
   * yieldCapability
@@ -957,12 +1073,6 @@ void waitForCapability (Capability **pCap, Task *task)
   * when either we know that the Capability should be given to another Task, or
   * there is nothing to do right now.  One of the following is true:
+  *
 - *    - The current Task is a worker, and there's a bound thread at the head of
 - *      the run queue (or vice versa)
 - *
 - *    - The run queue is empty.  We'll be woken up again when there's work to
 - *      do.
 - *
   *    - Another Task is trying to do parallel GC (pending_sync == SYNC_GC_PAR).
   *      We should become a GC worker for a while.
+  *
@@ -970,12 +1080,21 @@ void waitForCapability (Capability **pCap, Task *task)
   *      SYNC_GC_PAR), either to do a sequential GC, forkProcess, or
   *      setNumCapabilities.  We should give up the Capability temporarily.
+  *
 + *    - There is a Task returning from a safe FFI call.
 + *
 + *    - The current Task is a worker, and there's a bound thread at the head of
 + *      the run queue (or vice versa)
 + *
 + *    - There is no work to do (empty run queue, inbox etc).  We'll be woken up
 + *      again when there's work to do.
 + *
   * When yieldCapability returns *pCap will have been updated to the new
   * capability held by the caller.
+  *
   * ------------------------------------------------------------------------- */
  #if defined(THREADED_RTS)
 +static bool tryEnqueueWorker (Capability* cap);
  /* See Note [GC livelock] in Schedule.c for why we have gcAllowed
     and return the bool */
@@ -1030,28 +1149,61 @@ yieldCapability
      // We must now release the capability and wait to be woken up again.
      task->wakeup = false;
 +    // What happens next is a bit complicated. It has the following outline:
 +    //
 +    //  1. take the cap->lock
 +    //  2. "various stuff part A", pre-releaseCapability_ holding cap->lock
 +    //  3. release the capability
 +    //  4. "various stuff part B", post-releaseCapability_ holding cap->lock
 +    //  5. release the cap->lock
 +    //  6. "various stuff part C", post release cap->lock
 +    //
 +    // Much of the "various stuff" is also conditional, which complicates
 +    // matters further. To try and maintain clarity we use the following
 +    // variables in the conditions for the in-between steps.
 +    //
 +    bool terminate_worker = false;
 +    bool task_is_worker   = isWorker(task);
 +    bool task_is_bound    = isBoundTask(task);
++
 +    // Step 1: take the cap->lock
      ACQUIRE_LOCK(&cap->lock);
 -    // If this is a worker thread, put it on the spare_workers queue
 -    if (isWorker(task)) {
 -        enqueueWorker(cap);
 +    // Step 2: "various stuff part A", pre-releaseCapability_ holding cap->lock
 +    if (task_is_worker) {
 +        // If this is a worker thread, try to put it on the spare_workers
 +        // queue or if it is surplus then we will terminate it.
 +        terminate_worker = !tryEnqueueWorker(cap);
+     }
 -    releaseCapability_(cap, false);
 +    // Step 3: release the capability
 +    releaseCapability_(cap);
 -    if (isWorker(task) || isBoundTask(task)) {
 -        RELEASE_LOCK(&cap->lock);
 -        cap = waitForWorkerCapability(task);
 -    } else {
 +    // Step 4: "various stuff part B", post-releaseCapability_ holding cap->lock
 +    if (terminate_worker) {
 +        // hold the lock until after workerTaskStop; c.f. scheduleWorker()
 +        workerTaskStop(task);
 +    } else if (!task_is_worker && !task_is_bound) {
          // Not a worker Task, or a bound Task.  The only way we can be woken up
          // again is to put ourselves on the returning_tasks queue, so that's
 -        // what we do.  We still hold cap->lock at this point
 -        // The Task waiting for this Capability does not have it
 -        // yet, so we can be sure to be woken up later. (see #10545)
 -        newReturningTask(cap,task);
 -        RELEASE_LOCK(&cap->lock);
 +        // what we do.  We still hold cap->lock at this point. The Task waiting
 +        // for this Capability does not have it yet, so we can be sure to be
 +        // woken up later. (see #10545)
 +        appendToReturningTaskQueue(cap,task);
 +    }
++
 +    // Step 5: release the cap->lock
 +    RELEASE_LOCK(&cap->lock);
++
 +    // Step 6. "various stuff part C", post release cap->lock
 +    if (terminate_worker) {
 +        shutdownThread();
 +    } else if (task_is_worker || task_is_bound) {
 +        cap = waitForWorkerCapability(task);
 +    } else {
          cap = waitForReturnCapability(task);
+     }
 +    // End of step 6.
      debugTrace(DEBUG_sched, "resuming capability %d", cap->no);
      ASSERT(cap->running_task == task);
@@ -1067,8 +1219,226 @@ yieldCapability
      return false;
+ }
 +// Returns true if it could enqueue, and false if the worker is surplus to
 +// requirements and should be terminated.
 +static bool tryEnqueueWorker (Capability* cap)
 +{
 +    Task *task = cap->running_task;
++
 +    // If the Task is stopped, we shouldn't be yielding, we should
 +    // be just exiting.
 +    ASSERT(!task->stopped);
 +    ASSERT(task->worker);
 +    ASSERT_LOCK_HELD(&cap->lock);
++
 +    if (cap->n_spare_workers < MAX_SPARE_WORKERS)
 +    {
 +        task->next = cap->spare_workers;
 +        cap->spare_workers = task;
 +        cap->n_spare_workers++;
 +        return true;
 +    }
 +    else
 +    {
 +        debugTrace(DEBUG_sched, "%d spare workers already, exiting",
 +                   cap->n_spare_workers);
 +        return false;
 +    }
 +}
++
  #endif /* THREADED_RTS */
++
 +/* -----------------------------------------------------------------------------
 + * stopAllCapabilities()
 + *
 + * Stop all Haskell execution.  This is used when we need to make some global
 + * change to the system, such as altering the number of capabilities, or
 + * forking.
 + *
 + * pCap may be NULL in the event that the caller doesn't yet own a capability.
 + *
 + * To resume after stopAllCapabilities(), use releaseAllCapabilities().
 + * -------------------------------------------------------------------------- */
++
 +#if defined(THREADED_RTS)
 +void stopAllCapabilities
 +    ( Capability **pCap     // [in/out] This thread's task's owned capability.
 +                            //          pCap may be NULL if no capability is owned.
 +                            //          Else *pCap != NULL
 +                            // On return, set to the task's newly owned
 +                            // capability (task->cap). Though, the Task will
 +                            // technically own all capabilities.
 +    , Task *task            // [in] This thread's task.
 +    )
 +{
 +    stopAllCapabilitiesWith(pCap, task, SYNC_OTHER);
 +}
++
 +void stopAllCapabilitiesWith (Capability **pCap, Task *task, SyncType sync_type)
 +{
 +    bool was_syncing;
 +    SyncType prev_sync_type;
++
 +    PendingSync sync = {
 +        .type = sync_type,
 +        .idle = NULL,
 +        .task = task
 +    };
++
 +    do {
 +        was_syncing = requestSync(pCap, task, &sync, &prev_sync_type);
 +    } while (was_syncing);
++
 +    acquireAllCapabilities(pCap ? *pCap : NULL, task);
++
 +    resetSync();
 +}
 +#endif
++
 +/* -----------------------------------------------------------------------------
 + * requestSync()
 + *
 + * Commence a synchronisation between all capabilities.  Normally not called
 + * directly, instead use stopAllCapabilities().  This is used by the GC, which
 + * has some special synchronisation requirements.
 + *
 + * Note that this can be called in two ways:
 + *
 + * - where *pcap points to a capability owned by the caller: in this case
 + *   *prev_sync_type will reflect the in-progress sync type on return, if one
 + *   *was found
 + *
 + *  - where pcap == NULL: in this case the caller doesn't hold a capability.
 + *    we only return whether or not a pending sync was found and prev_sync_type
 + *    is unchanged.
 + *
 + * Returns:
 + *    false if we successfully got a sync
 + *    true  if there was another sync request in progress,
 + *             and we yielded to it.  The value returned is the
 + *             type of the other sync request.
 + * -------------------------------------------------------------------------- */
++
 +#if defined(THREADED_RTS)
 +bool requestSync
 +    ( Capability **pcap         // [in/out] This thread's task's owned capability.
 +                                // May change if there is an existing sync (true is returned).
 +                                // Precondition:
 +                                //      pcap may be NULL
 +                                //      *pcap != NULL
 +    , Task *task                // [in] This thread's task.
 +    , PendingSync *new_sync     // [in] The new requested sync.
 +    , SyncType *prev_sync_type  // [out] Only set if there is an existing sync (true is returned).
 +    )
 +{
 +    PendingSync *sync;
++
 +    sync = (PendingSync*)cas((StgVolatilePtr)&pending_sync,
 +                             (StgWord)NULL,
 +                             (StgWord)new_sync);
++
 +    if (sync != NULL)
 +    {
 +        // sync is valid until we have called yieldCapability().
 +        // After the sync is completed, we cannot read that struct any
 +        // more because it has been freed.
 +        *prev_sync_type = sync->type;
 +        if (pcap == NULL) {
 +            // The caller does not hold a capability (e.g. may be a concurrent
 +            // mark thread). Consequently we must wait until the pending sync is
 +            // finished before proceeding to ensure we don't loop.
 +            // TODO: Don't busy-wait
 +            ACQUIRE_LOCK(&sync_finished_mutex);
 +            while (pending_sync) {
 +                waitCondition(&sync_finished_cond, &sync_finished_mutex);
 +            }
 +            RELEASE_LOCK(&sync_finished_mutex);
 +        } else {
 +            do {
 +                debugTrace(DEBUG_sched, "someone else is trying to sync (%d)...",
 +                          sync->type);
 +                ASSERT(*pcap);
 +                yieldCapability(pcap,task,true);
 +                sync = SEQ_CST_LOAD(&pending_sync);
 +            } while (sync != NULL);
 +        }
++
 +        // NOTE: task->cap might have changed now
 +        return true;
 +    }
 +    else
 +    {
 +        return false;
 +    }
 +}
++
 +void resetSync (void)
 +{
 +    RELAXED_STORE(&pending_sync, 0);
 +    signalCondition(&sync_finished_cond);
 +}
 +#endif
++
 +/* -----------------------------------------------------------------------------
 + * acquireAllCapabilities()
 + *
 + * Grab all the capabilities except the one we already hold (cap may be NULL if
 + * the caller does not currently hold a capability). Used when synchronising
 + * before a single-threaded GC (SYNC_SEQ_GC), and before a fork (SYNC_OTHER).
 + *
 + * Only call this after requestSync(), otherwise a deadlock might
 + * ensue if another thread is trying to synchronise.
 + * -------------------------------------------------------------------------- */
++
 +#if defined(THREADED_RTS)
 +void acquireAllCapabilities(Capability *cap, Task *task)
 +{
 +    Capability *tmpcap = NULL;
 +    uint32_t i;
++
 +    ASSERT(SEQ_CST_LOAD(&pending_sync) != NULL);
 +    for (i=0; i < getNumCapabilities(); i++) {
 +        debugTrace(DEBUG_sched, "grabbing all the capabilities (%d/%d)",
 +                   i, getNumCapabilities());
 +        tmpcap = getCapability(i);
 +        if (tmpcap != cap) {
 +            task->cap = tmpcap;
 +            waitForCapability_(task, true /*high_priority*/);
++
 +            // Note that waitForCapability only waits for the capability
 +            // the task is associated with. There's no task migration here.
 +            ASSERT(task->cap == tmpcap);
 +        }
 +    }
 +    ASSERT(tmpcap != NULL);
 +    task->cap = cap == NULL ? tmpcap : cap;
 +}
 +#endif
++
 +/* -----------------------------------------------------------------------------
 + * releaseAllCapabilities()
 + *
 + * Assuming this thread holds all the capabilities, release them all (except for
 + * the one passed in as keep_cap, if non-NULL).
 + * -------------------------------------------------------------------------- */
++
 +#if defined(THREADED_RTS)
 +void releaseAllCapabilities(uint32_t n, Capability *keep_cap, Task *task)
 +{
 +    uint32_t i;
 +    ASSERT( task != NULL);
 +    for (i = 0; i < n; i++) {
 +        Capability *tmpcap = getCapability(i);
 +        if (keep_cap != tmpcap) {
 +            task->cap = tmpcap;
 +            releaseCapability(tmpcap);
 +        }
 +    }
 +    task->cap = keep_cap;
 +}
 +#endif
++
  /*
   * Note [migrated bound threads]
   * ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -1117,7 +1487,14 @@ prodCapability (Capability *cap)
+ {
      ACQUIRE_LOCK(&cap->lock);
      if (!cap->running_task) {
 -        releaseCapability_(cap,true);
 +        /* We have to use always_wakeup here because when prodCapability is
 +         * used for ctl-c, releaseCapability__ does not know about pending
 +         * signals (or the I/O managers generally) as one of the set of
 +         * conditions to look for when deciding if a Task should be woken up
 +         * to run the Capability. This is a bit of a design wart.
 +         */
 +        releaseCapability__(cap, true  /*always_wakeup*/,
 +                                 false /*wakeup_worker*/);
+     }
      RELEASE_LOCK(&cap->lock);
+ }
@@ -1238,7 +1615,7 @@ shutdownCapability (Capability *cap USED_IF_THREADS,
          if (!emptyRunQueue(cap) || cap->spare_workers) {
              debugTrace(DEBUG_sched,
                         "runnable threads or workers still alive, yielding");
 -            releaseCapability_(cap,false); // this will wake up a worker
 +            releaseCapability_(cap); // this will wake up a worker
              RELEASE_LOCK(&cap->lock);
              yieldThread();
              continue;

rts/Capability.h

@@ -254,16 +254,13 @@ void moreCapabilities (uint32_t from, uint32_t to);
  // ASSUMES: cap->running_task is the current Task.
  //
  #if defined(THREADED_RTS)
 -void releaseCapability           (Capability* cap);
 -void releaseAndWakeupCapability  (Capability* cap);
 -void releaseCapability_ (Capability* cap, bool always_wakeup);
 +void releaseCapability  (Capability* cap);
 +void releaseCapability_ (Capability* cap);
  // assumes cap->lock is held
  #else
  // releaseCapability() is empty in non-threaded RTS
  INLINE_HEADER void releaseCapability  (Capability* cap STG_UNUSED) {};
 -INLINE_HEADER void releaseAndWakeupCapability  (Capability* cap STG_UNUSED) {};
 -INLINE_HEADER void releaseCapability_ (Capability* cap STG_UNUSED,
 -                                       bool always_wakeup STG_UNUSED) {};
 +INLINE_HEADER void releaseCapability_ (Capability* cap STG_UNUSED) {};
  #endif
  // declared in rts/include/rts/Threads.h:
@@ -283,46 +280,22 @@ INLINE_HEADER Capability *getCapability(uint32_t i)
      return RELAXED_LOAD(&capabilities[i]);
+ }
 +// Acquire the task's associated capability, waiting as necessary.
  //
 -// Types of global synchronisation
 -//
 -typedef enum {
 -    SYNC_OTHER,
 -    SYNC_GC_SEQ,
 -    SYNC_GC_PAR,
 -    SYNC_FLUSH_UPD_REM_SET,
 -    SYNC_FLUSH_EVENT_LOG
 -} SyncType;
+-
 -//
 -// Details about a global synchronisation
 -//
 -typedef struct {
 -    SyncType type;              // The kind of synchronisation
 -    bool *idle;                 // Array of size n_capabilities. idle[i] is true
 -                                // if capability i will be idle during this GC
 -                                // cycle. Only available when doing GC (when
 -                                // type is SYNC_GC_*).
 -    Task *task;                 // The Task performing the sync
 -} PendingSync;
+-
 -//
 -// Indicates that the RTS wants to synchronise all the Capabilities
 -// for some reason.  All Capabilities should stop and return to the
 -// scheduler.
 +// Note this acquires the specific capability the task is already associated
 +// with (i.e. task->cap must be set). Alternatively, use waitForSomeCapability
 +// to acquire some appropriate choice of capability.
  //
 -extern PendingSync * volatile pending_sync;
 +void waitForCapability (Task *task);
 -// Acquires a capability at a return point.  If *cap is non-NULL, then
 -// this is taken as a preference for the Capability we wish to
 -// acquire.
 +// If the task is not already associated with a capability, select some
 +// reasonable choice of capability and associate the task with the chosen
 +// capability (i.e. the task->cap is set to the new choice). Then acquire the
 +// chosen capability, waiting as necessary.
  //
 -// OS threads waiting in this function get priority over those waiting
 -// in waitForCapability().
 +// Returns the now-acquired cap that was newly associated with the task.
  //
 -// On return, *cap is non-NULL, and points to the Capability acquired.
 -//
 -void waitForCapability (Capability **cap/*in/out*/, Task *task);
 +Capability *waitForSomeCapability (Task *task);
  EXTERN_INLINE void recordMutableCap (const StgClosure *p, Capability *cap,
                                          uint32_t gen);
@@ -391,6 +364,50 @@ INLINE_HEADER void contextSwitchCapability(Capability *cap, bool immediately);
  void interruptAllCapabilities(void);
  INLINE_HEADER void interruptCapability(Capability *cap);
 +#if defined(THREADED_RTS)
 +//
 +// Types of global synchronisation
 +//
 +typedef enum {
 +    SYNC_OTHER,
 +    SYNC_GC_SEQ,
 +    SYNC_GC_PAR,
 +    SYNC_FLUSH_UPD_REM_SET,
 +    SYNC_FLUSH_EVENT_LOG
 +} SyncType;
++
 +void stopAllCapabilities (Capability **pCap, Task *task);
 +void stopAllCapabilitiesWith (Capability **pCap, Task *task, SyncType sync_type);
 +void acquireAllCapabilities(Capability *cap, Task *task);
 +void releaseAllCapabilities(uint32_t n, Capability *keep_cap, Task *task);
++
 +//
 +// Details about a global synchronisation
 +//
 +typedef struct {
 +    SyncType type;              // The kind of synchronisation
 +    bool *idle;                 // Array of size n_capabilities. idle[i] is true
 +                                // if capability i will be idle during this GC
 +                                // cycle. Only available when doing GC (when
 +                                // type is SYNC_GC_*).
 +    Task *task;                 // The Task performing the sync
 +} PendingSync;
++
 +// Used by scheduler for GC. Other use cases should prefer stopAllCapabilities.
 +bool requestSync (Capability **pcap,
 +                  Task *task,
 +                  PendingSync *new_sync,
 +                  SyncType *prev_sync_type);
 +void resetSync (void);
++
 +// Is there a current pending sync? This is true after a successful requestSync
 +// and before the sync completes.
 +INLINE_HEADER bool anyPendingSync(void);
++
 +// Private, use anyPendingSync() accessor.
 +extern PendingSync *pending_sync;
 +#endif
++
  // Free all capabilities
  void freeCapabilities (void);
@@ -516,6 +533,10 @@ INLINE_HEADER bool emptyInbox(Capability *cap)
              RELAXED_LOAD(&cap->putMVars) == NULL);
+ }
 +INLINE_HEADER bool anyPendingSync(void)
 +{
 +    return RELAXED_LOAD(&pending_sync);
 +}
  #endif
  #include "EndPrivate.h"

rts/Messages.c

@@ -49,11 +49,7 @@ void sendMessage(Capability *from_cap, Capability *to_cap, Message *msg)
      recordClosureMutated(from_cap,(StgClosure*)msg);
      if (to_cap->running_task == NULL) {
 -        /* Precond for releaseCapability_ is: running_task || always_wakeup.
 -         * We have running_task == NULL, hence we must use always_wakeup. This
 -         * is ok since the inbox is now non-empty, so we wake a task anyway.
 -         */
 -        releaseCapability_(to_cap, true /*always_wakeup*/);
 +        releaseCapability_(to_cap);
      } else {
          interruptCapability(to_cap);
+     }

rts/RtsAPI.c

@@ -654,8 +654,7 @@ rts_lock (void)
+     }
  #endif
 -    cap = NULL;
 -    waitForCapability(&cap, task);
 +    cap = waitForSomeCapability(task);
      if (task->incall->prev_stack == NULL) {
        // This is a new outermost call from C into Haskell land.
@@ -692,7 +691,7 @@ rts_unlock (Capability *cap)
      // random point in the future, which causes problems for
      // freeTaskManager().
      ACQUIRE_LOCK(&cap->lock);
 -    releaseCapability_(cap,false);
 +    releaseCapability_(cap);
      // Finally, we can release the Task to the free list.
      exitMyTask();

rts/Schedule.c

@@ -112,18 +112,6 @@ Mutex sched_mutex;
  #define FORKPROCESS_PRIMOP_SUPPORTED
  #endif
 -/*
 - * sync_finished_cond allows threads which do not own any capability (e.g. the
 - * concurrent mark thread) to participate in the sync protocol. In particular,
 - * if such a thread requests a sync while sync is already in progress it will
 - * block on sync_finished_cond, which will be signalled when the sync is
 - * finished (by releaseAllCapabilities).
 - */
 -#if defined(THREADED_RTS)
 -static Condition sync_finished_cond;
 -static Mutex sync_finished_mutex;
 -#endif
+-
  /* -----------------------------------------------------------------------------
   * static function prototypes
@@ -141,9 +129,6 @@ static void scheduleFindWork (Capability **pcap);
  static void scheduleYield (Capability **pcap, Task *task);
  #endif
  #if defined(THREADED_RTS)
 -static bool requestSync (Capability **pcap, Task *task,
 -                         PendingSync *sync_type, SyncType *prev_sync_type);
 -static void acquireAllCapabilities(Capability *cap, Task *task);
  static void startWorkerTasks (uint32_t from USED_IF_THREADS,
                                uint32_t to USED_IF_THREADS);
  #endif
@@ -676,7 +661,7 @@ shouldYieldCapability (Capability *cap, Task *task, bool didGcLast)
      // n_returning_tasks. However, since this is an approximate predicate we can
      // use a RELAXED ordering.
 -    return ((RELAXED_LOAD(&pending_sync) && !didGcLast) ||
 +    return ((anyPendingSync() && !didGcLast) ||
              RELAXED_LOAD(&cap->n_returning_tasks) != 0 ||
              (!emptyRunQueue(cap) && (task->incall->tso == NULL
                                       ? peekRunQueue(cap)->bound != NULL
@@ -875,13 +860,9 @@ schedulePushWork(Capability *cap USED_IF_THREADS,
          // release the capabilities
          for (i = 0; i < n_free_caps; i++) {
              task->cap = free_caps[i];
 -            if (sparkPoolSizeCap(cap) > 0) {
 -                // If we have sparks to steal, wake up a worker on the
 -                // capability, even if it has no threads to run.
 -                releaseAndWakeupCapability(free_caps[i]);
 -            } else {
 -                releaseCapability(free_caps[i]);
 -            }
 +            // If there are sparks available, this will wake up a Task to run
 +            // the Capability, even if it has no threads to run.
 +            releaseCapability(free_caps[i]);
+         }
+     }
      task->cap = cap; // reset to point to our Capability.
@@ -1412,194 +1393,6 @@ scheduleNeedHeapProfile( bool ready_to_gc )
+     }
+ }
 -/* -----------------------------------------------------------------------------
 - * stopAllCapabilities()
 - *
 - * Stop all Haskell execution.  This is used when we need to make some global
 - * change to the system, such as altering the number of capabilities, or
 - * forking.
 - *
 - * pCap may be NULL in the event that the caller doesn't yet own a capability.
 - *
 - * To resume after stopAllCapabilities(), use releaseAllCapabilities().
 - * -------------------------------------------------------------------------- */
+-
 -#if defined(THREADED_RTS)
 -void stopAllCapabilities
 -    ( Capability **pCap     // [in/out] This thread's task's owned capability.
 -                            //          pCap may be NULL if no capability is owned.
 -                            //          Else *pCap != NULL
 -                            // On return, set to the task's newly owned
 -                            // capability (task->cap). Though, the Task will
 -                            // technically own all capabilities.
 -    , Task *task            // [in] This thread's task.
 -    )
 -{
 -    stopAllCapabilitiesWith(pCap, task, SYNC_OTHER);
 -}
+-
 -void stopAllCapabilitiesWith (Capability **pCap, Task *task, SyncType sync_type)
 -{
 -    bool was_syncing;
 -    SyncType prev_sync_type;
+-
 -    PendingSync sync = {
 -        .type = sync_type,
 -        .idle = NULL,
 -        .task = task
 -    };
+-
 -    do {
 -        was_syncing = requestSync(pCap, task, &sync, &prev_sync_type);
 -    } while (was_syncing);
+-
 -    acquireAllCapabilities(pCap ? *pCap : NULL, task);
+-
 -    RELAXED_STORE(&pending_sync, 0);
 -    signalCondition(&sync_finished_cond);
 -}
 -#endif
+-
 -/* -----------------------------------------------------------------------------
 - * requestSync()
 - *
 - * Commence a synchronisation between all capabilities.  Normally not called
 - * directly, instead use stopAllCapabilities().  This is used by the GC, which
 - * has some special synchronisation requirements.
 - *
 - * Note that this can be called in two ways:
 - *
 - * - where *pcap points to a capability owned by the caller: in this case
 - *   *prev_sync_type will reflect the in-progress sync type on return, if one
 - *   *was found
 - *
 - *  - where pcap == NULL: in this case the caller doesn't hold a capability.
 - *    we only return whether or not a pending sync was found and prev_sync_type
 - *    is unchanged.
 - *
 - * Returns:
 - *    false if we successfully got a sync
 - *    true  if there was another sync request in progress,
 - *             and we yielded to it.  The value returned is the
 - *             type of the other sync request.
 - * -------------------------------------------------------------------------- */
+-
 -#if defined(THREADED_RTS)
 -static bool requestSync
 -    ( Capability **pcap         // [in/out] This thread's task's owned capability.
 -                                // May change if there is an existing sync (true is returned).
 -                                // Precondition:
 -                                //      pcap may be NULL
 -                                //      *pcap != NULL
 -    , Task *task                // [in] This thread's task.
 -    , PendingSync *new_sync     // [in] The new requested sync.
 -    , SyncType *prev_sync_type  // [out] Only set if there is an existing sync (true is returned).
 -    )
 -{
 -    PendingSync *sync;
+-
 -    sync = (PendingSync*)cas((StgVolatilePtr)&pending_sync,
 -                             (StgWord)NULL,
 -                             (StgWord)new_sync);
+-
 -    if (sync != NULL)
 -    {
 -        // sync is valid until we have called yieldCapability().
 -        // After the sync is completed, we cannot read that struct any
 -        // more because it has been freed.
 -        *prev_sync_type = sync->type;
 -        if (pcap == NULL) {
 -            // The caller does not hold a capability (e.g. may be a concurrent
 -            // mark thread). Consequently we must wait until the pending sync is
 -            // finished before proceeding to ensure we don't loop.
 -            // TODO: Don't busy-wait
 -            ACQUIRE_LOCK(&sync_finished_mutex);
 -            while (pending_sync) {
 -                waitCondition(&sync_finished_cond, &sync_finished_mutex);
 -            }
 -            RELEASE_LOCK(&sync_finished_mutex);
 -        } else {
 -            do {
 -                debugTrace(DEBUG_sched, "someone else is trying to sync (%d)...",
 -                          sync->type);
 -                ASSERT(*pcap);
 -                yieldCapability(pcap,task,true);
 -                sync = SEQ_CST_LOAD(&pending_sync);
 -            } while (sync != NULL);
 -        }
+-
 -        // NOTE: task->cap might have changed now
 -        return true;
 -    }
 -    else
 -    {
 -        return false;
 -    }
 -}
 -#endif
+-
 -/* -----------------------------------------------------------------------------
 - * acquireAllCapabilities()
 - *
 - * Grab all the capabilities except the one we already hold (cap may be NULL is
 - * the caller does not currently hold a capability). Used when synchronising
 - * before a single-threaded GC (SYNC_SEQ_GC), and before a fork (SYNC_OTHER).
 - *
 - * Only call this after requestSync(), otherwise a deadlock might
 - * ensue if another thread is trying to synchronise.
 - * -------------------------------------------------------------------------- */
+-
 -#if defined(THREADED_RTS)
 -static void acquireAllCapabilities(Capability *cap, Task *task)
 -{
 -    Capability *tmpcap;
 -    uint32_t i;
+-
 -    ASSERT(SEQ_CST_LOAD(&pending_sync) != NULL);
 -    for (i=0; i < getNumCapabilities(); i++) {
 -        debugTrace(DEBUG_sched, "grabbing all the capabilities (%d/%d)",
 -                   i, getNumCapabilities());
 -        tmpcap = getCapability(i);
 -        if (tmpcap != cap) {
 -            // we better hope this task doesn't get migrated to
 -            // another Capability while we're waiting for this one.
 -            // It won't, because load balancing happens while we have
 -            // all the Capabilities, but even so it's a slightly
 -            // unsavoury invariant.
 -            task->cap = tmpcap;
 -            waitForCapability(&tmpcap, task);
 -            if (tmpcap->no != i) {
 -                barf("acquireAllCapabilities: got the wrong capability");
 -            }
 -        }
 -    }
 -    task->cap = cap == NULL ? tmpcap : cap;
 -}
 -#endif
+-
 -/* -----------------------------------------------------------------------------
 - * releaseAllCapabilities()
 - *
 - * Assuming this thread holds all the capabilities, release them all (except for
 - * the one passed in as keep_cap, if non-NULL).
 - * -------------------------------------------------------------------------- */
+-
 -#if defined(THREADED_RTS)
 -void releaseAllCapabilities(uint32_t n, Capability *keep_cap, Task *task)
 -{
 -    uint32_t i;
 -    ASSERT( task != NULL);
 -    for (i = 0; i < n; i++) {
 -        Capability *tmpcap = getCapability(i);
 -        if (keep_cap != tmpcap) {
 -            task->cap = tmpcap;
 -            releaseCapability(tmpcap);
 -        }
 -    }
 -    task->cap = keep_cap;
 -}
 -#endif
+-
  /* -----------------------------------------------------------------------------
   * Perform a garbage collection if necessary
   * -------------------------------------------------------------------------- */
@@ -1665,7 +1458,7 @@ scheduleDoGC (Capability **pcap, Task *task USED_IF_THREADS,
      // GC, we synchronise all the running threads using requestSync().
      //
      // Other capabilities are prevented from running yet more Haskell threads if
 -    // pending_sync is set. Tested inside yieldCapability() and
 +    // anyPendingSync() holds. Tested inside yieldCapability() and
      // releaseCapability() in Capability.c
      PendingSync sync = {
@@ -1802,7 +1595,7 @@ scheduleDoGC (Capability **pcap, Task *task USED_IF_THREADS,
                      if (i != cap->no && idle_cap[i]) {
                          Capability *tmpcap = getCapability(i);
                          task->cap = tmpcap;
 -                        waitForCapability(&tmpcap, task);
 +                        waitForCapability(task);
                          n_idle_caps++;
+                     }
+                 }
@@ -1910,10 +1703,10 @@ delete_threads_and_gc:
      };
  #if defined(THREADED_RTS)
 -    // reset pending_sync *before* GC, so that when the GC threads
 +    // reset the pending sync *before* GC, so that when the GC threads
      // emerge they don't immediately re-enter the GC.
 -    RELAXED_STORE(&pending_sync, 0);
 -    signalCondition(&sync_finished_cond);
 +    resetSync();
++
      config.parallel = gc_type == SYNC_GC_PAR;
      GarbageCollect(config, cap, idle_cap);
  #else
@@ -2074,8 +1867,7 @@ forkProcess(HsStablePtr *entry
      task = newBoundTask();
 -    cap = NULL;
 -    waitForCapability(&cap, task);
 +    cap = waitForSomeCapability(task);
  #if defined(THREADED_RTS)
      stopAllCapabilities(&cap, task);
@@ -2132,7 +1924,7 @@ forkProcess(HsStablePtr *entry
  #endif
          for (i=0; i < n_capabilities; i++) {
 -            releaseCapability_(getCapability(i),false);
 +            releaseCapability_(getCapability(i));
              RELEASE_LOCK(&getCapability(i)->lock);
+         }
@@ -2534,7 +2326,7 @@ suspendThread (StgRegTable *reg, bool interruptible)
    suspendTask(cap,task);
    cap->in_haskell = false;
 -  releaseCapability_(cap,false);
 +  releaseCapability_(cap);
    RELEASE_LOCK(&cap->lock);
@@ -2567,10 +2359,7 @@ resumeThread (void *task_)
      task->cap = cap;
      // Wait for permission to re-enter the RTS with the result.
 -    waitForCapability(&cap,task);
 -    // we might be on a different capability now... but if so, our
 -    // entry on the suspended_ccalls list will also have been
 -    // migrated.
 +    waitForCapability(task);
      // Remove the thread from the suspended list
      recoverSuspendedTask(cap,task);
@@ -2721,7 +2510,7 @@ void scheduleWorker (Capability *cap, Task *task)
      // Capability has been shut down.
      //
      ACQUIRE_LOCK(&cap->lock);
 -    releaseCapability_(cap,false);
 +    releaseCapability_(cap);
      workerTaskStop(task);
      RELEASE_LOCK(&cap->lock);
+ }
@@ -2767,8 +2556,6 @@ initScheduler(void)
     * the scheduler. */
  #if defined(THREADED_RTS)
    initMutex(&sched_mutex);
 -  initMutex(&sync_finished_mutex);
 -  initCondition(&sync_finished_cond);
  #endif
    ACQUIRE_LOCK(&sched_mutex);
@@ -2804,8 +2591,7 @@ exitScheduler (bool wait_foreign USED_IF_THREADS)
      // If we haven't killed all the threads yet, do it now.
      if (getSchedState() < SCHED_SHUTTING_DOWN) {
          setSchedState(SCHED_INTERRUPTING);
 -        Capability *cap = task->cap;
 -        waitForCapability(&cap,task);
 +        Capability *cap = waitForSomeCapability(task);
          scheduleDoGC(&cap,task,true,false,false,true);
          ASSERT(task->incall->tso == NULL);
          releaseCapability(cap);
@@ -2850,17 +2636,19 @@ freeScheduler( void )
  static void
  performGC_(bool force_major, bool nonconcurrent)
+ {
 -    Task *task;
 -    Capability *cap = NULL;
+-
 -    // We must grab a new Task here, because the existing Task may be
 -    // associated with a particular Capability, and chained onto the
 -    // suspended_ccalls queue.
 -    task = newBoundTask();
 +    // We must use a new InCall for our Task here, because the existing Task's
 +    // InCall may be chained onto the suspended_ccalls queue.
 +    //
 +    // If we could guarante there's an existing Task then we could just use
 +    // newInCall/endInCall. But it's not clear the calling thread necessarily
 +    // has an existing Task, so we play it safe and use newBoundTask/exitMyTask.
 +    // This does the same newInCall/endInCall when there is an initialised Task,
 +    // but will also do the right thing if there's no existing Task.
 +    Task *task = newBoundTask();
      // TODO: do we need to traceTask*() here?
 -    waitForCapability(&cap,task);
 +    Capability *cap = waitForSomeCapability(task);
      scheduleDoGC(&cap,task,force_major,false,false,nonconcurrent);
      releaseCapability(cap);
      exitMyTask();

rts/Schedule.h

@@ -55,12 +55,6 @@ StgWord findAtomicallyFrameHelper (Capability *cap, StgTSO *tso);
  /* Entry point for a new worker */
  void scheduleWorker (Capability *cap, Task *task);
 -#if defined(THREADED_RTS)
 -void stopAllCapabilitiesWith (Capability **pCap, Task *task, SyncType sync_type);
 -void stopAllCapabilities (Capability **pCap, Task *task);
 -void releaseAllCapabilities(uint32_t n, Capability *keep_cap, Task *task);
 -#endif
+-
  /* The state of the scheduler.  This is used to control the sequence
   * of events during shutdown.  See Note [shutdown] in Schedule.c.
   */