-
ac870ad5
by Simon Jakobi at 2026-10-28T04:40:42-04:00
X86 NCG: use 32-bit POPCNT for sub-word popcounts
A 32-bit write zero-extends the result to the full register, so we
don't need to emit additional code for this.
We also skip emitting zero-extension instructions for the argument
when it is already visibly zero-extended.
For the byte-wise popcount loop from #24813, this shrinks the popcount
sequence from 5 instructions to 3.
Assisted-by: Claude Fable 5.1
-
53087373
by Facundo Domínguez at 2026-10-28T04:41:32-04:00
Use a ShortByteString in Serialized instead of [Word8]
This makes the in-memory representation 40x more compact for all values
stored in annotations.
A test is added to measure the representation size of annotations
specified in the compiler input.
The GHC API is modified. All functions that previously handled annotations
in [Word8] form now use ShortByteString. The following definitions have
modified types:
* GHC.findGlobalAnns
* GHC.Tc.Gen.Splice.getAnnotationsByTypeRep
* GHC.Core.Opt.Monad.getAnnotations
* GHC.Core.Opt.Monad.getFirstAnnotations
* GHC.Types.Annotations.deserializeAnns
* GHC.Types.Annotations.findAnns
* GHC.Types.Annotations.findAnnsByTypeRep
* GHC.Serialized.Serialized
* GHC.Serialized.toSerialized
* GHC.Serialized.fromSerialized
* GHC.Serialized.serializeWithData
* GHC.Serialized.deserializeWithData
* GHCi.Message.ReifyAnnotations
Fixes #28002.
-
e4e3bb3c
by Rodrigo Mesquita at 2026-10-28T15:21:12+00:00
refactor: Keep the interactive Ids directly rather than Name
Removes things like
names <- map idName ids
map (\i -> let id = find (\x -> i==idName x) names) idso
and is more straightforward because there's less state going around (no
more updating HscEnv as an output)
It also kills `printTypeOfNames` completely, since now we just have the
Ids. No longer `lookupName` names to get back the ids...
```
printTypeOfName :: GHC.GhcMonad m => Name -> m ()
printTypeOfName n
= do maybe_tything <- GHC.lookupName n
case maybe_tything of
Nothing -> return ()
Just thing -> printTyThing thing
```
now is just `printTyThing . AnId`
Also:
```diff
- tythings <- catMaybes `liftM` mapM GHC.lookupName namesSorted
- docs <- mapM pprTypeAndContents [i | AnId i <- tythings]
+ docs <- mapM pprTypeAndContents idsSorted
```
(and yes, in ExecBreaks there are only Ids, no TyThings, bc they are
only constructed by the interpreter execution which can only bind ids.
for TyThings use runDecls which doesn't return execBreaks.)
There's some care to be had to make sure we use the "most refined" Ids,
whereas when using `Name`s we'd always fetch the most refined Id from
scratch.
Fixes #28052
-
7a0c80db
by Rodrigo Mesquita at 2026-11-02T17:10:26+00:00
interp: Make resumeExec and ExecBreak use `Resume` directly
Previously, `Resume` contexts were just put directly in the global
interactive environment state, so they were only accessible through it.
However, in the context of multi-threaded debugging, in that many
threads may hit breakpoints, it no longer suffices to have a global
linear list of breakpoints hit. We need a per-breakpoint Resume context
and need to be able to wield it directly rather than going through a
global context.
The driving changes of this commit are:
- Put the Resume data in ExecBreak
- Make resumeExec receive a Resume as an argument
This also simplifies the interpreter in that we no longer necessarily
rely on the global interactive state to resume a thread. We can do it
directly from the ExecBreak.
The interactive context still maintains a stack of resumes, but
`resumeExec` can no longer pop the resume it resumes off of the
breakpoints stack, since it is no longer guaranteed to be resuming the
first in that list. Hence, callers of `resumeExec` must update ic_resume
on their own.
Fixes #28058
-
55ce2ade
by Rodrigo Mesquita at 2026-11-02T17:10:29+00:00
Per-thread break-on exception (TSO_STOP_ON_EXCEPTION)
The `-fbreak-on-exception` flag allows the interactive interpreter to stop
when a synchronous exception is raised.
However, this is a global toggle backed by a global variable in the RTS
(rts_stop_on_exception) which is not compatible with a multi-threaded
debugger where we may want to stop if some threads raise a synchronous
exception but not other.
This commit makes -break-on-exception similar to step-next and step-out
where there is a per-thread toggle on each TSO (TSO_STOP_ON_EXCEPTION),
and that break-on-exception set in EvalOpts when resuming a statement
only affect the thread being resumed.
Fixes #28068
-
eee86f8d
by Rodrigo Mesquita at 2026-11-02T17:10:29+00:00
Asynchronous RPC with external interpreter
There's a single pipe to connect the host compiler to the external
interpreter. On this pipe, we have historically done synchronous
request/reply communication to invoke methods on the external
interpreter.
However, with the advent of multi-threaded debugging, it has become
necessary to do asynchronous calls on the external interpreter, where we
may block on the host waiting for a reply, but the external interpreter
should handle the request asynchronously and reply when ready. For
instance, we want to start the evaluation of the debuggee but not block
the interpreter waiting for it to finish or hit a breakpoint. Or e.g. we
want to resume two threads of the debuggee simulatenously instead of
blocking waiting for the resume result of each one sequentially.
And it turns out, we were already doing some ad-hoc asynchronous
communication to evaluate Template Haskell on the external interpreter.
It used to go as an ad-hoc "send request without expecting reply",
followed by handling on the host a set of 'THMessage's sent by the
interpreter, followed by "expect a response to arrive now". It worked
because there was still a fixed order of messages, but it fits an async
model in a non-ad-hoc way.
Asynchronous communication that unblocks the external interpreter main
thread will also make it possible to resolve issues such as #18852
(exceptions leaving the external interpreter in an unrecoverrable state)
The new design
--------------
The new invariant: every single request should receive a matching reply.
The pipe is bidirectional (the ext. interpreter can receive requests,
but it can also send requests for TH evaluation; the host can send
requests, but it can also receive TH requests from the interpreter).
When a request is sent in whichever direction, it always contains a
correlation-id header. The Sender places an empty MVar on a
pending-reply map keyed by id and waits on it for a response.
See 'remoteCall' and 'remoteTHCall'.
The pipe is associated to a thread which drains the read end. Because
it's bidirectional, two things may be read off the pipe:
- A Reply, with a correlation-id header, which we match against the
pending requests map to unblock the corresponding pending-reply MVar
- A Request, which is queued to the pending-request queue.
To handle a message, we take a Request from the pending-request queue
and run a handler that produces a Reply. Currently, the handler is run
synchronously in the thread, but it is trivial to make the handler run
in a forked thread for certain ("interruptible") messages in an upcoming
patch (when it is necessary for the multi-threaded debugger).
See 'handleRemoteCall' and 'handleRemoteTHCall'.
The old interp lock now just protect the pipe writing (reads don't lock
bc they are done exclusively by drainPipe), it doesn't need nor should
lock the full remote procedure call.
Fixes #27960
-
533ff7d9
by Rodrigo Mesquita at 2026-11-02T17:10:29+00:00
interp: Add separate mailboxes for TH async RPC
When the host sends a `RunTH` command to the interpreter, it expects a
few "reverse requests" from the interpreter (e.g. `NewName`) which must
be run in the same context that initiated the `RunTH`.
When the host is evaluating many splices at once, it will send many
`RunTH` commands to the interpreter and start listening for reverse
requests in the meantime. However, a received request previously had no
way to be associated to the scope of a particular `RunTH` invocation, so
the reverse commands could get replied to from incorrect contexts, or
e.g. a `RunTHDone` from the interp. could terminate early the wrong
`RunTH` computation.
In this commit we add independent mailboxes to the interpreter
asynchronous RPC infrastructure, allowing a RunTH to announce a
particular `InboxId` that all reverse requests for that RunTH should
use. On the host, we create that mailbox id and have
`handleRemoteTHCall` listen on that specific mailbox rather than for
requests without a specific mailbox address.
-
7c78175f
by Rodrigo Mesquita at 2026-11-02T17:10:29+00:00
refactor: Move Async RPC abstraction into GHCi.Message.Async
Carves out the Async RPC over the Pipe abstraction to a new module,
making the GHCi.Message more about the specific interpreter messages
rather than about the IPC protocol.
-
2346f2da
by Rodrigo Mesquita at 2026-11-02T17:10:29+00:00
interp: If we have pending reads in the queue we can deliver them
We may still be able to reply even if the read end was closed.
As long as we have pending messages we should try to deliver them.
-
a2284cc8
by Andrea Vezzosi at 2026-11-02T17:10:30+00:00
interp: close Handles when done with Pipe
Ensures we close the pipe handles (if created using 'mkPipeFromHandles')
on `closePipe`, and do `closePipe` on `Shutdown`.
The thread draining the read end always `'finally'` closes the read end.
And `closePipe` kills the draining thread and closes the write end.
When `remoteAsyncReq`, we throw PipeReadClosedException if the pipe read is
closed, since we wouldn't be able to receive the reply.
It's fine for the drain thread not to close the write end, even if it
dies without `closePipe` (which does close the write end), because:
- requests pending a reply will get PipeClosedException (we'll not be
able to read any further "real" reply)
- new calls to `remoteAsyncReq` will throw PipeReadClosedException
- pending pipe writes, in response to an already read request, might still succeed
And the clean up code should call close pipe to ensure the write end is
closed as well.
So, if the drain thread dies naturally and the read end is closed, may
still be attempted to be used, but it will throw accordingly. The user
likely wants to catch this to close the write end as well.
-
abd1c98a
by Zubin Duggal at 2026-11-02T17:10:30+00:00
interp: fix external interpreter shutdown deadlock on Windows
On shutdown both GHC and iserv call `closePipe`, which stopped the drain
thread before closing the write end. On Windows the drain thread reads
with a safe foreign call to _read, which `throwTo' can't interrupt, so
the two sides were stuck in the deadlock deadlock.
Closing the write end first is not enough on its own. On Windows a read
on a pipe only unblocks once all write handles to it are closed [1].
However, all all four pipe ends were inheritable and we didn't close the
child's ends in the parent. This means a close on any one side wouldn't
unblock any reads, becuase the other side had another copy open.
* runWithPipes now uses System.Process.CommunicationHandle [2], which is
built for passing pipes to a child process: only the child's ends are
inheritable, and it simplifies our mess of CPP. Additionally, it is
important to close our copy of the endpoints passed to the child so
each side only keeps open copies it needs to allow a close to unblock
reads.
* readGhcHandle makes the handle non-inheritable so we don't have
additional references floating around that stop reads from unblocking
on close.
* In closePipe the ordering is subtle and important. We first fail any
requests still waiting for a reply, then close the write end so the
other side sees EOF. We take the write lock to close it, so we never
close under a concurrent write [3]. Only then do we stop the drain
thread, which closes the read end itself once its read has finished.
Bump bound to process >= 1.6.20 as this is the version with
System.Process.CommunicationHandle
[1] https://learn.microsoft.com/en-us/windows/win32/ipc/anonymous-pipe-operations
[2] https://github.com/haskell/process/pull/308
[3] https://gitlab.haskell.org/ghc/ghc/-/work_items/27801
-
16a07999
by Andrea Vezzosi at 2026-11-02T17:10:30+00:00
GHCi.Server: handle RunTH and RunModFinalizers in forked thread
This commit makes the external interpreter able to many interpret
template haskell splices in parallel without blocking the main server
thread. Simply: fork off a thread to handle a RunTH request. The handler
will issue the reverse-requests on the given RunTH mailbox, to keep each
RunTH isolated from the others.
`handleRemoteCall` was generalised slightly by allowing `WriteReply` to
be used linearly in the handler directly, so that the caller of
`handleRemoteCall` may `WriteReply` in the forked thread rather than in
the main thread.
The only question is how can we guarantee `WriteReply` is used linearly
if we want to use it inside a forked thread? We need two guarantees:
1. `WriteReply` is used in the forked thread regardless of whether an
exception is thrown to that thread.
2. The forked thread is always awaited for, so that the main program
doesn't exit before the thread has a chance to actually use
`WriteReply`.
We introduced a small `ThreadManager` which provides a `spawnFinally`
that provides these two guarantees:
1. is guaranteed by `forkFinally`, which always runs the clean up action
exactly once.
2. is guaranteed by `ThreadManager` being abstract and the only exposed
function that introduces `ThreadManager` being `withThreadManager`,
which always awaits for all spawned threads.
Co-authored-by: Rodrigo Mesquita <rodrigo.m.mesquita@gmail.com>
-
32f3e510
by Andrea Vezzosi at 2026-11-02T17:10:30+00:00
interp: spawn fillPipe thread, writePipe now safely interruptible
Previously an async exception during a call to `writePipe` could have caused a
partial write, making any further write impossible to parse from the other end.
Now `writePipe` merely pushes the message onto a queue, and the actual write
happens in a separate `fillPipe` thread. As a result of this and previous
changes, `interpCmd` no longer needs `interruptibleMask` to preserve the state
of the protocol.
`writePipe` waits for the actual write to complete, and rethrows exceptions
raised in the write attempt, if any.
-
2d2a2fe9
by Andrea Vezzosi at 2026-11-02T17:10:30+00:00
Follow `Note [Exceptions in TH]` in `qReport` implementation
`qReport b msg` should use `seqList msg` to trigger any exceptions in the
`String`, as explained in the note.
Through refactors the `seqList msg` call followed the bulk of the implementation
in `GHC.Tc.Gen.Splice.report`, but the call should be in `qReport` itself, so
that, even with -fexternal-interpreter, `msg` gets forced right away and any
exceptions get reported as a compilation error.
Follow up to 'interp: spawn fillPipe thread, writePipe now safely
interruptible'.
-
7949cc15
by Rodrigo Mesquita at 2026-11-02T17:10:30+00:00
test: make T3171 not fragile and add version for ext interp too
- actually test the sleep of the user-given expression gets interrupted
- the ext interp version tests two things:
- Requests can be cancelled without terminating the ext interp (#18852)
- The stmt being evaluated gets interrupted in the ext interp (that
is, ^C gets sent to the thread running the user-given expression
and doesn't affect the ext interp main thread)
- testing ext interp requires sending ^C to whole process group
Fixes #15383
-
7aa73b46
by Andrea Vezzosi at 2026-11-02T17:10:30+00:00
interp: cancel requests if async exceptions while waiting for reply.
-
a46421a3
by Andrea Vezzosi at 2026-11-02T17:10:30+00:00
interp: do not terminate ext. interp. process on exception
`sendMessage` throwing an exception does not warrant terminating the external
interpreter process anymore, because communication can keep going no matter what.
`stopInterp` now resets the whole `Interp` state, so that a new
ExtInterpInstance can be later created and work correctly.
`withExtInterp` now calls `stopInterp` on a `PipeClosedException` and rethrows.
Fixes #18852
-
348c4cd5
by Rodrigo Mesquita at 2026-11-02T17:19:05+00:00
add break033 checks continue after unsetting break on exception works
-
6630f5d4
by Rodrigo Mesquita at 2026-11-02T17:19:05+00:00
Add test checking we don't stop at breakpoints in TH
-
ff887d8b
by Rodrigo Mesquita at 2026-11-02T17:19:05+00:00
Test that :abandon doesn't hang when -fbreak-on-error
-
e17dfb2c
by Rodrigo Mesquita at 2026-11-02T17:19:05+00:00
run conc033 in ghci too
-
6e0b4e0a
by Rodrigo Mesquita at 2026-11-02T17:19:05+00:00
Add test checking a ^C interrupt of a :force interrupts the expression
-
2bbbb5ba
by Rodrigo Mesquita at 2026-11-02T17:19:05+00:00
test T3171 with -fno-ghci-sandbox too
-
602c4111
by Rodrigo Mesquita at 2026-11-02T17:20:50+00:00
Global break action which uses shared thread map
the status Var must actually be global across all threads, otherwise all
of them will block trying to write some Status Var no one is looking at
at all
Build EvalOpts once per exec rather than per breakpoint stop
in interpreter_steplocal the allocations increased because now we're
updating `eval_opts` after constructing them in `handleRunStatus` (which
is run many times since in step-local most often we don't stop, just
check).
So drop the update to the eval_opts, and, in fact, pull their
construction out. the loop always uses the same eval_opts, so no point
in constructing them inside from scratch every time.
-3.6% from this branch which had the regressions in
interpreter_steplocal
Look up a thread's ResumeContext without taking a lock
modifyMVar is unnecessarily expensive and blocking when most often we
just care about reading without modifying. optimize for default case
(thread context already exists). It is called every single time a
breakpoint is hit (regardless of whether we stop there, e.g. in
interpreter_steplocal we do not stop there often)
-3.3% improvement over original regression
Every thread resume-context keeps a var for its EvalStatus
Kill the global mapping from thread id to EvalStatus
Instead, keep the EvalStatus in the ResumeContext associated to a thread
in the threadContextVar (where entries are created on breakpoints being
hit and when writing an EvalComplete result)
Before, every breakpoint hit meant reading the global map TVar and
deleting a map entry which did a lot of allocations. Now, we have a
Nothing variable which we overwrite to "take" rather than deleting from
a Map and allocating the GlobalDbgMap too. Note that for every
breakpoint hit we'd have to insert the ThreadId entry in the map again
(because we deleted it last time we resumed), again allocating
unnecessarily. Now it's only a look up on the map and an update on the
variable directly.
And a LOT of breakpoints were hit on interpreter_steplocal, stress
testing the cost of hitting a breakpoint
The isolated threads move to their own top-level TVar Set.
-4.4% improvement over the regression
Move whether a thread is isolated to the same global map
Stops us from reconstructing the set every time this changed instead to
being a per thread variable just toggled on/off, by default non-isolated
like before (before it was Set.empty, Set.insert for True, Set.delete
for FALSE). This is more straightforward -- there's always an isolation
mode per thread and we can toggle it directly
reduces slightly allocations, but the main advantage is killing a second
global map and making it more uniform with just one thread map and no
membership check for toggledness.
Prefer foldl' in readAnyThreadEvalBreak txn
retry 'orElse' next is just next, so we might as well use a strict fold
TODO comment on breakOnException being global rather than per-thread
Set global breakpoint action in sandboxIO rather than withBreakAction
WIP: Asynchronous eval/resume
Fix commentary
clean up some comments in GHCi.Signals
unmask runIt only
Move cmmtn
internal interp: redirect interrupts on waits to the thread we're waiting for
Do redirectInterrupts to ext interp too
get rid of installSignalHandlers and discardCtrlC and uninterruptibleMask
fixes
Manage thread status stopped vs running and allow many threads to read the same result off of a stoped thread
abstracrt resumeStoppedThread
and use it in abandonStmt too.
fix redirect todo on seq/resumeseq
Add inline TODO about improving efficiency of readyAnyThreadBreak
It is currently O(N) in number of threads that have hit breakpoints
It should be number of *live* threads that have hit breaks, bc we still need to find a way to evacuate threads from the map
setBreakAction should also unset if flag is off.
(e.g. we don't want a :continue to keep flags if the :continue resume
request has unset things)
Fix writeCtxEvalStatus can overwrite existing breakpoint because of exceptions
kill EvalStatus_ for EvalStatus
resume :abandon until complete
Don't hang if we try to resume an already finished thread (and test it)
catch exception thrown while stopped at breakpoint