[Git][ghc/ghc][wip/romes/async-rpc] 2 commits: interp: spawn fillPipe thread, writePipe now safely interruptible
Andrea Vezzosi pushed to branch wip/romes/async-rpc at Glasgow Haskell Compiler / GHC Commits: 256dcae4 by Andrea Vezzosi at 2026-10-22T17:48:37+02:00 interp: spawn fillPipe thread, writePipe now safely interruptible Previously an async exception during a call to `writePipe` could have caused a partial write, making any further write impossible to parse from the other end. Now `writePipe` merely pushes the message onto a queue, and the actual write happens in a separate `fillPipe` thread. As a result of this and previous changes, `interpCmd` no longer needs `interruptibleMask` to preserve the state of the protocol. `writePipe` waits for the actual write to complete, and rethrows exceptions raised in the write attempt, if any. - - - - - 77a50ed6 by Andrea Vezzosi at 2026-10-23T14:41:26+02:00 Follow `Note [Exceptions in TH]` in `qReport` implementation `qReport b msg` should use `seqList msg` to trigger any exceptions in the `String`, as explained in the note. Through refactors the `seqList msg` call followed the bulk of the implementation in `GHC.Tc.Gen.Splice.report`, but the call should be in `qReport` itself, so that, even with -fexternal-interpreter, `msg` gets forced right away and any exceptions get reported as a compilation error. Follow up to 'interp: spawn fillPipe thread, writePipe now safely interruptible'. - - - - - 8 changed files: - compiler/GHC/Runtime/Interpreter.hs - compiler/GHC/Tc/Errors/Types.hs - compiler/GHC/Tc/Gen/Splice.hs - libraries/ghci/GHCi/Message/Async.hs - libraries/template-haskell/Language/Haskell/TH/Syntax.hs - + testsuite/tests/th/T8987b.hs - + testsuite/tests/th/T8987b.stderr - testsuite/tests/th/all.T Changes: ===================================== compiler/GHC/Runtime/Interpreter.hs ===================================== @@ -197,9 +197,7 @@ interpCmd interp msg = case interpInstance interp of InternalInterp -> run msg -- Just run it directly #endif ExternalInterp ext -> withExtInterp ext $ \inst -> - uninterruptibleMask_ $ -- Note [uninterruptibleMask_ and interpCmd] - sendMessage inst msg - + sendMessage inst msg withExtInterp :: ExceptionMonad m => ExtInterp -> (forall d. ExtInterpInstance d -> m a) -> m a withExtInterp ext action = case ext of @@ -213,15 +211,6 @@ withExtInterpStatus ext action = case ext of ExtWasm i -> action $ interpStatus i ExtIServ i -> action (interpStatus i) --- Note [uninterruptibleMask_ and interpCmd] --- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ --- If we receive an async exception, such as ^C, while communicating --- with the iserv process then we will be out-of-sync and not be able --- to recover. Thus we use uninterruptibleMask_ during --- communication. A ^C will be delivered to the iserv process (because --- signals get sent to the whole process group) which will interrupt --- the running computation and return an EvalException result. - -- | Grab a lock on the 'IServ' and do something with it. -- Overloaded because this is used from TcM as well as IO. withIServ ===================================== compiler/GHC/Tc/Errors/Types.hs ===================================== @@ -7091,6 +7091,7 @@ data SpliceFailReason th/T5976 th/T7276a th/T8987 + th/T8987b th/TH_exn1 th/TH_exn2 th/TH_runIO ===================================== compiler/GHC/Tc/Gen/Splice.hs ===================================== @@ -1472,11 +1472,9 @@ when showing an error message. To call runQ in the Tc monad, we need to make TcM an instance of Quasi: -} --- 'msg' is forced to ensure exceptions don't escape, --- see Note [Exceptions in TH] report :: Bool -> [Char] -> TcM () -report True msg = seqList msg $ addErr $ TcRnTHError $ ReportCustomQuasiError True msg -report False msg = seqList msg $ addDiagnostic $ TcRnTHError $ ReportCustomQuasiError False msg +report True msg = addErr $ TcRnTHError $ ReportCustomQuasiError True msg +report False msg = addDiagnostic $ TcRnTHError $ ReportCustomQuasiError False msg addTopDecls :: [TH.Dec] -> TcM () addTopDecls thds = do @@ -1575,7 +1573,7 @@ metaHandlersTcM runInIO = TH.MetaHandlers { ; let i = toInteger (getKey u) ; return (TH.mkNameU s i) } - , mReport = fmap runInIO . report + , mReport = \ b msg -> runInIO (report b msg) , mLocation = runInIO location ===================================== libraries/ghci/GHCi/Message/Async.hs ===================================== @@ -77,6 +77,10 @@ data Pipe = Pipe , pipeReadClosed :: !(TVar Bool) -- ^ After the pipe is closed, messages can no longer be received from the -- other end, but the 'AsyncManager' queues might not be empty yet. + , pipeWriteClosed :: !(TVar Bool) + -- ^ After the write end of the pipe is closed we can no longer submit + -- requests or respond to ones we receive, but we might still receive + -- replies. } -- | Make a 'Pipe' from a 'Handle' to read and a 'Handle' to write. @@ -91,7 +95,13 @@ mkPipeFromHandles pipeRead pipeWrite = do -- | Make a 'Pipe' from a reader function and a writer function. -- The pipe should be closed with 'closePipe', which will call 'pipeCloseRead' and 'pipeCloseWrite'. -mkPipeFromContinuations :: IO ByteString -> (B.Builder -> IO ()) +-- +-- Arguments 'getSome' and 'putAll' should throw an `IOException` when communication fails +-- or is no longer possible. +mkPipeFromContinuations :: IO ByteString + -- ^ getSome + -> (B.Builder -> IO ()) + -- ^ putAll -> IO () -- ^ pipeCloseRead -> IO () @@ -101,6 +111,7 @@ mkPipeFromContinuations getSome putAll pipeCloseRead pipeCloseWrite = mdo pipeLeftovers <- newIORef Nothing pipeLock <- newMVar () pipeReadClosed <- newTVarIO False + pipeWriteClosed <- newTVarIO False mdo pipeAsyncMngr <- newAsyncManager p let p = Pipe { .. } @@ -114,20 +125,34 @@ mkPipeFromContinuations getSome putAll pipeCloseRead pipeCloseWrite = mdo -- write. Only then do we stop the drain thread, which closes the read end -- itself once its read has finished. On Windows its read can't be -- interrupted, so this waits for the other side to close its write end too. +-- Finally we stop the write thread, which could be blocked waiting for more to write. closePipe :: Pipe -> IO () closePipe p@Pipe{..} = do readAlreadyClosed <- atomically $ do closed <- readTVar pipeReadClosed failPendingReplies p (toException PipeReadClosedException) pure closed - withLock p pipeCloseWrite `finally` - unless readAlreadyClosed - (throwTo (asyncDrainTid pipeAsyncMngr) PipeReadClosedException) + + writeAlreadyClosed <- atomically $ do + closed <- readTVar pipeWriteClosed + failPendingWrites p (toException PipeWriteClosedException) + pure closed + + unless writeAlreadyClosed (withLock p pipeCloseWrite) + `finally` do + unless readAlreadyClosed + (throwTo (asyncDrainTid pipeAsyncMngr) PipeReadClosedException) + unless writeAlreadyClosed + (throwTo (asyncFillTid pipeAsyncMngr) PipeWriteClosedException) -- | We've closed the pipe read end on this process data PipeReadClosedException = PipeReadClosedException deriving Show instance Exception PipeReadClosedException +-- | We've closed the pipe write end on this process +data PipeWriteClosedException = PipeWriteClosedException deriving Show +instance Exception PipeWriteClosedException + -------------------------------------------------------------------------------- -- ** Pipe thread-safe read/write -------------------------------------------------------------------------------- @@ -135,7 +160,14 @@ instance Exception PipeReadClosedException -- and identify whether they are requests or replies. writePipe :: Pipe -> IddMsg -> IO () -writePipe p@Pipe{..} = withLock p . putAll . execPut . put +writePipe Pipe{..} msg = do + var <- newEmptyTMVarIO + atomically $ do + readTVar pipeWriteClosed >>= \case + True -> throwSTM PipeWriteClosedException + False -> pure () + writeTQueue (writeQueue pipeAsyncMngr) (msg, var) + either throwIO pure =<< atomically (takeTMVar var) -- | Only the 'drainPipe' thread should read from the pipe directly. Other -- functions should wait for the matching reply to be delivered asynchronously. @@ -181,6 +213,8 @@ withLock Pipe{..} = bracket (takeMVar pipeLock) (putMVar pipeLock) . const -------------------------------------------------------------------------------- -- | Send an asynchronous request and block waiting for a matching reply +-- +-- See Note [Exceptions during serialization] about exceptions the 'Put' argument might raise. remoteAsyncReq :: Pipe -> Put -> (Maybe InboxId) -> IO (Either SomeException LB.ByteString) remoteAsyncReq pipe msg ii = do @@ -222,6 +256,8 @@ remoteAsyncReq pipe msg ii = do -- It must be used exactly once (all requests expect a reply, and you can't -- reply more than once). Linear types enforce the handler replies in every -- branch, and if the handler otherwise throws, it is handled by 'handleAsyncReq'. +-- +-- See Note [Exceptions during serialization] about exceptions the 'Put' argument might raise. handleAsyncReq :: Pipe -> Get a -> (Maybe InboxId) -> (WriteReply Put %1 -> a -> IO r) -> IO r handleAsyncReq pipe getit mii k = mask $ \restore -> do let am = pipeAsyncMngr pipe @@ -229,6 +265,10 @@ handleAsyncReq pipe getit mii k = mask $ \restore -> do -- Block waiting for request mbreq <- atomically $ do + readTVar (pipeWriteClosed pipe) >>= \case + -- We will not be able to write the reply, so no point in continuiting. + True -> throwSTM PipeWriteClosedException + False -> pure () isClosed <- readTVar (pipeReadClosed pipe) taken <- case mii of Nothing -> tryReadTQueue $ requestsQueue am @@ -320,7 +360,10 @@ data AsyncManager = AsyncManager , mailboxQueues :: !(TVar (IntMap (TQueue (CorrelationId, LB.ByteString)))) -- ^ Requests for a specific 'InboxId' that can be read using -- @'handleAsyncReq' p get (Just <mailboxid>)@. + , writeQueue :: !(TQueue (IddMsg, TMVar (Either SomeException ()))) + -- ^ Messages to write on the pipe. , asyncDrainTid :: !ThreadId + , asyncFillTid :: !ThreadId , asyncFreshId :: !(IORef Int) } @@ -363,11 +406,16 @@ newAsyncManager pipe = do pendingReply <- newTVarIO IntMap.empty requestsQueue <- newTQueueIO mailboxQueues <- newTVarIO IntMap.empty + writeQueue <- newTQueueIO -- Fork thread to drain read end. -- (cleaned up on 'closePipe') asyncDrainTid <- forkIOWithUnmask (drainPipe pipe) + -- Fork thread to fill write end. + -- (cleaned up on 'closePipe') + asyncFillTid <- forkIOWithUnmask (fillPipe pipe) + let am = AsyncManager{..} pure am @@ -417,6 +465,55 @@ drainPipe p unmask = catch (unmask $ loop `finally` pipeCloseRead p) replyToPend am = pipeAsyncMngr p +{- +Note [Exceptions during serialization] +----------------------------------------------------------------------------- + +For efficiency reasons, any `IddMsg` to be sent is forced only when already +writing onto the Pipe, so we cannot discern whether any raised exception +originated due to forcing during serialization or from interacting with the +Pipe. + +If the exceptions happen to be `SomeAsyncException` or `IOException` the write +end of the `Pipe` will be closed while rethrowing them, as those are likely to +indicate something is wrong with the `Pipe` itself. Same for +PipeWriteClosedException, used by `closePipe`. + +It is recommended to fully force messages at the source, to avoid needlessly +interrupting communication. +-} + + +-- | "Fill"s write end of the Pipe with queued messages. +-- +-- Because Pipe is written to from this thread rather than 'writePipe', calls to +-- the latter can be interrupted without compromising the protocol. +-- +-- Closes the Pipe write end when stopping, notifying any messages still queued to write. +fillPipe :: Pipe -> (forall x. IO x -> IO x) -> IO () +fillPipe p unmask = catch (loop `finally` pipeCloseWrite p) (atomically . failPendingWrites p) where + loop = do + (msg,var) <- atomically $ readTQueue (writeQueue am) + success <- handle (\ e -> atomically (putTMVar var (Left e)) >> rethrowIfPipeFatal e >> pure False) $ do + unmask $ withLock p (putAll p . execPut . put $ msg) + pure True + when success $ atomically $ putTMVar var (Right ()) + loop + + -- See Note [Exceptions during serialization] + rethrowIfPipeFatal e + | Just SomeAsyncException{} <- fromException e = throwIO e + | Just (_ :: IOException) <- fromException e = throwIO e + | Just PipeWriteClosedException <- fromException e = throwIO e + | otherwise = pure () + am = pipeAsyncMngr p + +failPendingWrites :: Pipe -> SomeException -> STM () +failPendingWrites p e = do + writeTVar (pipeWriteClosed p) True + writes <- flushTQueue (writeQueue (pipeAsyncMngr p)) + mapM_ (`writeTMVar` Left e) (map snd writes) + failPendingReplies :: Pipe -> SomeException -> STM () failPendingReplies p e = do -- reqs are only added if pipe not closed; ===================================== libraries/template-haskell/Language/Haskell/TH/Syntax.hs ===================================== @@ -528,7 +528,9 @@ class (MonadIO m, MonadFail m) => Quasi m where -- | Report an error (True) or warning (False) -- ...but carry on; use 'fail' to stop. See 'report'. qReport :: Bool -> String -> m () - qReport b s = qRunQ $ report b s + -- 'msg' is forced to ensure exceptions don't escape, + -- see Note [Exceptions in TH] + qReport b msg = seqList msg $ qRunQ $ Internal.report b msg -- | See 'recover'. qRecover :: m a -- ^ the error handler @@ -665,13 +667,18 @@ instance Quasi Q where -- | Report an error (True) or warning (False), -- but carry on; use 'fail' to stop. report :: Bool -> String -> Q () -report = Internal.report +report = qReport {-# DEPRECATED report "Use reportError or reportWarning instead" #-} -- deprecated in 7.6 -- | Report an error to the user, but allow the current splice's computation to carry on. To abort the computation, use 'fail'. reportError :: String -> Q () -reportError = report True +reportError = qReport True -- | Report a warning to the user, and carry on. reportWarning :: String -> Q () -reportWarning = report False +reportWarning = qReport False + +-- | copied from GHC.Utils.Misc +seqList :: [a] -> b -> b +seqList [] b = b +seqList (x:xs) b = x `seq` seqList xs b ===================================== testsuite/tests/th/T8987b.hs ===================================== @@ -0,0 +1,8 @@ +{-# LANGUAGE TemplateHaskell #-} + +module T8987b where +import Language.Haskell.TH +import System.IO.Error +import Control.Exception + +$(reportWarning ['1', throw (userError "IOError in warning")] >> return []) ===================================== testsuite/tests/th/T8987b.stderr ===================================== @@ -0,0 +1,6 @@ +T8987b.hs:1:1: error: [GHC-87897] + Exception when trying to run compile-time code: + user error (IOError in warning) + Code: (reportWarning ['1', throw (userError "IOError in warning")] + >> return []) + ===================================== testsuite/tests/th/all.T ===================================== @@ -306,6 +306,7 @@ test('T8884', normal, compile, ['-v0']) test('T8954', normal, compile, ['-v0']) test('T8932', normal, compile_fail, ['-v0']) test('T8987', normal, compile_fail, ['-v0']) +test('T8987b', normal, compile_fail, ['-v0']) test('T7241', normal, compile_fail, ['-v0']) test('T9262', normal, compile, ['-v0']) test('T9199', normal, compile, ['-v0']) View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/c7a4eba097c6b36069cd497c562a788... -- View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/c7a4eba097c6b36069cd497c562a788... You're receiving this email because of your account on gitlab.haskell.org. Manage all notifications: https://gitlab.haskell.org/-/profile/notifications | Help: https://gitlab.haskell.org/help
participants (1)
-
Andrea Vezzosi (@trac-Saizan)