Hi Phil, Thanks for the reply, however that just gives me a forced deadlock removal as before. new bound thread (1) cap 0: schedule() cap 0: running thread 1 (ThreadRunGHC) cap 0: thread 1 stopped (blocked on an MVar) thread 1 @ 0000000003205388 is blocked on an MVar @ 00000000032040c8 (TSO_DIRTY) deadlocked, forcing major GC... all threads: threads on capability 0: other threads: thread 1 @ 0000000003205388 is blocked on an MVar @ 00000000032040c8 (TSO_DIRTY) cap 0: starting GC I don't believe any solution involving MVars will work for the non-threaded RTS. Though I'd love to be wrong here... Regards, Tamar On Sun, Jan 6, 2019 at 9:38 PM Phil Ruffwind wrote:

What if you wrap the MVar in a foreign closure?

import Control.Concurrent.MVar (newEmptyMVar, putMVar, takeMVar) import Control.Exception (bracket) import Foreign.Ptr (FunPtr, freeHaskellFunPtr)

foreign import ccall "wrapper" wrapAwaken :: IO () -> IO (FunPtr (IO ()))

main = do mvar <- newEmptyMVar bracket (wrapAwaken (putMVar mvar ())) freeHaskellFunPtr $ \ awaken -> do -- giveToExternalCode awaken takeMVar mvar

On Sun, Jan 6, 2019, at 10:37, Phyx wrote:

...

Hi All,

I'm looking for a way to block a task indefinitely until it is woken up by an external event in both the threaded and non-threaded RTS and returns a value that was stored/passed. MVar works great for the threaded RTS, but for the non-threaded there's a bunch of deadlock detection in the scheduler that would forcibly free the lock and resume the task with an opaque exception. This means that MVar and anything derived from it is not usable.

STMs are more expensive but also have the same deadlock code. So again no go. The reason it looks like a deadlock to the RTS is that the "Wake-up" call in the non-threaded rts will come from C code running inside the RTS. The RTS essentially just sees all tasks blocked on it's main capability and (usually rightly so) assumes a deadlock occurred.

You have other states like BlockedOnForeign etc but those are not usable as a primitive. Previous iterations of I/O managers have each invented primitives for this such as asyncRead#, but they are not general and can't be re-used, and requires a different solution for threaded and non-threaded.

I have started making a new primitive IOPort for this, based on the MVar code, but this is not trivial... (currently I'm getting a segfault *somewhere* in the primitive's cmm code). The reason is that the semantics are decidedly different from what MVars guarantee. I should also mention that this is meant to be internal to base (i.e no exported).

So before I continue down this path and some painful debugging..., does anyone know of a way to block a task, unblock it later and pass a value back? It does not need to support anything complicated such as multiple take/put requests etc.

Cheers, Tamar _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

I have simply copy pasted the code you provided. Note that my actual code code doesn't pass anything to a foreign interface. It stores everything in a Haskell mutable object. The RTS on completion of actions simply schedules a task which inspects this objects and wakes up as many blocked tasks as possible.

Strange, how could the scheduler assume a deadlock if the MVar could be called from a closure that is still alive? Can you show the code that you're testing?

Because as far as I can tell, it doesn't care. When it comes to MVars and STMs the scheduler assumes a deadlock when all tasks on all capabilities are blocked. For the threaded runtime it has an early exit condition from this code in the case where there has been any activity in a complete timeslice or when you're blocked on specific calls such as I/O. On the non-threaded runtime the timeslice case doesn't apply and you only have one capability, it will force a GC to try to revive some tasks, and if at the end of this the tasks are still blocked it will release one in order to attempt to proceed. In short, as far as I can tell I don't think it considers reach-ability at all, not for MVars. This is why things e.g. doing a takeMVar will also process pending puts etc, so that if you actually enter a blocked state, you know you had no other choice. On Mon, Jan 7, 2019 at 11:24 PM Phil Ruffwind wrote:

Strange, how could the scheduler assume a deadlock if the MVar could be called from a closure that is still alive? Can you show the code that you're testing?

On Mon, Jan 7, 2019, at 14:09, Phyx wrote:

...

Hi Phil,

Thanks for the reply, however that just gives me a forced deadlock removal as before.

new bound thread (1) cap 0: schedule() cap 0: running thread 1 (ThreadRunGHC) cap 0: thread 1 stopped (blocked on an MVar) thread 1 @ 0000000003205388 is blocked on an MVar @ 00000000032040c8 (TSO_DIRTY) deadlocked, forcing major GC... all threads: threads on capability 0: other threads: thread 1 @ 0000000003205388 is blocked on an MVar @ 00000000032040c8 (TSO_DIRTY) cap 0: starting GC

I don't believe any solution involving MVars will work for the non-threaded RTS. Though I'd love to be wrong here...

Regards, Tamar

Maybe that should be considered a false positive (bug) for the deadlock checker? Just because the Haskell runtime has a single thread, that doesn't imply the whole program is necessarily single-threaded (in the presence of foreign things). I'd think this is a legitimate use case for MVars.

Perhaps, I can see it both ways really :(. The papers and docs for MVars put a strong emphasis on this deadlock guarantee (though I've really only skimmed the paper), taking foreign calls into consideration does somewhat weaken this of course. I do agree with you that this wasn't the behavior I expected from MVars and STMs, particularly because they are such low level building blocks, for instance QSem is constructed with MVar, the docs make no mention of this but I suspect QSem and the rest all don't do what you'd expect on the non-threaded RTS. I did try removing this check to see, but it really didn't like that. It caused GC to be triggered over and over again as the RTS tried desperately to find something to do, doesn't seem to consider "do nothing" as a valid state. On Tue, Jan 8, 2019 at 6:59 AM Phil Ruffwind wrote:

Okay, I skimmed rts/Schedule.c and now see the problem you mentioned :(

...

On the non-threaded runtime the timeslice case doesn't apply and you only have one capability, it will force a GC to try to revive some tasks, and if at the end of this the tasks are still blocked it will release one in order to attempt to proceed. In short, as far as I can tell I don't think it considers reach-ability at all, not for MVars.

Maybe that should be considered a false positive (bug) for the deadlock checker? Just because the Haskell runtime has a single thread, that doesn't imply the whole program is necessarily single-threaded (in the presence of foreign things). I'd think this is a legitimate use case for MVars.

Oh, I see :( I guess it's not that easy of a fix then. Perhaps the RTS could use a new intrinsic for blocking on foreign state

Yeah, that's what I was/am currently working on, "IOPort" has much of the same property of MVar but doesn't have this deadlock guarantee and only supports a single put/take at a time. But debugging CMM is... not fun :( so I was wondering if I was just missing something with the existing mechanisms. On Tue, Jan 8, 2019 at 8:23 AM Phil Ruffwind wrote:

...

I did try removing this check to see, but it really didn't like that. It caused GC to be triggered over and over again as the RTS tried desperately to find something to do, doesn't seem to consider "do nothing" as a valid state.

Oh, I see :( I guess it's not that easy of a fix then. Perhaps the RTS could use a new intrinsic for blocking on foreign state.

I'm trying to model an Asynchronous I/O call/interface against a synchronous Haskell call. I want the behavior of an unsafe blocking foreign call, without blocking in the foreign call itself. On Tue, Jan 8, 2019 at 6:24 PM Carter Schonwald wrote:

What’s the underlying problem you’re trying to model?

On Tue, Jan 8, 2019 at 3:56 AM Phyx wrote:

...

...

Oh, I see :( I guess it's not that easy of a fix then. Perhaps the RTS could use a new intrinsic for blocking on foreign state

Yeah, that's what I was/am currently working on, "IOPort" has much of the same property of MVar but doesn't have this deadlock guarantee and only supports a single put/take at a time. But debugging CMM is... not fun :( so I was wondering if I was just missing something with the existing mechanisms.

On Tue, Jan 8, 2019 at 8:23 AM Phil Ruffwind wrote:

...

...

I did try removing this check to see, but it really didn't like that. It caused GC to be triggered over and over again as the RTS tried desperately to find something to do, doesn't seem to consider "do nothing" as a valid state.

Oh, I see :( I guess it's not that easy of a fix then. Perhaps the RTS could use a new intrinsic for blocking on foreign state.

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Hi John, I can, but the only reason that would work is because a blocking foreign call is except from the deadlock detection code. I can't block on the Handle itself as all the I/O calls are non-blocking for performance reasons. Instead what I want to do is just pause the current task, which means I don't have to allocate OS level locks for each task and have a mutable state for each task, which is essentially what an MVar does. My other concern is that using OS primitives would hit performance a bit, which increases as the load increases. So I'd much rather do it purely in the Haskell world. Kind regards, Tamar On Mon, Jan 7, 2019 at 5:44 AM John Lato wrote:

Can you use an os-level structure? E.g. block on a file descriptor, socket, or something like that?

On Sun, Jan 6, 2019, 10:37 Phyx

...

Hi All,

I'm looking for a way to block a task indefinitely until it is woken up by an external event in both the threaded and non-threaded RTS and returns a value that was stored/passed. MVar works great for the threaded RTS, but for the non-threaded there's a bunch of deadlock detection in the scheduler that would forcibly free the lock and resume the task with an opaque exception. This means that MVar and anything derived from it is not usable.

STMs are more expensive but also have the same deadlock code. So again no go. The reason it looks like a deadlock to the RTS is that the "Wake-up" call in the non-threaded rts will come from C code running inside the RTS. The RTS essentially just sees all tasks blocked on it's main capability and (usually rightly so) assumes a deadlock occurred.

You have other states like BlockedOnForeign etc but those are not usable as a primitive. Previous iterations of I/O managers have each invented primitives for this such as asyncRead#, but they are not general and can't be re-used, and requires a different solution for threaded and non-threaded.

I have started making a new primitive IOPort for this, based on the MVar code, but this is not trivial... (currently I'm getting a segfault *somewhere* in the primitive's cmm code). The reason is that the semantics are decidedly different from what MVars guarantee. I should also mention that this is meant to be internal to base (i.e no exported).

So before I continue down this path and some painful debugging..., does anyone know of a way to block a task, unblock it later and pass a value back? It does not need to support anything complicated such as multiple take/put requests etc.

Cheers, Tamar _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs