On Fri, Jun 08, 2012 at 12:41:29PM +0300, Michael Snoyman wrote:

On Jun 8, 2012 11:37 AM, "Simon Marlow" wrote:

...

I'd like to add a higher-level concurrency API to the base package.

What's the advantage to putting this in base instead of a separate package? If it goes in base, it will make it more difficult to upgrade, and take longer for this module to be adopted at all. If possible, I'd opt for a standalone package.

I agree. This doesn't sound like it needs to be a package shipped with GHC, let alone in the base package. Thanks Ian

On Jun 8, 2012 1:07 PM, "Simon Marlow" wrote:

On 08/06/2012 10:41, Michael Snoyman wrote:

...

What's the advantage to putting this in base instead of a separate package? If it goes in base, it will make it more difficult to upgrade, and take longer for this module to be adopted at all. If possible, I'd opt for a standalone package.

Uploading it to Hackage is certainly an option, and there are arguments

in both directions. My thinking was:

- it's a bit small for a package by itself. There's a lot of overhead for a package (github repo, Haskell Platform proposal, issue tracker, blah blah)

- To avoid further fragmentation, I would like this package to be more visible, especially if we go to the trouble as a community of building some consensus around it.

- The obvious package name 'async' is already taken

Cheers, Simon

Maybe we could address some of those issues: perhaps Peter Simons would be willing to forfeit the package name in favor of this updated one. That would provide more consolidation, and let the community-driven package use the obvious name. Michael

+1

What's the advantage to putting this in base instead of a separate

...

package? If it goes in base, it will make it more difficult to upgrade, and take longer for this module to be adopted at all. If possible, I'd opt for a standalone package.

I thought the idea was that we would try to point people towards using these as a best practice as opposed to using forkIO? I find that a helper function to fork a child thread with error handling is something I'm always pasting into various projects [1], which I take as a sign it should be easily available. I never really thought about looking for better threads packages on hackage, because I didn't want to add an extra dependency (or deal with something big and complex). But if it were standard I would have picked it up! -Ryan [1] P.S. Non-sequitor: The other little helper function I'm ALWAYS pasting into files is a function to print large numbers with interspersed commas. I'd like to see THAT in the standard ;-), but I realize there are internationalization issues.

Thomas Schilling wrote:

On 8 June 2012 11:07, Simon Marlow wrote:

...

Uploading it to Hackage is certainly an option, and there are arguments in both directions. My thinking was: [..] - it's a bit small for a package by itself. There's a lot of overhead for a package (github repo, Haskell Platform proposal, issue tracker, blah blah)

Github provides much of the infrastructure by default, like an issue tracker.

I am very much against putting things into base unless there is a *very* good argument to do this. If it is supposed to be *the* standard package, then it belongs into the platform, not base. If the platform process is too heavy-weight to discourage this path, then we need to do something about that rather than sneak things in through the base library. It already is a pain to work around issues in the base library (and many other core packages) since it's (a) takes a long time for an update to be released, and (b) cannot be upgraded independently from GHC (pulling in all the changes to other libraries as well).

I'm glad that the name issues is resolved. If it is a small library a platform addition proposal shouldn't be too difficult either. We just have to beware of too much bike shedding.

(BTW, have we had another platform proposal since the text package?)

+1 for a separate package that is included in the Haskell Platform. If merging a package into the base library is easier than the process of including it in the Haskell Platform, then the latter needs to be fixed. Best regards, Heinrich Apfelmus -- http://apfelmus.nfshost.com

On 6/9/12 4:32 AM, Herbert Valerio Riedel wrote:

Simon Marlow writes:

...

- To avoid further fragmentation, I would like this package to be more visible, especially if we go to the trouble as a community of building some consensus around it.

...would it help to advertise the new `async` package in the appropriate places in the Haddoc documentation for `base` to make it more visible?

I wonder if it'd be worthwhile to have a "metabase package" which includes all the modules which are, used to be, or 'should' be in base. What I mean is not to have an actual package (in the ghc/hackage sense), but just to have a single unified Haddock site that covers and advertises all of: * base * the things broken out from base-3.0 (array, bytestring, containers, directory, old-locale, old-time, packedstring, pretty, process, random)--- modulo things like packedstring which have been deprecated. * things broken out from base-4.0 (syb,...) * things which 'should' be part of base (text? vector? async?) * but not including all of the HP Perhaps this already exists in an official incarnation somewhere, but if so then it's poorly advertised. The url should be listed in the description of the base package on Hackage, at the very least. -- Live well, ~wren

On 08/06/2012 17:31, Thomas Schilling wrote:

I fixed the docs a tiny bit in: https://github.com/simonmar/async/pull/1

Other comments:

- Module introduction is missing.

Now added.

- There are no tests whatsoever. I would like to have a few stress tests that throw lots of asynchronous exceptions to make sure that exception masking etc behaves correctly.

Also added, including a couple of stress tests. (but the test coverage isn't great, I'll try to improve it in due course)

- Why is there no waitAnyCancel? - Same with waitEitherCancel

Both added. I think that now addresses all the comments that people had. There's one more change I'm thinking about: perhaps instead of wait :: Async a -> IO (Either SomeException a) waitThrow :: Async a -> IO a we should reverse the naming scheme, rename waitThrow to wait and wait to something else (waitCatch?). Rationale: waitThrow seems to be the version we need most often, and it's simpler to use. wait :: Async a -> IO a waitCatch :: Async a -> IO (Either SomeException a) Thoughts? Better names for waitCatch? Latest Haddocks: http://community.haskell.org/~simonmar/async/ Repo: https://github.com/simonmar/async Cheers, Simon

On 8 June 2012 09:37, Simon Marlow wrote:

...

I'd like to add a higher-level concurrency API to the base package. This API has grown while I've been working on examples for my book, and I'm convinced that we should have something like this in the standard libraries. Here's the API I propose:

http://community.haskell.org/~simonmar/async-stm/Control-Concurrent-Async.ht...

In fact it already exists in at least two packages on Hackage: 'async' and 'threads'. My version is a superset of both of these, except for a few small differences (which are up for discussion, of course).

One good reason to have this package is that it avoids some common pitfalls, such as forgetting to handle exceptions in your child threads. In the Async API, you get to choose whether to (a) receive the result as 'Either SomeException a', or (b) have the exception re-thrown in the waiting thread. You don't get to ignore it altogether.

Another common pitfall avoided by this library is accidentally leaving threads running in the background. This is handled by withAsync:

withAsync :: IO a -> (Async a -> IO b) -> IO b

the child thread is always cancelled (i.e. killed) when the second argument returns or throws an exception, if it hasn't finished already.

I'm opening this up for discussion so that we can tidy up any inconsistent details and add any functions that people feel are missing. Naming is obviously up for discussion too.

Cheers, Simon

_______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries

On 12/06/2012 12:38, Gábor Lehel wrote:

On Tue, Jun 12, 2012 at 12:58 PM, Simon Marlow wrote:

...

I think that now addresses all the comments that people had.

There's one more change I'm thinking about: perhaps instead of

wait :: Async a -> IO (Either SomeException a) waitThrow :: Async a -> IO a

we should reverse the naming scheme, rename waitThrow to wait and wait to something else (waitCatch?). Rationale: waitThrow seems to be the version we need most often, and it's simpler to use.

wait :: Async a -> IO a waitCatch :: Async a -> IO (Either SomeException a)

Thoughts? Better names for waitCatch?

Latest Haddocks:

http://community.haskell.org/~simonmar/async/

Two small ones:

- In the documentation for cancel: "Has no effect if the Async has already", it seems like something might have been cut off.

- Can we find a more accurate name for tryWait? It doesn't try and possibly fail to wait, it avoids waiting altogether. Maybe don'tWait, although that doesn't say what it does do. The only other idea I have is 'check' which is not very descriptive...

Good point. How about "poll"? Cheers, Simon

...

Repo:

https://github.com/simonmar/async

Cheers, Simon

...

On 8 June 2012 09:37, Simon Marlow wrote:

...

I'd like to add a higher-level concurrency API to the base package. This API has grown while I've been working on examples for my book, and I'm convinced that we should have something like this in the standard libraries. Here's the API I propose:

http://community.haskell.org/~simonmar/async-stm/Control-Concurrent-Async.ht...

In fact it already exists in at least two packages on Hackage: 'async' and 'threads'. My version is a superset of both of these, except for a few small differences (which are up for discussion, of course).

One good reason to have this package is that it avoids some common pitfalls, such as forgetting to handle exceptions in your child threads. In the Async API, you get to choose whether to (a) receive the result as 'Either SomeException a', or (b) have the exception re-thrown in the waiting thread. You don't get to ignore it altogether.

Another common pitfall avoided by this library is accidentally leaving threads running in the background. This is handled by withAsync:

withAsync :: IO a -> (Async a -> IO b) -> IO b

the child thread is always cancelled (i.e. killed) when the second argument returns or throws an exception, if it hasn't finished already.

I'm opening this up for discussion so that we can tidy up any inconsistent details and add any functions that people feel are missing. Naming is obviously up for discussion too.

Cheers, Simon

_______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries

_______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries

On Tue, Jun 12, 2012 at 3:37 PM, Gábor Lehel wrote:

On Tue, Jun 12, 2012 at 2:26 PM, Simon Marlow wrote:

...

...

- Can we find a more accurate name for tryWait? It doesn't try and possibly fail to wait, it avoids waiting altogether. Maybe don'tWait, although that doesn't say what it does do. The only other idea I have is 'check' which is not very descriptive...

Good point.  How about "poll"?

Definitely better.

Just ideas: - waitAny and company could be generalized to any Foldable. Downside: it makes the type signatures less obvious. Upside: if people will be taking the completed Async and removing it from the list (and doing this frequently), they might prefer something more efficient than a list. - If we want to follow the analogy, waitAny could be called 'select'.

I think we should distinguish two cases for "cancel". I emphasized one case and rambled a bit in my previous message. I hope this is clearer. On 2012/06/12 11:33, Simon Marlow wrote:

I appreciate that sometimes cancel may block and that you may want a non-blocking cancel, but I don't think that should be the default.

Firstly, the user might find it surprising if when withAsync has returned, the Async still performs some more operations (that are not inside an exception handler). I think it's an important guarantee that this doesn't happen - otherwise, what can you say with any certainty about the behaviour of withAsync?

I totally 100% have always agreed with that -- for withAsync. That is the correct semantics to minimize surprise when sequencing the two IO actions. But... you also include "race" and "concurrently" where the user has explicitly asked to go crazy with lots of non-deterministic interleaving. It may be that some are talking to blocking FFI things and thus it may be nice to have a version of these that calls asyncCancel internally:

Secondly, asynchronous cancel is easily implemented by the user, as

asyncCancel = async . cancel

If a user wanted race + asyncCancel then she could copy-paste your module's code with the changes. This is certainly something the user could do -- and the documentation can point out when it would be needed. I hope I have been clearer. I was not being crazy and says withAsync needed asyncCancel. -- Chris

On 09/06/12 17:50, Iavor Diatchki wrote:

Hello, A nice library, I've had to re-implement similar functionality, so this would be useful!

Some questions about the waiting combinators:

1. I was expecting that `waitAny` and friends, would also return the result of the first computation that completed. Is there a reason not to do so? 2. Is `waitAny` waiting for one of the Asyncs to complete _successfully_ or is throwing an exception considered as completion? I am guessing the former, with `waitAnyThrow` providing the latter functionality? 3. As far as I understand, `waitAny` does not stop the other asyncs (it'd be nice to add this note to its documentation, explicitly). If that's the case, perhaps it's type should be: `waitAny :: [Async a] -> IO (a,[Async a])`, so that we can keep on waiting for more asyncs, if needed?

All good points. I don't have many use cases for the waitAny* family yet so I'd been holding off refining the API until I had a clearer picture of what would be useful for clients. There are lots of possibilities here. But in light of your comments I have changed the type of waitAny to this: waitAny :: [Async a] -> IO (Async a, Either SomeException a) so that it now returns both the Async that completed and its result. Since Async supports Eq, the caller can remove the completed one from the list and call waitAny again if necessary. Similarly waitAnyThrow now has this type: waitAnyThrow :: [Async a] -> IO (Async a, a) I've also added a non-blocking wait: tryWait :: Async a -> IO (Maybe (Either SomeException a)) so that after a waitAny the caller can check all of the Asyncs for completion and remove multiple completed Asyncs from the list. Latest Haddocks here: http://community.haskell.org/~simonmar/async/ Cheers, Simon

-Iavor

On Fri, Jun 8, 2012 at 1:37 AM, Simon Marlow mailto:marlowsd@gmail.com> wrote:

I'd like to add a higher-level concurrency API to the base package. This API has grown while I've been working on examples for my book, and I'm convinced that we should have something like this in the standard libraries. Here's the API I propose:

http://community.haskell.org/~__simonmar/async-stm/Control-__Concurrent-Asyn... http://community.haskell.org/~simonmar/async-stm/Control-Concurrent-Async.ht...

In fact it already exists in at least two packages on Hackage: 'async' and 'threads'. My version is a superset of both of these, except for a few small differences (which are up for discussion, of course).

One good reason to have this package is that it avoids some common pitfalls, such as forgetting to handle exceptions in your child threads. In the Async API, you get to choose whether to (a) receive the result as 'Either SomeException a', or (b) have the exception re-thrown in the waiting thread. You don't get to ignore it altogether.

Another common pitfall avoided by this library is accidentally leaving threads running in the background. This is handled by withAsync:

withAsync :: IO a -> (Async a -> IO b) -> IO b

the child thread is always cancelled (i.e. killed) when the second argument returns or throws an exception, if it hasn't finished already.

I'm opening this up for discussion so that we can tidy up any inconsistent details and add any functions that people feel are missing. Naming is obviously up for discussion too.

Cheers, Simon

_________________________________________________ Libraries mailing list Libraries@haskell.org mailto:Libraries@haskell.org http://www.haskell.org/__mailman/listinfo/libraries http://www.haskell.org/mailman/listinfo/libraries

On 13/06/2012 22:58, Sjoerd Visscher wrote:

Hi Simon,

This a really nice and clean library!

I was wondering if it would be useful to have an applicative interface for the concurrently function? Something like this:

(page1, page2, page3) <- runConcurrently $ (,,) <$> Concurrently (getURL "url1") <*> Concurrently (getURL "url2") <*> Concurrently (getURL "url3")

More code here: https://gist.github.com/2926572

I'm not sure about this. What you get with the above code is a strange nesting of concurrently calls, whereas what the user might expect is for it to behave like the existing concurrently but on 3-tuples instead of pairs. I like the idea of generalising to arbitrary Traversable structures though. Maybe we could provide something like doConcurrently :: Traversable t => t (IO a) -> IO (t a) I haven't tried to write it, but it looks like it ought to be possible. Cheers, Simon

Thanks, Sjoerd

On Jun 8, 2012, at 10:37 AM, Simon Marlow wrote:

...

I'd like to add a higher-level concurrency API to the base package. This API has grown while I've been working on examples for my book, and I'm convinced that we should have something like this in the standard libraries. Here's the API I propose:

http://community.haskell.org/~simonmar/async-stm/Control-Concurrent-Async.ht...

In fact it already exists in at least two packages on Hackage: 'async' and 'threads'. My version is a superset of both of these, except for a few small differences (which are up for discussion, of course).

One good reason to have this package is that it avoids some common pitfalls, such as forgetting to handle exceptions in your child threads. In the Async API, you get to choose whether to (a) receive the result as 'Either SomeException a', or (b) have the exception re-thrown in the waiting thread. You don't get to ignore it altogether.

Another common pitfall avoided by this library is accidentally leaving threads running in the background. This is handled by withAsync:

withAsync :: IO a -> (Async a -> IO b) -> IO b

the child thread is always cancelled (i.e. killed) when the second argument returns or throws an exception, if it hasn't finished already.

I'm opening this up for discussion so that we can tidy up any inconsistent details and add any functions that people feel are missing. Naming is obviously up for discussion too.

Cheers, Simon

_______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries

-- Sjoerd Visscher https://github.com/sjoerdvisscher/blog

On 14/06/2012 16:10, Ross Paterson wrote:

On Fri, Jun 08, 2012 at 09:37:16AM +0100, Simon Marlow wrote:

...

I'm opening this up for discussion so that we can tidy up any inconsistent details and add any functions that people feel are missing. Naming is obviously up for discussion too.

I'd expect the arguments of cancelWith to be the other way round. Then you could say

cancel = cancelWith ThreadKilled

Agreed.

An alternative, equivalent interface for waitEither and friends would be

waitEither :: Async a -> Async a -> IO (Either SomeException a) waitEitherThrow :: Async a -> Async a -> IO a

This might be conceptually simpler, leaving it to the user to decide whether to mark the results with Left/Right, or something else, or not. Similarly the Async now returned by waitAny and friends could be left to the user. That would leave the two sets of functions more in line.

I liked this suggestion at first, but later concluded that it isn't as nice. The problem is that to convert an 'Async a' to an 'Async (Either a b)' has to be done when you create the Async in the first place, or else you have to write some STM code to lift the result. If you're writing some STM code anyway, then there's little point in using waitEither. So I think the waitEither and waitAny families complement each other; waitEither handles the lifting of the results into Either for you, which is sometimes handy. Cheers, Simon

On 18/06/2012 14:56, Ross Paterson wrote:

On Fri, Jun 15, 2012 at 02:26:51PM +0100, Simon Marlow wrote:

...

On 14/06/2012 16:10, Ross Paterson wrote:

...

An alternative, equivalent interface for waitEither and friends would be

waitEither :: Async a -> Async a -> IO (Either SomeException a) waitEitherThrow :: Async a -> Async a -> IO a

This might be conceptually simpler, leaving it to the user to decide whether to mark the results with Left/Right, or something else, or not. Similarly the Async now returned by waitAny and friends could be left to the user. That would leave the two sets of functions more in line.

I liked this suggestion at first, but later concluded that it isn't as nice. The problem is that to convert an 'Async a' to an 'Async (Either a b)' has to be done when you create the Async in the first place, or else you have to write some STM code to lift the result. If you're writing some STM code anyway, then there's little point in using waitEither.

The list functions have a similar issue: you have to give them the right type when you create them if you want them together in a list. Perhaps a convenient interface for the multiplexing part would be

data AsyncF a = forall r. AsyncF (Async r) (r -> a)

instance Functor AsyncF where fmap f (AsyncF a k) = AsyncF a (f . k)

waitSTMF :: AsyncF a -> STM (Either SomeException a) waitSTMF (AsyncF a k) = fmap (fmap k) (waitSTM a)

Good idea. In fact, I've added a Functor instance for Async itself - it required a bit of refactoring but didn't add much overhead. Cheers, Simon

On Tue, Jul 03, 2012 at 02:17:46PM +0100, Ross Paterson wrote:

On Tue, Jul 03, 2012 at 01:53:16PM +0100, Simon Marlow wrote:

...

Good idea. In fact, I've added a Functor instance for Async itself - it required a bit of refactoring but didn't add much overhead.

So now you can simplify the interfaces of waitEither/waitAny et al?

Ah, that won't work with the Catch versions.

On Wed, Jul 4, 2012 at 12:44 PM, Simon Marlow wrote:

Hackage has the docs now:

http://hackage.haskell.org/packages/archive/async/2.0.1.0/doc/html/Control-C...

Cheers,         Simon

Perhaps it might make sense to reduce the duplication among the asyncOn/WithUnmask/etc. variants using an ADT? e.g. type Unmask a = (forall b. IO b -> IO b) -> IO a data Fork a = ForkIO (IO a) | ForkOS (IO a) | ForkOn Int (IO a) | ForkIOWithUnmask (Unmask a) | ForkOnWithUnmask Int (Unmask a) async :: IO a -> IO (Async a) asyncUsingFork :: Fork a -> IO (Async a) withAsync :: IO a -> (Async a -> IO b) -> IO b withAsyncUsingFork :: Fork a -> (Async a -> IO b) -> IO b (Didn't put much thought into names.) -- Your ship was caught in a monadic eruption. Game over.

On 6/15/12 9:31 AM, Simon Marlow wrote:

On 14/06/2012 22:54, Lauri Alanko wrote:

...

Quoting "Simon Marlow" :

...

Naming is obviously up for discussion too.

I feel that "Async" is a bit too generic and doesn't very precisely characterize this particular construct. How about "Future", as similar things are called in e.g. Alice http://www.ps.uni-saarland.de/alice/manual/futures.html and Java http://docs.oracle.com/javase/6/docs/api/java/util/concurrent/FutureTask.htm...?

"Future" evokes notions of parallelism for me, rather than concurrency. I think the term is more often used in a parallel setting.

There's a precedent for using 'async' for concurrency: see the new C# and F# async extensions:

http://msdn.microsoft.com/en-us/library/hh191443%28v=vs.110%29.aspx

But naming is hard, and if everyone wanted to use "future" instead I wouldn't object very strongly.

I object to "future" as using an overly general term for one very particular instantiation of it. At the very least it must be qualified as "concurrent future". Though I'm not particularly opposed to "async"; and if there's precedent, then all the better. -- Live well, ~wren

On 6/14/12 5:54 PM, Lauri Alanko wrote:

Quoting "Simon Marlow" :

...

Naming is obviously up for discussion too.

I feel that "Async" is a bit too generic and doesn't very precisely characterize this particular construct. How about "Future", as similar things are called in e.g. Alice http://www.ps.uni-saarland.de/alice/manual/futures.html and Java http://docs.oracle.com/javase/6/docs/api/java/util/concurrent/FutureTask.htm...?

"Future" is a generic term for a particular way of thinking about the life cycle of variables (both in AliceML, Mozart/Oz, and in the literature). From this perspective *all* variables are futures. For example, in the AliceML url you provide, they specifically qualify the different kinds of futures: * concurrent futures --- something like Async * lazy futures --- what Haskell has by default * promised futures --- for tying the knot; akin to "static" in Java * immediate futures (not mentioned on the AliceML page) --- the standard all-at-once way of introducing, binding, and evaluating variables. Of course, the concurrent/lazy/promised qualifiers can be combined since they define orthogonal axes of temporal distinction. -- Live well, ~wren

On 15/06/12 15:38, Simon Marlow wrote:

On 15/06/2012 13:52, Twan van Laarhoven wrote:

...

On 08/06/12 10:37, Simon Marlow wrote: monad-par is for pure deterministic parallelism only, since it is based around

runPar :: Par a -> a

So I'm slightly confused by your comment that it can be used for IO as well - what do you mean exactly?

I base this on the documentation of Control.Monad.Par.Unsafe: unsafeParIO :: IO a -> p a Lift an IO operation into the Par monad. These are unsafe (in the normal, Haskell sense) when used with a runPar of type `Par a -> a`. If used with a runParIO that stays in the IO monad, then they are simply dangerous. Twan

Simon, Let me know if you want to proceed with this proposal after this initial batch of comments. If so I'll try to track the open issues more closely on the proposal's wiki page. -- Johan

On Tue, Jul 3, 2012 at 7:08 AM, Simon Marlow wrote:

I'm definitely keen to have the package in the platform. The main point against is that the package hasn't had a lot of testing in the wild; however I think the API is small and obvious enough that this isn't a problem.

If you could help with tracking the open issues that would be great!

I'm a bit swamped this week but I could try to jot down some notes next week. In the meantime: If everyone could please consider the package for inclusion into HP (i.e. try to separate nice to haves from must haves) and list things that you think must be addressed before inclusion that'd be great. -- Johan