On 6/9/12 5:28 PM, Simon Marlow wrote:
All good points. I don't have many use cases for the waitAny* family yet so I'd been holding off refining the API until I had a clearer picture of what would be useful for clients. There are lots of possibilities here.
The way I typically use it is as follows: The main thread spawns off a bunch of heterogeneous workers--- the kind that interact with one another in an event-loop like manner (not of the fire-and-forget nor the web-server kind). But, the workers may fail/terminate for various reasons--- including thrown exceptions, but also more benign things like expected failure modes (encoded in normal ADTs or passed along via TChans/TVars/etc), and temporary failures that just require some 'manual' help to get unstuck. After firing them off to do the work, the main thread's job is to monitor the workers in order to (a) handle any of the failure modes it can, (b) ensure clean and safe shutdown in the event of irrecoverable errors. A related but somewhat different style of using waitAny* parallelism is for things like grid managers. Again, there are a bunch of heterogeneous worker threads. But this time, rather than focusing on errorful termination as above, the focus here is on proper termination. In particular, once some job A has finished, we can now start jobs B, C, and D which depend on the output of A. In the small scale, you can just have A finish by firing off the other jobs; but in the large scale that doesn't work. You need to (a) be able to specify the job dependencies independently of the work that job is doing, (b) be able to handle failures and restarts as above, (c) perform load balancing[1], and so on. [1] For a proper grid manager this is "real" load balancing by deciding which machines a particular job should be run on. But even within a single OS process, we can know that certain tasks require more or less memory or other resources, and so can alter how aggressively to fire off jobs in order to keep from overloading the machine. It's this extreme heterogeneity of jobs which is a core part of being a "grid manager" IMO; when the jobs are relatively homogeneous (even if doing different things) then a lot of these complications go away. -- Live well, ~wren