Re: Handling multiple fds with GHC

2015-10-07 21:17 GMT+02:00 David Turner :

On 7 October 2015 at 18:16, Markus Ongyerth wrote:

...

2015-10-07 18:30 GMT+02:00 David Turner :

...

Hi,

Why the non-threaded runtime, out of interest?

Mostly because i am used to the poll/select method I mentioned and that one works without any threading. I don't really mind using the threaded runtime though, it's more habit.

...

Threads forked with forkIO are pretty lightweight, and although things look like blocking calls from the Haskell point of view, as I understand it under the hood it's all done with events of one form or another. Thus even with the non-threaded runtime you will see forkIO-threads behaving as if they're running concurrently. In particular, you have two threads blocked trying to read from two different Handles and each will be awoken just when there's data to read, and the rest of the runtime will carry on even while they're blocked. Try it!

Yeah, I know and I tried that. As far as I can see, that's actually why things break with GHC.Event. The Event system tries to register the Fd while it was registered by me and encounters an EEXIST from epoll.

Ah, ok, so you can either do your epolling through the Haskell runtime or with your bare hands but you can't do both on a single FD.

Ah, I didn't to it with bare hands, I did it with GHC.Event registerFd. Running my own epoll might work (according to the epoll man page), but I really don't want to do that.

...

...

If you're dealing with FDs that you've acquired from elsewhere, the function unix:System.Posix.IO.ByteString.fdToHandle can be used to import them and then they work like normal Handles in terms of blocking operations etc.

Whenever I've had to deal with waking up for one of a number of reasons (not all of which are FDs) I've found the simplicity of STM is hard to beat. Something like:

atomically ((Left <$> waitForFirstThing) <|> (Right <$> waitForSecondThing))

Looks like I should look up STM. Does this scale easily? I don't really need huge amounts, but I don't have any knowledge about the number of Fds I will have.

Waiting on arbitrarily many things is pretty much as simple (as long as they all have the same type so you can put them in a list):

atomically (asum listOfWaitingThings)

Oh, I didn't see asum, but "came up" with the same implementation.

In terms of code complexity that scales just fine! I'm afraid I've no real idea what the performance characteristics of such a device would be without trying it out in your use case. Whenever I've been doing this kind of thing I've always found myself IO-bound rather than CPU-bound so I've never found myself worrying too much about the efficiency of the code itself.

If you're used to doing select/poll things yourself then it may help to think of Haskell threads blocking on Handles as basically a way to do an epoll-based event loop on the underlying FDs but with a much nicer syntax and less mucking around with explicit continuations. Similarly, if you're used to dealing with task scheduling at a low level then it may help to think of STM transactions blocking as a way to muck around with the run queues in the scheduler but with a much nicer syntax and less mucking around with explicit continuations.

For my current project the speed does not really matter, but I tend to do some research anyway, since I might get to a point where I need it. The one thing I am not sure about right now, is how to use threadWaitReadSTM. Can I reuse the STM? I have two Fds I can test it with right now, and one of them works, the other one doesn't seem to work for me. I looked into the source and to me it looks like the STM should not be reused, since the content of the TVar used internally will be set to True. Thanks for the help, ongy