January 2005 - Haskell-Cafe

RE: [Haskell-cafe] Re: Hugsvs GHC (again)was: Re: Somerandomnewbiequestions
by Simon Marlow 20 Jan '05

20 Jan '05

On 20 January 2005 09:56, Keean Schupke wrote: > Why is disk a special case? I have never heard that all processes > under linux wait for a disk read... You were talking about Haskell threads, not processes! These are quite different things. > The kernel most certainly does not busy wait for disks > to respond, so the only alternative is that the process that needs to > wait (and only that process) is put to sleep. In which case a second > thread would be unaffected. > > Linux does not busy wait in the Kernel! (don't forget the kernel does > read-ahead, > so it could be that read really does return 'immediately' and without > any delay > apart from at the end of file - In which case asynchronous IO just > slows you down with extra context switches). The question is whether Haskell threads block when one of them is reading from the disk, and this has nothing to do with what happens in the kernel. I never said that Linux busy waits in the kernel. The reason that disk I/O is different is because select() treats it differently, and select() is what GHC's runtime currently uses to multiplex I/O. Cheers, Simon

2 1

RE: [Haskell-cafe]Re: Hugsvs GHC (again)was: Re: Somerandomnewbiequestions
by Simon Marlow 20 Jan '05

20 Jan '05

On 19 January 2005 20:31, Glynn Clements wrote: > Keean Schupke wrote: > >>>> Okay, my ignorance of Posix is showing again. Is it currently the >>>> case, then, that every GHC thread will stop running while a disk >>>> read is in progress in any thread? Is this true on all platforms? >>> >>> It's true on Unix-like systems, I believe. Even with -threaded. It >>> might not be true on Win32. >> >> I think this is not true on linux, where a thread is just a process >> created with special flags to keep the same fds and memory. >> >> As threads on linux are scheduled like processes, one thread >> blocking should not affect the others? > > That should be true of all POSIX-like thread implementations > (including Linux, whose threads aren't quite POSIX-compliant, e.g. in > regard to signal handling, but aren't that far off). > > Essentially, blocking system calls only block the calling kernel > thread. > > OTOH, if you are implementing multiple user-space threads within a > single kernel thread, if that kernel thread blocks, all of the > user-space threads within it will be blocked. We're getting a bit confused here. Keean: the original question was about whether a disk read will stop all other *Haskell* threads. Not OS threads. The two are quite different beasts in GHC. Cheers, Simon

2 1

RE: [Haskell-cafe]Re: Hugsvs GHC (again)was: Re: Somerandomnewbiequestions
by Simon Marlow 20 Jan '05

20 Jan '05

On 19 January 2005 20:33, Glynn Clements wrote: > Simon Marlow wrote: > >> We do use a thread pool. But you still need as many OS threads as >> there are blocked read() calls, unless you have a single thread >> doing select() as I described. > > How does the select() help? AFAIK, select() on a regular file or block > device will always indicate that it is readable, even if a subsequent > read() would have to read the data from disk. Again we've lost track of the discussion a bit. Yes you're right, but I was answering a different question. Cheers, Simon

1 0

Re: Why is getArgs in the IO monad?
by Ashley Yakeley 20 Jan '05

20 Jan '05

In article <41ECD49F.9020601(a)imperial.ac.uk>, Keean Schupke <k.schupke(a)imperial.ac.uk> wrote: > Surely both requirements can be satisfied if the programs arguments are made > parameters of main: > > main :: [String] -> IO () > > Keean. Better yet, it should be an implicit parameter so as not to break existing programs. main :: (?args :: [String]) => IO () You could do the same with standard input and output. -- Ashley Yakeley, Seattle WA

6 13

RE: [Haskell-cafe] Re: Hugsvs GHC (again)was: Re: Somerandomnewbiequestions
by Simon Marlow 19 Jan '05

19 Jan '05

We do use a thread pool. But you still need as many OS threads as there are blocked read() calls, unless you have a single thread doing select() as I described. BTW our Haskell Workshop paper from last year describes this stuff: http://www.haskell.org/~simonmar/papers/conc-ffi.ps.gz Cheers, Simon On 19 January 2005 15:07, Keean Schupke wrote: > Why not use a thread-pool, and a "safe" call to read, provided there > is an OS thread available, > defaulting to "unsafe" if no thread is available... You could make the > thread pool size an argument... > > Keean. > > Simon Marlow wrote: > >> On 19 January 2005 13:50, William Lee Irwin III wrote: >> >> >> >>> On 19 January 2005 09:45, Ben Rudiak-Gould wrote: >>> >>> >>>>> Okay, my ignorance of Posix is showing again. Is it currently the >>>>> case, then, that every GHC thread will stop running while a disk >>>>> read is in progress in any thread? Is this true on all platforms? >>>>> >>>>> >>> On Wed, Jan 19, 2005 at 01:39:05PM -0000, Simon Marlow wrote: >>> >>> >>>> It's true on Unix-like systems, I believe. Even with -threaded. >>>> It might not be true on Win32. >>>> >>>> >>> How does forkOS fit into this picture? It's described in the >>> documentation as allowing concurrent execution of system calls >>> and other activity by other threads. >>> >>> >> >> forkOS doesn't fix this. It forks another OS thread which can be >> used to make concurrent foreign calls, if they are not marked >> "unsafe". However, the standard I/O library, in -threaded mode, does >> read like this: >> >> - non-blocking, "unsafe", read() to see what's there >> - if read() would block, then hand off to another >> Haskell thread which does select() on all the outstanding >> IO requests. >> >> This scheme is just for efficiency. We could (and used to) just call >> "safe" read() for every read - that would give you the right >> concurrency with -threaded, but unfortunately you'd really notice >> the difference if you had 1000s of threads all doing IO, because >> each one would need its own OS thread. The current scheme is rather >> snappy (even snappier than non-threaded, as it happens). >> >> You can always do System.Posix.fileRead to get around it. >> >> Cheers, >> Simon >> _______________________________________________ >> Haskell-Cafe mailing list >> Haskell-Cafe(a)haskell.org >> http://www.haskell.org/mailman/listinfo/haskell-cafe

3 2

RE: [Haskell-cafe] Re: Hugsvs GHC (again)was: Re: Somerandomnewbiequestions
by Simon Marlow 19 Jan '05

19 Jan '05

On 19 January 2005 09:45, Ben Rudiak-Gould wrote: > Glynn Clements wrote: > > >Ben Rudiak-Gould wrote: > > > >>GHC really needs non-blocking > >>I/O to support its thread model, and memory-mapped I/O always > blocks. > > >If, by "blocks", you mean that execution will be suspended until the > >data has been read from the device into the buffer cache, then Unix > >non-blocking I/O (i.e. O_NONBLOCK) also blocks. > > Okay, my ignorance of Posix is showing again. Is it currently the > case, then, that every GHC thread will stop running while a disk read > is in progress in any thread? Is this true on all platforms? It's true on Unix-like systems, I believe. Even with -threaded. It might not be true on Win32. > (By the way, are the GHC folks aware that the description of Win32 I/O > at [2] is wrong? It seems to assume that ReadFile doesn't return until > the buffer is full.) I don't think that description corresponds with the implementation, but then I'm not terribly familiar with that part of the code. It doesn't get used when -threaded is on. Cheers, Simon

4 3

tree with labeled edges as a monad
by Andrew Pimlott 19 Jan '05

19 Jan '05

This is a "have you seen this monad?" post. I was trying to construct a search tree, and decided I wanted to do it in a monad (so I could apply StateT and keep state as I explored the space). I discovered that a tree with labeled leaves is a monad, but I wanted to label internal nodes, and such a tree (eg, Data.Tree.Tree) is not a monad (because it can only have one result type). Finally, I realized I could get a similar effect by labeling the edges and the leaves with different types: data Tree l a = Leaf a | Branch [(l, Tree l a)] instance Monad (Tree l) where return = Leaf Leaf a >>= f = f a Branch c >>= f = Branch [(l, t >>= f) | (l, t) <- c] You might use it as turn = do board <- getBoard move <- lift (Branch [(move, return move) | move <- findMoves board]) applyMove move turn I found this quite pleasing, though I hadn't run across trees as monads before. Has anyone else found this useful? Is it in a library somewhere? Andrew

3 2

RE: [Haskell-cafe] Re: Hugsvs GHC (again)was: Re: Somerandomnewbiequestions
by Simon Marlow 19 Jan '05

19 Jan '05

On 19 January 2005 16:49, Keean Schupke wrote: > But what about the continuing computation... we do not want > the fastest IO system, but we want the program to comlete the > fastest... So ideally we want 2 threads! > > One runs the Haskell code that is not waiting for IO. (IE other > Haskell threads)... The other runs a select loop as you suggest! > > This way the number of threads is fixed (2) and execution never > 'blocks' for IO. (Simon, what about this scheme?) This is what GHC does, if I understand you correctly. The thread running select() does so in its own OS thread, while another OS thread runs the Haskell code. As long as you use -threaded, that is. Oh, and before GHC 6.4 it was done a different way - the scheduler used to do the select() between running Haskell threads. Cheers, Simon

2 1

RE: [Haskell-cafe] Re: Hugsvs GHC (again)was: Re: Somerandomnewbiequestions
by Simon Marlow 19 Jan '05

19 Jan '05

On 19 January 2005 13:50, William Lee Irwin III wrote: > On 19 January 2005 09:45, Ben Rudiak-Gould wrote: >>> Okay, my ignorance of Posix is showing again. Is it currently the >>> case, then, that every GHC thread will stop running while a disk >>> read is in progress in any thread? Is this true on all platforms? > > On Wed, Jan 19, 2005 at 01:39:05PM -0000, Simon Marlow wrote: >> It's true on Unix-like systems, I believe. Even with -threaded. It >> might not be true on Win32. > > How does forkOS fit into this picture? It's described in the > documentation as allowing concurrent execution of system calls > and other activity by other threads. forkOS doesn't fix this. It forks another OS thread which can be used to make concurrent foreign calls, if they are not marked "unsafe". However, the standard I/O library, in -threaded mode, does read like this: - non-blocking, "unsafe", read() to see what's there - if read() would block, then hand off to another Haskell thread which does select() on all the outstanding IO requests. This scheme is just for efficiency. We could (and used to) just call "safe" read() for every read - that would give you the right concurrency with -threaded, but unfortunately you'd really notice the difference if you had 1000s of threads all doing IO, because each one would need its own OS thread. The current scheme is rather snappy (even snappier than non-threaded, as it happens). You can always do System.Posix.fileRead to get around it. Cheers, Simon

3 3

RE: [Haskell-cafe] GHC for .NET?
by Simon Peyton-Jones 19 Jan '05

19 Jan '05

| "The GHC compiler for .NET is currently under development at | Microsoft Research, Cambridge". | | Hmm. That location sounds familiar :-) Does anyone know if this is | actually going to happen? Or if there's any code anywhere, however | experimental, to try? It'd make a lot of sense to give GHC a .NET back end, and it's a question that comes up regularly. The reason that we haven't done it here, at GHC HQ, is because it's a more substantial undertaking than might at first appear (see below). Furthermore, it'd permanently add a complete new back-end platform for us to maintain. Given our rather limited development effort (= Simon and me), we have so far not bitten the bullet, and we have no immediate plans to do so. It'd be a good, well-defined project for someone else to tackle, and there is some good groundwork already done: * Sigbjorn Finne did a simple interop implementation that allows a Haskell program to be compiled to native code (as now) but to call .NET programs via a variant of the FFI. I don't think this work is in active use, and I'd be surprised if it worked out of the box, but it could probably be revived with modest effort * Andre Santos and his colleagues at UFPE in Brazil are working on a .NET back end, that generates CLR IL, though I don't know where they are up to. * GHC.Net would be extra attractive if there was a Visual Studio integration for GHC. Substantial progress on this has been made in 2004 by Simon Marlow, Krasimir Angelov, and Andre Santos and colleagues. There may be others that I don't know of. If anyone wants to join in this effort, do contact the above folk. And please keep us informed! Simon Here's a summary of why it's a non-trivial thing to do: - The first thing is to generate native CLR Intermediate Language (IL). That's not really hard. Requires thinking about representations for thunks and functions, and it may not be particularly efficient, but it can surely be done. An open question is about whether to generate verifiable IL or not. The trouble here is that Haskell's type system is more expressive than the CLR's in some ways, notably the use of higher-kinded type variables. So, to generate verifiable IL one is bound to need some run-time casts, and it's not clear how to minimise these. At first blush this is *all* you need do. But it isn't! - Next, you need to think about how to inter-operate with .NET libraries. You don't really want to write "foreign import..." for each and every import. You'd like GHC to read the CLR meta-data directly. But there are lots of tricky issues here; see the paper that Mark Shields and I wrote about "Object-oriented style overloading for Haskell". - Now you need to figure out how to implement GHC's primitive operations: the I/O monad arbitrary precision arithmetic concurrency exceptions finalisers stable pointers software transactional memory Not all of these are necessary, of course, but many are used in the libraries. The CLR supports many of them (e.g. concurrency) but with a very different cost model. - Last, you have to figure out what to do for the libraries. GHC has a pretty large library, and you either have to implement the primops on which the library is based (see previous point), or re-implement it. For example, GHC's implementation of I/O uses mutable state, concurrency, and more besides. For each module, you need to decide either to re-implement it using .NET primitives, or to implement the stuff the module is based on. These challenges are mostly broad rather than deep. But to get a production quality implementation that runs a substantial majority of Haskell programs "out of the box" requires a decent stab at all of them.

2 1