runhaskell a parallel program
Hi, I've got a program which I'd like to run on multiple threads. If I compile it with ghc --make -threaded, then run with +RTS -N2 it runs on 2 cores very nicely. If however I run it with runhaskell Test.hs +RTS -N2 I get told the -N2 flag isn't supported. Is there a way to runhaskell a program on multiple cores? Is this a bug that it doesn't work, a feature request I'm making, or is there some trick to getting it working I haven't thought of? I'll raise a bug report if that turns out to be the right thing. Thanks Neil
On 06/05/2009 17:19, Neil Mitchell wrote:
I've got a program which I'd like to run on multiple threads. If I compile it with ghc --make -threaded, then run with +RTS -N2 it runs on 2 cores very nicely.
If however I run it with runhaskell Test.hs +RTS -N2 I get told the -N2 flag isn't supported. Is there a way to runhaskell a program on multiple cores? Is this a bug that it doesn't work, a feature request I'm making, or is there some trick to getting it working I haven't thought of? I'll raise a bug report if that turns out to be the right thing.
runhaskell is picking up the +RTS options instead of giving them to GHC itself. Unfortunately there isn't a good way to pass these options to GHC, because it is the GHC runtime interpreting them rather than runhaskell itself. Trying the GHCRTS environment variable doesn't work, because runhaskell isn't compiled with -threaded so it rejects -N2. As a workaround you could use 'ghc -e main foo.hs +RTS -N2'. What's interesting to me is whether the byte-code interpreter will work right with +RTS -N2. It's not disabled, and in theory there's no good reason why it shouldn't work, but it hasn't been tested. Still, parallelism is about performance, and if you want performance you should start by compiling your program. Cheers, Simon
Hi
If however I run it with runhaskell Test.hs +RTS -N2 I get told the -N2 flag isn't supported. Is there a way to runhaskell a program on multiple cores? Is this a bug that it doesn't work, a feature request I'm making, or is there some trick to getting it working I haven't thought of? I'll raise a bug report if that turns out to be the right thing.
As a workaround you could use 'ghc -e main foo.hs +RTS -N2'.
That works great :-) Perhaps this trick should be documented? I thought runhaskell was just sugar over ghc -e, so couldn't it share the same mechanism?
What's interesting to me is whether the byte-code interpreter will work right with +RTS -N2
Isn't ghc -e using the byte-code interpreter?
Still, parallelism is about performance, and if you want performance you should start by compiling your program.
This is a test framework that spawns system commands. My guess is the Haskell accounts for a few milliseconds of execution per hour. Running two system commands in parallel gives a massive boost. A related question I wanted to ask. Is there any way to have my Haskell program support -j3, which is equivalent to +RTS -N3 -RTS. At the moment I've set this up with a shell script to translate the -j3, but a nicer method would be preferable. Even something as sledgehammer like as restartWithNProcessors :: Int -> IO (), which aborted the program entirely and restarted main with a completely fresh heap but a given number of processors. Thanks Neil
Hello Neil, Thursday, May 7, 2009, 2:27:34 PM, you wrote:
This is a test framework that spawns system commands. My guess is the Haskell accounts for a few milliseconds of execution per hour. Running two system commands in parallel gives a massive boost.
A related question I wanted to ask. Is there any way to have my Haskell program support -j3, which is equivalent to +RTS -N3 -RTS. At
my own program creates a lot of parallel threads without using -N the secret is using of forkOS plus C code in threads. since you spend time in system calls this should also work for you -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com
On 07/05/2009 11:37, Bulat Ziganshin wrote:
Hello Neil,
Thursday, May 7, 2009, 2:27:34 PM, you wrote:
This is a test framework that spawns system commands. My guess is the Haskell accounts for a few milliseconds of execution per hour. Running two system commands in parallel gives a massive boost.
A related question I wanted to ask. Is there any way to have my Haskell program support -j3, which is equivalent to +RTS -N3 -RTS. At
my own program creates a lot of parallel threads without using -N
the secret is using of forkOS plus C code in threads. since you spend time in system calls this should also work for you
Are you sure you need forkOS, and not just forkIO? Why? Cheers, Simon
Hello Simon, Thursday, May 7, 2009, 5:27:04 PM, you wrote:
my own program creates a lot of parallel threads without using -N
the secret is using of forkOS plus C code in threads. since you spend time in system calls this should also work for you
Are you sure you need forkOS, and not just forkIO? Why?
well, 4 years ago or so i discovered that my program can perform compression (C) threads simultaneously only if i use forkOS. i don't tried to jump to forkIO later ... just recompiled program with forkIO - it still works. at least on this quad-core cpu. so it was just my laziness, sorry for all those 4-year noise :( -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com
On 07/05/2009 11:27, Neil Mitchell wrote:
Hi
If however I run it with runhaskell Test.hs +RTS -N2 I get told the -N2 flag isn't supported. Is there a way to runhaskell a program on multiple cores? Is this a bug that it doesn't work, a feature request I'm making, or is there some trick to getting it working I haven't thought of? I'll raise a bug report if that turns out to be the right thing. As a workaround you could use 'ghc -e main foo.hs +RTS -N2'.
That works great :-) Perhaps this trick should be documented? I thought runhaskell was just sugar over ghc -e, so couldn't it share the same mechanism?
What's interesting to me is whether the byte-code interpreter will work right with +RTS -N2
Isn't ghc -e using the byte-code interpreter?
Yes; apparently it "works", though we still haven't stress-tested it running real parallel programs using GHCi with +RTS -N2.
Still, parallelism is about performance, and if you want performance you should start by compiling your program.
This is a test framework that spawns system commands. My guess is the Haskell accounts for a few milliseconds of execution per hour. Running two system commands in parallel gives a massive boost.
That still doesn't explain why you need +RTS -N2. You can spawn multiple processes by making multiple calls to runProcess or whatever. If you want to wait for multiple processes simultaneously, compile with -threaded and use forkIO.
A related question I wanted to ask. Is there any way to have my Haskell program support -j3, which is equivalent to +RTS -N3 -RTS. At the moment I've set this up with a shell script to translate the -j3, but a nicer method would be preferable. Even something as sledgehammer like as restartWithNProcessors :: Int -> IO (), which aborted the program entirely and restarted main with a completely fresh heap but a given number of processors.
We don't have anything like that right now. Personally I think the shell script method is quite reasonable. Cheers, Simon
Hello Simon, Thursday, May 7, 2009, 5:24:54 PM, you wrote:
A related question I wanted to ask. Is there any way to have my Haskell program support -j3, which is equivalent to +RTS -N3 -RTS. At the moment I've set this up with a shell script to translate the -j3, but a nicer method would be preferable. Even something as sledgehammer like as restartWithNProcessors :: Int -> IO (), which aborted the program entirely and restarted main with a completely fresh heap but a given number of processors.
Neil, you can implement it by yourself - convert -j3 in cmdline to +RTS -N3 -RTS and run program itself. alternatively, you can use defaultsHook() although i'm not sure that it can change number of Capabilities -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com
Hi Bulat,
Neil, you can implement it by yourself - convert -j3 in cmdline to +RTS -N3 -RTS and run program itself. alternatively, you can use defaultsHook() although i'm not sure that it can change number of Capabilities
Can I run a program itself? getProgName doesn't give me enough to invoke myself (unfortunately) I don't want to write any C - that's just more hassle than a shell script wrapping the command. Thanks Neil
Hi
Isn't ghc -e using the byte-code interpreter?
Yes; apparently it "works", though we still haven't stress-tested it running real parallel programs using GHCi with +RTS -N2.
It seemed perfectly stable when I tried, on a few examples I had knocking around.
Still, parallelism is about performance, and if you want performance you should start by compiling your program.
This is a test framework that spawns system commands. My guess is the Haskell accounts for a few milliseconds of execution per hour. Running two system commands in parallel gives a massive boost.
That still doesn't explain why you need +RTS -N2. You can spawn multiple processes by making multiple calls to runProcess or whatever. If you want to wait for multiple processes simultaneously, compile with -threaded and use forkIO.
I do need to wait for them - so I don't end up firing too many at once. I was hoping to avoid the compile, which the ghc -e will give me.
A related question I wanted to ask. Is there any way to have my Haskell program support -j3, which is equivalent to +RTS -N3 -RTS. At the moment I've set this up with a shell script to translate the -j3, but a nicer method would be preferable. Even something as sledgehammer like as restartWithNProcessors :: Int -> IO (), which aborted the program entirely and restarted main with a completely fresh heap but a given number of processors.
We don't have anything like that right now. Personally I think the shell script method is quite reasonable.
My shell script also does -g1, so for the moment it's not too bad. But for example I added parallel execution to HLint over the weekend - I don't want to add a shell script to invoke it, and I'm not entirely comfortable with documenting +RTS -N2 -RTS as the way to -j3 it. If a nicer way ever turned out to be relatively pain free to implement, it would be nice. I'll raise a feature request bug. Thanks Neil
On Thu, 2009-05-07 at 15:12 +0100, Neil Mitchell wrote:
This is a test framework that spawns system commands. My guess is the Haskell accounts for a few milliseconds of execution per hour. Running two system commands in parallel gives a massive boost.
That still doesn't explain why you need +RTS -N2. You can spawn multiple processes by making multiple calls to runProcess or whatever. If you want to wait for multiple processes simultaneously, compile with -threaded and use forkIO.
I do need to wait for them - so I don't end up firing too many at once. I was hoping to avoid the compile, which the ghc -e will give me.
Right, it works because ghc itself is compiled with the threaded rts. So you don't need the +RTS -N when you call ghc -e. Note that for the next ghc release the process library will use a different implementation of waitForProcess (at least on Unix) so will not need multiple OS threads to wait for multiple processes simultaneously. Duncan
On 07/05/2009 23:58, Duncan Coutts wrote:
Note that for the next ghc release the process library will use a different implementation of waitForProcess (at least on Unix) so will not need multiple OS threads to wait for multiple processes simultaneously.
It will if I ever get it finished... Cheers, Simon
On May 7, 2009, at 06:27 , Neil Mitchell wrote:
If however I run it with runhaskell Test.hs +RTS -N2 I get told the -N2 flag isn't supported. Is there a way to runhaskell a program on
As a workaround you could use 'ghc -e main foo.hs +RTS -N2'.
That works great :-) Perhaps this trick should be documented? I thought runhaskell was just sugar over ghc -e, so couldn't it share the same mechanism?
I think the problem is that runhaskell invokes the runghc executable in the GHC libdir, which is itself a non-threaded Haskell program and therefore consumes the RTS options. -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH
On 07/05/2009 18:12, Brandon S. Allbery KF8NH wrote:
On May 7, 2009, at 06:27 , Neil Mitchell wrote:
If however I run it with runhaskell Test.hs +RTS -N2 I get told the -N2 flag isn't supported. Is there a way to runhaskell a program on
As a workaround you could use 'ghc -e main foo.hs +RTS -N2'.
That works great :-) Perhaps this trick should be documented? I thought runhaskell was just sugar over ghc -e, so couldn't it share the same mechanism?
I think the problem is that runhaskell invokes the runghc executable in the GHC libdir, which is itself a non-threaded Haskell program and therefore consumes the RTS options.
Exactly. Simon
participants (5)
-
Brandon S. Allbery KF8NH -
Bulat Ziganshin -
Duncan Coutts -
Neil Mitchell -
Simon Marlow