I'm just tried writing a function to allow convenient embedding of shell commands, but I'm running into behavior I don't really understand somewhere at the intersection of lazy evaluation, IO, and threading. The function in question is: sh :: String -> String -> IO String sh cmd = \input -> do (stdin, stdout, _, pid) <- runInteractiveCommand cmd forkIO $ hPutStr stdin input >> hClose stdin -- get exit status to prevent zombie accumulation forkIO $ waitForProcess pid >> return () hGetContents stdout Simple uses like this work (prints "hi" and returns): return "hi" >>= sh "cat" >>= putStrLen However, this blocks: return "hi" >>= sh "cat" >>= sh "cat" >>= putStrLen this prints "hi\nblar" and returns return "hi" >>= sh "cat" >>= (\s -> putStrLn "blar" >> sh "cat" s) >>= putStrLn and this always returns but sometimes prints nothing and sometimes prints "hi\n" sh "echo hi" "there" >>= sh "cat" >>= putStrLn All these were tried with ghc 6.4.1 compiled with -threaded Just wondering if anybody has a good explanation of this range of behavior and/or a working rewrite of this function. -George
Hello George, Thursday, September 21, 2006, 7:44:22 AM, you wrote:
sh :: String -> String -> IO String sh cmd = \input -> do (stdin, stdout, _, pid) <- runInteractiveCommand cmd forkIO $ hPutStr stdin input >> hClose stdin -- get exit status to prevent zombie accumulation forkIO $ waitForProcess pid >> return () hGetContents stdout
1. when 'main' finishes, all the threads created by forkIO, silently dies. typically you need to wait until they are finished. ghc libraries don't provide any provision for it, they are too low-level. you can consider writing your own simple wrapper using MVar: myFork action = do v <- newEmptyMVar forIO $ action >>= putMVar v return v finishHim v = takeMVar v which also returns value returned by action. usage: subProcess <- myFork $ do-something-here ... finishHim subProcess or you can use some simple libraries which use more developed solutions of this problem: http://freearc.narod.ru/Process.tar.gz (my own) http://www-i2.informatik.rwth-aachen.de/~stolz/Haskell/CA.hs 2. afaiu, you don't want to wait for process finishing before calling getContents? in this case you need to collect all 'pids' returned by myFork function in some list and finish them all before main returns. or, better, develop more complex interface that will allow to kill processes whose output is no more required, in order to help 'head'-like processes also look at http://www.haskell.org/~petersen/haskell/popenhs/popenhs-1.00.0.tar.gz http://www.volker-wysk.de/hsshellscript/dist/hsshellscript-2.5.0.tar.gz and file i attached also, look at "Tackling the awkward squad: monadic input/output, concurrency, exceptions, and foreign-language calls in Haskell" http://research.microsoft.com/Users/simonpj/papers/marktoberdorf/marktoberdo... paper, which is a definitive source of information about GHC concurrency implementation -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com
On Thu, 21 Sep 2006, George Brewster wrote:
I'm just tried writing a function to allow convenient embedding of shell commands, but I'm running into behavior I don't really understand somewhere at the intersection of lazy evaluation, IO, and threading.
You may also find some unwelcome surprises in the area of pipes and buffered I/O, that aren't specifically about Haskell. Are you on a UNIX platform? I "rewrote" your function by removing both instances of forkIO, and it worked about like I expected. (The last one encounters an error "broken pipe" when it tries to write "there" to the "echo hi" shell process, because that process exits instead of reading from its input.) I can't say whether you really need forkIO, or whether it's really going to do what you need - not only do I not know enough about the thread model, neither do I know what you're really trying to do. Donn Cave, donn@drizzle.com
Donn Cave wrote:
On Thu, 21 Sep 2006, George Brewster wrote:
I'm just tried writing a function to allow convenient embedding of shell commands, but I'm running into behavior I don't really understand somewhere at the intersection of lazy evaluation, IO, and threading.
You may also find some unwelcome surprises in the area of pipes and buffered I/O, that aren't specifically about Haskell. Are you on a UNIX platform?
I "rewrote" your function by removing both instances of forkIO, and it worked about like I expected. (The last one encounters an error "broken pipe" when it tries to write "there" to the "echo hi" shell process, because that process exits instead of reading from its input.)
I can't say whether you really need forkIO, or whether it's really going to do what you need - not only do I not know enough about the thread model, neither do I know what you're really trying to do.
Donn Cave, donn@drizzle.com
Yup, I'm on linux. My goal is to have a function which gives me a lazy string which is the output of a shell command, and takes a string as input. In particular, I'd like the shell command to behave lazily (read only as much input as is needed, and write only as much output as needed), so things like this work (should terminate and print a few lines of "hi"): main = sh "yes hi" "there" >>= sh "head" >>= putStrLn I tried your suggestion out (remove forkIO), and when I do that, this example returns but doesn't print anything. Interestingly, my original implementation usually behaves this way also, but occasionally works as I would like. I had forked a seperate thread for writing the input string to the command so that the writing wouldn't block if the process filled up its write buffer and read buffer before we were done writing the string to it -- this doesn't seem to really have worked for me though. For reference, here is the funciton in question: sh :: String -> String -> IO String sh cmd = \input -> do (stdin, stdout, _, pid) <- runInteractiveCommand cmd forkIO $ hPutStr stdin input >> hClose stdin forkIO $ waitForProcess pid >> return () hGetContents stdout -George
On Thu, 21 Sep 2006, George Brewster wrote: ...
Yup, I'm on linux. My goal is to have a function which gives me a lazy string which is the output of a shell command, and takes a string as input. In particular, I'd like the shell command to behave lazily (read only as much input as is needed, and write only as much output as needed), so things like this work (should terminate and print a few lines of "hi"):
main = sh "yes hi" "there" >>= sh "head" >>= putStrLn
Good luck!
I tried your suggestion out (remove forkIO), and when I do that, this example returns but doesn't print anything.
For me, it hangs - since I left the wait in. If I omit the wait, it's "broken pipe", I think because we're trying to write data to "head" after it exits. Donn Cave, donn@drizzle.com
sh :: String -> String -> IO String sh cmd = \input -> do (stdin, stdout, _, pid) <- runInteractiveCommand cmd forkIO $ hPutStr stdin input >> hClose stdin forkIO $ waitForProcess pid >> return () hGetContents stdout
participants (3)
-
Bulat Ziganshin -
Donn Cave -
George Brewster