runInteractiveCommand: program ends before writing or reading all the output
Hi all, It's the first time I use the runInteractiveCommand and I was probably bitten by laziness. When I run the following program and send its output to a file using '>' redirection I get the full output of the called process. But if I run it in the console I get only half of the output. As console is slower than disk I assume the called process terminates before all data has been read from it or the main process terminates before data has been written to stdout. I thought using waitForProcess, closing called process output and flushing stdout would solve the problem but it doesn't.
-- Compile with -threaded option module Main where
import Control.Concurrent (forkIO) import System.Environment (getArgs) import System.FilePath (dropExtension, takeFileName) import System.IO (Handle, hClose, hFlush, hGetContents, stdout) import System.Process (runInteractiveCommand, waitForProcess)
main :: IO () main = do (file:_) <- getArgs (_, out, _, pid) <- runInteractiveCommand $ "dumpbin /EXPORTS " ++ file forkIO (createDefFile file out) waitForProcess pid hClose out hFlush stdout
createDefFile :: String -> Handle -> IO () createDefFile file inp = do putStrLn $ "LIBRARY " ++ (dropExtension . takeFileName) file ++ ".dll" putStrLn "EXPORTS" text <- hGetContents inp mapM_ writeExport $ keepExports $ map words $ lines text where keepExports :: [[String]] -> [String] keepExports = map head . filter (not . null) . takeWhile (["Summary"]/=) . drop 1 . dropWhile (["ordinal","name"]/=) writeExport ('_':xs) = putStrLn xs writeExport xs = putStrLn xs
Any idea regarding the cause of this problem? Thanks, Olivier.
2008/5/15 Olivier Boudry
Hi all,
It's the first time I use the runInteractiveCommand and I was probably bitten by laziness.
When I run the following program and send its output to a file using '>' redirection I get the full output of the called process. But if I run it in the console I get only half of the output. As console is slower than disk I assume the called process terminates before all data has been read from it or the main process terminates before data has been written to stdout. I thought using waitForProcess, closing called process output and flushing stdout would solve the problem but it doesn't.
-- Compile with -threaded option module Main where
import Control.Concurrent (forkIO) import System.Environment (getArgs) import System.FilePath (dropExtension, takeFileName) import System.IO (Handle, hClose, hFlush, hGetContents, stdout) import System.Process (runInteractiveCommand, waitForProcess)
main :: IO () main = do (file:_) <- getArgs (_, out, _, pid) <- runInteractiveCommand $ "dumpbin /EXPORTS " ++ file forkIO (createDefFile file out) waitForProcess pid hClose out hFlush stdout
createDefFile :: String -> Handle -> IO () createDefFile file inp = do putStrLn $ "LIBRARY " ++ (dropExtension . takeFileName) file ++ ".dll" putStrLn "EXPORTS" text <- hGetContents inp mapM_ writeExport $ keepExports $ map words $ lines text where keepExports :: [[String]] -> [String] keepExports = map head . filter (not . null) . takeWhile (["Summary"]/=) . drop 1 . dropWhile (["ordinal","name"]/=) writeExport ('_':xs) = putStrLn xs writeExport xs = putStrLn xs
Any idea regarding the cause of this problem?
I think I've encountered the same problem several times, and it was because I was reading from the handle lazily, like this: (_, out, _, pid) <- runInteractiveProcess ... str <- hGetContents out waitForProcess pid But I didn't use 'str' until after the process finishes. My solution was to use strict IO, usually by replacing String with a strict ByteString. I hear there is now a library that lets you do strict IO with Strings.... Hope this helps.
Thanks,
Olivier.
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
It looks like a simple race condition to me. You are using "waitForProcess pid" to wait for runInteractiveCommand to finish, but you don't seem to have anything that waits for createDefFile to finish.
main :: IO () main = do (file:_) <- getArgs (_, out, _, pid) <- runInteractiveCommand $ "dumpbin /EXPORTS " ++ file forkIO (createDefFile file out) waitForProcess pid hClose out hFlush stdout
On Thu, May 15, 2008 at 11:42 AM, Ronald Guida
It looks like a simple race condition to me. You are using "waitForProcess pid" to wait for runInteractiveCommand to finish, but you don't seem to have anything that waits for createDefFile to finish.
Whoops, sorry, I didn't read the original post closely enough.
main :: IO () main = do (file:_) <- getArgs (_, out, _, pid) <- runInteractiveCommand $ "dumpbin /EXPORTS " ++ file forkIO (createDefFile file out) waitForProcess pid hClose out hFlush stdout
Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
On Thu, May 15, 2008 at 2:42 PM, Ronald Guida
It looks like a simple race condition to me. You are using "waitForProcess pid" to wait for runInteractiveCommand to finish, but you don't seem to have anything that waits for createDefFile to finish.
Thanks Ronald, As I could not find a function to wait on a ThreadId I used a MVar to synchronize both threads.
sync <- newEmptyMVar forkIO (createDefFile sync file out) waitForProcess pid takeMVar sync
and at the end of the forked thread:
putMVar sync ()
Is this normal or have I missed the `waitOnThreadId` function? Thanks for all the comments received on this thread, Olivier.
Hello Olivier, Thursday, May 15, 2008, 11:06:19 PM, you wrote:
As I could not find a function to wait on a ThreadId I used a MVar to synchronize both threads.
Is this normal or have I missed the `waitOnThreadId` function?
yes, it's common idiom -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com
On Thu, 2008-05-15 at 13:40 -0400, Olivier Boudry wrote:
Hi all,
It's the first time I use the runInteractiveCommand and I was probably bitten by laziness.
Yes. I think Philip diagnosed the problem correctly. As an example let me show you as an example how we use it in Cabal: rawSystemStdout :: Verbosity -> FilePath -> [String] -> IO String rawSystemStdout verbosity path args = do Exception.bracket (runInteractiveProcess path args Nothing Nothing) (\(inh,outh,errh,_) -> hClose inh >> hClose outh >> hClose errh) $ \(_,outh,errh,pid) -> do -- We want to process the output as text. hSetBinaryMode outh False -- fork off a thread to pull on (and discard) the stderr -- so if the process writes to stderr we do not block. -- NB. do the hGetContents synchronously, otherwise the outer -- bracket can exit before this thread has run, and hGetContents -- will fail. err <- hGetContents errh forkIO $ do evaluate (length err); return () -- wait for all the output output <- hGetContents outh evaluate (length output) -- wait for the program to terminate exitcode <- waitForProcess pid -- on failure, throw the exit code as an exception unless (exitCode == ExitSuccess) $ exitWith exitCode return (output, exitcode) So as you can see there are two subtleties. One is the issue that we have to make sure we get all the output before we wait for the program to finish. Using evaluate is the trick there. The other is that we also have to pull on the stderr of the process. Otherwise the process may be trying to output to stderr but if the pipe buffer fills up then writing to stderr will block and so it will not be able to continue writing to stdout we'll have deadlocked the process. So we have to forkIO a thread to pull on stderr. This is one reason that runInteractiveProcess is hard to use and especially hard to use portably due to the use of preemptable threads. It would be possible to do without threads if we used non-blocking IO and interleaved between reading from stdout and stderr. Duncan
On Thu, May 15, 2008 at 6:23 PM, Duncan Coutts
On Thu, 2008-05-15 at 13:40 -0400, Olivier Boudry wrote: As an example let me show you as an example how we use it in Cabal:
Hi Duncan, I tried to place a "length text `seq`" before the mapM_ writeExport to force the process output to be read but the result was even worst (only one line printed). Apparently withtout the `evaluate` function it causes more troubles than it solves. The example you provided is really helpful. I learned a lot reading it. Thanks, Olivier.
On Thu, 2008-05-15 at 21:04 -0400, Olivier Boudry wrote:
I tried to place a "length text `seq`" before the mapM_ writeExport to force the process output to be read but the result was even worst (only one line printed). Apparently withtout the `evaluate` function it causes more troubles than it solves.
Yes, you should prefer evaluate over seq (or return $!) in the IO monad because you sometimes have to distinguish between the evaluation that happens when you construct your IO actions, and the evaluation that happens when you run your IO actions. The evaluate action does the latter and its ordered with respect to the other IO actions. Duncan
participants (5)
-
Bulat Ziganshin -
Duncan Coutts -
Olivier Boudry -
Philip Weaver -
Ronald Guida