Bug in IO libraries when sending data through a pipe?
Hello There seems to be a bug in the IO libraries. I'm using the following procedure to call an external program and send it data through a pipe. pipeto :: String -> String -> [String] -> IO () pipeto txt prog par = do catch (do -- create pipe (zu, von) <- createPipe vonh <- fdToHandle von hSetBuffering vonh NoBuffering mpid <- forkProcess case mpid of Nothing -> do -- child -- connect pipe's read end to stdin -- and close its write end dupTo zu (intToFd 0) fdClose zu hClose vonh executeFile prog True par Nothing ... -- (print error message) Just pid -> do -- parent fdClose zu -- close pipe's read end -- ** here ** hPutStr vonh txt -- write text to forked process hClose vonh -- close pipe's write end -- wait for child process to finish (Just ps) <- getProcessStatus True True pid if ps == Exited ExitSuccess then return () else ...) -- (error message) (\err -> ...) -- print error message The problem is that the child process doesn't receive all the data which the parent sends. It's as if "hPutStr vonh txt" sends the data lazily somehow, and "hClose vonh" closes the pipe prematurely. It varies from run to run exactly which data gets through. If I cause the child process to read all its input immediately, the problem doesn't seem to occur. Normally, it does so gradually, which takes a few seconds. I'm using GHC 5.02.2 Volker
Volker Wysk <post@volker-wysk.de> writes:
(zu, von) <- createPipe vonh <- fdToHandle von hSetBuffering vonh NoBuffering mpid <- forkProcess case mpid of Nothing -> do -- child -- connect pipe's read end to stdin -- and close its write end dupTo zu (intToFd 0) fdClose zu hClose vonh executeFile prog True par Nothing ... -- (print error message) Just pid -> do -- parent fdClose zu -- close pipe's read end -- ** here ** hPutStr vonh txt -- write text to forked process hClose vonh -- close pipe's write end -- wait for child process to finish (Just ps) <- getProcessStatus True True pid if ps == Exited ExitSuccess then return () else ...) -- (error message)
The problem is that the child process doesn't receive all the data which the parent sends. It's as if "hPutStr vonh txt" sends the data lazily somehow, and "hClose vonh" closes the pipe prematurely.
It varies from run to run exactly which data gets through. If I cause the child process to read all its input immediately, the problem doesn't seem to occur. Normally, it does so gradually, which takes a few seconds.
I'm using GHC 5.02.2
Quite possibly could be a bug. Lazy IO is rather subtle I think, specially when done across pipes. I faced some similar problem with in POpen recently. You can see how I solved it (worked round it?) by comparing the latest release 1.00 with the previous one 0.00.1: http://www.01.246.ne.jp/~juhp/haskell/popenhs/ In comparison Posix.runProcess allows attaching file handles to the in, out and error pipes, which can be written to and read from eagerly I suppose. Jens
Jens Petersen <petersen@redhat.com> writes:
The problem is that the child process doesn't receive all the data which the parent sends. It's as if "hPutStr vonh txt" sends the data lazily somehow, and "hClose vonh" closes the pipe prematurely.
It varies from run to run exactly which data gets through. If I cause the child process to read all its input immediately, the problem doesn't seem to occur. Normally, it does so gradually, which takes a few seconds.
I'm using GHC 5.02.2
Quite possibly could be a bug. Lazy IO is rather subtle I think, specially when done across pipes. I faced some similar problem with in POpen recently. You can see how I solved it (worked round it?) by comparing the latest release 1.00 with the previous one 0.00.1:
http://www.01.246.ne.jp/~juhp/haskell/popenhs/
In comparison Posix.runProcess allows attaching file handles to the in, out and error pipes, which can be written to and read from eagerly I suppose.
Also I just rediscovered Manuel Chakravarty's HPL (Haskell Ports Library), which provides a rather elegant, sophisticated approach. http://www.cse.unsw.edu.au/~chak/haskell/ports/ It compiles fine under ghc-5.02.2, and using the BufferMode patch included at the end output seems to be ok, but input of more than 2048 bytes doesn't seem to be being handled reliably. Eg with the test program below: % test-processes ping localhost works (with the aforementioned patch below to Processes.hs), but % cat 4096 | test-processes cat [4096 is a file of 4096 chars] mostly gives no output, but occasionally I see Warning: Ports.listenToPort: Attempted to listen to a closed port! Needs some debugging I guess. :) Jens -- ghc -o test-processes `ports-config --cflags --libs` test-processes.hs module Main where import Processes import Ports import IO (openFile, hGetContents, IOMode(..), hSetBuffering, BufferMode(..)) import Monad (unless) main :: IO() main = do inpt <- getContents withPorts [] $ \ (cmd:args) -> do outpt <- newPort ' ' errpt <- newPort ' ' let p = proc cmd args p inpt outpt errpt putStrLn "output:" out <- listenToPort outpt mapM_ putStrLn $ lines out putStrLn "error:" errclosed <- isClosedPort errpt unless errclosed $ do err <- listenToPort errpt putStr err putStrLn "test finished" Index: Processes.hs =================================================================== RCS file: /home/chakcvs/cvs/ports/lib/Processes.hs,v retrieving revision 1.7 diff -u -r1.7 Processes.hs --- Processes.hs 2001/07/04 16:15:52 1.7 +++ Processes.hs 2002/03/20 05:28:33 @@ -145,6 +145,8 @@ stdoutWriteHandle <- fdToHandle stdoutWriteFD stderrReadHandle <- fdToHandle stderrReadFD stderrWriteHandle <- fdToHandle stderrWriteFD + hSetBuffering stdoutReadHandle LineBuffering + hSetBuffering stderrReadHandle LineBuffering -- -- the child must close the pipe ends that it doesn't use (especially, the -- write end)
On Mit, 2002-03-20 at 07:00, Jens Petersen wrote:
Jens Petersen <petersen@redhat.com> writes:
The problem is that the child process doesn't receive all the data which the parent sends. It's as if "hPutStr vonh txt" sends the data lazily somehow, and "hClose vonh" closes the pipe prematurely.
It varies from run to run exactly which data gets through. If I cause the child process to read all its input immediately, the problem doesn't seem to occur. Normally, it does so gradually, which takes a few seconds.
I'm using GHC 5.02.2
Quite possibly could be a bug. Lazy IO is rather subtle I think, specially when done across pipes. I faced some similar problem with in POpen recently. You can see how I solved it (worked round it?) by comparing the latest release 1.00 with the previous one 0.00.1:
POpen-1.0.0 contains the same bug which I made. It doesn't ensure that the values which are needed after the call of forkProcess, before that of executeFile, are fully evaluated. So, if they are read lazily from a stream, the newly spawned child process reads data from a stream which it shares with its parent, making it disappear from the parent's input. In this situation, this sure isn't intended. Inserting the following lines just before the line "pid <- forkProcess", in POpen.hs, would force the corresponding values to be evaluated, so no data will be lost. seq (length path) $ seq (sum (map length args)) $ return () when (isJust env) $ seq (sum (map (\(a,b) -> length a + length b) (fromJust env))) $ return () when (isJust dir) $ seq (length (fromJust dir)) $ return () I'm also not sure what this part is supposed to do: inr <- if (isJust inpt) then do (inr', inw) <- createPipe hin <- fdToHandle inw hPutStr hin $ fromJust inpt hClose hin return $ Just inr' else return Nothing Doesn't it write the input data to a pipe which no process reads from..?? Volker
Hi Volker, Volker Wysk <post@volker-wysk.de> writes:
On Mit, 2002-03-20 at 07:00, Jens Petersen wrote:
Jens Petersen <petersen@redhat.com> writes:
The problem is that the child process doesn't receive all the data which the parent sends. It's as if "hPutStr vonh txt" sends the data lazily somehow, and "hClose vonh" closes the pipe prematurely.
It varies from run to run exactly which data gets through. If I cause the child process to read all its input immediately, the problem doesn't seem to occur. Normally, it does so gradually, which takes a few seconds.
I'm using GHC 5.02.2
Quite possibly could be a bug. Lazy IO is rather subtle I think, specially when done across pipes. I faced some similar problem with in POpen recently. You can see how I solved it (worked round it?) by comparing the latest release 1.00 with the previous one 0.00.1:
POpen-1.0.0 contains the same bug which I made. It doesn't ensure that the values which are needed after the call of forkProcess, before that of executeFile, are fully evaluated. So, if they are read lazily from a stream, the newly spawned child process reads data from a stream which it shares with its parent, making it disappear from the parent's input. In this situation, this sure isn't intended.
Perhaps you could give an explicit example?
Inserting the following lines just before the line "pid <- forkProcess", in POpen.hs, would force the corresponding values to be evaluated, so no data will be lost.
seq (length path) $ seq (sum (map length args)) $ return () when (isJust env) $ seq (sum (map (\(a,b) -> length a + length b) (fromJust env))) $ return () when (isJust dir) $ seq (length (fromJust dir)) $ return ()
Hmmm, I don't really see why this is necessary. Don't the lazy values of "path", "env" and "dir" just get evaluated when they're needed here as normal? (If what you say is true though it would be simpler just to use "$!" or "!"s for strict evaluation I guess.) I would be more worried about the input stream string not being complete when the input handle is closed.
I'm also not sure what this part is supposed to do:
inr <- if (isJust inpt) then do (inr', inw) <- createPipe hin <- fdToHandle inw hPutStr hin $ fromJust inpt hClose hin return $ Just inr' else return Nothing
It returns the output end of a pipe containing the input string if one is given.
Doesn't it write the input data to a pipe which no process reads from..??
Nope, "doTheBusiness" dup2's it to the stdin of the subprocess: (outr, outw) <- createPipe (errr, errw) <- createPipe pid <- forkProcess case pid of Nothing -> doTheBusiness inr outw errw -- *** Just p -> do -- close other end of pipes in here when (isJust inr) $ fdClose $ fromJust inr fdClose outw fdClose errw hout <- fdToHandle outr outstrm <- hGetContents hout herr <- fdToHandle errr errstrm <- hGetContents herr return (outstrm, errstrm , p) where doTheBusiness :: Maybe Fd -- stdin -> Fd -- stdout -> Fd -- stderr -> IO (String, String, ProcessID) -- (stdout, stderr) doTheBusiness inr outw errw = do maybeChangeWorkingDirectory dir when (isJust inr) $ dupTo (fromJust inr) stdInput -- *** dupTo outw stdOutput dupTo errw stdError executeFile path True args env -- for typing, should never actually run error "executeFile failed!" Jens
Hi On 21 Mar 2002, Jens Petersen wrote:
Volker Wysk <post@volker-wysk.de> writes:
POpen-1.0.0 contains the same bug which I made. It doesn't ensure that the values which are needed after the call of forkProcess, before that of executeFile, are fully evaluated. So, if they are read lazily from a stream, the newly spawned child process reads data from a stream which it shares with its parent, making it disappear from the parent's input. In this situation, this sure isn't intended.
Perhaps you could give an explicit example?
I haven't tried it, but it's exactly the same thing.
Inserting the following lines just before the line "pid <- forkProcess", in POpen.hs, would force the corresponding values to be evaluated, so no data will be lost.
seq (length path) $ seq (sum (map length args)) $ return () when (isJust env) $ seq (sum (map (\(a,b) -> length a + length b) (fromJust env))) $ return () when (isJust dir) $ seq (length (fromJust dir)) $ return ()
Hmmm, I don't really see why this is necessary. Don't the lazy values of "path", "env" and "dir" just get evaluated when they're needed here as normal? (If what you say is true though it would be simpler just to use "$!" or "!"s for strict evaluation I guess.)
Yes, and that's *after* forkProcess. So when they are computed from the lazily read contents of a stream, the newly spawned child will read data from a stream which it shares with its parent.
I'm also not sure what this part is supposed to do:
inr <- if (isJust inpt) then do (inr', inw) <- createPipe hin <- fdToHandle inw hPutStr hin $ fromJust inpt hClose hin return $ Just inr' else return Nothing
It returns the output end of a pipe containing the input string if one is given.
Doesn't it write the input data to a pipe which no process reads from..??
Nope, "doTheBusiness" dup2's it to the stdin of the subprocess: [...]
But hPutStr, followed by hClose, won't complete until all the input string has been written, while no process is listening. Volker
Volker Wysk <post@volker-wysk.de> writes:
On 21 Mar 2002, Jens Petersen wrote:
Volker Wysk <post@volker-wysk.de> writes:
POpen-1.0.0 contains the same bug which I made. It doesn't ensure that the values which are needed after the call of forkProcess, before that of executeFile, are fully evaluated. So, if they are read lazily from a stream, the newly spawned child process reads data from a stream which it shares with its parent, making it disappear from the parent's input. In this situation, this sure isn't intended.
Ok, I agree this is a potential if unlikely problem. I can't really think of any useful examples though. I guess using "$!"s in the call to popen would solve this part.
Perhaps you could give an explicit example?
I haven't tried it, but it's exactly the same thing.
Well, an explicit example using your "pipeto" or popen would be helpful.
Inserting the following lines just before the line "pid <- forkProcess", in POpen.hs, would force the corresponding values to be evaluated, so no data will be lost.
seq (length path) $ seq (sum (map length args)) $ return () when (isJust env) $ seq (sum (map (\(a,b) -> length a + length b) (fromJust env))) $ return () when (isJust dir) $ seq (length (fromJust dir)) $ return ()
I would prefer not to add strict evaluation to POpen unless it's absolutely necessary. I guess I really need a testcase for this problem. If you have one please send it to me. I should really add some unit tests to popenhs.
Hmmm, I don't really see why this is necessary. Don't the lazy values of "path", "env" and "dir" just get evaluated when they're needed here as normal? (If what you say is true though it would be simpler just to use "$!" or "!"s for strict evaluation I guess.)
Yes, and that's *after* forkProcess. So when they are computed from the lazily read contents of a stream, the newly spawned child will read data from a stream which it shares with its parent.
Btw I guess one can say that popen inherits this problem from "Posix.runProcess". But usually "path", "env" and "dir" are not streams, just strings, right? Even for args I feel pushed to think of a real example where it could be a problem. Something like "xargs" taking a long stream of arguments from stdin, but arguments instead?? (Most shells have restrictions on the size of argv I think though.) Are you're referring to these comments from Posix.lhs, or you've rediscovered them? -- ***NOTE***: make sure you completely force the evaluation of the path -- and arguments to the child before calling runProcess. If you don't do -- this *and* the arguments from runProcess are read in from a file lazily, -- be prepared for some rather weird parent-child file I/O behaviour. -- -- [If you don't force the args, consider the case where the -- arguments emanate from a file that is read lazily, using -- hGetContents or some such. Since a child of a fork() -- inherits the opened files of the parent, the child can -- force the evaluation of the arguments and read them off the -- file without any problems. The problem is that while the -- child share a file table with the parent, it has separate -- buffers, so a child may fill up its (copy of) the buffer, -- but only read it partially. When the *parent* tries to read -- from the shared file again, the (shared) file offset will -- have been stepped on by whatever number of chars that was -- copied into the file buffer of the child. i.e., the unused -- parts of the buffer will *not* be seen, resulting in -- random/unpredicatable results. -- -- Based on a true (, debugged :-) story. -- ] Reading this again I start to understand the problem you're describing a little better. Perhaps something does need to be done about it, as you say. ;-)
But hPutStr, followed by hClose, won't complete until all the input string has been written, while no process is listening.
Oops, you're right! Indeed popen hangs on input greater than 4kB on my system. Thank you very much for reporting this. Jens
participants (3)
-
Jens Petersen -
Jens Petersen -
Volker Wysk