Reading and writing to handles made with System.Process far too slow
Hi, I'm have some business in piping some data and reading some data back out of a socket so I thought that I'd just use the ‘socat’ tool. I went off to System.Process just to find out that reading and writing are taking far too long. I put together a small example which only requires that you have ‘cat’ on your system: {-# LANGUAGE UnicodeSyntax #-} module Uzbl.WithSource where import GHC.IO.Handle ( hPutStr, hGetContents, hSetBuffering , BufferMode(..)) import System.Process ( createProcess, proc , StdStream(CreatePipe), std_out, std_in) gs ∷ IO String gs = do let sp = (proc "cat" []) { std_out = CreatePipe, std_in = CreatePipe } (Just hin, Just hout, _, _) ← createProcess sp -- hSetBuffering hin NoBuffering -- hSetBuffering hout NoBuffering hPutStr hin "Test data" hGetContents hout All this should effectively do is to give you back "Test data". While it *does* do that, it takes far too long. When I run ‘gs’, it will start to (lazily) print the result, printing nothing but opening ‘"’ and then after about 2-3 seconds printing the rest and finishing. If we set buffering on the in-handle (hin) to NoBuffering, we get a slightly different behaviour: pretty much straight away we'll have ‘"Test data’ but then it will wait for the same amount of time to conclude that it's the end of the response. Changing buffering mode on ‘hout’ seems to make no difference. Setting precise number in a BlockBuffering seems to be no improvement and in the actual application I will not know how long the data I'm piping in and out will be. GHC 7.8.2, process-1.2.2.0; I'm running ‘gs’ in GHCi. It seems that if I change the module name to Main, make ‘main = gs >>= putStrLn’, compile the file and run it, it just hangs there! If I add a newline at the end, it will print but the program will not finish. This makes me think that perhaps I should be closing handles somewhere (but if I try inside the function, I get no output, thanks lazy I/O). What I would expect this program to do is to produce same result as ‘print "Test data" | cat’. -- Mateusz K.
On 05/12/2014 04:56 PM, Mateusz Kowalczyk wrote:
Hi,
I'm have some business in piping some data and reading some data back out of a socket so I thought that I'd just use the ‘socat’ tool. I went off to System.Process just to find out that reading and writing are taking far too long.
I put together a small example which only requires that you have ‘cat’ on your system:
{-# LANGUAGE UnicodeSyntax #-} module Uzbl.WithSource where
import GHC.IO.Handle ( hPutStr, hGetContents, hSetBuffering , BufferMode(..)) import System.Process ( createProcess, proc , StdStream(CreatePipe), std_out, std_in)
gs ∷ IO String gs = do let sp = (proc "cat" []) { std_out = CreatePipe, std_in = CreatePipe } (Just hin, Just hout, _, _) ← createProcess sp -- hSetBuffering hin NoBuffering -- hSetBuffering hout NoBuffering hPutStr hin "Test data" hGetContents hout
All this should effectively do is to give you back "Test data". While it *does* do that, it takes far too long. When I run ‘gs’, it will start to (lazily) print the result, printing nothing but opening ‘"’ and then after about 2-3 seconds printing the rest and finishing.
If we set buffering on the in-handle (hin) to NoBuffering, we get a slightly different behaviour: pretty much straight away we'll have ‘"Test data’ but then it will wait for the same amount of time to conclude that it's the end of the response. Changing buffering mode on ‘hout’ seems to make no difference. Setting precise number in a BlockBuffering seems to be no improvement and in the actual application I will not know how long the data I'm piping in and out will be.
GHC 7.8.2, process-1.2.2.0; I'm running ‘gs’ in GHCi. It seems that if I change the module name to Main, make ‘main = gs >>= putStrLn’, compile the file and run it, it just hangs there! If I add a newline at the end, it will print but the program will not finish. This makes me think that perhaps I should be closing handles somewhere (but if I try inside the function, I get no output, thanks lazy I/O).
What I would expect this program to do is to produce same result as ‘print "Test data" | cat’.
As it often happens, I solved it straight away after posting to the list. Here's the program that behaves how I wanted it to from the start: gs ∷ IO String gs = do let sp = (proc "cat" []) { std_out = CreatePipe, std_in = CreatePipe } (Just hin, Just hout, _, _) ← createProcess sp hPutStr hin "Test data" hClose hin c ← hGetContents hout length c `seq` hClose hout return c -- Mateusz K.
Bear in mind that this program will also hang if you write enough data to
it. There is an implicit buffer when piping data between processes. When it
gets filled the process trying to write to it will simply wait forever. The
"cat" you are spawning will wait too because you don't actually read any
data. The solution is to perform writing and reading in concurrent fashion.
Just try this program:
module Main where
import System.Process
import System.IO
import Control.Concurrent
r n = replicate n '.'
gs :: Int -> IO String
gs n = do
print n
let sp = (proc "cat" [])
{ std_out = CreatePipe, std_in = CreatePipe }
(Just hin, Just hout, _, _) <- createProcess sp
let cb = do
hPutStr hin (r n)
hClose hin
-- forkIO cb
cb
c <- hGetContents hout
length c `seq` hClose hout
return c
main = do
print "welcome"
mapM_ gs [ 2 ^ x | x <- [0..20]]
print "goodbye"
Without forkIO it hangs on my system with n = 2^18. If you replace "cb"
with "forkIO cb" it will finish without hanging.
Best regards,
Krzysztof Skrzętnicki
On Mon, May 12, 2014 at 5:00 PM, Mateusz Kowalczyk
On 05/12/2014 04:56 PM, Mateusz Kowalczyk wrote:
Hi,
I'm have some business in piping some data and reading some data back out of a socket so I thought that I'd just use the ‘socat’ tool. I went off to System.Process just to find out that reading and writing are taking far too long.
I put together a small example which only requires that you have ‘cat’ on your system:
{-# LANGUAGE UnicodeSyntax #-} module Uzbl.WithSource where
import GHC.IO.Handle ( hPutStr, hGetContents, hSetBuffering , BufferMode(..)) import System.Process ( createProcess, proc , StdStream(CreatePipe), std_out, std_in)
gs ∷ IO String gs = do let sp = (proc "cat" []) { std_out = CreatePipe, std_in = CreatePipe } (Just hin, Just hout, _, _) ← createProcess sp -- hSetBuffering hin NoBuffering -- hSetBuffering hout NoBuffering hPutStr hin "Test data" hGetContents hout
All this should effectively do is to give you back "Test data". While it *does* do that, it takes far too long. When I run ‘gs’, it will start to (lazily) print the result, printing nothing but opening ‘"’ and then after about 2-3 seconds printing the rest and finishing.
If we set buffering on the in-handle (hin) to NoBuffering, we get a slightly different behaviour: pretty much straight away we'll have ‘"Test data’ but then it will wait for the same amount of time to conclude that it's the end of the response. Changing buffering mode on ‘hout’ seems to make no difference. Setting precise number in a BlockBuffering seems to be no improvement and in the actual application I will not know how long the data I'm piping in and out will be.
GHC 7.8.2, process-1.2.2.0; I'm running ‘gs’ in GHCi. It seems that if I change the module name to Main, make ‘main = gs >>= putStrLn’, compile the file and run it, it just hangs there! If I add a newline at the end, it will print but the program will not finish. This makes me think that perhaps I should be closing handles somewhere (but if I try inside the function, I get no output, thanks lazy I/O).
What I would expect this program to do is to produce same result as ‘print "Test data" | cat’.
As it often happens, I solved it straight away after posting to the list. Here's the program that behaves how I wanted it to from the start:
gs ∷ IO String gs = do let sp = (proc "cat" []) { std_out = CreatePipe, std_in = CreatePipe } (Just hin, Just hout, _, _) ← createProcess sp hPutStr hin "Test data" hClose hin c ← hGetContents hout length c `seq` hClose hout return c
-- Mateusz K. _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
I hope you don't mind a bit of self-promotion. My process-streaminghttp://hackage.haskell.org/package/process-streaminglibrary provides helper functions for process http://hackage.haskell.org/package/process based on the pipeshttp://hackage.haskell.org/package/pipesstreaming library. The cat example would be written like this (taken verbatim from the tutorial): example6 = exitCode show $ execute3 (shell "cat") show (surely . useProducer $ yield "aaaaaa\naaaaa") (separate (encoding T.decodeIso8859_1 ignoreLeftovers $ surely $ T.toLazyM) nop ) Returns:
*Right ((),("aaaaaa\naaaaa",()))*
Writing stdin is done concurrently with the reading of stdout. When an exception is encountered, the library ensures that the handles are closed, the extant concurrent threads terminated, and the external process killed. Also stderr is drained even if you ignore it, to avoid deadlocks due to full buffers. http://hackage.haskell.org/package/process-streaming-0.0.1.1 http://hackage.haskell.org/package/process-streaming-0.0.1.1/docs/System-Pro... http://productivedetour.blogspot.com.es/2014/02/process-pipes-process-stream... On Monday, May 12, 2014 4:56:46 PM UTC+2, Mateusz Kowalczyk wrote:
Hi,
I'm have some business in piping some data and reading some data back out of a socket so I thought that I'd just use the ‘socat’ tool. I went off to System.Process just to find out that reading and writing are taking far too long.
I put together a small example which only requires that you have ‘cat’ on your system:
{-# LANGUAGE UnicodeSyntax #-} module Uzbl.WithSource where
import GHC.IO.Handle ( hPutStr, hGetContents, hSetBuffering , BufferMode(..)) import System.Process ( createProcess, proc , StdStream(CreatePipe), std_out, std_in)
gs ∷ IO String gs = do let sp = (proc "cat" []) { std_out = CreatePipe, std_in = CreatePipe } (Just hin, Just hout, _, _) ← createProcess sp -- hSetBuffering hin NoBuffering -- hSetBuffering hout NoBuffering hPutStr hin "Test data" hGetContents hout
All this should effectively do is to give you back "Test data". While it *does* do that, it takes far too long. When I run ‘gs’, it will start to (lazily) print the result, printing nothing but opening ‘"’ and then after about 2-3 seconds printing the rest and finishing.
If we set buffering on the in-handle (hin) to NoBuffering, we get a slightly different behaviour: pretty much straight away we'll have ‘"Test data’ but then it will wait for the same amount of time to conclude that it's the end of the response. Changing buffering mode on ‘hout’ seems to make no difference. Setting precise number in a BlockBuffering seems to be no improvement and in the actual application I will not know how long the data I'm piping in and out will be.
GHC 7.8.2, process-1.2.2.0; I'm running ‘gs’ in GHCi. It seems that if I change the module name to Main, make ‘main = gs >>= putStrLn’, compile the file and run it, it just hangs there! If I add a newline at the end, it will print but the program will not finish. This makes me think that perhaps I should be closing handles somewhere (but if I try inside the function, I get no output, thanks lazy I/O).
What I would expect this program to do is to produce same result as ‘print "Test data" | cat’.
-- Mateusz K. _______________________________________________ Haskell-Cafe mailing list Haskel...@haskell.org javascript: http://www.haskell.org/mailman/listinfo/haskell-cafe
participants (3)
-
Daniel Díaz -
Krzysztof Skrzętnicki -
Mateusz Kowalczyk