Hi, I'm using POpen to shell out to a command several hundreds or thousands of times per call (none of them simultaneous, though, this is completely serial). After running my program for a while, I get: Fail: resource exhausted Action: forkProcess Reason: Resource temporarily unavailable which basically seems to be telling me that the program hasn't been closing the old processes, even though they're definitely not in use anymore. Does anyone have any suggestions how to get around this? Thanks! - Hal -- Hal Daume III | hdaume@isi.edu "Arrest this man, he talks in maths." | www.isi.edu/~hdaume
Hal Daume III wrote:
I'm using POpen to shell out to a command several hundreds or thousands of times per call (none of them simultaneous, though, this is completely serial). After running my program for a while, I get:
Fail: resource exhausted Action: forkProcess Reason: Resource temporarily unavailable
which basically seems to be telling me that the program hasn't been closing the old processes, even though they're definitely not in use anymore.
Does anyone have any suggestions how to get around this?
I note two facts regarding POpen.popen:
1. There is no corresponding pclose function.
2. It uses lazy I/O (hGetContents).
Also, I can't see wait/waitpid getting called anywhere (although there
might be other mechanims involved, e.g. SIGCHLD handlers; I haven't
looked that closely).
What does the output from "ps" indicate?
If you have any "live" processes (S or R state), it's probably because
the process' output hasn't been consumed, so the program hasn't
exit()ed yet. OTOH, if you have zombies (Z state), the program has
terminated but the parent (your program) hasn't called wait/waitpid
(the Haskell interface is getProcessStatus, getProcessGroupStatus or
getAnyProcessStatus).
--
Glynn Clements
On Thu, 8 Jan 2004, Glynn Clements wrote:
What does the output from "ps" indicate?
It lists all the processes as defunct: 19981 pts/5 Z 0:00 [suffixtree <defunct>] 19982 pts/5 Z 0:00 [suffixtree <defunct>] 19983 pts/5 Z 0:00 [suffixtree <defunct>] 19984 pts/5 Z 0:00 [suffixtree <defunct>] 19985 pts/5 Z 0:00 [suffixtree <defunct>] ...
If you have any "live" processes (S or R state), it's probably because the process' output hasn't been consumed, so the program hasn't exit()ed yet. OTOH, if you have zombies (Z state), the program has terminated but the parent (your program) hasn't called wait/waitpid (the Haskell interface is getProcessStatus, getProcessGroupStatus or getAnyProcessStatus).
I don't mind evaluating the contents returned strictly, but I can't figure out how to force the process into a dead state...I don't see how any of these three functions accomplishes that...what am I missing? -- Hal Daume III | hdaume@isi.edu "Arrest this man, he talks in maths." | www.isi.edu/~hdaume
Hal Daume III
On Thu, 8 Jan 2004, Glynn Clements wrote:
What does the output from "ps" indicate?
It lists all the processes as defunct:
19981 pts/5 Z 0:00 [suffixtree <defunct>] 19982 pts/5 Z 0:00 [suffixtree <defunct>] 19983 pts/5 Z 0:00 [suffixtree <defunct>] 19984 pts/5 Z 0:00 [suffixtree <defunct>] 19985 pts/5 Z 0:00 [suffixtree <defunct>] ...
if you have zombies (Z state), the program has terminated but the parent (your program) hasn't called wait/waitpid (the Haskell interface is getProcessStatus, getProcessGroupStatus or getAnyProcessStatus).
I don't mind evaluating the contents returned strictly, but I can't figure out how to force the process into a dead state...
They are dead, just not acknowledged by their parent, so the OS keeps their exit statuses around.
I don't see how any of these three functions accomplishes that... what am I missing?
These functions read the exit status, and the OS will clean up. If you start them sequentially, then simply insert a call to the appropriate function at a point where the previous child is already finished. Or spawn a thread which sequence_s nonblocking getAnyProcessStatuses... :) Never tried myself. man waitpid Feri.
Hal Daume III wrote:
What does the output from "ps" indicate?
It lists all the processes as defunct:
19981 pts/5 Z 0:00 [suffixtree <defunct>] 19982 pts/5 Z 0:00 [suffixtree <defunct>] 19983 pts/5 Z 0:00 [suffixtree <defunct>] 19984 pts/5 Z 0:00 [suffixtree <defunct>] 19985 pts/5 Z 0:00 [suffixtree <defunct>] ...
If you have any "live" processes (S or R state), it's probably because the process' output hasn't been consumed, so the program hasn't exit()ed yet. OTOH, if you have zombies (Z state), the program has terminated but the parent (your program) hasn't called wait/waitpid (the Haskell interface is getProcessStatus, getProcessGroupStatus or getAnyProcessStatus).
I don't mind evaluating the contents returned strictly, but I can't figure out how to force the process into a dead state...I don't see how any of these three functions accomplishes that...what am I missing?
A "zombie" process (such as the above) is a process which has
terminated but which can't actually be removed from the system's
process table until the parent has retrieved its exit status.
That's where getProcessStatus etc (wait/waitpid at the C level) come
in; these functions block until a suitable[1] process has terminated,
and return its exit status. After which, the process can finally be
deleted (this is termed "reaping").
[1] getProcessStatus waits for a specific process,
getProcessGroupStatus waits for any process in a specific process
group, and getAnyProcessStatus waits for any child process.
So, you probably want something like:
do
(stdout, stderr, pid) <- popen cmd args (Just input)
-- consume stdout + stderr, e.g.:
writeFile "/dev/null" stdout
writeFile "/dev/null" stderr
getProcessStatus True False pid
You need to ensure that the output is consumed before calling
getProcessStatus, otherwise getProcessStatus will block indefinitely
(i.e. deadlock).
IMNSHO, this is one area where lazy I/O sucks even more than usual.
It's bad enough having it tie up descriptors, let alone processes.
It probably works fine for "simple, stupid programs", which spawn a
handful of child processes (which they don't bother to reap) and then
terminate shortly thereafter (a process whose parent has terminated is
"adopted" by the init process, which can be relied upon to reap it
when it terminates).
--
Glynn Clements
On Thu, Jan 08, 2004 at 09:33:29AM -0800, Hal Daume III wrote:
Hi,
I'm using POpen to shell out to a command several hundreds or thousands of times per call (none of them simultaneous, though, this is completely serial). After running my program for a while, I get:
Fail: resource exhausted Action: forkProcess Reason: Resource temporarily unavailable
which basically seems to be telling me that the program hasn't been closing the old processes, even though they're definitely not in use anymore.
Does anyone have any suggestions how to get around this?
I had a similar problem, and finally I created my own solution that doesn't leave zombies and doesn't block when the launched process writes too much to stderr. I tested it in GHC 6.0. For 6.2 you'd have to change the use of forkProcess. Usage: launch :: String -> [String] -> String -> IO (ProcessStatus, String, String) (status, out, err) <- launch prog args progInput Example: *Shell> (status, out, err) <- launch "tr" ["a-z", "A-Z"] (unlines (replicate 10000 "Haskell")) *Shell> status Exited ExitSuccess *Shell> length out 80000 *Shell> mapM_ putStrLn (take 10 (lines out)) HASKELL HASKELL HASKELL HASKELL HASKELL HASKELL HASKELL HASKELL HASKELL HASKELL Best regards, Tom -- .signature: Too many levels of symbolic links
Tomasz Zielonka
I had a similar problem, and finally I created my own solution that doesn't leave zombies and doesn't block when the launched process writes too much to stderr.
Pretty neat, I've got an application idea for that code! Couldn't it be include in the standard libraries? Anyway, thanks for posting it. Feri.
Tomasz, Your code looks great, but where do you find the library documentation, like what the arguments for executeFile are all about? (I'd guessed the Maybe thing was an environment, but what's the Bool?) I've been trying to do similar stuff, but have been stumbling in the dark rather. -- Mark
Mark Carroll wrote:
Your code looks great, but where do you find the library documentation, like what the arguments for executeFile are all about? (I'd guessed the Maybe thing was an environment, but what's the Bool?) I've been trying to do similar stuff, but have been stumbling in the dark rather.
Source code (hslibs/posix/PosixProcPrim.lhs):
executeFile :: FilePath -- Command
-> Bool -- Search PATH?
-> [String] -- Arguments
-> Maybe [(String, String)] -- Environment
-> IO ()
executeFile path search args Nothing = do
[snip]
if search
then throwErrnoIfMinus1_ "executeFile" (c_execvp s arr)
else throwErrnoIfMinus1_ "executeFile" (c_execv s arr)
executeFile path search args (Just env) = do
[snip]
if search
then throwErrnoIfMinus1_ "executeFile" (c_execvpe s arg_arr env_arr)
else throwErrnoIfMinus1_ "executeFile" (c_execve s arg_arr env_arr)
IOW:
search env function
False Nothing execv
True Nothing execvp
False Just _ execve
True Just _ execvpe [*]
[*] execvpe() isn't a standard library function; it is implemented in
hslibs/posix/cbits/execvpe.c using execve().
--
Glynn Clements
On Sat, Jan 10, 2004 at 04:50:28PM -0500, Mark Carroll wrote:
Tomasz,
Your code looks great,
Thanks :) It was written in haste for a particular purpose and then tweaked a bit, so I would be pleasantly surprised if it didn't contain any bugs (besides the one mentioned in code). It isn't suitable for exchanging bigger amounts of data between processes.
but where do you find the library documentation, like what the arguments for executeFile are all about? (I'd guessed the Maybe thing was an environment, but what's the Bool?)
That was about a month ago, so I don't remember exactly, but knowing me I guess I must have looked into the source code. You can find some good, but sometimes not haddockised, comments there. Glynn Clements already cited some relevant code, but I think I was rather looking at libraries/unix/System/Posix/Process.hsc which is the one I used in my code, probably more recent. Best regards, Tom -- .signature: Too many levels of symbolic links
participants (5)
-
Ferenc Wagner -
Glynn Clements -
Hal Daume III -
Mark Carroll -
Tomasz Zielonka