Bug in runInteractiveProcess?

newer
Re: [Haskell-cafe] On the verge of...

John Goerzen

16 Oct 2007 16 Oct '07

8:48 p.m.

Hi everyone, I have been trying to implement a Haskell-like version of shell pipelines using runInteractiveProcess. I am essentially using hGetContents to grab the output from one command, and passing that to (forkIO $ hPutStr) to write to the next. Slow, but this is just an experiment. This works OK to link together two external commands, but deadlocks when linking together three processes. After staring at straces all afternoon, I think I have found the culprit. I observed that when I have three commands linked together, the second never sees EOF on its input. This even though I saw an explicit close on the other end of the pipe from the Haskell side. Why is this, I wondered? I stared at the source for the C runInteractiveProcess function for a bit, but then the answer was right there: this function does not sanitize FDs. What that means is that the write end of the stdin pipe for command 2 was open in the Haskell process. The Haskell process then forked off for command 3 later on. This write end of the stdin pipe for command 2 was never closed in the command 3 child environment. Therefore, closing it in the Haskell thread did not cause command 2 to get an EOF. Does that make sense? Many systems will just try to close *all* FDs except the ones they need after a fork(). Another approach would be to maintain a global list of FDs that the Haskell thread is using, and close all of them except the pipe ends in the child. Without something like this, it is not possible to use runInteractiveProcess more than twice simultaneously in a single program. That renders it almost useless for some tasks. Does this make sense to everyone? If so, I'll submit the bug on GHC. -- John

Show replies by date

Jules Bean

16 Oct 16 Oct

9:31 p.m.

John Goerzen wrote:

...

Many systems will just try to close *all* FDs except the ones they need after a fork(). Another approach would be to maintain a global list of FDs that the Haskell thread is using, and close all of them except the pipe ends in the child.

...

Does this make sense to everyone? If so, I'll submit the bug on GHC.

Yes, it does make sense. On POSIX systems it should suffice to just have haskell set CLOSE_ON_EXEC on all its fds except std{in,out,err} and special pipe fds, shouldn't it? Jules

John Goerzen

9:58 p.m.

On 2007-10-16, Jules Bean wrote:

...

John Goerzen wrote:

...
Many systems will just try to close *all* FDs except the ones they need after a fork(). Another approach would be to maintain a global list of FDs that the Haskell thread is using, and close all of them except the pipe ends in the child.

...
Does this make sense to everyone? If so, I'll submit the bug on GHC.

Yes, it does make sense.

On POSIX systems it should suffice to just have haskell set CLOSE_ON_EXEC on all its fds except std{in,out,err} and special pipe fds, shouldn't it?

Do you mean FD_CLOEXEC, which can be set with fcntl()? If so, it's not defined in POSIX according to the Linux manpage. I couldn't find CLOSE_ON_EXEC in either open(2) or fcntl(2). If FD_CLOEXEC is set in the Haskell process atomically, before anything has a chance to fork off and use it, this would be an ideal solution on platforms that support it. -- John

Richard A. O'Keefe

17 Oct 17 Oct

1:40 a.m.

On 17 Oct 2007, at 10:58 am, John Goerzen wrote:

...

Do you mean FD_CLOEXEC, which can be set with fcntl()? If so, it's not defined in POSIX according to the Linux manpage. I couldn't find CLOSE_ON_EXEC in either open(2) or fcntl(2).

F_GETFD and F_SETFD are the things to look for; FD_CLOEXEC is a fancy way of saying 1 in historic UNIXes. The OSF/1 /usr/include/sys/ fcntl.h says that FD_CLOEXEC is "POSIX REQUIRED". This facility is most certainly part of the Single Unix Specification. The MacOS 10.4 manual page for fcntl() doesn't mention FD_CLOEXEC, but it *does* mention F_GETFD and F_SETFD and identifies the close-on-execute flag as being the "low-order bit" of that flags word, so what may possibly be missing from some editions of POSIX is the *name* FD_CLOEXEC but not the facility (F_SETFD) or the value (1).

Brandon S. Allbery KF8NH

2:31 a.m.

On Oct 16, 2007, at 21:40 , Richard A. O'Keefe wrote:

...

F_GETFD and F_SETFD are the things to look for; FD_CLOEXEC is a fancy way of saying 1 in historic UNIXes. The OSF/1 /usr/include/sys/ fcntl.h says that FD_CLOEXEC is "POSIX REQUIRED". This facility is most certainly part of the Single Unix Specification. The MacOS 10.4 manual page for fcntl() doesn't mention FD_CLOEXEC, but it *does* mention F_GETFD and F_SETFD and identifies the close-on-execute flag as being the "low-order bit" of that flags word, so what may possibly be missing from some editions of POSIX is the *name* FD_CLOEXEC but not the facility (F_SETFD) or the value (1).

I could dig for official confirmation, but this is my understanding of both POSIX and SUS, and portable C programs generally #define FD_CLOEXEC to 1 if it doesn't already exist, since the value *is* standard even though the name is not. -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH

Donn Cave

4:39 a.m.

On Oct 16, 2007, at 7:31 PM, Brandon S. Allbery KF8NH wrote:

...

I could dig for official confirmation, but this is my understanding of both POSIX and SUS, and portable C programs generally #define FD_CLOEXEC to 1 if it doesn't already exist, since the value *is* standard even though the name is not.

I find `runInteractiveProcess' in System.Process, in the documentation - not System.Posix.Process. Possibly the C macro is less important than the cross-platform semantics relating to this problem. I.e., what happens on Microsoft Windows. I sure wouldn't know. Does a process even inherit pipe file descriptors? Or pseudo-ttys - or does it even have them? A design with a two or three-way pipe connection like this is asking for trouble - begging for it, with a name like `runInteractiveProcess' - because most of the commands that you might invoke will block-buffer output, confounding the application that expects to conduct a dialogue with a forked command. Pseudo-ttys are pipe-like devices that won't be block-buffered, and they're really the only more or less reliable way to have an `interactive' I/O exchange with another command that wasn't written specifically to support this. As for closing file descriptors explicitly - if I remember right what I've seen in the NetBSD source, the UNIX popen() implementation may years ago have closed all file descriptors, but now it keeps track of the ones it created, and only closes them. I think that's the way to go, if closing fds. Donn Cave, donn@drizzle.com

Brandon S. Allbery KF8NH

4:52 a.m.

On Oct 17, 2007, at 0:39 , Donn Cave wrote:

...

On Oct 16, 2007, at 7:31 PM, Brandon S. Allbery KF8NH wrote:

...
I could dig for official confirmation, but this is my understanding of both POSIX and SUS, and portable C programs generally #define FD_CLOEXEC to 1 if it doesn't already exist, since the value *is* standard even though the name is not.

I find `runInteractiveProcess' in System.Process, in the documentation - not System.Posix.Process. Possibly the C macro is less important than the cross-platform semantics relating to this problem. I.e., what happens on Microsoft Windows. I sure wouldn't know. Does a process even inherit pipe file descriptors?

The context of my message was a proposal for how to fix it on POSIX systems. I do not claim to have sufficient understanding of Win32 APIs to fix that implementation --- but I suspect that there are already many other implementation differences between the POSIX and Win32 versions of System.Process, if only because popen(), fork(), and exec() are entirely foreign concepts in Win32.

...

As for closing file descriptors explicitly - if I remember right what I've seen in the NetBSD source, the UNIX popen() implementation may years ago have closed all file descriptors, but now it keeps track of the ones it created, and only closes them. I think that's the way to go, if closing fds.

Either implementation causes problems; security folks tend to prefer that all file descriptors other than 0-2 (0-4 on Windows?) be closed, and 0-2(4) be forced open (on /dev/null if they're not already open). But in this case, the idea is to set FD_CLOEXEC on (and only on) file descriptors opened by the Haskell runtime, so you would get the same effect as tracking file descriptors manually. -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH

Donn Cave

5:32 a.m.

On Oct 16, 2007, at 9:52 PM, Brandon S. Allbery KF8NH wrote:

...

On Oct 17, 2007, at 0:39 , Donn Cave wrote: ...

...
As for closing file descriptors explicitly - if I remember right what I've seen in the NetBSD source, the UNIX popen() implementation may years ago have closed all file descriptors, but now it keeps track of the ones it created, and only closes them. I think that's the way to go, if closing fds.

Either implementation causes problems; security folks tend to prefer that all file descriptors other than 0-2 (0-4 on Windows?) be closed, and 0-2(4) be forced open (on /dev/null if they're not already open). But in this case, the idea is to set FD_CLOEXEC on (and only on) file descriptors opened by the Haskell runtime, so you would get the same effect as tracking file descriptors manually.

I can't speak for security folks, but for me, the way you put it goes way too far. The file descriptors at issue were opened by runInteractiveProcess, and FD_CLOEXEC on them would solve the whole problem (I think.) Is that what you mean? To set this flag routinely on all file descriptors opened in any way would require a different justification, and it would have to be a pretty good one! Donn Cave, donn@drizzle.com

Simon Marlow

8:35 a.m.

Donn Cave wrote:

...

On Oct 16, 2007, at 9:52 PM, Brandon S. Allbery KF8NH wrote:

...
On Oct 17, 2007, at 0:39 , Donn Cave wrote: ...

...
As for closing file descriptors explicitly - if I remember right what I've seen in the NetBSD source, the UNIX popen() implementation may years ago have closed all file descriptors, but now it keeps track of the ones it created, and only closes them. I think that's the way to go, if closing fds.

Either implementation causes problems; security folks tend to prefer that all file descriptors other than 0-2 (0-4 on Windows?) be closed, and 0-2(4) be forced open (on /dev/null if they're not already open). But in this case, the idea is to set FD_CLOEXEC on (and only on) file descriptors opened by the Haskell runtime, so you would get the same effect as tracking file descriptors manually.

I can't speak for security folks, but for me, the way you put it goes way too far. The file descriptors at issue were opened by runInteractiveProcess, and FD_CLOEXEC on them would solve the whole problem (I think.) Is that what you mean? To set this flag routinely on all file descriptors opened in any way would require a different justification, and it would have to be a pretty good one!

Setting FD_CLOEXEC on just the pipes created by runInteractiveProcess sounds right to me. Certainly we don't want to set the flag on *all* FDs created in Haskell, in particular users of System.Posix.openFd probably want to choose whether they set FD_CLOEXEC or not. Would someone like to create a bug report? Cheers, Simon

John Goerzen

8:17 p.m.

On 2007-10-17, Simon Marlow wrote:

...

they set FD_CLOEXEC or not.

Would someone like to create a bug report?

http://hackage.haskell.org/trac/ghc/ticket/1780

Brandon S. Allbery KF8NH

1:53 p.m.

On Oct 17, 2007, at 1:32 , Donn Cave wrote:

...

On Oct 16, 2007, at 9:52 PM, Brandon S. Allbery KF8NH wrote:

...
Either implementation causes problems; security folks tend to prefer that all file descriptors other than 0-2 (0-4 on Windows?) be closed, and 0-2(4) be forced open (on /dev/null if they're not already open). But in this case, the idea is to set FD_CLOEXEC on (and only on) file descriptors opened by the Haskell runtime, so you would get the same effect as tracking file descriptors manually.

I can't speak for security folks, but for me, the way you put it goes way too far. The file descriptors at issue were opened by runInteractiveProcess, and FD_CLOEXEC on them would solve the whole problem (I think.) Is that what you mean? To set this flag routinely on all file descriptors opened in any way would require a different justification, and it would have to be a pretty good one!

Well, security folks (professional paranoids :) tend to consider passing anything other than standard file descriptors to arbitrary subprocesses to be a potential uncontrolled information leak. There *are* times when you want to care about this, but in general there is a tradeoff between secure and usable so most practical systems take the middle road and make the programmer do fd swizzling by hand if they need special behavior in either direction (either more or less sharing, that is). (Early Unix, on the other hand, erred toward the permissive/promiscuous, cf. your NetBSD source comparison.) -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH

Donn Cave

3:18 p.m.

On Wed, 17 Oct 2007, Brandon S. Allbery KF8NH wrote: ...

...

Well, security folks (professional paranoids :) tend to consider passing anything other than standard file descriptors to arbitrary subprocesses to be a potential uncontrolled information leak. There *are* times when you want to care about this, but in general there is a tradeoff between secure and usable so most practical systems take the middle road and make the programmer do fd swizzling by hand if they need special behavior in either direction (either more or less sharing, that is). (Early Unix, on the other hand, erred toward the permissive/promiscuous, cf. your NetBSD source comparison.)

My source observations may have been ambiguous. Old NetBSD popen closed all fds, current NetBSD popen closes only popen fds. Donn Cave, donn@drizzle.com

Donn Cave

5:04 a.m.

On Oct 16, 2007, at 1:48 PM, John Goerzen wrote:

...

I have been trying to implement a Haskell-like version of shell pipelines using runInteractiveProcess. I am essentially using hGetContents to grab the output from one command, and passing that to (forkIO $ hPutStr) to write to the next. Slow, but this is just an experiment.

As an aside, I personally would look to System.Posix.Process for this. Something like this would deal with the file descriptors in the fork ... fdfork fn dupfds closefds = do pid <- forkProcess $ execio return pid where dupe (a, b) = do dupTo a b closeFd a execio = do mapM_ dupe dupfds mapM_ closeFd closefds fn ... and then you can put the pipes directly between the processes ... -- date | tr '[A-Z]' '[a-z]' | read (a0, a1) <- createPipe (b0, b1) <- createPipe p1 <- fdfork tr [(b0, 0), (a1, 1)] [a0, b1] closeFd a1 p2 <- fdfork date [(b1, 1)] [a0, b0] closeFd b1 readfd a0 -- implementation left to reader where date = executeFile "/bin/date" False [] Nothing tr = executeFile "/usr/bin/tr" False ["[A-Z]", "[a-z]"] Nothing There's probably a nice way to wrap that up, so you're not keeping track of the file descriptors for all the pipes. Donn Cave, donn@drizzle.com

Simon Marlow

8:56 a.m.

Donn Cave wrote:

...

On Oct 16, 2007, at 1:48 PM, John Goerzen wrote:

...
I have been trying to implement a Haskell-like version of shell pipelines using runInteractiveProcess. I am essentially using hGetContents to grab the output from one command, and passing that to (forkIO $ hPutStr) to write to the next. Slow, but this is just an experiment.

As an aside, I personally would look to System.Posix.Process for this. Something like this would deal with the file descriptors in the fork ...

fdfork fn dupfds closefds = do pid <- forkProcess $ execio return pid where dupe (a, b) = do dupTo a b closeFd a execio = do mapM_ dupe dupfds mapM_ closeFd closefds fn

Note that forkProcess doesn't currently work with +RTS -N2 (or any value larger than 1), and it isn't likely to in the future. I suspect forkProcess should be deprecated. The POSIX spec is pretty strict about what system calls you can make in the child process of a fork in a multithreaded program, and those same restrictions apply in Haskell, and they apply not only to the Haskell code but also to the runtime (e.g. what if the child process needs more memory and the runtime calls mmap(), that's not allowed). We get away with it most of the time because the OSs we run on are less strict than POSIX. However, in general I think forking should be restricted to C code that you invoke via the FFI. Cheers, Simon

Donn Cave

3:46 p.m.

On Wed, 17 Oct 2007, Simon Marlow wrote: ...

...

Note that forkProcess doesn't currently work with +RTS -N2 (or any value larger than 1), and it isn't likely to in the future. I suspect forkProcess should be deprecated.

The POSIX spec is pretty strict about what system calls you can make in the child process of a fork in a multithreaded program, and those same restrictions apply in Haskell, and they apply not only to the Haskell code but also to the runtime (e.g. what if the child process needs more memory and the runtime calls mmap(), that's not allowed). We get away with it most of the time because the OSs we run on are less strict than POSIX. However, in general I think forking should be restricted to C code that you invoke via the FFI.

Just to be precise about it, though, there's nothing about Haskell per se that causes trouble with fork, right? This is a GHC implementation issue. Donn Cave, donn@drizzle.com

Stefan O'Rear

18 Oct 18 Oct

1:11 a.m.

On Wed, Oct 17, 2007 at 08:46:41AM -0700, Donn Cave wrote:

...

On Wed, 17 Oct 2007, Simon Marlow wrote: ...

...
Note that forkProcess doesn't currently work with +RTS -N2 (or any value larger than 1), and it isn't likely to in the future. I suspect forkProcess should be deprecated.

The POSIX spec is pretty strict about what system calls you can make in the child process of a fork in a multithreaded program, and those same restrictions apply in Haskell, and they apply not only to the Haskell code but also to the runtime (e.g. what if the child process needs more memory and the runtime calls mmap(), that's not allowed). We get away with it most of the time because the OSs we run on are less strict than POSIX. However, in general I think forking should be restricted to C code that you invoke via the FFI.

Just to be precise about it, though, there's nothing about Haskell per se that causes trouble with fork, right? This is a GHC implementation issue.

Forking in the presense of multiple threads is a semantic nightmare. Anything any Haskell implementation does is going to be wrong in at least two ways, so it's probably best not to try. Write your fork-child code in C. Stefan

Donn Cave

6:37 a.m.

On Oct 17, 2007, at 6:11 PM, Stefan O'Rear wrote:

...

...
Just to be precise about it, though, there's nothing about Haskell per se that causes trouble with fork, right? This is a GHC implementation issue.

Forking in the presense of multiple threads is a semantic nightmare. Anything any Haskell implementation does is going to be wrong in at least two ways, so it's probably best not to try. Write your fork- child code in C.

Forking in the presence of multiple threads is not my problem - I don't want multiple threads. Any Haskell implementation will do fine as long as, like C, it doesn't use threads to solve internal runtime problems. I know that for some, the UNIX process machinery - fork, the file descriptors, execve, etc. - is an appalling and gruesome spectacle that surely ought to have faded away in the lovely new order of things, but it's really a fairly elegant system in its antique way, and UNIX (et al.) is built around it on a big scale. It isn't going away. Donn Cave, donn@drizzle.com

John Goerzen

17 Oct 17 Oct

8:11 p.m.

On 2007-10-17, Simon Marlow wrote:

...

Note that forkProcess doesn't currently work with +RTS -N2 (or any value larger than 1), and it isn't likely to in the future. I suspect forkProcess should be deprecated.

That would be most annoying, and would render HSH unable to function without using FFI to import fork from C. I suspect that forkProcess has some more intelligence to it than that, though I haven't looked. System.Process is not powerful enough to do serious work on POSIX, and perhaps it never can be. The mechanism for setting up a direct pipeline with more than 2 processes is extremely inconvenient at best, and it does not seem possible to create n-process pipelines using System.Process without having to resort to copying data in the Haskell process at some point. (This even putting aside the instant bug) Not only that, but the ProcessHandle system doesn't: * Let me get the child process's PID * Send arbitrary signals to the child process * Handle SIGCHLD in a custom and sane way * Get full exit status information (stopped by a particular signal, etc) Now, there are likely perfectly valid cross-platform reasons that it doesn't do this. I am merely trying to point out that removing forkProcess in favor of System.Process will shut out a large number of very useful things. Don't forget either that there are a whole class of programs whose multithreading needs may be better addressed by forkProcess, executeFile, and clever management of subprograms rather than by a threaded RTS. forkProcess, with the ability to dupTo, closeFd, and executeFile is still mighty useful in my book.

Simon Marlow

18 Oct 18 Oct

8:33 a.m.

John Goerzen wrote:

...

On 2007-10-17, Simon Marlow wrote:

...
Note that forkProcess doesn't currently work with +RTS -N2 (or any value larger than 1), and it isn't likely to in the future. I suspect forkProcess should be deprecated.

That would be most annoying, and would render HSH unable to function without using FFI to import fork from C. I suspect that forkProcess has some more intelligence to it than that, though I haven't looked.

System.Process is not powerful enough to do serious work on POSIX, and perhaps it never can be. The mechanism for setting up a direct pipeline with more than 2 processes is extremely inconvenient at best, and it does not seem possible to create n-process pipelines using System.Process without having to resort to copying data in the Haskell process at some point. (This even putting aside the instant bug)

Not only that, but the ProcessHandle system doesn't:

* Let me get the child process's PID * Send arbitrary signals to the child process * Handle SIGCHLD in a custom and sane way * Get full exit status information (stopped by a particular signal, etc)

Now, there are likely perfectly valid cross-platform reasons that it doesn't do this.

Yes, absolutely. Although I *would* like there to be a more general version of runInteractiveProcess where for each FD the caller can choose whether to supply an existing Handle or to have a new pipe generated. This would let you pipe multiple processes together directly, which can't be done at the moment (we've discussed this before, I think).

...

I am merely trying to point out that removing forkProcess in favor of System.Process will shut out a large number of very useful things.

I wasn't intending to push users towards System.Process instead, rather to forking in C where it can be done safely. I completely agree that System.Process isn't a replacement for everything you might want to do with fork.

...

Don't forget either that there are a whole class of programs whose multithreading needs may be better addressed by forkProcess, executeFile, and clever management of subprograms rather than by a threaded RTS.

Ok, the non-threaded RTS can indeed support forkProcess without any difficulties. I'm not sure where that leaves us; the non-threaded RTS also cannot support waitForProcess in a multithreaded Haskell program, and it can't do non-blocking FFI calls in general. While we have no immediate plans to get rid of the non-threaded RTS, I'd really like to. The main reasons being that it adds another testing dimension, and it contains a completely separate implementation of non-blocking IO that we have to maintain.

...

forkProcess, with the ability to dupTo, closeFd, and executeFile is still mighty useful in my book.

Noted! Cheers, Simon

6477

Age (days ago)

6479

Last active (days ago)

List overview

Download

18 comments

7 participants

participants (7)

Brandon S. Allbery KF8NH
Donn Cave
John Goerzen
Jules Bean
Richard A. O'Keefe
Simon Marlow
Stefan O'Rear