named pipe interface

Serge D. Mechveliani

12 Jan 2012 12 Jan '12

1:53 p.m.

People, (I wonder: is this for beginners@haskell.org ?) I need to organize a string interface for a Haskell function Main.axiom and a C program fifoFromA.c via a pair of named pipes (in Linux, UNIX). The pipes are created before running, by the commands > mkfifo toA > mkfifo fromA Main.axiom outputs a string to toA and inputs the respond string from fromA as the result. fifoFromA inputs a string from toA, converts it to the string resStr, outputs resStr to fromA. Main.axiom must be able to be applied in a loop, and there must be avoided repeated opening of a file/channel in a loop. As an example, the string conversion in fifoFromA.c is put the conversion of each character to the lower case: -------------------------- fifoFromA.c ------------------------------ #include #include #include #define BOUND 64 int main() { int toA, fromA, i, numread; char buf[BOUND]; toA = open("toA", O_RDONLY); fromA = open("fromA", O_WRONLY); for (;;) { numread = read(toA, buf, BOUND); buf[numread] = '\0'; // printf("A: Read from toA: %s\n", buf); i = 0; // convert the string to the lower case while (i < numread) {buf[i] = tolower(buf[i]); i++;} write(fromA, buf, strlen(buf)); } } ----------------------------------------------------------------------- For the "to-A" part writen in C (instead of Haskell), this interface loop works all right. With Haskell, I manage to process only a single string in the loop, and then it ends with an error. Main.hs is given below. I never dealt with such an IO in Haskell. Can you, please, fix the code or give comments? Please, copy the response to mechvel@botik.ru (I am not in the list). Thank you in advance for your notes, ------ Sergei mechvel@botik.ru ------------------------------------------------------------------- import System.IO (IOMode(..), IO(..), Handle, openFile, hPutStr, hGetLine, hFlush) import System.IO.Unsafe (unsafePerformIO) dir = showString "/home/mechvel/ghc/axiomInterface/byLowerLevel/" toA_IO = openFile (dir "toA") WriteMode :: IO Handle fromA_IO = openFile (dir "fromA") ReadMode -- used as global values toA = unsafePerformIO toA_IO -- fromA = unsafePerformIO fromA_IO -- axiomIO :: String -> IO String axiomIO str = do hPutStr toA str hFlush toA hGetLine fromA axiom :: String -> String -> String axiom str = showString (unsafePerformIO $ axiomIO str) -- Examples of usage -------------------------------------------- -- main = putStr (axiom "ABC1" "\n") -- I -- putStr (axiom "ABC1" $ showChar '\n' $ axiom "ABC2" "\n") -- II {- III: putStr (shows resPairs "\n") where n = 9000 str0 = "ABC" strings = [str0 ++ (show i) | i <- [1 .. n]] resPairs = [(str, axiom str "") | str <- strings] -} ----------------------------------------------------------------- I use Glasgow Haskell ghc-7.01. Build: > gcc -o fifoFromA fifoFromA.c > ghc --make Main Running: first, command > ./fifoFromA on terminal-2, then command > ./Main on terminal-1. Now, the example (I) works -- in the sense that terminal-1 shows the result "abc1" after the program on terminal-2 is interrupted. II and III do not work. And the aim is the examples like III.

Show replies by date

Steffen Schuldenzucker

12 Jan 12 Jan

2:44 p.m.

On 01/12/2012 07:53 PM, Serge D. Mechveliani wrote:

...

[...]

For the "to-A" part writen in C (instead of Haskell), this interface loop works all right. With Haskell, I manage to process only a single string in the loop, and then it ends with an error.

Main.hs is given below.

I never dealt with such an IO in Haskell. Can you, please, fix the code or give comments?

Please, copy the response to mechvel@botik.ru (I am not in the list).

[...] ------------------------------------------------------------------- import System.IO (IOMode(..), IO(..), Handle, openFile, hPutStr, hGetLine, hFlush) import System.IO.Unsafe (unsafePerformIO)

dir = showString "/home/mechvel/ghc/axiomInterface/byLowerLevel/"

toA_IO = openFile (dir "toA") WriteMode :: IO Handle fromA_IO = openFile (dir "fromA") ReadMode -- used as global values toA = unsafePerformIO toA_IO -- fromA = unsafePerformIO fromA_IO --

axiomIO :: String -> IO String axiomIO str = do hPutStr toA str hFlush toA hGetLine fromA

axiom :: String -> String -> String axiom str = showString (unsafePerformIO $ axiomIO str)

-- Examples of usage --------------------------------------------

tl;dr, but did you try to write your program without using unsafePerformIO? It's "considered harmful" for a reason. Cheers, Steffen

Donn Cave

3:22 p.m.

Quoth "Serge D. Mechveliani" ,

...

(I wonder: is this for beginners@haskell.org ?)

No, not really! As already mentioned, the use of UnsafePerformIO goes a little beyond what I think is its intended purpose, and I think you might have better luck here with a test program that illustrates the problem and doesn't depend on that. But while you're looking into that ... you might be interested to know that there's a problem with named pipes in GHC (beyond the problems that afflict anyone who tries to work with named pipes.) GHC I/O routinely sets file descriptors to non-blocking, unlike the IO libraries for C and other languages, and this could add to your troubles with named pipes. You can read up on that and possibly find clues to the present problem. Donn

Chaddaï Fouché

13 Jan 13 Jan

10:34 a.m.

On Thu, Jan 12, 2012 at 7:53 PM, Serge D. Mechveliani wrote:

...

People,

(I wonder: is this for beginners@haskell.org ?)

I don't think so.

...

I need to organize a string interface for a Haskell function Main.axiom and a C program fifoFromA.c

via a pair of named pipes (in Linux, UNIX). The pipes are created before running, by the commands > mkfifo toA > mkfifo fromA

Main.axiom outputs a string to toA and inputs the respond string from fromA as the result. fifoFromA inputs a string from toA, converts it to the string resStr, outputs resStr to fromA.

Now that seems interesting, but just to be clear : did you choose this solution (and why won't you use the FFI instead) or is this just to see how to work it out ? -- Jedaï

Serge D. Mechveliani

12:25 p.m.

On Fri, Jan 13, 2012 at 04:34:37PM +0100, Chadda?? Fouch?? wrote:

...

On Thu, Jan 12, 2012 at 7:53 PM, Serge D. Mechveliani wrote:

...
People,

(I wonder: is this for beginners@haskell.org ?)

I don't think so.

...
I need to organize a string interface for a Haskell function Main.axiom and a C program fifoFromA.c

via a pair of named pipes (in Linux, UNIX). The pipes are created before running, by the commands > mkfifo toA > mkfifo fromA

Main.axiom outputs a string to toA and inputs the respond string from fromA as the result. fifoFromA inputs a string from toA, converts it to the string resStr, outputs resStr to fromA.

Now that seems interesting, but just to be clear : did you choose this solution (and why won't you use the FFI instead) or is this just to see how to work it out ?

Because it is a direct and the simplest approach. Why does one need a foreign language, if all the needed functions are in the standard Haskell library? Sergei.

Brandon Allbery

12:39 p.m.

On Fri, Jan 13, 2012 at 12:25, Serge D. Mechveliani wrote:

...

On Fri, Jan 13, 2012 at 04:34:37PM +0100, Chadda?? Fouch?? wrote:

...
Now that seems interesting, but just to be clear : did you choose this solution (and why won't you use the FFI instead) or is this just to see how to work it out ?

Because it is a direct and the simplest approach. Why does one need a foreign language, if all the needed functions are in the standard Haskell library?

I hope you are aware of the many "gotchas" involved with FIFOs. There are very good reasons why they are not widely used. -- brandon s allbery allbery.b@gmail.com wandering unix systems administrator (available) (412) 475-9364 vm/sms

Michael Craig

1:23 p.m.

Brandon, can you elaborate? Are you talking about UNIX named pipes or FIFO/queue data structures in general? Mike Craig On Fri, Jan 13, 2012 at 12:39 PM, Brandon Allbery wrote:

...

On Fri, Jan 13, 2012 at 12:25, Serge D. Mechveliani wrote:

...
On Fri, Jan 13, 2012 at 04:34:37PM +0100, Chadda?? Fouch?? wrote:

...
Now that seems interesting, but just to be clear : did you choose this solution (and why won't you use the FFI instead) or is this just to see how to work it out ?

Because it is a direct and the simplest approach. Why does one need a foreign language, if all the needed functions are in the standard Haskell library?

I hope you are aware of the many "gotchas" involved with FIFOs. There are very good reasons why they are not widely used.

-- brandon s allbery allbery.b@gmail.com wandering unix systems administrator (available) (412) 475-9364 vm/sms

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Serge D. Mechveliani

1:52 p.m.

I (Sergei) am invesigating (so far) only Unix named pipes.

...

...
I hope you are aware of the many "gotchas" involved with FIFOs. There are very good reasons why they are not widely used.

At least in C <-> C, the Unix named pipes do work. And I am trying to replace the first end with Haskell. On Fri, Jan 13, 2012 at 01:23:24PM -0500, Michael Craig wrote:

...

Brandon, can you elaborate? Are you talking about UNIX named pipes or FIFO/queue data structures in general?

Mike Craig

On Fri, Jan 13, 2012 at 12:39 PM, Brandon Allbery wrote:

...
On Fri, Jan 13, 2012 at 12:25, Serge D. Mechveliani wrote:

...
On Fri, Jan 13, 2012 at 04:34:37PM +0100, Chadda?? Fouch?? wrote:

...
Now that seems interesting, but just to be clear : did you choose this solution (and why won't you use the FFI instead) or is this just to see how to work it out ?

Because it is a direct and the simplest approach. Why does one need a foreign language, if all the needed functions are in the standard Haskell library?

I hope you are aware of the many "gotchas" involved with FIFOs. There are very good reasons why they are not widely used.

-- brandon s allbery allbery.b@gmail.com wandering unix systems administrator (available) (412) 475-9364 vm/sms

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Brandon Allbery

2:03 p.m.

On Fri, Jan 13, 2012 at 13:23, Michael Craig wrote:

...

Brandon, can you elaborate? Are you talking about UNIX named pipes or FIFO/queue data structures in general?

I mean POSIX named pipes. They work, but they don't do what most people think they do, and they're rather annoying to work with in general. -- brandon s allbery allbery.b@gmail.com wandering unix systems administrator (available) (412) 475-9364 vm/sms

Donn Cave

1:08 p.m.

Quoth "Serge D. Mechveliani" , [ ... why in Haskell instead of FFI ... ]

...

Because it is a direct and the simplest approach. Why does one need a foreign language, if all the needed functions are in the standard Haskell library?

The GHC Haskell library makes some compromises with normal I/O functionality for the sake of its runtime thread system. As difficult as named pipes can be in any case, they can be even trickier in GHC Haskell for this reason. I didn't suggest an FFI approach myself, in my previous follow-up, only because if you haven't worked with the FFI it's a significant initial investment, but I believe it's what I would do. (If I were somehow compelled to used named pipes.) Donn

Serge D. Mechveliani

2 p.m.

On Fri, Jan 13, 2012 at 10:08:04AM -0800, Donn Cave wrote:

...

Quoth "Serge D. Mechveliani" , [ ... why in Haskell instead of FFI ... ]

...
Because it is a direct and the simplest approach. Why does one need a foreign language, if all the needed functions are in the standard Haskell library?

The GHC Haskell library makes some compromises with normal I/O functionality for the sake of its runtime thread system. As difficult as named pipes can be in any case, they can be even trickier in GHC Haskell for this reason. I didn't suggest an FFI approach myself, in my previous follow-up, only because if you haven't worked with the FFI it's a significant initial investment, but I believe it's what I would do. (If I were somehow compelled to used named pipes.)

Initially, I did the example by the Foreign Function Interface for C. But then, I thought "But this is unnatural! Use plainly the standard Haskell IO, it has everything". So, your advice is "return to FFI" ? ------ Sergei mechvel@botik.ru

Donn Cave

3:19 p.m.

Quoth "Serge D. Mechveliani" , ...

...

Initially, I did the example by the Foreign Function Interface for C. But then, I thought "But this is unnatural! Use plainly the standard Haskell IO, it has everything".

So, your advice is "return to FFI" ?

Well, it turns out that the I/O system functions in System.Posix.IO may work for your purposes. I was able to get your example to work with these functions, which correspond to open(2), read(2), write(2). I would also use these functions in C, as you did in your C program. Haskell I/O functions like hGetLine are analogous to C library I/O like fgets(3) - in particular, they're buffered, and I would guess that's why they don't work for you here. Specifically, openFile "toA" WriteOnly Nothing defaultFileFlags openFile "fromA" ReadOnly Nothing defaultFileFlags fdWrite toA str (str, len) <- fdRead fromA 64 return str Donn

Serge D. Mechveliani

14 Jan 14 Jan

6:11 a.m.

On Fri, Jan 13, 2012 at 12:19:34PM -0800, Donn Cave wrote:

...

Quoth "Serge D. Mechveliani" , ...

...
Initially, I did the example by the Foreign Function Interface for C. But then, I thought "But this is unnatural! Use plainly the standard Haskell IO, it has everything".

So, your advice is "return to FFI" ?

Well, it turns out that the I/O system functions in System.Posix.IO may work for your purposes. I was able to get your example to work with these functions, which correspond to open(2), read(2), write(2).

I would also use these functions in C, as you did in your C program. Haskell I/O functions like hGetLine are analogous to C library I/O like fgets(3) - in particular, they're buffered, and I would guess that's why they don't work for you here.

Specifically, openFile "toA" WriteOnly Nothing defaultFileFlags openFile "fromA" ReadOnly Nothing defaultFileFlags

fdWrite toA str (str, len) <- fdRead fromA 64 return str

Great! Thank you very much. As I find, Posix.IO is not of the standard, but it is of GHC. Anyway, it fits my purpose. By openFile you, probably, mean openFd. Another point is the number of open files, for a long loop. I put toA_IO = openFd "toA" WriteOnly Nothing defaultFileFlags fromA_IO = openFd "fromA" ReadOnly Nothing defaultFileFlags axiomIO :: String -> IO String axiomIO str = do toA <- toA_IO fromA <- fromA_IO fdWrite toA str (str, _len) <- fdRead fromA 64 return str When applying axiomIO in a loop of 9000 strings, it breaks: "too many open files". I do not understand why it is so, because toA_IO and fromA_IO are global constants (I have not any experience with `do'). Anyway, I have changed this to toA = unsafePerformIO toA_IO fromA = unsafePerformIO fromA_IO axiomIO :: String -> IO String axiomIO str = do fdWrite toA str (str, _len) <- fdRead fromA 64 return str And now, it works in loong loops too (I need to understand further whether my usage of unsafePerformIO really damages the project). Its performance is 9/10 of the C <-> C performance (ghc -O, gcc -O, Linux Debian). It is still slow: 120000 small strings/second on a 2 GHz machine. But this is something to start with.

...

I was able to get your example to work with these functions, which correspond to open(2), read(2), write(2).

I would also use these functions in C, as you did in your C program. Haskell I/O functions like hGetLine are analogous to C library I/O like fgets(3) - in particular, they're buffered, and I would guess that's why they don't work for you here.

Indeed. Initially, I tried C <-> C, and used fgets, fputs, fflush. And it did not work, it required to open/close files inside a loop; I failed with attempts. Again, do not understand, why (do they wait till the buffer is full?). Then, I tried read/write, as it is in fifoFromA.c which I posted. And it works. Now, Haskell <-> C gives a hope. Nice. Thanks, ------ Sergei mechvel@botik.ru

Donn Cave

11:57 a.m.

Quoth "Serge D. Mechveliani" ,

...

By openFile you, probably, mean openFd.

Yes, sorry!

...

Another point is the number of open files, for a long loop. ... toA_IO = openFd "toA" WriteOnly Nothing defaultFileFlags ... When applying axiomIO in a loop of 9000 strings, it breaks: "too many open files". I do not understand why it is so, because toA_IO and fromA_IO are global constants (I have not any experience with `do').

toA_IO is a global constant of type "IO Fd", not "Fd". You now see the importance of this distinction - the action actually transpires at "toA <- toA_IO", and each time that executes, you get a new file descriptor.

...

Anyway, I have changed this to

toA = unsafePerformIO toA_IO fromA = unsafePerformIO fromA_IO ... (I need to understand further whether my usage of unsafePerformIO really damages the project).

It's actually similar to the way some libraries initialize global values, but there are some additional complexities and it isn't clear to me that it's all guaranteed to work anyway. You can read much more about this here: http://www.haskell.org/haskellwiki/Top_level_mutable_state I'm no expert in this, but they're sure here on haskell-cafe, so if you want to take this up, you might start a new topic here, something like "global initialization with unsafePerformIO", describe what you're doing and explain why you can't just pass the open file descriptors as function parameters. ...

...

Indeed. Initially, I tried C <-> C, and used fgets, fputs, fflush. And it did not work, it required to open/close files inside a loop; I failed with attempts. Again, do not understand, why (do they wait till the buffer is full?).

I don't know. When I was younger, I used to track these problems down and try to explain in detail why buffered I/O is a bad bet with pipes, sockets etc. I don't think anyone listened. I think I am going to experiment with "I am old, so listen to me" and see if it works any better. Donn

Brandon Allbery

12:41 p.m.

On Sat, Jan 14, 2012 at 11:57, Donn Cave wrote:

...

I don't know. When I was younger, I used to track these problems down and try to explain in detail why buffered I/O is a bad bet with pipes, sockets etc. I don't think anyone listened. I think I am going to experiment with "I am old, so listen to me" and see if it works any better.

If nothing else, you can transfer one of those explanations into the wiki and point people to it when it comes up. (Nobody pays attention to stuff not directly affecting them. This is annoying but somewhat understandable, especially given that haskell-cafe tends toward information overload / death by a thousand mathematical constructs :) -- brandon s allbery allbery.b@gmail.com wandering unix systems administrator (available) (412) 475-9364 vm/sms

Jean-Marie Gaillourdet

16 Jan 16 Jan

3:38 a.m.

Hi, On 14.01.2012, at 12:11, Serge D. Mechveliani wrote:

...

On Fri, Jan 13, 2012 at 12:19:34PM -0800, Donn Cave wrote:

...
Quoth "Serge D. Mechveliani" , ...

...
Initially, I did the example by the Foreign Function Interface for C. But then, I thought "But this is unnatural! Use plainly the standard Haskell IO, it has everything".

So, your advice is "return to FFI" ?

Well, it turns out that the I/O system functions in System.Posix.IO may work for your purposes. I was able to get your example to work with these functions, which correspond to open(2), read(2), write(2).

I would also use these functions in C, as you did in your C program. Haskell I/O functions like hGetLine are analogous to C library I/O like fgets(3) - in particular, they're buffered, and I would guess that's why they don't work for you here.

Specifically, openFile "toA" WriteOnly Nothing defaultFileFlags openFile "fromA" ReadOnly Nothing defaultFileFlags

fdWrite toA str (str, len) <- fdRead fromA 64 return str

Great! Thank you very much. As I find, Posix.IO is not of the standard, but it is of GHC. Anyway, it fits my purpose. By openFile you, probably, mean openFd.

Another point is the number of open files, for a long loop. I put toA_IO = openFd "toA" WriteOnly Nothing defaultFileFlags fromA_IO = openFd "fromA" ReadOnly Nothing defaultFileFlags

axiomIO :: String -> IO String axiomIO str = do toA <- toA_IO fromA <- fromA_IO fdWrite toA str (str, _len) <- fdRead fromA 64 return str

When applying axiomIO in a loop of 9000 strings, it breaks: "too many open files". I do not understand why it is so, because toA_IO and fromA_IO are global constants (I have not any experience with `do').

Anyway, I have changed this to

toA = unsafePerformIO toA_IO fromA = unsafePerformIO fromA_IO axiomIO :: String -> IO String axiomIO str = do fdWrite toA str (str, _len) <- fdRead fromA 64 return str

And now, it works in loong loops too (I need to understand further whether my usage of unsafePerformIO really damages the project).

I'd say this use of unsafePerformIO is *not* safe. E.g. a Haskell compiler is allowed to evaluate the right hand side of toA and fromA multiple times. If you aren't 100% sure that it is ok to use unsafePerformIO, don't use it! Try something like the following:

...

initIO = do to <- toA_IO from <- fromA_IO return (to, from)

...

axiomIO (to,from) str = do fdWrite to str (str, _len) <- fdRead from 64 return str

...

main = do fds <- initIO … res <- axiomIO fds "…"

Obviously I didn't compile and test this code.

...

Its performance is 9/10 of the C <-> C performance (ghc -O, gcc -O, Linux Debian). It is still slow: 120000 small strings/second on a 2 GHz machine. But this is something to start with.

...
I was able to get your example to work with these functions, which correspond to open(2), read(2), write(2).

I would also use these functions in C, as you did in your C program. Haskell I/O functions like hGetLine are analogous to C library I/O like fgets(3) - in particular, they're buffered, and I would guess that's why they don't work for you here.

Indeed. Initially, I tried C <-> C, and used fgets, fputs, fflush. And it did not work, it required to open/close files inside a loop; I failed with attempts. Again, do not understand, why (do they wait till the buffer is full?).

Then, I tried read/write, as it is in fifoFromA.c which I posted. And it works. Now, Haskell <-> C gives a hope. Nice.

Thanks,

------ Sergei mechvel@botik.ru

_______________________________________________ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

Serge D. Mechveliani

7:01 a.m.

To my question about safety of

...

...
toA_IO = openFd "toA" WriteOnly Nothing defaultFileFlags fromA_IO = openFd "fromA" ReadOnly Nothing defaultFileFlags toA = unsafePerformIO toA_IO fromA = unsafePerformIO fromA_IO

axiomIO :: String -> IO String axiomIO str = do fdWrite toA str (str, _) <- fdRead fromA 64 return str

Jean-Marie Gaillourdet wrote on Jan 16, 2012

...

[..]

I'd say this use of unsafePerformIO is *not* safe. E.g. a Haskell compiler is allowed to evaluate the right hand side of toA and fromA multiple times. If you aren't 100% sure that it is ok to use unsafePerformIO, don't use it! [..]

For example, consider in the above context the program main = do (str1, _) <- fdRead fromA 64 (str2, _) <- fdRead fromA 64 putStr (str1 ++ str2) Is this possible that str1 and str2 are input from different file descriptors? Does this effect depend on in-lining of fromA_IO by the compiler ? Will {-# NOTINLINE "fromA_IO" #-} make it safe? Please, copy the respond to mechvel@botik.ru Regards, ------ Sergei mechvel@botik.ru

Serge D. Mechveliani

13 Jan 13 Jan

2:29 p.m.

On Fri, Jan 13, 2012 at 04:34:37PM +0100, Chadda?? Fouch?? wrote:

...

On Thu, Jan 12, 2012 at 7:53 PM, Serge D. Mechveliani wrote:

...
[..] I need to organize a string interface for a Haskell function Main.axiom and a C program fifoFromA.c

via a pair of named pipes (in Linux, UNIX). The pipes are created before running, by the commands > mkfifo toA > mkfifo fromA

Main.axiom outputs a string to toA and inputs the respond string from fromA as the result. fifoFromA inputs a string from toA, converts it to the string resStr, outputs resStr to fromA.

Now that seems interesting, but just to be clear : did you choose this solution (and why won't you use the FFI instead) or is this just to see how to work it out ?

I am trying to interface my large DoCon program for algebra to (a much larger algebra program) Axiom (the FriCAS license allows this). And for many reasons, there is no real way for this except the string interface. Further, the first candidate for the string interface is Unix named pipes. If the GHC IO cannot sufficiently work with named pipes, I would return to the attempt with the Foreign Function Interface and C <-> C exchange. Sergei. mechvel@botik.ru

4990

Age (days ago)

4994

Last active (days ago)

List overview

Download

17 comments

7 participants

participants (7)

Brandon Allbery
Chaddaï Fouché
Donn Cave
Jean-Marie Gaillourdet
Michael Craig
Serge D. Mechveliani
Steffen Schuldenzucker