On Sat, 2008-07-19 at 23:55 -0500, Galchin, Vasili wrote:

yes Duncan I am trying to pass-by-value. I am familiar with ForeignPtr; however, I don't comprehend what you and Brandon are suggesting to do. Could either of you provide a code illustration or point at existing code to illustrate your approach?

Take a look at John Meacham's RSA example. So at the moment you're using using Storable and a Haskell record, say: data AIOCB = AIOCB { ... } and we're suggesting instead: newtype AIOCB = AIOCB (ForeignPtr AIOCB) then to access a member use hsc2hs: getBlah :: AIOCB -> IO Blah getBlah (AIOCB fptr) = withForeignPtr fptr $ \ptr -> {# peek aiocb,blah #} ptr So you only access the parts you need and keep the aiocb C struct allocated on the heap (use mallocForeignPtr). Duncan

Hi Duncan and Brandon, I am moving to the ForeignPtr strategy. However, I always try to learn from where I am before going to a new approach. I have discovered by debugging that the AIOCB peek is working; however, passing the "peeked" AIOCB back to the caller(i.e. the test program) is not working .. please let me try to demonstrate below. I have been staring at my Haskell code many, many times .... sigh ... 0) IN PEEK CODE .. aioErrorCode => 115 IN PEEK CODE .. aioReturnValue => 0 aioWrite after aio_write IN PEEK CODE .. aioErrorCode => 115 IN PEEK CODE .. aioReturnValue => 0 *************aiocb dump*************** fd => 3 opcode => 1 prio => 0 offset => 0 nbytes => 20 next => 0x00000000 absprio => 0 policy => 0 errocode => 115 <<<< correct ... INPROGRESS errno return value => 0 <<<<<<<<<<<<<<<<<<<< return from call of Haskell function aioWrite here ... below "errocode" has changed from 115 to 0 ... somehow my "return AIOCB" is corrupting the "state"/value of AIOCB .... *****************aioWrite dumpAIOCB *************aiocb dump*************** fd => 3 opcode => 0 prio => 0 offset => 0 nbytes => 20 next => 0x00000000 absprio => 0 policy => 0 errocode => 0 <<<< incorrect "Errno" ... should still be IN PROGRESS. return value => 0 1) aioWrite ... the function marshalling(poke) and unmarshalling(peek) an AIOCB: aioWrite :: AIOCB -> IO AIOCB aioWrite aiocb = do allocaBytes (#const sizeof(struct aiocb)) $ \ p_aiocb -> do poke p_aiocb aiocb putStrLn "aioWrite before aio_write" aiocb1 <- peek p_aiocb dumpAIOCB aiocb1 throwErrnoIfMinus1 "aioWrite" (c_aio_write p_aiocb) aiocb <- peek p_aiocb putStrLn "aioWrite after aio_write" aiocb <- peek p_aiocb dumpAIOCB aiocb -- putStrLn "aioWrite after aio_write" -- aiocb1 <- peek p_aiocb -- dumpAIOCB aiocb1 return (aiocb) foreign import ccall safe "aio.h aio_write" c_aio_write :: Ptr AIOCB -> IO CInt 2) an AIOCB: data LioOps = LioRead | LioWrite | LioNop data AIOCB = AIOCB { aioFd :: Fd, aioLioOpcode :: Int, aioReqPrio :: Int, aioOffset :: FileOffset, aioBuf :: Ptr Word8, aioBytes :: ByteCount, aioSigevent :: Sigevent, -- Internal members aioNext :: Ptr AIOCB, aioAbsPrio :: Int, aioPolicy :: Int, aioErrorCode :: Int, aioReturnValue :: ByteCount } 3) poke/peek instance Storable AIOCB where sizeOf _ = (#const sizeof (struct aiocb)) alignment _ = 1 poke p_AIOCB (AIOCB aioFd aioLioOpcode aioReqPrio aioOffset aioBuf aioBytes aioSigevent aioNext aioAbsPrio aioPolicy aioErrorCode aioReturnValue) = do (#poke struct aiocb, aio_fildes) p_AIOCB aioFd (#poke struct aiocb, aio_lio_opcode) p_AIOCB aioLioOpcode (#poke struct aiocb, aio_reqprio) p_AIOCB aioReqPrio (#poke struct aiocb, aio_offset) p_AIOCB aioOffset (#poke struct aiocb, aio_buf) p_AIOCB aioBuf (#poke struct aiocb, aio_nbytes) p_AIOCB aioBytes (#poke struct aiocb, aio_sigevent) p_AIOCB aioSigevent (#poke struct aiocb, __next_prio) p_AIOCB aioNext (#poke struct aiocb, __abs_prio) p_AIOCB aioAbsPrio (#poke struct aiocb, __policy) p_AIOCB aioPolicy (#poke struct aiocb, __error_code) p_AIOCB aioErrorCode (#poke struct aiocb, __return_value) p_AIOCB aioReturnValue peek p_AIOCB = do aioFd <- (#peek struct aiocb, aio_fildes) p_AIOCB aioLioOpcode <- (#peek struct aiocb, aio_lio_opcode) p_AIOCB aioReqPrio <- (#peek struct aiocb, aio_reqprio) p_AIOCB aioOffset <- (#peek struct aiocb, aio_offset) p_AIOCB aioBuf <- (#peek struct aiocb, aio_buf) p_AIOCB aioBytes <- (#peek struct aiocb, aio_nbytes) p_AIOCB aioSigevent <- (#peek struct aiocb, aio_sigevent) p_AIOCB aioNext <- (#peek struct aiocb, __next_prio) p_AIOCB aioAbsPrio <- (#peek struct aiocb, __abs_prio) p_AIOCB aioPolicy <- (#peek struct aiocb, __policy) p_AIOCB aioErrorCode <- (#peek struct aiocb, __error_code) p_AIOCB putStrLn ("IN PEEK CODE .. aioErrorCode => " ++ (show aioErrorCode)) aioReturnValue <- (#peek struct aiocb, __return_value) p_AIOCB putStrLn ("IN PEEK CODE .. aioReturnValue => " ++ (show aioReturnValue)) return (AIOCB aioFd aioLioOpcode aioReqPrio aioOffset aioBuf aioBytes aioSigevent aioNext aioAbsPrio aioPolicy aioErrorCode aioReturnValue) Kind regards, Vasili On Sun, Jul 20, 2008 at 6:51 AM, Duncan Coutts wrote:

On Sat, 2008-07-19 at 23:55 -0500, Galchin, Vasili wrote:

...

yes Duncan I am trying to pass-by-value. I am familiar with ForeignPtr; however, I don't comprehend what you and Brandon are suggesting to do. Could either of you provide a code illustration or point at existing code to illustrate your approach?

Take a look at John Meacham's RSA example.

So at the moment you're using using Storable and a Haskell record, say:

data AIOCB = AIOCB { ... }

and we're suggesting instead:

newtype AIOCB = AIOCB (ForeignPtr AIOCB)

then to access a member use hsc2hs:

getBlah :: AIOCB -> IO Blah getBlah (AIOCB fptr) = withForeignPtr fptr $ \ptr -> {# peek aiocb,blah #} ptr

So you only access the parts you need and keep the aiocb C struct allocated on the heap (use mallocForeignPtr).

Duncan

Hi Brandon, So even if I go to ForeignPtr is a problem? And/Or is this a "by reference" vs "by value" issue? Kind regards, Vasili On Mon, Jul 28, 2008 at 1:09 AM, Brandon S. Allbery KF8NH < allbery@ece.cmu.edu> wrote:

On 2008 Jul 28, at 1:54, Galchin, Vasili wrote:

allocaBytes (#const sizeof(struct aiocb)) $ \ p_aiocb -> do poke p_aiocb aiocb

As I understand it, you can't do this;you must use the same aiocb throughout (that is, the same chunk of memory, not merely the same values; there is quite possibly a kernel mapping to it which you can't change or copy, or a pointer to the original aiocb is kept as internal state in the aio library so it can update the errno and result on the fly).

-- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH

Brandon, Your reading on my code is quite correct. So you are suggesting that the Haskell aiocb contain a ForeignPtr to actual aiocb that is passed the C functions. In this scenario, whose responsibility to allocate the "chunk" of memory for the aiocb? Vasili On Mon, Jul 28, 2008 at 2:04 AM, Brandon S. Allbery KF8NH < allbery@ece.cmu.edu> wrote:

On 2008 Jul 28, at 2:36, Galchin, Vasili wrote:

Hi Brandon,

So even if I go to ForeignPtr is a problem? And/Or is this a "by reference" vs "by value" issue?

As I read your code, you're allocating a C object, poking the Haskell fields into it, and passing it on, then peeking the values back out. This won't work; the C pointer value passed to aio_write() is the value that must be passed to subsequent operations on that aiocb (such as aio_return()). More to the point: the exact chunk of memory passed to aio_write(), unmodified, must be passed to any other aio functions checking for or blocking on completion of the write. You may not move it around or arbitrarily change values within it.

You could do this if your Haskell aiocb also retained the ForeignPtr to the originally allocated C object... but after the initial pokes, the only thing you can safely do with that object is pass it to C and peek the current values out of it, unless the C API specifically says you can modify fields within it while I/O operations are pending (and I'd be rather surprised if it did).

-- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH

On 2008 Jul 28, at 2:41, Galchin, Vasili wrote:

So based on what you are saying I kind of need a Haskell AIO "imperative"/monadic function that basically returns a handle that is associated with this AIOCB chunk-of-memory .... This handle gets passed around during an AIO I/O session?? Sorry for talking too "imperatively" ;^) <<< smiley face ;^)

I/O *is* monadic in Haskell, so you're kinda there anyway. I would in fact use a custom state (AIO = StateT ForeignPtr IO) if I were doing it; I would hide the constructor so that the only way to alter values is to call specific function(s) returning filled-in aiocbs as initialized and passed to aio_read/aio_write, then provide accessors for the contents (which if necessary can call aio_return, aio_suspend, or aio_error). Returning from the monad would invoke aio_suspend to wait for completion or aio_cancel to abort. (Hm. Could be argued that we want ContT here to represent the two possibilities.) I note from my local documentation that (a) indeed you must not modify the aiocb after passing it to aio_read/aio_write and (b) the offset value should not be read, much less modified, because it could change during the async I/O (and not in a reliably useful fashion; consider buffering). And as I said earlier, the exact same memory block (not merely a copy of it) must be used for the same aiocb. -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH

2008/7/28 Galchin, Vasili :

...

and we're suggesting instead:

newtype AIOCB = AIOCB (ForeignPtr AIOCB)

^^^ I am somewhat new to Haskell. Not a total newbie! But what exactly does the above mean? Are the three references of "AIOCB" in different namespaces?

The first and the third are the type AIOCB, the second is the type constructor AIOCB. That is, it is equivalent (up to renaming) to newtype T = C (ForeignPtr T) Now, why use Type in Type's definition? It is obvious that if we were creating data T = D T it would be pretty useless, however the type that ForeignPtr requires is just a phantom type. In other words, the ForeignPtr will never use the C constructor. An analogy to C: if you have typeA *pa; typeB *pb; then of course pa and pb have different types, however their internal representation are the same: an integral type of 32/64 bits. The C compiler only uses the type to provide warnings, to know the fields' offsets, the size of the structure, etc. The same goes for Haskell, if you have pa :: ForeignPtr A pb :: ForeignPtr B then both pa and pb have different types, but again they have the same internal representation. However, for example, if you allocate memory for pa via Storable then the compiler will find the correct sizeOf definition because will gave the type hint. The compiler also won't you let mix pa and pb like in [pa,pb]. So, if you declare newtype T = C (ForeignPtr T) you are: 1) Hiding the ForeignPtr from the users of your library if you don't export C. 2) Having type safeness by using ForeignPtr T instead of something generic like ForeignPtr () -- the same as using typeA* instead of void*. 3) Not needing to create a different type, like data InternalT = InternalT newtype T = C (ForeignPtr InternalT) Well.. did it help at all? =) -- Felipe.

what does a datatype with no constructors mean? E.g. data RSAStruct data EVP_PKEY data EVP_CIPHER data EVP_CIPHER_CTX data EVP_MD_CTX data EVP_MD data BIGNUM On Mon, Jul 28, 2008 at 8:01 PM, Felipe Lessa wrote:

2008/7/28 Galchin, Vasili :

...

...

and we're suggesting instead:

newtype AIOCB = AIOCB (ForeignPtr AIOCB)

^^^ I am somewhat new to Haskell. Not a total newbie! But what exactly does the above mean? Are the three references of "AIOCB" in different namespaces?

The first and the third are the type AIOCB, the second is the type constructor AIOCB. That is, it is equivalent (up to renaming) to

newtype T = C (ForeignPtr T)

Now, why use Type in Type's definition? It is obvious that if we were creating

data T = D T

it would be pretty useless, however the type that ForeignPtr requires is just a phantom type. In other words, the ForeignPtr will never use the C constructor.

An analogy to C: if you have

typeA *pa; typeB *pb;

then of course pa and pb have different types, however their internal representation are the same: an integral type of 32/64 bits. The C compiler only uses the type to provide warnings, to know the fields' offsets, the size of the structure, etc. The same goes for Haskell, if you have

pa :: ForeignPtr A pb :: ForeignPtr B

then both pa and pb have different types, but again they have the same internal representation. However, for example, if you allocate memory for pa via Storable then the compiler will find the correct sizeOf definition because will gave the type hint. The compiler also won't you let mix pa and pb like in [pa,pb].

So, if you declare

newtype T = C (ForeignPtr T)

you are:

1) Hiding the ForeignPtr from the users of your library if you don't export C. 2) Having type safeness by using ForeignPtr T instead of something generic like ForeignPtr () -- the same as using typeA* instead of void*. 3) Not needing to create a different type, like

data InternalT = InternalT newtype T = C (ForeignPtr InternalT)

Well.. did it help at all? =)

-- Felipe.

ok guys .. what is this "phantom type" concept? Is it a type theory thing or just Haskell type concept? Vasili On Mon, Jul 28, 2008 at 8:53 PM, Bryan Donlan wrote:

On Mon, Jul 28, 2008 at 08:48:23PM -0500, Galchin, Vasili wrote:

...

what does a datatype with no constructors mean?

E.g.

data RSAStruct data EVP_PKEY data EVP_CIPHER data EVP_CIPHER_CTX data EVP_MD_CTX data EVP_MD data BIGNUM

It's simply a datatype that can never have a value - a so-called 'phantom type'. They're useful when you need a type (eg as the argument for a ForeignPointer) but no need for an actual value.

You can of course create values of these types using 'undefined', 'error' and friends, but this is perhaps not very useful most of the time :)

"Galchin, Vasili" writes:

ok guys .. what is this "phantom type" concept? Is it a type theory thing or just Haskell type concept?

Here's another example. Say you want to use bytestrings with different encodings. You obviously don't want to concatenate a string representing Latin characters with a string in Cyrillic. One way to do this, is to define phantom types for the encodings, and a bytestring type that takes additional type parameter data KOI8 data ISO8859_1 : data Bytestring enc = MkBS ... Operations like concat work on same-typed bytestrings: concat :: Bytestring e -> Bytestring e -> Bytestring e The parameter (enc) isn't used on the right hand side, so all Bytestrings will have the same representation, but Bytestring KOI8 and Bytestring ISO8859_1 will have different types, so although the runtime won't know the difference, trying to 'concat' them will give you a type error at compile time. -k -- If I haven't seen further, it is by standing in the footprints of giants

On Sat, 2008-07-19 at 10:45 -0400, Brandon S. Allbery KF8NH wrote:

On 2008 Jul 19, at 2:40, Galchin, Vasili wrote:

...

My viewpoint is that the above "Internal members" must be "carried" around in a Haskell program. Am I correct?? If I am correct, then the Linux implementation of Posix AIO is not portable to say Solaris? In hindsight, if I am correct, it seems that

You are correct --- but Solaris also has its own addenda, and its standard fields are not at the same offsets as in the Linux aiocb. The only safe way to do this is to use an opaque aiocb on the Haskell side and accessors in C via FFI.

You can do field accessors using an FFI pre-processor like c2hs or hsc2hs which will calculate the correct field offsets for the current platform. No need for C wrappers. Duncan

On 2008 Jul 19, at 16:42, Galchin, Vasili wrote:

You lost me .... can you explain what an opaque aiocb would look like?

In Haskell, it's a ForeignPtr --- you can't see "inside" it except by FFI calls. Although Duncan is correct and you can use an FFI preprocessor to access the portable fields, and simply not provide access to the rest from Haskell. -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH

I think it may be time for a little clarity about aoicb's. From the Single Unix Specification: "The header shall define the aiocb structure which shall include AT LEAST the following members: int aio_fildes File descriptor. off_t aio_offset File offset. volatile void *aio_buf Location of buffer. size_t aio_nbytes Length of transfer. int aio_reqprio Request priority offset. struct sigevent aio_sigevent Signal number and value. int aio_lio_opcode Operation to be performed. " The "AT LEAST" here means that - a portable program may rely on these members being present - a portable program MUST assume that an unknown number of additional members are also present - a portable program may freely copy such a record, but may only pass it to a library function if that function is expecting to initialise it For asynchronous I/O, this means that - you can allocate an aiocb object - an aiocb passed to aio_suspend, aio_error, aio_return, or aio_cancel should have been filled in by aio_read or aio_write and should be EXACTLY THE SAME object, not a copy of it.

On Mon, Jul 21, 2008 at 01:05:48PM +1200, Richard A. O'Keefe wrote:

I think it may be time for a little clarity about aoicb's. From the Single Unix Specification: "The header shall define the aiocb structure which shall include AT LEAST the following members: int aio_fildes File descriptor. off_t aio_offset File offset. volatile void *aio_buf Location of buffer. size_t aio_nbytes Length of transfer. int aio_reqprio Request priority offset. struct sigevent aio_sigevent Signal number and value. int aio_lio_opcode Operation to be performed. " The "AT LEAST" here means that - a portable program may rely on these members being present - a portable program MUST assume that an unknown number of additional members are also present - a portable program may freely copy such a record, but may only pass it to a library function if that function is expecting to initialise it

For asynchronous I/O, this means that - you can allocate an aiocb object - an aiocb passed to aio_suspend, aio_error, aio_return, or aio_cancel should have been filled in by aio_read or aio_write and should be EXACTLY THE SAME object, not a copy of it.

Yes. This is pretty standard as far as what you can count on in terms of standard C structures. The method I use in the RSA.hsc module I posted is compatible with these assumptions. Generally this is pretty much exactly the thing hsc2hs was made to solve. Just a note, if you are doing manual explicit frees of the aiocb structure then you only need a 'Ptr', if you want the structure to be automatically garbage collected when all haskell references to it disappear, then you need to use 'ForeignPtr'. John -- John Meacham - ⑆repetae.net⑆john⑈