Hello Simon, Friday, February 10, 2006, 3:26:30 PM, you wrote:
as i understand this idea, transformer implementing async i/o should intercept vGetBuf/vPutBuf calls for the FDs, start the appropriate
type FD = Int vGetBuf_async :: FD -> Ptr a -> Int -> IO Int vPutBuf_async :: FD -> Ptr a -> Int -> IO Int
EK> Please don't fix FD = Int, this is not true on some systems, EK> and when implementing efficient sockets one usually wants EK> to hold more complex state.
the heart of the library is class Stream. both "File" and "Socket" should implement this interface. just now i use plain "FD" to represent files, but that is temporary solution - really file also must carry additional information: filename, open mode, open/closed state. This "File" will be an abstract datatype, what can be based not on FD in other operating systems.
The same applies to the "Socket". it can be any type what carry enough information to work with network i/o.
implementation of async i/o should have a form of Stream Transformer, which intercepts only the vGetBuf/vPutBuf operations and pass other operations as is:
SM> I don't think async I/O is a stream transformer, fitting it into the SM> stream hierarchy seems artificial to me. yes, it is possible - what i'm trying to implement everything as tranformer, independent of real necessity. i really thinks that idea of transformers fit every need in extending functionality it is a list of my reasons to implement this as transformer: 1) there is no "common FD" interface. module System.FD implements something, but it is a really interface only for file i/o. it's used partially in System.MMFile, implementing memory-mapped files, and i think these fd* operations will be used to partially implement Socket operations, but something will be different, including using recv/send instead of read/write to implement GetBuf/PutBuf operations. so, there is no common "instance Stream FD", but different instances for files, memory-mapped files and sockets. As Einar just mentioned, Socket dataype will include information what absent in File datatype. So, these 3 types have in common using FD to implement some of its operations, but some operations will be different and internal dataype structures will be different. Transformer is an ideal way to just reimplement vGetBuf/vPutBuf operations while passing through all the rest. Without it, instead of 3 methods of doing I/O (mmap/read/recv) you will need to implement all the 5 (mmap/read/recv/readAsync/recvAsync) - it's even without counting selct, epoll and kqueue separately 2) as you can see in epoll()-based implementation of async i/o in alt-network library, Einar attaches additional data (read/write queues) to the FD to support epoll() interface. These data will be different for select, epoll, kqueue and other methods of async i/o. At least, without async i/o no information should be needed. Transformer is an ideal way to attach additional data to the file/socket without changing of "raw" datatype. Again, otherwise you will need to attach all these data to the raw file, duplicate this work with the raw socket and then repeat this for select, epoll and other async i/o methods on the other side, reasons for your proposal, as i see: 1) if FD will incorporate async i/o support, the System.FD library will become much more useful - anyone using low-level fd* functions will get async i/o support for free but there is another defeciency in the System.FD library - it doesn't include support for the files>4Gb and files with unicode filenames under Windows. it seems natural to include this support in fd* too. now let's see. you are proposing to include in fd* implementation support for files, sockets, various async i/o methods and what's not all. are you not think that this library will become a successor of Handle library, implementing all possible fucntionality and don't giving 3rd-party libraries chances to change anything partially? i propose instead to divide library into the small manageable pieces what can be easily stidied/modified/replaced and that brings something really usefull only when used together. if what means that low-level fd* interface can't be used even to work with raw files without great restrictions (no Unicode filenames in windows, no async i/o) then it will mean just this. SM> It is just another way of doing I/O directly to/from file descriptors. SM> If your basic operation to read from an FD is SM> readFD :: FD -> Int -> Ptr Word8 -> IO Int SM> then an async I/O layer simply provides you with the exact same SM> interface, but with an implementation that doesn't block other threads. SM> It is part of the file descriptor interface, not a stream transformer. SM> Also, you probably need SM> readNonBlockingFD :: FD -> Int -> Ptr Word8 -> IO Int SM> isReadyFD :: FD -> IO Bool SM> in fact, I think this should be the basic API, since you can implement SM> readFD in terms of it. (readNonBlockingFD always reads at least one SM> byte, blocking until some data is available). This is used to partially SM> fill an input buffer with the available data, for example. this can be in basic API, but not in basic implementation :))) really, i think that you mix two things - readNonBlockingFD call that can fill buffer only partially and readAsync call that use some I/O manager to perform other Haskell threads while data are read well, i agree that should be two GetBuf variants in the Stream interface - greedy and non-greedy. say, vGetBuf and vGetBufNonBlocking. vPutBuf also need two variants? then, may be LineBuffering and BlockBuffering should use vGetBufNonBlocking and vGetBuf, respectively? but i don't know anything about implementation. is the difference between readNonBlockingFD and readFD calls only in the O_NONBLOCK mode of file handle, or different functions are used? what for Windows? for sockets? how this interacts with the async i/o? SM> One problem here is that in order to implement readNonBlockingFD on Unix SM> you have to put the FD into O_NONBLOCK mode, which due to misdesign of SM> the Unix API affects other users of the same file descriptor, including SM> other programs. GHC suffers from this problem. what means that it is better to decide at "open" stage whether this file will be used with readNonBlockingFD or with simple readFD? -- Best regards, Bulat mailto:bulatz@HotPOP.com