August 2001 - Libraries

Re: I Hate IO
by Marcin 'Qrczak' Kowalczyk 13 Aug '01

13 Aug '01

Fri, 10 Aug 2001 18:31:24 -0700, Ashley Yakeley <ashley(a)semantic.org> pisze: > I can only tell you what's useful for me as an API user. Since there are > peculiar little differences between different kinds of sockets, my ideal > would be every type of socket to have its own API functions, and then > common patterns brought together with classes. It makes harder to program polymorphically. In particular having a heterogeneous datastructure requires wrapping different types in either function closures or existential types, thus awkwardly emulating the current interface. Items which are used interchangeably shouldn't have different types in a language like Haskell which doesn't provide subtyping. Too bad that one can't make his own object with type Handle and arbitrary behavior. It means that even Handle is a type too much tied to the implementation. > But at the very least, separate out files from sockets. I'm really > not sure why files have a "file pointer" stream-based API at all. Because it's convenient. Files are usually read sequentially. Would you separate terminals from these? Terminals are sequential by nature and they are used interchangeably with files. How would you express this in your interface? f <- if filename == "-" then return stdin else openFile filename ReadMode -- __("< Marcin Kowalczyk * qrczak(a)knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK

1 0

RE: I Hate IO
by Ashley Yakeley 11 Aug '01

11 Aug '01

At 2001-08-10 06:26, Simon Marlow wrote: >There's lots of useful shared functionality between file I/O and socket >I/O. For example: buffering/flushing and lazy I/O (getContents) are >shared for files and sockets in GHC, I wouldn't want to have to provide >N separate implementations of these things for the various different >types of streams and files. I can only tell you what's useful for me as an API user. Since there are peculiar little differences between different kinds of sockets, my ideal would be every type of socket to have its own API functions, and then common patterns brought together with classes. But at the very least, separate out files from sockets. I'm really not sure why files have a "file pointer" stream-based API at all. It seems very forced to me, but I guess everyone is so used to it from C/Unix etc. that it seems natural. ... >Does doing I/O in anything other than the IO monad make sense? Probably not. But Sources and Sinks are useful general-purpose patterns. >Why isn't writeEnd the same as close? On a TCP connection, one can still receive data after sending an end. My "close" is intended to be the last thing done with the connection as an object destructor: if called before sending and receiving end, it would simply forcibly close the connection. >Is there anything that is always a Source but not a Sink (or vice >versa)? Oh yes, lots of things. Standard input/output/error for instance. One might write code that did something like pipe from a Source to a Sink, and then one might want a Source constructed from a constant array. Or something. Stream-based encoding algorithms such as UTF-8 might make use of them: encodeUTF8:: SourceChar -> SourceWord8 etc. > Why separate these two classes? I think they are cleanly conceptually separable. I don't see a loss: by making TCPSocket an instance of both, the API user still has only one object to deal with. -- Ashley Yakeley, Seattle WA

1 0

RE: I Hate IO
by Simon Marlow 10 Aug '01

10 Aug '01

> -- > type TCPPortNumber = Word16 > type IPAddress = Word32 > data TCPPort = MkTCPPort IPAddress TCPPortNumber > data TCPSocket = ... > > openTCP :: TCPPort -> IO TCPSocket > closeTCP :: TCPSocket -> IO () > listenTCP :: TCPPortNumber -> (TCPConnection -> IO ()) -> IO () > -- not sure about listenTCP > > getRemoteAddressTCP :: TCPSocket -> IO TCPPort > getLocalAddressTCP :: TCPSocket -> IO TCPPort > > receiveTCP :: TCPSocket -> Integer -> Integer -> IO (Maybe [Word8]) > -- "receiveTCP socket waittime arraylength" > -- return "Nothing" if received end with no octets pending > -- return "Just array" where array is as many octets up to length as > received by waittime > -- set waittime to 0 to not wait but return only available bytes in > buffer. > > receiveWaitForeverTCP :: TCPSocket -> Integer -> IO (Maybe [Word8]) > -- Same, with infinite wait-time. > -- only useful if it can be interrupted by another thread > > sendDataTCP :: TCPSocket -> [Word8] -> IO () > sendEndTCP :: TCPSocket -> IO () > -- > > OK, this is just off-the-cuff and probably has many flaws, > and I offer it > merely for discussion, not as a proposal. It does, however, > represent the > kind of basic functionality I'd look for, or abstract, if > writing a TCP > application. There's lots of useful shared functionality between file I/O and socket I/O. For example: buffering/flushing and lazy I/O (getContents) are shared for files and sockets in GHC, I wouldn't want to have to provide N separate implementations of these things for the various different types of streams and files. But there is a big difference when it comes to seeking, which is why we have hIsSeekable. I guess it just comes down to how much there is in common between I/O on files and I/O on streams. I think I like using the same API for both. But it might be nice to abstract the interface a little, something like class Seekable h where hSeek :: h -> SeekMode -> Integer -> IO () class IO h where hPutStr hGetLine hGetContents ... instance Seekable FileHandle where { ... } instance IO FileHandle where { ... } instance IO Stream where { ... } > If programmers want to abstract away the differences between > different > types of network connections, well, this is Haskell so they > can always > write classes like this: > > class (Monad m) => Closable s m | s -> m where > close :: s -> m () > > class (Monad m) => Source s m d | s -> m d where > read :: s -> Integer -> Integer -> m (Maybe [d]) > readWaitForever :: s -> Integer -> m (Maybe [d]) > > class (Monad m) => Sink s m d | s -> m d where > writeData :: s -> [d] -> m () > writeEnd :: s -> m () > > write s Nothing = writeEnd s > write s (Just d) = writeData s d Does doing I/O in anything other than the IO monad make sense? Why isn't writeEnd the same as close? Is there anything that is always a Source but not a Sink (or vice versa)? Why separate these two classes? Cheers, SImon

1 0

RE: I Hate IO
by Ashley Yakeley 10 Aug '01

10 Aug '01

At 2001-08-09 03:01, Simon Marlow wrote: >I considered providing a different API for bidirectional streams, or >perhaps requiring that bidirectional streams use separate Handles for >read and write, but came to the conclusion that the user really doesn't >care whether under the hood a single Handle is using separate buffers >for read and write or just a single buffer, how much locking is going on >or whatever. The fact that these things are awkward to implement >shouldn't show through in the library interface. > >It's definitely more convenient from the programmer's point of view to >be able to use the *same* handle object for both read and write, >otherwise you have to explain to people why they can have a read/write >file handle but not a read/write handle for a TCP socket. Well, how about something like this: -- type TCPPortNumber = Word16 type IPAddress = Word32 data TCPPort = MkTCPPort IPAddress TCPPortNumber data TCPSocket = ... openTCP :: TCPPort -> IO TCPSocket closeTCP :: TCPSocket -> IO () listenTCP :: TCPPortNumber -> (TCPConnection -> IO ()) -> IO () -- not sure about listenTCP getRemoteAddressTCP :: TCPSocket -> IO TCPPort getLocalAddressTCP :: TCPSocket -> IO TCPPort receiveTCP :: TCPSocket -> Integer -> Integer -> IO (Maybe [Word8]) -- "receiveTCP socket waittime arraylength" -- return "Nothing" if received end with no octets pending -- return "Just array" where array is as many octets up to length as received by waittime -- set waittime to 0 to not wait but return only available bytes in buffer. receiveWaitForeverTCP :: TCPSocket -> Integer -> IO (Maybe [Word8]) -- Same, with infinite wait-time. -- only useful if it can be interrupted by another thread sendDataTCP :: TCPSocket -> [Word8] -> IO () sendEndTCP :: TCPSocket -> IO () -- OK, this is just off-the-cuff and probably has many flaws, and I offer it merely for discussion, not as a proposal. It does, however, represent the kind of basic functionality I'd look for, or abstract, if writing a TCP application. If programmers want to abstract away the differences between different types of network connections, well, this is Haskell so they can always write classes like this: -- class (Monad m) => Closable s m | s -> m where close :: s -> m () class (Monad m) => Source s m d | s -> m d where read :: s -> Integer -> Integer -> m (Maybe [d]) readWaitForever :: s -> Integer -> m (Maybe [d]) class (Monad m) => Sink s m d | s -> m d where writeData :: s -> [d] -> m () writeEnd :: s -> m () write s Nothing = writeEnd s write s (Just d) = writeData s d -- etc. -- Ashley Yakeley, Seattle WA

1 0

System.Directory & Posix
by Simon Marlow 09 Aug '01

09 Aug '01

Hi Folks, We currently have the Haskell 98 Directory library as System.IO.Directory in the new hierarchy. I'd prefer to move it to System.Directory if no-one has any objections. I'm currently looking at the Posix libraries. There were two suggested placings for the Posix libraries: System Posix IO Process Files System IO Posix Directory Posix Process Posix I'd go with the second layout on the grounds that it follows the convention of organising libraries by functionality. Comments? There are also various generic Posix bits (eg. types) that don't have a good home. I'm currently planning to put these in System.Posix. Cheers, Simon

1 0

RE: I Hate IO
by Ashley Yakeley 09 Aug '01

09 Aug '01

At 2001-08-09 03:53, Ashley Yakeley wrote: >-- >data OpenFile = ... > >openFileReadOnly :: FilePath -> IO OpenFile >openFileReadWrite :: FilePath -> IO OpenFile >closeFile :: OpenFile -> IO () > >fileWritable :: OpenFile -> IO Bool > >fileLength :: OpenFile -> IO Integer > >setFileLength :: OpenFile -> Integer -> IO () >-- fill with zeros if extending > >readFileBlock :: OpenFile -> Integer -> Integer -> IO [Word8] >-- "readFileBlock file start length", return truncated array if past file >end > >writeFileBlock :: OpenFile -> Integer -> [Word8] -> IO () >-- would extend file if writing past end >-- Note that any number of threads could be permitted to call 'read' operations (fileLength,readFileBlock) simultaneously, but if a thread is calling a 'write' operation (setFileLength,writeFileBlock), other threads must wait on both read and write operations. It's _so_ much simpler when you don't have that pesky "file pointer" to worry about. -- Ashley Yakeley, Seattle WA

1 0

RE: I Hate IO
by Ashley Yakeley 09 Aug '01

09 Aug '01

At 2001-08-09 03:01, Simon Marlow wrote: >Ok, I agree so far. Are you suggesting the IO library should be >changed? Yes. >How? Hmm... well the file part to use Word8s and not be stream-based. Perhaps something like this: -- data OpenFile = ... openFileReadOnly :: FilePath -> IO OpenFile openFileReadWrite :: FilePath -> IO OpenFile closeFile :: OpenFile -> IO () fileWritable :: OpenFile -> IO Bool fileLength :: OpenFile -> IO Integer setFileLength :: OpenFile -> Integer -> IO () -- fill with zeros if extending readFileBlock :: OpenFile -> Integer -> Integer -> IO [Word8] -- "readFileBlock file start length", return truncated array if past file end writeFileBlock :: OpenFile -> Integer -> [Word8] -> IO () -- would extend file if writing past end -- This covers ReadMode and ReadWriteMode. I don't really understand WriteMode, but depending on what operations that implies, that could be included too. AppendMode is by nature a weird sink-based thing and would have a separate interface: -- data AppendableFile = ... openFileAppendable :: FilePath -> IO AppendableFile closeFileAppendable :: AppendableFile -> IO () appendToFile :: AppendableFile -> [Word8] -> IO (); -- Of course, on some implementations the AppendableFile functions could be implemented with the OpenFile functions. In any case, network functions would not use the same datatypes at all. If you want to provide functions that work with different things that follow a single pattern, use classes. -- Ashley Yakeley, Seattle WA

1 0

RE: I Hate IO
by Steinitz, Dominic J 09 Aug '01

09 Aug '01

You definitely need to get to the octets when implementing communications protocols. Does this mean that if I send or receive certain octets or sequences of octets these will be converted into something else? Dominic. > 2. A file is not made of "Char"s. A file is made of octets ("bytes"), > i.e. Word8s. What is a "Char" anyway? Sometimes it's a seven- or > eight-bit quantity with a _vague_ implication of interpretation as > textual character; sometimes it's a 16-, 20.087- or 31-bit > quantity with > a much stronger implication of interpretation as textual character > (strictly, Unicode "codepoint"). Is an ASCII 'r' the same as > an EBCDIC > 'r'? Or is an ASCII code 57 the same as an EBCDIC code 57? > > As for streams, mostly they are streams of octets. But of > course streams > of anything might be useful. There's an implicit conversion step, between whatever is the on-disk encoding of character streams and Unicode. GHC currently only supports a straightforward ISO 8851 encoding. I agree there ought to be a way to get at the raw bytes too. ------------------------------------------------------------------------------------------------- 21st century air travel http://www.britishairways.com

1 0

RE: I Hate IO
by Simon Marlow 09 Aug '01

09 Aug '01

> 1. A file is not a stream. It really isn't anything like a > stream. Sure, > you can _make_ a stream based on a file but that's a > different thing. A > file is a list (ignoring for the moment meta-information), > accessible at > any point. By contrast, streams access either incoming or outgoing > entities, optionally with "end of stream" support. For > incoming, one may > 'skip' but not 'seek'. For outgoing, one may send a series of > predefined > 'zero' values. Call that "seek" if you want, I don't. I couldn't agree more. I came across exactly these issues while rewriting GHC's I/O library recently, so now GHC's Handle type internally has two constructors: - FileHandle (a handle to a file, seekable, with a single file pointer and a single buffer that contains either pending read or write data but not both). - DuplexHandle (a read/write stream, not seekable, with two completely independent buffers, and the two ends can be closed independently). strangely, a FileHandle is also used for a uni-directional stream, because it only needs a single buffer. For Concurrent Haskell there's a locking issue here: you can't expect two threads to read and write simultaneously to the same file, but it is entirely reasonable for two threads to be simultaneously reading and writing on the same socket. Hence a FileHandle has a single lock, and a DuplexHandle has one lock for each channel. In effect there's one lock per buffer. > 2. A file is not made of "Char"s. A file is made of octets ("bytes"), > i.e. Word8s. What is a "Char" anyway? Sometimes it's a seven- or > eight-bit quantity with a _vague_ implication of interpretation as > textual character; sometimes it's a 16-, 20.087- or 31-bit > quantity with > a much stronger implication of interpretation as textual character > (strictly, Unicode "codepoint"). Is an ASCII 'r' the same as > an EBCDIC > 'r'? Or is an ASCII code 57 the same as an EBCDIC code 57? > > As for streams, mostly they are streams of octets. But of > course streams > of anything might be useful. There's an implicit conversion step, between whatever is the on-disk encoding of character streams and Unicode. GHC currently only supports a straightforward ISO 8851 encoding. I agree there ought to be a way to get at the raw bytes too. > 3. Output streams ("sinks") are different from input streams > ("sources"). > That POSIX entity known as "standard output" is a sink of octets. A > "bi-directional" stream such as a TCP connection is nothing > but a source > and a sink considered together. Indeed, for TCP it's possible to send > "end of [outgoing] stream" without affecting the incoming > stream. This is > rather different from a contrived "file-access"-type stream, where > reading and writing are operations affect each other. Yup, see above. > There is no such thing as "I/O" unless, as in Haskell, one > means _all_ > imperative action. There are various entities out in the > world accessible > in a variety of different ways. Sources, sinks, lists, etc. > are but some > of the models useful for accessing them. Ok, I agree so far. Are you suggesting the IO library should be changed? How? I considered providing a different API for bidirectional streams, or perhaps requiring that bidirectional streams use separate Handles for read and write, but came to the conclusion that the user really doesn't care whether under the hood a single Handle is using separate buffers for read and write or just a single buffer, how much locking is going on or whatever. The fact that these things are awkward to implement shouldn't show through in the library interface. It's definitely more convenient from the programmer's point of view to be able to use the *same* handle object for both read and write, otherwise you have to explain to people why they can have a read/write file handle but not a read/write handle for a TCP socket. Cheers, Simon

1 0

I Hate IO
by Ashley Yakeley 09 Aug '01

09 Aug '01

1. A file is not a stream. It really isn't anything like a stream. Sure, you can _make_ a stream based on a file but that's a different thing. A file is a list (ignoring for the moment meta-information), accessible at any point. By contrast, streams access either incoming or outgoing entities, optionally with "end of stream" support. For incoming, one may 'skip' but not 'seek'. For outgoing, one may send a series of predefined 'zero' values. Call that "seek" if you want, I don't. 2. A file is not made of "Char"s. A file is made of octets ("bytes"), i.e. Word8s. What is a "Char" anyway? Sometimes it's a seven- or eight-bit quantity with a _vague_ implication of interpretation as textual character; sometimes it's a 16-, 20.087- or 31-bit quantity with a much stronger implication of interpretation as textual character (strictly, Unicode "codepoint"). Is an ASCII 'r' the same as an EBCDIC 'r'? Or is an ASCII code 57 the same as an EBCDIC code 57? As for streams, mostly they are streams of octets. But of course streams of anything might be useful. 3. Output streams ("sinks") are different from input streams ("sources"). That POSIX entity known as "standard output" is a sink of octets. A "bi-directional" stream such as a TCP connection is nothing but a source and a sink considered together. Indeed, for TCP it's possible to send "end of [outgoing] stream" without affecting the incoming stream. This is rather different from a contrived "file-access"-type stream, where reading and writing are operations affect each other. There is no such thing as "I/O" unless, as in Haskell, one means _all_ imperative action. There are various entities out in the world accessible in a variety of different ways. Sources, sinks, lists, etc. are but some of the models useful for accessing them. -- Ashley Yakeley, Seattle WA

1 0