The other day I was reading the Haskell i18n debate in the list archives, and started thinking about possible replacements for the existing Haskell file I/O model. It occurred to me that the Haskell community has really dropped the ball on this one. Haskell's design has always emphasized doing the right thing, not merely doing the thing that everyone else happens to be doing. It's that philosophy that led to the invention of the monadic I/O model, among other things. And yet, what do we choose for our I/O primitives? The same old crocks that everyone else was using. We open and close files (whatever that's supposed to mean); we expose file handles to the user; we even maintain a current position in the file, which is an unnecessary global state variable if I've ever seen one. The proposal below is the result of a few hours spent thinking about how the file system would be accessed if it were actually implemented in Haskell, instead of behind a weird C API. I'm very interested in hearing comments and criticism. In particular, I want to know if there's enough interest in this model that I should actually try to implement it. The most important idea in this design as far as i18n is concerned is the separation of random-access files from input and output streams. Most of the ugliness of the usual file I/O interface comes from conflating these three concepts, which are almost totally unrelated. In particular, there's no need in this model to worry about the meaning of reading or seeking in a "text file." Text encoding and decoding apply to streams, not files. To read text from a file you layer an input stream on it, apply a text parser to that, and read characters. If you need to "seek" to a new location, you create a new stream which starts at that location in the underlying file.
module System.ProposedNewIOModel (...) where
I assume that all I/O occurs in terms of octets. I think that this holds true of every platform on which Haskell is implemented or is likely to be implemented.
type Octet = Word8
File offsets are 64 bits on all platforms. This model never uses negative offsets, so there's no need for a signed type. (But perhaps it would be better to use one anyway?) BlockLength should be something appropriate to the architecture's address space.
type FilePos = Word64 type BlockLength = Int
A value of type File represents a file, which is essentially a resizable strict array of octets. Two values of type File compare equal if they are the same file -- that is, if they have the same contents and changes to one also appear in the other. ("File" is a bad name for this. For one thing, NTFS and HFS can associate more than one chunk of data with each directory entry, and "file" usually refers to all the chunks together. "Fork" would be more accurate, but it doesn't sound much like what it's supposed to represent.) (Should files be buffered at the application level? I'm not convinced it's necessary. Streams will be buffered, of course.)
data File -- abstract
A value of type InputStream or OutputStream represents an input or output stream: that is, an octet source or sink. Two InputStreams or OutputStreams compare equal iff reading/writing one also reads/writes the other. (Should I call these "ports" instead of "streams"? How about "OctetSource" and "OctetSink"?)
data InputStream -- abstract data OutputStream -- abstract
Fundamental operations on files. XXX represents some sort of memory buffer (an IOUArray or a Ptr?). All reads and writes supply absolute offsets. (Note that there's no such thing as reading fewer bytes than requested -- the read either succeeds or it throws an exception. This may be problematic because the file size could decrease between a call to fGetSize and a later call to fRead. One solution is to make fRead treat bytes beyond EOF as zeroes, which is consistent with the usual behavior for sparse files. fWrite should probably expand the file automatically if asked to write beyond its bounds, for similar reasons.) (A serious unaddressed problem here is locking. There should be a way to e.g. prevent someone else from writing to a file while I'm appending to it, to avoid race conditions. But for whatever reason, the usual OS interface only allows this kind of lock to be acquired when opening the file -- and I can't generally reopen the file because its name might have changed. Meanwhile, another kind of lock, the kind I don't want, can be acquired and released on an already-opened file. I don't know why it's done this way, or how to work around it.) (And what about asynchronous I/O?)
fGetSize :: File -> IO FilePos fSetSize :: File -> FilePos -> IO () fRead :: File -> FilePos -> BlockLength -> XXX -> IO () fWrite :: File -> FilePos -> BlockLength -> XXX -> IO () fCheckRead :: File -> FilePos -> BlockLength -> IO Bool fCheckWrite :: File -> FilePos -> BlockLength -> IO Bool
Fundamental operations on streams. "Maybe Octet" is supposed to represent "Octet or EOS," though I'm not sure this is enough for proper EOS handling. isPeek might be useful for text parsers. isUnGet is another possibility -- it's more versatile but harder to implement efficiently. Or this may be a moot point because text parsers will probably want to use isGetBlock anyway for efficiency.
isGet :: InputStream -> IO (Maybe Octet) isPeek :: InputStream -> IO (Maybe Octet) isGetBlock :: InputStream -> BlockLength -> XXX -> IO BlockLength -- efficiency hack
osPut :: OutputStream -> Octet -> IO () osPuts :: OutputStream -> [Octet] -> IO () osPutBlock :: OutputStream -> BlockLength -> XXX -> IO () osFlush :: OutputStream -> IO ()
Standard streams. (How to deal with line buffering?)
stdin :: InputStream stdout, stderr :: OutputStream
Streams can be layered on top of files. Each stream keeps track of its own independent position within the file. The effects of overlapped stream reading and writing on the same part of a file are unspecified, since streams will be buffered in practice.
fileToInputStreamFrom :: File -> FilePos -> (IO?) InputStream fileToOutputStreamFrom :: File -> FilePos -> (IO?) OutputStream
fileToInputStream f = fileToInputStreamFrom f 0 fileToOutputStreamAppend f = fGetSize f >>= fileToOutputStreamFrom f
InputStreams can be read as lazy lists. You're not allowed to use the InputStream again after you call this.
isGetContents :: InputStream -> IO [Octet]
A value of type Directory represents a directory, which is essentially a mutable associative array from names (Unicode strings) to File/Directory/Stream values plus some metadata. This is supposed to represent a directory *independently of its name*, so it's robust against changes at higher levels of the filesystem hierarchy. I don't know if this can be implemented on Posix or Win32. (The old Mac OS and the NT API do support it.)
data Directory -- abstract
Operations on directories. I suggest dGetContentsWithMetadata rather than dLookupMetadata because using the latter to scan a directory on a FAT32 filesystem takes O(n^2) time.
dGetContents :: Directory -> IO [String] dGetContentsWithMetadata :: Directory -> IO [(String,???)] dLookup :: Directory -> String -> IO (Either File Directory) -- Also should handle named pipes here dCreateFile :: Directory -> String -> (metadata?) -> IO File -- etc...
Pathnames don't fit well with this model, but they're not going to go away, so I provide these functions. If the pathname argument names a directory, that directory and Nothing are returned. Otherwise, a directory and string are returned such that a subsequent call to dLookup or dCreateFile with these values will refer to the same thing as the original pathname. lookupPathname is dLookupPathname from the current directory.
dLookupPathname :: Directory -> String -> IO (Directory, Maybe String) lookupPathname :: String -> IO (Directory, Maybe String) getCurrentDirectory :: IO Directory -- no corresponding set!
Convenient shortcuts for common cases.
lookupFileByPathname :: String -> IO File lookupInputStreamByPathname :: String -> IO InputStream -- at least as likely to succeed as lookupFileByPathname
-- Ben
Ben, I live in a different universe, but over here I prefer to represent files purely as memory-mapped objects. In this view, there is no difference between a read-only file and an immutable array of bytes (a byte being a natural number between 0 to 255). A read-write file is then equivalant to a mutable array (or a reference to a mutable array on a heap) of the same. Treating these all as heap references tends to be cleaner, because you can compare the references for equality, which is significant even for read-only files, because two files which contain the same exact data are not necessarily the same file, whereas opening the same file in two different places should result in equal references. This approach greatly simplifies lots of things now that all modern operating systems can perform file-mapping efficiently with the virtual memory subsystem paging pieces in and out as necessary. It gets rid of pieces of information which are redundent from the low-level file system point of view (the file handle itself, the current file pointer, etc). The typical C/Unix approach is to deal with network (TCP or UDP) connections as streams, too. Obviously, memory-mapped files aren't a good way of exposing them -- doing so would require buffering all past data, as well as blocking when waiting on yet-unreceived data (when you really want to be able to query whether there is incoming data available). Instead, I prefer to conceptualize network connections as a socket / packet-based interface, with functions to open/close sockets, send a complete packet (being an array of bytes) to a socket, receive a packet from a socket, and query packet availability. With this approach, there is no redundency or missing information; everything that is observable from the protocol point of view is an observable in the language interface, and nothing more. In this manner, it's possible to get rid of all remnants of Unix-like streams from a language's IO interface. -Tim ----- Original Message ----- From: "Ben Rudiak-Gould" <benrg@dark.darkweb.com> To: <haskell@haskell.org> Sent: Sunday, July 27, 2003 11:35 PM Subject: Proposal for a new I/O library design
The other day I was reading the Haskell i18n debate in the list archives, and started thinking about possible replacements for the existing Haskell file I/O model.
It occurred to me that the Haskell community has really dropped the ball on this one. Haskell's design has always emphasized doing the right thing, not merely doing the thing that everyone else happens to be doing. It's that philosophy that led to the invention of the monadic I/O model, among other things. And yet, what do we choose for our I/O primitives? The same old crocks that everyone else was using. We open and close files (whatever that's supposed to mean); we expose file handles to the user; we even maintain a current position in the file, which is an unnecessary global state variable if I've ever seen one.
The proposal below is the result of a few hours spent thinking about how the file system would be accessed if it were actually implemented in Haskell, instead of behind a weird C API. I'm very interested in hearing comments and criticism. In particular, I want to know if there's enough interest in this model that I should actually try to implement it.
The most important idea in this design as far as i18n is concerned is the separation of random-access files from input and output streams. Most of the ugliness of the usual file I/O interface comes from conflating these three concepts, which are almost totally unrelated. In particular, there's no need in this model to worry about the meaning of reading or seeking in a "text file." Text encoding and decoding apply to streams, not files. To read text from a file you layer an input stream on it, apply a text parser to that, and read characters. If you need to "seek" to a new location, you create a new stream which starts at that location in the underlying file.
module System.ProposedNewIOModel (...) where
I assume that all I/O occurs in terms of octets. I think that this holds true of every platform on which Haskell is implemented or is likely to be implemented.
type Octet = Word8
File offsets are 64 bits on all platforms. This model never uses negative offsets, so there's no need for a signed type. (But perhaps it would be better to use one anyway?) BlockLength should be something appropriate to the architecture's address space.
type FilePos = Word64 type BlockLength = Int
A value of type File represents a file, which is essentially a resizable strict array of octets. Two values of type File compare equal if they are the same file -- that is, if they have the same contents and changes to one also appear in the other.
("File" is a bad name for this. For one thing, NTFS and HFS can associate more than one chunk of data with each directory entry, and "file" usually refers to all the chunks together. "Fork" would be more accurate, but it doesn't sound much like what it's supposed to represent.)
(Should files be buffered at the application level? I'm not convinced it's necessary. Streams will be buffered, of course.)
data File -- abstract
A value of type InputStream or OutputStream represents an input or output stream: that is, an octet source or sink. Two InputStreams or OutputStreams compare equal iff reading/writing one also reads/writes the other.
(Should I call these "ports" instead of "streams"? How about "OctetSource" and "OctetSink"?)
data InputStream -- abstract data OutputStream -- abstract
Fundamental operations on files. XXX represents some sort of memory buffer (an IOUArray or a Ptr?). All reads and writes supply absolute offsets.
(Note that there's no such thing as reading fewer bytes than requested -- the read either succeeds or it throws an exception. This may be problematic because the file size could decrease between a call to fGetSize and a later call to fRead. One solution is to make fRead treat bytes beyond EOF as zeroes, which is consistent with the usual behavior for sparse files. fWrite should probably expand the file automatically if asked to write beyond its bounds, for similar reasons.)
(A serious unaddressed problem here is locking. There should be a way to e.g. prevent someone else from writing to a file while I'm appending to it, to avoid race conditions. But for whatever reason, the usual OS interface only allows this kind of lock to be acquired when opening the file -- and I can't generally reopen the file because its name might have changed. Meanwhile, another kind of lock, the kind I don't want, can be acquired and released on an already-opened file. I don't know why it's done this way, or how to work around it.)
(And what about asynchronous I/O?)
fGetSize :: File -> IO FilePos fSetSize :: File -> FilePos -> IO () fRead :: File -> FilePos -> BlockLength -> XXX -> IO () fWrite :: File -> FilePos -> BlockLength -> XXX -> IO () fCheckRead :: File -> FilePos -> BlockLength -> IO Bool fCheckWrite :: File -> FilePos -> BlockLength -> IO Bool
Fundamental operations on streams. "Maybe Octet" is supposed to represent "Octet or EOS," though I'm not sure this is enough for proper EOS handling.
isPeek might be useful for text parsers. isUnGet is another possibility -- it's more versatile but harder to implement efficiently. Or this may be a moot point because text parsers will probably want to use isGetBlock anyway for efficiency.
isGet :: InputStream -> IO (Maybe Octet) isPeek :: InputStream -> IO (Maybe Octet) isGetBlock :: InputStream -> BlockLength -> XXX -> IO BlockLength -- efficiency hack
osPut :: OutputStream -> Octet -> IO () osPuts :: OutputStream -> [Octet] -> IO () osPutBlock :: OutputStream -> BlockLength -> XXX -> IO () osFlush :: OutputStream -> IO ()
Standard streams. (How to deal with line buffering?)
stdin :: InputStream stdout, stderr :: OutputStream
Streams can be layered on top of files. Each stream keeps track of its own independent position within the file. The effects of overlapped stream reading and writing on the same part of a file are unspecified, since streams will be buffered in practice.
fileToInputStreamFrom :: File -> FilePos -> (IO?) InputStream fileToOutputStreamFrom :: File -> FilePos -> (IO?) OutputStream
fileToInputStream f = fileToInputStreamFrom f 0 fileToOutputStreamAppend f = fGetSize f >>= fileToOutputStreamFrom f
InputStreams can be read as lazy lists. You're not allowed to use the InputStream again after you call this.
isGetContents :: InputStream -> IO [Octet]
A value of type Directory represents a directory, which is essentially a mutable associative array from names (Unicode strings) to File/Directory/Stream values plus some metadata.
This is supposed to represent a directory *independently of its name*, so it's robust against changes at higher levels of the filesystem hierarchy. I don't know if this can be implemented on Posix or Win32. (The old Mac OS and the NT API do support it.)
data Directory -- abstract
Operations on directories. I suggest dGetContentsWithMetadata rather than dLookupMetadata because using the latter to scan a directory on a FAT32 filesystem takes O(n^2) time.
dGetContents :: Directory -> IO [String] dGetContentsWithMetadata :: Directory -> IO [(String,???)] dLookup :: Directory -> String -> IO (Either File Directory) -- Also should handle named pipes here dCreateFile :: Directory -> String -> (metadata?) -> IO File -- etc...
Pathnames don't fit well with this model, but they're not going to go away, so I provide these functions. If the pathname argument names a directory, that directory and Nothing are returned. Otherwise, a directory and string are returned such that a subsequent call to dLookup or dCreateFile with these values will refer to the same thing as the original pathname. lookupPathname is dLookupPathname from the current directory.
dLookupPathname :: Directory -> String -> IO (Directory, Maybe
String)
lookupPathname :: String -> IO (Directory, Maybe String) getCurrentDirectory :: IO Directory -- no corresponding set!
Convenient shortcuts for common cases.
lookupFileByPathname :: String -> IO File lookupInputStreamByPathname :: String -> IO InputStream -- at least as likely to succeed as lookupFileByPathname
-- Ben
_______________________________________________ Haskell mailing list Haskell@haskell.org http://www.haskell.org/mailman/listinfo/haskell
Tim Sweeney wrote:
I live in a different universe, but over here I prefer to represent files purely as memory-mapped objects. [...]
I'd prefer "official" support for this in Haskell, too. The need for this and other I/O-related stuff was recognized by the Java community, too: It resulted in the java.nio package family, see e.g. http://java.sun.com/j2se/1.4.2/docs/api/ I suggest we should have a look at this before re-inventing the wheel. Not that all Java libraries are perfect in their first incarnation, far from it, see e.g. java.awt, java.net.URL, java.io, ... :-P But the second try (javax.swing, java.net.URI, java.nio, ...) is often quite usable, so we can probably learn some useful concepts from the latter. Cheers, S.
On Mon, Jul 28, 2003 at 12:56:04PM -0500, Tim Sweeney wrote:
Ben,
I live in a different universe, but over here I prefer to represent files purely as memory-mapped objects. In this view, there is no difference between a read-only file and an immutable array of bytes (a byte being a natural number between 0 to 255). A read-write file is then equivalant to a mutable array (or a reference to a mutable array on a heap) of the same. Treating these all as heap references tends to be cleaner, because you can compare the references for equality, which is significant even for read-only files, because two files which contain the same exact data are not necessarily the same file, whereas opening the same file in two different places should result in equal references.
[...]
In this manner, it's possible to get rid of all remnants of Unix-like streams from a language's IO interface.
You certainly can't always mmap the whole file into memory at once (on a 32-bit architecture at least), because: 1) there are files that won't fit into 32-bit address space 2) ... and usually you don't have the whole address space for you. I mean there are mmaped libraries, stack, allocated memory, etc. so the address space can be somewhat fragmented. 3) after mmaping many big files (each fitting into 32-bit address space) you can run out of address space I've been bitten by all this problems and now I mmap my files in parts, mapping and unmapping them as needed. It can be done, but it is no longer that simple.
-Tim
Best regards, Tom -- .signature: Too many levels of symbolic links
Tomasz Zielonka wrote:
You certainly can't always mmap the whole file into memory at once (on a 32-bit architecture at least), because: [...]
I think all these issues are handled by java.nio.Buffer and friends. Are there any people on this list with real-world war stories about java.nio? So far I had a look at those packages from an implementation perspective only... Cheers, S.
Ben Rudiak-Gould (Sun, Jul 27, 2003 at 09:35:41PM -0700):
module System.ProposedNewIOModel (...) where
I assume that all I/O occurs in terms of octets. I think that this holds true of every platform on which Haskell is implemented or is likely to be implemented.
type Octet = Word8
If it should be really generall the base type should be Bool.
File offsets are 64 bits on all platforms. This model never uses negative offsets, so there's no need for a signed type. (But perhaps it would be better to use one anyway?) BlockLength should be something appropriate to the architecture's address space.
type FilePos = Word64 type BlockLength = Int
type FilePos = Integer type BlockLength = Integer
data File -- abstract
I would prefer: data ImmutableStore -- abstract data MutableStore -- abstract A note about buffering: Actually, current UNIX kernels do not support non-blocking descriptors; they support non-blocking open files. Furthermore, many programs will break if they encounter non-blocking mode. This means that you must not [change blocking mode] for a descriptor inherited from another program. See <http://cr.yp.to/lib/io.html>.
A value of type InputStream or OutputStream represents an input or output stream: that is, an octet source or sink. Two InputStreams or OutputStreams compare equal iff reading/writing one also reads/writes the other.
(Should I call these "ports" instead of "streams"? How about "OctetSource" and "OctetSink"?)
data InputStream -- abstract data OutputStream -- abstract
Use data OctetSource -- abstract (or BitSource, s.a.) data OctetSink -- abstract (or BitSink, s.a.) for octets (or bits/bools) and data PacketSource -- abstract data PacketSink -- abstract to send complete packets of data by the latter. Sincerly, -- Stefan
participants (5)
-
Ben Rudiak-Gould -
Stefan Karrmann -
Sven Panne -
Tim Sweeney -
Tomasz Zielonka