July 2003 - Libraries

Subprocesses (was: [ANNOUNCE] shell-haskell 1.0.0)
by Simon Marlow 08 Jul '04

08 Jul '04

Libraries people, There are currently two ways to create and communicate with external processes: - System.Cmd.system (very basic, doesn't give you access to the process's output but you can use shell primitives to redirect its input/output). - Posix.runProcess (can specify Handles for the process's input/output, but it doesn't create pipes so you can't actually communicate with the subprocess using these Handles.). - POpen.popen (can specify an input String, and gives you lazy output Strings. No interactive communication is possible.) Ok, *three* ways. System.Cmd.system, Posix.runProcess, POpen.popen, and shell-haskell. Er. *Four* ways. David Sankel's shell-haskell creates pipes for stdin/stdout/stderr, and lets you communicate interactively with the sub-process via Handles. It could do with a way to specify the environment and working directory of the sub-process (ala POpen and runProcess), but apart from that it seems to be the most flexible of the four. It could be simplified slightly by using one Handle rather than two for stdin/stdout. We should think about what functionality we want to provide in the core libraries. My preference would be to go for a full version with three pipes (like shell-haskell) and perhaps one or two more specialised versions covering common cases (I don't know what these should be, though). What do people think? Cheers, Simon -----Original Message----- From: haskell-admin(a)haskell.org [mailto:haskell-admin@haskell.org] On Behalf Of David Sankel Sent: 06 May 2003 03:06 To: haskell(a)haskell.org Subject: [ANNOUNCE] shell-haskell 1.0.0 =================== shell-haskell 1.0.0 =================== shell-haskell provides utilities to communicate with other process's via Haskell code. This library can be used for something as simple as getting the output of another program or as complex as interacting with an interpreter. see http://www.electronconsulting.com/shell-haskell for details. David J. Sankel _______________________________________________ Haskell mailing list Haskell(a)haskell.org http://www.haskell.org/mailman/listinfo/haskell

5 10

Text I/O library proposal, first draft
by Ben Rudiak-Gould 02 Aug '03

02 Aug '03

[Crossposted to Haskell and Libraries. Replies to Libraries.] {- Good things about this text library design: * Efficient implementation should be straightforward * Character coder interface is public, so users can supply their own encodings, or write coder transformers (there are some in the proposal) Bad things: * There's no way to implement fgetpos/fsetpos type functionality, because coders don't expose their internal state. (In fact, there would need to be a way to explicitly copy the state, since it may well include IORefs, Ptrs, etc.) Is this a serious problem? -} module System.TextIOFirstDraft (...) where -- A BlockRecoder takes source and destination buffers and does some sort -- of translation between them. It returns the number of values (not -- bytes!) consumed and the number of values produced. It does not have to -- empty the input buffer or fill the output buffer on each call, but it -- must do something (i.e. it's not acceptable to return (0,0)). Coders -- will in general have internal state which is updated on each call. type BlockRecoder from to = Ptr from -> BlockLength -> Ptr to -> BlockLength -> IO (BlockLength,BlockLength) type TextEncoder = BlockRecoder Word32 Octet type TextDecoder = BlockRecoder Octet Word32 -- IO TextEncoder and IO TextDecoder below denote "coder factories" which -- produce a new coder in its initial state each time they're called. compatibilityEncoder :: IO TextEncoder -- "mod 256" currentLocaleEncoder :: IO TextEncoder iso88591Encoder :: IO TextEncoder latin1Encoder = iso88591Encoder utf8Encoder, utf16BEEncoder, utf16LEEncoder, utf32BEEncoder, utf32LEEncoder :: IO TextEncoder compatibilityDecoder :: IO TextDecoder currentLocaleDecoder :: IO TextDecoder iso88591Decoder :: IO TextDecoder latin1Decoder = iso88591Decoder utf8Decoder, utf16BEDecoder, utf16LEDecoder, utf32BEDecoder, utf32LEDecoder :: IO TextDecoder -- An attempt at supporting setlocale-type locale strings. lookupEncoder :: String -> Maybe (IO TextEncoder) lookupDecoder :: String -> Maybe (IO TextDecoder) -- prependBOM takes an existing UTF encoder and causes it to prepend -- a BOM (byte-order mark) to its output. autodetectUTF takes an existing -- decoder and modifies it to check for a BOM, switching to the -- appropriate type of UTF decoding if one is found. prependBOM :: IO TextEncoder -> IO TextEncoder autodetectUTF :: IO TextDecoder -> IO TextDecoder -- Attaches a TextInputChannel to the supplied InputChannel. After -- this operation the InputChannel should be considered owned by the -- TextInputChannel; any attempt to use it directly will cause -- unpredictable results. This takes a decoder factory rather than a -- decoder to prevent the error of attaching the same decoder to more than -- one channel. icAttachTextDecoder :: InputChannel -> IO TextDecoder -> IO TextInputChannel ticGetChar :: TextInputChannel -> IO Char ticGetLine :: TextInputChannel -> IO String ticLazyGetContents :: TextInputChannel -> IO String ocAttachTextEncoder :: OutputChannel -> IO TextEncoder -> IO TextOutputChannel tocPutChar :: TextOutputChannel -> Char -> IO () tocPutStr :: TextOutputChannel -> String -> IO () tocPutStrLn :: TextOutputChannel -> String -> IO () -- ... etc ... -- Ben

5 6

Library deprecation (was: RE: monad library)
by Simon Marlow 31 Jul '03

31 Jul '03

Malcolm Wallace writes: > ... so the question here > is really about the process for deprecating one existing API and > replacing it with a new API. I don't have a solution to this issue. > However, it is clear that allocating a new namespace every time an > API is improved, is a recipe for confusion amongst users when they > search for "the" API without a knowledge of the social history. Agreed. It seems to me that we do need a way to have multiple versions of an API available, however. We can do this two ways: 1. Use a top-level name 'Old', for old versions of libraries. The contents of Old would mirror the top-level hierarchy, so eg. Old.Control.Monad would be the previous version of the Control.Monad library. This extends nicely to allowing even older versions: Old.Old.Control.Monad :-) 2. Use a suffix instead. For example, Control.Monad.Old would be the previous version, and would import Control.Monad.Trans.Old, etc. (1) means you can retain the structure of the original hierarchy inside Old, and different versions of APIs are completely separate. (2) means less moving around of source files. On balance, think I prefer (1). Cheers, Simon

4 3

RE: monad library
by Simon Marlow 31 Jul '03

31 Jul '03

Iavor Diatchki writes: > ... by the way i don't think this would be a problem if > relative names were added to the module system, as a porgrammer could > use an absolute name for the system library, and a relative > name for the > project monads. but we have already discussed that. The topic of relative module names is still open... > it seems that most people don't want the library in Monad.* > so i guess > we should keep it in Control.Monad.*. then the next > question is when > to replace the current library with the modified one. i am > not sure of > the user base of the monad library, but i would rather do it > sooner than > later. i guess i should point out that the "new" library is not much > different from the old one, and with exception of resumptions and the > instances for the Monad* classes for continuations added > after another > transformer things should work fine (those two were not in > the previous > library anyways). if we keep the library under Unstable*, i > doubt that > anyone would use it, and this will slow down tracking of bugs etc. So are you saying that it won't break much code, or it won't break any code? Either way, I'd be happy for it to be brought in without providing the old version - we make minor changes to library APIs all the time (only in new major releases of GHC, though). > there is also the problem of it being available only from > CVS. i could > make a package available from my web-page (or some other > page), but then > it would clash with the library in base. do you think it would be a > good idea to split it from the base package? base seems > huge as it is already. Definitely! Ideally I'd like to see it distributed as a completely separate package (pending progress on the library distribution work that's going on). For now, moving it into a separate package under fptools/libraries/ would be fine. Cheers, Simon

8 9

RE: Text I/O library proposal, first draft
by Hal Daume 31 Jul '03

31 Jul '03

Hi Ben, > Bad things: > > * There's no way to implement fgetpos/fsetpos type functionality, > because coders don't expose their internal state. (In fact, there > would need to be a way to explicitly copy the state, since it may > well include IORefs, Ptrs, etc.) Is this a serious problem? Yes! This is an enormously serious problem. At least for me. It's not a problem for writing files, but I really really really need this functionality when reading files. Reason: I'm often tooling around in very large (1gb or greater) files which happen to be sorted on some sort of index and I need to do binary search in them. To load all the file into Haskell or to do linear search is impossible. Other than that, I rather like the design. - Hal

3 2

Raw I/O library proposal, second (more pragmatic) draft
by Ben Rudiak-Gould 31 Jul '03

31 Jul '03

[Crossposted to Haskell and Libraries. Replies to Libraries.] -- More comments, please. Bad names? Important missing functionality? -- Still unimplementable? module System.RawIOSecondDraft (...) where data File -- now essentially a file handle data InputChannel -- renamed for less confusing function prefixes data OutputChannel -- | A Moniker is a slight generalization of a pathname. In principle this -- could be extended to support URIs, host:port SSL connections, memory -- buffers, and so on, though it might be better to give them their own -- functions. -- "Compatibility" refers to compatibility with existing Haskell -- implementations, which treat Char values like octets. RawPathname is -- there to avoid an octet -> string -> octet translation in cases where -- that's undesirable. This is slightly problematic on Win32, where -- pathnames really *are* strings. type Moniker = Pathname String | CompatibilityPathname String | RawPathname [Octet] {- if supported, maybe: | DirectoryEntry Directory String | ResourceFork Moniker | NamedFork String Moniker -} -- * File and channel creation functions -- I still use "lookup" rather than "open" to emphasize that there's an -- irreversible and time-dependent translation going on here. lookupFileRO :: Moniker -> IO File lookupFileRW :: Moniker -> IO File lookupInputChannel :: Moniker -> IO InputChannel lookupOutputChannelAppend :: Moniker -> IO OutputChannel createOutputChannel :: Moniker -> IO OutputChannel -- * File access functions -- | Indicates that a File value will never be used again. Calling -- fRelease is not mandatory, but it assists the implementation in -- closing file handles as early as possible. It is an error to use the -- argument in any way after making this call. fRelease :: File -> IO () -- | Returns a new File value which is identical to the argument except -- that calling fRelease on one does not affect the other. fDup :: File -> IO File -- | Returns True if fRelease has not been called on the File value. fIsValid :: File -> IO Bool -- | Reads from a file to a memory buffer. If the read extends beyond the -- end of the file, the extra buffer space is filled with zeroes and the -- return value is the number of octets which actually came from the file. -- Otherwise the return value is the same as the length argument. fReadBuf :: File -> FilePos -> BlockLength -> Ptr Word8 -> IO BlockLength -- | Writes to a file from a memory buffer. fWriteBuf :: File -> FilePos -> BlockLength -> Ptr Word8 -> IO () -- | Returns the current size of the file. fGetSize :: File -> IO FilePos -- | If the size argument is less than the current file size, the file size -- is reduced, destroying any octets beyond the new end of the file. If the -- size argument is greater than the current file size, the file may or may -- not be increased in length. If it is increased, the contents of the file -- beyond the old end of file are undefined. fSetSize :: File -> FilePos -> IO () fIsReadable :: File -> IO Bool fIsWritable :: File -> IO Bool -- * Layering channels on files -- | Opens an input channel (octet source) which reads from the specified -- file, starting at the specified offset. Effectively fDups the File -- argument, so the original File value may be released at any later time -- without affecting the channel. fInputChannelFrom :: File -> FilePos -> IO InputChannel -- | Opens an output channel (octet source) which writes to the specified -- file, starting at the specified offset. Because input and output -- channels may be buffered, the result of an fRead call on a file region fOutputChannelFrom :: File -> FilePos -> IO OutputChannel -- | Like fInputChannelFrom, but stops after reading the octet which -- immediately precedes the second FilePos argument, or at end of file, -- whichever comes first. fInputChannelFromTo :: File -> FilePos -> FilePos -> IO InputChannel -- * Standard channels stdin :: InputChannel stdout, stderr :: OutputChannel -- * Input channel access functions -- | Indicates that no more octets will be read from the InputChannel. -- It is not necessary to call this function if all available octets -- have already been read. icRelease :: InputChannel -> IO () icGet :: InputChannel -> IO Octet icGetBuf :: InputChannel -> Ptr a -> BlockLength -> IO BlockLength icAtEnd :: InputChannel -> IO Bool -- | Returns the File associated with its argument, if any. May fail even -- on a channel created with fInputChannel, since the channel is permitted -- to release its internal File value on reaching end-of-input. May succeed -- on a channel not created with fInputChannel, such as stdin. The caller -- is responsible for releasing the returned File. icTellFile :: InputChannel -> IO Maybe File -- | Returns the current read offset in the underlying file, if any. May -- succeed even if icTellFile fails. Always succeeds if the channel was -- created on top of a File. icTellPos :: InputChannel -> IO (Maybe FilePos) icPeek :: InputChannel -> IO Octet icReady :: InputChannel -> IO Bool icGetBuffering :: InputChannel -> IO BufferMode icSetBuffering :: InputChannel -> BufferMode -> IO () icIsTerminal :: InputChannel -> IO Bool -- | Read an InputChannel lazily as a list of octets. It is an error to -- read from the InputChannel directly after this is called. icLazyGetContents :: InputChannel -> IO [Octet] -- * Output channel access functions ocRelease :: OutputChannel -> IO () ocPut :: OutputChannel -> Octet -> IO () ocPuts :: OutputChannel -> [Octet] -> IO () ocPutBuf :: OutputChannel -> Ptr a -> BlockLength -> IO () ocTellFile :: OutputChannel -> IO (Maybe File) ocTellPos :: OutputChannel -> IO (Maybe FilePos) ocGetBuffering :: OutputChannel -> IO BufferMode ocSetBuffering :: OutputChannel -> BufferMode -> IO () ocFlush :: OutputChannel -> IO () ocIsTerminal :: OutputChannel -> IO Bool -- Ben

1 0

Time library
by Simon Marlow 31 Jul '03

31 Jul '03

The discussion on the new Time library has petered out a little; I want to get this settled and implemented, so let's try to round it off. When we last left the discussion, the conclusion was that having a ClockTime defined in terms of TAI was not implementable, because we can't tell whether the system clock is running POSIX time_t or a variant that is correct and includes leap seconds. (please correct me if I'm wrong). If the system clock is running POSIX time_t, then it is possible to determine the correct TAI time, given a table of leap seconds. There was some feeling that it shouldn't be our responsibility to do this, that the system should provide us with correct time in the first place. I'm inclined to agree (it's less work for those of us who have to implement this stuff after all :-). So, given this, I've updated the proposal to include everything discussed so far and to note the fact that having a correct ClockTime is at the mercy of the system. Complete proposal below, please comment. Cheers, Simon -- ------------------------------------------------------------------------ - -- * ClockTime -- | A representation of absolute time, measured as picoseconds since -- the epoch, where the epoch is 1 January 1970 00:10 TAI. data ClockTime -- abstract instance of (Eq, Ord, Num, Enum, Integral, Show, Read) -- | returns the current absolute time getClockTime :: IO ClockTime {- Rationale: - Our ClockTime is defined in terms of TAI, because this provides an absolute time scale and can be used for accurate time calculations. However, this is not always implementable. Many systems run their system clocks on a time scale that ignores leap seconds. For example, POSIX's time_t uses a broken notion of "seconds since the epoch", defined by a formula in terms of UTC time ignoring leap seconds. The effect of time_t is that the epoch moves forward in absolute terms each time there is a leap second. A system whose clock is following time_t must move its clock back by one second when a leap second occurs (the NTP client usually does this). ClockTime uses the system time, and on systems which run their system clocks according to POSIX time_t, ClockTime will be equivalent to time_t. Of course, regardless of whether ClockTime is based on TAI or not, conversions between ClockTime and CalendarTime will yield the correct results. If your system clock is set correctly, then obtaining the current time as a CalendarTime will give you the correct local time. -} -- ------------------------------------------------------------------------ - -- * Timezone data Timezone -- abstract -- | Make a 'Timezone' from an offset, in seconds relative to UTC, -- which must be smaller in magnitude than @+/-12*60*60@. timezoneFromUTCOffset :: Int -> Timezone -- | Make a 'Timezone' from a standard timezone name (eg. GMT, PDT). -- TAI is a valid timezone name. timezoneFromName :: String -> Maybe Timezone -- | Return the offset in seconds of the specified timezone relative -- to UTC. If the timezone is TAI, returns 'Nothing', because TAI -- cannot be represented as a fixed offset relative to UTC. timezoneUTCOffset :: Timezone -> Maybe Int -- | Return the timezone name corresponding to a 'Timezone' value. -- -- Some timezones may not correspond to a name, or the name of the timezone -- may not be known (some systems cannot convert easily from UTC offsets to -- timezone names), in which case 'timezoneName' returns 'Nothing'. timezoneName :: Timezone -> Maybe String -- | Returns the current timezone from the environment. On Unix, the -- current timezone is taken from the @TZ@ environment variable, or -- the system default if @TZ@ is not set. getCurrentTimezone :: IO Timezone ------------------------------------------------------------------------ ---- -- * CalendarTime data CalendarTime = CalendarTime { ctYear :: Int, ctMonth :: Month, ctDay :: Int, ctHour :: Int, ctMin :: Int, ctSec :: Int, ctPicosec :: Integer, ctTZ :: Timezone } deriving (Eq, Ord, Read, Show) -- | Converts a 'ClockTime' to a 'CalendarTime' in UTC. -- -- Note that this function may produce unpredictable results for -- times sufficiently far in the future, because it is not known -- when leap seconds will need to be added to or subtracted from -- UTC. Note that this doesn't apply if the timezone is TAI. -- clockTimeToUTCTime :: ClockTime -> CalendarTime -- | Converts a 'ClockTime' to a 'CalendarTime' in the current timezone. -- Caveats for 'clockTimeToUTCTime' also apply here. clockTimeToCalendarTime :: ClockTime -> IO CalendarTime -- | Converts a 'ClockTime' to a 'CalendarTime' in the specified timezone. -- Caveats for 'clockTimeToUTCTime' also apply here. clockTimeToCalendarTimeTZ :: Timezone -> ClockTime -> CalendarTime -- | Convert a 'CalendarTime' to a 'ClockTime'. Some values of -- 'CalendarTime' do not represent a valid 'ClockTime', hence this -- function returns a 'Maybe' type. calendarTimeToClockTime :: CalendarTime -> Maybe ClockTime {- TODO: add isDSTCalendarTime? (returns True if the specified CalendarTime is in daylight savings). How do we say "what's the current timezone in X", taking into account DST? -} {- TODO: should we have getLeapSeconds :: [ClockTime] a possibly infinite list of leap seconds in strictly increasing order. This would allow simple conversion between TAI and UTC. -} {- OPTIONAL: these are hard to implement, and require careful specification (see rationale below): addPicoseconds :: CalendarTime -> Integer -> CalendarTime addSeconds :: CalendarTime -> Integer -> CalendarTime addMinutes :: CalendarTime -> Integer -> CalendarTime addDays :: CalendarTime -> Integer -> CalendarTime addWeeks :: CalendarTime -> Integer -> CalendarTime addMonths :: CalendarTime -> Integer -> CalendarTime addYears :: CalendarTime -> Integer -> CalendarTime Rationale: - Adding "irregular" time differences should be done on CalendarTimes, because these operations depend on the timezone. - Need to define the meaning when the offset doesn't exist. eg. adding a day at the end of the month clearly rolls over into the next month. But what about adding a month to January 31st? - Note that addPicoseconds and addSeconds cannot be implemented without access to leap second tables. However, all the others can be implemented using simple calendar arithmetic (including leap years). If the timezone is TAI, then addPicoseconds and addSeconds can be implemented without leap second knowledge, of course. OR: we could provide normalizeCalendarTime :: CalendarTime -> CalendarTime where the following invariant holds: forall t . isJust (calendarTimeToClockTime (normalizeCalendarTime t)) that is, normalizeCalendarTime turns a possibly invalid CalendarTime into a valid one. The intention is that addDays could be implemented as: addDays t days = normalizeCalendarTime t{ ctDays = ctDays t + days } We still need to specify what exactly normalizeCalendarTime does, however. Presumably it needs to know about leap seconds, for example, but only for rolling over the seconds and picoseconds fields. The other fields of CalendarTime can be normalised using ordinary calendar calculations. -}

2 2

RE: monad library
by Simon Marlow 31 Jul '03

31 Jul '03

> On Thu, Jul 31, 2003 at 09:40:55AM +0100, Simon Marlow wrote: > > Iavor Diatchki writes: > > > do you think it would be a > > > good idea to split [monad transformers] from the base package? > > > base seems huge as it is already. > > > > Definitely! Ideally I'd like to see it distributed as a completely > > separate package (pending progress on the library distribution work > > that's going on). For now, moving it into a separate package under > > fptools/libraries/ would be fine. > > Maybe it would make sense to move parsec out too. Yes, and QuickCheck. Cheers, Simon

1 0

Re: Libraries digest, Vol 1 #272 - 12 msgs (SSL)
by Dominic Steinitz 30 Jul '03

30 Jul '03

> Message: 12 > Subject: Re: monad library > From: Martin =?ISO-8859-1?Q?Sj=F6gren?= <martin(a)strakt.com> > To: libraries(a)haskell.org > Date: 30 Jul 2003 08:59:30 +0200 > > > --=-Gr+al8yxZidrCfU3E9Nj > Content-Type: text/plain; charset=ISO-8859-1 > Content-Transfer-Encoding: quoted-printable > > ons 2003-07-30 klockan 04.26 skrev Andrew J Bromage: > > Our taxonomies are, for better or worse, DAG-shaped. SSL, for > > example, can legitimately go under both "network communication" and > > "cryptography", which are quite different fields of computer science. > > For the record, I don't agree with this. I could agree to having SSL > under some sort of "security" category, but it's completely orthogonal > to cryptography. I frequently use SSL without encryption because I just > need authentication. > You are still using encryption for the authentication (RSA) even if you are not encrypting the stream. (RC4). However, I think the classification ought to put RSA and RC4 into Codec.Encryption but SSL into Network. Dominic.

1 0

monad library
by Iavor Diatchki 30 Jul '03

30 Jul '03

hello, i was wondering what people think about the following two issues with the monad library: 1. there are the map* functions for different transformers (mapReader, mapWriter, etc.). does anyone use them for anything? while there is something common between them, i am not quite sure exactly what it is. if anyone is using them i would be interested if the same can be achieved by just using the "standard" functionality for the appropriate transformer. i am inclined to remove them from the library, if there are no objections. 2. currently the monad library is under Unstable.Control.Monad, and i would like to move it to some other place, so that we can use it, without the need to later rename everything. the core of it should be more or less stable. since moving libraries at the moment is quite work intensive, it would be nice if we agreed on where to put it, so that we don't have to move it again. i propose that we put all the monads under Monad, thus one would import Monad.State, Monad.ReaderT, or Monad.Transformers. in this way the old monad library can stay under Control.Monad and programs that are using it need not change. at some point in the future we can decide to remove the old library or it can just stay there. and i like the shorter names (not to mention that it is not clear what some of the monads have to do with control). any comments? iavor -- ================================================== | Iavor S. Diatchki, Ph.D. student | | Department of Computer Science and Engineering | | School of OGI at OHSU | | http://www.cse.ogi.edu/~diatchki | ==================================================

6 12