June 2011 - Libraries

PROPOSAL: add tryFdSeek to unix
by wren ng thornton 29 Jun '11

29 Jun '11

Hello all, I'd like to propose adding a variant of fdSeek which returns errors directly instead of throwing an exception. -- unix:System.Posix.IO tryFdSeek :: Fd -> SeekMode -> FileOffset -> IO (Either Errno FileOffset) The naming convention and desire for non-exceptional versions follows the design of unix-bytestring, which was released as a non-core library for exploring ways to clean up the System.Posix.IO API (since noone was satisfied with the String-based API when adding ByteString variants to unix was proposed[1]). However, this function is more appropriate to add to unix itself, and has been requested by some folks planning to move to unix-bytestring. The proposed implementation is the same as the current implementation of fdSeek, except using eitherErrnoIfMinus1 instead of throwErrnoIfMinus1. Functions like eitherErrnoIfMinus1 can be found here[2] and are straightforward modifications of the exception-throwing variants in Foreign.C.Error. A local copy of eitherErrnoIfMinus1 could be defined in unix, though it makes sense to propose adding it (or eitherErrnoIf) to base instead. While in there, it would be sensible to add eitherErrnoIfRetry as well. For the sake of completeness, the following suite of functions would provide a non exception-throwing variant of the current API: -- base:Foreign.C.Error eitherErrnoIf :: (a -> Bool) -> IO a -> IO (Either Errno a) eitherErrnoIfRetry :: (a -> Bool) -> IO a -> IO (Either Errno a) eitherErrnoIfMinus1 :: (Num a) => IO a -> IO (Either Errno a) eitherErrnoIfMinus1Retry :: (Num a) => IO a -> IO (Either Errno a) eitherErrnoIfNull :: IO (Ptr a) -> IO (Either Errno (Ptr a)) eitherErrnoIfNullRetry :: IO (Ptr a) -> IO (Either Errno (Ptr a)) eitherErrnoIfRetryMayBlock :: (a -> Bool) -> IO a -> IO b -> IO (Either Errno a) eitherErrnoIfMinus1RetryMayBlock :: (Num a) => IO a -> IO b -> IO (Either Errno a) eitherErrnoIfNullRetryMayBlock :: IO (Ptr a) -> IO b -> IO (Either Errno (Ptr a)) eitherErrnoPathIf :: (a -> Bool) -> FilePath -> IO a -> IO (Either Errno a) eitherErrnoPathIfNull :: FilePath -> IO (Ptr a) -> IO (Either Errno (Ptr a)) eitherErrnoPathIfMinus1 :: (Num a) => FilePath -> IO a -> IO (Either Errno a) -- plus all the foo_ variants That's a lot of functions! And most of them are trivial (e.g., all the Minus1, Null, and _ variants). Personally I think the whole suite should be modified into a simple combinator-based approach. However, doing all of that is *not* part of the current proposal === The Proposal === (1) Add tryFdSeek to unix:System.Posix.IO with a local copy of eitherErrnoIfMinus1. (2) Instigate discussion about the fate of base:Foreign.C.Error for future proposals. (3) Tell me where the current unix repo lives so I can make a formal patch. I know a bunch of things were moved from darcs to git recently, but Hackage still reports the darcs repo for unix and I'm not sure if it was one of the ones that moved. Discussion time: 2 weeks. === Links === [1] http://www.haskell.org/pipermail/libraries/2011-February/015970.html (et seq.) [2] http://hackage.haskell.org/packages/archive/unix-bytestring/0.3.4.1/doc/htm… -- Live well, ~wren

3 2

Re: Error in documentation for Control.Exception (base-4.*)
by wren ng thornton 29 Jun '11

29 Jun '11

On 6/29/11 1:37 AM, Ivan Lazar Miljenovic wrote: > This was caught by mbuf on #haskell: the example for the Exception > class in Control.Exception as of base-4 needs some extra escaping for > Haddock markup. It currently has: > > @ > *Main> throw ThisException `catch` \e -> putStrLn (\"Caught \" ++ show > (e :: MyException)) > Caught ThisException > @ > > It should be (untested; I'm not sure of the backticks but I think it'll work): > > @ > *Main> throw ThisException \`catch\` \\e -> putStrLn (\"Caught \" ++ > show (e :: MyException)) > Caught ThisException > @ You should also escape the >. At least on older Haddock it was required, not sure about newer Haddock (though it doesn't complain). -- Live well, ~wren

1 0

genBits: small addition to random API + largely new implementation that needs code review
by Ryan Newton 29 Jun '11

29 Jun '11

[CC'ing some people involved in the relevant ticket discussions or showing up in git blame] Hello all, So it seems random was due for an overhaul. I would like to initiate a discussion period to talk about what changes should happen before the next major release. I think this is timely because there is *already* a pending backwards-incompatible change in the API (factoring out the SplittableGen class) <http://hackage.haskell.org/trac/ghc/ticket/4314>. We might as well make any other fixes now and make all the changes at once. For reference, I've attached a list of the open tickets I know about pertaining to System.Random at the end of this email. There implementation has two major pieces: (1) Sources of random bits (class RandomGen) (2) Instances of class Random that map use random bits to create Haskell types For now I've left (1) alone and have made heavy changes to part (2) in a branch to fix correctness and performance problems: https://github.com/haskell/random/tree/new_api These new changes require a small API change to RandomGen. The issue is that the legacy RandomGen API isn't very good at generating random *bits*. Rather, it next creates numbers in an arbitrary range: genRange. First of all, I would argue that -- completely aside from performance considerations -- this extra flexibility for RandomGens increases the likelihood of errors in the clients of the API. As supporting evidence I would cite tickets #5278 <http://hackage.haskell.org/trac/ghc/ticket/5278> and #5280<http://hackage.haskell.org/trac/ghc/ticket/5280> -- it looks like even use of the API in System/Random.hs itself was incorrect! *My proposed API addition/restriction:* I propose that we add a "genBits" function that reports how many bits of randomness are created by a generator. class RandomGen g where ... genBits :: g -> Int I added a default implementation of genBits that computes the answer based on genRange. This need not be a backwards-incompatible change. What about generators that have a genRange which is not a power of two? We could consider making genBits return a Maybe value to distinguish these. But I would prefer that we* *instead* restrict RandomGen instances, requiring that they generate a clean number of random bits*. This requires that genRange be (0,2^N-1) or (-2^N,2^N-1). But we can tolerate slight deviations; for example, in the new_api branch<https://github.com/haskell/random/tree/new_api>, stdGen has a genRange of (0,2^31 - 86). Probably close enough. (In any case, in the future I'd like to see stdGen replaced with something better anyway.) *An alternative would be to go even further and require every RandomGen instances to generate the full range of Ints.* I think any future replacement for stdGen will probably meet this additional requirement, though I would be interested in important generators that do not, and in any references on the original API design of RandomGen. As long as we have genBits I don't feel strongly about it but am interested in other opinions. Anyway, I've used genBits to reimplement the random and randomR methods. Let me know what you think. This new version should fix tickets #5278<http://hackage.haskell.org/trac/ghc/ticket/5278>, #5280 <http://hackage.haskell.org/trac/ghc/ticket/5280>, #5133<http://hackage.haskell.org/trac/ghc/ticket/5133>. But give me a code review and help me head off future new bugs! Regarding other tickets: #3575 <http://hackage.haskell.org/trac/ghc/ticket/3575> and #3620<http://hackage.haskell.org/trac/ghc/ticket/3620>will have to wait for stdGen to get replaced. Whereas #427 <http://hackage.haskell.org/trac/ghc/ticket/427> and #2280<http://hackage.haskell.org/trac/ghc/ticket/2280> are complaints about the performance; they will also require a new stdGen to be adequately addressed. However, the current batch of changes *does* change the performance of some Random instances significantly. (See appendix.) There's plenty of room for improvement, and in particular, the new random approach would work better if #4102<http://hackage.haskell.org/trac/ghc/ticket/4102> were implemented in Data.Bits. The excessive laziness problems are still there. (#4218<http://hackage.haskell.org/trac/ghc/ticket/4218>is still unfixed.) Perhaps that's something we can fix in this audit. I'd also be happy to hear other proposals for API changes and additions. System.Random can't depend on bytestring, so I doubt we want block RNG in the RandomGen class. I could, however, imagine having nextWord16, nextWord32, etc. Something worthwhile if we're doing an API audit is to look at some of the other Hackage packages. Here are some that I am familiar with: http://hackage.haskell.org/package/DRBG-0.2.2 http://hackage.haskell.org/package/mersenne-random-pure64 http://hackage.haskell.org/package/mwc-random http://hackage.haskell.org/package/intel-aes And some that I'm not: http://hackage.haskell.org/package/random-fu http://hackage.haskell.org/package/gsl-random http://hackage.haskell.org/package/cprng-aes Pardon the long email! Cheers, -Ryan *Appendix A: List of tickets* #5278 System.Random.randomIvalInteger makes invalid assumptions about RandomGen #5280 System.Random commits (rand `mod` base) error. #5133 Random instance for Float can generate values out of requested range #4218 System.Random is way too lazy #427 Random.StdGen slowness #2280 randomR too slow #3575 mkStdGen and split conspire to make some programs predictable #3620 The seeds generated by split are not independent libraries/rando #3352 Data.Word should define instances of Random for each type *Appendix B: Performance data* Here's an "S curve" showing the speedup or slowdown of the new code when generating different types of data with random or randomR (the R labeled ones are randomR). They range from from some massive speedups on floats and doubles as well as a hefty performance regression when Integer is restricted to a small range by randomR. (I'm looking into it.) speedup of new over old: new_api revision f85c6a55b731d5e380c1d6e880105c9933aa9d48 old, revision 130e421e912d394653a43c987be99 161.87843858371693 "Float" 120.29838090036381 "Float R" 54.66538723769602 "CDouble" 41.01849643134359 "CDouble R" 5.546651773234119 "Int" 5.250176145938512 "Integer" 2.4963231822924556 "Word16" 2.340365231384218 "Bool R" 2.2810769806267412 "Bool" 2.1576446209293216 "Double R" 2.059792819214518 "Double" 1.4283216783216783 "Integer R Big" 1.235562774008646 "Word16 R" * 1.008346493029544 "stdGen/next"* 0.713868706483777 "Int R" 0.6738365001610773 "Char R" 0.5974339051794343 "Char" 0.16546783206240073 "Integer R" The baseline stdGen/next hasn't been changed. Thus it's "speedup" is 1.0.

5 10

Should Data.Bits have a bit scan operation; can the compiler generate a BSR instruction?
by Ryan Newton 27 Jun '11

27 Jun '11

It would be nice to have something equivalent to GCC's __builtin_clz intrinsic that could generate a BSR instruction on x86. Is there any way to get GHC to generate such an instruction presently? If Data.Bits had bitScan methods they would have straightforward default implementations but would also open up the possibility of generating efficient code for some instances. Cheers, -Ryan P.S. Here would be example default implementations: -- The number of leading zero bits: bitScanReverse :: Bits a => a -> Int bitScanReverse num = loop (size - 1) where size = bitSize num loop i | i < 0 = size | testBit num i = size - 1 - i | otherwise = loop (i-1) -- The number of trailing zero bits: bitScanForward :: Bits a => a -> Int bitScanForward num = loop 0 where size = bitSize num loop i | i == size = size | testBit num i = i | otherwise = loop (i+1)

2 1

Containers -- how to synchronize development and lagging repo
by Milan Straka 25 Jun '11

25 Jun '11

Hi GHC HQ, with the new library submission process, I think we should have development repo for containers (maybe on github?). Is there any standard way of dealing with lagging repos? I am not sure if updating the lagging repo and releasing cabal packages is responsibility of GHC HQ of the maintainers. With the new library process, there will probably be more people affected by this issue. BTW, we can also do the development on different branch of the repo, or the lagging repo can be a different branch. But it sounds a bit tricky... Cheers, Milan

5 5

HEADS UP: cabal-install repo being merged into cabal repo
by Duncan Coutts 24 Jun '11

24 Jun '11

All, In theory this will not break anything... But just in case, here's an explanation of what's going on so you will be aware in case you hit any oddities. What's actually happening ------------------------- I am about to push over 700 patches into the cabal darcs repository. This will bring cabal-install into the same repository as the Cabal library. Each package will live in a subdirectory in the same repository. The cabal-install repository will then no longer be used. Existing checkouts of the cabal repo should not be affected, you'll just notice a lot of patches next time you pull. Any outstanding patches not yet applied to the Cabal library should also not be affected. Any outstanding patches not yet applied to the cabal-install tool *will* be affected. They will no longer apply cleanly in the new merged repository. I'm not aware of any outstanding unapplied cabal-install patches but if you do have any that are ready to merge then send them to me and I'll sort them out. The background -------------- I decided that it would be useful for the Cabal and cabal-install packages to live in the same repository. There's a few reasons for this. In future we want to split both Cabal and cabal-install into more packages (in particular we want to split the Cabal package into the declarative bits and the "Simple" build system). Once there are a bunch of closely related packages it's easier to manage if they are together in one repo. Also, we want to do some renaming: it's a constant source of confusion to new users that the 'cabal' program lives not in the 'Cabal' package but in the 'cabal-install' package. Having all the cabal-related packages in one repo will make it easier to manage renaming individual packages. Now cabal-install originally did start out in the main Cabal repo but we forked the repositories back in 2007. So in a sense what we have here is a very very long running branch (4 years and 700+ patches) that we're now re-merging back into the primary cabal branch. But because the forked branch was never intended to be re-merged, we did things that make it non-trivial to re-merge like deleting the cabal-install files in the main cabal branch, and deleting the cabal files in the cabal-install branch (and other things like creating a README file in each). darcs rebase ------------ Enter darcs rebase. Ganesh has been working on a new darcs rebase feature for some time [1]. The basic idea is that as an alternative to merging and resolving conflicts, we modify the patches we want to apply so that they apply cleanly. [1]: http://wiki.darcs.net/Ideas/RebaseStatus In our case we want to rebase all the cabal-install patches so that they apply cleanly in the main cabal repo. Partly as a large-scale test of the new feature, Ganesh has kindly helped by doing the rebase for me. (While I was able to get most of the way using the current darcs rebase UI, it turned out to be quite a complex use case due to some clashing files so there were a few hairy steps that don't yet have a nice UI which is why it was easier for Ganesh to do it rather than me. Obviously the hope is in future the UI can be fleshed out so that mortals can do it too.) Since we are rebasing the cabal-install patches then any patches outside of the cabal-install repo will no longer apply in the merged repo because the identity of the patches will have changed. On the other hand, patches to the main cabal repo will apply just fine. Apart from one cabal-install branch that I am aware of and have handled specially, I don't think there are any other outstanding patches to cabal-install. If any turn up we'll just have to move them over manually. Duncan

2 1

Stdcall v. ccall on Windows FFI
by Mauricio CA 24 Jun '11

24 Jun '11

Hi, Haskell FFI addendum, when talking about calling conventions, says that[1]: ccall Calling convention of the standard C compiler on a system stdcall Calling convention of the Win32 API (matches Pascal conventions) I wonder: what would correspond to such "standard C compiler" in Windows? Is it, say, Visual Studio? And which convention does it use by default? Is it safe to assume that whatever is contained in a C file compiled inside a Cabal package in Windows will have a calling convention matched by 'ccall'? (The reason I'm asking this is because I maintain a FFI macro package[2] that is supposed to be portable between OSs, and I would like to be sure I give a good solution to Windows portability.) Thanks, Maurício [1] http://www.cse.unsw.edu.au/~chak/haskell/ffi/ffi/ffise3.html#x6-110003.1 [2] http://hackage.haskell.org/package/bindings-DSL

4 4

GHC and Haskell 98
by Simon Peyton-Jones 24 Jun '11

24 Jun '11

Friends, this is to ask your opinion about a possible change in GHC 7.2. The current implementation in GHC 7.2 is Plan A below. Plan A is a bit easier for us, but I think it may be a bit draconian, and therefore propose Plan B as an alternative. Opinions? Simon ============================================== With GHC 7.0 if you say ghc -c Main.hs the import of Prelude, whether implicit or explicit, will come from package 'base'. This is also true if you say ghc -c Main.hs -package haskell98 The package 'haskell98' exposes 'List' and 'Random' but not 'Prelude'. That still comes from 'base'. This isn't good, because it means that if in the future we change 'Prelude' in package 'base', a Haskell98 module might fail to compile. And later iterations of Haskell might well want to expand or change the Prelude. There appear to be two alternatives: (Plan A) Add a module 'Prelude' to package 'haskell98'. Now you can compile a pure H98 program thus: ghc -c Main.hs -hide-all-packages -package haskell98 (Cabal puts the -hide-all-packages in for you.) And this will continue to work even if later iterations of Haskell change the Prelude. BUT (A) means that this module becomes un-compilable: module Main where import Random import Data.List Why? Because 'Random' comes from 'haskell98' and 'Data.List' comes from 'base'. But if you say ghc -c Main.hs -package base -package haskell98 then the (implicit) import of 'Prelude' will say "Ambiguous module name: Prelude", because it's exported by both 'base' and 'haskell98'. To fix this you have to change 'import Random' to 'import System.Random'. But the latter's API is different, so you may have to change the source code that uses it. So the second alternative is this: (Plan B) Like Plan A, but in addition, if you say "{-# LANGUAGE Haskell98 #-}" in the file, or -XHaskell98 on the command line, the implicit import of Prelude comes from package 'haskell98', provided -package haskell98 is specified, but regardless of what other in-scope packages expose a Prelude module. Variation: an explicit 'import Prelude' is similarly directed to package 'haskell98' as well. So: Under Plan A, some Hackage packages will become un-compilable, and will require source code changes to fix them. I do not have any idea how many Hackage packages would fail in this way. Unser Plan B, Hackage package that compile now will continue to compile, if their Cabal file is altered. No source code changes. (Well, unless they depend on some innards of GHC.IO or something like that.) But Plan A is simpler. And by breaking packages it will encourage [force] libraries that use a mixture of H98 and more modern modules to move towards the more modern story.

11 12

Re: Proposal: Foldable typeclass: make foldl' and foldr' class methods
by wren ng thornton 21 Jun '11

21 Jun '11

On 6/20/11 1:02 PM, Duncan Coutts wrote: > The proposal is simple: move these two functions into the Foldable type class itself. +1. -- Live well, ~wren

2 1

Proposal: Foldable typeclass: make foldl' and foldr' class methods
by Duncan Coutts 20 Jun '11

20 Jun '11

All, This issue was brought up again recently by Milan's questions about what to do with folds for the containers package. Currently the Foldable type class has: class Foldable t where fold :: Monoid m => t m -> m foldMap :: Monoid m => (a -> m) -> t a -> m foldr :: (a -> b -> b) -> b -> t a -> b foldl :: (a -> b -> a) -> a -> t b -> a foldr1 :: (a -> a -> a) -> t a -> a foldl1 :: (a -> a -> a) -> t a -> a with default implementations for each in terms of the others. Then it defines: foldr' :: Foldable t => (a -> b -> b) -> b -> t a -> b foldr' f z0 xs = foldl f' id xs z0 where f' k x z = k $! f x z foldl' :: Foldable t => (a -> b -> a) -> a -> t b -> a foldl' f z0 xs = foldr f' id xs z0 where f' x k z = k $! f z x That is, they are fixed definitions so specialised implementations cannot be provided. Note also that these are the classic higher-order "foldl in terms of foldr" definitions. Current releases of GHC cannot optimises these higher-order definitions into efficient versions using accumulating parameters. Since one of the main purposes of foldl' is performance (other purpose being to avoid space leaks) then that's rather unfortunate. The proposal is simple: move these two functions into the Foldable type class itself. They would keep their existing default definitions but since they are then class methods they can have efficient implementations provided by the class instances. This should not break much code. In particular it should not break existing type class instance declarations since there is a default definition for instances that don't defined the new methods. The only potential breakage is that foldl' and foldr' are exported via Foldable(..) rather than directly. This could affect modules that use explicit imports. (I consider this fact to be a slightly unfortunate quirk of the Haskell module system). Patch attached. Deadline: 2 weeks: Monday 4th July. Unresolved: what is a good concise specification of foldr' to use in the documentation? For foldl' we can say: foldl' f z = List.foldl' f z . toList Related issues not covered by this simple proposal: providing foldl1' and foldr1', updating instances to define foldl' and foldr' if possible (e.g. array could provide an efficient impl of foldr'). Duncan

6 10