- Libraries - Haskell.org

Vector memory usage
by Ben Gamari 24 Feb '12

24 Feb '12

Recently I stumbled upon a rather non-obvious behavior in what seems like a fairly common use a vector[1]. It appears that, despite the claim of the documentation, dropWhile can result in copies, resulting in a 1MB vector ballooning to several gigabytes within a few seconds. This has been reproduced with ghc 7.4.1 and 7.0.3 with -O and -prof. My knowledge of the internals of vector is quite limited, so I'll leave potential explanations to those with better understanding; below I've attached some relevant discussion from #haskell. Let me know if there's anything at all I could do to help. Cheers, - Ben [1] http://hackage.haskell.org/trac/ghc/ticket/5893 Test case: import qualified Data.Vector.Unboxed as V type Time = Int spansPhotons :: V.Vector Time -> [(Time,Time)] -> [V.Vector Time] spansPhotons ts spans = map (\(start,_)->V.takeWhile (<start) ts) spans main = do let times = V.generate 1000000 (*100) spans = [(10000*i, 10000*i+5000) | i <- [1..10000]] let bursts = spansPhotons times spans print $ sum $ map V.length bursts print $ map (const 1) $ bursts 00:16 < rwbarton> there the problem is that the bursts aren't sharing the same memory 00:17 < rwbarton> like they would if you wrote a similar program with ByteString 00:17 < rwbarton> the thing is that the Vector implementation of dropWhile is good for the stream fusion situation where you have a pipeline of operations 00:17 < quintessence> Yeah, I don't think dropWhile can participate in stream fusion and not copy 00:18 < rwbarton> but not good when you want to dropWhile many different times on the same input 00:18 < rwbarton> right 00:18 < quintessence> because when you have dropWhile p . map f the thing you want to share with doesn't exist 00:18 < rwbarton> so I don't know what the resolution would be 00:18 < bgamari> ouch 00:18 < rwbarton> besides giving some more manual control over stream fusion perhaps 00:19 < bgamari> Yep 00:20 < bgamari> That is a tricky one 00:20 < dmwit> dropWhileFusable + dropWhileCopy 00:20 < dmwit> err... 00:21 < dmwit> dropWhileFusable + dropWhileDoesn'tCopy, I guess 00:21 < quintessence> Could you implement dropWhile the non-fusing way and then have a RULE for dropWhile p . unstream = unstream . S.dropWhile p? 00:21 < rwbarton> at the least you could implement dropWhile' the non-fusing way; maybe that already exists somewhere in Data.Vector 00:21 < tibbe> Isn't there a function to explicitly copy? Like ByteString's clone? 00:21 < rwbarton> there is 00:21 < rwbarton> but the goal here is to not copy 00:22 < tibbe> I'm joining this discussion late I'm afraid :) 00:22 < tibbe> it should be possible to write rules that only fire when fusion doesn't happen 00:22 < rwbarton> the issue is a program like http://hpaste.org/64262#a64263 ends up allocating a bunch of nonoverlapping vectors 00:22 < tibbe> they could do sharing, but the performance model might turn out to be confusing 00:22 < rwbarton> when it could share them (and the documentation would have you believe it does) 00:23 < tibbe> how would you document dropWhile? Might share the underlying vector? 00:23 < bgamari> tibbe: the performance model is already a bit confusing ;) 00:23 < tibbe> :) 00:24 < bgamari> I'm surprised I'm the first person to encounter this 00:25 < bgamari> The bug report is http://hackage.haskell.org/trac/ghc/ticket/5893#comment:1 00:29 < bgamari> rwbarton, Do you mind if I quote you in a message to libraries@?

2 1

Hackage documentation down?
by wren ng thornton 24 Feb '12

24 Feb '12

Hello all, I posted a minor update to unix-bytestring a few days ago in order to deal with the Num not implying Eq thing in GHC 7.4, but it looks like the documentation still hasn't been made (nor the built-on field). Are the Hackage builders down intentionally since the recent crash, or accidentally? -- Live well, ~wren

1 0

Wait... System.Posix.IO.ByteString reads data into a *String*?
by Ryan Newton 23 Feb '12

23 Feb '12

I was very surprised to find "fdRead" returning a String in the ByteString version of the library: http://hackage.haskell.org/packages/archive/unix/2.5.1.0/doc/html/System-Po… Unlike, for example the fdRead in unix-bytestring: http://hackage.haskell.org/packages/archive/unix-bytestring/0.3.5/doc/html/… I belatedly found the below thread when trying to figure out what went down with unix-2.5.1.0, and it seems that the impetus for the ".ByteString" modules was to change file path representation, not the representation of file contents. But I'm missing why this would be. If it is a distinct module namespace, why would there be any backwards compatibility concerns? And if taking this one step towards ByteString why not go all the way? (Especially as file contents are much larger and therefore more performance critical than file names.) -Ryan On Fri, Nov 11, 2011 at 11:23 AM, Simon Marlow <marlowsd(a)gmail.com> wrote: > I propose to commit the attached patch to the unix package and release it > with GHC 7.4.1. The commit log is reproduced below. Comments please! > > The unix version number will of course be bumped appropriately. > > Cheers, > Simon > > commit d5e43be90d3c6f8869dd2b0c65800c**9a6dd0ac70 > Author: Simon Marlow <marlowsd(a)gmail.com> > Date: Fri Nov 11 16:18:48 2011 +0000 > > Provide a raw ByteString version of FilePath and environment APIs > > The new module System.Posix.ByteString provides exactly the same API > as System.Posix, except that: > > - There is a new type: RawFilePath = ByteString > > - All functions mentioning FilePath in the System.Posix API > use RawFilePath in the System.Posix.ByteString API > > - RawFilePaths are not subject to Unicode locale encoding and > decoding, unlike FilePaths. They are the exact bytes passed to > and returned from the underlying POSIX API. > > - Similarly for functions that deal in environment > strings (System.Posix.Env): these use untranslated ByteStrings > in System.Posix.Environment > > - There is a new function > > System.Posix.ByteString.**getArgs :: [ByteString] > > returning the raw untranslated arguments as passed to exec() > when the program was started. > > _______________________________________________ > Libraries mailing list > Libraries(a)haskell.org > http://www.haskell.org/mailman/listinfo/libraries > >

4 5

Num instances for Sum and Product
by Dan Burton 22 Feb '12

22 Feb '12

Data.Monoid in the base package specifies newtypes Sum and Product. It would be convenient if these newtypes had appropriate Num instances, which are trivial to write in plain Haskell: import Data.Monoid liftTy2 wrap unwrap op x y = wrap $ unwrap x `op` unwrap yliftTy wrap unwrap f = wrap . f . unwrap liftSum = liftTy Sum getSumliftSum2 = liftTy2 Sum getSum instance (Num a) => Num (Sum a) where (+) = liftSum2 (+) (-) = liftSum2 (-) (*) = liftSum2 (*) abs = liftSum abs negate = liftSum negate signum = liftSum signum fromInteger = Sum . fromInteger liftProd = liftTy Product getProductliftProd2 = liftTy2 Product getProduct instance (Num a) => Num (Product a) where (+) = liftProd2 (+) (-) = liftProd2 (-) (*) = liftProd2 (*) abs = liftProd abs negate = liftProd negate signum = liftProd signum fromInteger = Product . fromInteger Or with a few extensions (as noted by Daniel Wagner): {-# LANGUAGE StandaloneDeriving, GeneralizedNewtypeDeriving #-} import Data.Monoid deriving instance Num a => Num (Sum a)deriving instance Num a => Num (Product a) -- Dan Burton

1 0

enummapset merge into containers
by Ben Gamari 22 Feb '12

22 Feb '12

Not so long ago, it was suggested[1] that enummapset[2] might be a good candidate for merge into containers. Having found this package to be very useful in the past, I would find this quite nice. Would others find this a reasonable proposal? If so, what needs to be done to make this happen? Cheers, - Ben [1] http://blog.ezyang.com/2011/08/changes-to-intmap/comment-page-1/#comment-28… [2] http://hackage.haskell.org/package/enummapset

4 6

Fwd: New STM?
by wren ng thornton 19 Feb '12

19 Feb '12

-------- Original Message -------- Message-ID: <4F3E9E3E.3070804(a)freegeek.org> Date: Fri, 17 Feb 2012 13:36:46 -0500 To: GHC-users List <glasgow-haskell-users(a)haskell.org> Hello all, I had some patches committed to STM a while back[1], and for some reason I was expecting them in the new GHC release. Does STM only ship with the Platform? If not, then any idea when the next version of STM will be released? [1] http://hackage.haskell.org/trac/ghc/ticket/5104 http://www.haskell.org/pipermail/cvs-libraries/2011-April/012914.html -- Live well, ~wren

1 0

Proposal: Remove Control.OldException
by Ian Lynagh 19 Feb '12

19 Feb '12

Hi all, We've had Control.OldException, and the new Control.Exception, since GHC 6.10.1, and since 6.12.1 using Control.OldException has given the deprecation message Future versions of base will not support the old exceptions style. Please switch to extensible exceptions. I propose that we now remove it from base. If some people still cannot use the new Control.Exception for some reason, then it could be maintained in an old-exception package. Discussion deadline: 18 Feb 2012. Thanks Ian

8 11

generalization of maybeToList
by Jeff Shaw 15 Feb '12

15 Feb '12

I find myself using this function quite a lot. I believe it should be included in Data.Maybe. import Control.Monad maybeTo :: (MonadPlus m) => Maybe a -> m a maybeTo = maybe mzero return

5 9

RE: Changes to Typeable
by Simon Peyton-Jones 14 Feb '12

14 Feb '12

| Should there perhaps be a NewTypeable module which could then be renamed | into Typeable once it is sufficiently well established? I started with that idea, but there would be a 2-stage process: * Step 1: (when PolyTypable becomes available) People change to import Data.PolyTypeable * Step 2: (when PolyTypeable becomes Typeable) People change back to Data.Typeable I was hoping to avoid that. Notice that even if we replace Typeable immediately with the new polymorphic one, doing so has very limited impact: for users who just use "deriving Typeable" and "typeOf", that will work fine with the new thing too. But I'm sure that some "seams" would show through. It's a balance. Simon

3 2

Something's weird about System.Directory.removeDirectoryRecursive
by John Millikin 09 Feb '12

09 Feb '12

http://hackage.haskell.org/packages/archive/directory/1.1.0.1/doc/html/Syst… The documentation says that removeDirectoryRecursive follows symlinks. However, the implementation doesn't (in most cases, see below). ------------------------------------------------------------------------------------------------------------ $ mkdir test-removeDirectoryRecursive $ cd test-removeDirectoryRecursive $ mkdir a b $ touch a/a.txt b/b.txt $ ln -s $PWD/b a/ $ ln -s $PWD/b/b.txt a/ $ ls -l a total 8.2k -rw-rw-r-- 1 john john 0 2012-01-27 23:33 a.txt lrwxrwxrwx 1 john john 65 2012-01-27 23:33 b -> /home/john/test-removeDirectoryRecursive/b lrwxrwxrwx 1 john john 71 2012-01-27 23:34 b.txt -> /home/john/test-removeDirectoryRecursive/b/b.txt # OK, a/ has a normal file and two symlinks in it. Let's recursively remove a/ and see what happens. $ ghci Prelude> import System.Directory Prelude System.Directory> removeDirectoryRecursive "a" Prelude System.Directory> Leaving GHCi. $ ls -l a b ls: cannot access a: No such file or directory b: total 0 -rw-rw-r-- 1 john john 0 2012-01-27 23:33 b.txt # a/ was removed -- good! # # b/ and its contents are untouched, good, but goes against the docs ------------------------------------------------------------------------------------------------------------ Now, there is one case where this function *will* follow symlinks. However, I believe it is a bug because it produces odd behavior: ------------------------------------------------------------------------------------------------------------ $ sudo mkdir a [sudo] password for john: $ sudo ln -s $PWD/b a/ $ ls -l a b a: total 4.1k lrwxrwxrwx 1 root root 65 2012-01-27 23:38 b -> /home/john/test-removeDirectoryRecursive/b b: total 0 -rw-rw-r-- 1 john john 0 2012-01-27 23:33 b.txt # Now a/ has a symlink, which cannot be deleted, because its containing directory is read-only to the current user. $ rm a/b rm: remove symbolic link `a/b'? y rm: cannot remove `a/b': Permission denied # What happens if removeDirectoryRecursive is called now? $ ghci Prelude> import System.Directory Prelude System.Directory> removeDirectoryRecursive "a" *** Exception: a/b: removeDirectory: permission denied (Permission denied) Prelude System.Directory> Leaving GHCi. $ ls -l a b a: total 4.1k lrwxrwxrwx 1 root root 65 2012-01-27 23:38 b -> /home/john/test-removeDirectoryRecursive/b b: total 0 # a/ is untouched, but b/ has been emptied! ------------------------------------------------------------------------------------------------------------ So what is the expected behavior of this function? What should it do in the presence of symlinks? IMO, the function should be documented as *not* following symlinks, and the directory check should be changed so that it returns False for symlink-to-directory.

3 2