July 2011 - Haskell-Cafe

compare iteratee with python's yield
by yi huang 01 Jul '11

01 Jul '11

I just read several tutorials on iteratee, i find that iteratee is similar to python's generator, both allow streamlined data processing. For example, i can implement enumFile and printChunks in python like this: EOF = None def enum_file(bufsize, filename): with open(filename) as input: while True: data = input.read(bufsize) if not data: break yield data yield EOF def print_chunks(print_empty, generator): for chunk in generator: if chunk==EOF: print 'EOF' return if len(chunk)==0 and not print_empty: continue print chunk print_chunks(True, enum_file(2, "data")) But i find iteratee far more complicated than python's generator, is that because iteratee can do something python's generator can't, or i simply need to be more familar with functional programming style. -- http://www.yi-programmer.com/blog/

3 2

Re: [Haskell-cafe] Parsing cabal files to calculate average number of dependencies
by Gwern Branwen 01 Jul '11

01 Jul '11

On Fri, Jul 1, 2011 at 4:49 PM, L Corbijn <aspergesoepje(a)gmail.com> wrote: > Is this including or exluding 'or'-ed dependency lists like > http://hackage.haskell.org/package/hugs2yc ? Excluding, it seems. When I run the script on that tarball: $ tar --wildcards "*.cabal" -Oxf `find . -name "*.tar.gz" | g hugs2yc` | runhaskell /home/gwern/deps.hs PackageName "mtl" PackageName "uniplate" PackageName "yhccore" PackageName "ycextra" PackageName "parsec" PackageName "directory" PackageName "filepath" No version of base or containers appears. (mtl appears in both branches and also the general build-depends list.) -- gwern http://www.gwern.net

1 0

Is there Functor instance of Enumerator'
by yi huang 01 Jul '11

01 Jul '11

Say i want to compose Enumerator ByteString m b and Iteratee Builder m b, so I try to transform the enum to Enumerator Builder m b, providing function ByteString -> Builder. It's like implement a Functor instance for Enumerator. But i failed, there are no way to make type system happy. Am I right that there is no Functor instance of Enumerator? -- http://www.yi-programmer.com/blog/

2 1

Twitter API?
by Dmitri O.Kondratiev 01 Jul '11

01 Jul '11

Hi, Does anybody work with any Haskell Twitter API library? In particular supporting OAuth authentication? Thanks!

2 2

Patterns for processing large but finite streams
by Eugene Kirpichov 01 Jul '11

01 Jul '11

Hi, I'm rewriting timeplot to avoid holding the whole input in memory, and naturally a problem arises: How to represent large but finite streams and functions that process them, returning other streams or some kinds of aggregate values? Examples: * Adjacent differences of a stream of numbers * Given a stream of numbers with times, split it into buckets by time of given width and produce a stream of (bucket, 50%,75% and 90% quantiles in this bucket) * Sum a stream of numbers Is this, perhaps, what comonads are for? Or iteratees? -- Eugene Kirpichov Principal Engineer, Mirantis Inc. http://www.mirantis.com/ Editor, http://fprog.ru/

7 13

Re: [Haskell-cafe] Patterns for processing large but finite streams
by John Lato 01 Jul '11

01 Jul '11

After the list discussion, I'm surprised nobody mentioned Max Rabkin/Conal Elliott's blog posts on folds and zips. http://squing.blogspot.com/2008/11/beautiful-folding.html http://conal.net/blog/posts/enhancing-a-zip/ They develop a formalism for zipping functions on lists. Iteratee's `zip` set of functions are somewhat similar, but not quite the same. Specifically they still require multiple traversals of the data, but only over a bounded portion of it, so they're much more efficient. Of course you could combine the above patterns with iteratees by creating functions as above, then just running them with a 'fold'. John L. > From: Eugene Kirpichov <ekirpichov(a)gmail.com> > > Thanks but I'm afraid that's still not quite what I'm looking for; > guess I'll have to define my desire by my implementation - so once > it's ready I'll show the result to cafe :) > > 2011/7/1 Alexey Khudyakov <alexey.skladnoy(a)gmail.com>: > > On Fri, Jul 1, 2011 at 12:54 PM, Eugene Kirpichov <ekirpichov(a)gmail.com> > wrote: > >> Alexey, your definition of "mean" does not look like "liftS2 (/) sum > >> length" - you have to manually "fuse" these computations. > >> > > Well it was fused for numerical stability > > > >> I'm asking for a formalism that does this fusion automatically (and > >> guaranteedly). > >> > > Joining accumulators is quite straightforward. So is joining of initial > > state. Just creating a > >> joinAcc :: (acc1 -> x -> acc1) -> (acc2 -> x -> acc2) -> (acc1,acc2) -> > x -> (acc1,acc2) > >> joinAcc f1 f2 (s1,s2) x = (f1 s1 x, f2 s2 x) > > > > Still you have to handle them separately. > >> sum' = foldl (+) 0 > >> len ?= foldl (\n _ -> n+1) 0 > >> sumLen = foldl (joinAcc (+) (\n _ -> n+1)) (0,0) > > > > There is more regular approach but it only works with statistics. > > (function which do not depend on order of elements in the sample) > > For every statistics monoid for its evaluation could be constructed. > > For example sum: > >> newtype Sum a = Sum a > >> instance Num a => Monoid (Sum a) where > >> ? mempty = Sum 0 > >> ? mappend (Sum a) (Sum b) = Sum (a+b) > > > > Composition of these monoids becomes trivial. Just use > > > > > > I pursued this approach in monoid-statistics[1] package. > > It's reasonably well documented > > > > ?[1] http://hackage.haskell.org/package/monoid-statistics > > > >

1 0

Re: [Haskell-cafe] Patterns for processing large but finite streams
by John Lato 01 Jul '11

01 Jul '11

> > From: Eugene Kirpichov <ekirpichov(a)gmail.com> > Subject: [Haskell-cafe] Patterns for processing large but finite > streams > To: Haskell Cafe <haskell-cafe(a)haskell.org> > Message-ID: <BANLkTikdsvQ2Wv4WJR+QMuvksoav0kTqXw(a)mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1 > > Hi, > > I'm rewriting timeplot to avoid holding the whole input in memory, and > naturally a problem arises: > > How to represent large but finite streams and functions that process > them, returning other streams or some kinds of aggregate values? > > Examples: > * Adjacent differences of a stream of numbers > * Given a stream of numbers with times, split it into buckets by time > of given width and produce a stream of (bucket, 50%,75% and 90% > quantiles in this bucket) > * Sum a stream of numbers > > Is this, perhaps, what comonads are for? Or iteratees? > This is exactly what iteratees are for. Specifically, enumeratees are stream transformers, iteratees are stream consumers, and enumerators are stream producers. Consider adjacent differences: Given the stream [a, b, c, d, e ....], you want to produce [b-a, c-b, d-c, e-d, ...] This is a stream transformer, so you need an enumeratee. Using iteratee, there are at least two obvious ways to produce it: 1) High-level, but probably not as good performance. Use Data.Iteratee.ListLike.roll > import Data.Iteratee as I > import Control.Applicative > > diff [x,y] = y-x > diff [x] = 0 > diffs = convStream (map diff <$> roll 2 1) > 2) somewhat explicit, probably better performance > import qualified Data.ListLike as LL > e' iter = do > h <- I.head > unfoldConvStream f h iter > where > f lastEl = do > c <- getChunk > if LL.null c > then return (lastEl, LL.empty) > else do > let h = LL.head c > t = LL.tail c > return (LL.last c, LL.cons (h-lastEl) (LL.zipWith (-) t (LL.init c))) either of these can be run by using an enumerator: *Main> enumPure1Chunk [1..10] (joinI $ e stream2list) >>= run [1,1,1,1,1,1,1,1,1,0] *Main> let e'2 = e' :: Enumeratee [Int] [Int] IO a *Main> enumPure1Chunk [1..10] (joinI $ e'2 stream2list) >>= run [1,1,1,1,1,1,1,1,1] I should optimize 'roll', it wouldn't be hard. Summing is easy; iteratee has "Data.Iteratee.ListLike.sum" built-in, but you could also use a fold. enumPure1Chunk is only useful for small amounts of data, but iteratee packages provide enumerators over files, handles, etc., as well as mechanisms by which you can create your own enumerators. The e' enumeratee is really just a model; I'd probably write one specific to whichever type of stream I wanted to work with. This one assumes a cheap 'cons', for example. For producing a stream of buckets, if the times are ordered it would be simple to do with Data.Iteratee.ListLike.breakE. If the times aren't ordered, I would probably use 'group' instead to collect a set number of samples. In my view, the biggest difference between iteratee and enumerator is the stream abstraction. Iteratee provides > I.Iteratee s m a where 's' is the stream type, e.g. [Int], String, ByteString, Vector Word8, etc. Although the library only processes a chunk at a time (where a chunk is a subsection of the stream), the type is that of the whole stream. Enumerator instead provides > E.Iteratee s m a here, 's' is the type of the chunk. Enumerator treats the stream as having type [s]. The implementations are different too, but ideally that would be hidden from most users. Although the iteratee implementation allows you to use >>= and $, whereas enumerator sometimes requires you to use >>== and $$. I think IterIO mostly follows the same type as Enumerator, although the implementation is again quite different. John

1 0

How to implement the mean function
by Ruohao Li 01 Jul '11

01 Jul '11

Hi guys, I just started learning some Haskell. I want to implement a mean function to compute the mean of a list. The signature of the function is: mean :: (Num a, Fractional b) => [a] -> b But when I implement this simple function, the compiler keep whining at me on type errors. I know this is wrong: mean xs = sum xs / length xs But how to get it right? Thanks.

5 7