February 2011 - Haskell-Cafe

Re: [Haskell-cafe] iteratee-compress space leak?
by John Lato 21 Feb '11

21 Feb '11

Hi Maciej, Thanks for looking in to this. > From: Maciej Piechotka <uzytkownik2(a)gmail.com> > > On Fri, 2011-02-18 at 17:27 +0300, Michael A Baikov wrote: > > I am trying to play with iteratee making parser for squid log files, but > found that my code do not run in constant space when it tries to process > compressed log files. So i simplified my code down to this snippet: > > > > import Data.ByteString (ByteString) > > import Data.Iteratee as I > > import Data.Iteratee.Char > > import Data.Iteratee.ZLib > > import System > > > > main = do > > args <- getArgs > > let fname = args !! 0 > > let blockSize = read $ args !! 1 > > > > fileDriver (leak blockSize) fname >>= print > > > > leak :: Int -> Iteratee ByteString IO () > > leak blockSize = joinIM $ enumInflate GZip defaultDecompressParams > chunkedRead > > where > > consChunk :: Iteratee ByteString IO String > > consChunk = (joinI $ I.take blockSize I.length) >>= return . show > > > > chunkedRead :: Iteratee ByteString IO () > > chunkedRead = joinI $ convStream consChunk printLines > > > > > > First argument - file name (/var/log/messages.1.gz will do) > > second - size of block to consume input. with low size (10 bytes) of > consumed blocks it leaks very fast, with larger blocks (~10000) it works > almost without leaks. > > > > So. Is it bugs within my code, or iteratee-compress should behave > differently? > > After looking into problem (or rather onto your code) - the problem have > nothing to do with iteratee-compress I believe. I get similar behaviour > and results when I replace "joinIM $ enumInflate GZip > defaultDecompressParams chunkedRead" by chunkedRead. (The memory is > smaller but it is due to decompression not iteratee fault). > This is due to "printLines". Whether it's a bug depends on what the correct behavior of "printLines" should be. "printLines" currently only prints lines that are terminated by an EOL (either "\n" or "\r\n"). This means that it needs to hold on to the entire stream received until it finds EOL, and then prints the stream, or drops it if it reaches EOF first. In your case, the stream generated by "convStream consChunk printLines" is just a stream of numbers without any EOL, where the length is dependent on the specified block size. This causes the space leak. If I change the behavior of "printLines" to print lines that aren't terminated by EOL, the leak could be fixed. Whether that behavior is more useful than the present, I don't know. Alternatively, if you insert some newlines into your stream this could be improved as well. As a result of investigating this, I realized that Data.Iteratee.ListLike.break can be very inefficient in cases where the predicate is not satisfied relatively early. I should actually provide an enumeratee interface for it. So thanks very much for (indirectly) suggesting that. Cheers, John L

1 0

PhD Studentships in Functional Programming
by Henrik Nilsson 21 Feb '11

21 Feb '11

Apologies for multiple copies. -- Henrik Nilsson School of Computer Science The University of Nottingham nhn(a)cs.nott.ac.uk

1 0

Re: [Haskell-cafe] Haskell for children? Any experience?
by John Lato 21 Feb '11

21 Feb '11

> > > Message: 7 > Date: Sun, 20 Feb 2011 04:34:56 +0900 > From: "Benjamin L. Russell" <DekuDekuplex(a)Yahoo.com> > Subject: Re: [Haskell-cafe] Haskell for children? Any experience? > To: haskell-cafe(a)haskell.org > Message-ID: <ijp613$9q5$1(a)dough.gmane.org> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Actually, I've been wishing for a high-level way of creating an > interactive three-dimensional virtual world in Haskell that doesn't > require explicit knowledge of linear algebra. Ideally, I'm looking for > a Haskell way of creating a functional counterpart to, say, Open Cobalt > (see http://www.opencobalt.org/) that is high-level-enough not to > require explicit manipulation of row and column vectors. > > One of the main problems, however, is the lack of reflection. Ideally, > I would like the project to be able to modify its own framework in real > time, so that, for example, within the virtual world, users would be > able to create portals to other virtual worlds, and then write code > while the project was running to change the configuration without > restarting the project. Then users would be able to write code in the > functional style to change the virtual environment _in situ._ > It's not Haskell, but this sounds a lot like Newspeak: http://bracha.org/Site/Newspeak.html John L.

1 0

Releasing head of lazy ByteString
by tsuraan 21 Feb '11

21 Feb '11

I have a streaming network protocol where each message in the stream is prefixed by a 64-bit message length marker. Lazy ByteStrings seem to be an elegant way to wrap network communications, so I'm using them. I have one concern though: how can I prevent my program from hanging on to the beginning of the stream? My code looks roughly like this: lazy <- getContents clientSock let (lenBS, rest1) = splitAt 8 lazy let length = runGet getWord64be lenBS let (msg, rest2) = splitAt (fromIntegral length) rest1 -- process msg The program never uses that initial "lazy" again, but it's there, and it's been assigned, so I assume that reference to the head of the stream will always be around, and thus always consuming memory. Is there a way to indicate to haskell that I want to "forget" about that assignment, so the runtime doesn't need to keep the stream around? In this case, I think I could do the stream processing in a tail-recursive loop, and thus lose the previous references to the stream, but is there a more specific way to tell haskell that I'm done with some data?

5 5

Please review my Xapian foreign function interface
by Oliver Charles 21 Feb '11

21 Feb '11

Hello! I've finally came up with some motivation for a project to get my feet wet using Haskell, and for this little pet project I need an interface to Xapian. After reading various documents on FFI in general, I've got a brief working implementation, and I'm now looking for how to better structure the public API. First, a quick bit of background if you're not familiar with Xapian. Xapian is a search engine, and provides a C++ API. You store documents in a database (handled by Xapian), and index documents by adding terms to them. Xapian provides stemming algorithms to help generate these terms from other data. Xapian also has an interface to queries (through a Xapian::Enquire object), and also a query parser to allow for natural language queries to be parsed and ran. For more information, you can check out the API at [1] - it's fairly small. As Xapian is C++, it seems my best option is to create my own simple C wrapper, which also lets me tailor my FFI to be easy to use from Haskell. You can see my C api on Github [2] - for now it's very stripped down; I've been wrapping stuff on a need-to-use basis. * * * Currently what I have is functional (in the sense that it works), but it's extremely tied to I/O and very little of the code is pure. For example, to create and index a document, you need to do something along the lines of: do document <- newDocument setDocumentData document "Document data" addPosting document "search_term" 1 addDocument database document (Assuming you already have an open database handle). How horrible imperative this all looks! :-) A document *feels* like it should be quite pure, however retrieving properties of a document performs I/O. For example, I'd like to have something like: data Document = Document { data :: String, postings :: [String] } do document <- getDocument database 123 -- Get doc #123 and have `document` refer to a pure Document object. I'm still stuck in the IO monad a bit, but at least I can write pure functions to operate on `Document` values now. The problem I see with this, is that I believe I'd have to retrieve all parts of document in my `getDocument` function (include the data and all postings), and I can't benefit from being lazy here. >From what I gather, all the methods on Xapian documents are lazy (such as getting the document data, and getting terms associated with documents), which would mean that my foreign imports would have to be `IO String`, for example. This tends to fairly quick cause the IO monad to propogate everywhere. * * * I think that's enough information to explain my current progress, and my concerns. It could well be that I'm overly worrying about everything being in the IO monad, but as I said - Haskell is new to me. All of my work is at [3], and I'd love any advice you have. Haddock documents have been exported to ocharles.org.uk [4]. Thanks for your time, Oliver Charles / ocharles -- [1]: http://xapian.org/docs/apidoc/html/annotated.html [2]: https://github.com/ocharles/Xapian-Haskell/blob/master/c/cxapian.h [3]: https://github.com/ocharles/Xapian-Haskell [4]: http://ocharles.org.uk/tmp/search-xapian/Search-Xapian.html

2 1

Alex -g
by Mihai Maruseac 20 Feb '11

20 Feb '11

Hi, When running Alex -g I get several warning telling me that a bang pattern is required and that the warning will be an error in GHC 6.14. It is something along the following lines: Warning: Bindings containing unlifted types must use an outermost bang pattern: { (base) = alexIndexInt32OffAddr alex_base s } *** This will be an error in GHC 6.14! Fix your code now! In the expression: let (base) = alexIndexInt32OffAddr alex_base s ((I# (ord_c))) = ord c (offset) = (base +# ord_c) .... I'm running it without the -g flag for now. Don't know yet where too look to solve it or if it is really needed. -- Mihai

2 2

iteratee-compress space leak?
by Michael A Baikov 20 Feb '11

20 Feb '11

I am trying to play with iteratee making parser for squid log files, but found that my code do not run in constant space when it tries to process compressed log files. So i simplified my code down to this snippet: import Data.ByteString (ByteString) import Data.Iteratee as I import Data.Iteratee.Char import Data.Iteratee.ZLib import System main = do args <- getArgs let fname = args !! 0 let blockSize = read $ args !! 1 fileDriver (leak blockSize) fname >>= print leak :: Int -> Iteratee ByteString IO () leak blockSize = joinIM $ enumInflate GZip defaultDecompressParams chunkedRead where consChunk :: Iteratee ByteString IO String consChunk = (joinI $ I.take blockSize I.length) >>= return . show chunkedRead :: Iteratee ByteString IO () chunkedRead = joinI $ convStream consChunk printLines First argument - file name (/var/log/messages.1.gz will do) second - size of block to consume input. with low size (10 bytes) of consumed blocks it leaks very fast, with larger blocks (~10000) it works almost without leaks. So. Is it bugs within my code, or iteratee-compress should behave differently?

2 2

Sub-optimal
by Andrew Coppin 20 Feb '11

20 Feb '11

I have a small program that fills a file with random numbers. If I compile it without optimisation, it runs in constant space. And yet, if I supply -O2 (or even just -O1), for large output files the program gobbles large amounts of RAM. Is this a known bug? (GHC 6.10.x)

6 17

Unable to get parallelism using `par`
by C K Kashyap 20 Feb '11

20 Feb '11

Hi, I tried the first example from "A tutorial on Parallel and Concurrent programming in Haskell" but I cant seem to get sparks to get converted to OS threads. Below is the program I am using. I did ghc --make -threaded program.hs then ./program +RTS -N2 I don't see any speed gain compared to N1. Am I missing something? import System.Time import Control.Parallel fib :: Int -> Int fib 0 = 0 fib 1 = 1 fib n = fib (n-1) + fib (n-2) mkList :: Int -> [Int] mkList n = [1..n-1] relPrime :: Int -> Int -> Bool relPrime x y = gcd x y == 1 euler :: Int -> Int euler n = length (filter (relPrime n) (mkList n)) sumEuler :: Int -> Int sumEuler = sum . (map euler) . mkList sumFibEuler :: Int -> Int -> Int sumFibEuler a b = fib a + sumEuler b parSumFibEuler :: Int -> Int -> Int parSumFibEuler a b = f `par` (e `pseq` (e + f)) where f = fib a e = sumEuler b secDiff :: ClockTime -> ClockTime -> Float secDiff (TOD secs1 psecs1) (TOD secs2 psecs2) = fromInteger (psecs2 - psecs1) / 1e12 + fromInteger (secs2 - secs1) r1 :: Int r1 = sumFibEuler 38 5300 main :: IO () main = do t0 <- getClockTime pseq r1 (return()) t1 <- getClockTime putStrLn ("Sum: " ++ show r1) putStrLn ("time: " ++ show (secDiff t0 t1) ++ " seconds")

5 7

Update2
by Patrick M 19 Feb '11

19 Feb '11

Ok so I wrote and update function update :: Node -> [Node] It takes in a Node and returns a list of the Nodes that result from assigning True to an unassigned atom in one case and False in the other (ie. a case split). The list returned has two nodes as elements. One node contains the formula with an atom assigned True and the model updated with this assignment, and the other contains the formula with the atom assigned False and the model updated to show this. The lists of unassigned atoms of each node are also updated accordingly. This function makes use of an assign function to make the assignments. It also uses the chooseAtom function to select the literal to assign. update :: Node -> [Node] update (formula, (atoms, model)) = [(assign (chooseAtom atoms, True) formula, (remove (chooseAtom atoms) atoms, ((chooseAtom atoms,True)) `insert` model)) , (assign (chooseAtom atoms, False) formula, (remove (chooseAtom atoms) atoms, ((chooseAtom atoms, False) `insert` model)) )] Now I have to do the same thing but this time I must implement a variable selection heuristic.this should replace the chooseAtom and I'm supposed to write a function update2 using it type Atom = String type Literal = (Bool,Atom) type Clause = [Literal] type Formula = [Clause] type Model = [(Atom, Bool)] type Node = (Formula, ([Atom], Model)) update2 :: Node -> [Node] update2 = undefined So my question is how can I create a heurestic and to implement it into the update2 function ,that shoud behave identical to the update function ?

2 1