- Haskell-Cafe - Haskell.org

Re: [Haskell-cafe] lazy-vs-strict, recursive IO (was: memory needed for SAX parsing XML)
by John Lato 23 Apr '10

23 Apr '10

(changed the thread title hoping to get some others to weigh in on the recursive I/O scheme presented here) > From: Daniil Elovkov <daniil.elovkov(a)googlemail.com> > John Lato wrote: >> >>> Another (additional) approach would be to encapsulate unsafeInterleaveIO >>> within some routine and not let it go out into the wild. >>> >>> lazilyDoWithIO :: IO a -> (a -> b) -> IO b >>> >>> It would use unsafeInterleave internally but catch all IO errors within >>> itself. >>> >>> I wonder if this is a reasonable idea? Maybe it's already done? >>> So the topic is shifting... >> >> doWithIO :: NFData b => IO a -> (a -> b) -> IO b >> doWithIO m f = liftM (\a -> let b = f a in b `deepseq` b) m >> >> It works (just stick it in a "try" block for error handling), but you >> need to write a lot of NFData instances. You also need to be careful >> that b is some sort of reduced structure, or you can end up forcing >> the whole file (or other data) into memory. > > I meant a different thing. In your example there is no unsafeInterleave > at all. I think you mean that 'm' argument is supposed to be an > unsafeInterleaved io action, like getContents, and deepseq'ing saves us > from it hanging somewhere for a long time. Ok Yes, that's what I intended. I understand now that it isn't what you meant. > > But I meant to have a routine that lets us use ordinary io actions in a > lazy way, and restricting that 'hanging' within bounds of this routine. > > But having thought about it a little more I understood that it's impossible. I think it's possible provided you have unsafeInterleaveIO and deepseq. As you discovered, to force evaluation through an unsafeInterleaveIO you need to fully evaluate data, and deepseq is the way to do so. > > Lazy io works now because unsafeInterleaveIO is sticked into getContents > itself, and is called repeatedly (via recursion). Or at least I can > think of this implementation, haven't looked into it. > > I realised that calling unsafeInterleaveIO for an ordinary io action > will not make it run lazily. It will, still, run all-at-once, just not > now, but later. Yes, exactly. unsafeInterleaveIO needs to be used recursively for getContents to function properly. > > So, to cope with it, I can think of exposing a little structure of an io > action. Normally it's completely opaque. But if we knew where its > recursion point lies, then we could control its course of execution. > > So, if we had something like > type RecIO a = IO a -> IO a > and io actions were like > getContents :: RecIO [Char] > getContents rec = do > c <- readOneChar > rest <- rec > return (c:rest) > > then we could either run them > normally :: RecIO a -> IO a > normally r = r (normally r) > or > lazily :: RecIO a -> IO a > lazily r = unsafeInterleavIO $ r (lazily r) > > And > lazilyDoWithIO :: RecIO a -> (a -> b) -> IO b > lazilyDoWithIO m f = do > a <- lazily m > return $ f a > > Hmm, but then, we would have to take special care to not let it out of > this function anyway... So here we come to deepSeq'ing you proposed. I think my understanding matches yours at this point, except I'm not sure I follow RecIO. Your getContents, for example, seems to only read one character. I think you mean something like this? recGetContents :: RecIO String recGetContents rec = do c <- getChar ise <- isEOF if ise then return [c] else liftM (c:) rec normally :: RecIO a -> IO a normally = fix lazily :: RecIO a -> IO a lazily rec = let f = unsafeInterleaveIO (rec f) in f This works, but I'm not sure that it accomplishes anything. (it doesn't work in ghci because EOF isn't sent through the terminal, but if you test for another character it works fine). > > And anyway, instead of re-writing pure functions to become iteratees we > will have to re-write io functions to adopt continuation passing style. > > Initially it looked better to me :) Re-writing pure functions to be iteratees doesn't have to be hard; many of the provided iteratees have the same names as standard functions so a lot of the work can be done by just changing imports. > > But with this approach we can run lazily any io action that has the form > of RecIO. Also, we can interleave normally and lazily based on the time > of day and other conditions :) > Originally I didn't see much use to this scheme, but I'm starting to change my mind. It would be nice for users to be able to control the evaluation of functions like getContents, hGetContents, etc., and this looks like it could be easier to maintain. > >> It also doesn't help with >> other IO effects, e.g. writing output. I consider this one of the >> nicest features of iteratee-based processing. > > Can you clarify what's the problem with writing? I think I just haven't > switched from the topic of gluing code. > > Because as for gluing code, type signature for io writing d -> IO () is > perfectly sufficient. The problem isn't with writing output per se, but with sequencing I/O effects with lazy data processing. If you wanted to write to a log file after processing each 100kbytes of a file, for example, doing so with bytestrings would require manually chunking the string, creating output, etc., which then hampers both modularity and readability. Adding this sort of monadic side-effect while data is only partially evaluated is pretty simple with iteratees. Actually, this recursive IO combinator idea you've sketched out would be somewhat helpful in this regard too. John

1 0

Re: [Haskell-cafe] memory needed for SAX parsing XML
by John Lato 23 Apr '10

23 Apr '10

> Message: 8 > Date: Tue, 20 Apr 2010 12:08:36 +0400 > From: Daniil Elovkov <daniil.elovkov(a)googlemail.com> > Subject: Re: [Haskell-cafe] memory needed for SAX parsing XML > To: Haskell-Cafe <haskell-cafe(a)haskell.org> > Message-ID: <4BCD6104.50508(a)googlemail.com> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Jason Dagit wrote: >> >> >> On Mon, Apr 19, 2010 at 3:01 AM, Daniil Elovkov >> <daniil.elovkov(a)googlemail.com <mailto:daniil.elovkov@googlemail.com>> >> wrote: > >> I think iteratees are slowly catching on as an alternative to lazy io. >> Basically, the iteratee approach uses a left fold style to stream the >> data and process it in chunks, including some exception handling. >> Unfortunately, I think it may also require a special sax parser that is >> specifically geared towards iteratee use. Having an iteratee based sax >> parser would make processing large xml streams very convenient in >> haskell. Hint, hint, if you want to write a library :) (Or, maybe it >> exists, I admit that I haven't checked.) > > > Iteratees seem like a natural thing if we want to completely avoid > unsafeInterleaveIO. But the presence of the latter is so good for > modularity. > > We can glue IO code and pure code of the type String -> a so seamlessly. > In case of iteratees, as I understand, pure functions of the type String > -> a would no longer be usable with IO String result. The signature (and > the code itself) would have to be changed to be left fold. To some extent, yes, although the amount of work required can vary. The general rule of thumb is that functions can be used directly with iteratees if they can work strictly. Functions that rely on laziness need some adaptation, although how much varies. If you've built your string handling functions out of a parser combinator library, e.g. parsec-3 or attoparsec, you can just lift the parser into an iteratee and use all your existing functions. Since you're using HaXml, this should work. The only missing part is a polyparse-iteratee converter. I haven't used polyparse, but it looks like the converter would be similar to the one used for attoparsec in the attoparsec-iteratee package. That said, it's not how I would do it. Since SAX is a stream processor, an iteratee-based SAX implementation would be a good fit. I would write a lexer and parser (using a parser combinator library) and then use those with Data.Iteratee.convStream. That's how I would write an iteratee-based SAX parser. HaXml already includes a suitable lexer and parser, but unfortunately they're not exposed. > > Another (additional) approach would be to encapsulate unsafeInterleaveIO > within some routine and not let it go out into the wild. > > lazilyDoWithIO :: IO a -> (a -> b) -> IO b > > It would use unsafeInterleave internally but catch all IO errors within > itself. > > I wonder if this is a reasonable idea? Maybe it's already done? > So the topic is shifting... doWithIO :: NFData b => IO a -> (a -> b) -> IO b doWithIO m f = liftM (\a -> let b = f a in b `deepseq` b) m It works (just stick it in a "try" block for error handling), but you need to write a lot of NFData instances. You also need to be careful that b is some sort of reduced structure, or you can end up forcing the whole file (or other data) into memory. It also doesn't help with other IO effects, e.g. writing output. I consider this one of the nicest features of iteratee-based processing. John

2 1

instance reification in TH
by Dmitry Olshansky 23 Apr '10

23 Apr '10

Hello cafe, Could you say can I determine in Template Haskell that datatype X has instance of class Y? Regards, Dmitry

2 1

ANNOUNCE: c2hs 0.16.2
by Duncan Coutts 23 Apr '10

23 Apr '10

All, c2hs version 0.16.2 is out and is available from hackage http://hackage.haskell.org/package/c2hs The only significant change in this release is that it works with ghc-6.12. Note that it is not currently possible to bind to C functions that use "long double" as the corresponding Haskell type (CLDouble) is not provided by the version of the base library that comes with ghc-6.12. See ghc ticket http://hackage.haskell.org/trac/ghc/ticket/3353 What is c2hs ============ The c2hs tool assists in the development of Haskell bindings to C libraries. It extracts interface information from C header files and generates Haskell code with foreign imports and marshaling. The major advantages of using c2hs compared to writing foreign imports by hand (or using hsc2hs) are: * Cross-language type safety: C functions are imported with the correct Haskell types. * Saves time: boilerplate marshaling code is generated. Links ===== Home page: http://www.cse.unsw.edu.au/~chak/haskell/c2hs/ Bug tracker: http://hackage.haskell.org/trac/c2hs/ Duncan

1 0

ANN: forkable-monad 0.1
by David Anderson 23 Apr '10

23 Apr '10

Dear Haskellers, I'm happy, and only slightly intimidated, to announce the initial release of forkable-monad. The short version is that forkable-monad exports a replacement forkIO that lets you do this: type MyMonad = ReaderT Config (StateT Ctx IO) startThread :: MyMonad ThreadId startThread = forkIO threadMain threadMain :: MyMonad () threadMain = forever $ liftIO $ putStrLn "Painless monad stack forking!" Note the lack of monad stack deconstruction and reconstruction to transport it over to the new thread. You'll find the details in the Haddock documentation for the module. forkable-monad is available: * On hackage: http://hackage.haskell.org/package/forkable-monad * Via cabal: cabal install forkable-monad * Source and issue tracker: http://code.google.com/p/forkable-monad/ Feedback is of course welcome. As this is my first published Haskell code and Hackage upload, I expect there will be quite a bit! - Dave

3 5

Haskell (wiki) Logo license
by Jens Petersen 23 Apr '10

23 Apr '10

Hi, We use the Haskell Logo in Fedora OS installer for the Haskell packages group: http://git.fedorahosted.org/git/?p=comps-extras.git;a=tree I trying to get the logo updated to the nice "new" logo designed by Jeff Wheeler: http://haskell.org/sitewiki/images/a/a8/Haskell-logo-60.png To do that we need to update the license in http://git.fedorahosted.org/git/?p=comps-extras.git;a=blob;f=COPYING Can someone tell me what the correct license for the above image file is? Is it the wiki license or something else? Thank you, Jens -- on behalf of Fedora Haskell SIG https://fedoraproject.org/wiki/Haskell_SIG

2 1

Does anyone know the reason for differeng measurements in profiling under -threaded?
by Jesper Louis Andersen 23 Apr '10

23 Apr '10

Hi, I am asking if anyone has seen the following behaviour from GHC under -threaded and heavy use of File I/O, network I/O and STM use. After having run Combinatorrent for a while and then terminating it, we get the following output from the RTS in GC statistics: INIT time 0.00s ( 0.00s elapsed) MUT time 0.22s ( 60.06s elapsed) GC time 0.05s ( 0.07s elapsed) RP time 0.00s ( 0.00s elapsed) PROF time 0.00s ( 0.00s elapsed) EXIT time 0.00s ( 0.00s elapsed) Total time 0.26s ( 60.12s elapsed) %GC time 18.2% (0.1% elapsed) Alloc rate 431,635,651 bytes per MUT second Productivity 80.3% of total user, 0.4% of total elapsed Note that I am running it for exactly 60 seconds of wall-clock time. Yet, the profile information reports: total time = 11.20 secs (560 ticks @ 20 ms) total alloc = 78,696,568 bytes (excludes profiling overheads) COST CENTRE MODULE %time %alloc ticks bytes MAIN MAIN 80.4 0.2 450 20033 PeerControl Process.Peer 11.2 1.0 63 95272 Sender Process.Peer.Sender 6.2 0.0 35 387 sha1_digest Digest 1.8 1.4 10 136377 checkPiece FS 0.2 92.3 1 9083658 TorrentManager Process.TorrentManager 0.2 1.8 1 179587 which is clearly wrong. Total time is nowhere near 11.20 secs with 560 ticks here. I am almost certain that the 450 ticks attributed to MAIN is wrong as well. I have SCCs at each possible forkIO-point in the program. -prof -auto-all -caf-all doesn't make any difference either, I've tried. Has anyone seen this kind of problem and in that case, what was the fix? Barring nobody knows, the only thing I have left to try is to dive into the GHC RTS and figure out what is going on in there. As an aside: Before writing combinatorrent, I wrote etorrent, A BitTorrent client in Erlang. The two clients follow more or less the same basic structure and largely does the same ting. Combinatorrent has better memory usage right now, mostly due to representation choices: Haskell can do much better data representation than erlang. On the other hand, etorrent still beats out combinatorrent in the CPU department, using less CPU. And that somewhat perplexes me. This is why I am looking at performance profiles and trying to find the culprit. But with the above problem, I can't find it, since I don't trust the profiler anymore to give me the right results. All logic suggest that Haskell is ripping apart Erlang in all parts that has no communication, mostly due to being compiled and having a static type system (The erlang code is interpreted, even though they have a HiPE JIT). In practice however, my guess is that the binary handling in Erlang is quite effective, as is the communication between processes, so this might turn the tides in the favor of the erlang-RTS. As a fun thing, I tried downloading the same file in the test environment and running strace -cf on both programs. Here are the top winners. I don't think you can trust all the columns: Erlang: % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 95.57 285.220815 2241 127300 10518 futex 2.16 6.453726 271 23828 9838 read 2.15 6.420512 279153 23 wait4 0.07 0.198851 3 64073 1 epoll_wait 0.02 0.048004 4000 12 poll 0.01 0.039131 2 16297 write 0.01 0.026917 0 88057 37842 recvfrom 0.01 0.018174 0 75803 36 epoll_ctl 0.00 0.008001 667 12 1 select 0.00 0.007133 1 5138 3 lseek 0.00 0.001931 64 30 6 execve 0.00 0.001912 0 5535 writev 0.00 0.001334 191 7 vfork 0.00 0.001209 0 2684 1995 open 0.00 0.001128 0 3094 2257 stat 0.00 0.000942 6 153 munmap Haskell/GHC: % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 32.20 15834.075616 176163 89883 10065 futex 21.72 10682.355809 176009 60692 26672 recvfrom 12.22 6009.064699 176023 34138 18 select 10.25 5038.622913 176009 28627 1 write 8.45 4153.652812 176010 23599 1 read 4.84 2380.702238 176009 13526 1 writev 3.59 1765.107536 175965 10031 rt_sigprocmask 1.95 961.185526 176009 5461 fstat 1.94 956.257032 176009 5433 lseek 1.80 883.829682 175922 5024 getrusage 0.85 419.781465 176009 2385 107 rt_sigreturn 0.04 19.713008 176009 112 fcntl 0.03 12.320630 176009 70 mmap 0.02 9.680495 176009 55 socket 0.02 9.504619 176011 54 53 connect 0.02 8.976459 176009 51 getsockopt 0.01 5.456279 176009 31 close Apart from Erlang using epoll, there is not too much difference from the perspective of the kernel, which I think is rather good :) -- J.

2 2

wxHaskell not able to link to libstdc++.so
by Ahn, Ki Yung 23 Apr '10

23 Apr '10

Dear Haskellers, I heard from a guy who was having problems with wxHaskell using GHC 6.10.x. I tried it myself and had the exact same problem occurring using GHC 6.12.x as well. I am using the debian unstable version of ghc6 package. kyagrd@kyagrd:~/tmp$ head bb.hs import Graphics.UI.WX import Graphics.UI.WXCore (bitmapGetSize) main = start viewer viewer :: IO () viewer = do bm <- bitmapCreateFromFile "~/sample.jpg" size <- bitmapGetSize bm f <- frame [text := "viewer", on paint := onPaint bm, clientSize :=size] kyagrd@kyagrd:~/tmp$ runghc bb.hs bb.hs: <command line>: can't load .so/.DLL for: stdc++ (libstdc++.so: cannot open shared object file: No such file or directory) kyagrd@kyagrd:~/tmp$ ghc bb.hs bb.o: In function `r43M_info': (.text+0x48): undefined reference to `wxzm0zi12zi1zi4_GraphicsziUIziWXziWindow_zdfDimensionsObject_closure' bb.o: In function `r43O_info': (.text+0xa8): undefined reference to `wxzm0zi12zi1zi4_GraphicsziUIziWXziWindow_zdfPaintObject_closure' ... The problem is there is no libstdc++.so in my Debian system as well as the guy who uses Ubunto. What exists is libstdc++.so.6. kyagrd@kyagrd:~/tmp$ ls /usr/lib/libstdc++.* /usr/lib/libstdc++.so.6@ /usr/lib/libstdc++.so.6.0.13 I wonder how wxHaskell was even able to successfully configure and compile at all. I didn't have wx package installed before, and I installed it just today and it successfully configured and compiled but unable to dynamically link to libstdc++.so. More serious problem is that wxHaskell doesn't work even making a symbolic link of libstdc++.so from libstdc++.so.6. I've also tried running ldconfig -v and ldconfig -p as a root, but still get the same problem. Is this a commom problem with the libraries linking to libstdc++.so or is this just the wxHaskell problem? Or, is this a ghc problem? -- Ahn, Ki Yung

1 1

PhD student / postdoc positions in PLT+AD
by Barak A. Pearlmutter 22 Apr '10

22 Apr '10

Seeking PhD students and postdocs interested in an elegant combination of functional programming and big-iron style numeric computing. Functional Programming and Automatic Differentiation PhD Studentships Postdoctoral Positions We are adding exact first-class derivative calculation operators (Automatic Differentiation or AD) to the lambda calculus, and embodying the combination into a production-quality optimising compiler. Our research prototype compiler generates object code competitive with the fastest current systems, which are based on FORTRAN. We are seeking PhD students and postdocs with interest and experience in relevant areas: programming language theory, numeric computing/numeric linear algebra, or differential geometry; and a burning drive to help lift big iron numeric computing out of the 1960s and into a newer higher order. Specific sub-projects include: compiler and numeric programming environment construction; writing, simplifying, and generalising numeric algorithms through the use of AD operators; and associated type/lambda calculus/PLT/real computation issues. The project headquarters will be in the Hamilton Institute, NUI Maynooth, Ireland, http://www.hamilton.ie/. Applications to: "Barak A. Pearlmutter" <barak%2Bad-fp-job(a)cs.nuim.ie> -- Prof Barak A. Pearlmutter Hamilton Institute & Dept Comp Sci, NUI Maynooth, Co. Kildare, Ireland http://www.bcl.hamilton.ie/~barak/

1 0

GHC lexer questions
by Phyx 22 Apr '10

22 Apr '10

Hi all, I have a few questions about the GHC lexer (currently using GHC Glasgow Haskell Compiler, Version 6.13.20100320, for Haskell 98, stage 2 booted by GHC version 6.12.1) And I notice that when invoking the lexer I get a few unexpected results that I was hoping someone could clarify. The first part has to do with braces. Opening braces are lexed fine, but the closing ones throw an error. I assume the lexer is keeping some kind of state and trying to do brace matching? (but isn't this supposed to be done by the parser?) ?parseLine("{")->tag cStatelessParseResultSOk ?parseLine("}")->tag cStatelessParseResultSFailed and ?parseLine("{-")->tag cStatelessParseResultSOk ?parseLine("-}")->tag cStatelessParseResultSFailed and in a statement "let foo = .." "foo" isn't lexed as an identifier, "f" is lexed as a ITvocurly and the "oo" as an identifier. I find this rather odd, and maybe it's me doing something wrong but doubt it. The original Haskell code I call from FFI is lexSourceString :: String -> IO (StatelessParseResult [Located Token]) lexSourceString source = do buffer <- stringToStringBuffer source let srcLoc = mkSrcLoc (mkFastString "internal:string") 1 1 let dynFlag = defaultDynFlags let result = lexTokenStream buffer srcLoc dynFlag return $ convert result -- | convert the build in ParseResult to out custom StateLessParseResult -- mainly because the State is difficult to marshal and we don't really seem to need it convert :: ParseResult a -> StatelessParseResult a convert (POk _ ty) = SOk ty convert (PFailed src msg) = SFailed src (showSDocDump msg)

1 0