June 2007 - Haskell-Cafe

Wikipedia archiving bot - code review
by Gwern Branwen 26 Jun '07

26 Jun '07

Hey everyone. So I've been learning Haskell for a while now, and I've found the best way to move from theory to practice is to just write something useful for yourself. Now, I'm keen on editing Wikipedia and I've long wanted some way to stop links to external websites from breaking on me. So I wrote this little program using the TagSoup library which will download Wikipedia articles, parse out external links, and then ask WebCite to archive them. But there's a problem: no matter how I look at it, it's just way too slow. Running on a measly 100 articles at a time, it'll eat up to half my processor time and RAM (according to top). I converted it over to ByteStrings since that's supposed to be a lot better than regular Strings, but that didn't seem to help much. So I'm curious: in what way could this code be better? How could it be more idiomatic or shorter? Particularly, how could it be more efficient either in space or time? Any comments are appreciate. {- Module : Main.hs License : public domain Maintainer : Gwern Branwen <gwern0(a)gmail.com> Stability : unstable Portability : portable Functionality: retrieve specified articles from Wikipedia and request WebCite to archive all URLs found. TODO: send an equivalent request to the Internet Archive. Not in any way rate-limited. BUGS: Issues redundant archive requests. Currently uses Data.ByteString.Lazy.Char8. If I'm understanding the documentation right, this barfs on the full UTF-8 character set, but Wikipedia definitely exercises the full UTF-8 set. USE: Print to stdin a succession of Wikipedia article names (whitespace in names should be escaped as '_'). A valid invocation might be, say: '$echo Fujiwara_no_Teika Fujiwara_no_Shunzei | archive-bot' All URLs in [[Fujiwara no Teika]] and [[Fujiwara no Shunzei]] would then be backed up. If you wanted to run this on all of Wikipedia, you could take the current 'all-titles-in-ns0' gzipped file from [[WP:DUMP]], gunzip it, and then pipe it into archive-bot. -} module Main where import Text.HTML.TagSoup (parseTags, Tag(TagOpen)) import Text.HTML.Download (openURL) import Data.List (isPrefixOf) import Monad (liftM) import Data.Set (toList, fromList) import qualified Data.ByteString.Lazy.Char8 as B (ByteString(), getContents, lines, unlines, pack, unpack, words) main :: IO () main = do mapM_ archiveURL =<< (liftM sortNub $ mapM fetchArticleText =<< (liftM B.words $ B.getContents)) where sortNub :: [[B.ByteString]] -> [B.ByteString] sortNub = toList . fromList . concat fetchArticleText :: B.ByteString -> IO [B.ByteString] fetchArticleText article = liftM (B.lines . extractURLs) (openURL(wikipedia ++ B.unpack article)) where wikipedia = "http://en.wikipedia.org/wiki/" extractURLs :: String -> B.ByteString extractURLs arg = B.unlines $ map B.pack ([x | TagOpen "a" atts <- (parseTags arg), (_,x) <- atts, "http://" `isPrefixOf` x]) archiveURL :: B.ByteString -> IO String archiveURL url = openURL("www.webcitation.org/archive?url=" ++ (B.unpack url) ++ emailAddress) where emailAddress = "&email=gwern0(a)gmail.com" -- gwern MAC10 M3 L34A1 Walther MPL AKS-74 HK-GR6 subsonic rounds ballistic media special

4 3

avoiding command window with wxHaskell on Windows?
by Dean Herington 26 Jun '07

26 Jun '07

I'm experimenting with wxHaskell. I created a small program with it on Windows. (Well, small in source form, not so small in binary form :-) The program runs fine when invoked at a Windows command line. But if I double-click the .exe file, I get a command window that hangs around (but doesn't appear to do anything, fortunately) until the program terminates. How can I avoid this command window? I'm using GHC 6.6 and wxHaskell 0.9.4 on Windows XP. Dean

2 2

Propositional logic question
by Dave Tapley 26 Jun '07

26 Jun '07

Hopefully this is just about on topic enough.. (Oh and it's not home work, I just can't bring myself to let it go!) Taken from "Simon Thompson: Type Theory and Functional Programming" Section 1.1 Exercise 1.3 Question: Give a proof of (A => (B => C)) => ((A /\ B) => C). Now I can easily perform (and verify, it's given earlier in the section) the proof of implication with the terms flipped around: ((A /\ B) => C) => (A => (B => C)) Thus: [A]2 [B]1 ----------- (/\ I) [(A /\ B) => C]3 A /\ B ---------------------------------------- (=> E) C --------- (=> I) 1 B => C ------------------ (=> I) 2 A => (B => C) --------------------------- (=> I) 3 ((A /\ B) => C) => (A => (B => C)) My problem arrives finding a solution to the exercise question, my approach is to basically run the above proof backwards. Thus: A => (B => C) A --------------------- (=> E) B B => C --------------------- (=> E) C Now at this point I thought "aha, I can use (=> I) to introduce (A /\ B)" and get: --------------------- (=> I) 1 (A /\ B) => C But here I am only entitled to discharge (A /\ B) in the preceding proof and not A and B on their own. What proof which would allow me to discharge my assumptions A and B? I can see in my head how it makes perfect sense, but can't jiggle a way to do it using only the given derivations. Many thanks, Dave

3 2

Re: [Haskell-cafe] Parallel + exceptions
by Andrew Coppin 25 Jun '07

25 Jun '07

Sebastian Sylvan wrote: > On 25/06/07, Andrew Coppin <andrewcoppin(a)btinternet.com> wrote: >> Does anybody know where I can find a version of this paper that actually >> has the diagrams in it? >> > > http://citeseer.ist.psu.edu/cache/papers/cs/8487/http:zSzzSzwww.cee.hw.ac.u… > Thanks! :-)

1 0

runInteractiveProcess leaks memory, runInteractiveCommand does not
by Andrea Rossato 25 Jun '07

25 Jun '07

Hi, after many test I found out that System.Process.runInteractiveProcess leaks memory while runInteractiveCommand does. The issue of memory leaks related to running external program was raised here: http://www.haskell.org/pipermail/haskell-cafe/2007-June/027234.html and Bryan noted that after a while the process hits a steady state: http://www.haskell.org/pipermail/haskell-cafe/2007-June/027278.html This is true for runInteractiveCommand, but not for runInceractiveProcess, even though, both relay on the same foreign C imported function. I think this is a ghc bug, but before reporting it I'd like to have someone to confirm it. I wrote this code that demonstrate my point. Could you please have a look? Thanks for you kind attention. Andrea here's the code.: module Main where import System.Process import System.Posix.IO import System.IO import Control.Concurrent readOutput rh = do str <- hGetLine rh return str runWith c f = do (i,o,e,p) <- f c exit <- waitForProcess p str <- readOutput o putStrLn str cHandles i o e threadDelay $ 100000 * 1 runWith c f runRunIntProcess c = do (inp,out,err,p) <- runInteractiveProcess c [] Nothing Nothing return (inp,out,err,p) runRunIntCommand c = do (inp,out,err,p) <- runInteractiveCommand c return (inp,out,err,p) cHandles i o e = do hClose i hClose o hClose e -- this does not leaks -- this reaches a steady state after a while: --virt --5528 3404 668 S 6.6 0.7 13:07.48 procRunInComm main = runWith "date" runRunIntProcess -- this keep on growing --10548 8436 676 S 8.6 1.6 13:15.24 procRunInProc main' = runWith "date" runRunIntCommand

1 0

Parallel + exceptions
by Andrew Coppin 25 Jun '07

25 Jun '07

The two things mentioned in the subject line are both things I've never tried with Haskell. I've seen a lot of papers about these things, but I don't really understand what the current state of play with this is. Are any of these things "production-ready" yet? And how do you use them in the real world?

5 8

Plugin Problem
by Daniel Fischer 25 Jun '07

25 Jun '07

Hi again, I thought, I'd try out the plugins package, cause lambdabot segfaulted while loading plugins (or immediately after). So I wrote a very simpleminded test for eval(_) and unsafeEval(_) from System.Eval.Haskell. Compiled, but: dafis@linux:~/Documents> ghc --make PlugTest [1 of 1] Compiling Main ( PlugTest.hs, PlugTest.o ) Linking PlugTest ... dafis@linux:~/Documents> PlugTest PlugTest: /tmp/MectNX2619.o: unknown symbol `pluginszm1zi0_AltDataziDynamic_toDyn_closure' PlugTest: user error (resolvedObjs failed.) What does that tell more knowledgeable persons? Thanks, Daniel

5 7

Fw: [Haskell-cafe] Re: Best idiom for avoiding Defaulting warnings with ghc -Wall -Werror ??
by Claus Reinke 25 Jun '07

25 Jun '07

i just noticed that i forgot to copy this message to the list.. ----- Original Message ----- From: "Claus Reinke" <claus.reinke(a)talk21.com> To: "apfelmus" <apfelmus(a)quantentunnel.de> Sent: Monday, June 25, 2007 12:20 PM Subject: Re: [Haskell-cafe] Re: Best idiom for avoiding Defaulting warnings with ghc -Wall -Werror ?? >>> there are at least two problems with embedding strongly and dynamically >>> typed languages into strongly and statically typed languages: >>> >>> - it assumes that there is a globally unique static stage, entirely >>> separate from a globally unique dynamic stage >>> - it assumes that there is a unique, unchanging (closed) world of types >>> >>> static typing alone is too static to deal with dynamically evolving >>> types or flexible ideas such as runtime compilation or dynamic linking.. >>> >>> the solution is long-known as type Dynamic >> >> Yes, that's what I meant. The "embedding" is a rather degenerate one, >> making everything to be of type Dynamic. > > one problem is that haskell doesn't have type Dynamic. only hacked > up approximations of it for monomorphic types. i don't recall whether > the following is the best reference (there were many papers with similar > titles), it is just the first one i find right now: > > M. Abadi, L. Cardelli, B. Pierce, G. Plotkin, D. R{\`e}my, > Dynamic Typing in Polymorphic Languages > http://citeseer.ist.psu.edu/abadi94dynamic.html > > the other problem is that there's no type registry outside of individual programs, so only > programs with identical type definitions can communicate. some of the issues are discussed in > 2003: 6. Martijn Vervoort and Rinus Plasmeijer. Lazy Dynamic Input/Output in the lazy > functional language Clean > http://www.st.cs.ru.nl/Onderzoek/Publicaties/publicaties.html > > some more discussion in earlier publications on orthogonal persistent systems. the only losely > related one i can find right now is > > Connor, RCH, Cutts, QI, Kirby, GNC, Morrison, R, > Using Persistence Technology to Control Schema Evolution > http://www-old.cs.st-andrews.ac.uk/research/publications/CCK+94a.php > > where "schema evolution" is the database community term for what > is the evolution of type definitions in an orthogonally persistent language > (another thing haskell doesn't address: try compiling old -previously type-correct!- haskell code > with new ghc/libraries where the library types and apis have changed..; the static typing of > haskell only ever addresses fragments of the lifetime of programs and data - between these > fragments, we're back to untyped strings/object code; so when i say i want Dynamics, i don't want > to weaken the type system, i want to extend its range - more safety, not less;-). > > [that paper also happens to feature some screenshots of an ide that > permits program parts expressed as source text to refer to program > parts that have already been run and stored, a kind of hyperlinked > program representation which might be of independent interest;-] > > imagine a language that can implement something like an advanced > hs-plugins: at runtime of program A, a string representing program B, including new type > definitions and code, is type checked, compiled, and linked into the running program A, so that > entities in A and B > can interact as if they'd been written as parts of a single program. > > the "static" type-checking of B is a "dynamic" type-check if viewed > from A. and if the contents of B are not known when A is compiled, > there's no way of writing down a universal sum type in A that usefully > encompasses all the possible types in B as well. at least not in current > haskell. > >>> if you have a strongly and dynamically typed language, you can embed >>> strongly and statically typed languages into it. by default, that means >>> you get more type-checks than necessary and type-errors later than you'd >>> wish, but you still get them. eliminating runtime type information and >>> runtime type-checks in a strongly and dynamically typed language is a >>> question of optimisation, similar to deforestation. >> >> Well, you can't embed the static checks into dynamic checks without >> loosing the fact that they're static. Which I think is the crucial >> point: "program testing can be used very effectively to show the >> presence of [type] bugs but never to show their absence." To me, dynamic >> typing is next to useless, i want a (partial) proof that the program >> works since a proof is really the only way to know whether it works. > > yes, and no. no, because there's nothing keeping the dynamically > typed system from doing as many type-checks as early as possible; > yes, because there's nothing in the dynamically typed language that > says "i want all my type-checks to be done before this point, and > a guarantee that no type-errors can occur after this point", so the > dynamically typed system can't really report the errors until just > before their execution. > > so, you're right. Dynamic and typecase are necessary extensions > not just to strongly and statically typed languages but also to strongly and dynamically typed > languages. in the former, they > extend the set of expressible programs, in the latter, they extend > the set of expressible restrictions/safety checks. > > i don't want dynamically typed or statically typed languages, i > want the safety of static types whereever possible, combined with the expressiveness of dynamic > types only where needed. and one known way of expressing the phase shift between dynamic and > static typing precisely and safely are Dynamic/ > typecase: > > - in an otherwise statically typed language, they permit me to write programs that couldn't be > expressed otherwise > > - in an otherwise dynamically typed language, they permit > me to extend type-safety over whole regions of code > at once in ways that couldn't be expressed otherwise > > claus >

1 0

Question about HList possibilities
by Scott West 25 Jun '07

25 Jun '07

Hello all, Given an HList (http://homepages.cwi.nl/~ralf/HList/) would it be possible to do the following: Create a class/function/magicks that would essentially do what hOccursMany does, except it would not return a list of elements, but a new HList. For example, would this allow us to be able to write more lax typing constraints and say extract only things that are in lists. ie) HCons "hi" (HCons [2.2,3.3] (HCons 'a' hNil)) -> HCons "hi" (HCons [2.2,3.3] hNil) (removing the Char element). I tried to write something like this but I did not get very far, is it even possible? I'm new to this type-level programming :) Scott

2 1

Haskell version of ray tracer code is much slower than the original ML
by Philip Armstrong 25 Jun '07

25 Jun '07

In odd spare moments, I took John Harrops simple ray tracer[1] & made a Haskell version: http://www.kantaka.co.uk/cgi-bin/darcsweb.cgi?r=ray darcs get http://www.kantaka.co.uk/darcs/ray It's pretty much a straight translation into idiomatic Haskell (as far as my Haskell is idiomatic anyway). Unfortunately, it's a lot slower than the ML version, despite turning all the optimisation options up as far as they'll go. Profiling suggests that much of the time is spent in the intersection' function, and that the code is creating (and garbage collecting) an awful lot of (-|) vector subtraction thunks. Trying to make intersection' or ray_sphere stricter (with seq) appears to have no effect whatsoever: the output of -ddump-simpl is unchanged (with the arguments all staying lazy). Am I missing anything obvious? I don't want to carry out herculean code rewriting efforts: that wouldn't really be in the spirit of the thing. cheers, Phil [1] http://www.ffconsultancy.com/languages/ray_tracer/comparison.html -- http://www.kantaka.co.uk/ .oOo. public key: http://www.kantaka.co.uk/gpg.txt

2 1