June 2011 - Haskell-Cafe

*GROUP HUG*
by Gregory Crosswhite 05 Jun '11

05 Jun '11

Hey everyone, Okay, this will sound silly, but I ventured into the Scala mailing list recently and asked an ignorant question on it, and I was shocked when people reacted not by enlightening me but by jumping on me and reacting with hostility. I bring this up not to badmouth the Scala community (they are apparently going through growing pains and will hopefully mature with time!) but just because it made me appreciate just how awesome you guys are, so I just feel the need to publicly express my admiration and thank to everyone on this list for having fostered such an incredibly professional, fanatically nonhostile, and generally pleasant place to talk about Haskell!!! *GROUP HUG* Okay, I'm done now. :-) Cheers, Greg

30 53

ANN: unordered-container 0.1.3.0
by Johan Tibell 04 Jun '11

04 Jun '11

Hi all, I've just uploaded a new version of the unordered-containers package, a package of fast hashing-based container types. Version 0.1.3.0 [1] adds: * the ability to take the union of two maps, * lower memory overhead per key/value pair (contributed by Jan-Willem Maessen), and * the beginning of a `Data.HashSet` module (contributed by Bryan O'Sullivan). If you want to contribute, you can get the source from the git repository: git clone https://github.com/tibbe/unordered-containers.git Alternatively, just fork the project on GitHub [2]. 1. http://http://hackage.haskell.org/package/unordered-containers-0.1.3.0 2. https://github.com/tibbe/unordered-containers

3 3

Problem with linker
by Alejandro Serrano Mena 04 Jun '11

04 Jun '11

Hi, I'm trying to use Hoogle as a library in my Summer of Code project. But when I try to compile using Cabal, I get the following error: Linking dist/build/scion-browser/scion-browser ... /home/serras/.cabal/lib/hoogle-4.2.4/ghc-7.0.2/libHShoogle-4.2.4.a(Serialise.o): In function `s1ePR_info': (.text+0x2015): undefined reference to `hooglezm4zi2zi4_Pathszuhoogle_version1_closure' /home/serras/.cabal/lib/hoogle-4.2.4/ghc-7.0.2/libHShoogle-4.2.4.a(Serialise.o): In function `s1f8a_info': (.text+0x338f): undefined reference to `__stginit_hooglezm4zi2zi4_Pathszuhoogle_' I've been looking for some information and it seems to be related with the fact that Hoogle first builds a library and then build an executable using the library, but all files get recompiled in that second pass. Is there any way to workaround this problem?

3 4

haskellwiki slow/unresponsive
by Greg Weber 03 Jun '11

03 Jun '11

I have been trying to make a few edits to the haskell wiki and find it an excruciating process when I press save and then have to wait a long time to see if the save will go through. I just clicked on the introduction page and it may have took an entire minute to load. Can we put at the top: this wiki written in php, not haskell! Seriously though, the wiki has performed poorly for a long time now. It gives a bad impression to newcomers and deters contributions. Can I (and others!) donate money so someone can make the wiki responsive? Thanks, Greg Weber

5 5

How on Earth Do You Reason about Space?
by Aleksandar Dimitrov 03 Jun '11

03 Jun '11

Dear Cafe, (Excuse the probably very ranty email; I am, unfortunately, at the end of my wits, and I hope that as fellow programmers, you will understand that this is among the most dreadful situations for our kind to be in.) Say, we have an input file that contains a word per line. I want to find all unigrams (unique words) in that file, and associate with them the amount of times they occurred in the file. This would allow me, for example, to make a list of word frequencies in a given text. Simple enough task. Here's an implementation using iteratees (lazy IO is evil) and unordered-containers' Data.HashMap.Strict, which enforces WHNF in values and keys: > import qualified Data.ByteString.Char8 as S > import qualified Data.Iteratee as I > import Data.Iteratee.IO > > import qualified Data.HashMap.Strict as T > > import Data.Iteratee.Char > import Data.List (foldl') > import Data.Char (toLower) > > import Data.Ord (comparing) > import Data.List (sortBy) > import System.Environment (getArgs) > > type Wordcounts = T.HashMap S.ByteString Int > > f' :: Monad m => I.Iteratee S.ByteString m Wordcounts > f' = I.joinI $ enumLinesBS (I.liftI $ step T.empty) > where step t (I.Chunk str) = I.liftI (step $! foldl' maybeIncrement t str) > step t stream = I.idone t stream > maybeIncrement t s > | s == S.empty = t > | otherwise = {-# SCC "m-I-other" #-} T.insertWith (+) s 1 t > > main :: IO () > main = getArgs >>= fileDriverVBuf 65536 f'.head >>= print.prettyList > where prettyList = -- sortBy (comparing snd) . T.toList > T.keys Some lines are empty, and I don't want them to be recorded, so that's why maybeIncrement is necessary. hpaste of this code: http://hpaste.org/47300/spaceleak (ignore convert, that's yet another issue.) Now, here's some observations: on a 75M input file (minuscule, compared to what I actually need) this program will eat 30M of heap space (says profiling) and return in 14 secs. I have two problems with that: a) that's too much heap space, b) the actual memory residency is *much* worse. ad b) average memory residency is at 38MB (this is OK, given heap consumption) but max residency is at 130MB, which is unacceptable to me (remember that I need to run this on files *much* bigger than just 75M.) <<ghc: 9013546200 bytes, 16876 GCs, 38232524/130572448 avg/max bytes residency (58 samples), 306M in use, 0.00 INIT (0.00 elapsed), 6.12 MUT (6.28 elapsed), 7.45 GC (7.50 elapsed) :ghc>> In fact, htop reports total memory residency of the program at around 320 MB at the end of its life-cycle (compiled and ran without profiling options.) I tried running this program on a 1.4GB file, and had to kill it when at 3.5GB memory consumption, my computer started paging. The actual hash-map, however, shouldn't be much bigger than in the 75MB case (I expect about twice the size,) since the input is natural language. I redirected the output of the program (showing a list of assoc pairs that were in the hash map) to a file, and that file measured 11MB in the case of a 75MB corpus, and 40MB when I ran the program on a 1.4GB corpus (I had to use a machine with 24GB of RAM to be able to run this.) ad a) heap consumption is too high for two reasons: firstly, the actual data I care about is much less than there's data on the heap. Secondly, about half the heap space is in LAG state. Here are profiles that will illustrate this: http://imgur.com/wBWmJ&XN1mW&ltYNR. - The first image shows 50% of the heap space being gobbled up with data that shouldn't be there anymore (LAG) - The second image shows the types that are in LAG state: ByteString and HashMap. So, it seems I'm keeping around hash maps? - The third image blames the cost centre m-i-other for the LAG. - Then there's the retainer profile: I have no idea how to read it. I thought it had something to do with iteratee reading in chunks of data, but after setting the chunk size to 4096 bytes and actually getting *less* spikes than before, I doubt that really is the case. It seems like half of the data this program puts on the heap is not cleaned up properly, and the otherwise-branch of maybeIncrement (i.e. the only really important line is this program) is to blame. I'm thinking there's too little strictness somewhere in the T.insertWith clause, but I have no idea how to make it better. Honestly, sprinkling around magic seq-dust every time something goes wrong doesn't feel good. I cannot reason about Haskell's space-performance even a little bit, and I have no idea how to change that. Some more remarks: - I tried using lazy IO first, but that was a total disaster. Iteratee is nice, and works very well, but maybe I don't know how to use it? - In terms of map-implementation, I observed that unordered-containers < Data.Map < bytestring-trie. It surprised me to see that Data.Map would perform better than bytestring-trie for this task (usually, Data.Map would be around twice as fast.) - I should probably look into using a mutable data structure as a map. I would find it a bit disenchanting to have to use a mutable data structure this way, but maybe that's just reality for you. RANT I have tried and tried again to avoid writing programs in Haskell that would leak space like BP likes to leak oil. However, I have yet to produce a single instance of a program that would do anything at all and at the same time consume less memory than there is actual data in the input file. It is very disconcerting to me that I seem to be unable, even after quite some practice, to identify space leaks in trivial programs like the above. I know of no good resource to educate myself in that regard. I have read the GHC manual, RWH's chapter on profiling, also "Inside T5"'s recent series on the Haskell heap, but no dice. Even if I can clearly see the exact line where at least some of the leaking happens (as I can in this case,) it seems impossible for me to prevent it. *thank you very much* for reading this far. This is probably a mostly useless email anyhow, I just had to get it off my chest. Maybe, just maybe, someone among you will have a crucial insight that will save Haskell for me :-) But currently, I see no justification to not start my next project in Lua, Python or Java. Sure, Haskell's code is pretty, and it's fun, but if I can't actually *run* it, why bother? (Yes, this isn't the first time I've ran into this problem …) Thanks, and kindest regards to you all, Aleks

9 19

[JOB] NYU's Center for Genomics and Systems Biology seeks Functional Programmer
by Ashish Agarwal 03 Jun '11

03 Jun '11

<Relevance to this list: The posting below is for an OCaml programmer but welcomes applications from Haskell programmers.> Dear Functional Programmers, I am pleased to announce a job opportunity at The Center for Genomics and Systems Biology (CGSB) at New York University (NYU), located in the heart of Manhattan. The position's main function will be to develop software in the OCaml language to manage, analyze, and display the vast amounts of data generated by next-generation sequencing technologies. NYU's strong commitment to this field is represented by its $100M investment in the brand new CGSB building, which houses the latest sequencing platforms and excellent high performance computing facilities. The position will support the computational needs of several experimental labs by designing and building: o A database for tracking samples, very large quantities of raw data, and complex analysis results o A website for users to submit new samples, monitor progress of their workflow, and visualize data o A system for distributing batch jobs to a cluster, accounting for dependencies between jobs and cached results All components are expected to follow good functional programming design. There are no formal experience or education requirements. Although the software will be written in OCaml, we welcome applicants with experience in other functional languages, especially Haskell, F#, and SML. There is flexibility in the exact nature of the position; you may expand your work to machine learning, parallel programming, or any of the many topics relevant to bioinformatics, so long as the core requirements of the position are fulfilled. Experience in the following areas is a plus but not required: bioinformatics, statistics, type theory, distributed computing, and UNIX systems administration. NYU researchers are using sequencing technologies to investigate basic questions about the nature of life and to address fundamental problems in human health. The very large datasets generated by these technologies pose significant computational challenges for which the robust principles of functional programming are ideally suited. I hope you will find this an exciting opportunity. If you are interested, please contact me with a CV and brief description of your background. Thank you. __________________________________________ Ashish Agarwal New York University http://ashishagarwal.org

1 0

License of hslogger, HDBC, etc.
by John Goerzen 03 Jun '11

03 Jun '11

Hi Jon & all, I've decided that I'm OK with re-licensing hslogger, HDBC, and well all of my Haskell libraries (not end programs) under 3-clause BSD. My schedule is extremely tight right now but if someone wants to send me patches for these things I will try to apply them within the week. -- John

4 4

Re: [Haskell-cafe] Status of Haskell + Mac + GUIs & graphics
by John Lato 02 Jun '11

02 Jun '11

> > Message: 17 > Date: Fri, 20 May 2011 15:59:51 -0700 > From: Evan Laforge <qdunkan(a)gmail.com> > Subject: Re: [Haskell-cafe] Status of Haskell + Mac + GUIs & graphics > To: Erik Hesselink <hesselink(a)gmail.com> > Cc: haskell-cafe(a)haskell.org > Message-ID: <BANLkTi=74mM6ORtU2e192jtoOT9g49fzTQ(a)mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1 > > > Note that it is supposed to be possible to build gtk2hs with gtk+osx, > > which will not use X11 but use the native OS X GUI. I've not been able > > to get this to work, but it's been a while since I tried. The Haskell > > wiki mentions it doesn't support Glade, but does support Cairo. If > > this were to work, gtk2hs would be a serious option as well. > > I've tried this 3 or 4 times, and failed every time. It's crazy > complicated. > I used to use this combination regularly. IME the difficulties are primarily with getting a working gtk+osx build. Once that was accomplished, gtk2hs was a straightforward install, provided you build from the src repo with the -quartz flag. Recently I've switched to using macports gtk2 with the quartz, no-x11 variant, which also uses native cocoa. This is much more reliable than gtk+osx. I don't know if it supports Glade though. My biggest problem with wx is that there's no support for building 64-bit wx applications on OS X. If that were fixed I might prefer it. John Lato

3 6

Re: [Haskell-cafe] Names for pretty-printing combinators
by Ivan Lazar Miljenovic 02 Jun '11

02 Jun '11

(Changing the subject as it's going off-topic from the original email :p) On 25 May 2011 22:45, Simon Meier <iridcode(a)gmail.com> wrote: > 2011/5/25 Ivan Lazar Miljenovic <ivan.miljenovic(a)gmail.com>: >> >> Also, by clashes with Applicative, are you referring to empty and <$> >> ? I'm not sure if a better name than "empty" can be found; as for >> <$>, maybe using pretty's notation of $$ and $+$ rather than <$> and >> <$$> ? > > What about 'emptyDoc'? It's a bit more of a mouthful, and getting a bit close to Hungarian Notation, isn't it? :p Also, empty is already used a fair amount in other libraries as it stands (admittedly, most of those are in modules that are typically imported qualified...). > Moreover, if you are changing the names of > combinators, then moving them away from Applicative and Arrow would be > a good idea; i.e., don't use <+>, as it already used by ArrowPlus. *sigh* We really need to get a list of used operators somewhere so that libraries can stake a claim :p > Moreover, if you can make a Monoid instance such that `mappend` equals > <>, you would also make the library compatible to a future > introduction of (<>) = mappend. I was thinking about that actually... -- Ivan Lazar Miljenovic Ivan.Miljenovic(a)gmail.com IvanMiljenovic.wordpress.com

4 3

ANN: fix-imports-0.1.0
by Evan Laforge 02 Jun '11

02 Jun '11

I just uploaded a new package called fix-imports. Maintaining the import block is one of the less fun things I do while writing haskell. So I wrote a little program to do that for me. Basically, I edit source, then hit ,a in vim, and it figures out which modules need to be imported, which imports are not needed, and sorts and formats the import list. I've been using it for the last week or so and it seems to be working ok, so I decided to upload it to hackage in case anyone else doesn't like maintaining import lists manually. The catch: it only works for qualified imports. Here's copy paste from the package description: A small standalone program to manage the import block of a haskell program. It will try to add import lines for qualified names with no corresponding import, remove unused import lines, and sort the import block according to some heuristic you can define. This only works for qualified imports! Unqualified imports are left untouched. It's most convenient if bound to an editor key.

1 0