November 2014 - Haskell-Cafe

How can I improve the pipes's performance with a huge file?
by zhangjun.julian 16 Nov '14

16 Nov '14

Dear cafe I have 2 file, I want zip the 2 file as couple, and then count each couple's repeat times? The file had more than 40M rows, I use pipe to write code as blow. When I test with 8768000 rows input, it take 30 secs When I test with 18768000 rows input, it take 74 secs But when I test with whole file (40M rows), it take more than 20 minutes and not finished yet. It take more than 9G memorys, and the disk is also busy all time. The result will less than 10k rows, so I had no idea why the memory is so huge. I had use the “http://hackage.haskell.org/package/visual-prof” to profile and improve the performance with the small file But I don’t know how to deal with the “hang” situation. Anyone can give me some help, Thanks. =================================== import System.IO import System.Environment import Pipes import qualified Pipes.Prelude as P import qualified Data.Map as DM import Data.List emptyMap = DM.empty::(DM.Map (String,String) Int) keyCount num = do readHandle1 <- openFile "dataByColumn/click" ReadMode readHandle2 <- openFile "dataByColumn/hour" ReadMode writeHadle <- openFile "output" AppendMode rCount num readHandle1 readHandle2 writeHadle hClose writeHadle hClose readHandle1 hClose readHandle2 mapToString::DM.Map (String,String) Int-> String mapToString m = unlines $ map eachItem itemList where itemList = DM.toList m eachItem ((x,y),i) = show x ++ "," ++ show y ++ "," ++ show i --rCount::Int -> [String] -> Handle->Handle -> IO() rCount num readHandle1 readHandle2 writeHadle = do rt <- P.fold (\x y -> DM.unionWith (+) x y) emptyMap id $ P.zipWith (\x y -> DM.singleton (x,y) 1) (P.fromHandle readHandle1) (P.fromHandle readHandle2) >-> P.take num hPutStr writeHadle $ mapToString rt main = do s<- getArgs let num = (read . head) s keyCount num

4 11

data analysis question
by Tobias Pflug 15 Nov '14

15 Nov '14

Hi, just the other day I talked to a friend of mine who works for an online radio service who told me he was currently looking into how best work with assorted usage data: currently 250 million entries as a 12GB in a csv comprising of information such as which channel was tuned in for how long with which user agent and what not. He accidentally ran into K and Q programming language (*1) which apparently work nicely for this as unfamiliar as it might seem. This certainly is not my area of expertise at all. I was just wondering how some of you would suggest to approach this with Haskell. How would you most efficiently parse such data evaluating custom queries ? Thanks for your time, Tobi [1] (http://en.wikipedia.org/wiki/K_(programming_language) [2] http://en.wikipedia.org/wiki/Q_(programming_language_from_Kx_Systems)

14 23

Re: [Haskell-cafe] Programming videogames in haskell
by sean.seefried＠gmail.com 15 Nov '14

15 Nov '14

Hi Quentin, I am currently building a game in Haskell that I plan to deploy to iOS and Android. It’s built using a combination of SDL2 and Cairo (for vector graphics) and Hipmunk (for physics via the Chipmunk C library). As in Ivan Perez's case I’m not using the Hackage versions of the SDL2 bindings. Instead I’m using the versions found on Github at [1] and [2]. I have recently managed to deploy my game to Android but this has not been a straightforward process. Building a GHC cross compiler for ARM was a time consuming process but greatly aided by the using the script found in the stable-ghc-snapshot branch of joehy’s repo on Github [3]. Building all the libraries I needed was even less straightforward. (I’ve written more about this in a post script) Deploying to iOS should be a much more straightforward process since there is official GHC support for iOS and you can download a binary distribution of the cross compiler at [4]. (Scroll down to find Luke Iannini’s distribution. Luke Iannini, Steven Blackheath put in a lot of hard work over many months to provide iOS support for GHC.) I’ve also had a play around with Helm which uses the Elerea FRP library, SDL2 and Cairo. I think it’s a pretty good little library for prototyping, although you will need to learn FRP (Functional Reactive Programming) in the process. Cheers, Sean p.s. Here is a little more detail on targeting ARM. Many of the libraries, notably the SDL2 binding, did not work out of the box with a cross compiler since they did not use Cabal’s Distribution.Simple and required that Setup.hs was compiled and run. Unfortunately this just does not work with a cross-compiler since you can’t run the code on your host machine. The work around is to build it using GHC targeting the host machine and to run that. This works okay, but a straight “cabal install” will not. A more serious problem I ran into was that the SDL2 binding requires the use of hsc2c. hsc2c has a --cross-compile flag but it does not support all of the hsc2c directives, specifically the “let” directive which is used by SDL2. [1] https://github.com/Lemmih/hsSDL2 [2] https://github.com/jdeseno/hs-sdl2-mixer [3] https://github.com/joeyh/ghc-android [4] https://www.haskell.org/ghc/download_ghc_7_8_3

1 0

Programming videogames in haskell
by Quentin Le Guennec 15 Nov '14

15 Nov '14

Hello, We're building a indie game studio with some friends. I am the lead developper and i am planning to write the game logic and graphics in haskell. I would like to know if some of you have some experience in game developpement in haskell, functionnal reactive programming in particular. Furthermore, knowing which tools/DSL's/libraries i could use and where to get good documentation could really help me. Elise Huard recently wrote an ebook about game programming in haskell, did some of you read it ? Is it worth buying it ? Thanks for your feedback. -- Quentin Le Guennec

5 4

Correct implementation of ListT besides the pipes package?
by Nicola Gigante 14 Nov '14

14 Nov '14

Hi! I understand the code on the wiki about what is wrong about the standard ListT and how to implement a correct version, but I’m wondering if there is already a package that provides such an implementation ready to use. I know the pipes package provides one, but since I don’t otherwise use pipes in my program I’d rather not depend on a big library like that only for a transformer. Is there any package that provides a correct ListT? Side question: is there a particular reason why Control.Monad.ListT doesn’t get fixed? Thank you, NIcola

2 1

Presenting at Royal Holloway Colloquium
by Dominic Steinitz 14 Nov '14

14 Nov '14

All, I have been invited to give a TED style talk (20 mins) at the Royal Holloway Hewlett Packard Information Security Colloquium: https://www.royalholloway.ac.uk/isg/externalengagement/hpday.aspx. Now I could give an uncontroversial talk about Internet banking security using triple DES, role based access control, etc. but I am thinking about being controversial (I think that is in the spirit of TED). I’d like to say that the Information Security community is solving the wrong problems by e.g. performing security audits of code, developing tools for finding buffer overflows, etc. and what they should really be doing is encouraging development in languages that prevent this sort of behaviour. E.g. if openssl were written in Haskell, heartbleed (http://en.wikipedia.org/wiki/Heartbleed) would never have happened. What do people think about this? Are there other examples I can draw on? Dominic Steinitz dominic(a)steinitz.org http://idontgetoutmuch.wordpress.com

3 4

Monad m => m (Maybe a) -> m (Maybe a) -> m (Maybe a)
by Evan Laforge 14 Nov '14

14 Nov '14

I've been using these functions lately: try :: Monad m => m (Maybe a) -> m (Maybe a) -> m (Maybe a) try action alternative = maybe alternative (return . Just) =<< action tries :: Monad m => [m (Maybe a)] -> m (Maybe a) tries = foldr try (return Nothing) It's sort of like (<|>) on Maybe, or MonadPlus, but within a monad. It seems like the sort of thing that should be already available, but hoogle shows nothing. I think 'm' has to be a monad, and I can't figure out how to generalize the Maybe to MonadPlus or Alternative. It's sort of a mirror image to another function I use a lot: justm :: Monad m => m (Maybe a) -> (a -> m (Maybe b)) -> m (Maybe b) justm op1 op2 = maybe (return Nothing) op2 =<< op1 ... which is just MaybeT for when I can't be bothered to put runMaybeT and lifts and hoists on everything. So you could say 'try' is like MaybeT with the exceptional case reversed. Is 'try' just the instantiation of some standard typeclass, or is it its own thing?

5 9

Mechanize in Haskell or Revive Shpider?
by Cody Goodman 14 Nov '14

14 Nov '14

What would be a good way to model mechanize in Haskell? For instance the python mechanize example is quite simple: def google_search(keyword): br = mechanize.Browser() br.open("https://www.google.com") br.select_form(name="gbqf") br["q"] = keyword response = br.submit() print html_to_text(response.read()) I was thinking I could just use a StateT monad transformer with the current pages html as a cursor. Then have something like: get "http://www.google.com" >>= (\c -> c $// form (Name "gbqf") >=> input (Name "q") &| submit . modifyVal "query" "Haskell") That's kinda horrible but it's a place to start. Or would I be better off trying to revive Shpider?

1 0

Fwd: Generating valid html
by Mike Meyer 14 Nov '14

14 Nov '14

On Thu, Nov 13, 2014 at 12:56 PM, Evan Laforge <qdunkan(a)gmail.com> wrote: > On Wed, Nov 12, 2014 at 2:44 PM, Wojtek Narczyński <wojtek(a)power.com.pl> > wrote: > > How to create a DSL for HTML generation, that would (statically) allow > only > > valid HTML to be generated? Parent - child relations, valid attributes > only. > > > > Or does such a DSL already exist, perhaps? > > WASH did that, a long time ago. They mentioned that they > intentionally relaxed the rules, since being totally correct was > annoying to actually use. There are docs here: > http://www2.informatik.uni-freiburg.de/~thiemann/WASH/ > Well, a lot of folks felt that any validation beyond "it works in my browser" was annoying to actually use. I, on the other hand, used NSGML in emacs and the actual DTD without much problem. But there were multiple DTDs, some of which also relaxed things because so many people felt the strict rules were annoying to actually use. It didn't seem to catch on, so maybe it was too heavy for its benefits. > Based on a decade of trying to get people to write valid HTML - so that it would work in browsers other than the most popular two or three on the most popular two or three platforms - most people don't care about "correct".. I've had people - including employees of the FSF, who I would expect otherwise from - tell me that their HTML was tested on some limited set of browser/platform pairs when I pointed out it was incorrect and hence broke on some obscure browser/platform combination. Or even on one of those browser/platform pairs with a configuration change. This also seems to be true in the Haskell community. The web platforms help catching the obvious problems you find testing against a browser. But - like most popular web platforms - break the tools that help with the creation of valid HTML Validating HTML against a DTD is much easier than doing so against browsers, but my clients never did it unless I set it up for them. Doing so helped find more than one really obscure bug, and on one occasion kept my clients web site working when a major browser tweaked how it handled broken code and broke a lot of web sites. I think this may be related to why ADTs are a hard sell. While they help a lot writing correct programs, but for just writing working programs not so much.

1 0

[ANN] Haskell FFI Tutorial
by Alex Petrov 14 Nov '14

14 Nov '14

Hi Everyone, I've recently started working on Haskell FFI Tutorial [1]. The repository already contains code that explains how to: * (expressively) represent C `struct` in Haskell code * call C code from Haskell * call Haskell code from C * operate on nested `struct`s * operate on `struct` arrays * decode `unions` * read and write C fixed-length strings and Pointer-type Strings * how to import functions from, for example, stdlib I'm still working on more story-like writeups, but the first one (about how to call Haskell from C) is already available [2]. If you have any feedback / comments / questions, know how to improve it or would like to see more things covered, just ping me. Thanks [1] https://github.com/ifesdjeen/haskell-ffi-tutorial [2] https://github.com/ifesdjeen/haskell-ffi-tutorial#calling-haskell-from-c -- Alex https://twitter.com/ifesdjeen http://clojurewerkz.org/

8 18