- Haskell-Cafe - Haskell.org

Caching Actions
by Yotam Ohad 25 Jan '18

25 Jan '18

Hi, I've been digging around the source code of reactive-banana and I found this code <https://github.com/HeinrichApfelmus/reactive-banana/blob/master/reactive-ba…>: data Cached m a = Cached (m a) runCached :: Cached m a -> m a runCached (Cached x) = x -- | An action whose result will be cached. -- Executing the action the first time in the monad will -- execute the side effects. From then on, -- only the generated value will be returned. {-# NOINLINE cache #-} cache :: (MonadFix m, MonadIO m) => m a -> Cached m a cache m = unsafePerformIO $ do key <- liftIO $ newIORef Nothing return $ Cached $ do ma <- liftIO $ readIORef key -- read the cached result case ma of Just a -> return a -- return the cached result. Nothing -> mdo liftIO $ -- write the result already writeIORef key (Just a) a <- m -- evaluate return a I'm trying to understand the reasom behind the use of mdo. Can't it be like this: do a <- m liftIO $ writeIORef key (Just a) return a Removing the need for a recursive definition? Yotam

3 2

Munich Haskell Meeting, 2018-01-29 @ 19:30
by Christian Neukirchen 25 Jan '18

25 Jan '18

Dear all, Next week, our monthly Munich Haskell Meeting will take place again on Monday, January 29 at **Cafe Puck** at 19h30. For details see here: http://muenchen.haskell.bayern/dates.html If you plan to join, please add yourself to this dudle so we can reserve enough seats! It is OK to add yourself to the dudle anonymously or pseudonymously. https://dudle.inf.tu-dresden.de/haskell-munich-jan-2018/ [Please note: I'll likely arrive 8-9pm this time since I need to wait for my water meter to get changed. I'll book for Neukirchen as usual.] Everybody is welcome! cu, -- Christian Neukirchen <chneukirchen(a)gmail.com> http://chneukirchen.org

1 0

Large JSON File Processing
by erik 25 Jan '18

25 Jan '18

Hello Haskell Cafe, I have written a small, pretty simple program but I am finding it hard to reason about its behavior (and also about the best way to do what I want), so I would like to ask you all for some suggestions. For reference, here's a Stack Overflow question <https://stackoverflow.com/questions/48330690/haskell-conduit-aeson-parsing-…> where I described what's going on, but I'll also describe it below. My program does the following: 1. Recursively list a directory, 2. Parse the JSON files from the directory list into identifiable objects/records, 3. Look for matching key-value pairs, and 4. Return filenames where matches have been found. A few details for more context: - I have to filter between 500,000 and 1 million files (I'm typically trying to reduce down to between 1,000 and 40,000 that represent a particular project). I usually just need the filenames. - Each file is quite large, some of them 5mb or 10mb, and it's not uncommon for them to have deeply nested keys (40,000 keys or so). My first version of this program was simple, synchronous, and as straightforward as I could come up with. However, the memory usage increased monotonically. Profiling, I found that most of the time was spent in JSON-parsing into Objects before my code could turn the objects into records (also, as you might imagine, tons of time in garbage collection). For my second version, I switched to conduit and it seemed to solve the increasing memory issue. My core function now looked like this: conduitFilesFilter :: ProjectFilter -> Path Abs Dir -> IO [Path Abs File] conduitFilesFilter projFilter dirname' = do (_, allFiles) <- listDirRecur dirname' C.runConduit $ C.yieldMany allFiles .| C.filterMC (filterMatchingFile projFilter) .| C.sinkList This was still slow and certainly still synchronous. What I really wanted was to run that "filterMatchingFile..." part in parallel across a number of CPUs. As an aside, my filtering function looks like this: filterMatchingFile :: ProjectFilter -> Path Abs File -> IO Bool filterMatchingFile (ProjectFilter filterFunc) fpath = do let fp = toFilePath fpath bs <- B.readFile fp case validImplProject bs of -- this is pretty much just `decodeStrict` Nothing -> pure False (Just proj') -> pure $ filterFunc proj' Here are the stats from running this: 115,961,554,600 bytes allocated in the heap 35,870,639,768 bytes copied during GC 56,467,720 bytes maximum residency (681 sample(s)) 1,283,008 bytes maximum slop 145 MB total memory in use (0 MB lost due to fragmentation) Tot time (elapsed) Avg pause Max pause Gen 0 108716 colls, 108716 par 76.915s 20.571s 0.0002s 0.0266s Gen 1 681 colls, 680 par 0.530s 0.147s 0.0002s 0.0009s Parallel GC work balance: 14.99% (serial 0%, perfect 100%) TASKS: 10 (1 bound, 9 peak workers (9 total), using -N4) SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled) INIT time 0.001s ( 0.007s elapsed) MUT time 34.813s ( 42.938s elapsed) GC time 77.445s ( 20.718s elapsed) EXIT time 0.000s ( 0.010s elapsed) Total time 112.260s ( 63.672s elapsed) Alloc rate 3,330,960,996 bytes per MUT second Productivity 31.0% of total user, 67.5% of total elapsed gc_alloc_block_sync: 188614 whitehole_spin: 0 gen[0].sync: 33 gen[1].sync: 811204 I thought about writing a plainer (non-conduit) parallel version but I was afraid of the memory issue. I tried to write a Conduit-plus-channels version but it didn't work. Finally, I wrote a version using stm-conduit, which I thought might be a bit more efficient. It seems to be slightly better, but it's not really the kind of parallelization I was imagining: conduitAsyncFilterFiles :: ProjectFilter -> Path Abs Dir -> IO [String] conduitAsyncFilterFiles projFilter dirname' = do (_, allFiles) <- listDirRecur dirname' buffer 10 (C.yieldMany allFiles .| (C.mapMC (readFileWithPath . toFilePath))) (C.mapC (filterProjForFilename projFilter) .| C.filterC isJust .| C.mapC fromJust .| C.sinkList) The first conduit passed to `buffer` does something like the following: parseStrict . B.readFile. This still wasn't too great, but after reading about handing garbage collection in smarter ways, I found that I could run my application like this: stack exec search-json -- --searchPath $FILES --name hello +RTS -s -A32m -n4m And the "productivity" would shoot up quite a lot presumably because I'm doing less frequent garbage collection. My program also got a bit faster: 36,379,265,096 bytes allocated in the heap 1,238,438,160 bytes copied during GC 22,996,264 bytes maximum residency (85 sample(s)) 3,834,152 bytes maximum slop 207 MB total memory in use (14 MB lost due to fragmentation) Tot time (elapsed) Avg pause Max pause Gen 0 211 colls, 211 par 1.433s 0.393s 0.0019s 0.0077s Gen 1 85 colls, 84 par 0.927s 0.256s 0.0030s 0.0067s Parallel GC work balance: 67.93% (serial 0%, perfect 100%) TASKS: 10 (1 bound, 9 peak workers (9 total), using -N4) SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled) INIT time 0.001s ( 0.004s elapsed) MUT time 12.636s ( 12.697s elapsed) GC time 2.359s ( 0.650s elapsed) EXIT time -0.015s ( 0.003s elapsed) Total time 14.982s ( 13.354s elapsed) Alloc rate 2,878,972,840 bytes per MUT second Productivity 84.2% of total user, 95.1% of total elapsed gc_alloc_block_sync: 9612 whitehole_spin: 0 gen[0].sync: 2044 gen[1].sync: 47704 Thanks for reading thus far. I now have three questions. 1. I understand that my program necessarily creates tons of garbage because it parses and then throws away 5mb of JSON 500,000 times. However, I don't really understand why this helps "+RTS -A32m -n4m" and I'm always reluctant to sprinkle in magic I don't fully understand. Can anyone help me understand what this means? 2. It seems that the allocation limit is really something I should be using, but I can't figure out how to successfully add it to my package.yml with the other options. From the documentation for GHC 8.2, I thought it needed to look like this but it never works, usually telling me that -A32m and -n4m are not recognizable flags (how do I add them in to my package.yml so I don't have to pass them when running the program?): ghc-options: - -threaded - -rtsopts - "-with-rtsopts=-N4 -A32m -n4m" 3. Finally, the most important question I have is this. When I run this program on OSX, it runs successfully through to completion. However, *a few minutes after terminating*, my terminal becomes unresponsive. I use emacs for my editor, typically launched from a terminal window and that too becomes unresponsive. This is not a typical outcome for any programs I write and it happens *every time* I run this particular application, so I know that this application is to blame. The crazy thing is that force quitting the terminal or logging out doesn't help: I have to actually restart my computer to use the terminal application again. Other details that may help: - This crash happens after the process id for my program has terminated. - Watching its progress in HTOP, it never comes close to running out of memory: the value hovers in the same place. I can't really deploy an application that has this potential-crashing problem, but I don't know to debug this issue. My total stab-in-the-dark idea is that heap allocations somehow are unrecoverable even after the process has terminated? Can anyone offer suggestions on things to look for or ways to debug and/or fix this issue? Finally, if anyone has suggestions on better ways to structure my application or parallelize the slow parts, I'll happily take those. Thanks again for reading. I appreciate any suggestions you may have. Best, -- Erik Aker

2 5

I really don't understand this
by Jean-Marc Alliot 25 Jan '18

25 Jan '18

I apologize if my question is stupid, but here is a simple Haskell program which never stops. However, if I comment the line (which is apparently useless): v<-acc The program works like a charm... I am using GHC 8.0.2 (debian sid) and multiset 0.3.3. If anyone has an idea, I would gladly hear it. Thanks in advance PS: don't try to understand what the program is doing; it is just the reduction to a few lines of a much larger code; I have tried to find a smaller subset which is not working "properly". PPS: The program can also be downloaded from: http://www.alliot.fr/tmp/example.hs import qualified Data.IntMultiSet as IMS b :: IMS.IntMultiSet b = IMS.fromList ([1, 2, 3, 4]) search :: IMS.IntMultiSet -> Int -> IO Bool search mynumbers nb = ins mynumbers (return False) where ins numbers acc = do v <- acc IMS.fold (\x acc1 -> let numbers2 = IMS.delete x numbers in IMS.fold (\y acc2 -> let numbers3 = IMS.delete y numbers2 res = x + y in if res == nb then (return True) else ins (IMS.insert res numbers3) acc2) acc1 numbers2) acc numbers main = do v <- search b 999999999 print v

2 5

Coding Haskell in a music start-up?
by Bas de Haas 24 Jan '18

24 Jan '18

Dear Haskellers, Chordify is hiring! Chordify is a young and fast growing music e-learning platform that helps musicians to play their favorite music. We automatically analyse the chords of a piece of music and display them in an intuitive player. Try it yourself at: https://chordify.net/ or download one of our apps https://chordify.net/app The cool thing is: our backend serving our apps and website has been written almost exclusively in Haskell. We hope to broaden our team with a functional programmer and a front-end developer. We are looking for people who are pro-active, independent, and creative to improve Chordify. You’d work in a small team of dedicated people with various backgrounds in which you can make a difference. If you are interested in working at Chordify, you can find the details here: https://chordify.homerun.co/ All the best, Bas de Haas

1 0

SLURP: a single unified registry for Haskell packages
by Simon Peyton Jones 24 Jan '18

24 Jan '18

Friends Hackage has been extraordinarily successful as a single repository through which to share Haskell packages. It has supported the emergence of variety of tools to locate Haskell packages, build them and install them (cabal-install, Stack, Nix, ...). But in recent years there has been increasing friction over, * Hackage's policies, especially concerning version bounds; * Hackage's guarantees, especially around durability of package content and metadata; * Hackage's features, especially the visual presentation and package documentation. If we do not resolve this friction, it seems likely that the Haskell library ecosystem will soon "fork", with two separate repositories, one optimised for Cabal and one for Stack. This would be extremely counter-productive for Haskell users. Thus motivated, over the last few months we have talked a lot to colleagues, including ones in the Hackage and Stack communities. We have emerged with SLURP, a proposal that could go a long way towards supporting the upsides of a diverse ecosystem, without the sad downsides of forking into mutually-exclusive sub-communities. Here is the SLURP proposal<https://github.com/simonmar/ecosystem-proposals/blob/slurp/proposals/0000-s…>. We invite the Haskell community to debate it. SLURP is meant to enable both Hackage and Stackage (and perhaps more services in the future) to in the future make choices autonomously without hurting other package services. But it will only work if the implementors of both Hackage and Stackage are willing to participate. We respect their autonomy in this matter, but we urge them to give this proposal serious consideration in the best interests of the community and Haskell's success. We have carefully designed SLURP to be as minimal and non-invasive as possible, so that it can be adopted without much trouble. Of course, we are open to debate about the specific details. We do have an offer from someone willing to implement SLURP. We also strongly urge members of the community to express clear views about the importance --- or otherwise --- of adopting something like SLURP. You are, after all, the community that GHC, Hackage, Stackage, Cabal, etc are designed to serve, so your views about what best meets your needs are critically important. Mathieu Boespflug (@mboes<https://github.com/boes>) Manuel Chakravarty (@mchakravarty<https://github.com/mchakravarty>) Simon Marlow (@simonmar<https://github.com/simonmar>) Simon Peyton Jones (@simonpj<https://github.com/simonpj>) Alan Zimmerman (@alanz<https://github.com/alanz>)

4 3

ANN: containers-0.5.11.0
by David Feuer 24 Jan '18

24 Jan '18

We have just released containers version 0.5.11.0. This is the first release since Matt Renaud joined the maintenance team. Matt has focused on documentation, code quality, and testing, and made some substantial contributions in these areas. This release comes with quite a few new functions and instances, written by quite a few different contributors. It also includes a complete overhaul of the sorting machinery in Data.Sequence thanks to Donnacha Oisín Kidney. And Matt Renaud has launched a brand new set of tutorial documentation at https://haskell-containers.readthedocs.io/en/latest/ Many thanks to all the contributors who made this release possible! The complete changelog can be found below. David Feuer Matt Renaud Wren Romano New functions and class instances * Add a `MonadFix` instance for `Data.Sequence`. * Add a `MonadFix` instance for `Data.Tree`. * Add `powerSet`, `cartesianProduct`, and `disjointUnion` for `Data.Set`. (Thanks, Edward Kmett.) * Add `disjoint` for `Data.Set` and `Data.IntSet`. (Thanks, Víctor López Juan.) * Add `lookupMin` and `lookupMax` to `Data.IntMap`. (Thanks, bwroga.) * Add `unzip` and `unzipWith` to `Data.Sequence`. Make unzipping build its results in lockstep to avoid certain space leaks. * Add carefully optimized implementations of `sortOn` and `unstableSortOn` to `Data.Sequence`. (Thanks, Donnacha Oisín Kidney.) Changes to existing functions and features * Make `Data.Sequence.replicateM` a synonym for `replicateA` for post-AMP `base`. * Rewrite the `IsString` instance head for sequences, improving compatibility with the list instance and also improving type inference. We used to have instance IsString (Seq Char) Now we commit more eagerly with instance a ~ Char => IsString (Seq a) * Make `>>=` for `Data.Tree` strict in the result of its second argument; being too lazy here is almost useless, and violates one of the monad identity laws. Specifically, `return () >>= \_ -> undefined` should always be `undefined`, but this was not the case. * Harmonize laziness details for `minView` and `maxView` between `Data.IntMap` and `Data.Map`. Performance improvement * Speed up both stable and unstable sorting for `Data.Sequence` (Thanks, Donnacha Oisín Kidney.) Other changes * Update for recent and upcoming GHC and Cabal versions (Thanks, Herbert Valerio Reidel, Simon Jakobi, and Ryan Scott.) * Improve external and internal documentation (Thanks, Oleg Grenrus and Benjamin Hodgson.) * Add tutorial-style documentation. * Add Haddock `@since` annotations for changes made since version 0.5.4 (Thanks, Simon Jakobi.) * Add a (very incomplete) test suite for `Data.Tree`. * Add structural validity checks to the test suites for `Data.IntMap` and `Data.IntSet` (Thanks to Joachim Breitner for catching an error in a first draft.)

2 1

[ANNOUNCE] GHC 8.4.1-alpha1 available
by Ben Gamari 22 Jan '18

22 Jan '18

The GHC development team is pleased to announce the first alpha release of the 8.4.1 release. The usual release artifacts are available from https://downloads.haskell.org/~ghc/8.4.1-alpha1 Note that this release drops compatibility with GCC 4.6 and earlier. While we generally try to place as few constraints on system toolchain as possible, this release depends upon the __atomic__ builtins provided by GCC 4.7 and later (see #14244). === Notes on release scheduling === The 8.4.1 release marks the first release where GHC will be adhering to its new, higher-cadence release schedule [1]. Under this new scheme, major releases will be made in 6-month intervals with interstitial minor releases as necessary. In order to minimize the likelihood of schedule slippage and to ensure adequate testing, each major release will be preceeded by a number of regular alpha releases. We will begin issuing these releases roughly three months before the final date of the major release and will issue roughly one every two weeks during this period. This high release cadence will allow us to quickly get fixes in to users hands and allow better feedback on the status of the release. GHC 8.4 is slated to be released in mid-February but, due to technical constraints, we are starting the alpha-release cycle a bit later than planned under the above schedule. For this reason, it would be greatly appreciated if users could put this alpha through its paces to make up for lost time. As always, do let us know if you encounter any trouble in the course of testing. Thanks for your help! Cheers, - Ben [1] https://ghc.haskell.org/trac/ghc/blog/2017-release-schedule

3 4

FRP: Why is Behavior not a Monad?
by martin 21 Jan '18

21 Jan '18

Hello all, it appears to be difficult to define a Monad instance for Behavior and in many (if not all) reactive libraries Behavior is not a Monad. Why is that so? If a Beahvior is conceptionally a function from Time to something, newtype Behavior a = Behavior (Time -> a) then its Monad instance should work like a Reader Monad. Alternatively one can look at the join function join :: Behavior (Behavior a) -> Behavior a. This corresponds to the situation where you switch between Channels on a TV remote controle and each Channel is a behavior of Image. Then join Behavior (Behavior image) should give you the video stream you see on the screen. How can one possibly live without this operation? I assume this has something to do with the actual implementation and not so much with the semantics, because semantically I don't see any problems. So, - why is Behavior not a Monad and - how would you express the TV-remote example without a Monad?

4 3

How to use FRP for Simulation
by martin 20 Jan '18

20 Jan '18

Hello all, the answer to the question "what is FRP good for?" often includes "simulation". I somewhat understand the "reactive" situation, where an FRP system receives Events from the outside world and computes Events to be sent to the outside world. To do so, the computation uses Behaviors which are essentially time-varying states. I also read, that the programmer should not be allowed to literally define Events and Behaviors, or she might end up writing bad prograns which exhibit time or space leaks. She should only be allowed to define transformations. Now in a discrete event simulation (DES) there is a lot of "event stuff" going on and FRP looks like a good paradigm. However, in a simulation nothing comes from the outside world and not much goes to the outside world. There is a simulation result, but it is just a result like any other and doesn't have much to do with Time. A DES is not really a reactive system at all. In DES there is typically a set of initial Events and more events are created as the simulation progresses. I cannot see how this can be done without the power to explicitly create Events. Also these Events point to the "future" and must therefore carry an explicit (simulated) Time, whereas (IIRC) in FRP, Events do not necessarily carry an explicit Time. They just occur, and if you need the time, you look at the clock (a Behavior Time). So what exactly is meant by "FRP is useful for simulation". Does it mean that a DES engine can be written in FRP style or does it mean something else? If one can indeed write a DES engine in FPR style, what is the correct way of thinking about it?

1 0