- Haskell-Cafe - Haskell.org

addFinalizer in GHC 7.10
by Kazu Yamamoto 31 Jan '18

31 Jan '18

Hello cafe, We, the network library maintainers, have questions about System.Mem.Weak.addFinalizer in GHC 7.10. We are trying to make the library extensible especially for the socket type and the socket address type. For this purpose, a proposed definition for Socket in version 3.0.0.0 is a just CInt: newtype Socket = Socket CInt deriving (Eq, Show) One problem is that unreachable Socket cannot be GCed if CInt (a file descriptor) is not closed. To let GC work, we are trying to add a finalizer via addFinalizer: socket family stype protocol = do fd <- c_socket ... ... let s = Socket fd addFinalizer s $ close s ruturn s This works well for many cases. Unfortunately, we *sometime* got the following error with GHC 7.10: tests/SimpleSpec.hs:56: 1) Simple.sendMany works well uncaught exception: IOException of type InvalidArgument (threadWait: invalid argument (Bad file descriptor)) It seems to us that Socket is GCed too early in GHC 7.10. We don't see this behavior in other versions of GHC. So, here are our questions: Q1) Do we use addFinalizer correctly? Q2) Is this a bug of GHC 7.10? Q3) If this is a bug of GHC 7.10, are there any workarounds? --Kazu

3 6

BayHac 2018 announcement
by Dan Burton 30 Jan '18

30 Jan '18

I am pleased to announce that BayHac 2018 will be held at the Takt office in San Francisco on the weekend of Friday, April 27 thru Sunday, April 29. BayHac is a Haskell hackathon weekend event for the San Francisco Bay Area and Silicon Valley. It will be a weekend of learning, hacking, and connecting with fellow Haskell enthusiasts. See the page for BayHac 2018 on the haskell.org wiki for details: https://wiki.haskell.org/BayHac2018 -- Dan Burton

1 0

What's the story behind "applicative" in "Constant Applicative Form"?
by sonne 29 Jan '18

29 Jan '18

Hi cafe. I've been trying to make the concept of CAFs get through to me, but the very name is cryptic enough to stupefy me. What does it mean for a thing to be "applicative"? Is it related to the concept of applicative functor (likely not)? What would a constant non-applicative form look like? A non-constant applicative form? An applicative non-form, in the end? I put a question on Stack Overflow about this, only to discover no one's really sure. I would appreciate either an answer put there right away or a permission to re-post or rephrase an answer there myself, but, if you do mind, I will of course keep the answer private to this mailing list. This is the link to the question: stackoverflow.com/questions/48489778 Thank you! P.S. I guess this is my first e-mail to this list, so please kindly let me know if it's unfit or in any way out of line.

2 1

Taking over maintainership of exact-pi
by Andrew Lelechenko 27 Jan '18

27 Jan '18

I would like to take over `exact-pi` package (https://hackage.haskell.org/package/exact-pi) Unfortunately, the package became incompatible with upcoming GHC 8.4 and needs an upgrade. The relevant pull request (https://github.com/dmcclean/exact-pi/pull/5) prepared by @konn, has been open for a month without response. I also tried to reach its maintainer Douglas McClean (cced) by email, but have not heard back. I am so interested in upgrading `exact-pi`, because it is a dependency of my package `arithmoi`. — Best regards, Andrew

3 2

LTS-10.* missing build-depends suggestion
by Rick Owens 25 Jan '18

25 Jan '18

Dear Haskell Café, I am a long-time Haskell developer with a proliferation of private projects. I typically make very heavy use of stack and LTS. Today, I tried to bootstrap my first project using LTS-10.* (with GHC-8.2.2) and I ran into an interesting problem (for me). In the past, with lts-9.* and below, when I started cranking on some code and writing a bunch of imports: > import Control.Monad.Trans.Class (MonadTrans) > import Data.Map (Map) I would try to build first without updating the cabal file build-depends, and I would get something like the following error messages: > /path/to/Module.hs:48:1: error: > Failed to load interface for ‘Data.Map’ > It is a member of the hidden package ‘containers-0.5.7.1’. > Perhaps you need to add ‘containers’ to the build-depends in your .cabal file. > Use -v to see a list of the files searched for. > > /path/to/Module.hs:21:1: error: > Failed to load interface for ‘Control.Monad.Trans.Class’ > It is a member of the hidden package ‘transformers-0.5.2.0’. > Perhaps you need to add ‘transformers’ to the build-depends in your .cabal file. > Use -v to see a list of the files searched for. This was good because the error contained a suggested solution which contained the missing package, and I could sort of lazily add the build-depends I needed without having to memorize a (Module -> Package) mapping. I even had developer tooling to do it automatically. With lts-10.*, however, the analogous error message looks like this: > /path/to/Module.hs:9:1: error: > Could not find module ‘Data.DoubleWord’ > Use -v to see a list of the files searched for. > | > 9 | import Data.DoubleWord (Word256(Word256), Word128(Word128)) > | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ I am wondering, is there a way to bring back the solution suggestion? Not having access to it has made me realize how heavily I was using that feature. What do others think about this missing bit of supplemental error information? Thanks, -Rick Owens

3 6

Multiple positions open
by Lee Pike 25 Jan '18

25 Jan '18

We have multiple positions open doing work at the intersection of Haskell, compilers, and machine learning, as well as Haskell-based infrastructure (CI, builds, etc.) Preferably Portland, Oregon or Palo Alto, California. Email me for details! Lee

1 0

Caching Actions
by Yotam Ohad 25 Jan '18

25 Jan '18

Hi, I've been digging around the source code of reactive-banana and I found this code <https://github.com/HeinrichApfelmus/reactive-banana/blob/master/reactive-ba…>: data Cached m a = Cached (m a) runCached :: Cached m a -> m a runCached (Cached x) = x -- | An action whose result will be cached. -- Executing the action the first time in the monad will -- execute the side effects. From then on, -- only the generated value will be returned. {-# NOINLINE cache #-} cache :: (MonadFix m, MonadIO m) => m a -> Cached m a cache m = unsafePerformIO $ do key <- liftIO $ newIORef Nothing return $ Cached $ do ma <- liftIO $ readIORef key -- read the cached result case ma of Just a -> return a -- return the cached result. Nothing -> mdo liftIO $ -- write the result already writeIORef key (Just a) a <- m -- evaluate return a I'm trying to understand the reasom behind the use of mdo. Can't it be like this: do a <- m liftIO $ writeIORef key (Just a) return a Removing the need for a recursive definition? Yotam

3 2

Munich Haskell Meeting, 2018-01-29 @ 19:30
by Christian Neukirchen 25 Jan '18

25 Jan '18

Dear all, Next week, our monthly Munich Haskell Meeting will take place again on Monday, January 29 at **Cafe Puck** at 19h30. For details see here: http://muenchen.haskell.bayern/dates.html If you plan to join, please add yourself to this dudle so we can reserve enough seats! It is OK to add yourself to the dudle anonymously or pseudonymously. https://dudle.inf.tu-dresden.de/haskell-munich-jan-2018/ [Please note: I'll likely arrive 8-9pm this time since I need to wait for my water meter to get changed. I'll book for Neukirchen as usual.] Everybody is welcome! cu, -- Christian Neukirchen <chneukirchen(a)gmail.com> http://chneukirchen.org

1 0

Large JSON File Processing
by erik 25 Jan '18

25 Jan '18

Hello Haskell Cafe, I have written a small, pretty simple program but I am finding it hard to reason about its behavior (and also about the best way to do what I want), so I would like to ask you all for some suggestions. For reference, here's a Stack Overflow question <https://stackoverflow.com/questions/48330690/haskell-conduit-aeson-parsing-…> where I described what's going on, but I'll also describe it below. My program does the following: 1. Recursively list a directory, 2. Parse the JSON files from the directory list into identifiable objects/records, 3. Look for matching key-value pairs, and 4. Return filenames where matches have been found. A few details for more context: - I have to filter between 500,000 and 1 million files (I'm typically trying to reduce down to between 1,000 and 40,000 that represent a particular project). I usually just need the filenames. - Each file is quite large, some of them 5mb or 10mb, and it's not uncommon for them to have deeply nested keys (40,000 keys or so). My first version of this program was simple, synchronous, and as straightforward as I could come up with. However, the memory usage increased monotonically. Profiling, I found that most of the time was spent in JSON-parsing into Objects before my code could turn the objects into records (also, as you might imagine, tons of time in garbage collection). For my second version, I switched to conduit and it seemed to solve the increasing memory issue. My core function now looked like this: conduitFilesFilter :: ProjectFilter -> Path Abs Dir -> IO [Path Abs File] conduitFilesFilter projFilter dirname' = do (_, allFiles) <- listDirRecur dirname' C.runConduit $ C.yieldMany allFiles .| C.filterMC (filterMatchingFile projFilter) .| C.sinkList This was still slow and certainly still synchronous. What I really wanted was to run that "filterMatchingFile..." part in parallel across a number of CPUs. As an aside, my filtering function looks like this: filterMatchingFile :: ProjectFilter -> Path Abs File -> IO Bool filterMatchingFile (ProjectFilter filterFunc) fpath = do let fp = toFilePath fpath bs <- B.readFile fp case validImplProject bs of -- this is pretty much just `decodeStrict` Nothing -> pure False (Just proj') -> pure $ filterFunc proj' Here are the stats from running this: 115,961,554,600 bytes allocated in the heap 35,870,639,768 bytes copied during GC 56,467,720 bytes maximum residency (681 sample(s)) 1,283,008 bytes maximum slop 145 MB total memory in use (0 MB lost due to fragmentation) Tot time (elapsed) Avg pause Max pause Gen 0 108716 colls, 108716 par 76.915s 20.571s 0.0002s 0.0266s Gen 1 681 colls, 680 par 0.530s 0.147s 0.0002s 0.0009s Parallel GC work balance: 14.99% (serial 0%, perfect 100%) TASKS: 10 (1 bound, 9 peak workers (9 total), using -N4) SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled) INIT time 0.001s ( 0.007s elapsed) MUT time 34.813s ( 42.938s elapsed) GC time 77.445s ( 20.718s elapsed) EXIT time 0.000s ( 0.010s elapsed) Total time 112.260s ( 63.672s elapsed) Alloc rate 3,330,960,996 bytes per MUT second Productivity 31.0% of total user, 67.5% of total elapsed gc_alloc_block_sync: 188614 whitehole_spin: 0 gen[0].sync: 33 gen[1].sync: 811204 I thought about writing a plainer (non-conduit) parallel version but I was afraid of the memory issue. I tried to write a Conduit-plus-channels version but it didn't work. Finally, I wrote a version using stm-conduit, which I thought might be a bit more efficient. It seems to be slightly better, but it's not really the kind of parallelization I was imagining: conduitAsyncFilterFiles :: ProjectFilter -> Path Abs Dir -> IO [String] conduitAsyncFilterFiles projFilter dirname' = do (_, allFiles) <- listDirRecur dirname' buffer 10 (C.yieldMany allFiles .| (C.mapMC (readFileWithPath . toFilePath))) (C.mapC (filterProjForFilename projFilter) .| C.filterC isJust .| C.mapC fromJust .| C.sinkList) The first conduit passed to `buffer` does something like the following: parseStrict . B.readFile. This still wasn't too great, but after reading about handing garbage collection in smarter ways, I found that I could run my application like this: stack exec search-json -- --searchPath $FILES --name hello +RTS -s -A32m -n4m And the "productivity" would shoot up quite a lot presumably because I'm doing less frequent garbage collection. My program also got a bit faster: 36,379,265,096 bytes allocated in the heap 1,238,438,160 bytes copied during GC 22,996,264 bytes maximum residency (85 sample(s)) 3,834,152 bytes maximum slop 207 MB total memory in use (14 MB lost due to fragmentation) Tot time (elapsed) Avg pause Max pause Gen 0 211 colls, 211 par 1.433s 0.393s 0.0019s 0.0077s Gen 1 85 colls, 84 par 0.927s 0.256s 0.0030s 0.0067s Parallel GC work balance: 67.93% (serial 0%, perfect 100%) TASKS: 10 (1 bound, 9 peak workers (9 total), using -N4) SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled) INIT time 0.001s ( 0.004s elapsed) MUT time 12.636s ( 12.697s elapsed) GC time 2.359s ( 0.650s elapsed) EXIT time -0.015s ( 0.003s elapsed) Total time 14.982s ( 13.354s elapsed) Alloc rate 2,878,972,840 bytes per MUT second Productivity 84.2% of total user, 95.1% of total elapsed gc_alloc_block_sync: 9612 whitehole_spin: 0 gen[0].sync: 2044 gen[1].sync: 47704 Thanks for reading thus far. I now have three questions. 1. I understand that my program necessarily creates tons of garbage because it parses and then throws away 5mb of JSON 500,000 times. However, I don't really understand why this helps "+RTS -A32m -n4m" and I'm always reluctant to sprinkle in magic I don't fully understand. Can anyone help me understand what this means? 2. It seems that the allocation limit is really something I should be using, but I can't figure out how to successfully add it to my package.yml with the other options. From the documentation for GHC 8.2, I thought it needed to look like this but it never works, usually telling me that -A32m and -n4m are not recognizable flags (how do I add them in to my package.yml so I don't have to pass them when running the program?): ghc-options: - -threaded - -rtsopts - "-with-rtsopts=-N4 -A32m -n4m" 3. Finally, the most important question I have is this. When I run this program on OSX, it runs successfully through to completion. However, *a few minutes after terminating*, my terminal becomes unresponsive. I use emacs for my editor, typically launched from a terminal window and that too becomes unresponsive. This is not a typical outcome for any programs I write and it happens *every time* I run this particular application, so I know that this application is to blame. The crazy thing is that force quitting the terminal or logging out doesn't help: I have to actually restart my computer to use the terminal application again. Other details that may help: - This crash happens after the process id for my program has terminated. - Watching its progress in HTOP, it never comes close to running out of memory: the value hovers in the same place. I can't really deploy an application that has this potential-crashing problem, but I don't know to debug this issue. My total stab-in-the-dark idea is that heap allocations somehow are unrecoverable even after the process has terminated? Can anyone offer suggestions on things to look for or ways to debug and/or fix this issue? Finally, if anyone has suggestions on better ways to structure my application or parallelize the slow parts, I'll happily take those. Thanks again for reading. I appreciate any suggestions you may have. Best, -- Erik Aker

2 5

I really don't understand this
by Jean-Marc Alliot 24 Jan '18

24 Jan '18

I apologize if my question is stupid, but here is a simple Haskell program which never stops. However, if I comment the line (which is apparently useless): v<-acc The program works like a charm... I am using GHC 8.0.2 (debian sid) and multiset 0.3.3. If anyone has an idea, I would gladly hear it. Thanks in advance PS: don't try to understand what the program is doing; it is just the reduction to a few lines of a much larger code; I have tried to find a smaller subset which is not working "properly". PPS: The program can also be downloaded from: http://www.alliot.fr/tmp/example.hs import qualified Data.IntMultiSet as IMS b :: IMS.IntMultiSet b = IMS.fromList ([1, 2, 3, 4]) search :: IMS.IntMultiSet -> Int -> IO Bool search mynumbers nb = ins mynumbers (return False) where ins numbers acc = do v <- acc IMS.fold (\x acc1 -> let numbers2 = IMS.delete x numbers in IMS.fold (\y acc2 -> let numbers3 = IMS.delete y numbers2 res = x + y in if res == nb then (return True) else ins (IMS.insert res numbers3) acc2) acc1 numbers2) acc numbers main = do v <- search b 999999999 print v

2 5