November 2005 - Haskell-Cafe

RE: [Haskell-cafe] Trashing of memory: How to troubleshoot and fix?
by Simon Marlow 07 Nov '05

07 Nov '05

On 05 November 2005 10:10, Joel Reymont wrote: > This is a I came up with a "scripting" environment for poker server > people to excercise their server. All it basically does is compose > and parse binary packets and let scripters send and receive them. > > The issue I have is that the "script" runs fine on Mac OSX (1Gb of > memory, though 6Gb VM) and runs out of memory on a Windows machine > with 512Mb of memory and 768Mb of VM. The memory error is from malloc > which leads me to believe memory fragmentation could be an issue. > Running on Windows is a customer requirement. GHC very rarely calls malloc(), so this is likely to be caused by heavy memory use in the C++ part of your app. It would be interesting to find out though: you can run gdb on the binary and get a backtrace when the memory error occurs (compiling the Haskell app with -debug will help). > I was wondering if peak memory usage was the issue but jugding from > the GC stats it does not seem to be that high. Maybe I'm wrong. What > do you guys make of this? > > ./logon +RTS -p -sblah > > GC summary: > -- > 5,695,379,620 bytes allocated in the heap > 1,274,038,800 bytes copied during GC > 69,790,544 bytes maximum residency (53 sample(s)) > > 20433 collections in generation 0 (230.29s) > 53 collections in generation 1 ( 5.15s) > > 152 Mb total memory in use > > INIT time 0.00s ( 0.04s elapsed) > MUT time 91.13s (250.79s elapsed) > GC time 235.44s (607.31s elapsed) > RP time 0.00s ( 0.00s elapsed) > PROF time 0.00s ( 0.00s elapsed) > EXIT time 0.00s ( 0.00s elapsed) > Total time 326.57s (858.14s elapsed) GC load is *very* high. But as you can see, GHC's storage manager is only using 152Mb peak - if you're running out with 512Mb, then something else is using the rest. To improve performance, I would first try to optimise the program to reduce the allocation rate, get any low-hanging fruit, and then tweak the GC settings. Most of your GC is generation 0 collections, which suggests that your program is allocating a lot of temporary memory and not getting much work done. Also, note that you got these figures from a version of the program compiled for profiling, which will have higher memory requirements than normal due to the profiling overheads. > --- > Time allocation and profiling report: > ---- > total time = 85.58 secs (4279 ticks @ 20 ms) > total alloc = 2,941,143,704 bytes (excludes profiling > overheads) > > COST CENTRE MODULE %time %alloc > > exp/evalU/cmd Script.Engine 50.0 66.4 > connect Main 14.1 11.1 > CAF Script.Engine 10.3 5.4 > exp/if Script.Engine 8.8 4.8 > send Main 6.4 6.7 > expect Main 5.8 2.7 > exp/evalU/sz Script.Engine 1.4 1.4 > startSSL Script.Engine 1.3 0.1 > --- Seems like you need to look at evalU: > cmd' <- {-# SCC "exp/evalU/cmd" #-}return $ evalU packet appU > -- unpickle command this is where most of the time/alloc seems to be going. Also, just looking at this line of code reminds me of SimonM's 203rd law of Haskell optimisation: when writing 'return $ E', consider writing 'return $! E' instead. Cheers, Simon

2 1

RE: [Haskell-cafe] Re: FPS: Finalizers not running (was Memory usageoutside of the Haskell heap)
by Simon Marlow 07 Nov '05

07 Nov '05

On 06 November 2005 18:38, Joel Reymont wrote: > You need to do something like this to force them to run, apparently > > main = do foo $ Prelude.concat $ replicate 100 data_ > > Just triggering the GC with performGC doesn't do it. > > The really odd thing is that if I setup a C finalizer (see below) > then it's triggered once and the whole thing crashes with a bus > error. Duncan reported that it segfaulted for him as well. > > installFinalizer fps(a)(P.PS fp _ _) = > addForeignPtrFinalizer finalizer fp > > foreign import ccall unsafe "finalizer" finalizer :: FinalizerPtr > Word8 > > where the finalizer itself is > > #include <stdio.h> > > void finalizer(char *p) { > printf("--- Finalized: %x\n", p); > } This crash is due to driver error, I'm afraid. You should have written: foreign import ccall unsafe "&finalizer" finalizer :: FinalizerPtr Word8 (note the "&"). Cheers, Simon

1 0

Memory usage outside of the Haskell heap
by Joel Reymont 07 Nov '05

07 Nov '05

Folks, How can I profile memory usage outside of the Haskell heap? I'm using FastStrings for everything and some folks on #haskell suspect it might be causing my problems. hGet :: Handle -> Int -> IO FastString hGet _ 0 = return empty hGet h i = do fp <- mallocForeignPtr i l <- withForeignPtr fp $ \p-> hGetBuf h p i return $ PS fp 0 l I do this a lot and the finalizers might not be running or something like that. Thanks, Joel -- http://wagerlabs.com/

6 13

RE: [Haskell-cafe] Re: FPS: Finalizers not running (was Memory usageoutside of the Haskell heap)
by Simon Marlow 07 Nov '05

07 Nov '05

Finalizers aren't guaranteed to be run. In particular, if the main thread exits, then we don't run any outstanding finalizers. This change was made recently, but it turned out that even prior to 6.4 we couldn't guarantee to run all outstanding finalizers. Does this explain it, or is there something else going on? BTW, when you addForeignPtrConcFinalizer to a ForeignPtr created with mallocForeignPtr, you're *creating* a finalizer, it doesn't have one to start with. It's pretty expensive to do this. Cheers, Simon On 07 November 2005 01:42, Donald Bruce Stewart wrote: > joelr1: >> David, >> >> I followed your suggestion and I think the finalizers for FPS are not >> running. > > Here are some experiments. Sometimes I can get the finalizers to run > as > expected, but only when putting memory pressure on the Haskell heap > with > normal haskell values. Works with 6.4.1 and 6.5 on OpenBSD or Linux. > > So here we generate a bunch of lists inside Haskell, then pack them. > Our > finalisers get run nicely as expected: > > import qualified Data.FastPackedString as P > import Foreign.Concurrent > import Data.Word > import Data.Char > > data_ = [ replicate 5000 i | i <- [1..100] ] :: [[Word8]] > > main = do > foo data_ > putStrLn "" > > foo [] = return () > foo (x:xs) = do > let fps = P.packWords x > installFinalizer fps > putStr "." > foo xs > > installFinalizer fps(a)(P.PS fp _ _) = > addForeignPtrFinalizer fp $ putStr (show.ord.P.head $ fps) > > > pill00$ ./a.out > 100 > > ..................................................1234567891011121314151 617181920212223242526272829303132333435363738394041424344454647484950... ........................................51525354555657585960616263646566 6768697071727374757677787980818283848586878889909192....... > > Now, if we instead allocate only outside the Haskell heap (with an > mmap packed > string), then we see something different: > > import qualified Data.FastPackedString as P > import Foreign.Concurrent > import Data.Word > import Data.Char > > main = do > foo 100 > putStrLn "" > > foo 0 = return () > foo n = do > fps <- P.mmapFile "/home/dons/tmp/tests/128k" > installFinalizer fps > putStr "." > foo (n-1) > > installFinalizer fps(a)(P.PS fp _ _) = > addForeignPtrFinalizer fp $ putStr (show.ord.P.head $ fps) > > No finalisers (even though more data is allocated)! > > $ ./a.out > > ........................................................................ .............. > > The GHC rts doesn't know how big the packed string is. It just sees > the few > bytes of the packed string (as dcoutts pointed out on irc). > > Strangely, the same effect is seen with P.readFile, which (from > darcs) allocates as so: > > mallocForeignPtr :: Int -> IO (ForeignPtr Word8) > mallocForeignPtr l = when (l > 1000000) performGC >> > mallocForeignPtrArray l > > where here mallocForeignPtrArray ends up calling > GHC.Prim.newPinnedByteArray# (i.e. on the Haskell heap). Hmm. I don't > understand that then, as the memory is > in the Haskell heap, yet still no finalisers are run. > > If we do something expensive with the packed string, then finalizers > get run: > > foo n = do > fps <- P.readFile "/home/dons/tmp/tests/128k" > installFinalizer fps > > -- unpack the string into a normal Haskell list: > forkIO (Control.Exception.evaluate ((length . P.unpack) fps) > >> return ()) putStr "." > foo (n-1) > > Also interestingly, calling unsafeFinalize didn't help. > > Simon, any thoughts on this? Is there some way we can ensure memory > outside the Haskell heap gets counted for its full weight? Also, why > don't we see any finalizers run with the pinned arrays? > > -- Don > _______________________________________________ > Haskell-Cafe mailing list > Haskell-Cafe(a)haskell.org > http://www.haskell.org/mailman/listinfo/haskell-cafe

2 1

RE: [Haskell-cafe] Re: Fixup too large
by Simon Marlow 07 Nov '05

07 Nov '05

On 04 November 2005 20:06, Joel Reymont wrote: > I was able to work around this by adding -O > > On Nov 4, 2005, at 6:57 PM, Joel Reymont wrote: > >> What does this mean exactly? >> >> ghc -prof --make -auto-all -o logon -package Script Logon.hs >> Chasing modules from: Logon.hs >> Compiling Main ( Logon.hs, Logon.o ) >> /tmp/ghc20361.s:17881:Fixup of 93184 too large for field width of 16 >> bits /tmp/ghc20361.s:17878:Fixup of 93196 too large for field width >> of 16 bits /tmp/ghc20361.s:17874:Fixup of 93160 too large for field >> width of 16 bits Can you send us the offending code? Cheers, Simon

1 0

Type classes and hFoldr from HList
by Greg Buchholz 06 Nov '05

06 Nov '05

I was playing around with the HList library from the paper... Strongly typed heterogeneous collections http://homepages.cwi.nl/~ralf/HList/ ...and I thought I'd try to fold the composition function (.) through a heterogeneous list of functions, using hFoldr... >{-# OPTIONS -fglasgow-exts #-} >{-# OPTIONS -fallow-undecidable-instances #-} > >import CommonMain > >main = print $ comp "abc" > >test = HCons ((+1)::(Int->Int)) (HCons ((*2)::(Int->Int)) (HCons length HNil)) > >comp = hFoldr (.) id test > >instance Apply (a -> b -> c -> d) (a, b) (c -> d) > where > apply f (a,b) = f a b ...but it fails with the following type error... ]Compiling Main ( compose.hs, interpreted ) ] ]compose.hs:10:7: ] No instances for (Apply ((b -> c) -> (a -> b) -> a -> c) ] (Int -> Int, r) ] ([Char] -> a3), ] Apply ((b -> c) -> (a -> b) -> a -> c) (Int -> Int, r1) r, ] Apply ((b -> c) -> (a -> b) -> a -> c) ([a2] -> Int, a1 ->a1) r1) ] arising from use of `hFoldr' at compose.hs:10:7-12 ] Probable fix: ] add an instance declaration for (Apply ((b -> c) -> (a -> b) -> a -> c) ] (Int -> Int, r) ] ([Char] -> a3), ] Apply ((b -> c) -> (a -> b) -> a -> c) ](Int -> Int, r1) r, ] Apply ((b -> c) -> (a -> b) -> a -> c) ]([a2] -> Int, a1 -> a1) r1) ] In the definition of `comp': comp = hFoldr (.) id test ...Anyway, I couldn't quite tell whether I was using hFoldr incorrectly, or if I needed to have more constraints placed on the construction of "test", or if needed some sort of type-level function that resolves... Apply ((b -> c) -> (a -> b) -> a -> c) ...into (a -> c), or something else altogether. I figured someone might be able to help point me in the right direction. Thanks, Greg Buchholz

1 0

Families of type classes
by Klaus Ostermann 06 Nov '05

06 Nov '05

Hi all, I am not a Haskell expert, and I am currently exploring type classes and need some advice. I want to define a family of mutually recursive types as a collection of type classes and then I want to be able to map these collections of types to a set of other types using instance declarations. For example, I have a type family for graphs, consisting of the types "Node" and "Edge". In another part of my application I have the types "Person" and "Likes" (a pair of persons), and I want to map the roles "Node" and "Edge" to "Person" and "Likes", respectively. It seems to me that functional dependencies could be a way to model it (maybe it can also be done much simpler, but I don't know how). Here is what I tried: class (Node n, Edge e) => Graph g n e | g -> n, g -> e where class Node n where isConnectedTo :: Graph g n e => g -> n -> e -> Bool class Edge e where n1 :: Graph g n e => g -> e -> n n2 :: Graph g n e => g -> e -> n type Person = String type Likes = (Person, Person) data DummyGraph = DummyGraph String instance Graph DummyGraph Person Likes where instance Node Person where isConnectedTo g n (p1,p2) = (p1 == n) || (p2 == n) instance Edge Likes where n1 g (p1,p2) = p1 n2 g (p1,p2) = p2 This "DummyGraph" thing is supposed to be used as a kind of "family object" which stands for a particular type class family. However, this is not yet quite right because I get the error message Couldn't match the rigid variable `e' against `(a, b)' `e' is bound by the type signature for `isConnectedTo' Expected type: e Inferred type: (a, b) When checking the pattern: (p1, p2) In the definition of `isConnectedTo': isConnectedTo g n (p1, p2) = (p1 == n) || (p2 == n) Similar error messages occur in the instance declaration for Edge/Likes. I don't understand exactly what my error is. Maybe I would need a completely different strategy to model this. Any help would be appreciated! Regards, Klaus

4 5

Information on B-Tree IO implemenations
by Scott Weeks 06 Nov '05

06 Nov '05

Hello all, I'm a newbie and while I grasp some of the basics of FP I have yet to really understand the guts of Haskell. The type system is starting to clear up for me and so are Monads but I have a long way to go. For a project that I'm working on I need to implement a B-Tree or similar disk based data structure for storing large amounts of data. I feel like I'm getting it and then I get hung up on disk IO operations and trying to figure out how they should fit into the mix. I'm not necessarily looking for someone to point me at an implementation (although I wouldn't mind) but if someone could point me in the direction of some material that could shed a light on things I would be much obliged. I've read some Okasaki and found a promising paper called "Database Manipulation in Haskell 1.3" by Kevin Hammond and Phil Trinder but while they talk to the data structures themselves they don't quite get around to mentioning how they achieve the IO part of the equation. Thanks, Scotty Weeks

1 0

Trashing of memory: How to troubleshoot and fix?
by Joel Reymont 05 Nov '05

05 Nov '05

Folks, This is a I came up with a "scripting" environment for poker server people to excercise their server. All it basically does is compose and parse binary packets and let scripters send and receive them. The issue I have is that the "script" runs fine on Mac OSX (1Gb of memory, though 6Gb VM) and runs out of memory on a Windows machine with 512Mb of memory and 768Mb of VM. The memory error is from malloc which leads me to believe memory fragmentation could be an issue. Running on Windows is a customer requirement. I'm sending and receiving deeply nested structures although that is not my call as the server is in C++. Most of the time just one or two fields will be needed after the command is unpickled. Most of the commands are converted to strings and written to a Chan for logging, though. I'm trying to launch 1000 scripts (separate threads) which just connect to the server, login and disconnect. The big issue is that the production scripts will run for hours, playing against one another. If I the maximum that I can handle is about 200 scripts that do nothing that I failed the project. I was wondering if peak memory usage was the issue but jugding from the GC stats it does not seem to be that high. Maybe I'm wrong. What do you guys make of this? ./logon +RTS -p -sblah GC summary: -- 5,695,379,620 bytes allocated in the heap 1,274,038,800 bytes copied during GC 69,790,544 bytes maximum residency (53 sample(s)) 20433 collections in generation 0 (230.29s) 53 collections in generation 1 ( 5.15s) 152 Mb total memory in use INIT time 0.00s ( 0.04s elapsed) MUT time 91.13s (250.79s elapsed) GC time 235.44s (607.31s elapsed) RP time 0.00s ( 0.00s elapsed) PROF time 0.00s ( 0.00s elapsed) EXIT time 0.00s ( 0.00s elapsed) Total time 326.57s (858.14s elapsed) %GC time 72.1% (70.8% elapsed) Alloc rate 62,497,307 bytes per MUT second Productivity 27.9% of total user, 10.6% of total elapsed --- Time allocation and profiling report: ---- total time = 85.58 secs (4279 ticks @ 20 ms) total alloc = 2,941,143,704 bytes (excludes profiling overheads) COST CENTRE MODULE %time %alloc exp/evalU/cmd Script.Engine 50.0 66.4 connect Main 14.1 11.1 CAF Script.Engine 10.3 5.4 exp/if Script.Engine 8.8 4.8 send Main 6.4 6.7 expect Main 5.8 2.7 exp/evalU/sz Script.Engine 1.4 1.4 startSSL Script.Engine 1.3 0.1 --- expect -- receives command and unpickles it from a FastPackedString. send -- pickles the command into a FastPackedString. hGet from FPS: --- hGet :: Handle -> Int -> IO FastString hGet _ 0 = return empty hGet h i = do fp <- mallocForeignPtr i l <- withForeignPtr fp $ \p-> hGetBuf h p i return $ PS fp 0 l -- The code for send: --- send :: Command -> EngineState () send cmd@(Command kind props) = do liftIO $ yield w <- get let (_, cmd') = checkEncryption cmd send_ cmd' trace $ "Sent: " ++ show (kindOf cmd') tracecmd cmd' --- The code for expect: --- expect :: [CmdType] -> EngineState () expect kind = do w <- get let h = socket w secs = timeout_seconds w (Right fps) <- liftIO $ timeout secs $ P.hGet h 4 trace $ "Size: " ++ show (P.unpackWords fps) state <- liftIO $ hIsEOF h trace $ "Connection closed: " ++ show state (Right size', _) <- {-# SCC "exp/evalU/sz" #-}return $ evalU fps appU let size = fromIntegral $ (unLE size' :: Word16) - 4 (Right packet) <- liftIO $ timeout secs $ P.hGet h size guard $ size == P.length packet cmd' <- {-# SCC "exp/evalU/cmd" #-}return $ evalU packet appU -- unpickle command case cmd' of (Left s, _) -> do trace $ "Error: " ++ s throwError $ "Cannot parse: " ++ show (P.unpackWords packet) (Right _, _) -> do return () let (Right cmd'', _) = cmd' let kind' = kindOf cmd'' let vars = (cmdKind := kind'):(propsOf cmd'') vars'= map (\p -> (attrName p, p)) vars -- update world {-# SCC "exp/putWorld" #-}put $ w { cmd = cmd'', cmdVars = M.fromList vars' } -- move on {-# SCC "processCbks" #-}processCallbacks {-# SCC "exp/if" #-}if elem kind' (skip w) then tracecmd cmd'' >> expect kind -- keep going else match kind cmd'' -- http://wagerlabs.com/

1 1

handling rank 2 types
by Andrew Pimlott 04 Nov '05

04 Nov '05

I am trying to use a rank 2 type to enforce a condition on a data structure. For the purpose of this message, I am trying to ensure that a list is empty (ignoring the possibility of bottom elements): > {-# OPTIONS -fglasgow-exts #-} > import Control.Monad > newtype Empty = Empty (forall a. [a]) I want a function > listToEmpty :: [v] -> IO Empty > listToEmpty l = liftM Empty (mapM no l) where > no :: a -> IO a' > no x = fail "element found" But this gives a "less polymorphic" error. I think the problem is that ghc will not instantiate the "a'" in "IO a'" as a higher-rank type because it occurs within a type contructor. I believe this is the restriction referred to at the end of 7.4.9[1]. If I instead write > uncheckedListToEmpty :: [v] -> Empty > uncheckedListToEmpty l = Empty (map no l) where > no :: a -> a' > no x = error "element found" it compiles, because "v'" can be instantiated as "forall v. Rule v", and even pass through map with the higher-rank type. Is there any way to make ghc accept the first definition? I found this workaround: > newtype Any = Any { unAny :: forall a. a } > listToEmptyAny :: [v] -> IO Empty > listToEmptyAny l = liftM (\l -> Empty (map unAny l)) (mapM no l) where > no :: a -> IO Any > no x = fail "element found" But needing a newtype wrapper Empty was bad enough; do I really need one for every intermediate result (that is inside another type constructor) as well? I could probably define a generic family of wrappers > newtype Forall = Forall (forall a. a) > newtype Forall1 c1 = Forall1 (forall a. c1 a) > newtype Forall2 c1 c2 = Forall2 (forall a. c1 c2 a) ... But I would still have to use them everywhere. Any other tricks? Andrew [1] http://haskell.org/ghc/docs/latest/html/users_guide/type-extensions.html#un…

1 1