December 2005 - Haskell-Cafe

RE: [Haskell-cafe] Optimizing a high-traffic network architecture
by Simon Marlow 16 Dec '05

16 Dec '05

On 16 December 2005 15:19, Lennart Augustsson wrote: > John Meacham wrote: >> On Thu, Dec 15, 2005 at 02:02:02PM -0000, Simon Marlow wrote: >> >>> With 2k connections the overhead of select() is going to start to >>> be a problem. You would notice the system time going up. >>> -threaded may help with this, because it calls select() less often. >> >> >> we should be using /dev/poll on systems that support it. > > And kqueue for systems that support that. Much, much more efficient > than select. Yeah, yeah. We know. We just haven't got around to doing anything about it :-( It's actually quite fiddly to hook this up to Handles - see Einar's implementation in Network.Alt for instance. Cheers, Simon (who wished he hadn't mentioned select() again)

1 0

RE: [Haskell-cafe] Battling laziness
by Simon Marlow 16 Dec '05

16 Dec '05

On 16 December 2005 12:42, Joel Reymont wrote: > On Dec 16, 2005, at 12:36 PM, Simon Marlow wrote: > >> If script#9 is the cost center attached to all of your leaking heap >> data, then you're already a long way to finding the problem. It'll >> help even more to find out whether it is just unevaluated copies of >> "takeEmptySeat Holdem affid []", or something else (-hd, -hy will >> help here). Try +RTS -hy -hcscript#9, for example. >> >> One obvious thing to try is replacing the '$' before {-# SCC >> "script#9" #-} with '$!'. And similarly in takeEmptySeat. > > Let me try these and report my findings. > >> (I should say that we definitely plan to update these for STM, but >> it's not completely trivial (I checked). Volunteers definitely >> welcome). > > I volunteer! Just need some pointers on where to get started. I learn > quickly but need to be guided ;-). Plus, I need this the most, right? I was slightly mistaken: lag/drag/void profiling is pretty easy. Take a look at ghc/rts/LdvProfile.c and add relevant cases for STM objects to processHeapClosureForDead(). If you fix this up and test it we should be able to get it into 6.4.2. Retainer profiling is much harder; the code is in RetainerProfile.c/RetainerSet.c. Cheers, Simon

2 2

RE: [Haskell-cafe] Optimizing a high-traffic network architecture
by Simon Marlow 16 Dec '05

16 Dec '05

On 15 December 2005 10:21, Joel Reymont wrote: > Here are statistics that I gathered. I'm almost done modifying the > program to use 1 timer thread instead of 1 per bot as well as writing > to the socket from the writer thread. This should reduce the number > of threads from 6k (2k x 3) to 2k plus change. > > It appears that +RTS -k3k does make a difference. As per Simon, 2-4k > avoids the thread being garbage collected because each thread gets > its own block in the storage manager. Simon, did I get that right? > > BTW, how does garbage-collecting a thread works in this scenario? My > threads are very long-running. > > The total is the number of bots launched, lobby is how many bots > connected to the lobby. Failed is mostly due to connection reset by > peer errors. The Windows C++ server uses IOCP and running a firewall > was apparently interfering with that somehow. I hate Windows :-(. > > --- Test#1 +RTS -k3k as per Simon. Keep-alive timeout of 9 minutes. > > Total: 1961, Lobby: 1961, Failed: 0 > Total: 2000, Lobby: 2000, Failed: 1 > > This test went smoothly and got to 2k connections very quickly. Maybe > within 30 minutes or so. I did not gather CPU usage, etc. statistics. > > --- Test #2, No thread stack increase, 1 minute keep-alive timeout, > more network traffic > > With a 1 minute timeout things run veeery slow. 86 physical and 158Mb > of VM with 1k bots, CPU 50-60%. Data sent/received is 60-70 packets > and 6-7kb/sec. Killed after a while. > > The statistics are phys/VM, CPU usage in % and #packets/transfer speed > > Total: 1345, Lobby: 1326, Failed: 0, 102/184, 50%, 90/8kb > Total: 1395, Lobby: 1367, Failed: 2 > Total: 1421, Lobby: 1394, Failed: 4 > Total: 1490, Lobby: 1463, Failed: 4, 108/194, 50%, 110/11Kb > Total: 1574, Lobby: 1546, Failed: 4, 113/202, 50%, 116/11kb Hmm, your machine is spending 50% of its time doing nothing, and the network traffic is very low. I wouldn't expect 2k connections to pose any problem at all, so further investigation is definitely required. With 2k connections the overhead of select() is going to start to be a problem. You would notice the system time going up. -threaded may help with this, because it calls select() less often. If that's not the cause, we should find out what your app is doing while it's idle. If there are runnable threads (eg. the lauchner), then the app should not be spending any of its time idle. Cheers, Simon

9 10

RE: [Haskell-cafe] Battling laziness
by Simon Marlow 16 Dec '05

16 Dec '05

On 16 December 2005 12:08, Joel Reymont wrote: > -hc points to script#9 below. > > script (_, _, affid) (Custom (JoinedTable 0)) = > do {-# SCC "script#8" #-}push "takeEmptySeat" $ > {-# SCC "script#9" #-}takeEmptySeat Holdem affid [] > {-# SCC "script#10" #-}return $ Eat $ Just Go > > What takeEmptySeat does it call pickTable > > takeEmptySeat game_type _ filters Go = > do push "pickTable" $ pickTable game_type filters > return $ Eat $ Just Go It's hard to pick out the cause of a space leak from just a fragment of the program, but I'll try to give you some pointers. If script#9 is the cost center attached to all of your leaking heap data, then you're already a long way to finding the problem. It'll help even more to find out whether it is just unevaluated copies of "takeEmptySeat Holdem affid []", or something else (-hd, -hy will help here). Try +RTS -hy -hcscript#9, for example. One obvious thing to try is replacing the '$' before {-# SCC "script#9" #-} with '$!'. And similarly in takeEmptySeat. > Overall, -hc does not help me figure out where my data is being > retained. My understanding is that I need to do -hbdrag,void fo > rthat. I did not try -hd and -hy, they would only help me narrow down > the producers, right? Not necessarily; lag/drag/void only tells you about certain kinds of space leaks. It's another tool in the box, and quite often you can get away without it. Retainer profiling similarly. (I should say that we definitely plan to update these for STM, but it's not completely trivial (I checked). Volunteers definitely welcome). > My program seems to spend 70% of the time collecting garbage. Notice > the HUGE overall allocations. This is my trying to launch 4k bots > over 8 hours. Only 1k bots were launched and just 300 of those got to > play. Maybe because they did not have time with all the garbage > collection :-). Note that your GC time is inflated quite a bit due to profiling (it makes every object larger). The plan to reduce GC time is, in this order: squash space leaks, reduce allocation (to reduce GC load), and then tweak GC parameters. Cheers, Simon

2 2

Battling laziness
by Joel Reymont 16 Dec '05

16 Dec '05

Folks, I have a huge space leak someplace and I suspect this code. The SrvServerInfo data structure is something like 50K compressed or uncompressed byte data before unpickling. My thousands of bots issue this request at least once and I almost run out of memory with 100 bots on a 1Gb machine on FreeBSD. Do I need deepSeq somewhere below? This is the read. read :: Handle -> (SSL, BIO, BIO) -> IO Command read h _ = do sa <- emptyByteArray 4 hGetArray h sa 4 (size', _) <- unpickle endian32 sa 0 let size = fromIntegral $ size' - 4 packet <- emptyByteArray size hGetArray h packet size unstuff packet 0 I suspect that I need to deepSeq cmd'' instead of return $! cmd'' unstuff :: MutByteArray -> Index -> IO Command unstuff array ix = do (kind, ix1) <- unpickle puCmdType array ix (cmd', _) <- unpickle (puCommand kind) array ix1 case cmd' of InvalidCommand -> do fail $ "unstuff: Cannot parse " ++ show array SrvCompressedCommands sz bytes -> do bytes' <- uncompress bytes (fromIntegral sz) cmd'' <- unstuff bytes' 4 return $! cmd'' _ -> return cmd' This is where the list of active tables is converted to a table id list of [Word32]. pickTable _ filters (Cmd cmd@(SrvServerInfo {})) = do let tables = filter (tableMatches filters) $ activeTables cmd ids = map tiTableID tables case tables of [] -> fail $ "pickTable: No tables found: " ++ show filters _ -> do pop stoptimer "pickTable" return $! Eat $! Just $! Custom $! Tables $! ids This is where the table id list of [Word32] is consumed. takeEmptySeat _ aff_id _ (Custom (Tables ids@(table:rest))) = do trace 85 $ "takeEmptySeat: " ++ show (length ids) ++ " tables found" trace 100 $ "takeEmptySeat: tables: " ++ showTables ids trace 85 $ "takeEmptySeat: trying table# " ++ show table w <- get put_ $ w { tables_to_try = rest } push "goToTable" $ goToTable table aff_id -- kick off goToTable return $ Eat $ Just Go This is the SrvServerInfo structure. | SrvServerInfo { activeTables :: ![TableInfo], -- Word16/ removedTables :: ![Word32], -- Word16/ version :: !Int32 } And this is the table info itself. data TableInfo = TableInfo { tiAvgPot :: !Word64, tiNumPlayers :: !Word16, tiWaiting :: !Word16, tiPlayersFlop :: !Word8, tiTableName :: !String, tiTableID :: !Word32, tiGameType :: !GameType, tiInfoMaxPlayers :: !Word16, tiIsRealMoneyTable :: !Bool, tiLowBet :: !Word64, tiHighBet :: !Word64, tiMinStartMoney :: !Word64, tiMaxStartMoney :: !Word64, tiGamesPerHour :: !Word16, tiTourType :: !TourType, tiTourID :: !Word32, tiBetType :: !BetType, tiCantReturnLess :: !Word32, tiAffiliateID :: ![Word8], tiLangID :: !Word32 } deriving (Show, Typeable) Thanks, Joel -- http://wagerlabs.com/

2 1

RE: [Haskell-cafe] Battling laziness
by Simon Marlow 16 Dec '05

16 Dec '05

What does ordinary heap profiling (-hc, -hd, -hy) tell you about what's in the heap? These options should work fine with STM. Cheers, Simon On 16 December 2005 11:44, Joel Reymont wrote: > Folks, > > I have a huge space leak someplace and I suspect this code. The > SrvServerInfo data structure is something like 50K compressed or > uncompressed byte data before unpickling. My thousands of bots issue > this request at least once and I almost run out of memory with 100 > bots on a 1Gb machine on FreeBSD. Do I need deepSeq somewhere below? > > This is the read. > > read :: Handle -> (SSL, BIO, BIO) -> IO Command > read h _ = > do sa <- emptyByteArray 4 > hGetArray h sa 4 > (size', _) <- unpickle endian32 sa 0 > let size = fromIntegral $ size' - 4 > packet <- emptyByteArray size > hGetArray h packet size > unstuff packet 0 > > I suspect that I need to deepSeq cmd'' instead of return $! cmd'' > > unstuff :: MutByteArray -> Index -> IO Command > unstuff array ix = > do (kind, ix1) <- unpickle puCmdType array ix > (cmd', _) <- unpickle (puCommand kind) array ix1 > case cmd' of > InvalidCommand -> do fail $ "unstuff: Cannot parse " ++ > show array > SrvCompressedCommands sz bytes -> > do bytes' <- uncompress bytes (fromIntegral sz) > cmd'' <- unstuff bytes' 4 > return $! cmd'' > _ -> return cmd' > > This is where the list of active tables is converted to a table id > list of [Word32]. > > pickTable _ filters (Cmd cmd@(SrvServerInfo {})) = > do let tables = filter (tableMatches filters) $ activeTables cmd > ids = map tiTableID tables > case tables of > [] -> fail $ "pickTable: No tables found: " ++ show filters > _ -> > do pop > stoptimer "pickTable" > return $! Eat $! Just $! Custom $! Tables $! ids > > This is where the table id list of [Word32] is consumed. > > takeEmptySeat _ aff_id _ (Custom (Tables ids@(table:rest))) = > do trace 85 $ "takeEmptySeat: " ++ show (length ids) > ++ " tables found" > trace 100 $ "takeEmptySeat: tables: " ++ showTables ids > trace 85 $ "takeEmptySeat: trying table# " ++ show table > w <- get > put_ $ w { tables_to_try = rest } > push "goToTable" $ goToTable table aff_id > -- kick off goToTable > return $ Eat $ Just Go > > This is the SrvServerInfo structure. > > | SrvServerInfo > { > activeTables :: ![TableInfo], -- Word16/ > removedTables :: ![Word32], -- Word16/ > version :: !Int32 > } > > And this is the table info itself. > > data TableInfo = TableInfo > { > tiAvgPot :: !Word64, > tiNumPlayers :: !Word16, > tiWaiting :: !Word16, > tiPlayersFlop :: !Word8, > tiTableName :: !String, > tiTableID :: !Word32, > tiGameType :: !GameType, > tiInfoMaxPlayers :: !Word16, > tiIsRealMoneyTable :: !Bool, > tiLowBet :: !Word64, > tiHighBet :: !Word64, > tiMinStartMoney :: !Word64, > tiMaxStartMoney :: !Word64, > tiGamesPerHour :: !Word16, > tiTourType :: !TourType, > tiTourID :: !Word32, > tiBetType :: !BetType, > tiCantReturnLess :: !Word32, > tiAffiliateID :: ![Word8], > tiLangID :: !Word32 > } deriving (Show, Typeable) > > Thanks, Joel

2 2

Non-STM bounded queue
by Joel Reymont 16 Dec '05

16 Dec '05

Folks, I need to move away from STM (unfortunately) until I profile my program to my satisfaction. Profiling is somewhat crippled with STM in 6.4.1. I cannot do void and drag profiling, for examplel. I decided to abstract my mailboxes in a Queue module so that I could switch between STM and non-STM as needed. Please let me know what you think of this bounded queue implementation. --- module Queue where import Control.Monad import Control.Concurrent data Queue a = Queue !QSem !(Chan a) newQ :: Int -> IO (Queue a) newQ size = do sem <- newQSem size q <- newChan return $! Queue sem q writeQ :: Queue a -> a -> IO () writeQ (Queue sem q) x = do waitQSem sem writeChan q x readQ :: Queue a -> IO a readQ (Queue sem q) = do x <- readChan q signalQSem sem return x Thank, Joel -- http://wagerlabs.com/

1 0

+RTS -M800M
by Joel Reymont 16 Dec '05

16 Dec '05

Folks, I'm trying to restrict GHC to 800Mb of heap at runtime by passing in +RTS -M800M, the machine has 1Gb of memory and top shows free physical memory dropping below 175Mb. I suppose I'm missing something obvious or paying attention to the wrong statistics, Unix has a good VM manager after all. Are my runtime options correct, though? Thanks, Joel -- http://wagerlabs.com/

1 0

RE: Re[2]: [Haskell-cafe] Unbound threads and FFI
by Simon Peyton-Jones 16 Dec '05

16 Dec '05

I've added a FAQ. Simon | -----Original Message----- | From: haskell-cafe-bounces(a)haskell.org [mailto:haskell-cafe-bounces@haskell.org] On Behalf Of | Simon Marlow | Sent: 15 December 2005 09:26 | To: Bulat Ziganshin | Cc: Haskell-Cafe Cafe | Subject: RE: Re[2]: [Haskell-cafe] Unbound threads and FFI | | On 14 December 2005 18:04, Bulat Ziganshin wrote: | | > Hello Simon, | > | > Wednesday, December 14, 2005, 7:39:43 PM, you wrote: | > | >> Do other Haskell threads get blocked during an FFI call? | > | >> | safe unsafe | >> --------------+---------------- | >> -threaded | NO YES | >> no -threaded | YES YES | > | > there is also `threadsafe` specifier. i don't understand from the docs | > what it exactly means? | > | > ghc commentary says: | > | > To signal that you want an external call to be serviced by a separate | > OS thread, you have to add the attribute threadsafe to a foreign | > import declaration, i.e., | > | > foreign import "bigComp" threadsafe largeComputation :: Int -> IO () | > | > The distinction between 'safe' and thread-safe C calls is made so that | > we may call external functions that aren't re-entrant but may cause a | > GC to occur. | > | > The threadsafe attribute subsumes safe. | | threadsafe is deprecated; it is currently exactly the same as safe. We | thought at one time that it would be useful to have the distinction, but | it turned out to be impractical to implement safe that wasn't also | threadsafe. | | Cheers, | Simon | _______________________________________________ | Haskell-Cafe mailing list | Haskell-Cafe(a)haskell.org | http://www.haskell.org/mailman/listinfo/haskell-cafe

1 0

RE: [Haskell-cafe] Unbound threads and FFI
by Simon Marlow 15 Dec '05

15 Dec '05

On 14 December 2005 15:34, Joel Reymont wrote: > On Dec 14, 2005, at 3:21 PM, Simon Marlow wrote: > >> this only happens in the -threaded RTS, in the non-threaded RTS a FFI >> call will block all the other Haskell threads. > > Just to be absolutely sure... Does the above apply to safe _and_ > unsafe FFI? > > I have been using the non-threaded runtime with unsafe FFI calls > (OpenSSL, Zlib) and I do not believe my runtime was being blocked. > Maybe I just didn't notice it, though. Maybe this little table will help: Do other Haskell threads get blocked during an FFI call? | safe unsafe --------------+---------------- -threaded | NO YES no -threaded | YES YES Cheers, Simon

3 2