About the only universal solution seems to pace the threads by passing trace to read. Even then I got 1 alert. Now, can someone explain why the lock eliminates the time leak? On Dec 21, 2005, at 2:41 AM, Cale Gibbard wrote:

Using forkOS with -threaded seems to work, at least on linux and openbsd.

-- http://wagerlabs.com/

I still get timeouts with 5k threads. Not as often as with 1k before, though. On Dec 21, 2005, at 3:35 AM, Donald Bruce Stewart wrote:

It looks like with the 1000s of threads that get run, the problem is just getting enough cpu time for each thread. All the solutions that appear to work involve some kind of blocking (on Handles, MVars, ffi calls), allowing the scheduler to switch.

So why don't we just increase the scheduler switch rate ourselves?

Using +RTS -C -RTS we can convince the rts to switch on every alloc, and with 4000 threads running this seems to remove the timeout alerts, and give all threads enough time to get things done.

-- http://wagerlabs.com/

Well, yes. I think that part of the problem you're having is that the threads compete very highly, and some eventually get ignored for longer than the timeouts. The reason why the (withMVar lock $ const $ putStrLn s) helped in the first place was not so much because of any evaluation of s (we tested that assumption), but rather because during the time that withMVar is active in one thread, any other threads can't pass that same point. So a bunch of the threads which were stealing too much of the time before, now end up blocking, which reduces the competition so the other threads which were being ignored by the scheduler can get some work done. Your problem is partly in the fact that the pickler/unpickler is a bit slow for the workload/machine you have it on, and partly in the fact that it's hard to get the processes to advance smoothly. (The pickling/unpickling doesn't really take very long in any one thread, but if the thread isn't scheduled for a long time in the first place...) A bit of locking is sort of what you want for that. Unfortunately, the way we have it here still affords no guarantees, it just serves as a mechanism to help keep things from getting too bad. It's unclear to me that you really don't want any kind of locking. I think that you might do quite well to decide which threads get to run yourself at least to some extent. After all, you know more about their priorities and how long they have to complete their tasks than the GHC scheduler does. - Cale On 21/12/05, Joel Reymont wrote:

The other thing worth noting is that by inserting a lock with a thread delay we are fooling ourselves. While the individual pickling time goes down, the threads are slowed down overall. Assuming that an external source was waiting for the unpickled packet _that_ source would get a timeout!

-- http://wagerlabs.com/

On 12/21/05, Joel Reymont wrote:

I don't want any kind of locking, true. I need all bots to respond in time otherwise the poker server will sit them out. Eliminating the timeout on pickling does not eliminate the timeout overall, it just passes it to a different place.

One thread will go through serialization quickly but it will be too late by the time it sends a response to the server since it waited a few seconds for the chance to have a go at serialization. I'm trying to build 6.5 right now (having trouble, you might have guessed ;)) and will positively study the scheduler. I do not believe thread priorities are supported in GHC, though.

I had a thought about using continuations but then I would also have to do selects on a few thousand file descriptors myself. Then I would have to decide which continuation to run based on priorities. I might as well patch the GHC scheduler to do what I need. Alternatively, I can just throw in the towel and rewrite the app in Erlang.

I haven't made a firm decision yet, I think I will make my records storable first as you can't get "any closer to the metal". If that does not work then I will just give up. I do have other apps to write in Haskell but it will be a pity if this one does not work out.

I've only skimmed through this, so I may miss the point, but it sounds like a latency vs bandwidth discussion. Let's say you push through 5000 requests in one second (i.e you start 5000 processes and then exactly one second later all 5000 complete simultaneously). Now 50000 in ten seconds is actually the exact same throughput, but if your timeout is three seconds, then you'll get problems. So your problem is that you only do a tiny bit of work for each processess over and over, think of the scheduler just looping through the processess giving it a tiny slice of time, over and over and over. It may take ten seconds for any individual process to complete, but the full throughput is still the same. When you increase the number of processes you won't see the additional processes timeout, but ALL processess (or at least many of them). My suggestion is this: Find out how many processes can be serviced at one time without getting timed out (i.e. find a good compromise between latency and bandwidth), then wrap up the computaitons in a semaphore containing exactly that many resources. I think you probably want this amount to be somwehere around the number of actual CPU cores you have. Having a process' computation wait for 99% of the timeout out to start and then complete it in the final 1% is no worse then having it slowly compute its result for the duration of the timeout. The difference is that if you run out of CPU juice, only some of the processess get hurt (they timout before they start), instead of all of them (the time it takes to compute each of them is more than the timeout because the CPU is spread too thin). /S -- Sebastian Sylvan +46(0)736-818655 UIN: 44640862

On Dec 21, 2005, at 7:45 AM, Bulat Ziganshin wrote:

1) i think that this method of deserialization may be just inefficient. what is plain deserialization time for this 50k?

No idea. I know it's inefficient but this is not the issue. The issue is that with some strange tweaks it runs fast.

2) try to insert lock around thread startup code, something like this:

withMVar x $ \_ -> send "Hello" data <- get cmd <- unpickle data

I have no "data <- get". Did you try this on your system to see if your fix works? We spent a lot of time tweaking this on #haskell yesterday and you need to be quite precise at what you tweak to get it right. I would appreciate if you could actually run the code. Also, what will this lock do and how far do you want it to extend? I do need the threads to run in parallel. Thanks, Joel -- http://wagerlabs.com/

This should not be the case. The amount of work is the same regardless and the issues seem to be with _timing_. Passing in trace that writes to the screen with a lock sort of slows things down. I encourage you to actually build the code and see it for yourself. Thanks, Joel On Dec 21, 2005, at 10:26 AM, Tomasz Zielonka wrote:

My guess is that without tweaks the threads allocate more, which increases the pressure on GC. Try to analyze the GC stats with and without tweaks.

-- http://wagerlabs.com/

From: Joel Reymont To: "Branimir Maksimovic" CC: haskell-cafe@haskell.org Subject: Re: [Haskell-cafe] Killer pickler combinators (was Time leak) Date: Wed, 21 Dec 2005 14:51:42 +0000

I'm not sure I buy this. Again, this helps:

{-# NOINLINE lock #-} lock :: MVar () lock = unsafePerformIO $ newMVar ()

trace s = withMVar lock $ const $ putStrLn s

and then in read_

cmd <- read h trace

trace is called _after_ all the timings in read so it should not affect the timings.

It does not affects timings directly , but indirectly. You have putStrLn which performs some work and that is point of serialization. Try no op instead of putStrLn and you should get timeouts again. i'm sure you'll get even better operation timings if you lock around whole timing operation instead. This will be valid only for single CPU only. In this way you actually get same throughoutput, with or without lock. It' just you have to measure cummulative reads/sec which will be same with/without lock for single CPU/worker thread. Again, only way to improve performance is to use more then one CPU/worker thread. Greetings, Bane.

You could basically say that the lock is at the end of read, after all the unpickling has been done. The other interesting thing is that replacing trace with

delay _ = threadDelay 1

does not solve the issue.

On Dec 21, 2005, at 2:33 PM, Branimir Maksimovic wrote:

...

In this code: do TOD time1 _ <- getClockTime (kind, ix1) <- unpickle puCmdType ptr ix TOD time2 _ <- getClockTime (cmd', _) <- unpickle (puCommand kind) ptr ix1 TOD time3 _ <- getClockTime you get time multiple times. So if you lock whole operation nothing else will be executed while in this code and therefore you don;t have timeouts. But, without lock more threads you have, more time have to wait for operation to finish and therefore timeouts.

-- http://wagerlabs.com/

_________________________________________________________________ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/