On Dec 25, 2005, at 10:13 AM, Bulat Ziganshin wrote:
Hello Joel, [...] so i think that your problems is due to bad design decisions caused by lack of experience. two weeks ago when you skipped my suggestions about improving this design and answered that you will use "systematic approach", i foresee that you will fail and say that Haskell is a bad language
Yes and no. The systematic approach that I used was profiling the serialization code and tweaking all that I could. I saved my profiling reports after each run and tracked the changes that I made. I will blog about it after Simon M. comes back and suggests how to squeeze the last bit out of it. Regardless of this, it looks to me like I could easily have around 4Mb of network traffic per second with about 4k threads and complicated nested structures to serialize and deserialize. Trying to tackle far less data suggests to me that it's not gonna happen. So I will try to take this as far as I can in Haskell, once I have the heavy artillery to back me up. If the results are good then I will use them in later applications of the same nature but in the meantime I'm rewriting this particular app in Erlang. There are other apps to write in Haskell but my overall lesson is that the choice of right tool for the task at hand will help far more than any optimizations you can produce. Specially when the natural approach to the problem will just produce the results required. I spent 3 months with this app, going from total Haskell newbie who has not seen Haskell before to someone who can read Haskell core code and tackle FFI and networking in a heavily concurrent environment. I still don't know what "the right thing" is. The Erlang rewrite should take me 2-3 days with FFI being the teeth- grinding bit. I need the FFI (which totally sucks compared to Haskell's) because my use of SSL and ZLib is non-standard, otherwise these are already provided to you. I expect that the app will just work. No memory or time leaks. The serialization approach that I'm using is based on the pickler combinators but slightly different. I think I'll backport it to Haskell to compare. The next project on my plate is real-time collusion detection in poker. I first thought of using the Dazzle approach and Bayesian networks but the focus again seems to be on high-throughput networking and scalability. What language will I use? Haskell is no sure bet. A disclaimer for those who might think that I'm a newbit disillutioned with Haskell... I totally love Haskell, it just seems that you need to be a guru to do what I'm trying to do and I have not yet reached this status. I will not rest until I'm able to see just _what_ kind of a performance can be squeezed from Haskell for my application. I'll also continue to seek enlightenment but I need to finish this project before New Year's. Joel -- http://wagerlabs.com/