Looking for architecture/infrastructure advice on coping with thousands of concurrent clients

Isak Hansen

9 Jul 2014 9 Jul '14

5:40 p.m.

Let's just imagine there aren't enough poker sites out there already and that it makes sense to build another one. Basically, there will be players connecting to one or more tables, most likely talking some kind of JSON based protocol with the server(s). #1 - What's a good way to set up and maintain compressed and encrypted connections from the clients to my cluster? I've been wanting to try out websockets, think that'll do? #2 - I'm new to Haskell and would appreciate thoughts on how to store and process game state. Imagine a plain application managing 1000 tables of poker, responding to player input (that magically appears) and timer events (e.g. folding players that fail to take action). What data structures and libraries should I be looking at here? Thoughts on concurrency and how I organize program flow? #3 - I'm thinking of ZMQ to wire components together. Would I be better off basing my cluster on Cloud Haskell or some other library for distributed work? Would love some quick pointers and feedback here, thanks in advance, Isak

Attachments:

attachment.html (text/html — 1.2 KB)

Show replies by date

Andrew Gibiansky

9 Jul 9 Jul

6:01 p.m.

I don't have any comments on the specifics of your application, but I've used zeromq4-haskell [0] for IHaskell and it's a solid library that works well. There's also zeromq-haskell (instead of zeromq4-haskell) if you want older ZeroMQ. I was using it for local communication, not networked though, so not sure if that changes anything. It is fairly low-level, though, so I imagine Cloud Haskell can give you a higher level of abstraction when dealing with networking stuff. [0] https://hackage.haskell.org/package/zeromq4-haskell On Wed, Jul 9, 2014 at 10:40 AM, Isak Hansen wrote:

...

Let's just imagine there aren't enough poker sites out there already and that it makes sense to build another one. Basically, there will be players connecting to one or more tables, most likely talking some kind of JSON based protocol with the server(s).

#1 - What's a good way to set up and maintain compressed and encrypted connections from the clients to my cluster? I've been wanting to try out websockets, think that'll do?

#2 - I'm new to Haskell and would appreciate thoughts on how to store and process game state. Imagine a plain application managing 1000 tables of poker, responding to player input (that magically appears) and timer events (e.g. folding players that fail to take action). What data structures and libraries should I be looking at here? Thoughts on concurrency and how I organize program flow?

#3 - I'm thinking of ZMQ to wire components together. Would I be better off basing my cluster on Cloud Haskell or some other library for distributed work?

Would love some quick pointers and feedback here, thanks in advance, Isak

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Wojtek Narczyński

6:52 p.m.

On 09.07.2014 19:40, Isak Hansen wrote: Thousands of connections, your smartphone wouldn't handle, not to mention a server.

...

#1 - What's a good way to set up and maintain compressed and encrypted connections from the clients to my cluster? I've been wanting to try out websockets, think that'll do?

http://caniuse.com/websockets - WebSockets browser compatibility, caveat emptor

...

#2 - I'm new to Haskell and would appreciate thoughts on how to store and process game state. Imagine a plain application managing 1000 tables of poker, responding to player input (that magically appears) and timer events (e.g. folding players that fail to take action). What data structures and libraries should I be looking at here? Thoughts on concurrency and how I organize program flow?

I would spawn a process for each game, and a process for each player.

...

#3 - I'm thinking of ZMQ to wire components together. Would I be better off basing my cluster on Cloud Haskell or some other library for distributed work?

Erlang/OTP is great, but I have a feeling that something was 'lost in translation' to Haskell, and that thing is simplicity. For example, I do not understand why such a small cabal package has been split into multiple tiny cabal packages. This is the fuel that cabal hell fire burns on. Despite this, I would give Cloud Haskell a try. If you don't you, will have to rediscover things like supervision hierarchy, not to mention timeouts as the first line of defense againtst all kinds of concurrency problems. Erlang/OTP really does get certain things right. -- Kind regards, Wojtek N.

Carter Schonwald

9:39 p.m.

the courier package on hackage seems to strike a better balance vs CH https://hackage.haskell.org/package/courier On Wed, Jul 9, 2014 at 2:52 PM, Wojtek Narczyński wrote:

...

On 09.07.2014 19:40, Isak Hansen wrote:

Thousands of connections, your smartphone wouldn't handle, not to mention a server.

#1 - What's a good way to set up and maintain compressed and encrypted

...
connections from the clients to my cluster? I've been wanting to try out websockets, think that'll do?

http://caniuse.com/websockets - WebSockets browser compatibility, caveat emptor

#2 - I'm new to Haskell and would appreciate thoughts on how to store and

...
process game state. Imagine a plain application managing 1000 tables of poker, responding to player input (that magically appears) and timer events (e.g. folding players that fail to take action). What data structures and libraries should I be looking at here? Thoughts on concurrency and how I organize program flow?

I would spawn a process for each game, and a process for each player.

...
off basing my cluster on Cloud Haskell or some other library for distributed work?

Erlang/OTP is great, but I have a feeling that something was 'lost in

#3 - I'm thinking of ZMQ to wire components together. Would I be better translation' to Haskell, and that thing is simplicity. For example, I do not understand why such a small cabal package has been split into multiple tiny cabal packages. This is the fuel that cabal hell fire burns on. Despite this, I would give Cloud Haskell a try. If you don't you, will have to rediscover things like supervision hierarchy, not to mention timeouts as the first line of defense againtst all kinds of concurrency problems. Erlang/OTP really does get certain things right.

-- Kind regards, Wojtek N.

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Wojtek Narczyński

10:05 p.m.

On 09.07.2014 23:39, Carter Schonwald wrote:

...

the courier package on hackage seems to strike a better balance vs CH https://hackage.haskell.org/package/courier

Well, no. There really is great value in the OTP part of Erlang/OTP, while this package only provides Erlang (messaging). -- Wojtek

Alois Cochard

6:53 p.m.

Hi Isak, On 9 July 2014 18:40, Isak Hansen wrote:

...

#2 - I'm new to Haskell and would appreciate thoughts on how to store and process game state. Imagine a plain application managing 1000 tables of poker, responding to player input (that magically appears) and timer events (e.g. folding players that fail to take action). What data structures and libraries should I be looking at here? Thoughts on concurrency and how I organize program flow?

If I had to build an system like this, the main design principle I would adopt is to not try to share the state of games among nodes, but instead redirect players of the same game on the same node (where you basically have a thread processing the event of the game and updating the state in memory). Then I would probably use some form of event sourcing to record game action which could be use in case of node crash to recreate a game state by replaying events. The event source could then be used as well to generate stats/views in near real-time. -- *A\ois* http://twitter.com/aloiscochard http://github.com/aloiscochard

Mikael Brockman

16 Jul 16 Jul

12:47 p.m.

New subject: Looking for architecture/infrastructure advice on coping with thousands of concurrent clients

Alois Cochard writes:

...

Hi Isak,

On 9 July 2014 18:40, Isak Hansen wrote:

#2 - I'm new to Haskell and would appreciate thoughts on how to store and process game state. Imagine a plain application managing 1000 tables of poker, responding to player input (that magically appears) and timer events (e.g. folding players that fail to take action). What data structures and libraries should I be looking at here? Thoughts on concurrency and how I organize program flow?

If I had to build an system like this, the main design principle I would adopt is to not try to share the state of games among nodes, but instead redirect players of the same game on the same node (where you basically have a thread processing the event of the game and updating the state in memory).

Then I would probably use some form of event sourcing to record game action which could be use in case of node crash to recreate a game state by replaying events.

The event source could then be used as well to generate stats/views in near real-time.

Some related thoughts: For poker, it seems reasonable to organize the program around a function like this: step :: Game -> Event -> (Game, Response) This would be the entry point for the game logic. It's obvious how to write tests for this function, and if you're sure that it does the right thing, then you'll be confident about your game loop. Then you need a way to produce a sequence of events, combined from different sources: timers, HTTP requests, etc. If every game is fed by a `TChan`, for example, then you can write events into that from different sources. But from the viewpoint of the core logic, this is just an abstract source of events. Given the performance of GHC's threads, I don't think thousands of concurrent clients is that much different from dozens or hundreds -- yet another reason to avoid premature optimization and focus on clarity and purity! By the way, I've had fun using `pipes`, `stm`, and `async` to implement an event sourcing server using JSON commands from web requests. Using pipe combinators and queues, the data/control flow of the server can be apparent from the code of the main function, which is something I've rarely seen in non-Haskell servers! -- Mikael Brockman

Alexander Kjeldaas

9 Jul 9 Jul

8:48 p.m.

I'd have user state serviced by a thread per user. Then have separate game state serviced by another thread. Then use zmq or cloud Haskell for interconnect between the two states. Scalability games then boils down to what cross sectional bandwidth you can get by optimizing the interconnect and the placement/locality of the state. Scaling the number of client connections can be done independently. Alexander On Jul 9, 2014 7:40 PM, "Isak Hansen" wrote:

...

Let's just imagine there aren't enough poker sites out there already and that it makes sense to build another one. Basically, there will be players connecting to one or more tables, most likely talking some kind of JSON based protocol with the server(s).

#1 - What's a good way to set up and maintain compressed and encrypted connections from the clients to my cluster? I've been wanting to try out websockets, think that'll do?

#2 - I'm new to Haskell and would appreciate thoughts on how to store and process game state. Imagine a plain application managing 1000 tables of poker, responding to player input (that magically appears) and timer events (e.g. folding players that fail to take action). What data structures and libraries should I be looking at here? Thoughts on concurrency and how I organize program flow?

#3 - I'm thinking of ZMQ to wire components together. Would I be better off basing my cluster on Cloud Haskell or some other library for distributed work?

Would love some quick pointers and feedback here, thanks in advance, Isak

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

4011

Age (days ago)

4018

Last active (days ago)

List overview

Download

7 comments

7 participants

participants (7)

Alexander Kjeldaas
Alois Cochard
Andrew Gibiansky
Carter Schonwald
Isak Hansen
Mikael Brockman
Wojtek Narczyński