Re: [web-devel] questions about ResponseEnumerator

On Sun, Oct 23, 2011 at 5:09 PM, Gregory Collins wrote:

On Sat, Oct 22, 2011 at 10:20 PM, Michael Snoyman wrote:

...

I think Greg's/Snap's approach of a separate timeout for the status and headers is right on the money. It should never take more than one timeout cycle to receive a full set of headers, regardless of how slow the user's connection, and given a reasonable timeout setting from the user (anything over 2 seconds should be fine I'd guess, and our default is 30 seconds).

That's fairly uncontroversial.

...

The bigger question is what we do about the request body. A simple approach might just be that if we receive a packet from the client which is less than a certain size (user defined, maybe 2048 bytes is a good default) it does not tickle the timeout at all. Obviously this means a malicious program could be devised to send precisely 2048 bytes per timeout cycle... but I don't think there's any way to do better than this.

This doesn't really work either. I've already posted code in this thread for what I think is the only reasonable option, which is rate limiting. The way we've implemented rate limiting is:

  1) any individual data packet must arrive within N seconds (the usual timeout)

  2) when you receive a packet, you compute the data rate in bytes per second -- if it's lower than X bytes/sec (where X is a policy decision left up to the user), the connection is killed

  3) the logic from 2) only kicks in after Y seconds, to cover cases where the client needs to do some expensive initial setup. Y is also a policy decision.

As you mention below, the approach I outline is fairly close to rate limiting. I would argue it's *more* correct. In the rate limiting case, a client could send a burst of a huge amount of data at the beginning of the connection, and then send single bytes every 20 seconds for the next few minutes, and rate limiting would allow it to happen. The approach below (call it minimum packet size?) wouldn't. Additionally, I think minimum packet size can be implemented much more efficiently, as there is significantly less data to track.

...

We *have* to err on the side of allowing attacks, otherwise we'll end up with disconnecting valid requests.

I don't agree with this. Some kinds of "valid" requests are indistinguishable from attacks. You need to decide what's more important: letting some guy on a 30-kilobit packet radio connection upload a big file, or letting someone DoS your server.

Fair enough, let me rephrase: there's a gray area between attack and valid request, and by default I'd like to draw the line so as to incorporate most of the gray zone.

...

In other words, here's how I'd see the timeout code working:

1. A timeout is created at the beginning of a connection, and not tickled at all until all the request headers are read in. 2. Every time X (default: 2048) bytes of the request body are read, the timeout is tickled.

Note that this is basically a crude form of rate-limiting (at X/T bytes per second). Why not do it "properly"?

G -- Gregory Collins