Did you suppose some link with [0] in your email? Kind regards, Kirill Zaborsky 2013/7/23 Erik de Castro Lopo

Michael Snoyman wrote:

...

- Remove the deprecated isSecure field from the Request data type. - Remove the misleading fields serverName and serverPort fields as well. - Possibly: hide the Request constructor so that new fields can be added without breaking backwards compatibility.

If anyone else has some ideas, please bring them up.

Warning, the following idea is not even half baked.

It would be really nice if the new Wai allowed the implementation (as a Wai application) of a full HTTP and HTTPS proxy capable of stream data in both directions with constant space usage.

I was able to implement a proxy like this [0] with a hacked version of Warp based on Enumerator and a version of Warp based on Conduit 0.2. For later versions of Conduit I was never able to get it to run in constant space.

Happy to help in any way I can.

Cheers, Erik -- ---------------------------------------------------------------------- Erik de Castro Lopo http://www.mega-nerd.com/

-- You received this message because you are subscribed to the Google Groups "Yesod Web Framework" group. To unsubscribe from this group and stop receiving emails from it, send an email to yesodweb+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.

On Tue, Jul 23, 2013 at 2:10 PM, Erik de Castro Lopo wrote:

Michael Snoyman wrote:

...

- Remove the deprecated isSecure field from the Request data type. - Remove the misleading fields serverName and serverPort fields as well. - Possibly: hide the Request constructor so that new fields can be added without breaking backwards compatibility.

If anyone else has some ideas, please bring them up.

Warning, the following idea is not even half baked.

It would be really nice if the new Wai allowed the implementation (as a Wai application) of a full HTTP and HTTPS proxy capable of stream data in both directions with constant space usage.

I was able to implement a proxy like this [0] with a hacked version of Warp based on Enumerator and a version of Warp based on Conduit 0.2. For later versions of Conduit I was never able to get it to run in constant space.

Happy to help in any way I can.

I'm still quite interested in getting this use case to work, but I'm also still not familiar enough with the current problems you've faced to have an idea on what to change.

On Tue, Jul 23, 2013 at 2:53 PM, Erik de Castro Lopo wrote:

Michael Snoyman wrote:

...

I'm still quite interested in getting this use case to work, but I'm also still not familiar enough with the current problems you've faced to have an idea on what to change.

Give me a couple of days to figure out the exact current state of play. Then we'll see it we can fix it.

Sounds good to me. I'm personally not in a huge rush to get the new version of WAI out the door, and if Kazu needs to get something out sooner for better performance testing, I think I could backport most of the performance improvement to Warp 1.4. In fact, maybe I'll play with that now :). Michael

On Tue, Jul 23, 2013 at 3:21 PM, Krzysztof Skrzętnicki wrote:

On Tue, Jul 23, 2013 at 12:04 PM, Michael Snoyman wrote:

...

- Possibly: hide the Request constructor so that new fields can be added without breaking backwards compatibility.

I strongly believe this isn't the best route to achieve stability. This

will end up in hacking up own versions of WAI with exported internals. At the very least please consider adding "Internal" module with stability warnings but actually exposing the constructor.

Point taken, that's a fair trade-off.

The reason we'd be removing those is that there *still* isn't any way to know this. If Warp is sitting behind a reverse proxy (which is virtually always the case, whether Amazon ELB, Nginx, Keter, or something else) there's no way to know what the actual server name, server port, or scheme are. On Tue, Jul 23, 2013 at 6:37 PM, Federico Mastellone wrote:

Hi,

Without isSecure and serverPort, will Applications be able to handle requests to different ports and HTTP or HTTPS in different ways? I won't be able to know if the login is made over HTTP and redirect to HTTPS with any Wai server.

On 23/07/2013, at 07:04, Michael Snoyman wrote:

Some of you may have seen the brief interchange between Kazu and myself regarding ResourceT in WAI. Let me give a very brief synopsis of the situation:

- In order to allow WAI applications to acquire scarce resources like file handles in an exception-safe way, WAI actions all live in a `ResourceT IO` monad, instead of just `IO`. - Kazu has mentioned to me in the past that, based on his profiling, monadic bind for ResourceT may currently be a bottleneck in WAI. - I made an experimental change to WAI so that, instead of actions living in `ResourceT IO`, the WAI Request value contains a ResourceT InternalState value, which can be used to accomplish the exact same thing as living in ResourceT (see [1]). - Kazu checked out the performance difference of this new branch, and it allows for some better optimizations.

If this was purely a performance optimization, I would probably just optimize Warp and not bother changing WAI. However, I think that this change makes WAI itself better as well, and therefore am in favor of making this as a breaking change, releasing it as WAI 2.0[2]. Firstly, are there any objections to this move?

So as long as we're making a breaking change, the question arises: what other changes should we be making to WAI? I know Kazu had mentioned adding fields to avoid the need for lookups, can you clarify that request a bit? Here are some other changes I can think of:

- Remove the deprecated isSecure field from the Request data type. - Remove the misleading fields serverName and serverPort fields as well. - Possibly: hide the Request constructor so that new fields can be added without breaking backwards compatibility.

If anyone else has some ideas, please bring them up.

[1] http://haddocks.fpcomplete.com/fp/7.4.2/20130704-120/resourcet/Control-Monad... [2] For the Yesod community: the change is minor enough that Yesod 1.2 can have some conditional compilation to support both the current WAI 1.4 and WAI 2.0, so this shouldn't affect most Yesod users at all.

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

It seems Warp's place to faithfully report what actually happened - what IP they connected to, what port, on what interface, and even what ciphers were used ideally. Interpretation of headers and such seems very much the place of middleware to me. I'm pretty sure when I've wanted to interpret X-Forward-For and related it has been in a way different from you - specifically I wanted to know how people were messing with it more then what the value should resolve as. Never mind in my setup I expect your prebaked config would just flat out fail. -davean On Tue, Jul 23, 2013 at 4:58 PM, Greg Weber wrote:

In Rack most of these things are known. In some cases this requires users to configure the proxy to send headers.

One solution would be having the user state what values are being depended on and having Warp fail on startup with an error message if it can't collect them. It is possible to expect some of these values to not change (and require a reboot of warp when they do) and have them be part of Warp rather than WAI.

But just wrapping these values in a Maybe could be another approach.

On Tue, Jul 23, 2013 at 1:13 PM, Michael Snoyman wrote:

...

The reason we'd be removing those is that there *still* isn't any way to know this. If Warp is sitting behind a reverse proxy (which is virtually always the case, whether Amazon ELB, Nginx, Keter, or something else) there's no way to know what the actual server name, server port, or scheme are.

On Tue, Jul 23, 2013 at 6:37 PM, Federico Mastellone wrote:

...

Hi,

Without isSecure and serverPort, will Applications be able to handle requests to different ports and HTTP or HTTPS in different ways? I won't be able to know if the login is made over HTTP and redirect to HTTPS with any Wai server.

On 23/07/2013, at 07:04, Michael Snoyman wrote:

Some of you may have seen the brief interchange between Kazu and myself regarding ResourceT in WAI. Let me give a very brief synopsis of the situation:

- In order to allow WAI applications to acquire scarce resources like file handles in an exception-safe way, WAI actions all live in a `ResourceT IO` monad, instead of just `IO`. - Kazu has mentioned to me in the past that, based on his profiling, monadic bind for ResourceT may currently be a bottleneck in WAI. - I made an experimental change to WAI so that, instead of actions living in `ResourceT IO`, the WAI Request value contains a ResourceT InternalState value, which can be used to accomplish the exact same thing as living in ResourceT (see [1]). - Kazu checked out the performance difference of this new branch, and it allows for some better optimizations.

If this was purely a performance optimization, I would probably just optimize Warp and not bother changing WAI. However, I think that this change makes WAI itself better as well, and therefore am in favor of making this as a breaking change, releasing it as WAI 2.0[2]. Firstly, are there any objections to this move?

So as long as we're making a breaking change, the question arises: what other changes should we be making to WAI? I know Kazu had mentioned adding fields to avoid the need for lookups, can you clarify that request a bit? Here are some other changes I can think of:

- Remove the deprecated isSecure field from the Request data type. - Remove the misleading fields serverName and serverPort fields as well. - Possibly: hide the Request constructor so that new fields can be added without breaking backwards compatibility.

If anyone else has some ideas, please bring them up.

[1] http://haddocks.fpcomplete.com/fp/7.4.2/20130704-120/resourcet/Control-Monad... [2] For the Yesod community: the change is minor enough that Yesod 1.2 can have some conditional compilation to support both the current WAI 1.4 and WAI 2.0, so this shouldn't affect most Yesod users at all.

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

-- You received this message because you are subscribed to the Google Groups "Yesod Web Framework" group. To unsubscribe from this group and stop receiving emails from it, send an email to yesodweb+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

Completely not what I was saying. "what IP they connected to, what port, on what interface, and even what ciphers were used ideally." None of that is information the user could ever pass to the application except in the truly trivial cases. They're inherently features of each connection separately. If it isn't passed in the Wai Request it can't be (reasonably) discovered. If some thing is going to try to derive for the user the pre-proxy IP based the connection information it should be a library that does such preferably or alternatively a middleware component, never the webserver. Furthermore WAI should contain sufficient information in the Request object to do this and more. -davean On Wed, Jul 24, 2013 at 6:13 AM, Michael Snoyman wrote:

I think Greg and davean are saying something similar here. If I may paraphrase (please correct me if I'm wrong): some information may not be derivable by the WAI handler (e.g. Warp) itself, but with a little bit of help from the user, that information could be included in the WAI standard.

If that's what you're getting at, I disagree. This would essentially mean we require users to provide information to Warp, so that Warp can provide that information to user applications. Warp (or any other handler) is just a dumb transport mechanism then. And since we can't rely on that information being present, we'd wrap it up with Maybe values, and therefore end up losing type safety.

The prime example of all this is application root. I still strongly believe that Yesod did this the right way: if the user needs to provide the information, then ask the user to provide the information to the application itself, and the WAI handler needn't know anything about it.

Is there some kind of data that the WAI handler could do a better job of passing onto the application than the user can do?

On Tue, Jul 23, 2013 at 11:58 PM, Greg Weber wrote:

...

In Rack most of these things are known. In some cases this requires users to configure the proxy to send headers.

One solution would be having the user state what values are being depended on and having Warp fail on startup with an error message if it can't collect them. It is possible to expect some of these values to not change (and require a reboot of warp when they do) and have them be part of Warp rather than WAI.

But just wrapping these values in a Maybe could be another approach.

On Tue, Jul 23, 2013 at 1:13 PM, Michael Snoyman wrote:

...

The reason we'd be removing those is that there *still* isn't any way to know this. If Warp is sitting behind a reverse proxy (which is virtually always the case, whether Amazon ELB, Nginx, Keter, or something else) there's no way to know what the actual server name, server port, or scheme are.

On Tue, Jul 23, 2013 at 6:37 PM, Federico Mastellone wrote:

...

Hi,

Without isSecure and serverPort, will Applications be able to handle requests to different ports and HTTP or HTTPS in different ways? I won't be able to know if the login is made over HTTP and redirect to HTTPS with any Wai server.

On 23/07/2013, at 07:04, Michael Snoyman wrote:

Some of you may have seen the brief interchange between Kazu and myself regarding ResourceT in WAI. Let me give a very brief synopsis of the situation:

- In order to allow WAI applications to acquire scarce resources like file handles in an exception-safe way, WAI actions all live in a `ResourceT IO` monad, instead of just `IO`. - Kazu has mentioned to me in the past that, based on his profiling, monadic bind for ResourceT may currently be a bottleneck in WAI. - I made an experimental change to WAI so that, instead of actions living in `ResourceT IO`, the WAI Request value contains a ResourceT InternalState value, which can be used to accomplish the exact same thing as living in ResourceT (see [1]). - Kazu checked out the performance difference of this new branch, and it allows for some better optimizations.

If this was purely a performance optimization, I would probably just optimize Warp and not bother changing WAI. However, I think that this change makes WAI itself better as well, and therefore am in favor of making this as a breaking change, releasing it as WAI 2.0[2]. Firstly, are there any objections to this move?

So as long as we're making a breaking change, the question arises: what other changes should we be making to WAI? I know Kazu had mentioned adding fields to avoid the need for lookups, can you clarify that request a bit? Here are some other changes I can think of:

- Remove the deprecated isSecure field from the Request data type. - Remove the misleading fields serverName and serverPort fields as well. - Possibly: hide the Request constructor so that new fields can be added without breaking backwards compatibility.

If anyone else has some ideas, please bring them up.

[1] http://haddocks.fpcomplete.com/fp/7.4.2/20130704-120/resourcet/Control-Monad... [2] For the Yesod community: the change is minor enough that Yesod 1.2 can have some conditional compilation to support both the current WAI 1.4 and WAI 2.0, so this shouldn't affect most Yesod users at all.

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

-- You received this message because you are subscribed to the Google Groups "Yesod Web Framework" group. To unsubscribe from this group and stop receiving emails from it, send an email to yesodweb+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.

-- You received this message because you are subscribed to the Google Groups "Yesod Web Framework" group. To unsubscribe from this group and stop receiving emails from it, send an email to yesodweb+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

Michael,

1. How does Warp know which interface, port, and IP address the connection comes in on? The port is specified by the user, so there's no purpose in Warp providing that to the application. The interface is (mostly) user specified as well. But it's not a request-specific piece of information anyway. If you wanted that information, why should it be part of the Request value? Warp could just provide it separately via a separate run function, e.g.:

I guess they want to know peer's port number. I don't know why it is important. However, I think that peer's IP address is important. If an application can obtain combination of peer's IP address and HTTP header, the application can tell whether or not communication is encrypted. --Kazu

Michael,

I'm definitely not talking about removing the peer information. Request has a remoteHost field which is of type SockAddr, and therefore provides both remote IP address and port number (the latter, as you mention, being mostly useless).

Please understand that I'm not opposing your opinion. I'm just trying to interpret user's opinions.

I'm not sure what you mean by telling if communication is encrypted using the headers + IP address, can you clarify?

Suppose a Yesod application receives X-Forwarded-For:. A bad client can insert X-Forwarded-For:. But if an IP address is provided and the application knows the IP address of the proxy, the application can tell whether or not the IP address can be trusted. I'm not sure that there is a standard header field to tell HTTPS. But if the proxy and the application shares such field: - The application can truct the field according peer's IP address - The application can tell the outside HTTP is encrypted Correct me if I misunderstand. --Kazu

Michael, I agree that you are theoretically correct not to include extra info for example about approot. On the other hand, when someone comes from using Rails we have to explain to them this theory about why they have to manually enter approot and can't tell if their app is over https, but they will still scratch their head since in Rails that information was always readily available. So I am wondering if there is a middle ground to be found where being theoretically correct doesn't mean zero programmer convenience. But perhaps this has nothing to do with the WAI standard and should be handled by middleware? On Wed, Jul 24, 2013 at 5:23 AM, Michael Snoyman wrote:

On Wed, Jul 24, 2013 at 3:08 PM, Kazu Yamamoto wrote:

...

Michael,

...

I'm definitely not talking about removing the peer information. Request has a remoteHost field which is of type SockAddr, and therefore provides both remote IP address and port number (the latter, as you mention, being mostly useless).

Please understand that I'm not opposing your opinion. I'm just trying to interpret user's opinions.

I appreciate the input, I just wanted to clarify my proposal.

...

...

I'm not sure what you mean by telling if communication is encrypted using the headers + IP address, can you clarify?

Suppose a Yesod application receives X-Forwarded-For:.

A bad client can insert X-Forwarded-For:. But if an IP address is provided and the application knows the IP address of the proxy, the application can tell whether or not the IP address can be trusted.

I'm not sure that there is a standard header field to tell HTTPS. But if the proxy and the application shares such field: - The application can truct the field according peer's IP address - The application can tell the outside HTTP is encrypted

Correct me if I misunderstand.

I'd never considered checking the IP address of the proxy. When I've used X-Forwarded-For, I've always made sure to set up a firewall to ensure the *only* connections come from the reverse proxy, and then x-forwarded-for can be trusted.

As far as determining secure/insecure, it seems like Amazon's ELB sets x-forwarded-proto[1]. Not sure if that's a well accepted standard, but it seems useful.

Michael

[1] http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/Termin...

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

Clients directly accessing the server also makes sense when you have tens of thousands of open connections at a time (that would be me). The proxy overhead is just silly. As for HTTPS, isSecure is interesting when using something like warp-tls so as to handle insecure connections more elegantly. While the server can't know about if a proxy received a secure connection it can know if it did. This is even interesting when using a proxy since if the proxy is on another system it should still connect with HTTPS for a large number of threat models. Of course one may put that in the Vault. And don't forget CDN updates where connections that were being sent as HTTPS start being sent as HTTP due to the CDN operator's error. Perhaps you still want to serve most connections when they come in as HTTP from the CDN, send a warning to the ops team, and deny some requests as too sensitive? Plenty of cases here. This isn't about the entire history of the connection, it is about conveying the webserver's information to the app so the app writer can make appropriate choices to figure out what the history actually is given their specific situation. -davean PS: The below happens BTW: client <-> their work's proxy <-> their ISP's proxy <-> CDN <-> provider's proxy <-> final HTTP server That is forwarded 4 times. Of course someone in there can also be lying. I've even seen some pretty clever goes at forging X-Forward-Fors through (or around) proxies. On Wed, Jul 24, 2013 at 9:59 AM, Federico Mastellone wrote:

Wai should provide "a common protocol for communication between web applications and web servers.", using proxies, load balancers or not. I use Wai in a small setup (measured by clients), where clients access the server directly. I was even going to request support for client certificates.

Because some fields are unknown when running behind Amazon ELB I don't think they should be taken out of the Wai interface. In that case, that fields could be Nothing, False, etc. I think it would be great to write an Application that can be run with any Wai compatible server, and maybe later warp supports this fields behind a reverse proxy.

On 24/07/2013, at 09:23, Michael Snoyman wrote:

On Wed, Jul 24, 2013 at 3:08 PM, Kazu Yamamoto wrote:

...

Michael,

...

I'm definitely not talking about removing the peer information. Request has a remoteHost field which is of type SockAddr, and therefore provides both remote IP address and port number (the latter, as you mention, being mostly useless).

Please understand that I'm not opposing your opinion. I'm just trying to interpret user's opinions.

I appreciate the input, I just wanted to clarify my proposal.

...

...

I'm not sure what you mean by telling if communication is encrypted using the headers + IP address, can you clarify?

Suppose a Yesod application receives X-Forwarded-For:.

A bad client can insert X-Forwarded-For:. But if an IP address is provided and the application knows the IP address of the proxy, the application can tell whether or not the IP address can be trusted.

I'm not sure that there is a standard header field to tell HTTPS. But if the proxy and the application shares such field: - The application can truct the field according peer's IP address - The application can tell the outside HTTP is encrypted

Correct me if I misunderstand.

I'd never considered checking the IP address of the proxy. When I've used X-Forwarded-For, I've always made sure to set up a firewall to ensure the *only* connections come from the reverse proxy, and then x-forwarded-for can be trusted.

As far as determining secure/insecure, it seems like Amazon's ELB sets x-forwarded-proto[1]. Not sure if that's a well accepted standard, but it seems useful.

Michael

[1] http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/Termin...

_______________________________________________

web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

I explicitly described a scenario in my email. Can you tell me more about what you want me to describe? Unless you consider my Vault alternative as not in the Request. Not sure what backend couldn't reliably tell whether *it* was connected to securely. None can make claims about other steps in the chain (Which is also exactly why we need this value). -davean On Wed, Jul 24, 2013 at 11:29 AM, Michael Snoyman wrote:

My concern is that, by throwing in as much information as possible in Request, we'll essentially be diluting the meaning of the values there. My approach was to only include values in Request that can be reliably reported by all backends, or at the very least would clearly be nonsense for certain backends[1]. My big concern is making it easy to write incorrect code. My question is: what is a concrete example where you would want the isSecure field in Request? If you're talking about a case where you're using warp-tls and no reverse proxies, then you can tell your application (via a config parameter, for example) that this is the case.

Maybe I just don't have the right mental model for what people are expecting for WAI. I want to understand the use cases better, so please keep up the discussion :).

[1] Example: wai-test clearly has to provide a fake IP address for the remote end of the connection.

On Wed, Jul 24, 2013 at 6:10 PM, davean wrote:

...

Clients directly accessing the server also makes sense when you have tens of thousands of open connections at a time (that would be me). The proxy overhead is just silly.

As for HTTPS, isSecure is interesting when using something like warp-tls so as to handle insecure connections more elegantly. While the server can't know about if a proxy received a secure connection it can know if it did. This is even interesting when using a proxy since if the proxy is on another system it should still connect with HTTPS for a large number of threat models. Of course one may put that in the Vault.

And don't forget CDN updates where connections that were being sent as HTTPS start being sent as HTTP due to the CDN operator's error. Perhaps you still want to serve most connections when they come in as HTTP from the CDN, send a warning to the ops team, and deny some requests as too sensitive? Plenty of cases here.

This isn't about the entire history of the connection, it is about conveying the webserver's information to the app so the app writer can make appropriate choices to figure out what the history actually is given their specific situation.

-davean

PS: The below happens BTW:

client <-> their work's proxy <-> their ISP's proxy <-> CDN <-> provider's proxy <-> final HTTP server

That is forwarded 4 times. Of course someone in there can also be lying. I've even seen some pretty clever goes at forging X-Forward-Fors through (or around) proxies.

On Wed, Jul 24, 2013 at 9:59 AM, Federico Mastellone wrote:

...

Wai should provide "a common protocol for communication between web applications and web servers.", using proxies, load balancers or not. I use Wai in a small setup (measured by clients), where clients access the server directly. I was even going to request support for client certificates.

Because some fields are unknown when running behind Amazon ELB I don't think they should be taken out of the Wai interface. In that case, that fields could be Nothing, False, etc. I think it would be great to write an Application that can be run with any Wai compatible server, and maybe later warp supports this fields behind a reverse proxy.

On 24/07/2013, at 09:23, Michael Snoyman wrote:

On Wed, Jul 24, 2013 at 3:08 PM, Kazu Yamamoto wrote:

...

Michael,

...

I'm definitely not talking about removing the peer information. Request has a remoteHost field which is of type SockAddr, and therefore provides both remote IP address and port number (the latter, as you mention, being mostly useless).

Please understand that I'm not opposing your opinion. I'm just trying to interpret user's opinions.

I appreciate the input, I just wanted to clarify my proposal.

...

...

I'm not sure what you mean by telling if communication is encrypted using the headers + IP address, can you clarify?

Suppose a Yesod application receives X-Forwarded-For:.

A bad client can insert X-Forwarded-For:. But if an IP address is provided and the application knows the IP address of the proxy, the application can tell whether or not the IP address can be trusted.

I'm not sure that there is a standard header field to tell HTTPS. But if the proxy and the application shares such field: - The application can truct the field according peer's IP address - The application can tell the outside HTTP is encrypted

Correct me if I misunderstand.

I'd never considered checking the IP address of the proxy. When I've used X-Forwarded-For, I've always made sure to set up a firewall to ensure the *only* connections come from the reverse proxy, and then x-forwarded-for can be trusted.

As far as determining secure/insecure, it seems like Amazon's ELB sets x-forwarded-proto[1]. Not sure if that's a well accepted standard, but it seems useful.

Michael

[1] http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/Termin...

_______________________________________________

web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

Perhaps the part that might help you understand what I mean is: -- | An action when a plain HTTP comes to HTTP over TLS/SSL port. data OnInsecure = DenyInsecure L.ByteString | AllowInsecure ( http://hackage.haskell.org/packages/archive/warp-tls/1.4.1.3/doc/html/Networ... ) "And don't forget CDN updates where connections that were being sent as HTTPS start being sent as HTTP due to the CDN operator's error. Perhaps you still want to serve most connections when they come in as HTTP from the CDN, send a warning to the ops team, and deny some requests as too sensitive? Plenty of cases here." HTTP can come in on an HTTPS port. Your HTTPS server can get HTTP requests. Though in theory you could run an HTTP and an HTTPS server on the same port I suppose ... at an immense technical complexity. -davean On Wed, Jul 24, 2013 at 11:46 AM, Michael Snoyman wrote:

What I'm looking for as far as use case is "I want to do X, and if you put Y in the Request value, I could do X better." For example, "If I knew that a request was insecure, I could automatically redirect to the secure URL." My reasoning here is that, for all such use cases I've heard, determining if it's a secure connection from the Request value is wrong. In this case, you have to run two Warp servers anyway, one over HTTP and one over HTTPS. So the more logical thing would be to just have two separate applications, one with the redirect logic and one without.

And you're probably right; I can't think of any handlers that can't determine if they were connected to securely. But again, my concern is about including misleading information in Request which, given its lack of reliability, can't really be used. If there *are* real use cases that I'm not thinking of, that's a different story.

On Wed, Jul 24, 2013 at 6:36 PM, davean wrote:

...

I explicitly described a scenario in my email. Can you tell me more about what you want me to describe? Unless you consider my Vault alternative as not in the Request.

Not sure what backend couldn't reliably tell whether *it* was connected to securely. None can make claims about other steps in the chain (Which is also exactly why we need this value).

-davean

On Wed, Jul 24, 2013 at 11:29 AM, Michael Snoyman wrote:

...

My concern is that, by throwing in as much information as possible in Request, we'll essentially be diluting the meaning of the values there. My approach was to only include values in Request that can be reliably reported by all backends, or at the very least would clearly be nonsense for certain backends[1]. My big concern is making it easy to write incorrect code. My question is: what is a concrete example where you would want the isSecure field in Request? If you're talking about a case where you're using warp-tls and no reverse proxies, then you can tell your application (via a config parameter, for example) that this is the case.

Maybe I just don't have the right mental model for what people are expecting for WAI. I want to understand the use cases better, so please keep up the discussion :).

[1] Example: wai-test clearly has to provide a fake IP address for the remote end of the connection.

On Wed, Jul 24, 2013 at 6:10 PM, davean wrote:

...

Clients directly accessing the server also makes sense when you have tens of thousands of open connections at a time (that would be me). The proxy overhead is just silly.

As for HTTPS, isSecure is interesting when using something like warp-tls so as to handle insecure connections more elegantly. While the server can't know about if a proxy received a secure connection it can know if it did. This is even interesting when using a proxy since if the proxy is on another system it should still connect with HTTPS for a large number of threat models. Of course one may put that in the Vault.

And don't forget CDN updates where connections that were being sent as HTTPS start being sent as HTTP due to the CDN operator's error. Perhaps you still want to serve most connections when they come in as HTTP from the CDN, send a warning to the ops team, and deny some requests as too sensitive? Plenty of cases here.

This isn't about the entire history of the connection, it is about conveying the webserver's information to the app so the app writer can make appropriate choices to figure out what the history actually is given their specific situation.

-davean

PS: The below happens BTW:

client <-> their work's proxy <-> their ISP's proxy <-> CDN <-> provider's proxy <-> final HTTP server

That is forwarded 4 times. Of course someone in there can also be lying. I've even seen some pretty clever goes at forging X-Forward-Fors through (or around) proxies.

On Wed, Jul 24, 2013 at 9:59 AM, Federico Mastellone wrote:

...

Wai should provide "a common protocol for communication between web applications and web servers.", using proxies, load balancers or not. I use Wai in a small setup (measured by clients), where clients access the server directly. I was even going to request support for client certificates.

Because some fields are unknown when running behind Amazon ELB I don't think they should be taken out of the Wai interface. In that case, that fields could be Nothing, False, etc. I think it would be great to write an Application that can be run with any Wai compatible server, and maybe later warp supports this fields behind a reverse proxy.

On 24/07/2013, at 09:23, Michael Snoyman wrote:

On Wed, Jul 24, 2013 at 3:08 PM, Kazu Yamamoto wrote:

...

Michael,

> I'm definitely not talking about removing the peer information. Request has > a remoteHost field which is of type SockAddr, and therefore provides both > remote IP address and port number (the latter, as you mention, being mostly > useless).

Please understand that I'm not opposing your opinion. I'm just trying to interpret user's opinions.

I appreciate the input, I just wanted to clarify my proposal.

...

> I'm not sure what you mean by telling if communication is encrypted using > the headers + IP address, can you clarify?

Suppose a Yesod application receives X-Forwarded-For:.

A bad client can insert X-Forwarded-For:. But if an IP address is provided and the application knows the IP address of the proxy, the application can tell whether or not the IP address can be trusted.

I'm not sure that there is a standard header field to tell HTTPS. But if the proxy and the application shares such field: - The application can truct the field according peer's IP address - The application can tell the outside HTTP is encrypted

Correct me if I misunderstand.

I'd never considered checking the IP address of the proxy. When I've used X-Forwarded-For, I've always made sure to set up a firewall to ensure the *only* connections come from the reverse proxy, and then x-forwarded-for can be trusted.

As far as determining secure/insecure, it seems like Amazon's ELB sets x-forwarded-proto[1]. Not sure if that's a well accepted standard, but it seems useful.

Michael

[1] http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/Termin...

_______________________________________________

web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

I agree, that is a pure duplication. It is in he headers also and that is where it should be because that is what it is. -davean On Wed, Jul 24, 2013 at 11:47 PM, Michael Snoyman wrote:

OK, fair enough, you've convinced me. We can address the concerns that I have about this value being misleading with clearer documentation.

However, given that serverName is essentially just "the Host header, or whatever else I feel like giving you," I still think it makes sense to remove that one.

On Wed, Jul 24, 2013 at 6:52 PM, davean wrote:

...

Perhaps the part that might help you understand what I mean is:

-- | An action when a plain HTTP comes to HTTP over TLS/SSL port.

data OnInsecure = DenyInsecure L.ByteString | AllowInsecure

( http://hackage.haskell.org/packages/archive/warp-tls/1.4.1.3/doc/html/Networ... )

"And don't forget CDN updates where connections that were being sent as HTTPS start being sent as HTTP due to the CDN operator's error. Perhaps you still want to serve most connections when they come in as HTTP from the CDN, send a warning to the ops team, and deny some requests as too sensitive? Plenty of cases here."

HTTP can come in on an HTTPS port. Your HTTPS server can get HTTP requests.

Though in theory you could run an HTTP and an HTTPS server on the same port I suppose ... at an immense technical complexity.

-davean

On Wed, Jul 24, 2013 at 11:46 AM, Michael Snoyman wrote:

...

What I'm looking for as far as use case is "I want to do X, and if you put Y in the Request value, I could do X better." For example, "If I knew that a request was insecure, I could automatically redirect to the secure URL." My reasoning here is that, for all such use cases I've heard, determining if it's a secure connection from the Request value is wrong. In this case, you have to run two Warp servers anyway, one over HTTP and one over HTTPS. So the more logical thing would be to just have two separate applications, one with the redirect logic and one without.

And you're probably right; I can't think of any handlers that can't determine if they were connected to securely. But again, my concern is about including misleading information in Request which, given its lack of reliability, can't really be used. If there *are* real use cases that I'm not thinking of, that's a different story.

On Wed, Jul 24, 2013 at 6:36 PM, davean wrote:

...

I explicitly described a scenario in my email. Can you tell me more about what you want me to describe? Unless you consider my Vault alternative as not in the Request.

Not sure what backend couldn't reliably tell whether *it* was connected to securely. None can make claims about other steps in the chain (Which is also exactly why we need this value).

-davean

On Wed, Jul 24, 2013 at 11:29 AM, Michael Snoyman wrote:

...

My concern is that, by throwing in as much information as possible in Request, we'll essentially be diluting the meaning of the values there. My approach was to only include values in Request that can be reliably reported by all backends, or at the very least would clearly be nonsense for certain backends[1]. My big concern is making it easy to write incorrect code. My question is: what is a concrete example where you would want the isSecure field in Request? If you're talking about a case where you're using warp-tls and no reverse proxies, then you can tell your application (via a config parameter, for example) that this is the case.

Maybe I just don't have the right mental model for what people are expecting for WAI. I want to understand the use cases better, so please keep up the discussion :).

[1] Example: wai-test clearly has to provide a fake IP address for the remote end of the connection.

On Wed, Jul 24, 2013 at 6:10 PM, davean wrote:

...

Clients directly accessing the server also makes sense when you have tens of thousands of open connections at a time (that would be me). The proxy overhead is just silly.

As for HTTPS, isSecure is interesting when using something like warp-tls so as to handle insecure connections more elegantly. While the server can't know about if a proxy received a secure connection it can know if it did. This is even interesting when using a proxy since if the proxy is on another system it should still connect with HTTPS for a large number of threat models. Of course one may put that in the Vault.

And don't forget CDN updates where connections that were being sent as HTTPS start being sent as HTTP due to the CDN operator's error. Perhaps you still want to serve most connections when they come in as HTTP from the CDN, send a warning to the ops team, and deny some requests as too sensitive? Plenty of cases here.

This isn't about the entire history of the connection, it is about conveying the webserver's information to the app so the app writer can make appropriate choices to figure out what the history actually is given their specific situation.

-davean

PS: The below happens BTW:

client <-> their work's proxy <-> their ISP's proxy <-> CDN <-> provider's proxy <-> final HTTP server

That is forwarded 4 times. Of course someone in there can also be lying. I've even seen some pretty clever goes at forging X-Forward-Fors through (or around) proxies.

On Wed, Jul 24, 2013 at 9:59 AM, Federico Mastellone < fmaste@gmail.com> wrote:

> Wai should provide "a common protocol for communication between web > applications and web servers.", using proxies, load balancers or not. I > use Wai in a small setup (measured by clients), where clients access the > server directly. I was even going to request support for client > certificates. > > Because some fields are unknown when running behind Amazon ELB I > don't think they should be taken out of the Wai interface. In that case, > that fields could be Nothing, False, etc. I think it would be great to > write an Application that can be run with any Wai compatible server, and > maybe later warp supports this fields behind a reverse proxy. > > On 24/07/2013, at 09:23, Michael Snoyman > wrote: > > > > > On Wed, Jul 24, 2013 at 3:08 PM, Kazu Yamamoto wrote: > >> Michael, >> >> > I'm definitely not talking about removing the peer information. >> Request has >> > a remoteHost field which is of type SockAddr, and therefore >> provides both >> > remote IP address and port number (the latter, as you mention, >> being mostly >> > useless). >> >> Please understand that I'm not opposing your opinion. I'm just >> trying >> to interpret user's opinions. >> >> > I appreciate the input, I just wanted to clarify my proposal. > > >> > I'm not sure what you mean by telling if communication is >> encrypted using >> > the headers + IP address, can you clarify? >> >> Suppose a Yesod application receives X-Forwarded-For:. >> >> A bad client can insert X-Forwarded-For:. But if an IP address is >> provided and the application knows the IP address of the proxy, the >> application can tell whether or not the IP address can be trusted. >> >> I'm not sure that there is a standard header field to tell HTTPS. >> But if the proxy and the application shares such field: >> - The application can truct the field according peer's IP address >> - The application can tell the outside HTTP is encrypted >> >> Correct me if I misunderstand. >> >> > I'd never considered checking the IP address of the proxy. When I've > used X-Forwarded-For, I've always made sure to set up a firewall to ensure > the *only* connections come from the reverse proxy, and then > x-forwarded-for can be trusted. > > As far as determining secure/insecure, it seems like Amazon's ELB > sets x-forwarded-proto[1]. Not sure if that's a well accepted standard, but > it seems useful. > > Michael > > [1] > http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/Termin... > > _______________________________________________ > > web-devel mailing list > web-devel@haskell.org > http://www.haskell.org/mailman/listinfo/web-devel > > > _______________________________________________ > web-devel mailing list > web-devel@haskell.org > http://www.haskell.org/mailman/listinfo/web-devel > >

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

No, assume there *is no proxy*. All I'm talking about is what the webserver its self sees. Anything to do with proxies or other infrastructure really isn't WAI's place. -davean On Wed, Jul 24, 2013 at 5:46 PM, Rico Moorman wrote:

Dear Davean,

I hope that I correctly interpret your mails. I guess you are talking about connection information available at the proxy when the user makes a connection. And usually this information is passed on to the backends using special headers such as x-forwarded-for and x-forwarded-proto. Right?

Isn't this information readily available using the requestHeaders function which well, should return those headers?

Furthermore I would think that the application in question would be responsible to interpret the available information correctly (this would be the framework on top of e.g. warp I guess).

The framework should then provide measures to correctly generate urls/links based on this information…as in fact done by various frameworks.

best regards,

Rico Moorman

On Jul 24, 2013, at 12:59 PM, davean wrote:

Completely not what I was saying.

"what IP they connected to, what port, on what interface, and even what ciphers were used ideally."

None of that is information the user could ever pass to the application except in the truly trivial cases. They're inherently features of each connection separately. If it isn't passed in the Wai Request it can't be (reasonably) discovered.

If some thing is going to try to derive for the user the pre-proxy IP based the connection information it should be a library that does such preferably or alternatively a middleware component, never the webserver. Furthermore WAI should contain sufficient information in the Request object to do this and more.

-davean

On Wed, Jul 24, 2013 at 6:13 AM, Michael Snoyman wrote:

...

I think Greg and davean are saying something similar here. If I may paraphrase (please correct me if I'm wrong): some information may not be derivable by the WAI handler (e.g. Warp) itself, but with a little bit of help from the user, that information could be included in the WAI standard.

If that's what you're getting at, I disagree. This would essentially mean we require users to provide information to Warp, so that Warp can provide that information to user applications. Warp (or any other handler) is just a dumb transport mechanism then. And since we can't rely on that information being present, we'd wrap it up with Maybe values, and therefore end up losing type safety.

The prime example of all this is application root. I still strongly believe that Yesod did this the right way: if the user needs to provide the information, then ask the user to provide the information to the application itself, and the WAI handler needn't know anything about it.

Is there some kind of data that the WAI handler could do a better job of passing onto the application than the user can do?

On Tue, Jul 23, 2013 at 11:58 PM, Greg Weber wrote:

...

In Rack most of these things are known. In some cases this requires users to configure the proxy to send headers.

One solution would be having the user state what values are being depended on and having Warp fail on startup with an error message if it can't collect them. It is possible to expect some of these values to not change (and require a reboot of warp when they do) and have them be part of Warp rather than WAI.

But just wrapping these values in a Maybe could be another approach.

On Tue, Jul 23, 2013 at 1:13 PM, Michael Snoyman wrote:

...

The reason we'd be removing those is that there *still* isn't any way to know this. If Warp is sitting behind a reverse proxy (which is virtually always the case, whether Amazon ELB, Nginx, Keter, or something else) there's no way to know what the actual server name, server port, or scheme are.

On Tue, Jul 23, 2013 at 6:37 PM, Federico Mastellone wrote:

...

Hi,

Without isSecure and serverPort, will Applications be able to handle requests to different ports and HTTP or HTTPS in different ways? I won't be able to know if the login is made over HTTP and redirect to HTTPS with any Wai server.

On 23/07/2013, at 07:04, Michael Snoyman wrote:

Some of you may have seen the brief interchange between Kazu and myself regarding ResourceT in WAI. Let me give a very brief synopsis of the situation:

- In order to allow WAI applications to acquire scarce resources like file handles in an exception-safe way, WAI actions all live in a `ResourceT IO` monad, instead of just `IO`. - Kazu has mentioned to me in the past that, based on his profiling, monadic bind for ResourceT may currently be a bottleneck in WAI. - I made an experimental change to WAI so that, instead of actions living in `ResourceT IO`, the WAI Request value contains a ResourceT InternalState value, which can be used to accomplish the exact same thing as living in ResourceT (see [1]). - Kazu checked out the performance difference of this new branch, and it allows for some better optimizations.

If this was purely a performance optimization, I would probably just optimize Warp and not bother changing WAI. However, I think that this change makes WAI itself better as well, and therefore am in favor of making this as a breaking change, releasing it as WAI 2.0[2]. Firstly, are there any objections to this move?

So as long as we're making a breaking change, the question arises: what other changes should we be making to WAI? I know Kazu had mentioned adding fields to avoid the need for lookups, can you clarify that request a bit? Here are some other changes I can think of:

- Remove the deprecated isSecure field from the Request data type. - Remove the misleading fields serverName and serverPort fields as well. - Possibly: hide the Request constructor so that new fields can be added without breaking backwards compatibility.

If anyone else has some ideas, please bring them up.

[1] http://haddocks.fpcomplete.com/fp/7.4.2/20130704-120/resourcet/Control-Monad... [2] For the Yesod community: the change is minor enough that Yesod 1.2 can have some conditional compilation to support both the current WAI 1.4 and WAI 2.0, so this shouldn't affect most Yesod users at all.

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

-- You received this message because you are subscribed to the Google Groups "Yesod Web Framework" group. To unsubscribe from this group and stop receiving emails from it, send an email to yesodweb+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.

-- You received this message because you are subscribed to the Google Groups "Yesod Web Framework" group. To unsubscribe from this group and stop receiving emails from it, send an email to yesodweb+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel

_______________________________________________ web-devel mailing list web-devel@haskell.org http://www.haskell.org/mailman/listinfo/web-devel