On Sun, Jan 22, 2012 at 11:07 PM, Myles C. Maxfield wrote:

Replies are inline. Thanks for the quick and thoughtful response!

On Sat, Jan 21, 2012 at 8:56 AM, Michael Snoyman wrote:

...

Hi Myles,

These sound like two solid features, and I'd be happy to merge in code to support it. Some comments below.

On Sat, Jan 21, 2012 at 8:38 AM, Myles C. Maxfield wrote:

...

To: Michael Snoyman, author and maintainer of http-conduit CC: haskell-cafe

Hello!

I am interested in contributing to the http-conduit library. I've been using it for a little while and reading through its source, but have felt that it could be improved with two features:

Allowing the caller to know the final URL that ultimately resulted in the HTTP Source. Because httpRaw is not exported, the caller can't even re-implement the redirect-following code themselves. Ideally, the caller would be able to know not only the final URL, but also the entire chain of URLs that led to the final request. I was thinking that it would be even cooler if the caller could be notified of these redirects as they happen in another thread. There are a couple ways to implement this that I have been thinking about:

A straightforward way would be to add a [W.Ascii] to the type of Response, and getResponse can fill in this extra field. getResponse already knows about the Request so it can tell if the response should be gunzipped.

What would be in the [W.Ascii], a list of all paths redirected to? Also, I'm not sure what gunzipping has to do with here, can you clarify?

Yes; my idea was to make the [W.Ascii] represent the list of all URLs redirected to, in order.

My comment about gunzipping is only tangentially related. I meant that in the latest version of the code on GitHub, the getResponse function already takes a Request as an argument. This means that the getResponse function already knows what URL its data is coming from, so modifying the getResponse function to return that URL is simple. (I mentioned gunzip because, as far as I can tell, the reason that getResponse already takes a Request is so that the function can tell if the request should be gunzipped.)

...

...

It would be nice for the caller to be able to know in real time what URLs the request is being redirected to. A possible way to do this would be for the 'http' function to take an extra argument of type (Maybe (Control.Concurrent.Chan W.Ascii)) which httpRaw can push URLs into. If the caller doesn't want to use this variable, they can simply pass Nothing. Otherwise, the caller can create an IO thread which reads the Chan until some termination condition is met (Perhaps this will change the type of the extra argument to (Maybe (Chan (Maybe W.Ascii)))). I like this solution, though I can see how it could be considered too heavyweight.

I do think it's too heavyweight. I think if people really want lower-level control of the redirects, they should turn off automatic redirect and allow 3xx responses.

Yeah, that totally makes more sense. As it stands, however, httpRaw isn't exported, so a caller has no way of knowing about each individual HTTP transaction. Exporting httpRaw solves the problem I'm trying to solve. If we export httpRaw, should we also make 'http' return the URL chain? Doing both is probably the best solution, IMHO.

What's the difference between calling httpRaw and calling http with redirections turned off?

...

...

Making the redirection aware of cookies. There are redirects around the web where the first URL returns a Set-Cookie header and a 3xx code which redirects to another site that expects the cookie that the first HTTP transaction set. I propose to add an (IORef to a Data.Set of Cookies) to the Manager datatype, letting the Manager act as a cookie store as well as a repository of available TCP connections. httpRaw could deal with the cookie store. Network.HTTP.Types does not declare a Cookie datatype, so I would probably be adding one. I would probably take it directly from Network.HTTP.Cookie.

Actually, we already have the cookie package for this. I'm not sure if putting the cookie store in the manager is necessarily the right approach, since I can imagine wanting to have separate sessions while reusing the same connections. A different approach could be adding a list of Cookies to both the Request and Response.

Ah, looks like you're the maintainer of that package as well! I didn't realize it existed. I should have, though; Yesod must need to know about cookies somehow.

As the http-conduit package stands, the headers of the original Request can be set, and the headers of the last Response can be read. Because cookies are implemented on top of headers, the caller knows about the cookies before and after the redirection chain. I'm more interested in the preservation of cookies within the redirection chain. As discussed earlier, exposing the httpRaw function allows the entire redirection chain to be handled by the caller, which alleviates the problem.

That being said, however, the simpleHttp function (and all functions built upon 'http' inside of http-conduit) should probably respect cookies inside redirection chains. Under the hood, Network.Browser does this by having the State monad keep track of these cookies (as well as the connection pool) and making HTTP requests mutate that State, but that's a pretty different architecture than Network.HTTP.Conduit.

One way I can think to do this would be to let the user supply a CookieStore (probably implemented as a (Data.Set Web.Cookie.SetCookie)) and receive a (different) CookieStore from the 'http' function. That way, the caller can manage the CookieStores independently from the connection pool. The downside is that it's one more bit of ugliness the caller has to deal with. How do you feel about this? You probably have a better idea :-)

The only idea was to implement an extra layer of cookie-away functions in a separate Browser module. That's been the running assumption for a while now, since HTTP does it, but I'm not opposed to taking a different approach. It could be that the big mistake in all this was putting redirection at the layer of the API that I did. Yitz Gale pointed out that in Python, they have the low-level API and the high-level API, the latter dealing with both redirection and cookies. Anyway, here's one possible approach to the whole situation: `Request` could have an extra record on it of type `Maybe (IORef (Set SetCookie))`. When `http` is called, if the record is `Nothing`, a new value is created. Every time a request is made, the value is updated accordingly. That way, redirects will respect cookies for the current sessions, and if you want to keep a longer-term session, you can keep reusing the record in different `Request`s. We can also add some convenience functions to automatically reuse the cookie set. Michael

...

I'd be happy to do both of these things, but I'm hoping for your input on how to go about this endeavor. Are these features even good to be pursuing? Should I be going about this entirely differently?

Thanks, Myles C. Maxfield

P.S. I'm curious about the lack of Network.URI throughout Network.HTTP.Conduit. Is there a particular design decision that led you to use raw ascii strings?

Because there are plenty of URIs that are valid that we don't handle at all, e.g., ftp.

I'm a little surprised by this, since you can easily test for unhandled URIs because they're already parsed. Whatever; It doesn't really matter to me, I was just surprised by it.

Michael

Thanks again for the feedback! I'm hoping to make a difference :]

--Myles

On Mon, Jan 23, 2012 at 8:31 AM, Myles C. Maxfield wrote:

1. Oops - I overlooked the fact that the redirectCount attribute of a Request is exported (it isn't listed on the documentation probably because the constructor itself isn't exported. This seems like a flaw in Haddock...). Silly me. No need to export httpRaw.

2. I think that stuffing many arguments into the 'http' function is ugly. However, I'm not sure that the number of arguments to 'http' could ever reach an unreasonably large amount. Perhaps I have bad foresight, but I personally feel that adding cookies to the http request will be the last thing that we will need to add. Putting a bound on this growth of arguments

I completely disagree here. If we'd followed this approach, rawBody, decompress, redirectCount, and checkStatus all would have been arguments. There's a reason we use a settings data type[1] here. [1] http://www.yesodweb.com/blog/2011/10/settings-types

makes me more willing to think about this option. On the other hand, using a BrowserAction to modify internal state is very elegant. Which approach do you think is best? I think I'm leaning toward the upper-level Browser module idea.

If there was to be a higher-level HTTP library, I would argue that the redirection code should be moved into it, and the only high-level function that the Network.HTTP.Conduit module would export is 'http' (or httpRaw). What do you think about this?

I actually don't want to move the redirection code out from where it is right now. I think that redirection *is* a basic part of HTTP. I'd be more in favor of just bundling cookies in with the current API, possibly with the IORef approach I'd mentioned (unless someone wants to give a different idea). Having a single API that provides both high-level and low-level approaches seems like a win to me. Michael

The only times cookies would be used would be: 1. If you explicitly use it. 2. If you have redirects turned on, and a page that redirects you also sets a cookie. I would think that we would want (2) to be on regardless of user setting, do you disagree? Michael On Mon, Jan 23, 2012 at 8:46 AM, Aristid Breitkreuz wrote:

Just make sure Cookie handling can be disabled completely.

Aristid

Am 23.01.2012 07:44 schrieb "Michael Snoyman" :

...

On Mon, Jan 23, 2012 at 8:31 AM, Myles C. Maxfield wrote:

...

1. Oops - I overlooked the fact that the redirectCount attribute of a Request is exported (it isn't listed on the documentation probably because the constructor itself isn't exported. This seems like a flaw in Haddock...). Silly me. No need to export httpRaw.

2. I think that stuffing many arguments into the 'http' function is ugly. However, I'm not sure that the number of arguments to 'http' could ever reach an unreasonably large amount. Perhaps I have bad foresight, but I personally feel that adding cookies to the http request will be the last thing that we will need to add. Putting a bound on this growth of arguments

I completely disagree here. If we'd followed this approach, rawBody, decompress, redirectCount, and checkStatus all would have been arguments. There's a reason we use a settings data type[1] here.

[1] http://www.yesodweb.com/blog/2011/10/settings-types

...

makes me more willing to think about this option. On the other hand, using a BrowserAction to modify internal state is very elegant. Which approach do you think is best? I think I'm leaning toward the upper-level Browser module idea.

If there was to be a higher-level HTTP library, I would argue that the redirection code should be moved into it, and the only high-level function that the Network.HTTP.Conduit module would export is 'http' (or httpRaw). What do you think about this?

I actually don't want to move the redirection code out from where it is right now. I think that redirection *is* a basic part of HTTP. I'd be more in favor of just bundling cookies in with the current API, possibly with the IORef approach I'd mentioned (unless someone wants to give a different idea). Having a single API that provides both high-level and low-level approaches seems like a win to me.

Michael

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Indeed, I disagree on 2. Sometimes there is an API and cookies are just not part of it (and redirects are). Aristid Am 23.01.2012 08:16 schrieb "Michael Snoyman" :

The only times cookies would be used would be:

1. If you explicitly use it. 2. If you have redirects turned on, and a page that redirects you also sets a cookie.

I would think that we would want (2) to be on regardless of user setting, do you disagree?

Michael

On Mon, Jan 23, 2012 at 8:46 AM, Aristid Breitkreuz wrote:

...

Just make sure Cookie handling can be disabled completely.

Aristid

Am 23.01.2012 07:44 schrieb "Michael Snoyman" :

...

On Mon, Jan 23, 2012 at 8:31 AM, Myles C. Maxfield wrote:

...

1. Oops - I overlooked the fact that the redirectCount attribute of a Request is exported (it isn't listed on the documentation probably because the constructor itself isn't exported. This seems like a flaw in Haddock...). Silly me. No need to export httpRaw.

2. I think that stuffing many arguments into the 'http' function is ugly. However, I'm not sure that the number of arguments to 'http' could

ever

...

...

reach an unreasonably large amount. Perhaps I have bad foresight, but I personally feel that adding cookies to the http request will be the last thing that we will need to add. Putting a bound on this growth of arguments

I completely disagree here. If we'd followed this approach, rawBody, decompress, redirectCount, and checkStatus all would have been arguments. There's a reason we use a settings data type[1] here.

[1] http://www.yesodweb.com/blog/2011/10/settings-types

...

makes me more willing to think about this option. On the other hand, using a BrowserAction to modify internal state is very elegant. Which approach do you think is best? I think I'm leaning toward the upper-level Browser module idea.

If there was to be a higher-level HTTP library, I would argue that the redirection code should be moved into it, and the only high-level function that the Network.HTTP.Conduit module would export is 'http' (or httpRaw). What do you think about this?

I actually don't want to move the redirection code out from where it is right now. I think that redirection *is* a basic part of HTTP. I'd be more in favor of just bundling cookies in with the current API, possibly with the IORef approach I'd mentioned (unless someone wants to give a different idea). Having a single API that provides both high-level and low-level approaches seems like a win to me.

Michael

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Rejecting cookies is not without precedent. If you must force cookie handling upon us, at least make it possible to selectively reject them. Aristid Am 23.01.2012 08:44 schrieb "Michael Snoyman" :

That's a violation of the spec. Having a server set a cookie and then "not really mean it" or something along those lines would be invalid. And having a server not set a cookie at all means having this feature would be irrelevant.

On Mon, Jan 23, 2012 at 9:42 AM, Aristid Breitkreuz wrote:

...

Indeed, I disagree on 2. Sometimes there is an API and cookies are just not part of it (and redirects are).

Aristid

Am 23.01.2012 08:16 schrieb "Michael Snoyman" :

...

The only times cookies would be used would be:

1. If you explicitly use it. 2. If you have redirects turned on, and a page that redirects you also sets a cookie.

I would think that we would want (2) to be on regardless of user setting, do you disagree?

Michael

On Mon, Jan 23, 2012 at 8:46 AM, Aristid Breitkreuz wrote:

...

Just make sure Cookie handling can be disabled completely.

Aristid

Am 23.01.2012 07:44 schrieb "Michael Snoyman" :

...

On Mon, Jan 23, 2012 at 8:31 AM, Myles C. Maxfield wrote:

...

1. Oops - I overlooked the fact that the redirectCount attribute

of a

...

...

Request is exported (it isn't listed on the documentation probably because the constructor itself isn't exported. This seems like a flaw in Haddock...). Silly me. No need to export httpRaw.

2. I think that stuffing many arguments into the 'http' function is ugly. However, I'm not sure that the number of arguments to 'http' could ever reach an unreasonably large amount. Perhaps I have bad foresight, but I personally feel that adding cookies to the http request will be the last thing that we will need to add. Putting a bound on this growth of arguments

I completely disagree here. If we'd followed this approach, rawBody, decompress, redirectCount, and checkStatus all would have been arguments. There's a reason we use a settings data type[1] here.

[1] http://www.yesodweb.com/blog/2011/10/settings-types

...

makes me more willing to think about this option. On the other hand, using a BrowserAction to modify internal state is very elegant. Which approach do you think is best? I think I'm leaning toward the upper-level Browser module idea.

If there was to be a higher-level HTTP library, I would argue that the redirection code should be moved into it, and the only high-level function that the Network.HTTP.Conduit module would export is 'http' (or httpRaw). What do you think about this?

I actually don't want to move the redirection code out from where it is right now. I think that redirection *is* a basic part of HTTP. I'd be more in favor of just bundling cookies in with the current API, possibly with the IORef approach I'd mentioned (unless someone wants to give a different idea). Having a single API that provides both high-level and low-level approaches seems like a win to me.

Michael

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

I have attached a patch to add a redirect chain to the Response datatype. Comments on this patch are very welcome. I was originally going to include the entire Request object in the redirection chain, but Request objects are parameterized with a type 'm', so including a 'Request m' field would force the Response type to be parameterized as well. I felt that would be too large a change, so I made the type of the redirection chain W.Ascii. Perhaps its worth using the 'forall' keyword to get rid of the pesky 'm' type parameter for Requests? data RequestBody = RequestBodyLBS L.ByteString | RequestBodyBS S.ByteString | RequestBodyBuilder Int64 Blaze.Builder | forall m. RequestBodySource Int64 (C.Source m Blaze.Builder) | forall m. RequestBodySourceChunked (C.Source m Blaze.Builder) --Myles On Mon, Jan 23, 2012 at 3:31 AM, Michael Snoyman wrote:

On Mon, Jan 23, 2012 at 1:20 PM, Aristid Breitkreuz wrote:

...

Rejecting cookies is not without precedent.

If you must force cookie handling upon us, at least make it possible to selectively reject them.

Aristid

If you turn off automatic redirects, then you won't have cookie handling. I'd be interested to hear of a use case where you would want to avoid passing cookies after a redirect.

Michael

On Mon, Jan 23, 2012 at 10:43 PM, Michael Snoyman wrote:

On Tue, Jan 24, 2012 at 8:37 AM, Myles C. Maxfield wrote:

...

I have attached a patch to add a redirect chain to the Response datatype. Comments on this patch are very welcome.

I thought that this isn't necessary since a client wanting to track all the redirects could just handle them manually by setting the redirect count to 0.

It seems like a lot of work to re-implement the redirection-following code, just to know which URL the bytes are coming from. I feel that adding this field makes the library easier to use, but it's your call.

...

I was originally going to include the entire Request object in the redirection chain, but Request objects are parameterized with a type 'm', so including a 'Request m' field would force the Response type to be parameterized as well. I felt that would be too large a change, so I made the type of the redirection chain W.Ascii.

Perhaps its worth using the 'forall' keyword to get rid of the pesky 'm' type parameter for Requests?

data RequestBody = RequestBodyLBS L.ByteString | RequestBodyBS S.ByteString | RequestBodyBuilder Int64 Blaze.Builder | forall m. RequestBodySource Int64 (C.Source m Blaze.Builder) | forall m. RequestBodySourceChunked (C.Source m Blaze.Builder)

There'd be no way to run the request body then (try compiling the code after that change).

Yeah, I never actually tried this change to see if it works. I'll try it tonight after work.

Michael

Thanks, Myles

Sorry, I don't think I'm following. What would the meaning of the value returned from checkRedirect be? --Myles On Tue, Jan 24, 2012 at 10:47 AM, Michael Snoyman wrote:

On Tue, Jan 24, 2012 at 6:57 PM, Myles C. Maxfield wrote:

...

On Mon, Jan 23, 2012 at 10:43 PM, Michael Snoyman wrote:

...

On Tue, Jan 24, 2012 at 8:37 AM, Myles C. Maxfield wrote:

...

I have attached a patch to add a redirect chain to the Response datatype. Comments on this patch are very welcome.

I thought that this isn't necessary since a client wanting to track all the redirects could just handle them manually by setting the redirect count to 0.

It seems like a lot of work to re-implement the redirection-following

...

just to know which URL the bytes are coming from. I feel that adding

code, this

...

field makes the library easier to use, but it's your call.

If that's the concern, I'd much rather just expose a function to help with dealing with redirects, rather than sticking a rather arbitrary [Ascii] in everyone's Response. I think a function along the lines of:

checkRedirect :: Response -> Maybe Request

would fit the bill, and could be extracted from the current `http` function.

Michael

Sorry, I think I'm still a little confused about this.

From the point of view of a library user, if I use the 'http' function, but want to know what final URL I ended up at, I would have to set redirects to 0, call http, call checkRedirect, and recurse until checkRedirect returns Nothing (or a count runs out). I would be handling the recursion of redirects myself.

On one hand, this solution is lightweight and easy to implement in the library. On the other hand, the caller has to run each individual request themselves, keeping track of the number of requests (so there isn't an infinite loop). The loop is already implemented in the http function - I think it's reasonable to modify the existing loop rather than expect the caller to re-implement that logic. However, it's probably just as reasonable to say "if you want to know what URL you end up at, you have to re-implement your own redirection-following logic." I do agree, however, that including an (possibly long, though explicitly bounded) [Ascii] along with every request is arbitrary, and probably not the best solution. Can you think of a solution which allows the caller to know the url chain (or possibly just the last URL - that's the important one) without having to re-implement the redirection-following logic themselves? It sounds like if you had to choose, you would rather force a caller to re-implement redirection-following rather than include a URL chain in every Response. Is this correct? Thanks for helping me out with this, Myles C. Maxfield On Tue, Jan 24, 2012 at 8:05 PM, Michael Snoyman wrote:

It would be the new request indicated by the server response, if the server gave a redirect response.

On Tue, Jan 24, 2012 at 9:05 PM, Myles C. Maxfield wrote:

...

Sorry, I don't think I'm following. What would the meaning of the value returned from checkRedirect be?

--Myles

On Tue, Jan 24, 2012 at 10:47 AM, Michael Snoyman wrote:

...

On Tue, Jan 24, 2012 at 6:57 PM, Myles C. Maxfield wrote:

...

On Mon, Jan 23, 2012 at 10:43 PM, Michael Snoyman <

michael@snoyman.com>

...

...

wrote:

...

On Tue, Jan 24, 2012 at 8:37 AM, Myles C. Maxfield wrote:

...

I have attached a patch to add a redirect chain to the Response datatype. Comments on this patch are very welcome.

I thought that this isn't necessary since a client wanting to track all the redirects could just handle them manually by setting the redirect count to 0.

It seems like a lot of work to re-implement the redirection-following code, just to know which URL the bytes are coming from. I feel that adding this field makes the library easier to use, but it's your call.

If that's the concern, I'd much rather just expose a function to help with dealing with redirects, rather than sticking a rather arbitrary [Ascii] in everyone's Response. I think a function along the lines of:

checkRedirect :: Response -> Maybe Request

would fit the bill, and could be extracted from the current `http` function.

Michael

The nice thing is that this way, nobody can force me to handle cookies. ;-) Might be that usage patterns emerge, which we can then codify into functions later. Am 25.01.2012 08:09 schrieb "Michael Snoyman" :

...

Sorry, I think I'm still a little confused about this.

From the point of view of a library user, if I use the 'http' function, but want to know what final URL I ended up at, I would have to set redirects to 0, call http, call checkRedirect, and recurse until checkRedirect returns Nothing (or a count runs out). I would be handling the recursion of redirects myself.

On one hand, this solution is lightweight and easy to implement in the library. On the other hand, the caller has to run each individual request themselves, keeping track of the number of requests (so there isn't an infinite loop). The loop is already implemented in the http function - I think it's reasonable to modify the existing loop rather than expect the caller to re-implement that logic.

However, it's probably just as reasonable to say "if you want to know what URL you end up at, you have to re-implement your own redirection-following logic."

I do agree, however, that including an (possibly long, though explicitly bounded) [Ascii] along with every request is arbitrary, and probably not

...

best solution. Can you think of a solution which allows the caller to know the url chain (or possibly just the last URL - that's the important one) without having to re-implement the redirection-following logic

On Wed, Jan 25, 2012 at 9:01 AM, Myles C. Maxfield wrote: the themselves?

...

It sounds like if you had to choose, you would rather force a caller to re-implement redirection-following rather than include a URL chain in

every

...

Response. Is this correct?

That's correct. I think knowing the final URL is a fairly arbitrary requirement, in the same boat as wanting redirect handling without supporting cookies. These to me fall well within the 20%: most people won't need them, so the API should not be optimized for them.

There's also the fact that [Ascii] isn't nearly enough information to properly follow the chain. Next someone's going to want to know if a request was GET or POST, or whether it was a permanent or temporary redirect, or the exact text of the location header, or a million other things involved. If someone wants this stuff, they should reimplement the redirect logic.

But I don't really see this as being such a burden. If we had the checkRedirect function, it would look something like:

myHttp req = do let req' = req { redirectCount = 0 } res <- http req' case checkRedirect res of Just req2 -> myHttp req2 Nothing -> return res

I don't think that's a terrible burden.

Michael

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

On Wed, Jan 25, 2012 at 8:18 PM, Myles C. Maxfield wrote:

Alright, that's fine. I just wanted to be explicit about the interface we'd be providing. Taking the Request construction code out of 'http' and putting it into its own function should be a quick change - I'll have it to you soon. One possible wrench - The existing code copies some fields (like the proxy) from the original request. In order to keep this functionality, the signature would have to be:

checkRedirect :: Request m -> Response -> Maybe (Request m)

Is that okay with you? I think I'd also like to call the function something different, perhaps 'getRedirectedRequest'. Is that okay? I'll also add an example to the documentation about how a caller would get the redirection chain by re-implementing redirection (by using the example in your previous email).

Sounds great.

As for cookie handling - I think Network.Browser has a pretty elegant solution to this. They allow a "CookieFilter" which has type of URI -> Cookie -> IO Bool. Cookies are only put in the cookie jar if the function returns True. There is a default CookieFilter, which behaves as we would expect, but the user can override this function. That way, if you don't want to support cookies, you can just pass in (\ _ _ -> return False).

Also sounds good.

If we're already expecting people that want specific functionality to re-implement the redirect-following code, this solution might be unnecessary. Do you think that such a concept would be beneficial for Network.HTTP.Conduit to implement?

Yes, I can imagine that some people would want more fine-grained control of which cookies are accepted.

Either way, I'll probably end up making a solution similar to your checkRedirect function that will just allow people to take SetCookies out of a Response and put Cookies into a Request. I'll probably also provide a default function which converts a SetCookie into a cookie by looking up the current time, inspecting the Request, etc. This will allow me to not have to change the type of Request or Response - the functions I'll be writing can deal with the raw Headers that are already in Requests and Responses. Modifying 'http' to use these functions will be straightforward.

How does this sound to you?

Sounds like a good plan to me. I'm not entirely certain how you're planning on implementing the cookie jar itself. In other words, if I make a request, have a cookie set, and then make another request later, where will the cookie be stored in the interim, and how will the second request know to use it? Michael

Here is a patch regarding getRedirectedRequest. Comments are very welcome. --Myles C. Maxfield On Wed, Jan 25, 2012 at 10:21 PM, Myles C. Maxfield < myles.maxfield@gmail.com> wrote:

I was planning on making the caller deal with keeping track of cookies between requests. My cookie idea only solves the problem of cookies persisting through a redirect chain - not between subsequent request chains.

Do you think that Network.HTTP.Conduit should have a persistent cookie jar between caller's requests? I don't really think so.

--Myles

On Wed, Jan 25, 2012 at 9:28 PM, Michael Snoyman wrote:

...

...

Alright, that's fine. I just wanted to be explicit about the interface we'd be providing. Taking the Request construction code out of 'http' and

...

it into its own function should be a quick change - I'll have it to you soon. One possible wrench - The existing code copies some fields (like

...

proxy) from the original request. In order to keep this functionality,

...

signature would have to be:

checkRedirect :: Request m -> Response -> Maybe (Request m)

Is that okay with you? I think I'd also like to call the function something different, perhaps 'getRedirectedRequest'. Is that okay? I'll also add an example to the documentation about how a caller would get the redirection chain by re-implementing redirection (by using the example in your

On Wed, Jan 25, 2012 at 8:18 PM, Myles C. Maxfield wrote: putting the the previous

...

email).

Sounds great.

...

As for cookie handling - I think Network.Browser has a pretty elegant solution to this. They allow a "CookieFilter" which has type of URI -> Cookie -> IO Bool. Cookies are only put in the cookie jar if the function returns True. There is a default CookieFilter, which behaves as we would expect, but the user can override this function. That way, if you don't want to support cookies, you can just pass in (\ _ _ -> return False).

Also sounds good.

...

If we're already expecting people that want specific functionality to re-implement the redirect-following code, this solution might be unnecessary. Do you think that such a concept would be beneficial for Network.HTTP.Conduit to implement?

Yes, I can imagine that some people would want more fine-grained control of which cookies are accepted.

...

Either way, I'll probably end up making a solution similar to your checkRedirect function that will just allow people to take SetCookies out of a Response and put Cookies into a Request. I'll probably also provide a default function which converts a SetCookie into a cookie by looking up the current time, inspecting the Request, etc. This will allow me to not have to change the type of Request or Response - the functions I'll be writing can deal with the raw Headers that are already in Requests and Responses. Modifying 'http' to use these functions will be straightforward.

How does this sound to you?

Sounds like a good plan to me. I'm not entirely certain how you're planning on implementing the cookie jar itself. In other words, if I make a request, have a cookie set, and then make another request later, where will the cookie be stored in the interim, and how will the second request know to use it?

Michael

the fromJust should never fail, beceause of the guard statement: | 300 <= code && code < 400 && isJust l'' && isJust l' = Just $ req Because of the order of the && operators, it will only evaluate fromJust after it makes sure that the argument isJust. That function in particular shouldn't throw any exceptions - it should only return Nothing. Knowing that, I don't quite think I understand what your concern is. Can you elaborate? Thanks, Myles On Thu, Jan 26, 2012 at 12:59 AM, Michael Snoyman wrote:

I'm a little worried about the use of `fromJust`, it will give users a very confusing error message, and the error might be trigged at the wrong point in the computation. I'd feel better if checkRedirect lived in either some Failure, an Either, or maybe even in IO itself. IO might make sense if we want to implement some cookie jar functionality in the future via mutable references.

Michael

...

Here is a patch regarding getRedirectedRequest. Comments are very welcome.

--Myles C. Maxfield

On Wed, Jan 25, 2012 at 10:21 PM, Myles C. Maxfield wrote:

...

I was planning on making the caller deal with keeping track of cookies between requests. My cookie idea only solves the problem of cookies persisting through a redirect chain - not between subsequent request

chains.

...

Do you think that Network.HTTP.Conduit should have a persistent cookie

jar

...

between caller's requests? I don't really think so.

--Myles

On Wed, Jan 25, 2012 at 9:28 PM, Michael Snoyman wrote:

...

On Wed, Jan 25, 2012 at 8:18 PM, Myles C. Maxfield wrote:

...

Alright, that's fine. I just wanted to be explicit about the

interface

...

...

we'd be providing. Taking the Request construction code out of 'http' and putting it into its own function should be a quick change - I'll have it to you soon. One possible wrench - The existing code copies some fields (like the proxy) from the original request. In order to keep this functionality, the signature would have to be:

checkRedirect :: Request m -> Response -> Maybe (Request m)

Is that okay with you? I think I'd also like to call the function something different, perhaps 'getRedirectedRequest'. Is that okay? I'll also add an example to the documentation about how a caller would get the redirection chain by re-implementing redirection (by using the example in your previous email).

Sounds great.

...

As for cookie handling - I think Network.Browser has a pretty elegant solution to this. They allow a "CookieFilter" which has type of URI -> Cookie -> IO Bool. Cookies are only put in the cookie jar if the function returns True. There is a default CookieFilter, which behaves as we would expect, but the user can override this function. That way, if you don't want to support cookies, you can just pass in (\ _ _ -> return False).

Also sounds good.

...

If we're already expecting people that want specific functionality to re-implement the redirect-following code, this solution might be unnecessary. Do you think that such a concept would be beneficial for Network.HTTP.Conduit to implement?

Yes, I can imagine that some people would want more fine-grained control of which cookies are accepted.

...

Either way, I'll probably end up making a solution similar to your checkRedirect function that will just allow people to take SetCookies out of a Response and put Cookies into a Request. I'll probably also

On Thu, Jan 26, 2012 at 10:29 AM, Myles C. Maxfield wrote: provide a

...

...

...

...

default function which converts a SetCookie into a cookie by looking up the current time, inspecting the Request, etc. This will allow me to not have to change the type of Request or Response - the functions I'll be writing can deal with the raw Headers that are already in Requests and Responses. Modifying 'http' to use these functions will be straightforward.

How does this sound to you?

Sounds like a good plan to me. I'm not entirely certain how you're planning on implementing the cookie jar itself. In other words, if I make a request, have a cookie set, and then make another request later, where will the cookie be stored in the interim, and how will the second request know to use it?

Michael

Ah, yes, you're completely right. I completely agree that moving the function into the Maybe monad increases readability. This kind of function is what the Maybe monad was designed for. Here is a revised patch. On Sat, Jan 28, 2012 at 8:28 AM, Michael Snoyman wrote:

On Sat, Jan 28, 2012 at 1:20 AM, Myles C. Maxfield wrote:

...

the fromJust should never fail, beceause of the guard statement:

| 300 <= code && code < 400 && isJust l'' && isJust l' = Just $ req

Because of the order of the && operators, it will only evaluate fromJust after it makes sure that the argument isJust. That function in particular shouldn't throw any exceptions - it should only return Nothing.

Knowing that, I don't quite think I understand what your concern is. Can you elaborate?

You're right, but I had to squint really hard to prove to myself that you're right. That's the kind of code that could easily be broken in future updates by an unwitting maintainer (e.g., me). To protect the world from me, I'd prefer if the code didn't have the fromJust. This might be a good place to leverage the Monad instance of Maybe.

Michael

Here are my initial ideas about supporting cookies. Note that I'm using Chrome for ideas since it's open source. - Network/HTTP/Conduit/Cookies.hs file - Exporting the following symbols: - type StuffedCookie = SetCookie - A regular SetCookie can have Nothing for its Domain and Path attributes. A StuffedCookie has to have these fields set. - type CookieJar = [StuffedCookie] - Chrome's cookie jar is implemented as (the C++ equivalent of) Map W.Ascii StuffedCookie. The key is the "eTLD+1" of the domain, so lookups for all cookies for a given domain are fast. - I think I'll stay with just a list of StuffedCookies just to keep it simple. Perhaps a later revision can implement the faster map. - getRelevantCookies :: Request m -> CookieJar -> UTCTime -> (CookieJar, Cookies) - Gets all the cookies from the cookie jar that should be set for the given Request. - The time argument is whatever "now" is (it's pulled out of the function so the function can remain pure and easily testable) - The function will also remove expired cookies from the cookie jar (given what "now" is) and return the filtered cookie jar - putRelevantCookies :: Request m -> CookieJar -> [StuffedCookie] -> CookieJar - Insert cookies from a server response into the cookie jar. - The first argument is only used for checking to see which cookies are valid (which cookies match the requested domain, etc, so site1.com can't set a cookie for site2.com) - stuffCookie :: Request m -> SetCookie -> StuffedCookie - If the SetCookie's fields are Nothing, fill them in given the Request from which it originated - getCookies :: Response a -> ([SetCookie], Response a) - Pull cookies out of a server response. Return the response with the Set-Cookie headers filtered out - putCookies :: Request a -> Cookies -> Request a - A wrapper around renderCookies. Inserts some cookies into a request. - Doesn't overwrite cookies that are already set in the request - These functions will be exported from Network.HTTP.Conduit as well, so callers can use them to re-implement redirection chains - I won't implement a cookie filtering function (like what Network.Browser has) - If you want to have arbitrary handling of cookies, re-implement redirection following. It's not very difficult if you use the API provided, and the 'http' function is open source so you can use that as a reference. - I will implement the functions according to RFC 6265 - I will also need to write the following functions. Should they also be exported? - canonicalizeDomain :: W.Ascii -> W.Ascii - turns "..a.b.c..d.com..." to "a.b.c.d.com" - Technically necessary for domain matching (Chrome does it) - Perhaps unnecessary for a first pass? Perhaps we can trust users for now? - domainMatches :: W.Ascii -> W.Ascii -> Maybe W.Ascii - Does the first domain match against the second domain? - If so, return the prefix of the first that isn't in the second - pathMatches :: W.Ascii -> W.Ascii -> Bool - Do the paths match? - In order to implement domain matching, I have to have knowledge of the Public Suffix Listhttp://mxr.mozilla.org/mozilla-central/source/netwerk/dns/effective_tld_name... so I know that sub1.sub2.pvt.k12.wy.us can set a cookie for sub2.pvt.k12.wy.us but not for k12.wy.us (because pvt.k12.wy.us is a "suffix"). There are a variety of ways to implement this. - As far as I can tell, Chrome does it by using a script (which a human periodically runs) which parses the list at creates a .cc file that is included in the build. - I might be wrong about the execution of the script; it might be a build step. If it is a build step, however, it is suspicious that a build target would try to download a file... - Any more elegant ideas? Feedback on any/all of the above would be very helpful before I go off into the weeds on this project. Thanks, Myles C. Maxfield On Sat, Jan 28, 2012 at 8:17 PM, Michael Snoyman wrote:

Thanks, looks great! I've merged it into the Github tree.

On Sat, Jan 28, 2012 at 8:36 PM, Myles C. Maxfield wrote:

...

Ah, yes, you're completely right. I completely agree that moving the function into the Maybe monad increases readability. This kind of function is what the Maybe monad was designed for.

Here is a revised patch.

On Sat, Jan 28, 2012 at 8:28 AM, Michael Snoyman wrote:

...

On Sat, Jan 28, 2012 at 1:20 AM, Myles C. Maxfield wrote:

...

the fromJust should never fail, beceause of the guard statement:

| 300 <= code && code < 400 && isJust l'' && isJust l' = Just $

req

...

...

Because of the order of the && operators, it will only evaluate

fromJust

...

after it makes sure that the argument isJust. That function in particular shouldn't throw any exceptions - it should only return Nothing.

Knowing that, I don't quite think I understand what your concern is. Can you elaborate?

You're right, but I had to squint really hard to prove to myself that you're right. That's the kind of code that could easily be broken in future updates by an unwitting maintainer (e.g., me). To protect the world from me, I'd prefer if the code didn't have the fromJust. This might be a good place to leverage the Monad instance of Maybe.

Michael

You mean you're *not* making this proposal? On Wed, Feb 1, 2012 at 7:30 AM, Myles C. Maxfield wrote:

Well, this is embarrassing. Please disregard my previous email. I should learn to read the RFC *before* submitting proposals.

--Myles

On Tue, Jan 31, 2012 at 6:37 PM, Myles C. Maxfield wrote:

...

Here are my initial ideas about supporting cookies. Note that I'm using Chrome for ideas since it's open source.

Network/HTTP/Conduit/Cookies.hs file Exporting the following symbols:

type StuffedCookie = SetCookie

A regular SetCookie can have Nothing for its Domain and Path attributes. A StuffedCookie has to have these fields set.

type CookieJar = [StuffedCookie]

Chrome's cookie jar is implemented as (the C++ equivalent of) Map W.Ascii StuffedCookie. The key is the "eTLD+1" of the domain, so lookups for all cookies for a given domain are fast. I think I'll stay with just a list of StuffedCookies just to keep it simple. Perhaps a later revision can implement the faster map.

getRelevantCookies :: Request m -> CookieJar -> UTCTime -> (CookieJar, Cookies)

Gets all the cookies from the cookie jar that should be set for the given Request. The time argument is whatever "now" is (it's pulled out of the function so the function can remain pure and easily testable) The function will also remove expired cookies from the cookie jar (given what "now" is) and return the filtered cookie jar

putRelevantCookies :: Request m -> CookieJar -> [StuffedCookie] -> CookieJar

Insert cookies from a server response into the cookie jar. The first argument is only used for checking to see which cookies are valid (which cookies match the requested domain, etc, so site1.com can't set a cookie for site2.com)

stuffCookie :: Request m -> SetCookie -> StuffedCookie

If the SetCookie's fields are Nothing, fill them in given the Request from which it originated

getCookies :: Response a -> ([SetCookie], Response a)

Pull cookies out of a server response. Return the response with the Set-Cookie headers filtered out

putCookies :: Request a -> Cookies -> Request a

A wrapper around renderCookies. Inserts some cookies into a request. Doesn't overwrite cookies that are already set in the request

These functions will be exported from Network.HTTP.Conduit as well, so callers can use them to re-implement redirection chains I won't implement a cookie filtering function (like what Network.Browser has)

If you want to have arbitrary handling of cookies, re-implement redirection following. It's not very difficult if you use the API provided, and the 'http' function is open source so you can use that as a reference.

I will implement the functions according to RFC 6265 I will also need to write the following functions. Should they also be exported?

canonicalizeDomain :: W.Ascii -> W.Ascii

turns "..a.b.c..d.com..." to "a.b.c.d.com" Technically necessary for domain matching (Chrome does it) Perhaps unnecessary for a first pass? Perhaps we can trust users for now?

domainMatches :: W.Ascii -> W.Ascii -> Maybe W.Ascii

Does the first domain match against the second domain? If so, return the prefix of the first that isn't in the second

pathMatches :: W.Ascii -> W.Ascii -> Bool

Do the paths match?

In order to implement domain matching, I have to have knowledge of the Public Suffix List so I know that sub1.sub2.pvt.k12.wy.us can set a cookie for sub2.pvt.k12.wy.us but not for k12.wy.us (because pvt.k12.wy.us is a "suffix"). There are a variety of ways to implement this.

As far as I can tell, Chrome does it by using a script (which a human periodically runs) which parses the list at creates a .cc file that is included in the build.

I might be wrong about the execution of the script; it might be a build step. If it is a build step, however, it is suspicious that a build target would try to download a file...

Any more elegant ideas?

Feedback on any/all of the above would be very helpful before I go off into the weeds on this project.

Thanks, Myles C. Maxfield

On Sat, Jan 28, 2012 at 8:17 PM, Michael Snoyman wrote:

...

Thanks, looks great! I've merged it into the Github tree.

On Sat, Jan 28, 2012 at 8:36 PM, Myles C. Maxfield wrote:

...

Ah, yes, you're completely right. I completely agree that moving the function into the Maybe monad increases readability. This kind of function is what the Maybe monad was designed for.

Here is a revised patch.

On Sat, Jan 28, 2012 at 8:28 AM, Michael Snoyman wrote:

...

On Sat, Jan 28, 2012 at 1:20 AM, Myles C. Maxfield wrote:

...

the fromJust should never fail, beceause of the guard statement:

    | 300 <= code && code < 400 && isJust l'' && isJust l' = Just $ req

Because of the order of the && operators, it will only evaluate fromJust after it makes sure that the argument isJust. That function in particular shouldn't throw any exceptions - it should only return Nothing.

Knowing that, I don't quite think I understand what your concern is. Can you elaborate?

You're right, but I had to squint really hard to prove to myself that you're right. That's the kind of code that could easily be broken in future updates by an unwitting maintainer (e.g., me). To protect the world from me, I'd prefer if the code didn't have the fromJust. This might be a good place to leverage the Monad instance of Maybe.

Michael

Alright. After reading the spec, I have these questions / concerns: - The spec supports the "Max-Age" cookie attribute, which Web.Cookies doesn't. - I see two possible solutions to this. The first is to have parseSetCookie take a UTCTime as an argument which will represent the current time so it can populate the setCookieExpires field by adding the Max-Age attribute to the current time. Alternatively, that function can return an IO SetCookie so it can ask for the current time by itself (which I think is inferior to taking the current time as an argument). Note that the spec says to prefer Max-Age over Expires. - Add a field to SetCookie of type Maybe DiffTime which represents the Max-Age attribute - Cookie code should be aware of the Public Suffix Listhttp://mxr.mozilla.org/mozilla-central/source/netwerk/dns/effective_tld_name... as a part of its domain verification. The cookie code only needs to be able to tell if a specific string is in the list (W.Ascii -> Bool) - I propose making an entirely unrelated package, public-suffix-list, with a module Network.PublicSuffixList, which will expose this function, as well as functions about parsing the list itself. Thoughts? - Web.Cookie doesn't have a "secure-only" attribute. Adding one in is straightforward enough. - The spec describes cookies as a property of HTTP, not of the World Wide Web. Perhaps "Web.Cookie" should be renamed? Just a thought; it doesn't really matter to me. As for Network.HTTP.Conduit.Cookie, the spec describes in section 5.3 "Storage Model" what fields a Cookie has. Here is my proposal for the functions it will expose: - receiveSetCookie :: SetCookie -> Req.Request m -> UTCTime -> Bool -> CookieJar -> CookieJar - Runs the algorithm described in section 5.3 "Storage Model" - The UTCTime is the current-time, the Bool is whether or not the caller is an HTTP-based API (as opposed to JavaScript or anything else) - updateCookieJar :: Res.Response a -> Req.Request m -> UTCTime -> CookieJar -> (CookieJar, Res.Response a) - Applies "receiveSetCookie" to a Response. The output CookieJar is stripped of any Set-Cookie headers. - Specifies "True" for the Bool in receiveSetCookie - computeCookieString :: Req.Request m -> CookieJar -> UTCTime -> Bool -> (W.Ascii, CookieJar) - Runs the algorithm described in section 5.4 "The Cookie Header" - The UTCTime and Bool are the same as in receiveSetCookie - insertCookiesIntoRequest :: Req.Request m -> CookieJar -> UTCTime -> (Req.Request m, CookieJar) - Applies "computeCookieString" to a Request. The output cookie jar has updated last-accessed-times. - Specifies "True" for the Bool in computeCookieString - evictExpiredCookies :: CookieJar -> UTCTime -> CookieJar - Runs the algorithm described in the last part of section 5.3 "Storage Model" This will make the relevant part of 'http' look like: go count req'' cookie_jar'' = do now <- liftIO $ getCurrentTime let (req', cookie_jar') = insertCookiesIntoRequest req'' (evictExpiredCookies cookie_jar'' now) now res' <- httpRaw req' manager let (cookie_jar, res) = updateCookieJar res' req' now cookie_jar' case getRedirectedRequest req' (responseHeaders res) (W.statusCode (statusCode res)) of Just req -> go (count - 1) req cookie_jar Nothing -> return res I plan to not allow for a user-supplied cookieFilter function. If they want that functionality, they can re-implement the redirection-following logic. Any thoughts on any of this? Thanks, Myles On Wed, Feb 1, 2012 at 5:19 PM, Myles C. Maxfield wrote:

Nope. I'm not. The RFC is very explicit about how to handle cookies. As soon as I'm finished making sense of it (in terms of Haskell) I'll send another proposal email. On Feb 1, 2012 3:25 AM, "Michael Snoyman" wrote:

...

You mean you're *not* making this proposal?

...

Well, this is embarrassing. Please disregard my previous email. I should learn to read the RFC *before* submitting proposals.

--Myles

On Tue, Jan 31, 2012 at 6:37 PM, Myles C. Maxfield wrote:

...

Here are my initial ideas about supporting cookies. Note that I'm using Chrome for ideas since it's open source.

Network/HTTP/Conduit/Cookies.hs file Exporting the following symbols:

type StuffedCookie = SetCookie

A regular SetCookie can have Nothing for its Domain and Path

attributes. A

...

StuffedCookie has to have these fields set.

type CookieJar = [StuffedCookie]

Chrome's cookie jar is implemented as (the C++ equivalent of) Map W.Ascii StuffedCookie. The key is the "eTLD+1" of the domain, so lookups for all cookies for a given domain are fast. I think I'll stay with just a list of StuffedCookies just to keep it simple. Perhaps a later revision can implement the faster map.

getRelevantCookies :: Request m -> CookieJar -> UTCTime -> (CookieJar, Cookies)

Gets all the cookies from the cookie jar that should be set for the given Request. The time argument is whatever "now" is (it's pulled out of the function so the function can remain pure and easily testable) The function will also remove expired cookies from the cookie jar (given what "now" is) and return the filtered cookie jar

putRelevantCookies :: Request m -> CookieJar -> [StuffedCookie] -> CookieJar

Insert cookies from a server response into the cookie jar. The first argument is only used for checking to see which cookies are valid (which cookies match the requested domain, etc, so site1.comcan't set a cookie for site2.com)

stuffCookie :: Request m -> SetCookie -> StuffedCookie

If the SetCookie's fields are Nothing, fill them in given the Request from which it originated

getCookies :: Response a -> ([SetCookie], Response a)

Pull cookies out of a server response. Return the response with the Set-Cookie headers filtered out

putCookies :: Request a -> Cookies -> Request a

A wrapper around renderCookies. Inserts some cookies into a request. Doesn't overwrite cookies that are already set in the request

These functions will be exported from Network.HTTP.Conduit as well, so callers can use them to re-implement redirection chains I won't implement a cookie filtering function (like what Network.Browser has)

If you want to have arbitrary handling of cookies, re-implement redirection following. It's not very difficult if you use the API

...

...

and the 'http' function is open source so you can use that as a reference.

I will implement the functions according to RFC 6265 I will also need to write the following functions. Should they also be exported?

canonicalizeDomain :: W.Ascii -> W.Ascii

turns "..a.b.c..d.com..." to "a.b.c.d.com" Technically necessary for domain matching (Chrome does it) Perhaps unnecessary for a first pass? Perhaps we can trust users for now?

domainMatches :: W.Ascii -> W.Ascii -> Maybe W.Ascii

Does the first domain match against the second domain? If so, return the prefix of the first that isn't in the second

pathMatches :: W.Ascii -> W.Ascii -> Bool

Do the paths match?

In order to implement domain matching, I have to have knowledge of the Public Suffix List so I know that sub1.sub2.pvt.k12.wy.us can set a cookie for sub2.pvt.k12.wy.us but not for k12.wy.us (because

...

...

is a "suffix"). There are a variety of ways to implement this.

As far as I can tell, Chrome does it by using a script (which a human periodically runs) which parses the list at creates a .cc file that is included in the build.

I might be wrong about the execution of the script; it might be a build step. If it is a build step, however, it is suspicious that a build target would try to download a file...

Any more elegant ideas?

Feedback on any/all of the above would be very helpful before I go off into the weeds on this project.

Thanks, Myles C. Maxfield

On Sat, Jan 28, 2012 at 8:17 PM, Michael Snoyman wrote:

...

Thanks, looks great! I've merged it into the Github tree.

On Sat, Jan 28, 2012 at 8:36 PM, Myles C. Maxfield wrote:

...

Ah, yes, you're completely right. I completely agree that moving the function into the Maybe monad increases readability. This kind of function is what the Maybe monad was designed for.

Here is a revised patch.

On Sat, Jan 28, 2012 at 8:28 AM, Michael Snoyman <

michael@snoyman.com>

...

...

wrote: > > On Sat, Jan 28, 2012 at 1:20 AM, Myles C. Maxfield > wrote: > > the fromJust should never fail, beceause of the guard statement: > > > > | 300 <= code && code < 400 && isJust l'' && isJust l' = Just $ > > req > > > > Because of the order of the && operators, it will only evaluate > > fromJust > > after it makes sure that the argument isJust. That function in > > particular > > shouldn't throw any exceptions - it should only return Nothing. > > > > Knowing that, I don't quite think I understand what your concern is. > > Can > > you > > elaborate? > > You're right, but I had to squint really hard to prove to myself

...

...

...

...

> you're right. That's the kind of code that could easily be broken in > future updates by an unwitting maintainer (e.g., me). To protect

On Wed, Feb 1, 2012 at 7:30 AM, Myles C. Maxfield wrote: provided, pvt.k12.wy.us that the

...

...

...

...

> world from me, I'd prefer if the code didn't have the fromJust. This > might be a good place to leverage the Monad instance of Maybe. > > Michael

Looks good to me too. I agree with Aristid: let's make the change to cookie itself. Do you want to send a pull request? I'm also considering making the SetCookie constructor hidden like we have for Request, so that if in the future we realize we need to add some other settings, it doesn't break the API. Chris: I would recommend compiling it into the module. Best bet would likely being converting the source file to Haskell source. Michael On Fri, Feb 3, 2012 at 6:32 AM, Myles C. Maxfield wrote:

Alright. After reading the spec, I have these questions / concerns:

The spec supports the "Max-Age" cookie attribute, which Web.Cookies doesn't.

I see two possible solutions to this. The first is to have parseSetCookie take a UTCTime as an argument which will represent the current time so it can populate the setCookieExpires field by adding the Max-Age attribute to the current time. Alternatively, that function can return an IO SetCookie so it can ask for the current time by itself (which I think is inferior to taking the current time as an argument). Note that the spec says to prefer Max-Age over Expires. Add a field to SetCookie of type Maybe DiffTime which represents the Max-Age attribute

Cookie code should be aware of the Public Suffix List as a part of its domain verification. The cookie code only needs to be able to tell if a specific string is in the list (W.Ascii -> Bool)

I propose making an entirely unrelated package, public-suffix-list, with a module Network.PublicSuffixList, which will expose this function, as well as functions about parsing the list itself. Thoughts?

Web.Cookie doesn't have a "secure-only" attribute. Adding one in is straightforward enough. The spec describes cookies as a property of HTTP, not of the World Wide Web. Perhaps "Web.Cookie" should be renamed? Just a thought; it doesn't really matter to me.

As for Network.HTTP.Conduit.Cookie, the spec describes in section 5.3 "Storage Model" what fields a Cookie has. Here is my proposal for the functions it will expose:

receiveSetCookie :: SetCookie -> Req.Request m -> UTCTime -> Bool -> CookieJar -> CookieJar

Runs the algorithm described in section 5.3 "Storage Model" The UTCTime is the current-time, the Bool is whether or not the caller is an HTTP-based API (as opposed to JavaScript or anything else)

updateCookieJar :: Res.Response a -> Req.Request m -> UTCTime -> CookieJar -> (CookieJar, Res.Response a)

Applies "receiveSetCookie" to a Response. The output CookieJar is stripped of any Set-Cookie headers. Specifies "True" for the Bool in receiveSetCookie

computeCookieString :: Req.Request m -> CookieJar -> UTCTime -> Bool -> (W.Ascii, CookieJar)

Runs the algorithm described in section 5.4 "The Cookie Header" The UTCTime and Bool are the same as in receiveSetCookie

insertCookiesIntoRequest :: Req.Request m -> CookieJar -> UTCTime -> (Req.Request m, CookieJar)

Applies "computeCookieString" to a Request. The output cookie jar has updated last-accessed-times. Specifies "True" for the Bool in computeCookieString

evictExpiredCookies :: CookieJar -> UTCTime -> CookieJar

Runs the algorithm described in the last part of section 5.3 "Storage Model"

This will make the relevant part of 'http' look like:

    go count req'' cookie_jar'' = do         now <- liftIO $ getCurrentTime         let (req', cookie_jar') = insertCookiesIntoRequest req'' (evictExpiredCookies cookie_jar'' now) now         res' <- httpRaw req' manager         let (cookie_jar, res) = updateCookieJar res' req' now cookie_jar'         case getRedirectedRequest req' (responseHeaders res) (W.statusCode (statusCode res)) of             Just req -> go (count - 1) req cookie_jar             Nothing -> return res

I plan to not allow for a user-supplied cookieFilter function. If they want that functionality, they can re-implement the redirection-following logic.

Any thoughts on any of this?

Thanks, Myles

On Wed, Feb 1, 2012 at 5:19 PM, Myles C. Maxfield wrote:

...

Nope. I'm not. The RFC is very explicit about how to handle cookies. As soon as I'm finished making sense of it (in terms of Haskell) I'll send another proposal email.

On Feb 1, 2012 3:25 AM, "Michael Snoyman" wrote:

...

You mean you're *not* making this proposal?

On Wed, Feb 1, 2012 at 7:30 AM, Myles C. Maxfield wrote:

...

Well, this is embarrassing. Please disregard my previous email. I should learn to read the RFC *before* submitting proposals.

--Myles

On Tue, Jan 31, 2012 at 6:37 PM, Myles C. Maxfield wrote:

...

Here are my initial ideas about supporting cookies. Note that I'm using Chrome for ideas since it's open source.

Network/HTTP/Conduit/Cookies.hs file Exporting the following symbols:

type StuffedCookie = SetCookie

A regular SetCookie can have Nothing for its Domain and Path attributes. A StuffedCookie has to have these fields set.

type CookieJar = [StuffedCookie]

Chrome's cookie jar is implemented as (the C++ equivalent of) Map W.Ascii StuffedCookie. The key is the "eTLD+1" of the domain, so lookups for all cookies for a given domain are fast. I think I'll stay with just a list of StuffedCookies just to keep it simple. Perhaps a later revision can implement the faster map.

getRelevantCookies :: Request m -> CookieJar -> UTCTime -> (CookieJar, Cookies)

Gets all the cookies from the cookie jar that should be set for the given Request. The time argument is whatever "now" is (it's pulled out of the function so the function can remain pure and easily testable) The function will also remove expired cookies from the cookie jar (given what "now" is) and return the filtered cookie jar

putRelevantCookies :: Request m -> CookieJar -> [StuffedCookie] -> CookieJar

Insert cookies from a server response into the cookie jar. The first argument is only used for checking to see which cookies are valid (which cookies match the requested domain, etc, so site1.com can't set a cookie for site2.com)

stuffCookie :: Request m -> SetCookie -> StuffedCookie

If the SetCookie's fields are Nothing, fill them in given the Request from which it originated

getCookies :: Response a -> ([SetCookie], Response a)

Pull cookies out of a server response. Return the response with the Set-Cookie headers filtered out

putCookies :: Request a -> Cookies -> Request a

A wrapper around renderCookies. Inserts some cookies into a request. Doesn't overwrite cookies that are already set in the request

These functions will be exported from Network.HTTP.Conduit as well, so callers can use them to re-implement redirection chains I won't implement a cookie filtering function (like what Network.Browser has)

If you want to have arbitrary handling of cookies, re-implement redirection following. It's not very difficult if you use the API provided, and the 'http' function is open source so you can use that as a reference.

I will implement the functions according to RFC 6265 I will also need to write the following functions. Should they also be exported?

canonicalizeDomain :: W.Ascii -> W.Ascii

turns "..a.b.c..d.com..." to "a.b.c.d.com" Technically necessary for domain matching (Chrome does it) Perhaps unnecessary for a first pass? Perhaps we can trust users for now?

domainMatches :: W.Ascii -> W.Ascii -> Maybe W.Ascii

Does the first domain match against the second domain? If so, return the prefix of the first that isn't in the second

pathMatches :: W.Ascii -> W.Ascii -> Bool

Do the paths match?

In order to implement domain matching, I have to have knowledge of the Public Suffix List so I know that sub1.sub2.pvt.k12.wy.us can set a cookie for sub2.pvt.k12.wy.us but not for k12.wy.us (because pvt.k12.wy.us is a "suffix"). There are a variety of ways to implement this.

As far as I can tell, Chrome does it by using a script (which a human periodically runs) which parses the list at creates a .cc file that is included in the build.

I might be wrong about the execution of the script; it might be a build step. If it is a build step, however, it is suspicious that a build target would try to download a file...

Any more elegant ideas?

Feedback on any/all of the above would be very helpful before I go off into the weeds on this project.

Thanks, Myles C. Maxfield

On Sat, Jan 28, 2012 at 8:17 PM, Michael Snoyman wrote:

...

Thanks, looks great! I've merged it into the Github tree.

On Sat, Jan 28, 2012 at 8:36 PM, Myles C. Maxfield wrote: > Ah, yes, you're completely right. I completely agree that moving > the > function into the Maybe monad increases readability. This kind of > function > is what the Maybe monad was designed for. > > Here is a revised patch. > > > On Sat, Jan 28, 2012 at 8:28 AM, Michael Snoyman > > wrote: >> >> On Sat, Jan 28, 2012 at 1:20 AM, Myles C. Maxfield >> wrote: >> > the fromJust should never fail, beceause of the guard statement: >> > >> >     | 300 <= code && code < 400 && isJust l'' && isJust l' = >> > Just $ >> > req >> > >> > Because of the order of the && operators, it will only evaluate >> > fromJust >> > after it makes sure that the argument isJust. That function in >> > particular >> > shouldn't throw any exceptions - it should only return Nothing. >> > >> > Knowing that, I don't quite think I understand what your concern >> > is. >> > Can >> > you >> > elaborate? >> >> You're right, but I had to squint really hard to prove to myself >> that >> you're right. That's the kind of code that could easily be broken >> in >> future updates by an unwitting maintainer (e.g., me). To protect >> the >> world from me, I'd prefer if the code didn't have the fromJust. >> This >> might be a good place to leverage the Monad instance of Maybe. >> >> Michael > >

Here is the patch to Web.Cookie. I didn't modify the tests at all because they were already broken - they looked like they hadn't been updated since SetCookie only had 5 parameters. I did verify by hand that the patch works, though. Thanks, Myles On Thu, Feb 2, 2012 at 11:26 PM, Myles C. Maxfield

wrote:

Alright, I'll make a small patch that adds 2 fields to SetCookie: setCookieMaxAge :: Maybe DiffTime setCookieSecureOnly :: Bool

I've also gotten started on those cookie functions. I'm currently writing tests for them.

@Chris: The best advice I can give is that Chrome (what I'm using as a source on all this) has the data baked into a .cc file. However, they have directions in a README and a script which will parse the list and generate that source file. I recommend doing this. That way, the Haskell module would have 2 source files: one file that reads the list and generates the second file, which is a very large source file that contains each element in the list. The list should export `elem`-type queries. I'm not quite sure how to handle wildcards that appear in the list - that part is up to you. Thanks for helping out with this :]

--Myles

On Thu, Feb 2, 2012 at 10:53 PM, Michael Snoyman wrote:

...

Looks good to me too. I agree with Aristid: let's make the change to cookie itself. Do you want to send a pull request? I'm also considering making the SetCookie constructor hidden like we have for Request, so that if in the future we realize we need to add some other settings, it doesn't break the API.

Chris: I would recommend compiling it into the module. Best bet would likely being converting the source file to Haskell source.

Michael

...

Alright. After reading the spec, I have these questions / concerns:

The spec supports the "Max-Age" cookie attribute, which Web.Cookies doesn't.

I see two possible solutions to this. The first is to have

...

take a UTCTime as an argument which will represent the current time so it can populate the setCookieExpires field by adding the Max-Age attribute to the current time. Alternatively, that function can return an IO SetCookie so it can ask for the current time by itself (which I think is inferior to taking the current time as an argument). Note that the spec says to

...

Max-Age over Expires. Add a field to SetCookie of type Maybe DiffTime which represents the Max-Age attribute

Cookie code should be aware of the Public Suffix List as a part of its domain verification. The cookie code only needs to be able to tell if a specific string is in the list (W.Ascii -> Bool)

I propose making an entirely unrelated package, public-suffix-list, with a module Network.PublicSuffixList, which will expose this function, as well as functions about parsing the list itself. Thoughts?

Web.Cookie doesn't have a "secure-only" attribute. Adding one in is straightforward enough. The spec describes cookies as a property of HTTP, not of the World Wide Web. Perhaps "Web.Cookie" should be renamed? Just a thought; it doesn't really matter to me.

As for Network.HTTP.Conduit.Cookie, the spec describes in section 5.3 "Storage Model" what fields a Cookie has. Here is my proposal for the functions it will expose:

receiveSetCookie :: SetCookie -> Req.Request m -> UTCTime -> Bool -> CookieJar -> CookieJar

Runs the algorithm described in section 5.3 "Storage Model" The UTCTime is the current-time, the Bool is whether or not the caller is an HTTP-based API (as opposed to JavaScript or anything else)

updateCookieJar :: Res.Response a -> Req.Request m -> UTCTime -> CookieJar -> (CookieJar, Res.Response a)

Applies "receiveSetCookie" to a Response. The output CookieJar is stripped of any Set-Cookie headers. Specifies "True" for the Bool in receiveSetCookie

computeCookieString :: Req.Request m -> CookieJar -> UTCTime -> Bool -> (W.Ascii, CookieJar)

Runs the algorithm described in section 5.4 "The Cookie Header" The UTCTime and Bool are the same as in receiveSetCookie

insertCookiesIntoRequest :: Req.Request m -> CookieJar -> UTCTime -> (Req.Request m, CookieJar)

Applies "computeCookieString" to a Request. The output cookie jar has updated last-accessed-times. Specifies "True" for the Bool in computeCookieString

evictExpiredCookies :: CookieJar -> UTCTime -> CookieJar

Runs the algorithm described in the last part of section 5.3 "Storage Model"

This will make the relevant part of 'http' look like:

go count req'' cookie_jar'' = do now <- liftIO $ getCurrentTime let (req', cookie_jar') = insertCookiesIntoRequest req'' (evictExpiredCookies cookie_jar'' now) now res' <- httpRaw req' manager let (cookie_jar, res) = updateCookieJar res' req' now cookie_jar' case getRedirectedRequest req' (responseHeaders res) (W.statusCode (statusCode res)) of Just req -> go (count - 1) req cookie_jar Nothing -> return res

I plan to not allow for a user-supplied cookieFilter function. If they want that functionality, they can re-implement the redirection-following logic.

Any thoughts on any of this?

Thanks, Myles

On Wed, Feb 1, 2012 at 5:19 PM, Myles C. Maxfield < myles.maxfield@gmail.com> wrote:

...

Nope. I'm not. The RFC is very explicit about how to handle cookies. As soon as I'm finished making sense of it (in terms of Haskell) I'll send another proposal email.

On Feb 1, 2012 3:25 AM, "Michael Snoyman" wrote:

...

You mean you're *not* making this proposal?

On Wed, Feb 1, 2012 at 7:30 AM, Myles C. Maxfield wrote:

...

Well, this is embarrassing. Please disregard my previous email. I should learn to read the RFC *before* submitting proposals.

--Myles

On Tue, Jan 31, 2012 at 6:37 PM, Myles C. Maxfield wrote: > > Here are my initial ideas about supporting cookies. Note that I'm > using > Chrome for ideas since it's open source. > > Network/HTTP/Conduit/Cookies.hs file > Exporting the following symbols: > > type StuffedCookie = SetCookie > > A regular SetCookie can have Nothing for its Domain and Path > attributes. A > StuffedCookie has to have these fields set. > > type CookieJar = [StuffedCookie] > > Chrome's cookie jar is implemented as (the C++ equivalent of) Map > W.Ascii > StuffedCookie. The key is the "eTLD+1" of the domain, so lookups

for

...

...

...

> all > cookies for a given domain are fast. > I think I'll stay with just a list of StuffedCookies just to keep it > simple. Perhaps a later revision can implement the faster map. > > getRelevantCookies :: Request m -> CookieJar -> UTCTime -> (CookieJar, > Cookies) > > Gets all the cookies from the cookie jar that should be set for the > given > Request. > The time argument is whatever "now" is (it's pulled out of the > function so > the function can remain pure and easily testable) > The function will also remove expired cookies from the cookie jar > (given > what "now" is) and return the filtered cookie jar > > putRelevantCookies :: Request m -> CookieJar -> [StuffedCookie] -> > CookieJar > > Insert cookies from a server response into the cookie jar. > The first argument is only used for checking to see which cookies are > valid (which cookies match the requested domain, etc, so site1.com > can't set > a cookie for site2.com) > > stuffCookie :: Request m -> SetCookie -> StuffedCookie > > If the SetCookie's fields are Nothing, fill them in given the Request > from > which it originated > > getCookies :: Response a -> ([SetCookie], Response a) > > Pull cookies out of a server response. Return the response with the > Set-Cookie headers filtered out > > putCookies :: Request a -> Cookies -> Request a > > A wrapper around renderCookies. Inserts some cookies into a request. > Doesn't overwrite cookies that are already set in the request > > These functions will be exported from Network.HTTP.Conduit as well, so > callers can use them to re-implement redirection chains > I won't implement a cookie filtering function (like what > Network.Browser > has) > > If you want to have arbitrary handling of cookies, re-implement > redirection following. It's not very difficult if you use the API > provided, > and the 'http' function is open source so you can use that as a > reference. > > I will implement the functions according to RFC 6265 > I will also need to write the following functions. Should they also be > exported? > > canonicalizeDomain :: W.Ascii -> W.Ascii > > turns "..a.b.c..d.com..." to "a.b.c.d.com" > Technically necessary for domain matching (Chrome does it) > Perhaps unnecessary for a first pass? Perhaps we can trust users for > now? > > domainMatches :: W.Ascii -> W.Ascii -> Maybe W.Ascii > > Does the first domain match against the second domain? > If so, return the prefix of the first that isn't in the second > > pathMatches :: W.Ascii -> W.Ascii -> Bool > > Do the paths match? > > In order to implement domain matching, I have to have knowledge of > the Public Suffix List so I know that sub1.sub2.pvt.k12.wy.us can set > a > cookie for sub2.pvt.k12.wy.us but not for k12.wy.us (because > pvt.k12.wy.us > is a "suffix"). There are a variety of ways to implement this. > > As far as I can tell, Chrome does it by using a script (which a human > periodically runs) which parses the list at creates a .cc file

...

...

...

...

> included in the build. > > I might be wrong about the execution of the script; it might be a > build > step. If it is a build step, however, it is suspicious that a build > target > would try to download a file... > > Any more elegant ideas? > > Feedback on any/all of the above would be very helpful before I go off > into the weeds on this project. > > Thanks, > Myles C. Maxfield > > On Sat, Jan 28, 2012 at 8:17 PM, Michael Snoyman < michael@snoyman.com> > wrote: >> >> Thanks, looks great! I've merged it into the Github tree. >> >> On Sat, Jan 28, 2012 at 8:36 PM, Myles C. Maxfield >> wrote: >> > Ah, yes, you're completely right. I completely agree that moving >> > the >> > function into the Maybe monad increases readability. This kind of >> > function >> > is what the Maybe monad was designed for. >> > >> > Here is a revised patch. >> > >> > >> > On Sat, Jan 28, 2012 at 8:28 AM, Michael Snoyman >> > >> > wrote: >> >> >> >> On Sat, Jan 28, 2012 at 1:20 AM, Myles C. Maxfield >> >> wrote: >> >> > the fromJust should never fail, beceause of the guard statement: >> >> > >> >> > | 300 <= code && code < 400 && isJust l'' && isJust l' = >> >> > Just $ >> >> > req >> >> > >> >> > Because of the order of the && operators, it will only evaluate >> >> > fromJust >> >> > after it makes sure that the argument isJust. That function in >> >> > particular >> >> > shouldn't throw any exceptions - it should only return Nothing. >> >> > >> >> > Knowing that, I don't quite think I understand what your concern >> >> > is. >> >> > Can >> >> > you >> >> > elaborate? >> >> >> >> You're right, but I had to squint really hard to prove to myself >> >> that >> >> you're right. That's the kind of code that could easily be broken >> >> in >> >> future updates by an unwitting maintainer (e.g., me). To

On Fri, Feb 3, 2012 at 6:32 AM, Myles C. Maxfield wrote: parseSetCookie prefer that is protect

...

...

...

...

>> >> the >> >> world from me, I'd prefer if the code didn't have the fromJust. >> >> This >> >> might be a good place to leverage the Monad instance of Maybe. >> >> >> >> Michael >> > >> > > >

That's a pretty reasonable thing to do. Didn't you say that I should keep the 'prefer max-age to expires' logic out of Web.Cookie? What do you think, Michael? --Myles On Sat, Feb 4, 2012 at 4:03 AM, Aristid Breitkreuz wrote:

Is it possible to have both an Expires and a Max-age? If not, maybe you should make a type like

data Expiry = NeverExpires | ExpiresAt UTCTime | ExpiresIn DiffTime

...

Here is the patch to Web.Cookie. I didn't modify the tests at all because they were already broken - they looked like they hadn't been updated since SetCookie only had 5 parameters. I did verify by hand that the patch works, though.

Thanks, Myles

On Thu, Feb 2, 2012 at 11:26 PM, Myles C. Maxfield wrote:

...

Alright, I'll make a small patch that adds 2 fields to SetCookie: setCookieMaxAge :: Maybe DiffTime setCookieSecureOnly :: Bool

I've also gotten started on those cookie functions. I'm currently

writing

...

tests for them.

@Chris: The best advice I can give is that Chrome (what I'm using as a source on all this) has the data baked into a .cc file. However, they have directions in a README and a script which will parse the list and generate that source file. I recommend doing this. That way, the Haskell module would have 2 source files: one file that reads the list and generates the second file, which is a very large source file that contains each element in

...

...

list. The list should export `elem`-type queries. I'm not quite sure how to handle wildcards that appear in the list - that part is up to you. Thanks for helping out with this :]

--Myles

On Thu, Feb 2, 2012 at 10:53 PM, Michael Snoyman wrote:

...

Looks good to me too. I agree with Aristid: let's make the change to cookie itself. Do you want to send a pull request? I'm also considering making the SetCookie constructor hidden like we have for Request, so that if in the future we realize we need to add some other settings, it doesn't break the API.

Chris: I would recommend compiling it into the module. Best bet would likely being converting the source file to Haskell source.

Michael

On Fri, Feb 3, 2012 at 6:32 AM, Myles C. Maxfield wrote:

...

Alright. After reading the spec, I have these questions / concerns:

The spec supports the "Max-Age" cookie attribute, which Web.Cookies doesn't.

I see two possible solutions to this. The first is to have parseSetCookie take a UTCTime as an argument which will represent the current time

so

...

...

it can populate the setCookieExpires field by adding the Max-Age attribute to the current time. Alternatively, that function can return an IO SetCookie so it can ask for the current time by itself (which I think is inferior to taking the current time as an argument). Note that the spec says to prefer Max-Age over Expires. Add a field to SetCookie of type Maybe DiffTime which represents the Max-Age attribute

Cookie code should be aware of the Public Suffix List as a part of its domain verification. The cookie code only needs to be able to tell if a specific string is in the list (W.Ascii -> Bool)

I propose making an entirely unrelated package, public-suffix-list, with a module Network.PublicSuffixList, which will expose this function, as well as functions about parsing the list itself. Thoughts?

Web.Cookie doesn't have a "secure-only" attribute. Adding one in is straightforward enough. The spec describes cookies as a property of HTTP, not of the World Wide Web. Perhaps "Web.Cookie" should be renamed? Just a thought; it doesn't really matter to me.

As for Network.HTTP.Conduit.Cookie, the spec describes in section 5.3 "Storage Model" what fields a Cookie has. Here is my proposal for the functions it will expose:

receiveSetCookie :: SetCookie -> Req.Request m -> UTCTime -> Bool -> CookieJar -> CookieJar

Runs the algorithm described in section 5.3 "Storage Model" The UTCTime is the current-time, the Bool is whether or not the caller is an HTTP-based API (as opposed to JavaScript or anything else)

updateCookieJar :: Res.Response a -> Req.Request m -> UTCTime -> CookieJar -> (CookieJar, Res.Response a)

Applies "receiveSetCookie" to a Response. The output CookieJar is stripped of any Set-Cookie headers. Specifies "True" for the Bool in receiveSetCookie

computeCookieString :: Req.Request m -> CookieJar -> UTCTime -> Bool -> (W.Ascii, CookieJar)

Runs the algorithm described in section 5.4 "The Cookie Header" The UTCTime and Bool are the same as in receiveSetCookie

insertCookiesIntoRequest :: Req.Request m -> CookieJar -> UTCTime -> (Req.Request m, CookieJar)

Applies "computeCookieString" to a Request. The output cookie jar has updated last-accessed-times. Specifies "True" for the Bool in computeCookieString

evictExpiredCookies :: CookieJar -> UTCTime -> CookieJar

Runs the algorithm described in the last part of section 5.3 "Storage Model"

This will make the relevant part of 'http' look like:

go count req'' cookie_jar'' = do now <- liftIO $ getCurrentTime let (req', cookie_jar') = insertCookiesIntoRequest req'' (evictExpiredCookies cookie_jar'' now) now res' <- httpRaw req' manager let (cookie_jar, res) = updateCookieJar res' req' now cookie_jar' case getRedirectedRequest req' (responseHeaders res) (W.statusCode (statusCode res)) of Just req -> go (count - 1) req cookie_jar Nothing -> return res

I plan to not allow for a user-supplied cookieFilter function. If

2012/2/4 Myles C. Maxfield : the they

...

...

...

...

want that functionality, they can re-implement the redirection-following logic.

Any thoughts on any of this?

Thanks, Myles

On Wed, Feb 1, 2012 at 5:19 PM, Myles C. Maxfield wrote:

...

Nope. I'm not. The RFC is very explicit about how to handle cookies. As soon as I'm finished making sense of it (in terms of Haskell) I'll send another proposal email.

On Feb 1, 2012 3:25 AM, "Michael Snoyman"

wrote:

...

> > You mean you're *not* making this proposal? > > On Wed, Feb 1, 2012 at 7:30 AM, Myles C. Maxfield > wrote: > > Well, this is embarrassing. Please disregard my previous email. I > > should > > learn to read the RFC *before* submitting proposals. > > > > --Myles > > > > > > On Tue, Jan 31, 2012 at 6:37 PM, Myles C. Maxfield > > wrote: > >> > >> Here are my initial ideas about supporting cookies. Note that I'm > >> using > >> Chrome for ideas since it's open source. > >> > >> Network/HTTP/Conduit/Cookies.hs file > >> Exporting the following symbols: > >> > >> type StuffedCookie = SetCookie > >> > >> A regular SetCookie can have Nothing for its Domain and Path > >> attributes. A > >> StuffedCookie has to have these fields set. > >> > >> type CookieJar = [StuffedCookie] > >> > >> Chrome's cookie jar is implemented as (the C++ equivalent of) Map > >> W.Ascii > >> StuffedCookie. The key is the "eTLD+1" of the domain, so lookups > >> for > >> all > >> cookies for a given domain are fast. > >> I think I'll stay with just a list of StuffedCookies just to keep > >> it > >> simple. Perhaps a later revision can implement the faster map. > >> > >> getRelevantCookies :: Request m -> CookieJar -> UTCTime -> > >> (CookieJar, > >> Cookies) > >> > >> Gets all the cookies from the cookie jar that should be set for > >> the > >> given > >> Request. > >> The time argument is whatever "now" is (it's pulled out of the > >> function so > >> the function can remain pure and easily testable) > >> The function will also remove expired cookies from the cookie jar > >> (given > >> what "now" is) and return the filtered cookie jar > >> > >> putRelevantCookies :: Request m -> CookieJar -> [StuffedCookie] -> > >> CookieJar > >> > >> Insert cookies from a server response into the cookie jar. > >> The first argument is only used for checking to see which cookies > >> are > >> valid (which cookies match the requested domain, etc, so site1.com > >> can't set > >> a cookie for site2.com) > >> > >> stuffCookie :: Request m -> SetCookie -> StuffedCookie > >> > >> If the SetCookie's fields are Nothing, fill them in given the > >> Request > >> from > >> which it originated > >> > >> getCookies :: Response a -> ([SetCookie], Response a) > >> > >> Pull cookies out of a server response. Return the response with > >> the > >> Set-Cookie headers filtered out > >> > >> putCookies :: Request a -> Cookies -> Request a > >> > >> A wrapper around renderCookies. Inserts some cookies into a > >> request. > >> Doesn't overwrite cookies that are already set in the request > >> > >> These functions will be exported from Network.HTTP.Conduit as > >> well, so > >> callers can use them to re-implement redirection chains > >> I won't implement a cookie filtering function (like what > >> Network.Browser > >> has) > >> > >> If you want to have arbitrary handling of cookies, re-implement > >> redirection following. It's not very difficult if you use the API > >> provided, > >> and the 'http' function is open source so you can use that as a > >> reference. > >> > >> I will implement the functions according to RFC 6265 > >> I will also need to write the following functions. Should they > >> also be > >> exported? > >> > >> canonicalizeDomain :: W.Ascii -> W.Ascii > >> > >> turns "..a.b.c..d.com..." to "a.b.c.d.com" > >> Technically necessary for domain matching (Chrome does it) > >> Perhaps unnecessary for a first pass? Perhaps we can trust users > >> for > >> now? > >> > >> domainMatches :: W.Ascii -> W.Ascii -> Maybe W.Ascii > >> > >> Does the first domain match against the second domain? > >> If so, return the prefix of the first that isn't in the second > >> > >> pathMatches :: W.Ascii -> W.Ascii -> Bool > >> > >> Do the paths match? > >> > >> In order to implement domain matching, I have to have knowledge of > >> the Public Suffix List so I know that sub1.sub2.pvt.k12.wy.uscan > >> set > >> a > >> cookie for sub2.pvt.k12.wy.us but not for k12.wy.us (because > >> pvt.k12.wy.us > >> is a "suffix"). There are a variety of ways to implement this. > >> > >> As far as I can tell, Chrome does it by using a script (which a > >> human > >> periodically runs) which parses the list at creates a .cc file > >> that is > >> included in the build. > >> > >> I might be wrong about the execution of the script; it might be a > >> build > >> step. If it is a build step, however, it is suspicious that a > >> build > >> target > >> would try to download a file... > >> > >> Any more elegant ideas? > >> > >> Feedback on any/all of the above would be very helpful before I go > >> off > >> into the weeds on this project. > >> > >> Thanks, > >> Myles C. Maxfield > >> > >> On Sat, Jan 28, 2012 at 8:17 PM, Michael Snoyman > >> > >> wrote: > >>> > >>> Thanks, looks great! I've merged it into the Github tree. > >>> > >>> On Sat, Jan 28, 2012 at 8:36 PM, Myles C. Maxfield > >>> wrote: > >>> > Ah, yes, you're completely right. I completely agree that > >>> > moving > >>> > the > >>> > function into the Maybe monad increases readability. This kind > >>> > of > >>> > function > >>> > is what the Maybe monad was designed for. > >>> > > >>> > Here is a revised patch. > >>> > > >>> > > >>> > On Sat, Jan 28, 2012 at 8:28 AM, Michael Snoyman > >>> > > >>> > wrote: > >>> >> > >>> >> On Sat, Jan 28, 2012 at 1:20 AM, Myles C. Maxfield > >>> >> wrote: > >>> >> > the fromJust should never fail, beceause of the guard > >>> >> > statement: > >>> >> > > >>> >> > | 300 <= code && code < 400 && isJust l'' && isJust l' = > >>> >> > Just $ > >>> >> > req > >>> >> > > >>> >> > Because of the order of the && operators, it will only > >>> >> > evaluate > >>> >> > fromJust > >>> >> > after it makes sure that the argument isJust. That function > >>> >> > in > >>> >> > particular > >>> >> > shouldn't throw any exceptions - it should only return > >>> >> > Nothing. > >>> >> > > >>> >> > Knowing that, I don't quite think I understand what your > >>> >> > concern > >>> >> > is. > >>> >> > Can > >>> >> > you > >>> >> > elaborate? > >>> >> > >>> >> You're right, but I had to squint really hard to prove to > >>> >> myself > >>> >> that > >>> >> you're right. That's the kind of code that could easily be > >>> >> broken > >>> >> in > >>> >> future updates by an unwitting maintainer (e.g., me). To > >>> >> protect > >>> >> the > >>> >> world from me, I'd prefer if the code didn't have the > >>> >> fromJust. > >>> >> This > >>> >> might be a good place to leverage the Monad instance of Maybe. > >>> >> > >>> >> Michael > >>> > > >>> > > >> > >> > >

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

1. The spec defines a grammar for the attributes. They're in uppercase. 2. Yes - 1.3 is the first version that lists DiffTime as an instance of RealFrac (so I can use the 'floor' function to pull out the number of seconds to render it) 3. I'll see what I can do. --Myles On Sat, Feb 4, 2012 at 9:06 PM, Michael Snoyman wrote:

Looks good, a few questions/requests:

1. Is there a reason to upper-case all the attributes? 2. Is the time >= 1.3 a requirements? Because that can cause a lot of trouble for people. 3. Can you send the patch as a Github pull request? It's easier to track that way.

Michael

...

Here is the patch to Web.Cookie. I didn't modify the tests at all because they were already broken - they looked like they hadn't been updated since SetCookie only had 5 parameters. I did verify by hand that the patch works, though.

Thanks, Myles

On Thu, Feb 2, 2012 at 11:26 PM, Myles C. Maxfield wrote:

...

Alright, I'll make a small patch that adds 2 fields to SetCookie: setCookieMaxAge :: Maybe DiffTime setCookieSecureOnly :: Bool

I've also gotten started on those cookie functions. I'm currently

writing

...

tests for them.

@Chris: The best advice I can give is that Chrome (what I'm using as a source on all this) has the data baked into a .cc file. However, they have directions in a README and a script which will parse the list and generate that source file. I recommend doing this. That way, the Haskell module would have 2 source files: one file that reads the list and generates the second file, which is a very large source file that contains each element in

...

...

list. The list should export `elem`-type queries. I'm not quite sure how to handle wildcards that appear in the list - that part is up to you. Thanks for helping out with this :]

--Myles

On Thu, Feb 2, 2012 at 10:53 PM, Michael Snoyman wrote:

...

Looks good to me too. I agree with Aristid: let's make the change to cookie itself. Do you want to send a pull request? I'm also considering making the SetCookie constructor hidden like we have for Request, so that if in the future we realize we need to add some other settings, it doesn't break the API.

Chris: I would recommend compiling it into the module. Best bet would likely being converting the source file to Haskell source.

Michael

On Fri, Feb 3, 2012 at 6:32 AM, Myles C. Maxfield wrote:

...

Alright. After reading the spec, I have these questions / concerns:

The spec supports the "Max-Age" cookie attribute, which Web.Cookies doesn't.

I see two possible solutions to this. The first is to have parseSetCookie take a UTCTime as an argument which will represent the current time

so

...

...

it can populate the setCookieExpires field by adding the Max-Age attribute to the current time. Alternatively, that function can return an IO SetCookie so it can ask for the current time by itself (which I think is inferior to taking the current time as an argument). Note that the spec says to prefer Max-Age over Expires. Add a field to SetCookie of type Maybe DiffTime which represents the Max-Age attribute

Cookie code should be aware of the Public Suffix List as a part of its domain verification. The cookie code only needs to be able to tell if a specific string is in the list (W.Ascii -> Bool)

I propose making an entirely unrelated package, public-suffix-list, with a module Network.PublicSuffixList, which will expose this function, as well as functions about parsing the list itself. Thoughts?

Web.Cookie doesn't have a "secure-only" attribute. Adding one in is straightforward enough. The spec describes cookies as a property of HTTP, not of the World Wide Web. Perhaps "Web.Cookie" should be renamed? Just a thought; it doesn't really matter to me.

As for Network.HTTP.Conduit.Cookie, the spec describes in section 5.3 "Storage Model" what fields a Cookie has. Here is my proposal for the functions it will expose:

receiveSetCookie :: SetCookie -> Req.Request m -> UTCTime -> Bool -> CookieJar -> CookieJar

Runs the algorithm described in section 5.3 "Storage Model" The UTCTime is the current-time, the Bool is whether or not the caller is an HTTP-based API (as opposed to JavaScript or anything else)

updateCookieJar :: Res.Response a -> Req.Request m -> UTCTime -> CookieJar -> (CookieJar, Res.Response a)

Applies "receiveSetCookie" to a Response. The output CookieJar is stripped of any Set-Cookie headers. Specifies "True" for the Bool in receiveSetCookie

computeCookieString :: Req.Request m -> CookieJar -> UTCTime -> Bool -> (W.Ascii, CookieJar)

Runs the algorithm described in section 5.4 "The Cookie Header" The UTCTime and Bool are the same as in receiveSetCookie

insertCookiesIntoRequest :: Req.Request m -> CookieJar -> UTCTime -> (Req.Request m, CookieJar)

Applies "computeCookieString" to a Request. The output cookie jar has updated last-accessed-times. Specifies "True" for the Bool in computeCookieString

evictExpiredCookies :: CookieJar -> UTCTime -> CookieJar

Runs the algorithm described in the last part of section 5.3 "Storage Model"

This will make the relevant part of 'http' look like:

go count req'' cookie_jar'' = do now <- liftIO $ getCurrentTime let (req', cookie_jar') = insertCookiesIntoRequest req'' (evictExpiredCookies cookie_jar'' now) now res' <- httpRaw req' manager let (cookie_jar, res) = updateCookieJar res' req' now cookie_jar' case getRedirectedRequest req' (responseHeaders res) (W.statusCode (statusCode res)) of Just req -> go (count - 1) req cookie_jar Nothing -> return res

I plan to not allow for a user-supplied cookieFilter function. If

On Sat, Feb 4, 2012 at 1:21 AM, Myles C. Maxfield wrote: the they

...

...

...

...

want that functionality, they can re-implement the redirection-following logic.

Any thoughts on any of this?

Thanks, Myles

On Wed, Feb 1, 2012 at 5:19 PM, Myles C. Maxfield wrote:

...

Nope. I'm not. The RFC is very explicit about how to handle cookies. As soon as I'm finished making sense of it (in terms of Haskell) I'll send another proposal email.

On Feb 1, 2012 3:25 AM, "Michael Snoyman"

wrote:

...

> > You mean you're *not* making this proposal? > > On Wed, Feb 1, 2012 at 7:30 AM, Myles C. Maxfield > wrote: > > Well, this is embarrassing. Please disregard my previous email. I > > should > > learn to read the RFC *before* submitting proposals. > > > > --Myles > > > > > > On Tue, Jan 31, 2012 at 6:37 PM, Myles C. Maxfield > > wrote: > >> > >> Here are my initial ideas about supporting cookies. Note that I'm > >> using > >> Chrome for ideas since it's open source. > >> > >> Network/HTTP/Conduit/Cookies.hs file > >> Exporting the following symbols: > >> > >> type StuffedCookie = SetCookie > >> > >> A regular SetCookie can have Nothing for its Domain and Path > >> attributes. A > >> StuffedCookie has to have these fields set. > >> > >> type CookieJar = [StuffedCookie] > >> > >> Chrome's cookie jar is implemented as (the C++ equivalent of) Map > >> W.Ascii > >> StuffedCookie. The key is the "eTLD+1" of the domain, so lookups > >> for > >> all > >> cookies for a given domain are fast. > >> I think I'll stay with just a list of StuffedCookies just to keep > >> it > >> simple. Perhaps a later revision can implement the faster map. > >> > >> getRelevantCookies :: Request m -> CookieJar -> UTCTime -> > >> (CookieJar, > >> Cookies) > >> > >> Gets all the cookies from the cookie jar that should be set for > >> the > >> given > >> Request. > >> The time argument is whatever "now" is (it's pulled out of the > >> function so > >> the function can remain pure and easily testable) > >> The function will also remove expired cookies from the cookie jar > >> (given > >> what "now" is) and return the filtered cookie jar > >> > >> putRelevantCookies :: Request m -> CookieJar -> [StuffedCookie] -> > >> CookieJar > >> > >> Insert cookies from a server response into the cookie jar. > >> The first argument is only used for checking to see which cookies > >> are > >> valid (which cookies match the requested domain, etc, so site1.com > >> can't set > >> a cookie for site2.com) > >> > >> stuffCookie :: Request m -> SetCookie -> StuffedCookie > >> > >> If the SetCookie's fields are Nothing, fill them in given the > >> Request > >> from > >> which it originated > >> > >> getCookies :: Response a -> ([SetCookie], Response a) > >> > >> Pull cookies out of a server response. Return the response with > >> the > >> Set-Cookie headers filtered out > >> > >> putCookies :: Request a -> Cookies -> Request a > >> > >> A wrapper around renderCookies. Inserts some cookies into a > >> request. > >> Doesn't overwrite cookies that are already set in the request > >> > >> These functions will be exported from Network.HTTP.Conduit as > >> well, so > >> callers can use them to re-implement redirection chains > >> I won't implement a cookie filtering function (like what > >> Network.Browser > >> has) > >> > >> If you want to have arbitrary handling of cookies, re-implement > >> redirection following. It's not very difficult if you use the API > >> provided, > >> and the 'http' function is open source so you can use that as a > >> reference. > >> > >> I will implement the functions according to RFC 6265 > >> I will also need to write the following functions. Should they > >> also be > >> exported? > >> > >> canonicalizeDomain :: W.Ascii -> W.Ascii > >> > >> turns "..a.b.c..d.com..." to "a.b.c.d.com" > >> Technically necessary for domain matching (Chrome does it) > >> Perhaps unnecessary for a first pass? Perhaps we can trust users > >> for > >> now? > >> > >> domainMatches :: W.Ascii -> W.Ascii -> Maybe W.Ascii > >> > >> Does the first domain match against the second domain? > >> If so, return the prefix of the first that isn't in the second > >> > >> pathMatches :: W.Ascii -> W.Ascii -> Bool > >> > >> Do the paths match? > >> > >> In order to implement domain matching, I have to have knowledge of > >> the Public Suffix List so I know that sub1.sub2.pvt.k12.wy.uscan > >> set > >> a > >> cookie for sub2.pvt.k12.wy.us but not for k12.wy.us (because > >> pvt.k12.wy.us > >> is a "suffix"). There are a variety of ways to implement this. > >> > >> As far as I can tell, Chrome does it by using a script (which a > >> human > >> periodically runs) which parses the list at creates a .cc file > >> that is > >> included in the build. > >> > >> I might be wrong about the execution of the script; it might be a > >> build > >> step. If it is a build step, however, it is suspicious that a > >> build > >> target > >> would try to download a file... > >> > >> Any more elegant ideas? > >> > >> Feedback on any/all of the above would be very helpful before I go > >> off > >> into the weeds on this project. > >> > >> Thanks, > >> Myles C. Maxfield > >> > >> On Sat, Jan 28, 2012 at 8:17 PM, Michael Snoyman > >> > >> wrote: > >>> > >>> Thanks, looks great! I've merged it into the Github tree. > >>> > >>> On Sat, Jan 28, 2012 at 8:36 PM, Myles C. Maxfield > >>> wrote: > >>> > Ah, yes, you're completely right. I completely agree that > >>> > moving > >>> > the > >>> > function into the Maybe monad increases readability. This kind > >>> > of > >>> > function > >>> > is what the Maybe monad was designed for. > >>> > > >>> > Here is a revised patch. > >>> > > >>> > > >>> > On Sat, Jan 28, 2012 at 8:28 AM, Michael Snoyman > >>> > > >>> > wrote: > >>> >> > >>> >> On Sat, Jan 28, 2012 at 1:20 AM, Myles C. Maxfield > >>> >> wrote: > >>> >> > the fromJust should never fail, beceause of the guard > >>> >> > statement: > >>> >> > > >>> >> > | 300 <= code && code < 400 && isJust l'' && isJust l' = > >>> >> > Just $ > >>> >> > req > >>> >> > > >>> >> > Because of the order of the && operators, it will only > >>> >> > evaluate > >>> >> > fromJust > >>> >> > after it makes sure that the argument isJust. That function > >>> >> > in > >>> >> > particular > >>> >> > shouldn't throw any exceptions - it should only return > >>> >> > Nothing. > >>> >> > > >>> >> > Knowing that, I don't quite think I understand what your > >>> >> > concern > >>> >> > is. > >>> >> > Can > >>> >> > you > >>> >> > elaborate? > >>> >> > >>> >> You're right, but I had to squint really hard to prove to > >>> >> myself > >>> >> that > >>> >> you're right. That's the kind of code that could easily be > >>> >> broken > >>> >> in > >>> >> future updates by an unwitting maintainer (e.g., me). To > >>> >> protect > >>> >> the > >>> >> world from me, I'd prefer if the code didn't have the > >>> >> fromJust. > >>> >> This > >>> >> might be a good place to leverage the Monad instance of Maybe. > >>> >> > >>> >> Michael > >>> > > >>> > > >> > >> > >

After all these commits have been flying around, I have yet another question: the 'HTTP' package defines Network.Browser which is a State monad which keeps state about a "browser" (i.e. a cookie jar, a proxy, redirection parameters, etc.) It would be pretty straightforward to implement this kind of functionality on top of http-conduit. I was originally going to do it and release it as its own package, but it may be beneficial to add such a module to the existing http-conduit package. Should I add it in to the existing package, or release it as its own package? --Myles On Mon, Feb 6, 2012 at 12:15 AM, Michael Snoyman wrote:

Just an FYI for everyone: Myles sent an (incredibly thorough) pull request to handle cookies:

https://github.com/snoyberg/http-conduit/pull/13

Thanks!

...

1. The spec defines a grammar for the attributes. They're in uppercase. 2. Yes - 1.3 is the first version that lists DiffTime as an instance of RealFrac (so I can use the 'floor' function to pull out the number of seconds to render it) 3. I'll see what I can do.

--Myles

On Sat, Feb 4, 2012 at 9:06 PM, Michael Snoyman wrote:

...

Looks good, a few questions/requests:

1. Is there a reason to upper-case all the attributes? 2. Is the time >= 1.3 a requirements? Because that can cause a lot of trouble for people. 3. Can you send the patch as a Github pull request? It's easier to track that way.

Michael

On Sat, Feb 4, 2012 at 1:21 AM, Myles C. Maxfield wrote:

...

Here is the patch to Web.Cookie. I didn't modify the tests at all because they were already broken - they looked like they hadn't been updated since SetCookie only had 5 parameters. I did verify by hand that the patch works, though.

Thanks, Myles

On Thu, Feb 2, 2012 at 11:26 PM, Myles C. Maxfield wrote:

...

Alright, I'll make a small patch that adds 2 fields to SetCookie: setCookieMaxAge :: Maybe DiffTime setCookieSecureOnly :: Bool

I've also gotten started on those cookie functions. I'm currently writing tests for them.

@Chris: The best advice I can give is that Chrome (what I'm using as

a

...

...

...

source on all this) has the data baked into a .cc file. However, they have directions in a README and a script which will parse the list and generate that source file. I recommend doing this. That way, the Haskell module would have 2 source files: one file that reads the list and generates the second file, which is a very large source file that contains each element in the list. The list should export `elem`-type queries. I'm not quite sure how to handle wildcards that appear in the list - that part is up to you. Thanks for helping out with this :]

--Myles

On Thu, Feb 2, 2012 at 10:53 PM, Michael Snoyman < michael@snoyman.com> wrote:

...

Looks good to me too. I agree with Aristid: let's make the change to cookie itself. Do you want to send a pull request? I'm also considering making the SetCookie constructor hidden like we have for Request, so that if in the future we realize we need to add some

other

...

settings, it doesn't break the API.

Chris: I would recommend compiling it into the module. Best bet would likely being converting the source file to Haskell source.

Michael

On Fri, Feb 3, 2012 at 6:32 AM, Myles C. Maxfield wrote: > Alright. After reading the spec, I have these questions / concerns: > > The spec supports the "Max-Age" cookie attribute, which Web.Cookies > doesn't. > > I see two possible solutions to this. The first is to have > parseSetCookie > take a UTCTime as an argument which will represent the current time > so > it > can populate the setCookieExpires field by adding the Max-Age > attribute > to > the current time. Alternatively, that function can return an IO > SetCookie so > it can ask for the current time by itself (which I think is inferior > to > taking the current time as an argument). Note that the spec says to > prefer > Max-Age over Expires. > Add a field to SetCookie of type Maybe DiffTime which represents

...

...

...

...

...

> Max-Age > attribute > > Cookie code should be aware of the Public Suffix List as a part of > its > domain verification. The cookie code only needs to be able to tell > if a > specific string is in the list (W.Ascii -> Bool) > > I propose making an entirely unrelated package,

...

...

...

...

...

> with a > module Network.PublicSuffixList, which will expose this function, as > well as > functions about parsing the list itself. Thoughts? > > Web.Cookie doesn't have a "secure-only" attribute. Adding one in is > straightforward enough. > The spec describes cookies as a property of HTTP, not of the World > Wide > Web. > Perhaps "Web.Cookie" should be renamed? Just a thought; it doesn't > really > matter to me. > > As for Network.HTTP.Conduit.Cookie, the spec describes in section > 5.3 > "Storage Model" what fields a Cookie has. Here is my proposal for > the > functions it will expose: > > receiveSetCookie :: SetCookie -> Req.Request m -> UTCTime -> Bool -> > CookieJar -> CookieJar > > Runs the algorithm described in section 5.3 "Storage Model" > The UTCTime is the current-time, the Bool is whether or not the > caller > is an > HTTP-based API (as opposed to JavaScript or anything else) > > updateCookieJar :: Res.Response a -> Req.Request m -> UTCTime -> > CookieJar > -> (CookieJar, Res.Response a) > > Applies "receiveSetCookie" to a Response. The output CookieJar is > stripped > of any Set-Cookie headers. > Specifies "True" for the Bool in receiveSetCookie > > computeCookieString :: Req.Request m -> CookieJar -> UTCTime -> Bool > -> > (W.Ascii, CookieJar) > > Runs the algorithm described in section 5.4 "The Cookie Header" > The UTCTime and Bool are the same as in receiveSetCookie > > insertCookiesIntoRequest :: Req.Request m -> CookieJar -> UTCTime -> > (Req.Request m, CookieJar) > > Applies "computeCookieString" to a Request. The output cookie jar > has > updated last-accessed-times. > Specifies "True" for the Bool in computeCookieString > > evictExpiredCookies :: CookieJar -> UTCTime -> CookieJar > > Runs the algorithm described in the last part of section 5.3 > "Storage > Model" > > This will make the relevant part of 'http' look like: > > go count req'' cookie_jar'' = do > now <- liftIO $ getCurrentTime > let (req', cookie_jar') = insertCookiesIntoRequest req'' > (evictExpiredCookies cookie_jar'' now) now > res' <- httpRaw req' manager > let (cookie_jar, res) = updateCookieJar res' req' now > cookie_jar' > case getRedirectedRequest req' (responseHeaders res) > (W.statusCode > (statusCode res)) of > Just req -> go (count - 1) req cookie_jar > Nothing -> return res > > I plan to not allow for a user-supplied cookieFilter function. If > they > want > that functionality, they can re-implement the redirection-following > logic. > > Any thoughts on any of this? > > Thanks, > Myles > > On Wed, Feb 1, 2012 at 5:19 PM, Myles C. Maxfield > > wrote: >> >> Nope. I'm not. The RFC is very explicit about how to handle >> cookies. >> As >> soon as I'm finished making sense of it (in terms of Haskell) I'll >> send >> another proposal email. >> >> On Feb 1, 2012 3:25 AM, "Michael Snoyman" >> wrote: >>> >>> You mean you're *not* making this proposal? >>> >>> On Wed, Feb 1, 2012 at 7:30 AM, Myles C. Maxfield >>> wrote: >>> > Well, this is embarrassing. Please disregard my previous email. >>> > I >>> > should >>> > learn to read the RFC *before* submitting proposals. >>> > >>> > --Myles >>> > >>> > >>> > On Tue, Jan 31, 2012 at 6:37 PM, Myles C. Maxfield >>> > wrote: >>> >> >>> >> Here are my initial ideas about supporting cookies. Note that >>> >> I'm >>> >> using >>> >> Chrome for ideas since it's open source. >>> >> >>> >> Network/HTTP/Conduit/Cookies.hs file >>> >> Exporting the following symbols: >>> >> >>> >> type StuffedCookie = SetCookie >>> >> >>> >> A regular SetCookie can have Nothing for its Domain and Path >>> >> attributes. A >>> >> StuffedCookie has to have these fields set. >>> >> >>> >> type CookieJar = [StuffedCookie] >>> >> >>> >> Chrome's cookie jar is implemented as (the C++ equivalent of) >>> >> Map >>> >> W.Ascii >>> >> StuffedCookie. The key is the "eTLD+1" of the domain, so >>> >> lookups >>> >> for >>> >> all >>> >> cookies for a given domain are fast. >>> >> I think I'll stay with just a list of StuffedCookies just to >>> >> keep >>> >> it >>> >> simple. Perhaps a later revision can implement the faster map. >>> >> >>> >> getRelevantCookies :: Request m -> CookieJar -> UTCTime -> >>> >> (CookieJar, >>> >> Cookies) >>> >> >>> >> Gets all the cookies from the cookie jar that should be set for >>> >> the >>> >> given >>> >> Request. >>> >> The time argument is whatever "now" is (it's pulled out of

...

...

...

...

...

>>> >> function so >>> >> the function can remain pure and easily testable) >>> >> The function will also remove expired cookies from the cookie >>> >> jar >>> >> (given >>> >> what "now" is) and return the filtered cookie jar >>> >> >>> >> putRelevantCookies :: Request m -> CookieJar -> [StuffedCookie] >>> >> -> >>> >> CookieJar >>> >> >>> >> Insert cookies from a server response into the cookie jar. >>> >> The first argument is only used for checking to see which >>> >> cookies >>> >> are >>> >> valid (which cookies match the requested domain, etc, so >>> >> site1.com >>> >> can't set >>> >> a cookie for site2.com) >>> >> >>> >> stuffCookie :: Request m -> SetCookie -> StuffedCookie >>> >> >>> >> If the SetCookie's fields are Nothing, fill them in given the >>> >> Request >>> >> from >>> >> which it originated >>> >> >>> >> getCookies :: Response a -> ([SetCookie], Response a) >>> >> >>> >> Pull cookies out of a server response. Return the response with >>> >> the >>> >> Set-Cookie headers filtered out >>> >> >>> >> putCookies :: Request a -> Cookies -> Request a >>> >> >>> >> A wrapper around renderCookies. Inserts some cookies into a >>> >> request. >>> >> Doesn't overwrite cookies that are already set in the request >>> >> >>> >> These functions will be exported from Network.HTTP.Conduit as >>> >> well, so >>> >> callers can use them to re-implement redirection chains >>> >> I won't implement a cookie filtering function (like what >>> >> Network.Browser >>> >> has) >>> >> >>> >> If you want to have arbitrary handling of cookies, re-implement >>> >> redirection following. It's not very difficult if you use the >>> >> API >>> >> provided, >>> >> and the 'http' function is open source so you can use that as a >>> >> reference. >>> >> >>> >> I will implement the functions according to RFC 6265 >>> >> I will also need to write the following functions. Should

...

...

...

...

...

>>> >> also be >>> >> exported? >>> >> >>> >> canonicalizeDomain :: W.Ascii -> W.Ascii >>> >> >>> >> turns "..a.b.c..d.com..." to "a.b.c.d.com" >>> >> Technically necessary for domain matching (Chrome does it) >>> >> Perhaps unnecessary for a first pass? Perhaps we can trust >>> >> users >>> >> for >>> >> now? >>> >> >>> >> domainMatches :: W.Ascii -> W.Ascii -> Maybe W.Ascii >>> >> >>> >> Does the first domain match against the second domain? >>> >> If so, return the prefix of the first that isn't in the second >>> >> >>> >> pathMatches :: W.Ascii -> W.Ascii -> Bool >>> >> >>> >> Do the paths match? >>> >> >>> >> In order to implement domain matching, I have to have knowledge >>> >> of >>> >> the Public Suffix List so I know that sub1.sub2.pvt.k12.wy.us >>> >> can >>> >> set >>> >> a >>> >> cookie for sub2.pvt.k12.wy.us but not for k12.wy.us (because >>> >> pvt.k12.wy.us >>> >> is a "suffix"). There are a variety of ways to implement

On Sun, Feb 5, 2012 at 8:20 AM, Myles C. Maxfield wrote: the public-suffix-list, the they this.

...

...

...

...

...

>>> >> >>> >> As far as I can tell, Chrome does it by using a script (which a >>> >> human >>> >> periodically runs) which parses the list at creates a .cc file >>> >> that is >>> >> included in the build. >>> >> >>> >> I might be wrong about the execution of the script; it might be >>> >> a >>> >> build >>> >> step. If it is a build step, however, it is suspicious that a >>> >> build >>> >> target >>> >> would try to download a file... >>> >> >>> >> Any more elegant ideas? >>> >> >>> >> Feedback on any/all of the above would be very helpful before I >>> >> go >>> >> off >>> >> into the weeds on this project. >>> >> >>> >> Thanks, >>> >> Myles C. Maxfield >>> >> >>> >> On Sat, Jan 28, 2012 at 8:17 PM, Michael Snoyman >>> >> >>> >> wrote: >>> >>> >>> >>> Thanks, looks great! I've merged it into the Github tree. >>> >>> >>> >>> On Sat, Jan 28, 2012 at 8:36 PM, Myles C. Maxfield >>> >>> wrote: >>> >>> > Ah, yes, you're completely right. I completely agree that >>> >>> > moving >>> >>> > the >>> >>> > function into the Maybe monad increases readability. This >>> >>> > kind >>> >>> > of >>> >>> > function >>> >>> > is what the Maybe monad was designed for. >>> >>> > >>> >>> > Here is a revised patch. >>> >>> > >>> >>> > >>> >>> > On Sat, Jan 28, 2012 at 8:28 AM, Michael Snoyman >>> >>> > >>> >>> > wrote: >>> >>> >> >>> >>> >> On Sat, Jan 28, 2012 at 1:20 AM, Myles C. Maxfield >>> >>> >> wrote: >>> >>> >> > the fromJust should never fail, beceause of the guard >>> >>> >> > statement: >>> >>> >> > >>> >>> >> > | 300 <= code && code < 400 && isJust l'' && isJust >>> >>> >> > l' = >>> >>> >> > Just $ >>> >>> >> > req >>> >>> >> > >>> >>> >> > Because of the order of the && operators, it will only >>> >>> >> > evaluate >>> >>> >> > fromJust >>> >>> >> > after it makes sure that the argument isJust. That >>> >>> >> > function >>> >>> >> > in >>> >>> >> > particular >>> >>> >> > shouldn't throw any exceptions - it should only return >>> >>> >> > Nothing. >>> >>> >> > >>> >>> >> > Knowing that, I don't quite think I understand what your >>> >>> >> > concern >>> >>> >> > is. >>> >>> >> > Can >>> >>> >> > you >>> >>> >> > elaborate? >>> >>> >> >>> >>> >> You're right, but I had to squint really hard to prove to >>> >>> >> myself >>> >>> >> that >>> >>> >> you're right. That's the kind of code that could easily be >>> >>> >> broken >>> >>> >> in >>> >>> >> future updates by an unwitting maintainer (e.g., me). To >>> >>> >> protect >>> >>> >> the >>> >>> >> world from me, I'd prefer if the code didn't have the >>> >>> >> fromJust. >>> >>> >> This >>> >>> >> might be a good place to leverage the Monad instance of >>> >>> >> Maybe. >>> >>> >> >>> >>> >> Michael >>> >>> > >>> >>> > >>> >> >>> >> >>> > > >

+1 On Mon, Feb 6, 2012 at 11:16 PM, Aristid Breitkreuz wrote:

I would say: if it adds no package dependencies, put it right in.

Aristid

Am 06.02.2012 22:09 schrieb "Myles C. Maxfield" :

...

After all these commits have been flying around, I have yet another question:

the 'HTTP' package defines Network.Browser which is a State monad which keeps state about a "browser" (i.e. a cookie jar, a proxy, redirection parameters, etc.) It would be pretty straightforward to implement this kind of functionality on top of http-conduit.

I was originally going to do it and release it as its own package, but it may be beneficial to add such a module to the existing http-conduit package. Should I add it in to the existing package, or release it as its own package?

--Myles

On Mon, Feb 6, 2012 at 12:15 AM, Michael Snoyman wrote:

...

Just an FYI for everyone: Myles sent an (incredibly thorough) pull request to handle cookies:

https://github.com/snoyberg/http-conduit/pull/13

Thanks!

On Sun, Feb 5, 2012 at 8:20 AM, Myles C. Maxfield wrote:

...

1. The spec defines a grammar for the attributes. They're in uppercase. 2. Yes - 1.3 is the first version that lists DiffTime as an instance of RealFrac (so I can use the 'floor' function to pull out the number of seconds to render it) 3. I'll see what I can do.

--Myles

On Sat, Feb 4, 2012 at 9:06 PM, Michael Snoyman wrote:

...

Looks good, a few questions/requests:

1. Is there a reason to upper-case all the attributes? 2. Is the time >= 1.3 a requirements? Because that can cause a lot of trouble for people. 3. Can you send the patch as a Github pull request? It's easier to track that way.

Michael

On Sat, Feb 4, 2012 at 1:21 AM, Myles C. Maxfield wrote:

...

Here is the patch to Web.Cookie. I didn't modify the tests at all because they were already broken - they looked like they hadn't been updated since SetCookie only had 5 parameters. I did verify by hand that the patch works, though.

Thanks, Myles

On Thu, Feb 2, 2012 at 11:26 PM, Myles C. Maxfield wrote: > > Alright, I'll make a small patch that adds 2 fields to SetCookie: > setCookieMaxAge :: Maybe DiffTime > setCookieSecureOnly :: Bool > > I've also gotten started on those cookie functions. I'm currently > writing > tests for them. > > @Chris: The best advice I can give is that Chrome (what I'm using > as a > source on all this) has the data baked into a .cc file. However, > they > have > directions in a README and a script which will parse the list and > generate > that source file. I recommend doing this. That way, the Haskell > module > would > have 2 source files: one file that reads the list and generates the > second > file, which is a very large source file that contains each element > in > the > list. The list should export `elem`-type queries. I'm not quite > sure > how to > handle wildcards that appear in the list - that part is up to you. > Thanks > for helping out with this :] > > --Myles > > > On Thu, Feb 2, 2012 at 10:53 PM, Michael Snoyman > > wrote: >> >> Looks good to me too. I agree with Aristid: let's make the change >> to >> cookie itself. Do you want to send a pull request? I'm also >> considering making the SetCookie constructor hidden like we have >> for >> Request, so that if in the future we realize we need to add some >> other >> settings, it doesn't break the API. >> >> Chris: I would recommend compiling it into the module. Best bet >> would >> likely being converting the source file to Haskell source. >> >> Michael >> >> On Fri, Feb 3, 2012 at 6:32 AM, Myles C. Maxfield >> wrote: >> > Alright. After reading the spec, I have these questions / >> > concerns: >> > >> > The spec supports the "Max-Age" cookie attribute, which >> > Web.Cookies >> > doesn't. >> > >> > I see two possible solutions to this. The first is to have >> > parseSetCookie >> > take a UTCTime as an argument which will represent the current >> > time >> > so >> > it >> > can populate the setCookieExpires field by adding the Max-Age >> > attribute >> > to >> > the current time. Alternatively, that function can return an IO >> > SetCookie so >> > it can ask for the current time by itself (which I think is >> > inferior >> > to >> > taking the current time as an argument). Note that the spec says >> > to >> > prefer >> > Max-Age over Expires. >> > Add a field to SetCookie of type Maybe DiffTime which represents >> > the >> > Max-Age >> > attribute >> > >> > Cookie code should be aware of the Public Suffix List as a part >> > of >> > its >> > domain verification. The cookie code only needs to be able to >> > tell >> > if a >> > specific string is in the list (W.Ascii -> Bool) >> > >> > I propose making an entirely unrelated package, >> > public-suffix-list, >> > with a >> > module Network.PublicSuffixList, which will expose this >> > function, as >> > well as >> > functions about parsing the list itself. Thoughts? >> > >> > Web.Cookie doesn't have a "secure-only" attribute. Adding one in >> > is >> > straightforward enough. >> > The spec describes cookies as a property of HTTP, not of the >> > World >> > Wide >> > Web. >> > Perhaps "Web.Cookie" should be renamed? Just a thought; it >> > doesn't >> > really >> > matter to me. >> > >> > As for Network.HTTP.Conduit.Cookie, the spec describes in >> > section >> > 5.3 >> > "Storage Model" what fields a Cookie has. Here is my proposal >> > for >> > the >> > functions it will expose: >> > >> > receiveSetCookie :: SetCookie -> Req.Request m -> UTCTime -> >> > Bool -> >> > CookieJar -> CookieJar >> > >> > Runs the algorithm described in section 5.3 "Storage Model" >> > The UTCTime is the current-time, the Bool is whether or not the >> > caller >> > is an >> > HTTP-based API (as opposed to JavaScript or anything else) >> > >> > updateCookieJar :: Res.Response a -> Req.Request m -> UTCTime -> >> > CookieJar >> > -> (CookieJar, Res.Response a) >> > >> > Applies "receiveSetCookie" to a Response. The output CookieJar >> > is >> > stripped >> > of any Set-Cookie headers. >> > Specifies "True" for the Bool in receiveSetCookie >> > >> > computeCookieString :: Req.Request m -> CookieJar -> UTCTime -> >> > Bool >> > -> >> > (W.Ascii, CookieJar) >> > >> > Runs the algorithm described in section 5.4 "The Cookie Header" >> > The UTCTime and Bool are the same as in receiveSetCookie >> > >> > insertCookiesIntoRequest :: Req.Request m -> CookieJar -> >> > UTCTime -> >> > (Req.Request m, CookieJar) >> > >> > Applies "computeCookieString" to a Request. The output cookie >> > jar >> > has >> > updated last-accessed-times. >> > Specifies "True" for the Bool in computeCookieString >> > >> > evictExpiredCookies :: CookieJar -> UTCTime -> CookieJar >> > >> > Runs the algorithm described in the last part of section 5.3 >> > "Storage >> > Model" >> > >> > This will make the relevant part of 'http' look like: >> > >> >     go count req'' cookie_jar'' = do >> >         now <- liftIO $ getCurrentTime >> >         let (req', cookie_jar') = insertCookiesIntoRequest req'' >> > (evictExpiredCookies cookie_jar'' now) now >> >         res' <- httpRaw req' manager >> >         let (cookie_jar, res) = updateCookieJar res' req' now >> > cookie_jar' >> >         case getRedirectedRequest req' (responseHeaders res) >> > (W.statusCode >> > (statusCode res)) of >> >             Just req -> go (count - 1) req cookie_jar >> >             Nothing -> return res >> > >> > I plan to not allow for a user-supplied cookieFilter function. >> > If >> > they >> > want >> > that functionality, they can re-implement the >> > redirection-following >> > logic. >> > >> > Any thoughts on any of this? >> > >> > Thanks, >> > Myles >> > >> > On Wed, Feb 1, 2012 at 5:19 PM, Myles C. Maxfield >> > >> > wrote: >> >> >> >> Nope. I'm not. The RFC is very explicit about how to handle >> >> cookies. >> >> As >> >> soon as I'm finished making sense of it (in terms of Haskell) >> >> I'll >> >> send >> >> another proposal email. >> >> >> >> On Feb 1, 2012 3:25 AM, "Michael Snoyman" >> >> wrote: >> >>> >> >>> You mean you're *not* making this proposal? >> >>> >> >>> On Wed, Feb 1, 2012 at 7:30 AM, Myles C. Maxfield >> >>> wrote: >> >>> > Well, this is embarrassing. Please disregard my previous >> >>> > email. >> >>> > I >> >>> > should >> >>> > learn to read the RFC *before* submitting proposals. >> >>> > >> >>> > --Myles >> >>> > >> >>> > >> >>> > On Tue, Jan 31, 2012 at 6:37 PM, Myles C. Maxfield >> >>> > wrote: >> >>> >> >> >>> >> Here are my initial ideas about supporting cookies. Note >> >>> >> that >> >>> >> I'm >> >>> >> using >> >>> >> Chrome for ideas since it's open source. >> >>> >> >> >>> >> Network/HTTP/Conduit/Cookies.hs file >> >>> >> Exporting the following symbols: >> >>> >> >> >>> >> type StuffedCookie = SetCookie >> >>> >> >> >>> >> A regular SetCookie can have Nothing for its Domain and >> >>> >> Path >> >>> >> attributes. A >> >>> >> StuffedCookie has to have these fields set. >> >>> >> >> >>> >> type CookieJar = [StuffedCookie] >> >>> >> >> >>> >> Chrome's cookie jar is implemented as (the C++ equivalent >> >>> >> of) >> >>> >> Map >> >>> >> W.Ascii >> >>> >> StuffedCookie. The key is the "eTLD+1" of the domain, so >> >>> >> lookups >> >>> >> for >> >>> >> all >> >>> >> cookies for a given domain are fast. >> >>> >> I think I'll stay with just a list of StuffedCookies just >> >>> >> to >> >>> >> keep >> >>> >> it >> >>> >> simple. Perhaps a later revision can implement the faster >> >>> >> map. >> >>> >> >> >>> >> getRelevantCookies :: Request m -> CookieJar -> UTCTime -> >> >>> >> (CookieJar, >> >>> >> Cookies) >> >>> >> >> >>> >> Gets all the cookies from the cookie jar that should be set >> >>> >> for >> >>> >> the >> >>> >> given >> >>> >> Request. >> >>> >> The time argument is whatever "now" is (it's pulled out of >> >>> >> the >> >>> >> function so >> >>> >> the function can remain pure and easily testable) >> >>> >> The function will also remove expired cookies from the >> >>> >> cookie >> >>> >> jar >> >>> >> (given >> >>> >> what "now" is) and return the filtered cookie jar >> >>> >> >> >>> >> putRelevantCookies :: Request m -> CookieJar -> >> >>> >> [StuffedCookie] >> >>> >> -> >> >>> >> CookieJar >> >>> >> >> >>> >> Insert cookies from a server response into the cookie jar. >> >>> >> The first argument is only used for checking to see which >> >>> >> cookies >> >>> >> are >> >>> >> valid (which cookies match the requested domain, etc, so >> >>> >> site1.com >> >>> >> can't set >> >>> >> a cookie for site2.com) >> >>> >> >> >>> >> stuffCookie :: Request m -> SetCookie -> StuffedCookie >> >>> >> >> >>> >> If the SetCookie's fields are Nothing, fill them in given >> >>> >> the >> >>> >> Request >> >>> >> from >> >>> >> which it originated >> >>> >> >> >>> >> getCookies :: Response a -> ([SetCookie], Response a) >> >>> >> >> >>> >> Pull cookies out of a server response. Return the response >> >>> >> with >> >>> >> the >> >>> >> Set-Cookie headers filtered out >> >>> >> >> >>> >> putCookies :: Request a -> Cookies -> Request a >> >>> >> >> >>> >> A wrapper around renderCookies. Inserts some cookies into a >> >>> >> request. >> >>> >> Doesn't overwrite cookies that are already set in the >> >>> >> request >> >>> >> >> >>> >> These functions will be exported from Network.HTTP.Conduit >> >>> >> as >> >>> >> well, so >> >>> >> callers can use them to re-implement redirection chains >> >>> >> I won't implement a cookie filtering function (like what >> >>> >> Network.Browser >> >>> >> has) >> >>> >> >> >>> >> If you want to have arbitrary handling of cookies, >> >>> >> re-implement >> >>> >> redirection following. It's not very difficult if you use >> >>> >> the >> >>> >> API >> >>> >> provided, >> >>> >> and the 'http' function is open source so you can use that >> >>> >> as a >> >>> >> reference. >> >>> >> >> >>> >> I will implement the functions according to RFC 6265 >> >>> >> I will also need to write the following functions. Should >> >>> >> they >> >>> >> also be >> >>> >> exported? >> >>> >> >> >>> >> canonicalizeDomain :: W.Ascii -> W.Ascii >> >>> >> >> >>> >> turns "..a.b.c..d.com..." to "a.b.c.d.com" >> >>> >> Technically necessary for domain matching (Chrome does it) >> >>> >> Perhaps unnecessary for a first pass? Perhaps we can trust >> >>> >> users >> >>> >> for >> >>> >> now? >> >>> >> >> >>> >> domainMatches :: W.Ascii -> W.Ascii -> Maybe W.Ascii >> >>> >> >> >>> >> Does the first domain match against the second domain? >> >>> >> If so, return the prefix of the first that isn't in the >> >>> >> second >> >>> >> >> >>> >> pathMatches :: W.Ascii -> W.Ascii -> Bool >> >>> >> >> >>> >> Do the paths match? >> >>> >> >> >>> >> In order to implement domain matching, I have to have >> >>> >> knowledge >> >>> >> of >> >>> >> the Public Suffix List so I know that >> >>> >> sub1.sub2.pvt.k12.wy.us >> >>> >> can >> >>> >> set >> >>> >> a >> >>> >> cookie for sub2.pvt.k12.wy.us but not for k12.wy.us >> >>> >> (because >> >>> >> pvt.k12.wy.us >> >>> >> is a "suffix"). There are a variety of ways to implement >> >>> >> this. >> >>> >> >> >>> >> As far as I can tell, Chrome does it by using a script >> >>> >> (which a >> >>> >> human >> >>> >> periodically runs) which parses the list at creates a .cc >> >>> >> file >> >>> >> that is >> >>> >> included in the build. >> >>> >> >> >>> >> I might be wrong about the execution of the script; it >> >>> >> might be >> >>> >> a >> >>> >> build >> >>> >> step. If it is a build step, however, it is suspicious that >> >>> >> a >> >>> >> build >> >>> >> target >> >>> >> would try to download a file... >> >>> >> >> >>> >> Any more elegant ideas? >> >>> >> >> >>> >> Feedback on any/all of the above would be very helpful >> >>> >> before I >> >>> >> go >> >>> >> off >> >>> >> into the weeds on this project. >> >>> >> >> >>> >> Thanks, >> >>> >> Myles C. Maxfield >> >>> >> >> >>> >> On Sat, Jan 28, 2012 at 8:17 PM, Michael Snoyman >> >>> >> >> >>> >> wrote: >> >>> >>> >> >>> >>> Thanks, looks great! I've merged it into the Github tree. >> >>> >>> >> >>> >>> On Sat, Jan 28, 2012 at 8:36 PM, Myles C. Maxfield >> >>> >>> wrote: >> >>> >>> > Ah, yes, you're completely right. I completely agree >> >>> >>> > that >> >>> >>> > moving >> >>> >>> > the >> >>> >>> > function into the Maybe monad increases readability. >> >>> >>> > This >> >>> >>> > kind >> >>> >>> > of >> >>> >>> > function >> >>> >>> > is what the Maybe monad was designed for. >> >>> >>> > >> >>> >>> > Here is a revised patch. >> >>> >>> > >> >>> >>> > >> >>> >>> > On Sat, Jan 28, 2012 at 8:28 AM, Michael Snoyman >> >>> >>> > >> >>> >>> > wrote: >> >>> >>> >> >> >>> >>> >> On Sat, Jan 28, 2012 at 1:20 AM, Myles C. Maxfield >> >>> >>> >> wrote: >> >>> >>> >> > the fromJust should never fail, beceause of the guard >> >>> >>> >> > statement: >> >>> >>> >> > >> >>> >>> >> >     | 300 <= code && code < 400 && isJust l'' && >> >>> >>> >> > isJust >> >>> >>> >> > l' = >> >>> >>> >> > Just $ >> >>> >>> >> > req >> >>> >>> >> > >> >>> >>> >> > Because of the order of the && operators, it will >> >>> >>> >> > only >> >>> >>> >> > evaluate >> >>> >>> >> > fromJust >> >>> >>> >> > after it makes sure that the argument isJust. That >> >>> >>> >> > function >> >>> >>> >> > in >> >>> >>> >> > particular >> >>> >>> >> > shouldn't throw any exceptions - it should only >> >>> >>> >> > return >> >>> >>> >> > Nothing. >> >>> >>> >> > >> >>> >>> >> > Knowing that, I don't quite think I understand what >> >>> >>> >> > your >> >>> >>> >> > concern >> >>> >>> >> > is. >> >>> >>> >> > Can >> >>> >>> >> > you >> >>> >>> >> > elaborate? >> >>> >>> >> >> >>> >>> >> You're right, but I had to squint really hard to prove >> >>> >>> >> to >> >>> >>> >> myself >> >>> >>> >> that >> >>> >>> >> you're right. That's the kind of code that could easily >> >>> >>> >> be >> >>> >>> >> broken >> >>> >>> >> in >> >>> >>> >> future updates by an unwitting maintainer (e.g., me). >> >>> >>> >> To >> >>> >>> >> protect >> >>> >>> >> the >> >>> >>> >> world from me, I'd prefer if the code didn't have the >> >>> >>> >> fromJust. >> >>> >>> >> This >> >>> >>> >> might be a good place to leverage the Monad instance of >> >>> >>> >> Maybe. >> >>> >>> >> >> >>> >>> >> Michael >> >>> >>> > >> >>> >>> > >> >>> >> >> >>> >> >> >>> > >> > >> > > >

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

I have been looking around at possibly making a Browser module for Network.HTTP.Conduit on top of Control.Monad.State. I came across this roadbump: In order to implement redirection following, client code must call 'http' with a redirection count of 0. If there is a redirect, 'http' will throw an exception. In order to catch this exception and continue on with the redirection chain, the 'catch' function must be called. The problem is that the 'catch' function has a type of (catch :: Exception e => IO a -> (e -> IO a) -> IO a) which means that it can only be used in the IO monad. A call to 'http' inside the first argument of 'catch' must be wrapped in a 'runResourceT' This has a couple implications: - The catch function cannot yield a Source, because any connections which 'http' opened will have to be immediately closed as soon as control reaches the 'catch' function (because of the runResourceT). The only way around this is to put the bind inside the catch block with some default sink, but this loses all of the constant-memory benefits of using a conduit in the first place. - The function that will be calling 'catch' operates as a State monad. Cookie state must be updated both before and after making the request (both inside the 'catch' block and outside). I could pass around an updated cookie jar out of the IO monad as an extra entry in a tuple, but this defeats the purpose of using a State monad in the first place. This kind of programming is very ugly. I see two solutions to these problems: - Make 'http' not throw a StatusCodeException if the _original_ redirection count = 0 - Make my internal module call the un-exported 'httpRaw' function. This has the problem that it doen't solve the problem for anyone else trying to use http-conduit; it only solves the problem for me. Perhaps this could be alleviated by exporting httpRaw. What do you think? --Myles On Tue, Feb 7, 2012 at 12:35 AM, Myles C. Maxfield

wrote:

Alright. I'll issue another pull request to you when it's done (expect in a couple weeks).

Thanks for your input so far, Aristid and Michael.

@Chris Wong: Do you want to talk about the Suffix List stuff some time?

--Myles

On Mon, Feb 6, 2012 at 10:14 PM, Michael Snoyman wrote:

...

+1

...

I would say: if it adds no package dependencies, put it right in.

Aristid

Am 06.02.2012 22:09 schrieb "Myles C. Maxfield" < myles.maxfield@gmail.com>:

...

After all these commits have been flying around, I have yet another question:

the 'HTTP' package defines Network.Browser which is a State monad which keeps state about a "browser" (i.e. a cookie jar, a proxy, redirection parameters, etc.) It would be pretty straightforward to implement this

kind

...

of functionality on top of http-conduit.

I was originally going to do it and release it as its own package, but it may be beneficial to add such a module to the existing http-conduit

...

...

Should I add it in to the existing package, or release it as its own package?

--Myles

On Mon, Feb 6, 2012 at 12:15 AM, Michael Snoyman wrote:

...

Just an FYI for everyone: Myles sent an (incredibly thorough) pull request to handle cookies:

https://github.com/snoyberg/http-conduit/pull/13

Thanks!

On Sun, Feb 5, 2012 at 8:20 AM, Myles C. Maxfield wrote:

...

1. The spec defines a grammar for the attributes. They're in

uppercase.

...

...

2. Yes - 1.3 is the first version that lists DiffTime as an instance of RealFrac (so I can use the 'floor' function to pull out the number of seconds to render it) 3. I'll see what I can do.

--Myles

On Sat, Feb 4, 2012 at 9:06 PM, Michael Snoyman < michael@snoyman.com> wrote: > > Looks good, a few questions/requests: > > 1. Is there a reason to upper-case all the attributes? > 2. Is the time >= 1.3 a requirements? Because that can cause a lot of > trouble for people. > 3. Can you send the patch as a Github pull request? It's easier to > track that way. > > Michael > > On Sat, Feb 4, 2012 at 1:21 AM, Myles C. Maxfield > wrote: > > Here is the patch to Web.Cookie. I didn't modify the tests at all > > because > > they were already broken - they looked like they hadn't been updated > > since > > SetCookie only had 5 parameters. I did verify by hand that the

...

...

...

...

> > works, > > though. > > > > Thanks, > > Myles > > > > > > On Thu, Feb 2, 2012 at 11:26 PM, Myles C. Maxfield > > wrote: > >> > >> Alright, I'll make a small patch that adds 2 fields to SetCookie: > >> setCookieMaxAge :: Maybe DiffTime > >> setCookieSecureOnly :: Bool > >> > >> I've also gotten started on those cookie functions. I'm currently > >> writing > >> tests for them. > >> > >> @Chris: The best advice I can give is that Chrome (what I'm using > >> as a > >> source on all this) has the data baked into a .cc file. However, > >> they > >> have > >> directions in a README and a script which will parse the list and > >> generate > >> that source file. I recommend doing this. That way, the Haskell > >> module > >> would > >> have 2 source files: one file that reads the list and generates

...

...

...

...

> >> second > >> file, which is a very large source file that contains each element > >> in > >> the > >> list. The list should export `elem`-type queries. I'm not quite > >> sure > >> how to > >> handle wildcards that appear in the list - that part is up to you. > >> Thanks > >> for helping out with this :] > >> > >> --Myles > >> > >> > >> On Thu, Feb 2, 2012 at 10:53 PM, Michael Snoyman > >> > >> wrote: > >>> > >>> Looks good to me too. I agree with Aristid: let's make the change > >>> to > >>> cookie itself. Do you want to send a pull request? I'm also > >>> considering making the SetCookie constructor hidden like we have > >>> for > >>> Request, so that if in the future we realize we need to add some > >>> other > >>> settings, it doesn't break the API. > >>> > >>> Chris: I would recommend compiling it into the module. Best bet > >>> would > >>> likely being converting the source file to Haskell source. > >>> > >>> Michael > >>> > >>> On Fri, Feb 3, 2012 at 6:32 AM, Myles C. Maxfield > >>> wrote: > >>> > Alright. After reading the spec, I have these questions / > >>> > concerns: > >>> > > >>> > The spec supports the "Max-Age" cookie attribute, which > >>> > Web.Cookies > >>> > doesn't. > >>> > > >>> > I see two possible solutions to this. The first is to have > >>> > parseSetCookie > >>> > take a UTCTime as an argument which will represent the current > >>> > time > >>> > so > >>> > it > >>> > can populate the setCookieExpires field by adding the Max-Age > >>> > attribute > >>> > to > >>> > the current time. Alternatively, that function can return an IO > >>> > SetCookie so > >>> > it can ask for the current time by itself (which I think is > >>> > inferior > >>> > to > >>> > taking the current time as an argument). Note that the spec says > >>> > to > >>> > prefer > >>> > Max-Age over Expires. > >>> > Add a field to SetCookie of type Maybe DiffTime which represents > >>> > the > >>> > Max-Age > >>> > attribute > >>> > > >>> > Cookie code should be aware of the Public Suffix List as a

...

...

...

...

> >>> > of > >>> > its > >>> > domain verification. The cookie code only needs to be able to > >>> > tell > >>> > if a > >>> > specific string is in the list (W.Ascii -> Bool) > >>> > > >>> > I propose making an entirely unrelated package, > >>> > public-suffix-list, > >>> > with a > >>> > module Network.PublicSuffixList, which will expose this > >>> > function, as > >>> > well as > >>> > functions about parsing the list itself. Thoughts? > >>> > > >>> > Web.Cookie doesn't have a "secure-only" attribute. Adding one in > >>> > is > >>> > straightforward enough. > >>> > The spec describes cookies as a property of HTTP, not of the > >>> > World > >>> > Wide > >>> > Web. > >>> > Perhaps "Web.Cookie" should be renamed? Just a thought; it > >>> > doesn't > >>> > really > >>> > matter to me. > >>> > > >>> > As for Network.HTTP.Conduit.Cookie, the spec describes in > >>> > section > >>> > 5.3 > >>> > "Storage Model" what fields a Cookie has. Here is my proposal > >>> > for > >>> > the > >>> > functions it will expose: > >>> > > >>> > receiveSetCookie :: SetCookie -> Req.Request m -> UTCTime -> > >>> > Bool -> > >>> > CookieJar -> CookieJar > >>> > > >>> > Runs the algorithm described in section 5.3 "Storage Model" > >>> > The UTCTime is the current-time, the Bool is whether or not

...

...

...

...

> >>> > caller > >>> > is an > >>> > HTTP-based API (as opposed to JavaScript or anything else) > >>> > > >>> > updateCookieJar :: Res.Response a -> Req.Request m -> UTCTime -> > >>> > CookieJar > >>> > -> (CookieJar, Res.Response a) > >>> > > >>> > Applies "receiveSetCookie" to a Response. The output CookieJar > >>> > is > >>> > stripped > >>> > of any Set-Cookie headers. > >>> > Specifies "True" for the Bool in receiveSetCookie > >>> > > >>> > computeCookieString :: Req.Request m -> CookieJar -> UTCTime -> > >>> > Bool > >>> > -> > >>> > (W.Ascii, CookieJar) > >>> > > >>> > Runs the algorithm described in section 5.4 "The Cookie Header" > >>> > The UTCTime and Bool are the same as in receiveSetCookie > >>> > > >>> > insertCookiesIntoRequest :: Req.Request m -> CookieJar -> > >>> > UTCTime -> > >>> > (Req.Request m, CookieJar) > >>> > > >>> > Applies "computeCookieString" to a Request. The output cookie > >>> > jar > >>> > has > >>> > updated last-accessed-times. > >>> > Specifies "True" for the Bool in computeCookieString > >>> > > >>> > evictExpiredCookies :: CookieJar -> UTCTime -> CookieJar > >>> > > >>> > Runs the algorithm described in the last part of section 5.3 > >>> > "Storage > >>> > Model" > >>> > > >>> > This will make the relevant part of 'http' look like: > >>> > > >>> > go count req'' cookie_jar'' = do > >>> > now <- liftIO $ getCurrentTime > >>> > let (req', cookie_jar') = insertCookiesIntoRequest req'' > >>> > (evictExpiredCookies cookie_jar'' now) now > >>> > res' <- httpRaw req' manager > >>> > let (cookie_jar, res) = updateCookieJar res' req' now > >>> > cookie_jar' > >>> > case getRedirectedRequest req' (responseHeaders res) > >>> > (W.statusCode > >>> > (statusCode res)) of > >>> > Just req -> go (count - 1) req cookie_jar > >>> > Nothing -> return res > >>> > > >>> > I plan to not allow for a user-supplied cookieFilter function. > >>> > If > >>> > they > >>> > want > >>> > that functionality, they can re-implement the > >>> > redirection-following > >>> > logic. > >>> > > >>> > Any thoughts on any of this? > >>> > > >>> > Thanks, > >>> > Myles > >>> > > >>> > On Wed, Feb 1, 2012 at 5:19 PM, Myles C. Maxfield > >>> > > >>> > wrote: > >>> >> > >>> >> Nope. I'm not. The RFC is very explicit about how to handle > >>> >> cookies. > >>> >> As > >>> >> soon as I'm finished making sense of it (in terms of Haskell) > >>> >> I'll > >>> >> send > >>> >> another proposal email. > >>> >> > >>> >> On Feb 1, 2012 3:25 AM, "Michael Snoyman" < michael@snoyman.com> > >>> >> wrote: > >>> >>> > >>> >>> You mean you're *not* making this proposal? > >>> >>> > >>> >>> On Wed, Feb 1, 2012 at 7:30 AM, Myles C. Maxfield > >>> >>> wrote: > >>> >>> > Well, this is embarrassing. Please disregard my previous > >>> >>> > email. > >>> >>> > I > >>> >>> > should > >>> >>> > learn to read the RFC *before* submitting proposals. > >>> >>> > > >>> >>> > --Myles > >>> >>> > > >>> >>> > > >>> >>> > On Tue, Jan 31, 2012 at 6:37 PM, Myles C. Maxfield > >>> >>> > wrote: > >>> >>> >> > >>> >>> >> Here are my initial ideas about supporting cookies. Note > >>> >>> >> that > >>> >>> >> I'm > >>> >>> >> using > >>> >>> >> Chrome for ideas since it's open source. > >>> >>> >> > >>> >>> >> Network/HTTP/Conduit/Cookies.hs file > >>> >>> >> Exporting the following symbols: > >>> >>> >> > >>> >>> >> type StuffedCookie = SetCookie > >>> >>> >> > >>> >>> >> A regular SetCookie can have Nothing for its Domain and > >>> >>> >> Path > >>> >>> >> attributes. A > >>> >>> >> StuffedCookie has to have these fields set. > >>> >>> >> > >>> >>> >> type CookieJar = [StuffedCookie] > >>> >>> >> > >>> >>> >> Chrome's cookie jar is implemented as (the C++ equivalent > >>> >>> >> of) > >>> >>> >> Map > >>> >>> >> W.Ascii > >>> >>> >> StuffedCookie. The key is the "eTLD+1" of the domain, so > >>> >>> >> lookups > >>> >>> >> for > >>> >>> >> all > >>> >>> >> cookies for a given domain are fast. > >>> >>> >> I think I'll stay with just a list of StuffedCookies just > >>> >>> >> to > >>> >>> >> keep > >>> >>> >> it > >>> >>> >> simple. Perhaps a later revision can implement the faster > >>> >>> >> map. > >>> >>> >> > >>> >>> >> getRelevantCookies :: Request m -> CookieJar -> UTCTime -> > >>> >>> >> (CookieJar, > >>> >>> >> Cookies) > >>> >>> >> > >>> >>> >> Gets all the cookies from the cookie jar that should be set > >>> >>> >> for > >>> >>> >> the > >>> >>> >> given > >>> >>> >> Request. > >>> >>> >> The time argument is whatever "now" is (it's pulled out of > >>> >>> >> the > >>> >>> >> function so > >>> >>> >> the function can remain pure and easily testable) > >>> >>> >> The function will also remove expired cookies from the > >>> >>> >> cookie > >>> >>> >> jar > >>> >>> >> (given > >>> >>> >> what "now" is) and return the filtered cookie jar > >>> >>> >> > >>> >>> >> putRelevantCookies :: Request m -> CookieJar -> > >>> >>> >> [StuffedCookie] > >>> >>> >> -> > >>> >>> >> CookieJar > >>> >>> >> > >>> >>> >> Insert cookies from a server response into the cookie jar. > >>> >>> >> The first argument is only used for checking to see which > >>> >>> >> cookies > >>> >>> >> are > >>> >>> >> valid (which cookies match the requested domain, etc, so > >>> >>> >> site1.com > >>> >>> >> can't set > >>> >>> >> a cookie for site2.com) > >>> >>> >> > >>> >>> >> stuffCookie :: Request m -> SetCookie -> StuffedCookie > >>> >>> >> > >>> >>> >> If the SetCookie's fields are Nothing, fill them in given > >>> >>> >> the > >>> >>> >> Request > >>> >>> >> from > >>> >>> >> which it originated > >>> >>> >> > >>> >>> >> getCookies :: Response a -> ([SetCookie], Response a) > >>> >>> >> > >>> >>> >> Pull cookies out of a server response. Return the response > >>> >>> >> with > >>> >>> >> the > >>> >>> >> Set-Cookie headers filtered out > >>> >>> >> > >>> >>> >> putCookies :: Request a -> Cookies -> Request a > >>> >>> >> > >>> >>> >> A wrapper around renderCookies. Inserts some cookies into a > >>> >>> >> request. > >>> >>> >> Doesn't overwrite cookies that are already set in the > >>> >>> >> request > >>> >>> >> > >>> >>> >> These functions will be exported from Network.HTTP.Conduit > >>> >>> >> as > >>> >>> >> well, so > >>> >>> >> callers can use them to re-implement redirection chains > >>> >>> >> I won't implement a cookie filtering function (like what > >>> >>> >> Network.Browser > >>> >>> >> has) > >>> >>> >> > >>> >>> >> If you want to have arbitrary handling of cookies, > >>> >>> >> re-implement > >>> >>> >> redirection following. It's not very difficult if you use > >>> >>> >> the > >>> >>> >> API > >>> >>> >> provided, > >>> >>> >> and the 'http' function is open source so you can use

...

...

...

...

> >>> >>> >> as a > >>> >>> >> reference. > >>> >>> >> > >>> >>> >> I will implement the functions according to RFC 6265 > >>> >>> >> I will also need to write the following functions. Should > >>> >>> >> they > >>> >>> >> also be > >>> >>> >> exported? > >>> >>> >> > >>> >>> >> canonicalizeDomain :: W.Ascii -> W.Ascii > >>> >>> >> > >>> >>> >> turns "..a.b.c..d.com..." to "a.b.c.d.com" > >>> >>> >> Technically necessary for domain matching (Chrome does it) > >>> >>> >> Perhaps unnecessary for a first pass? Perhaps we can

...

...

...

...

> >>> >>> >> users > >>> >>> >> for > >>> >>> >> now? > >>> >>> >> > >>> >>> >> domainMatches :: W.Ascii -> W.Ascii -> Maybe W.Ascii > >>> >>> >> > >>> >>> >> Does the first domain match against the second domain? > >>> >>> >> If so, return the prefix of the first that isn't in the > >>> >>> >> second > >>> >>> >> > >>> >>> >> pathMatches :: W.Ascii -> W.Ascii -> Bool > >>> >>> >> > >>> >>> >> Do the paths match? > >>> >>> >> > >>> >>> >> In order to implement domain matching, I have to have > >>> >>> >> knowledge > >>> >>> >> of > >>> >>> >> the Public Suffix List so I know that > >>> >>> >> sub1.sub2.pvt.k12.wy.us > >>> >>> >> can > >>> >>> >> set > >>> >>> >> a > >>> >>> >> cookie for sub2.pvt.k12.wy.us but not for k12.wy.us > >>> >>> >> (because > >>> >>> >> pvt.k12.wy.us > >>> >>> >> is a "suffix"). There are a variety of ways to implement > >>> >>> >> this. > >>> >>> >> > >>> >>> >> As far as I can tell, Chrome does it by using a script > >>> >>> >> (which a > >>> >>> >> human > >>> >>> >> periodically runs) which parses the list at creates a .cc > >>> >>> >> file > >>> >>> >> that is > >>> >>> >> included in the build. > >>> >>> >> > >>> >>> >> I might be wrong about the execution of the script; it > >>> >>> >> might be > >>> >>> >> a > >>> >>> >> build > >>> >>> >> step. If it is a build step, however, it is suspicious

...

...

...

...

> >>> >>> >> a > >>> >>> >> build > >>> >>> >> target > >>> >>> >> would try to download a file... > >>> >>> >> > >>> >>> >> Any more elegant ideas? > >>> >>> >> > >>> >>> >> Feedback on any/all of the above would be very helpful > >>> >>> >> before I > >>> >>> >> go > >>> >>> >> off > >>> >>> >> into the weeds on this project. > >>> >>> >> > >>> >>> >> Thanks, > >>> >>> >> Myles C. Maxfield > >>> >>> >> > >>> >>> >> On Sat, Jan 28, 2012 at 8:17 PM, Michael Snoyman > >>> >>> >> > >>> >>> >> wrote: > >>> >>> >>> > >>> >>> >>> Thanks, looks great! I've merged it into the Github

...

...

...

...

> >>> >>> >>> > >>> >>> >>> On Sat, Jan 28, 2012 at 8:36 PM, Myles C. Maxfield > >>> >>> >>> wrote: > >>> >>> >>> > Ah, yes, you're completely right. I completely agree > >>> >>> >>> > that > >>> >>> >>> > moving > >>> >>> >>> > the > >>> >>> >>> > function into the Maybe monad increases readability. > >>> >>> >>> > This > >>> >>> >>> > kind > >>> >>> >>> > of > >>> >>> >>> > function > >>> >>> >>> > is what the Maybe monad was designed for. > >>> >>> >>> > > >>> >>> >>> > Here is a revised patch. > >>> >>> >>> > > >>> >>> >>> > > >>> >>> >>> > On Sat, Jan 28, 2012 at 8:28 AM, Michael Snoyman > >>> >>> >>> > > >>> >>> >>> > wrote: > >>> >>> >>> >> > >>> >>> >>> >> On Sat, Jan 28, 2012 at 1:20 AM, Myles C. Maxfield > >>> >>> >>> >> wrote: > >>> >>> >>> >> > the fromJust should never fail, beceause of the guard > >>> >>> >>> >> > statement: > >>> >>> >>> >> > > >>> >>> >>> >> > | 300 <= code && code < 400 && isJust l'' && > >>> >>> >>> >> > isJust > >>> >>> >>> >> > l' = > >>> >>> >>> >> > Just $ > >>> >>> >>> >> > req > >>> >>> >>> >> > > >>> >>> >>> >> > Because of the order of the && operators, it will > >>> >>> >>> >> > only > >>> >>> >>> >> > evaluate > >>> >>> >>> >> > fromJust > >>> >>> >>> >> > after it makes sure that the argument isJust. That > >>> >>> >>> >> > function > >>> >>> >>> >> > in > >>> >>> >>> >> > particular > >>> >>> >>> >> > shouldn't throw any exceptions - it should only > >>> >>> >>> >> > return > >>> >>> >>> >> > Nothing. > >>> >>> >>> >> > > >>> >>> >>> >> > Knowing that, I don't quite think I understand what > >>> >>> >>> >> > your > >>> >>> >>> >> > concern > >>> >>> >>> >> > is. > >>> >>> >>> >> > Can > >>> >>> >>> >> > you > >>> >>> >>> >> > elaborate? > >>> >>> >>> >> > >>> >>> >>> >> You're right, but I had to squint really hard to

...

...

...

...

> >>> >>> >>> >> to > >>> >>> >>> >> myself > >>> >>> >>> >> that > >>> >>> >>> >> you're right. That's the kind of code that could easily > >>> >>> >>> >> be > >>> >>> >>> >> broken > >>> >>> >>> >> in > >>> >>> >>> >> future updates by an unwitting maintainer (e.g., me). > >>> >>> >>> >> To > >>> >>> >>> >> protect > >>> >>> >>> >> the > >>> >>> >>> >> world from me, I'd prefer if the code didn't have

On Mon, Feb 6, 2012 at 11:16 PM, Aristid Breitkreuz wrote: package. patch the part the that trust that tree. prove the

...

...

...

...

> >>> >>> >>> >> fromJust. > >>> >>> >>> >> This > >>> >>> >>> >> might be a good place to leverage the Monad instance of > >>> >>> >>> >> Maybe. > >>> >>> >>> >> > >>> >>> >>> >> Michael > >>> >>> >>> > > >>> >>> >>> > > >>> >>> >> > >>> >>> >> > >>> >>> > > >>> > > >>> > > >> > >> > >

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

On Wed, Feb 8, 2012 at 6:28 AM, Myles C. Maxfield wrote:

I have been looking around at possibly making a Browser module for Network.HTTP.Conduit on top of Control.Monad.State. I came across this roadbump:

In order to implement redirection following, client code must call 'http' with a redirection count of 0. If there is a redirect, 'http' will throw an exception. In order to catch this exception and continue on with the

Actually, this is just a setting: you can override with checkStatus[1]. [1] http://hackage.haskell.org/packages/archive/http-conduit/1.2.4/doc/html/Netw...

redirection chain, the 'catch' function must be called. The problem is that the 'catch' function has a type of (catch :: Exception e => IO a -> (e -> IO a) -> IO a) which means that it can only be used in the IO monad. A call to 'http' inside the first argument of 'catch' must be wrapped in a 'runResourceT'

This has a couple implications:

The catch function cannot yield a Source, because any connections which 'http' opened will have to be immediately closed as soon as control reaches the 'catch' function (because of the runResourceT). The only way around this is to put the bind inside the catch block with some default sink, but this loses all of the constant-memory benefits of using a conduit in the first place. The function that will be calling 'catch' operates as a State monad. Cookie state must be updated both before and after making the request (both inside the 'catch' block and outside). I could pass around an updated cookie jar out of the IO monad as an extra entry in a tuple, but this defeats the purpose of using a State monad in the first place. This kind of programming is very ugly.

I see two solutions to these problems:

Make 'http' not throw a StatusCodeException if the _original_ redirection count = 0 Make my internal module call the un-exported 'httpRaw' function. This has the problem that it doen't solve the problem for anyone else trying to use http-conduit; it only solves the problem for me. Perhaps this could be alleviated by exporting httpRaw.

What do you think? --Myles

On Tue, Feb 7, 2012 at 12:35 AM, Myles C. Maxfield wrote:

...

Alright. I'll issue another pull request to you when it's done (expect in a couple weeks).

Thanks for your input so far, Aristid and Michael.

@Chris Wong: Do you want to talk about the Suffix List stuff some time?

--Myles

On Mon, Feb 6, 2012 at 10:14 PM, Michael Snoyman wrote:

...

+1

On Mon, Feb 6, 2012 at 11:16 PM, Aristid Breitkreuz wrote:

...

I would say: if it adds no package dependencies, put it right in.

Aristid

Am 06.02.2012 22:09 schrieb "Myles C. Maxfield" :

...

After all these commits have been flying around, I have yet another question:

the 'HTTP' package defines Network.Browser which is a State monad which keeps state about a "browser" (i.e. a cookie jar, a proxy, redirection parameters, etc.) It would be pretty straightforward to implement this kind of functionality on top of http-conduit.

I was originally going to do it and release it as its own package, but it may be beneficial to add such a module to the existing http-conduit package. Should I add it in to the existing package, or release it as its own package?

--Myles

On Mon, Feb 6, 2012 at 12:15 AM, Michael Snoyman wrote:

...

Just an FYI for everyone: Myles sent an (incredibly thorough) pull request to handle cookies:

https://github.com/snoyberg/http-conduit/pull/13

Thanks!

On Sun, Feb 5, 2012 at 8:20 AM, Myles C. Maxfield wrote: > 1. The spec defines a grammar for the attributes. They're in > uppercase. > 2. Yes - 1.3 is the first version that lists DiffTime as an > instance of > RealFrac (so I can use the 'floor' function to pull out the number > of > seconds to render it) > 3. I'll see what I can do. > > --Myles > > > On Sat, Feb 4, 2012 at 9:06 PM, Michael Snoyman > > wrote: >> >> Looks good, a few questions/requests: >> >> 1. Is there a reason to upper-case all the attributes? >> 2. Is the time >= 1.3 a requirements? Because that can cause a lot >> of >> trouble for people. >> 3. Can you send the patch as a Github pull request? It's easier to >> track that way. >> >> Michael >> >> On Sat, Feb 4, 2012 at 1:21 AM, Myles C. Maxfield >> wrote: >> > Here is the patch to Web.Cookie. I didn't modify the tests at >> > all >> > because >> > they were already broken - they looked like they hadn't been >> > updated >> > since >> > SetCookie only had 5 parameters. I did verify by hand that the >> > patch >> > works, >> > though. >> > >> > Thanks, >> > Myles >> > >> > >> > On Thu, Feb 2, 2012 at 11:26 PM, Myles C. Maxfield >> > wrote: >> >> >> >> Alright, I'll make a small patch that adds 2 fields to >> >> SetCookie: >> >> setCookieMaxAge :: Maybe DiffTime >> >> setCookieSecureOnly :: Bool >> >> >> >> I've also gotten started on those cookie functions. I'm >> >> currently >> >> writing >> >> tests for them. >> >> >> >> @Chris: The best advice I can give is that Chrome (what I'm >> >> using >> >> as a >> >> source on all this) has the data baked into a .cc file. >> >> However, >> >> they >> >> have >> >> directions in a README and a script which will parse the list >> >> and >> >> generate >> >> that source file. I recommend doing this. That way, the Haskell >> >> module >> >> would >> >> have 2 source files: one file that reads the list and generates >> >> the >> >> second >> >> file, which is a very large source file that contains each >> >> element >> >> in >> >> the >> >> list. The list should export `elem`-type queries. I'm not quite >> >> sure >> >> how to >> >> handle wildcards that appear in the list - that part is up to >> >> you. >> >> Thanks >> >> for helping out with this :] >> >> >> >> --Myles >> >> >> >> >> >> On Thu, Feb 2, 2012 at 10:53 PM, Michael Snoyman >> >> >> >> wrote: >> >>> >> >>> Looks good to me too. I agree with Aristid: let's make the >> >>> change >> >>> to >> >>> cookie itself. Do you want to send a pull request? I'm also >> >>> considering making the SetCookie constructor hidden like we >> >>> have >> >>> for >> >>> Request, so that if in the future we realize we need to add >> >>> some >> >>> other >> >>> settings, it doesn't break the API. >> >>> >> >>> Chris: I would recommend compiling it into the module. Best >> >>> bet >> >>> would >> >>> likely being converting the source file to Haskell source. >> >>> >> >>> Michael >> >>> >> >>> On Fri, Feb 3, 2012 at 6:32 AM, Myles C. Maxfield >> >>> wrote: >> >>> > Alright. After reading the spec, I have these questions / >> >>> > concerns: >> >>> > >> >>> > The spec supports the "Max-Age" cookie attribute, which >> >>> > Web.Cookies >> >>> > doesn't. >> >>> > >> >>> > I see two possible solutions to this. The first is to have >> >>> > parseSetCookie >> >>> > take a UTCTime as an argument which will represent the >> >>> > current >> >>> > time >> >>> > so >> >>> > it >> >>> > can populate the setCookieExpires field by adding the >> >>> > Max-Age >> >>> > attribute >> >>> > to >> >>> > the current time. Alternatively, that function can return an >> >>> > IO >> >>> > SetCookie so >> >>> > it can ask for the current time by itself (which I think is >> >>> > inferior >> >>> > to >> >>> > taking the current time as an argument). Note that the spec >> >>> > says >> >>> > to >> >>> > prefer >> >>> > Max-Age over Expires. >> >>> > Add a field to SetCookie of type Maybe DiffTime which >> >>> > represents >> >>> > the >> >>> > Max-Age >> >>> > attribute >> >>> > >> >>> > Cookie code should be aware of the Public Suffix List as a >> >>> > part >> >>> > of >> >>> > its >> >>> > domain verification. The cookie code only needs to be able >> >>> > to >> >>> > tell >> >>> > if a >> >>> > specific string is in the list (W.Ascii -> Bool) >> >>> > >> >>> > I propose making an entirely unrelated package, >> >>> > public-suffix-list, >> >>> > with a >> >>> > module Network.PublicSuffixList, which will expose this >> >>> > function, as >> >>> > well as >> >>> > functions about parsing the list itself. Thoughts? >> >>> > >> >>> > Web.Cookie doesn't have a "secure-only" attribute. Adding >> >>> > one in >> >>> > is >> >>> > straightforward enough. >> >>> > The spec describes cookies as a property of HTTP, not of the >> >>> > World >> >>> > Wide >> >>> > Web. >> >>> > Perhaps "Web.Cookie" should be renamed? Just a thought; it >> >>> > doesn't >> >>> > really >> >>> > matter to me. >> >>> > >> >>> > As for Network.HTTP.Conduit.Cookie, the spec describes in >> >>> > section >> >>> > 5.3 >> >>> > "Storage Model" what fields a Cookie has. Here is my >> >>> > proposal >> >>> > for >> >>> > the >> >>> > functions it will expose: >> >>> > >> >>> > receiveSetCookie :: SetCookie -> Req.Request m -> UTCTime -> >> >>> > Bool -> >> >>> > CookieJar -> CookieJar >> >>> > >> >>> > Runs the algorithm described in section 5.3 "Storage Model" >> >>> > The UTCTime is the current-time, the Bool is whether or not >> >>> > the >> >>> > caller >> >>> > is an >> >>> > HTTP-based API (as opposed to JavaScript or anything else) >> >>> > >> >>> > updateCookieJar :: Res.Response a -> Req.Request m -> >> >>> > UTCTime -> >> >>> > CookieJar >> >>> > -> (CookieJar, Res.Response a) >> >>> > >> >>> > Applies "receiveSetCookie" to a Response. The output >> >>> > CookieJar >> >>> > is >> >>> > stripped >> >>> > of any Set-Cookie headers. >> >>> > Specifies "True" for the Bool in receiveSetCookie >> >>> > >> >>> > computeCookieString :: Req.Request m -> CookieJar -> UTCTime >> >>> > -> >> >>> > Bool >> >>> > -> >> >>> > (W.Ascii, CookieJar) >> >>> > >> >>> > Runs the algorithm described in section 5.4 "The Cookie >> >>> > Header" >> >>> > The UTCTime and Bool are the same as in receiveSetCookie >> >>> > >> >>> > insertCookiesIntoRequest :: Req.Request m -> CookieJar -> >> >>> > UTCTime -> >> >>> > (Req.Request m, CookieJar) >> >>> > >> >>> > Applies "computeCookieString" to a Request. The output >> >>> > cookie >> >>> > jar >> >>> > has >> >>> > updated last-accessed-times. >> >>> > Specifies "True" for the Bool in computeCookieString >> >>> > >> >>> > evictExpiredCookies :: CookieJar -> UTCTime -> CookieJar >> >>> > >> >>> > Runs the algorithm described in the last part of section 5.3 >> >>> > "Storage >> >>> > Model" >> >>> > >> >>> > This will make the relevant part of 'http' look like: >> >>> > >> >>> >     go count req'' cookie_jar'' = do >> >>> >         now <- liftIO $ getCurrentTime >> >>> >         let (req', cookie_jar') = insertCookiesIntoRequest >> >>> > req'' >> >>> > (evictExpiredCookies cookie_jar'' now) now >> >>> >         res' <- httpRaw req' manager >> >>> >         let (cookie_jar, res) = updateCookieJar res' req' >> >>> > now >> >>> > cookie_jar' >> >>> >         case getRedirectedRequest req' (responseHeaders res) >> >>> > (W.statusCode >> >>> > (statusCode res)) of >> >>> >             Just req -> go (count - 1) req cookie_jar >> >>> >             Nothing -> return res >> >>> > >> >>> > I plan to not allow for a user-supplied cookieFilter >> >>> > function. >> >>> > If >> >>> > they >> >>> > want >> >>> > that functionality, they can re-implement the >> >>> > redirection-following >> >>> > logic. >> >>> > >> >>> > Any thoughts on any of this? >> >>> > >> >>> > Thanks, >> >>> > Myles >> >>> > >> >>> > On Wed, Feb 1, 2012 at 5:19 PM, Myles C. Maxfield >> >>> > >> >>> > wrote: >> >>> >> >> >>> >> Nope. I'm not. The RFC is very explicit about how to handle >> >>> >> cookies. >> >>> >> As >> >>> >> soon as I'm finished making sense of it (in terms of >> >>> >> Haskell) >> >>> >> I'll >> >>> >> send >> >>> >> another proposal email. >> >>> >> >> >>> >> On Feb 1, 2012 3:25 AM, "Michael Snoyman" >> >>> >> >> >>> >> wrote: >> >>> >>> >> >>> >>> You mean you're *not* making this proposal? >> >>> >>> >> >>> >>> On Wed, Feb 1, 2012 at 7:30 AM, Myles C. Maxfield >> >>> >>> wrote: >> >>> >>> > Well, this is embarrassing. Please disregard my previous >> >>> >>> > email. >> >>> >>> > I >> >>> >>> > should >> >>> >>> > learn to read the RFC *before* submitting proposals. >> >>> >>> > >> >>> >>> > --Myles >> >>> >>> > >> >>> >>> > >> >>> >>> > On Tue, Jan 31, 2012 at 6:37 PM, Myles C. Maxfield >> >>> >>> > wrote: >> >>> >>> >> >> >>> >>> >> Here are my initial ideas about supporting cookies. >> >>> >>> >> Note >> >>> >>> >> that >> >>> >>> >> I'm >> >>> >>> >> using >> >>> >>> >> Chrome for ideas since it's open source. >> >>> >>> >> >> >>> >>> >> Network/HTTP/Conduit/Cookies.hs file >> >>> >>> >> Exporting the following symbols: >> >>> >>> >> >> >>> >>> >> type StuffedCookie = SetCookie >> >>> >>> >> >> >>> >>> >> A regular SetCookie can have Nothing for its Domain and >> >>> >>> >> Path >> >>> >>> >> attributes. A >> >>> >>> >> StuffedCookie has to have these fields set. >> >>> >>> >> >> >>> >>> >> type CookieJar = [StuffedCookie] >> >>> >>> >> >> >>> >>> >> Chrome's cookie jar is implemented as (the C++ >> >>> >>> >> equivalent >> >>> >>> >> of) >> >>> >>> >> Map >> >>> >>> >> W.Ascii >> >>> >>> >> StuffedCookie. The key is the "eTLD+1" of the domain, >> >>> >>> >> so >> >>> >>> >> lookups >> >>> >>> >> for >> >>> >>> >> all >> >>> >>> >> cookies for a given domain are fast. >> >>> >>> >> I think I'll stay with just a list of StuffedCookies >> >>> >>> >> just >> >>> >>> >> to >> >>> >>> >> keep >> >>> >>> >> it >> >>> >>> >> simple. Perhaps a later revision can implement the >> >>> >>> >> faster >> >>> >>> >> map. >> >>> >>> >> >> >>> >>> >> getRelevantCookies :: Request m -> CookieJar -> UTCTime >> >>> >>> >> -> >> >>> >>> >> (CookieJar, >> >>> >>> >> Cookies) >> >>> >>> >> >> >>> >>> >> Gets all the cookies from the cookie jar that should be >> >>> >>> >> set >> >>> >>> >> for >> >>> >>> >> the >> >>> >>> >> given >> >>> >>> >> Request. >> >>> >>> >> The time argument is whatever "now" is (it's pulled out >> >>> >>> >> of >> >>> >>> >> the >> >>> >>> >> function so >> >>> >>> >> the function can remain pure and easily testable) >> >>> >>> >> The function will also remove expired cookies from the >> >>> >>> >> cookie >> >>> >>> >> jar >> >>> >>> >> (given >> >>> >>> >> what "now" is) and return the filtered cookie jar >> >>> >>> >> >> >>> >>> >> putRelevantCookies :: Request m -> CookieJar -> >> >>> >>> >> [StuffedCookie] >> >>> >>> >> -> >> >>> >>> >> CookieJar >> >>> >>> >> >> >>> >>> >> Insert cookies from a server response into the cookie >> >>> >>> >> jar. >> >>> >>> >> The first argument is only used for checking to see >> >>> >>> >> which >> >>> >>> >> cookies >> >>> >>> >> are >> >>> >>> >> valid (which cookies match the requested domain, etc, >> >>> >>> >> so >> >>> >>> >> site1.com >> >>> >>> >> can't set >> >>> >>> >> a cookie for site2.com) >> >>> >>> >> >> >>> >>> >> stuffCookie :: Request m -> SetCookie -> StuffedCookie >> >>> >>> >> >> >>> >>> >> If the SetCookie's fields are Nothing, fill them in >> >>> >>> >> given >> >>> >>> >> the >> >>> >>> >> Request >> >>> >>> >> from >> >>> >>> >> which it originated >> >>> >>> >> >> >>> >>> >> getCookies :: Response a -> ([SetCookie], Response a) >> >>> >>> >> >> >>> >>> >> Pull cookies out of a server response. Return the >> >>> >>> >> response >> >>> >>> >> with >> >>> >>> >> the >> >>> >>> >> Set-Cookie headers filtered out >> >>> >>> >> >> >>> >>> >> putCookies :: Request a -> Cookies -> Request a >> >>> >>> >> >> >>> >>> >> A wrapper around renderCookies. Inserts some cookies >> >>> >>> >> into a >> >>> >>> >> request. >> >>> >>> >> Doesn't overwrite cookies that are already set in the >> >>> >>> >> request >> >>> >>> >> >> >>> >>> >> These functions will be exported from >> >>> >>> >> Network.HTTP.Conduit >> >>> >>> >> as >> >>> >>> >> well, so >> >>> >>> >> callers can use them to re-implement redirection chains >> >>> >>> >> I won't implement a cookie filtering function (like >> >>> >>> >> what >> >>> >>> >> Network.Browser >> >>> >>> >> has) >> >>> >>> >> >> >>> >>> >> If you want to have arbitrary handling of cookies, >> >>> >>> >> re-implement >> >>> >>> >> redirection following. It's not very difficult if you >> >>> >>> >> use >> >>> >>> >> the >> >>> >>> >> API >> >>> >>> >> provided, >> >>> >>> >> and the 'http' function is open source so you can use >> >>> >>> >> that >> >>> >>> >> as a >> >>> >>> >> reference. >> >>> >>> >> >> >>> >>> >> I will implement the functions according to RFC 6265 >> >>> >>> >> I will also need to write the following functions. >> >>> >>> >> Should >> >>> >>> >> they >> >>> >>> >> also be >> >>> >>> >> exported? >> >>> >>> >> >> >>> >>> >> canonicalizeDomain :: W.Ascii -> W.Ascii >> >>> >>> >> >> >>> >>> >> turns "..a.b.c..d.com..." to "a.b.c.d.com" >> >>> >>> >> Technically necessary for domain matching (Chrome does >> >>> >>> >> it) >> >>> >>> >> Perhaps unnecessary for a first pass? Perhaps we can >> >>> >>> >> trust >> >>> >>> >> users >> >>> >>> >> for >> >>> >>> >> now? >> >>> >>> >> >> >>> >>> >> domainMatches :: W.Ascii -> W.Ascii -> Maybe W.Ascii >> >>> >>> >> >> >>> >>> >> Does the first domain match against the second domain? >> >>> >>> >> If so, return the prefix of the first that isn't in the >> >>> >>> >> second >> >>> >>> >> >> >>> >>> >> pathMatches :: W.Ascii -> W.Ascii -> Bool >> >>> >>> >> >> >>> >>> >> Do the paths match? >> >>> >>> >> >> >>> >>> >> In order to implement domain matching, I have to have >> >>> >>> >> knowledge >> >>> >>> >> of >> >>> >>> >> the Public Suffix List so I know that >> >>> >>> >> sub1.sub2.pvt.k12.wy.us >> >>> >>> >> can >> >>> >>> >> set >> >>> >>> >> a >> >>> >>> >> cookie for sub2.pvt.k12.wy.us but not for k12.wy.us >> >>> >>> >> (because >> >>> >>> >> pvt.k12.wy.us >> >>> >>> >> is a "suffix"). There are a variety of ways to >> >>> >>> >> implement >> >>> >>> >> this. >> >>> >>> >> >> >>> >>> >> As far as I can tell, Chrome does it by using a script >> >>> >>> >> (which a >> >>> >>> >> human >> >>> >>> >> periodically runs) which parses the list at creates a >> >>> >>> >> .cc >> >>> >>> >> file >> >>> >>> >> that is >> >>> >>> >> included in the build. >> >>> >>> >> >> >>> >>> >> I might be wrong about the execution of the script; it >> >>> >>> >> might be >> >>> >>> >> a >> >>> >>> >> build >> >>> >>> >> step. If it is a build step, however, it is suspicious >> >>> >>> >> that >> >>> >>> >> a >> >>> >>> >> build >> >>> >>> >> target >> >>> >>> >> would try to download a file... >> >>> >>> >> >> >>> >>> >> Any more elegant ideas? >> >>> >>> >> >> >>> >>> >> Feedback on any/all of the above would be very helpful >> >>> >>> >> before I >> >>> >>> >> go >> >>> >>> >> off >> >>> >>> >> into the weeds on this project. >> >>> >>> >> >> >>> >>> >> Thanks, >> >>> >>> >> Myles C. Maxfield >> >>> >>> >> >> >>> >>> >> On Sat, Jan 28, 2012 at 8:17 PM, Michael Snoyman >> >>> >>> >> >> >>> >>> >> wrote: >> >>> >>> >>> >> >>> >>> >>> Thanks, looks great! I've merged it into the Github >> >>> >>> >>> tree. >> >>> >>> >>> >> >>> >>> >>> On Sat, Jan 28, 2012 at 8:36 PM, Myles C. Maxfield >> >>> >>> >>> wrote: >> >>> >>> >>> > Ah, yes, you're completely right. I completely agree >> >>> >>> >>> > that >> >>> >>> >>> > moving >> >>> >>> >>> > the >> >>> >>> >>> > function into the Maybe monad increases readability. >> >>> >>> >>> > This >> >>> >>> >>> > kind >> >>> >>> >>> > of >> >>> >>> >>> > function >> >>> >>> >>> > is what the Maybe monad was designed for. >> >>> >>> >>> > >> >>> >>> >>> > Here is a revised patch. >> >>> >>> >>> > >> >>> >>> >>> > >> >>> >>> >>> > On Sat, Jan 28, 2012 at 8:28 AM, Michael Snoyman >> >>> >>> >>> > >> >>> >>> >>> > wrote: >> >>> >>> >>> >> >> >>> >>> >>> >> On Sat, Jan 28, 2012 at 1:20 AM, Myles C. Maxfield >> >>> >>> >>> >> wrote: >> >>> >>> >>> >> > the fromJust should never fail, beceause of the >> >>> >>> >>> >> > guard >> >>> >>> >>> >> > statement: >> >>> >>> >>> >> > >> >>> >>> >>> >> >     | 300 <= code && code < 400 && isJust l'' && >> >>> >>> >>> >> > isJust >> >>> >>> >>> >> > l' = >> >>> >>> >>> >> > Just $ >> >>> >>> >>> >> > req >> >>> >>> >>> >> > >> >>> >>> >>> >> > Because of the order of the && operators, it will >> >>> >>> >>> >> > only >> >>> >>> >>> >> > evaluate >> >>> >>> >>> >> > fromJust >> >>> >>> >>> >> > after it makes sure that the argument isJust. >> >>> >>> >>> >> > That >> >>> >>> >>> >> > function >> >>> >>> >>> >> > in >> >>> >>> >>> >> > particular >> >>> >>> >>> >> > shouldn't throw any exceptions - it should only >> >>> >>> >>> >> > return >> >>> >>> >>> >> > Nothing. >> >>> >>> >>> >> > >> >>> >>> >>> >> > Knowing that, I don't quite think I understand >> >>> >>> >>> >> > what >> >>> >>> >>> >> > your >> >>> >>> >>> >> > concern >> >>> >>> >>> >> > is. >> >>> >>> >>> >> > Can >> >>> >>> >>> >> > you >> >>> >>> >>> >> > elaborate? >> >>> >>> >>> >> >> >>> >>> >>> >> You're right, but I had to squint really hard to >> >>> >>> >>> >> prove >> >>> >>> >>> >> to >> >>> >>> >>> >> myself >> >>> >>> >>> >> that >> >>> >>> >>> >> you're right. That's the kind of code that could >> >>> >>> >>> >> easily >> >>> >>> >>> >> be >> >>> >>> >>> >> broken >> >>> >>> >>> >> in >> >>> >>> >>> >> future updates by an unwitting maintainer (e.g., >> >>> >>> >>> >> me). >> >>> >>> >>> >> To >> >>> >>> >>> >> protect >> >>> >>> >>> >> the >> >>> >>> >>> >> world from me, I'd prefer if the code didn't have >> >>> >>> >>> >> the >> >>> >>> >>> >> fromJust. >> >>> >>> >>> >> This >> >>> >>> >>> >> might be a good place to leverage the Monad >> >>> >>> >>> >> instance of >> >>> >>> >>> >> Maybe. >> >>> >>> >>> >> >> >>> >>> >>> >> Michael >> >>> >>> >>> > >> >>> >>> >>> > >> >>> >>> >> >> >>> >>> >> >> >>> >>> > >> >>> > >> >>> > >> >> >> >> >> > > >

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Thanks for the help, everyone. The browser is coming along nicely :] On Tue, Feb 7, 2012 at 10:05 PM, Michael Snoyman wrote:

On Wed, Feb 8, 2012 at 6:28 AM, Myles C. Maxfield wrote:

...

I have been looking around at possibly making a Browser module for Network.HTTP.Conduit on top of Control.Monad.State. I came across this roadbump:

In order to implement redirection following, client code must call 'http' with a redirection count of 0. If there is a redirect, 'http' will throw an exception. In order to catch this exception and continue on with the

Actually, this is just a setting: you can override with checkStatus[1].

[1] http://hackage.haskell.org/packages/archive/http-conduit/1.2.4/doc/html/Netw...

...

redirection chain, the 'catch' function must be called. The problem is that the 'catch' function has a type of (catch :: Exception e => IO a -> (e -> IO a) -> IO a) which means that it can only be used in the IO monad. A call to 'http' inside the first argument of 'catch' must be wrapped in a 'runResourceT'

This has a couple implications:

The catch function cannot yield a Source, because any connections which 'http' opened will have to be immediately closed as soon as control reaches the 'catch' function (because of the runResourceT). The only way around this is to put the bind inside the catch block with some default sink, but this loses all of the constant-memory benefits of using a conduit in the first place. The function that will be calling 'catch' operates as a State monad. Cookie state must be updated both before and after making the request (both inside the 'catch' block and outside). I could pass around an updated cookie jar out of the IO monad as an extra entry in a tuple, but this defeats the purpose of using a State monad in the first place. This kind of programming is very ugly.

I see two solutions to these problems:

Make 'http' not throw a StatusCodeException if the _original_ redirection count = 0 Make my internal module call the un-exported 'httpRaw' function. This has the problem that it doen't solve the problem for anyone else trying to use http-conduit; it only solves the problem for me. Perhaps this could be alleviated by exporting httpRaw.

What do you think? --Myles

On Tue, Feb 7, 2012 at 12:35 AM, Myles C. Maxfield wrote:

...

Alright. I'll issue another pull request to you when it's done (expect

...

a couple weeks).

Thanks for your input so far, Aristid and Michael.

@Chris Wong: Do you want to talk about the Suffix List stuff some time?

--Myles

On Mon, Feb 6, 2012 at 10:14 PM, Michael Snoyman wrote:

...

+1

On Mon, Feb 6, 2012 at 11:16 PM, Aristid Breitkreuz wrote:

...

I would say: if it adds no package dependencies, put it right in.

Aristid

Am 06.02.2012 22:09 schrieb "Myles C. Maxfield" :

...

After all these commits have been flying around, I have yet another question:

the 'HTTP' package defines Network.Browser which is a State monad which keeps state about a "browser" (i.e. a cookie jar, a proxy,

redirection

...

...

...

parameters, etc.) It would be pretty straightforward to implement

...

...

...

...

kind of functionality on top of http-conduit.

I was originally going to do it and release it as its own package, but it may be beneficial to add such a module to the existing http-conduit package. Should I add it in to the existing package, or release it as its own package?

--Myles

On Mon, Feb 6, 2012 at 12:15 AM, Michael Snoyman < michael@snoyman.com> wrote: > > Just an FYI for everyone: Myles sent an (incredibly thorough) pull > request to handle cookies: > > https://github.com/snoyberg/http-conduit/pull/13 > > Thanks! > > On Sun, Feb 5, 2012 at 8:20 AM, Myles C. Maxfield > wrote: > > 1. The spec defines a grammar for the attributes. They're in > > uppercase. > > 2. Yes - 1.3 is the first version that lists DiffTime as an > > instance of > > RealFrac (so I can use the 'floor' function to pull out the number > > of > > seconds to render it) > > 3. I'll see what I can do. > > > > --Myles > > > > > > On Sat, Feb 4, 2012 at 9:06 PM, Michael Snoyman > > > > wrote: > >> > >> Looks good, a few questions/requests: > >> > >> 1. Is there a reason to upper-case all the attributes? > >> 2. Is the time >= 1.3 a requirements? Because that can cause a lot > >> of > >> trouble for people. > >> 3. Can you send the patch as a Github pull request? It's easier to > >> track that way. > >> > >> Michael > >> > >> On Sat, Feb 4, 2012 at 1:21 AM, Myles C. Maxfield > >> wrote: > >> > Here is the patch to Web.Cookie. I didn't modify the tests at > >> > all > >> > because > >> > they were already broken - they looked like they hadn't been > >> > updated > >> > since > >> > SetCookie only had 5 parameters. I did verify by hand that the > >> > patch > >> > works, > >> > though. > >> > > >> > Thanks, > >> > Myles > >> > > >> > > >> > On Thu, Feb 2, 2012 at 11:26 PM, Myles C. Maxfield > >> > wrote: > >> >> > >> >> Alright, I'll make a small patch that adds 2 fields to > >> >> SetCookie: > >> >> setCookieMaxAge :: Maybe DiffTime > >> >> setCookieSecureOnly :: Bool > >> >> > >> >> I've also gotten started on those cookie functions. I'm > >> >> currently > >> >> writing > >> >> tests for them. > >> >> > >> >> @Chris: The best advice I can give is that Chrome (what I'm > >> >> using > >> >> as a > >> >> source on all this) has the data baked into a .cc file. > >> >> However, > >> >> they > >> >> have > >> >> directions in a README and a script which will parse the list > >> >> and > >> >> generate > >> >> that source file. I recommend doing this. That way, the Haskell > >> >> module > >> >> would > >> >> have 2 source files: one file that reads the list and generates > >> >> the > >> >> second > >> >> file, which is a very large source file that contains each > >> >> element > >> >> in > >> >> the > >> >> list. The list should export `elem`-type queries. I'm not quite > >> >> sure > >> >> how to > >> >> handle wildcards that appear in the list - that part is up to > >> >> you. > >> >> Thanks > >> >> for helping out with this :] > >> >> > >> >> --Myles > >> >> > >> >> > >> >> On Thu, Feb 2, 2012 at 10:53 PM, Michael Snoyman > >> >> > >> >> wrote: > >> >>> > >> >>> Looks good to me too. I agree with Aristid: let's make the > >> >>> change > >> >>> to > >> >>> cookie itself. Do you want to send a pull request? I'm also > >> >>> considering making the SetCookie constructor hidden like we > >> >>> have > >> >>> for > >> >>> Request, so that if in the future we realize we need to add > >> >>> some > >> >>> other > >> >>> settings, it doesn't break the API. > >> >>> > >> >>> Chris: I would recommend compiling it into the module. Best > >> >>> bet > >> >>> would > >> >>> likely being converting the source file to Haskell source. > >> >>> > >> >>> Michael > >> >>> > >> >>> On Fri, Feb 3, 2012 at 6:32 AM, Myles C. Maxfield > >> >>> wrote: > >> >>> > Alright. After reading the spec, I have these questions / > >> >>> > concerns: > >> >>> > > >> >>> > The spec supports the "Max-Age" cookie attribute, which > >> >>> > Web.Cookies > >> >>> > doesn't. > >> >>> > > >> >>> > I see two possible solutions to this. The first is to have > >> >>> > parseSetCookie > >> >>> > take a UTCTime as an argument which will represent the > >> >>> > current > >> >>> > time > >> >>> > so > >> >>> > it > >> >>> > can populate the setCookieExpires field by adding the > >> >>> > Max-Age > >> >>> > attribute > >> >>> > to > >> >>> > the current time. Alternatively, that function can return an > >> >>> > IO > >> >>> > SetCookie so > >> >>> > it can ask for the current time by itself (which I think is > >> >>> > inferior > >> >>> > to > >> >>> > taking the current time as an argument). Note that the spec > >> >>> > says > >> >>> > to > >> >>> > prefer > >> >>> > Max-Age over Expires. > >> >>> > Add a field to SetCookie of type Maybe DiffTime which > >> >>> > represents > >> >>> > the > >> >>> > Max-Age > >> >>> > attribute > >> >>> > > >> >>> > Cookie code should be aware of the Public Suffix List as a > >> >>> > part > >> >>> > of > >> >>> > its > >> >>> > domain verification. The cookie code only needs to be able > >> >>> > to > >> >>> > tell > >> >>> > if a > >> >>> > specific string is in the list (W.Ascii -> Bool) > >> >>> > > >> >>> > I propose making an entirely unrelated package, > >> >>> > public-suffix-list, > >> >>> > with a > >> >>> > module Network.PublicSuffixList, which will expose this > >> >>> > function, as > >> >>> > well as > >> >>> > functions about parsing the list itself. Thoughts? > >> >>> > > >> >>> > Web.Cookie doesn't have a "secure-only" attribute. Adding > >> >>> > one in > >> >>> > is > >> >>> > straightforward enough. > >> >>> > The spec describes cookies as a property of HTTP, not of

...

...

...

...

> >> >>> > World > >> >>> > Wide > >> >>> > Web. > >> >>> > Perhaps "Web.Cookie" should be renamed? Just a thought; it > >> >>> > doesn't > >> >>> > really > >> >>> > matter to me. > >> >>> > > >> >>> > As for Network.HTTP.Conduit.Cookie, the spec describes in > >> >>> > section > >> >>> > 5.3 > >> >>> > "Storage Model" what fields a Cookie has. Here is my > >> >>> > proposal > >> >>> > for > >> >>> > the > >> >>> > functions it will expose: > >> >>> > > >> >>> > receiveSetCookie :: SetCookie -> Req.Request m -> UTCTime -> > >> >>> > Bool -> > >> >>> > CookieJar -> CookieJar > >> >>> > > >> >>> > Runs the algorithm described in section 5.3 "Storage Model" > >> >>> > The UTCTime is the current-time, the Bool is whether or not > >> >>> > the > >> >>> > caller > >> >>> > is an > >> >>> > HTTP-based API (as opposed to JavaScript or anything else) > >> >>> > > >> >>> > updateCookieJar :: Res.Response a -> Req.Request m -> > >> >>> > UTCTime -> > >> >>> > CookieJar > >> >>> > -> (CookieJar, Res.Response a) > >> >>> > > >> >>> > Applies "receiveSetCookie" to a Response. The output > >> >>> > CookieJar > >> >>> > is > >> >>> > stripped > >> >>> > of any Set-Cookie headers. > >> >>> > Specifies "True" for the Bool in receiveSetCookie > >> >>> > > >> >>> > computeCookieString :: Req.Request m -> CookieJar -> UTCTime > >> >>> > -> > >> >>> > Bool > >> >>> > -> > >> >>> > (W.Ascii, CookieJar) > >> >>> > > >> >>> > Runs the algorithm described in section 5.4 "The Cookie > >> >>> > Header" > >> >>> > The UTCTime and Bool are the same as in receiveSetCookie > >> >>> > > >> >>> > insertCookiesIntoRequest :: Req.Request m -> CookieJar -> > >> >>> > UTCTime -> > >> >>> > (Req.Request m, CookieJar) > >> >>> > > >> >>> > Applies "computeCookieString" to a Request. The output > >> >>> > cookie > >> >>> > jar > >> >>> > has > >> >>> > updated last-accessed-times. > >> >>> > Specifies "True" for the Bool in computeCookieString > >> >>> > > >> >>> > evictExpiredCookies :: CookieJar -> UTCTime -> CookieJar > >> >>> > > >> >>> > Runs the algorithm described in the last part of section 5.3 > >> >>> > "Storage > >> >>> > Model" > >> >>> > > >> >>> > This will make the relevant part of 'http' look like: > >> >>> > > >> >>> > go count req'' cookie_jar'' = do > >> >>> > now <- liftIO $ getCurrentTime > >> >>> > let (req', cookie_jar') = insertCookiesIntoRequest > >> >>> > req'' > >> >>> > (evictExpiredCookies cookie_jar'' now) now > >> >>> > res' <- httpRaw req' manager > >> >>> > let (cookie_jar, res) = updateCookieJar res' req' > >> >>> > now > >> >>> > cookie_jar' > >> >>> > case getRedirectedRequest req' (responseHeaders res) > >> >>> > (W.statusCode > >> >>> > (statusCode res)) of > >> >>> > Just req -> go (count - 1) req cookie_jar > >> >>> > Nothing -> return res > >> >>> > > >> >>> > I plan to not allow for a user-supplied cookieFilter > >> >>> > function. > >> >>> > If > >> >>> > they > >> >>> > want > >> >>> > that functionality, they can re-implement the > >> >>> > redirection-following > >> >>> > logic. > >> >>> > > >> >>> > Any thoughts on any of this? > >> >>> > > >> >>> > Thanks, > >> >>> > Myles > >> >>> > > >> >>> > On Wed, Feb 1, 2012 at 5:19 PM, Myles C. Maxfield > >> >>> > > >> >>> > wrote: > >> >>> >> > >> >>> >> Nope. I'm not. The RFC is very explicit about how to handle > >> >>> >> cookies. > >> >>> >> As > >> >>> >> soon as I'm finished making sense of it (in terms of > >> >>> >> Haskell) > >> >>> >> I'll > >> >>> >> send > >> >>> >> another proposal email. > >> >>> >> > >> >>> >> On Feb 1, 2012 3:25 AM, "Michael Snoyman" > >> >>> >> > >> >>> >> wrote: > >> >>> >>> > >> >>> >>> You mean you're *not* making this proposal? > >> >>> >>> > >> >>> >>> On Wed, Feb 1, 2012 at 7:30 AM, Myles C. Maxfield > >> >>> >>> wrote: > >> >>> >>> > Well, this is embarrassing. Please disregard my

...

...

...

...

> >> >>> >>> > email. > >> >>> >>> > I > >> >>> >>> > should > >> >>> >>> > learn to read the RFC *before* submitting proposals. > >> >>> >>> > > >> >>> >>> > --Myles > >> >>> >>> > > >> >>> >>> > > >> >>> >>> > On Tue, Jan 31, 2012 at 6:37 PM, Myles C. Maxfield > >> >>> >>> > wrote: > >> >>> >>> >> > >> >>> >>> >> Here are my initial ideas about supporting cookies. > >> >>> >>> >> Note > >> >>> >>> >> that > >> >>> >>> >> I'm > >> >>> >>> >> using > >> >>> >>> >> Chrome for ideas since it's open source. > >> >>> >>> >> > >> >>> >>> >> Network/HTTP/Conduit/Cookies.hs file > >> >>> >>> >> Exporting the following symbols: > >> >>> >>> >> > >> >>> >>> >> type StuffedCookie = SetCookie > >> >>> >>> >> > >> >>> >>> >> A regular SetCookie can have Nothing for its Domain and > >> >>> >>> >> Path > >> >>> >>> >> attributes. A > >> >>> >>> >> StuffedCookie has to have these fields set. > >> >>> >>> >> > >> >>> >>> >> type CookieJar = [StuffedCookie] > >> >>> >>> >> > >> >>> >>> >> Chrome's cookie jar is implemented as (the C++ > >> >>> >>> >> equivalent > >> >>> >>> >> of) > >> >>> >>> >> Map > >> >>> >>> >> W.Ascii > >> >>> >>> >> StuffedCookie. The key is the "eTLD+1" of the domain, > >> >>> >>> >> so > >> >>> >>> >> lookups > >> >>> >>> >> for > >> >>> >>> >> all > >> >>> >>> >> cookies for a given domain are fast. > >> >>> >>> >> I think I'll stay with just a list of StuffedCookies > >> >>> >>> >> just > >> >>> >>> >> to > >> >>> >>> >> keep > >> >>> >>> >> it > >> >>> >>> >> simple. Perhaps a later revision can implement the > >> >>> >>> >> faster > >> >>> >>> >> map. > >> >>> >>> >> > >> >>> >>> >> getRelevantCookies :: Request m -> CookieJar -> UTCTime > >> >>> >>> >> -> > >> >>> >>> >> (CookieJar, > >> >>> >>> >> Cookies) > >> >>> >>> >> > >> >>> >>> >> Gets all the cookies from the cookie jar that should be > >> >>> >>> >> set > >> >>> >>> >> for > >> >>> >>> >> the > >> >>> >>> >> given > >> >>> >>> >> Request. > >> >>> >>> >> The time argument is whatever "now" is (it's pulled out > >> >>> >>> >> of > >> >>> >>> >> the > >> >>> >>> >> function so > >> >>> >>> >> the function can remain pure and easily testable) > >> >>> >>> >> The function will also remove expired cookies from

...

...

...

...

> >> >>> >>> >> cookie > >> >>> >>> >> jar > >> >>> >>> >> (given > >> >>> >>> >> what "now" is) and return the filtered cookie jar > >> >>> >>> >> > >> >>> >>> >> putRelevantCookies :: Request m -> CookieJar -> > >> >>> >>> >> [StuffedCookie] > >> >>> >>> >> -> > >> >>> >>> >> CookieJar > >> >>> >>> >> > >> >>> >>> >> Insert cookies from a server response into the cookie > >> >>> >>> >> jar. > >> >>> >>> >> The first argument is only used for checking to see > >> >>> >>> >> which > >> >>> >>> >> cookies > >> >>> >>> >> are > >> >>> >>> >> valid (which cookies match the requested domain, etc, > >> >>> >>> >> so > >> >>> >>> >> site1.com > >> >>> >>> >> can't set > >> >>> >>> >> a cookie for site2.com) > >> >>> >>> >> > >> >>> >>> >> stuffCookie :: Request m -> SetCookie -> StuffedCookie > >> >>> >>> >> > >> >>> >>> >> If the SetCookie's fields are Nothing, fill them in > >> >>> >>> >> given > >> >>> >>> >> the > >> >>> >>> >> Request > >> >>> >>> >> from > >> >>> >>> >> which it originated > >> >>> >>> >> > >> >>> >>> >> getCookies :: Response a -> ([SetCookie], Response a) > >> >>> >>> >> > >> >>> >>> >> Pull cookies out of a server response. Return the > >> >>> >>> >> response > >> >>> >>> >> with > >> >>> >>> >> the > >> >>> >>> >> Set-Cookie headers filtered out > >> >>> >>> >> > >> >>> >>> >> putCookies :: Request a -> Cookies -> Request a > >> >>> >>> >> > >> >>> >>> >> A wrapper around renderCookies. Inserts some cookies > >> >>> >>> >> into a > >> >>> >>> >> request. > >> >>> >>> >> Doesn't overwrite cookies that are already set in the > >> >>> >>> >> request > >> >>> >>> >> > >> >>> >>> >> These functions will be exported from > >> >>> >>> >> Network.HTTP.Conduit > >> >>> >>> >> as > >> >>> >>> >> well, so > >> >>> >>> >> callers can use them to re-implement redirection chains > >> >>> >>> >> I won't implement a cookie filtering function (like > >> >>> >>> >> what > >> >>> >>> >> Network.Browser > >> >>> >>> >> has) > >> >>> >>> >> > >> >>> >>> >> If you want to have arbitrary handling of cookies, > >> >>> >>> >> re-implement > >> >>> >>> >> redirection following. It's not very difficult if you > >> >>> >>> >> use > >> >>> >>> >> the > >> >>> >>> >> API > >> >>> >>> >> provided, > >> >>> >>> >> and the 'http' function is open source so you can use > >> >>> >>> >> that > >> >>> >>> >> as a > >> >>> >>> >> reference. > >> >>> >>> >> > >> >>> >>> >> I will implement the functions according to RFC 6265 > >> >>> >>> >> I will also need to write the following functions. > >> >>> >>> >> Should > >> >>> >>> >> they > >> >>> >>> >> also be > >> >>> >>> >> exported? > >> >>> >>> >> > >> >>> >>> >> canonicalizeDomain :: W.Ascii -> W.Ascii > >> >>> >>> >> > >> >>> >>> >> turns "..a.b.c..d.com..." to "a.b.c.d.com" > >> >>> >>> >> Technically necessary for domain matching (Chrome does > >> >>> >>> >> it) > >> >>> >>> >> Perhaps unnecessary for a first pass? Perhaps we can > >> >>> >>> >> trust > >> >>> >>> >> users > >> >>> >>> >> for > >> >>> >>> >> now? > >> >>> >>> >> > >> >>> >>> >> domainMatches :: W.Ascii -> W.Ascii -> Maybe W.Ascii > >> >>> >>> >> > >> >>> >>> >> Does the first domain match against the second domain? > >> >>> >>> >> If so, return the prefix of the first that isn't in

...

...

...

...

> >> >>> >>> >> second > >> >>> >>> >> > >> >>> >>> >> pathMatches :: W.Ascii -> W.Ascii -> Bool > >> >>> >>> >> > >> >>> >>> >> Do the paths match? > >> >>> >>> >> > >> >>> >>> >> In order to implement domain matching, I have to have > >> >>> >>> >> knowledge > >> >>> >>> >> of > >> >>> >>> >> the Public Suffix List so I know that > >> >>> >>> >> sub1.sub2.pvt.k12.wy.us > >> >>> >>> >> can > >> >>> >>> >> set > >> >>> >>> >> a > >> >>> >>> >> cookie for sub2.pvt.k12.wy.us but not for k12.wy.us > >> >>> >>> >> (because > >> >>> >>> >> pvt.k12.wy.us > >> >>> >>> >> is a "suffix"). There are a variety of ways to > >> >>> >>> >> implement > >> >>> >>> >> this. > >> >>> >>> >> > >> >>> >>> >> As far as I can tell, Chrome does it by using a

in this the previous the the script

...

...

...

...

> >> >>> >>> >> (which a > >> >>> >>> >> human > >> >>> >>> >> periodically runs) which parses the list at creates a > >> >>> >>> >> .cc > >> >>> >>> >> file > >> >>> >>> >> that is > >> >>> >>> >> included in the build. > >> >>> >>> >> > >> >>> >>> >> I might be wrong about the execution of the script; it > >> >>> >>> >> might be > >> >>> >>> >> a > >> >>> >>> >> build > >> >>> >>> >> step. If it is a build step, however, it is suspicious > >> >>> >>> >> that > >> >>> >>> >> a > >> >>> >>> >> build > >> >>> >>> >> target > >> >>> >>> >> would try to download a file... > >> >>> >>> >> > >> >>> >>> >> Any more elegant ideas? > >> >>> >>> >> > >> >>> >>> >> Feedback on any/all of the above would be very helpful > >> >>> >>> >> before I > >> >>> >>> >> go > >> >>> >>> >> off > >> >>> >>> >> into the weeds on this project. > >> >>> >>> >> > >> >>> >>> >> Thanks, > >> >>> >>> >> Myles C. Maxfield > >> >>> >>> >> > >> >>> >>> >> On Sat, Jan 28, 2012 at 8:17 PM, Michael Snoyman > >> >>> >>> >> > >> >>> >>> >> wrote: > >> >>> >>> >>> > >> >>> >>> >>> Thanks, looks great! I've merged it into the Github > >> >>> >>> >>> tree. > >> >>> >>> >>> > >> >>> >>> >>> On Sat, Jan 28, 2012 at 8:36 PM, Myles C. Maxfield > >> >>> >>> >>> wrote: > >> >>> >>> >>> > Ah, yes, you're completely right. I completely agree > >> >>> >>> >>> > that > >> >>> >>> >>> > moving > >> >>> >>> >>> > the > >> >>> >>> >>> > function into the Maybe monad increases readability. > >> >>> >>> >>> > This > >> >>> >>> >>> > kind > >> >>> >>> >>> > of > >> >>> >>> >>> > function > >> >>> >>> >>> > is what the Maybe monad was designed for. > >> >>> >>> >>> > > >> >>> >>> >>> > Here is a revised patch. > >> >>> >>> >>> > > >> >>> >>> >>> > > >> >>> >>> >>> > On Sat, Jan 28, 2012 at 8:28 AM, Michael Snoyman > >> >>> >>> >>> > > >> >>> >>> >>> > wrote: > >> >>> >>> >>> >> > >> >>> >>> >>> >> On Sat, Jan 28, 2012 at 1:20 AM, Myles C. Maxfield > >> >>> >>> >>> >> wrote: > >> >>> >>> >>> >> > the fromJust should never fail, beceause of the > >> >>> >>> >>> >> > guard > >> >>> >>> >>> >> > statement: > >> >>> >>> >>> >> > > >> >>> >>> >>> >> > | 300 <= code && code < 400 && isJust l'' && > >> >>> >>> >>> >> > isJust > >> >>> >>> >>> >> > l' = > >> >>> >>> >>> >> > Just $ > >> >>> >>> >>> >> > req > >> >>> >>> >>> >> > > >> >>> >>> >>> >> > Because of the order of the && operators, it will > >> >>> >>> >>> >> > only > >> >>> >>> >>> >> > evaluate > >> >>> >>> >>> >> > fromJust > >> >>> >>> >>> >> > after it makes sure that the argument isJust. > >> >>> >>> >>> >> > That > >> >>> >>> >>> >> > function > >> >>> >>> >>> >> > in > >> >>> >>> >>> >> > particular > >> >>> >>> >>> >> > shouldn't throw any exceptions - it should only > >> >>> >>> >>> >> > return > >> >>> >>> >>> >> > Nothing. > >> >>> >>> >>> >> > > >> >>> >>> >>> >> > Knowing that, I don't quite think I understand > >> >>> >>> >>> >> > what > >> >>> >>> >>> >> > your > >> >>> >>> >>> >> > concern > >> >>> >>> >>> >> > is. > >> >>> >>> >>> >> > Can > >> >>> >>> >>> >> > you > >> >>> >>> >>> >> > elaborate? > >> >>> >>> >>> >> > >> >>> >>> >>> >> You're right, but I had to squint really hard to > >> >>> >>> >>> >> prove > >> >>> >>> >>> >> to > >> >>> >>> >>> >> myself > >> >>> >>> >>> >> that > >> >>> >>> >>> >> you're right. That's the kind of code that could > >> >>> >>> >>> >> easily > >> >>> >>> >>> >> be > >> >>> >>> >>> >> broken > >> >>> >>> >>> >> in > >> >>> >>> >>> >> future updates by an unwitting maintainer (e.g., > >> >>> >>> >>> >> me). > >> >>> >>> >>> >> To > >> >>> >>> >>> >> protect > >> >>> >>> >>> >> the > >> >>> >>> >>> >> world from me, I'd prefer if the code didn't have > >> >>> >>> >>> >> the > >> >>> >>> >>> >> fromJust. > >> >>> >>> >>> >> This > >> >>> >>> >>> >> might be a good place to leverage the Monad > >> >>> >>> >>> >> instance of > >> >>> >>> >>> >> Maybe. > >> >>> >>> >>> >> > >> >>> >>> >>> >> Michael > >> >>> >>> >>> > > >> >>> >>> >>> > > >> >>> >>> >> > >> >>> >>> >> > >> >>> >>> > > >> >>> > > >> >>> > > >> >> > >> >> > >> > > > > >

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Yeah, a more combinatorial approach to making HTTP requests would be good. So +1 for checkRedirect or anything similar. 2012/1/24 Michael Snoyman :

On Tue, Jan 24, 2012 at 6:57 PM, Myles C. Maxfield wrote:

...

On Mon, Jan 23, 2012 at 10:43 PM, Michael Snoyman wrote:

...

On Tue, Jan 24, 2012 at 8:37 AM, Myles C. Maxfield wrote:

...

I have attached a patch to add a redirect chain to the Response datatype. Comments on this patch are very welcome.

I thought that this isn't necessary since a client wanting to track all the redirects could just handle them manually by setting the redirect count to 0.

It seems like a lot of work to re-implement the redirection-following code, just to know which URL the bytes are coming from.  I feel that adding this field makes the library easier to use, but it's your call.

If that's the concern, I'd much rather just expose a function to help with dealing with redirects, rather than sticking a rather arbitrary [Ascii] in everyone's Response. I think a function along the lines of:

checkRedirect :: Response -> Maybe Request

would fit the bill, and could be extracted from the current `http` function.

Michael

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe