Re: ETag support broken on hackage.haskell.org

Hi, I spent some time looking into this and was able to get etags checking to work, see [1] [1] https://github.com/haskell/hackage-server/pull/236 However, my approach only deals with etags only for the index tarball. I poked around for quite a while seeing if there is a more general solution, but could not find anything useful. The thing is, support for computing etags is hand-coded for the index tarball, so a more generic mechanism does not make much sense. Plus, servePackagesIndex is the only logical place to handle etags in the request, since further down we do not have access to the request and further up we do not have access to the tarball object which embeds the md5sum. On Tue, Jun 10, 2014 at 11:38 AM, Duncan Coutts < duncan.coutts@googlemail.com> wrote:

I'm pretty sure it's the hackage-server. I think we simply don't have code yet to react to the if-none-match. Happstack has some util for the time-based conditionals, but not the etag one. So it should just be a matter of us adding a combinator that both serves the etag and reacts to the if-none-match header. Then we'd go and use that in all the places where we currently serve etags. At the same time, it's probably worth making that caching combinator also take an optional timestamp and "valid for" time, so we can do all our caching things in one call in each resource handler.

On 10 June 2014 08:54, Johan Tibell wrote:

...

We need to figure out if nginx or the hackage-server is to blame here.

On Tue, Jun 10, 2014 at 1:34 AM, Gintautas Miliauskas wrote:

...

Hi,

first of all, a shout out to all ZuriHac2014 folks, thanks for the

gentle

...

introduction to the deeper trenches of Haskell infrastructure!

I spent a bit of time looking into why "cabal update" always redownloads the entire package list even though the code has explicit support for etags, and it looks like etags are broken on the server side. The server does serve "Etag" headers, but does not react to If-None-Match:

$ curl -I --header 'If-None-Match: "1ea6c9a3168b9c186d5bbafd065ce5ec"' hackage.haskell.org/packages/index.tar.gz HTTP/1.1 200 OK Server: nginx/1.6.0 Date: Mon, 09 Jun 2014 23:22:49 GMT Content-Type: application/x-gzip Content-Length: 7193838 Connection: keep-alive Content-MD5: 1ea6c9a3168b9c186d5bbafd065ce5ec ETag: "1ea6c9a3168b9c186d5bbafd065ce5ec" Last-modified: Mon, 09 Jun 2014 23:10:44 GMT

In this case the server is supposed to return a 304 code because the ETag matches, but it returns the usual 200.

Here's a well-behaved server for comparison:

$ curl -I --header 'If-None-Match: f89ef04efbcb1bbd413c9170523beeb1' http://upload.wikimedia.org/ HTTP/1.1 304 Not Modified Last-Modified: Thu, 03 Oct 2013 22:29:34 GMT Etag: f89ef04efbcb1bbd413c9170523beeb1 X-Timestamp: 1380839374.33426 X-Object-Meta-Mtime: 1366745378.0 Content-Type: text/html X-Varnish: 2552484147 2541352860, 2937266545 2853346814, 523731362 Via: 1.1 varnish, 1.1 varnish, 1.1 varnish Accept-Ranges: bytes Date: Mon, 09 Jun 2014 23:29:36 GMT Age: 38954 Connection: keep-alive X-Cache: cp1064 hit (336), cp3010 hit (1791), cp3007 frontend miss (0) Access-Control-Allow-Origin: * Access-Control-Expose-Headers: Age, Content-Length, Date, X-Cache, X-Varnish

Note the 304 return code.

I do not know about the configuration in use on the server, but something seems to be wrong there. There are some nginx bug reports [1] about similar issues, not sure if they are relevant here.

[1] https://github.com/rgrove/rawgit/issues/13

Best regards, -- Gintautas Miliauskas

_______________________________________________ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel

_______________________________________________ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel

-- Gintautas Miliauskas