michael:

2010/11/20 José Romildo Malaquias :

...

In order to download a given web page, I wrote the attached program. The problem is that the page is not being full downloaded. It is being somehow intettupted.

Any clues on how to solve this problem?

My guess is that there's a character encoding issue. Another approach would be using the http-enumerator package[1]. The equivalent program is:

module Main where

import Network.HTTP.Enumerator (simpleHttp) import qualified Data.ByteString.Lazy as L

main = do src <- simpleHttp "http://www.adorocinema.com/common/search/search_by_film/?criteria=Bourne" L.writeFile "test.html" src L.putStrLn src

FWIW, with this url, I get the same problem using the Curl package (via the download-curl): import Network.Curl.Download import qualified Data.ByteString as B main = do edoc <- openURI "http://www.adorocinema.com/common/search/search_by_film/?criteria=Bourne" case edoc of Left err -> print err Right doc -> B.writeFile "test.html" doc Not a problem on e.g. http://haskell.org -- Don

On Sat, Nov 20, 2010 at 10:26:49PM +0100, Daniel Fischer wrote:

On Saturday 20 November 2010 21:47:52, Don Stewart wrote:

...

...

2010/11/20 José Romildo Malaquias :

...

In order to download a given web page, I wrote the attached program. The problem is that the page is not being full downloaded. It is being somehow intettupted.

Any clues on how to solve this problem?

FWIW, with this url, I get the same problem using the Curl package

Just for the record, wget also gets a truncated (at the same point) file, so it's not a Haskell problem.

Web browsers like Firefox and Opera does not seem to have the same problem with this web page. I would like to be able to download this page from Haskell. Romildo

On Nov 20, 2010, at 5:10 PM, Yitzchak Gale wrote:

José Romildo Malaquias wrote:

...

Web browsers like Firefox and Opera does not seem to have the same problem with this web page. I would like to be able to download this page from Haskell.

Hi Romildo,

This web page serves the head, including a lot of JavaScript, and the first few hundred bytes of the body, then pauses. That causes web browsers to begin loading and executing the JavaScript. Apparently, the site only continues serving the rest of the page if the JavaScript is actually loaded and executed. If not, it aborts.

Actually, I think it's just a misconfigured proxy. The curl executable fails, at the same point, but a curl --compressed call succeeds. The curl bindings don't allow you to automatically get and decompress gzip data, so you could either set the accept: gzip header yourself, then pipe the output through the appropriate decompression routine, or, more simply, just get the page via using System.Process to drive the curl binary directly. Cheers, Sterl

Albert Y. C. Lai wrote:

...truncates when the web server chooses the identity encoding The server chooses identity when your request's Accept-Encoding field specifies identity or simply your request has no Accept-Encoding field

Excellent work!

My methodology of discovering and confirming this is a great lesson in the triumph of the scientific methodology (over the prevailing opinionative methodology, for example).

Haha, indeed!

Actually the truncation strongly suggests that javascript is not involved: the truncation ends with an incomplete end-tag "

Well, no, the theory was that the server sends some random number of bytes from the body to ensure that the browser starts loading the scripts in the head. So it could stop anywhere. In the end, I think you didn't really need the W3C validator. You also could have triangulated on the headers sent by your own browser. So, there you have it, folks. The Haskell community debugs a broken web server, without being asked, and without access to the server.