Network.Download stops after 10 downloads
I have tried the Network.Download library in order to download CSV files from Yahoo Finance, but after the 10th download every next download fails. Does Network.Download keep connections open? I did consume the result of the openURI function (even with deepseq). Any ideas? Thanks in advance! Stéphane
On Mon, Feb 6, 2017 at 8:49 PM, Stéphane Thibaud <snthibaud@gmail.com> wrote:
I have tried the Network.Download library in order to download CSV files from Yahoo Finance, but after the 10th download every next download fails. Does Network.Download keep connections open? I did consume the result of the openURI function (even with deepseq). Any ideas?
Can you get any details about why/how it fails? It could well be something as simple as Yahoo throttling you for too many connections in too short a time or something; in that case, the error from Yahoo should indicate it, but if all you get back is a 4xx or 5xx code then it won't be obvious. (Which may well mean something like running it under strace with +RTS -V0 to see the actual error returned.) -- brandon s allbery kf8nh sine nomine associates allbery.b@gmail.com ballbery@sinenomine.net unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net
Thank you for your answer! I get 200 OK status codes (I think after I get one 404 first) and it does seem like some (pieces of) data are sent (as tcpflow was printing data when the requests were failing - even when I started tcpflow after the failing requests started). Network.Download gives back the following error: Left "Failed to connect: Not Found" I have attached my code. Best regards, Stéphane 2017年2月7日(火) 10:53 Brandon Allbery <allbery.b@gmail.com>:
On Mon, Feb 6, 2017 at 8:49 PM, Stéphane Thibaud <snthibaud@gmail.com> wrote:
I have tried the Network.Download library in order to download CSV files from Yahoo Finance, but after the 10th download every next download fails. Does Network.Download keep connections open? I did consume the result of the openURI function (even with deepseq). Any ideas?
Can you get any details about why/how it fails? It could well be something as simple as Yahoo throttling you for too many connections in too short a time or something; in that case, the error from Yahoo should indicate it, but if all you get back is a 4xx or 5xx code then it won't be obvious. (Which may well mean something like running it under strace with +RTS -V0 to see the actual error returned.)
-- brandon s allbery kf8nh sine nomine associates allbery.b@gmail.com ballbery@sinenomine.net unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net
I tried the same thing with the Network.Download.Curl library and that worked! I gueas something was wrong with the Network.Download library. Best regards, Stéphane 2017年2月8日(水) 11:33 Stéphane Thibaud <snthibaud@gmail.com>:
Thank you for your answer! I get 200 OK status codes (I think after I get one 404 first) and it does seem like some (pieces of) data are sent (as tcpflow was printing data when the requests were failing - even when I started tcpflow after the failing requests started). Network.Download gives back the following error: Left "Failed to connect: Not Found" I have attached my code.
Best regards,
Stéphane
2017年2月7日(火) 10:53 Brandon Allbery <allbery.b@gmail.com>:
On Mon, Feb 6, 2017 at 8:49 PM, Stéphane Thibaud <snthibaud@gmail.com> wrote:
I have tried the Network.Download library in order to download CSV files from Yahoo Finance, but after the 10th download every next download fails. Does Network.Download keep connections open? I did consume the result of the openURI function (even with deepseq). Any ideas?
Can you get any details about why/how it fails? It could well be something as simple as Yahoo throttling you for too many connections in too short a time or something; in that case, the error from Yahoo should indicate it, but if all you get back is a 4xx or 5xx code then it won't be obvious. (Which may well mean something like running it under strace with +RTS -V0 to see the actual error returned.)
-- brandon s allbery kf8nh sine nomine associates allbery.b@gmail.com ballbery@sinenomine.net unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net
participants (2)
-
Brandon Allbery -
Stéphane Thibaud