Hello, I'd like to download 1,000 web pages with up to 6 six concurrent downloads at a time. How can I express such a thread limit within the orc EDSL? Günther
Excerpts from Günther Schmidt's message of Fri Jul 30 16:16:38 -0400 2010:
I'd like to download 1,000 web pages with up to 6 six concurrent downloads at a time.
How can I express such a thread limit within the orc EDSL?
One solution that comes to mind is place all 1000 web pages in an MVar containing a queue of URLs to process (a list will probably suffice), and then use Orc to orchestrate six threads that pull a page from the queue and make a download. Admittedly, Orc doesn't buy you very much in this scenario until you add timeout handling and such. Cheers, Edward
Dear Edward, I hope that there is a more orc integrated solution. I think the scenario I described here is quite common. Günther Am 31.07.10 00:16, schrieb Edward Z. Yang:
Excerpts from Günther Schmidt's message of Fri Jul 30 16:16:38 -0400 2010:
I'd like to download 1,000 web pages with up to 6 six concurrent downloads at a time.
How can I express such a thread limit within the orc EDSL?
One solution that comes to mind is place all 1000 web pages in an MVar containing a queue of URLs to process (a list will probably suffice), and then use Orc to orchestrate six threads that pull a page from the queue and make a download. Admittedly, Orc doesn't buy you very much in this scenario until you add timeout handling and such.
Cheers, Edward
participants (2)
-
Edward Z. Yang -
Günther Schmidt