How to make reading an array from disk more efficient
Hi, I have some code (http://www.acooke.org/andrew/ReadTest.hs) that reads data from a file (an image in ppm format; example data (256*256 pixels) at http://www.acooke.org/andrew/test.ppm) and stores it in an array of Word8 values. The aim is to read a file that contains 5000 * 5000 * 3 Word8 values. I think this should take under 100Mb of memory, if the Array implementation is efficient. However, when I run the code on a file of that size it looks like it will need several days to complete. This seems rather slow - the GIMP can read the same file maybe 30 seconds). How do I make the Haskell code faster while keeping it flexible? My requirements are: - machine limited to 1Gb memory - display "status bar" - the possibility to filter the pixel stream so that the image is subsampled (see "everyN" in the code) - the possibility to filter the pixel strean so that a subsection of the image is selected. All that is possible in the code I have now, but it's slow. Thanks, Andrew -- personal web site: http://www.acooke.org/andrew personal mail list: http://www.acooke.org/andrew/compute.html
(1) use unboxed arrays, otherwise you're wasting too much space with pointers. that is, unless you need laziness on the elements, which i don't think you do based on your list (2) (maybe) use imperative arrays; this will help you ensure that everything is being evaluated quickly. On Wed, 24 Dec 2003, andrew cooke wrote:
Hi,
I have some code (http://www.acooke.org/andrew/ReadTest.hs) that reads data from a file (an image in ppm format; example data (256*256 pixels) at http://www.acooke.org/andrew/test.ppm) and stores it in an array of Word8 values. The aim is to read a file that contains 5000 * 5000 * 3 Word8 values. I think this should take under 100Mb of memory, if the Array implementation is efficient. However, when I run the code on a file of that size it looks like it will need several days to complete. This seems rather slow - the GIMP can read the same file maybe 30 seconds).
How do I make the Haskell code faster while keeping it flexible? My requirements are:
- machine limited to 1Gb memory - display "status bar" - the possibility to filter the pixel stream so that the image is subsampled (see "everyN" in the code) - the possibility to filter the pixel strean so that a subsection of the image is selected.
All that is possible in the code I have now, but it's slow.
Thanks, Andrew
-- Hal Daume III | hdaume@isi.edu "Arrest this man, he talks in maths." | www.isi.edu/~hdaume
Thanks. I should have added that I will only use the array for reading once it's created. I don't mind whether creating is lazy or eager (it's currently eager because I was fighting a space leak, but I think that was down to some other error). I don't fully understand how either of the suggestions you make will speed up creation, though (I guess allocating less memory is faster). I'm looking for a factor of tens of thousands improvement. But I will try what you suggest. Cheers, Andrew PS Thanks for the very speedy reply and thanks to whoever maintains the list archive for updating the archive for each message (I'm pretty sure it used to be made daily, which was a nuisance if you wanted to forward URLs to interesting discussions). Hal Daume III said:
(1) use unboxed arrays, otherwise you're wasting too much space with pointers. that is, unless you need laziness on the elements, which i don't think you do based on your list
(2) (maybe) use imperative arrays; this will help you ensure that everything is being evaluated quickly.
-- personal web site: http://www.acooke.org/andrew personal mail list: http://www.acooke.org/andrew/compute.html
one other thing you might find useful is to read it imperatively and then use unsafeFreezeArray (i think that's the name) to get a pure array out of it. since all you'll be doing is reading, this should work nicely for you. On Wed, 24 Dec 2003, andrew cooke wrote:
Thanks. I should have added that I will only use the array for reading once it's created. I don't mind whether creating is lazy or eager (it's currently eager because I was fighting a space leak, but I think that was down to some other error).
I don't fully understand how either of the suggestions you make will speed up creation, though (I guess allocating less memory is faster). I'm looking for a factor of tens of thousands improvement. But I will try what you suggest.
Cheers, Andrew
PS Thanks for the very speedy reply and thanks to whoever maintains the list archive for updating the archive for each message (I'm pretty sure it used to be made daily, which was a nuisance if you wanted to forward URLs to interesting discussions).
Hal Daume III said:
(1) use unboxed arrays, otherwise you're wasting too much space with pointers. that is, unless you need laziness on the elements, which i don't think you do based on your list
(2) (maybe) use imperative arrays; this will help you ensure that everything is being evaluated quickly.
-- Hal Daume III | hdaume@isi.edu "Arrest this man, he talks in maths." | www.isi.edu/~hdaume
On Wed, 24 Dec 2003, Hal Daume III wrote:
one other thing you might find useful is to read it imperatively and then use unsafeFreezeArray (i think that's the name) to get a pure array out of it. since all you'll be doing is reading, this should work nicely for you.
I must say, this sort of reply is EXTREMELY DISTURBING. It is ridiculous that one needs to get so intimate with implementation issues just to read a file! Is this inherent in Haskell or is it just that no one has bothered to create a the Haskell equivalent of BufferedReader? <http://java.sun.com/j2se/1.4.2/docs/api/java/io/BufferedReader.html> And if so, is there something about Haskell that makes a Haskell implementation of such a lib difficult? -Alex- _________________________________________________________________ S. Alexander Jacobson mailto:me@alexjacobson.com tel:917-770-6565 http://alexjacobson.com
On Wed, 24 Dec 2003, andrew cooke wrote:
Thanks. I should have added that I will only use the array for reading once it's created. I don't mind whether creating is lazy or eager (it's currently eager because I was fighting a space leak, but I think that was down to some other error).
I don't fully understand how either of the suggestions you make will speed up creation, though (I guess allocating less memory is faster). I'm looking for a factor of tens of thousands improvement. But I will try what you suggest.
Cheers, Andrew
PS Thanks for the very speedy reply and thanks to whoever maintains the list archive for updating the archive for each message (I'm pretty sure it used to be made daily, which was a nuisance if you wanted to forward URLs to interesting discussions).
Hal Daume III said:
(1) use unboxed arrays, otherwise you're wasting too much space with pointers. that is, unless you need laziness on the elements, which i don't think you do based on your list
(2) (maybe) use imperative arrays; this will help you ensure that everything is being evaluated quickly.
-- Hal Daume III | hdaume@isi.edu "Arrest this man, he talks in maths." | www.isi.edu/~hdaume
_______________________________________________ Haskell mailing list Haskell@haskell.org http://www.haskell.org/mailman/listinfo/haskell
On Thu, Dec 25, 2003 at 02:53:00PM -0500, S. Alexander Jacobson wrote:
I must say, this sort of reply is EXTREMELY DISTURBING.
Why are you shouting? I find your reply EXTREMELY UNHELPFUL.
It is ridiculous that one needs to get so intimate with implementation issues just to read a file!
Its quite easy to read a file, but it can be difficult to handle big files efficiently. Been there, done that, been satisfied with the results.
Is this inherent in Haskell or is it just that no one has bothered to create a the Haskell equivalent of BufferedReader? <http://java.sun.com/j2se/1.4.2/docs/api/java/io/BufferedReader.html>
Handles are buffered by default. It's not the case here. There are other factors involved (eg. laziness, ineffective file format).
And if so, is there something about Haskell that makes a Haskell implementation of such a lib difficult?
Not really. Tom. -- .signature: Too many levels of symbolic links
participants (4)
-
andrew cooke -
Hal Daume III -
S. Alexander Jacobson -
Tomasz Zielonka