Re: [Haskell-cafe] out of core computing in haskell
The main two use cases I have in mind are
1) really really really big abstract syntax trees or proof trees (a la compilers or theorem provers)
2) random access to numerical data
does that help clarify what i'm asking about? In each case, is there a standard way of dealing with this?
in the case of (1) the sensible way seems to me to do some sort of zipper representation which loads adjacent nodes to some depth surrounding your current location in the tree,
and in the case of (2) it seems like the sensible way would be load subsequences of the data into memory.
----- Original Message ----
From: Andrew Coppin
Hello Everyone, I'm not quite sure if I'm posing this question correctly, but what facilities currently exist in haskell to nicely deal with datastructures that won't fit within a given machine's ram? And if there are no such facilities, what would it take to fix that?
If you just want to process a big chunk of data from one end to the other without loading it all into RAM at once... that's fairly easy. Haskell is a lazy language. By playing with functions such as getContents, you can automatically load the data as you access it. No tricky programming required. If you're asking for something more specific -- e.g., "how do I talk to a standard database like Oracle / MySql / et al.", there are a couple of libraries for that. (Unfortunately, no one standard one.) See Stefan's answer. I'd you'd like to be more specific about what you'd like to do... _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
In the second case, if your numerical data is just a big array, why not just mmap it? Unless you're running on a 32 bit machine, or have a *seriously* large amount of data, that seems like the easiest option. Although if you're talking about mutable data, that's a whole 'nother can of worms... but that's true even if your data fits into memory. You could look at Data.ByteString to get some idea as to how to wrap a pure interface around a disk-backed chunk of read-only memory. Or just read about the ffi, it's not too hard to figure out. David On Mon, Aug 13, 2007 at 01:27:03PM -0700, Carter T Schonwald wrote:
The main two use cases I have in mind are 1) really really really big abstract syntax trees or proof trees (a la compilers or theorem provers) 2) random access to numerical data
does that help clarify what i'm asking about? In each case, is there a standard way of dealing with this? in the case of (1) the sensible way seems to me to do some sort of zipper representation which loads adjacent nodes to some depth surrounding your current location in the tree, and in the case of (2) it seems like the sensible way would be load subsequences of the data into memory.
participants (2)
-
Carter T Schonwald -
David Roundy