On Jan 23, 2006, at 3:33 AM, Bulat Ziganshin wrote:
what you mean? afaik, there is no standard FastPackedString implementation, but there is some library that with minimal modifications used in darcs, jhc and many other apps
I considered the version at Don Stewart's web site to be the standard, perhaps that was silly of me.
JD> still reports large amounts of memory used. Does ghc use mmap to JD> allocate memory instead of malloc?
afaik, the answer is yes. look at ghc-6.4.1-src\ghc\rts\MBlock.c
I've been doing the bulk of my testing in linux on i386, and if I'm reading the ifdefs correctly that is mmap'd anonymously. So at least I would expect Ian's memory tool to report the memory allocations correctly on that platform.
JD> I'm looking for advice or help in optimizing darcs in this case. I JD> guess this could be viewed as a challenge for people that felt like JD> the micro benchmarks of the shootout were unfair to Haskell. Can we JD> demonstrate that Haskell provides good performance in the real- world JD> when working with large files? Ideally, darcs could easily work with JD> a patch that is 10GB in size using only a few megs of ram if need be JD> and doing so in about the time it takes read the file once or twice JD> and gzip it.
it seems that Ian just used this as memory/time-efficient alternative for hGetContents. reading from memory-mapped file may be done as pure computation if the whole file is mapped. is this used in darcs?
I'm not sure, I have looked at the code but I can't tell. I think that was the point with the mmap'd files. There are several layers of abstraction at work here. Slurpies, PackedStrings, (custom) Lazy Reader monad, and maybe some others.
if not, then we can map small block at the time or even just use hGetBuf - at least, for first version. what you will say about this?
I'd expect it to be horribly slow, but if it gives guarantees about memory usage then I bet we could tune it up and make it tolerable.
i'll be happy to help, but i'm not ready to study the darcs code. i'm ready to solve more concrete problems - such as providing new input mechanism with given external interface or optimizing something rather small
That reminds me, you wrote FastIO right? I haven't looked at FastIO, do you think it would apply here? Thanks, Jason