Compiling an extremely large Haskell file (in GHC)

I have an extremely large source file of about 11 MB. It's the all-pairs shortest paths data for a map of the Hyde Park area of Chicago (no real reason, really). I generated information in Scheme and printed the result to a Haskell source file as a list. I then edited the file to initialized an array with the data. GHC, with a 200MB stack, took up 1 hour and 1.3 GB of memory before getting killed by the system. How would I compile something of this size? I need to have the array of all-pairs shortest paths pre-computed. Any suggestions? Thanks. -Arjun

Arjun Guha wrote:
I have an extremely large source file of about 11 MB. It's the all-pairs shortest paths data for a map of the Hyde Park area of Chicago (no real reason, really). I generated information in Scheme and printed the result to a Haskell source file as a list. I then edited the file to initialized an array with the data.
GHC, with a 200MB stack, took up 1 hour and 1.3 GB of memory before getting killed by the system. How would I compile something of this size? I need to have the array of all-pairs shortest paths pre-computed. Any suggestions?
Don't do that. ;) Seriously though, you would probably have better luck parsing a data file at runtime rather than trying to compile it in. It might even be that the "read" implementation for lists will be sufficent for what you want.

On 6/27/05, Arjun Guha
It's the all-pairs shortest paths data for a map of the Hyde Park area of Chicago (no real reason, really).
I wonder: is there really no way to do Floyd-Warshall in Haskell? -- regards, radu http://rgrig.blogspot.com/

On Tue, 2005-06-28 at 12:11 +0300, Radu Grigore wrote:
On 6/27/05, Arjun Guha
wrote: It's the all-pairs shortest paths data for a map of the Hyde Park area of Chicago (no real reason, really).
I wonder: is there really no way to do Floyd-Warshall in Haskell?
Indeed I understand from a colleague who implemented an all-pairs shortest paths algorithm in Haskell over the weekend for a map of the Hyde Park area of Chicago (no real reason, really!), that it takes about 0.1 seconds to execute (if you compile with -O), which is well within the 5 second limit... I wonder why everyone is so interested in the distances between intersections in the Hyde Park area of Chicago all of a sudden... http://icfpc.plt-scheme.org/ Duncan

As a self-taught Haskell programmer of about a year, I'm really interested in seeing your colleague's code. I'd like to know what I did wrong. How about after two weeks? I think that's reasonable! -Arjun On Jun 28, 2005, at 20:05 , Duncan Coutts wrote:
On Tue, 2005-06-28 at 12:11 +0300, Radu Grigore wrote:
On 6/27/05, Arjun Guha
wrote: It's the all-pairs shortest paths data for a map of the Hyde Park area of Chicago (no real reason, really).
I wonder: is there really no way to do Floyd-Warshall in Haskell?
Indeed I understand from a colleague who implemented an all-pairs shortest paths algorithm in Haskell over the weekend for a map of the Hyde Park area of Chicago (no real reason, really!), that it takes about 0.1 seconds to execute (if you compile with -O), which is well within the 5 second limit...
I wonder why everyone is so interested in the distances between intersections in the Hyde Park area of Chicago all of a sudden...
Duncan
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

On Tue, 2005-06-28 at 20:13 -0400, Arjun Guha wrote:
As a self-taught Haskell programmer of about a year, I'm really interested in seeing your colleague's code. I'd like to know what I did wrong. How about after two weeks? I think that's reasonable!
I'm sure we'll publish our entry after the end of the second round, we did last year (http://urchin.earth.li/icfpcontest/2004/). However I would not expect it to be something beautiful to behold; code written in haste seldom is. In the meantime, try looking up the standard algorithms for all-pairs shortest paths. Duncan

Hi, On Wednesday 29 June 2005 10:05, Duncan Coutts wrote:
On Tue, 2005-06-28 at 12:11 +0300, Radu Grigore wrote:
On 6/27/05, Arjun Guha
wrote: It's the all-pairs shortest paths data for a map of the Hyde Park area of Chicago (no real reason, really).
I wonder: is there really no way to do Floyd-Warshall in Haskell?
Indeed I understand from a colleague who implemented an all-pairs shortest paths algorithm in Haskell over the weekend for a map of the Hyde Park area of Chicago (no real reason, really!), that it takes about 0.1 seconds to execute (if you compile with -O), which is well within the 5 second limit...
you can just use dijkstra from the fgl (http://www.haskell.org/ghc/docs/latest/html/libraries/fgl/Data.Graph.Inducti...) for the paths you are interested. We used it *a lot* and we didn't have any performance problems. There is really no need to precompute all-pairs shortest path! Cheers, Stefan -- Stefan Wehr Web: http://www.stefanwehr.de PGP: Key is available from pgp.mit.edu, ID 0B9F5CE4

On 6/29/05, Duncan Coutts
Indeed I understand from a colleague who implemented an all-pairs shortest paths algorithm in Haskell over the weekend for a map of the Hyde Park area of Chicago (no real reason, really!), that it takes about 0.1 seconds to execute (if you compile with -O), which is well within the 5 second limit...
Well, that is about two times slower than c++ code :p ... I mean: it's way faster that what I would have expected :) Anyway, I am interested in how DP-like algorithms are implemented in Haskell (with the same complexities), not in this particular problem. A week ago I tried LCS, now the ICFP contest fed me with another algorithm that isn't trivial to implement in this language. For ICFPC you could use some other solution than Floyd-Warshall but, again, I am asking about _this_ particular algorithm.
I wonder why everyone is so interested in the distances between intersections in the Hyde Park area of Chicago all of a sudden...
You are right: I'll ask again in 2 weeks. So please don't answer to this post :) -- regards, radu http://rgrig.blogspot.com/
participants (6)
-
Arjun Guha
-
Arjun Guha
-
Duncan Coutts
-
Radu Grigore
-
robert dockins
-
Stefan Wehr