On 21/09/11 02:39, Heinrich Apfelmus wrote:

Tim Docker wrote:

...

I'm getting a stack overflow exception in code like this:

-- applyAction :: A -> IO [B]

.... vs <- fmap concat $ mapM applyAction sas return vs

I don't get it if I change the code to this:

-- applyAction :: A -> IO [B]

.... mapM_ applyAction sas return []

But of course, I need the results from the actions. I know that the returned list contains approximately 1 million items.

Any suggestions on how I should rewrite the first code snippet to not blow the stack?

Of course, a list of 1 million items is going to take a lot of memory, unless you generate it lazily. Unfortunately mapM cannot generate its result lazily because it has to execute all IO actions before returning the list of results.

I'm OK with it taking a lot of memory. I should have enough. It's the stack overflow exception I'm struggling with.

I'm not entirely sure whether the stack overflow happens in this part of your code, though. What happens if you change it to

map_ applyAction sas return [1..1000000]

? If this still throws a stack overflow, then problem is in the part of the code that consumes said list.

I believe the error is happening in the concat because there are subsequent IO actions that fail to execute. ie the code is equivalent to: vs <- fmap concat $ mapM applyAction sas someOtherAction consume vs and someOtherAction seems not to be run. However, to be sure, I'll confirm with code akin to what you suggest above. Tim

On Wed, Sep 21, 2011 at 3:39 AM, Heinrich Apfelmus wrote:

Of course, a list of 1 million items is going to take a lot of memory, unless you generate it lazily. Unfortunately  mapM  cannot generate its result lazily because it has to execute all IO actions before returning the list of results.

That's oversimplifying a bit. The outer list cannot be generated lazily, but the inner values (in this case inner lists) can be generated lazily. On Wed, Sep 21, 2011 at 7:00 PM, Tim Docker wrote:

I believe the error is happening in the concat because there are subsequent IO actions that fail to execute. ie the code is equivalent to:

       vs <- fmap concat $ mapM applyAction sas        someOtherAction        consume vs

and someOtherAction seems not to be run. However, to be sure, I'll confirm with code akin to what you suggest above.

The error shouldn't be happening in either concat or mapM. Are you sure that someOtherAction isn't being run? Might it be writing to a file and the result isn't getting flushed? GHC has no inherent limit on the stack size, though using extremely large amounts of stack is usually indicative of an error. You can up the stack limit with the -Ksize RTS option, and I think there is a way it can be disabled entirely. You might try upping your stack size and profiling your program to see if that's helpful: the -xt profiling option might be useful, but I haven't played with it much. I suspect the issue is that one of your applyAction is creating a thunk that blows the stack when it's evaluated, and "return []" ensures that the thunk is never evaluated. Though it's not clear to me why it'd be getting evaluated in this new scenario with the information you've provided, assuming you really truly aren't running someOtherAction. Best, Leon

On Wed, Sep 21, 2011 at 6:04 AM, Ketil Malde wrote:

Tim Docker writes:

...

        mapM_ applyAction sas

Maybe you could try a lazy version of mapM?  E.g., I think this would do it:

Another option is to use a version of mapM that accumulates the result on the heap. Maybe this would do the trick: mapM' f = go [] where go acc [] = return (reverse acc) go acc (a:as) = do {x <- a; go (x:acc) as} HTH, -- Felipe.