The blocked on black hole message is very suspicious.  It means that thread 7 is blocked waiting for another thread to evaluate a thunk.  But in this case, it&#39;s thread 7 that created that thunk and is supposed to be doing the evaluating.  This is some evidence that Gregory&#39;s theory is correct and your encode function loops somewhere.<br><br><div class="gmail_quote">On Wed Nov 12 2014 at 11:25:30 AM Michael Jones &lt;<a href="mailto:mike@proclivis.com">mike@proclivis.com</a>&gt; wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>Gregory,</div><div><br></div><div>The options in the 7.8.3 user guide says in the -Msize option that by default the heap is unlimited. I have several applications, and they all have messages like:</div><div><br></div><div><div>7fddc7bcd700: cap 2: waking up thread 7 on cap 2</div><div>7fddc7bcd700: cap 2: thread 4 stopped (yielding)</div><div>7fddcaad6740: cap 2: running thread 7 (ThreadRunGHC)</div><div>7fddcaad6740: cap 2: thread 7 stopped (heap overflow)</div><div>7fddcaad6740: cap 2: requesting parallel GC</div><div>7fddc5ffe700: cap 0: starting GC</div><div>7fddc57fd700: cap 1: starting GC</div><div>7fdda77fe700: cap 3: starting GC</div><div>7fddcaad6740: cap 2: starting GC</div></div><div><br></div><div>I assumed that when the heap ran out of space, it caused a GC, or it enlarged the heap. The programs that have these messages run for very long periods of time, and when I heap profile them, they use about 500KM to 1MB over long periods of time, and are quite stable.</div><div><br></div><div>As a test, I ran the hang application with profiling to see if memory jumps up before or after the hang.</div><div><br></div><div>What I notice is the app moves along using about 800KB, then there is a spike to 2MB at the hang. So I believe you, but I am confused about the RTS behavior and how I can have all these overflow messages in a normal application and how to tell the difference between these routine messages vs a real heap problem.</div><div><br></div><div>So, I dug deeper into the log. A normal execution for sending a command looks like:</div><div><br></div><div><div>7f99e6ffd700: cap 0: running thread 7 (ThreadRunGHC)</div><div>7f99e6ffd700: cap 0: thread 7 stopped (heap overflow)</div><div>7f99e6ffd700: cap 0: requesting parallel GC</div><div>7f99e6ffd700: cap 0: starting GC</div><div>7f99e6ffd700: cap 0: GC working</div><div>7f99e6ffd700: cap 0: GC idle</div><div>7f99e6ffd700: cap 0: GC done</div><div>7f99e6ffd700: cap 0: GC idle</div><div>7f99e6ffd700: cap 0: GC done</div><div>7f99e6ffd700: cap 0: all caps stopped for GC</div><div>7f99e6ffd700: cap 0: finished GC</div><div>7f99e6ffd700: cap 0: running thread 7 (ThreadRunGHC)</div><div>7f99e6ffd700: cap 0: sendCommand</div><div>7f99e6ffd700: cap 0: sendCommand: encoded</div><div>7f99e6ffd700: cap 0: sendCommand: size 4</div><div>7f99e6ffd700: cap 0: sendCommand: unpacked</div><div>7f99e6ffd700: cap 0: Sending command of size 4</div><div>7f99e6ffd700: cap 0: Sending command of size &quot;\NUL\EOT&quot;</div><div>7f99e6ffd700: cap 0: sendCommand: sent</div><div>7f99e6ffd700: cap 0: sendCommand: flushed</div><div>7f99e6ffd700: cap 0: thread 7 stopped (blocked on an MVar)</div><div>7f99e6ffd700: cap 0: running thread 2 (ThreadRunGHC)</div><div>7f99e6ffd700: cap 0: thread 2 stopped (yielding)</div><div>7f99e6ffd700: cap 0: running thread 45 (ThreadRunGHC)</div><div>7f99e6ffd700: cap 0: fetchTelemetryServer</div><div>7f99e6ffd700: cap 0: fetchTelemetryServer: got lock</div></div><div><br></div><div>The thread is run, overflows, GC, runs, then blocks on an MVAr.</div><div><br></div><div>For a the hang case:</div><div><br></div><div>7f99e6ffd700: cap 0: running thread 7 (ThreadRunGHC)</div><div><div>7f99e6ffd700: cap 0: sendCommand</div><div>7f99e6ffd700: cap 0: thread 7 stopped (heap overflow)</div><div>7f99e6ffd700: cap 0: requesting parallel GC</div><div>7f99e6ffd700: cap 0: starting GC</div><div>7f99e6ffd700: cap 0: GC working</div><div>7f99e6ffd700: cap 0: GC idle</div><div>7f99e6ffd700: cap 0: GC done</div><div>7f99e6ffd700: cap 0: GC idle</div><div>7f99e6ffd700: cap 0: GC done</div><div>7f99e6ffd700: cap 0: all caps stopped for GC</div><div>7f99e6ffd700: cap 0: finished GC</div><div>7f9a05362a40: cap 0: running thread 1408 (ThreadRunGHC)</div><div>7f9a05362a40: cap 0: thread 1408 stopped (yielding)</div><div>7f99e6ffd700: cap 0: running thread 7 (ThreadRunGHC)</div><div>7f99e6ffd700: cap 0: thread 7 stopped (heap overflow)</div><div>7f99e6ffd700: cap 0: requesting parallel GC</div><div>7f99e6ffd700: cap 0: starting GC</div><div>7f99e6ffd700: cap 0: GC working</div><div>7f99e6ffd700: cap 0: GC idle</div><div>7f99e6ffd700: cap 0: GC done</div><div>7f99e6ffd700: cap 0: GC idle</div><div>7f99e6ffd700: cap 0: GC done</div><div>7f99e6ffd700: cap 0: all caps stopped for GC</div><div>7f99e6ffd700: cap 0: finished GC</div><div>7f99e6ffd700: cap 0: running thread 7 (ThreadRunGHC)</div><div>7f99e6ffd700: cap 0: thread 7 stopped (yielding)</div><div>7f99e6ffd700: cap 0: running thread 2 (ThreadRunGHC)</div><div>7f99e6ffd700: cap 0: thread 2 stopped (yielding)</div><div>7f99e6ffd700: cap 0: running thread 45 (ThreadRunGHC)</div><div>7f99e6ffd700: cap 0: fetchTelemetryServer</div><div>7f99e6ffd700: cap 0: fetchTelemetryServer: got lock</div><div>...</div><div>7f99e6ffd700: cap 0: fetchTelemetryServer: got lock</div><div>7f99e6ffd700: cap 0: fetchTelemetryServer: unlock</div><div>7f99e6ffd700: cap 0: fetchTelemetryServer</div><div>7f99e6ffd700: cap 0: fetchTelemetryServer: got lock</div><div>7f99e6ffd700: cap 0: fetchTelemetryServer: unlock</div><div>7f99e6ffd700: cap 0: fetchTelemetryServer</div><div>7f99e6ffd700: cap 0: thread 45 stopped (yielding)</div><div>7f9a05362a40: cap 0: running thread 1408 (ThreadRunGHC)</div><div>7f9a05362a40: cap 0: thread 1408 stopped (suspended while making a foreign call)</div><div>7f9a05362a40: cap 0: running thread 1408 (ThreadRunGHC)</div><div>7f9a05362a40: cap 0: thread 1408 stopped (suspended while making a foreign call)</div><div>7f99e6ffd700: cap 0: running thread 7 (ThreadRunGHC)</div><div>7f99e6ffd700: cap 0: thread 7 stopped (blocked on black hole owned by thread 7)</div><div>7f9a05362a40: cap 0: running thread 1408 (ThreadRunGHC)</div><div>7f9a05362a40: cap 0: thread 1408 stopped (suspended while making a foreign call)</div></div><div><br></div><div>run, overflow, GC, run, overflow, GC, run, yield, other stuff, run blocked on black hole</div><div><br></div><div>And that is the last activity for thread 7.</div><div><br></div><div>I found a few links about black holes and such, so I’ll go off and read those and try to learn what the RTS is doing and why it can hang on a black hole. But if you have some hits, let me know. </div><div><br></div><div>Mike</div></div><div style="word-wrap:break-word"><div><br></div><div><br></div><div><br></div><div><div><div><div>On Nov 11, 2014, at 4:01 PM, Gregory Collins &lt;<a href="mailto:greg@gregorycollins.net" target="_blank">greg@gregorycollins.net</a>&gt; wrote:</div><br><blockquote type="cite"><div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Nov 11, 2014 at 2:06 PM, Michael Jones <span dir="ltr">&lt;<a href="mailto:mike@proclivis.com" target="_blank">mike@proclivis.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Those are all over the log even when it runs properly. So I assume the runtime is resizing the heap or something.</blockquote></div><br>No, it means you&#39;re exhausting the heap (maybe the runtime stack for the thread running &quot;encode&quot;), probably because &quot;encode&quot; is infinite-looping. I think Occam&#39;s razor applies here, check that any recursion you&#39;re doing is actually reducing the recursive argument. Perhaps you could post the code (e.g. <a href="http://gist.github.com/" target="_blank">http://gist.github.com/</a>)?</div><div class="gmail_extra"><br></div><div class="gmail_extra">G<br><br clear="all"><div><br></div>-- <br><div>Gregory Collins &lt;<a href="mailto:greg@gregorycollins.net" target="_blank">greg@gregorycollins.net</a>&gt;</div>
</div></div>
</blockquote></div><br></div></div></div>______________________________<u></u>_________________<br>
Glasgow-haskell-users mailing list<br>
<a href="mailto:Glasgow-haskell-users@haskell.org" target="_blank">Glasgow-haskell-users@haskell.<u></u>org</a><br>
<a href="http://www.haskell.org/mailman/listinfo/glasgow-haskell-users" target="_blank">http://www.haskell.org/<u></u>mailman/listinfo/glasgow-<u></u>haskell-users</a><br>
</blockquote></div>