Hi Ben, The windows build is unusable. The settings file has $TopDir expanded and hard-coded to the build path on drydock. Tamar On Mon, 10 Apr 2017, 08:14 Boespflug, Mathieu, wrote:

Hi Ben,

this is great news! I'm particularly keen on learning more about two points you mentioned in your email:

* Compiler performance: do you have any numbers to quantify what 8.0 vs 8.2 is likely to look like? How much has the work that's been done affect performance across the board? Or are these mostly pathological cases (e.g. ghc --make with high number of cores, large number of definitions in a module, large amount of let nesting in definitions, etc) * DWARF support: could you clarify at a very high-level what typical uses cases can be expected to work and which ones won't? Would be eager to read any resources you could point me at to help me understand what still needs to be done on this front.

Many thanks,

-- Mathieu Boespflug Founder at http://tweag.io.

On 10 April 2017 at 05:50, Ben Gamari wrote:

Hello everyone,

The GHC team is very pleased to announce the first candidate of the 8.2.1 release of the Glasgow Haskell Compiler. Source and binary distributions are available at

https://downloads.haskell.org/~ghc/8.2.1-rc1/

This is the first of a few release candidates leading up the final 8.2.1 release. This release will feature,

* A new type-indexed Typeable implementation

* The long awaited Backpack

* Deriving strategies for disambiguating DeriveAnyClass, GeneralizedNewtypeDeriving, and stock mechanisms

* Overloaded record fields

* Improved compiler performance

* Better code generation through more robust tracking of join points

* Compact regions for more efficient garbage collection and serialization

* Better support for machines with non-uniform memory architectures

* More robust support for levity (e.g. RuntimeRep) polymorphism

* A simple interface for streaming eventlog data from live processes

* Further refinement of DWARF support

Unfortunately there are still a few known issues in this release, including a few compiler panics (#13233, #13509) and a memory leak in the simplifier (#13426) which may adversely affect heap sizes and compilation time for some modules. This memory leak unfortunately made it impossible to provide a 32-bit Windows distribution for this candidate; this will be resolved in -rc2.

As always, please let us know if you have difficulty. Thanks to everyone who has contributed to this release!

Happy testing,

- Ben

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

"Boespflug, Mathieu" writes:

Hi Ben,

this is great news! I'm particularly keen on learning more about two points you mentioned in your email:

* Compiler performance: do you have any numbers to quantify what 8.0 vs 8.2 is likely to look like?

I'm afraid the best I can provide at the moment is [1]. On closer inspection of these I'm a bit suspicious of the 8.0 numbers; I'll try to reproduce them (and characterize the current ghc-8.2 branch, which fixes a significant memory leak present in -rc1) shortly. That being said, there were a few major fixes in 8.2.

How much has the work that's been done affect performance across the board? Or are these mostly pathological cases (e.g. ghc --make with high number of cores, large number of definitions in a module, large amount of let nesting in definitions, etc)

The fixes span the gamut but I generally tried to concentrate on issues that looked like they might affect a large fraction of programs. I fixed at least one major regression present in 8.0 which significantly inflated allocations of typeclass instance matching. I've also done quite a bit of refactoring around our handling of names. These include improvements in interface file serialization efficiency and name lookup. These are just some of the major reworks; there are also numerous smaller fixes which I don't have time to cover here. Compilation performance has also generally improved as a result of some of the work in the simplifier. In particular, GHC now performs an early inlining phase which, quite surprisingly, tends to result in smaller programs earlier in simplification, reducing the amount of work that the compiler needs to carry out. Simon summarized some of his preliminary numbers here [2].

* DWARF support: could you clarify at a very high-level what typical uses cases can be expected to work and which ones won't? Would be eager to read any resources you could point me at to help me understand what still needs to be done on this front.

At this point if GHC compiles your program with -g you should have a pretty good assurance that the resulting DWARF information is reasonable. This means that users can use the ExecutionStack mechanism, RTS stack-trace functionality, and GDB without fear of the act of capturing a stacktrace leading to a crash. After starting to write more here, I realized that you will likely not be the last person to ask about this and updated the DWARF status page with additional discussion [3]. Let me know if you have any questions. Cheers, - Ben [1] https://gist.github.com/bgamari/fbd3e55047bd041d8208ebe820c0f493 [2] https://mail.haskell.org/pipermail/ghc-devs/2017-February/013818.html [3] https://ghc.haskell.org/trac/ghc/wiki/DWARF/Status

"Boespflug, Mathieu" writes:

Hi Ben,

thank you for this summary. The DWARF status page is helpful. Something was unclear to me though. There are three main potential use cases for DWARF that I see:

1. debugging, possibly with gdb 2. stack traces on exceptions 3. stack sampling, which is a form of performance profiling.

Forgive me for these naive questions, but... Is (1) possible at all at this point?

If you break into a program with gdb you will indeed get a usable stacktrace. However, you cannot probe in-scope bindings as you can in other languages.

If I compile GHC with all the right options as explained in the status page, do I magically get backtraces associated to all my exceptions? What should I do to get those?

Ahh, thanks for pointing this out; this should be discussed on the status page. I have a proposal allowing this here [1] and there is a ticket here [2]. It turns out that this is actually a slightly tricky thing due to two facts, * stack unwinding can be expensive * exceptions (especially asynchronous exceptions) are sometimes used as a mechanism for normal control flow. Compatibility is quite tricky as under the current plan existing code would drop callstack information if it catches and rethrows an exception. I don't see any way to resolve this in a backwards compatible manner.

It's my understanding that (3) is possible and works fine, even when FFI code is invoked, but slower than for C programs due to running user-level code to inspect the stack for each sample. Is that right?

It's possible with my statistical profiling patches, but these are currently not merged and the project is on hold due to lack of time. You can, however, use perf. --callgraph will be broken in Haskell code, however (e.g. you will only see the first stack frame). Cheers, - Ben [1] https://ghc.haskell.org/trac/ghc/wiki/Exceptions/StackTraces [2] https://ghc.haskell.org/trac/ghc/ticket/12096

"Boespflug, Mathieu" writes:

Hi Ben,

it sounds like some of the remaining limitations around DWARF support (e.g. finishing the stack sampling work, local bindings in GDB, ...) could make for a good Haskell Summer of Code project. Have you considered writing this up as one or two project ideas here: https://summer.haskell.org/ideas.html? As HSoC sponsors, we'd be more than happy to see someone pick this up.

I haven't until now. To be honest I think the local-bindings-in-GDB issue is a hard problem and is likely well beyond a summer project. You would need not just the ability track the in-scope bindings in every context of the source program, but also the ability to reconstitute their values after simplification. It's also not clear to me how useful this feature would be given how much simplified code often differs from the code the user writes. However, I do see there might be room for a project on the statistical profiler itself or its associated tooling. We just need to come to a conclusion on which direction is most appropriate for GHC. For this having some concrete use-cases would be quite helpful. How do you envision using statistical profiling on Haskell projects? What is the minimal set of features that would make for a useful profiler? Cheers, - Ben

"Boespflug, Mathieu" writes:

...

However, I do see there might be room for a project on the statistical profiler itself or its associated tooling. We just need to come to a conclusion on which direction is most appropriate for GHC.

You mean - given a choice between somehow reusing perf or going down the route of fully custom tooling?

Right.

...

For this having some concrete use-cases would be quite helpful. How do you envision using statistical profiling on Haskell projects? What is the minimal set of features that would make for a useful profiler?

That sounds like a good way to approach this. Here goes...

I'd really prefer seeing a Haskell program as a black box, that I can profile using the same tools as C programs or native code generated from any other language. It shouldn't matter that the source is Haskell. In my ideal workflow, I have a *vanilla* Haskell program compiled with debug symbols by a *vanilla* GHC (no special ./configure options as prereqs), that I can hook up perf to, e.g.

$ perf record -g ./mybinary

Then I should be able to use perf report to analyze the results. Or indeed use existing pipelines to obtain other visualizations (flame graphs etc).

I'm not particularly interested in integration with the event log, though others might have a need for that.

I'm also interested in hotspot analysis, à la perf annotate.

As Brendan Gregg says, "perf isn't some random tool: it's part of the Linux kernel, and is actively developed and enhanced."

I need accurate and informative stack samples (no STG internal details in the output that I can't connect back to source locations) for programs that include all manner of FFI calls. Better still if time spent in the GC doesn't pollute my stack samples.

The tricky part is that for flame graphs you need to sample stacks, and for that you need to teach perf how to collect that data somehow, since the C stack isn't used for haskell activation frames and we have a call-by-need evaluation strategy anyways. But the slow option you mention in the status page sounds okayish to me, and using eBPF to perform stack sampling entirely from the kernel looks like a promising direction.

Indeed that is the engineering side of the trickiness. However, there is also a theoretically difficult aspect which Peter articulates nicely in Chapter 2 of his thesis. I refer you to the thesis for the full explanation but, in short, reasoning about causality in lazy languages is quite difficult (which is why this whole endeavour was worthy of a thesis). This leads to some amount of impedance mismatch with existing tooling, which is the reason I started down the road of a Haskell-centric solution. To elaborate, while in an imperative language it is relatively easy to say "instruction at address $A arose from line $N in $FILE.c", this sort of unambiguous statement is not possible in Haskell. Instead, we have to be satisfied with *sets* of source locations. That is, "instruction at address $A arose from line $N in $FILE, and line $N' in $FILE', and ...". Unfortunately, essentially none of the existing profiling and debugging infrastructure (DWARF and perf included) was designed with this model in mind. In particular, DWARF's line information encoding requires that GHC choose a single location to attribute the cost of each instruction to. This is, as you might expect, a rather tricky choice to make and while GHC has a few heuristics, we will inevitably be wrong in some circumstances. From experience, I can say that attempting to analyse profiles naively, using only GHC's heuristically-guided guess of the appropriate source location, can lead to rather perplexing results. For this reason, GHC uses [1] DWARF's extensibility mechanisms to export an additional set of line information which can be consumed by Haskell-aware tooling which captures the full richness of GHC's source ticks. In section 5.8 of his thesis Peter proposes a scheme of "fuzzing" for making use of this location information. I haven't tried Peter's fuzzing approach, but I suspect it will make the profiler output significantly easier to reason about. All this being said, I totally agree that being able to use widely-known, well-maintained native tools is a significant advantage. This is a large part of why I put the profiling project on hold. I have wondered whether we might be able to provide a preprocessor which would take sample data from GHC's statistical profiler and export it to perf's own format, performing fuzzing and any other necessary Haskell-specific preprocessing. This would at least allow us to tap in to the ecosystem that has arisen around perf. Alternatively, we could try to contribute patches to perf's upstream, although I'm not sure how likely acceptance would be. Cheers, - Ben [1] https://phabricator.haskell.org/D1279#R?query=D1279

Hi Ben, devs, Thanks for your hard work! Could you also include the following commits? These are about user's guide. * https://phabricator.haskell.org/D3850 * https://phabricator.haskell.org/D3852 Regards, Takenobu 2017-10-01 20:17 GMT+09:00 Ben Gamari :

Hello everyone,

The GHC team is very pleased to announce the first candidate of the 8.2.2 release of the Glasgow Haskell Compiler. Source and binary distributions are available at

https://downloads.haskell.org/~ghc/8.2.2-rc1/

Currently there is a slightly reduced set of binary distributions available; do let me know if there is a missing platform you would like to see built.

This is the first of two release candidates leading up the final 8.2.2 release. This release fixes approximately fifty bugs present in 8.2.1. See [1] for the full list.

As always, please report any issues you encounter.

Happy testing,

- Ben

[1] https://ghc.haskell.org/trac/ghc/query?status=closed& milestone=8.2.2&col=id&col=summary&col=status&col=type& col=priority&col=milestone&col=component&order=priority

_______________________________________________ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users