- ghc-devs - Haskell.org

Call for GHC Maintainers
by Moritz Angermann 18 Aug '20

18 Aug '20

Hi there! As it stands right now, Ben is the one who works tirelessly trying to cut releases. Not just for the most recent version, but also for previous versions. Most recently 8.10.2, but we have 9.0 coming up as well. I know that there are some people who deeply care for personal or professional reasons for older releases, 8.4, 8.6, 8.8, ... Some of them have stacks of patches applied, or proprietary extensions. I'd argue that most of those applied patches are backports of bug fixes and rarely language features, as language features will break compatibility (due to ghc, base, and other library versions anyway). I would therefore like drum up a group of people who will take care (ideally 2+ per release) of backporting and making minor partch releases. This does not have to go on forever, but it would take much needed load off of Ben to focus on what ever happens in ghc HEAD. So what would this work actually look like? It would consist of - going through the list of MRs and tagging those which are relevant for backporting to a certain release. - backport MRs where the MR does not cleanly apply. - fixup any test-suite failures. - agree on a date to cut/make the release. This is not a permanent commitment. I hope we can attract more people to the ghc release managers. I'm looking forward to great many responses. And I'm sure Ben will be able to help mentor us through cutting the first releases. I'll volunteer to be part of the 8.6 branch maintainers for now. Cheers, Moritz PS: There is a slightly related discussion about release cadence and versions and how other projects deal with this in this ticket: https://gitlab.haskell.org/ghc/ghc/-/issues/18222

5 6

The curious case of #367: Infinite loops can hang Concurrent Haskell
by Moritz Angermann 17 Aug '20

17 Aug '20

Hi there! While working on a NCG, I eventually came across #367[0], which make GHC produce code that looks similar to this: ``` label: [non-branch-instructions]* brach-instruction label ``` so essentially an uninterruptible loop. The solution for GHC to produce code that can be interrupted is to pass -fno-omit-yields. So far so good. Out of curiosity, I did add a small piece of code to detect this to my NCG to complain if code like the above was generated[1]. Three weeks ago, I kind of maneuvered myself into a memory blow up corner, and then life happened, but this weekend I managed to find some time to revert some memory blow up and continue working on the NCG. Turns out I can build a stage2 "quick" flavour of the NCG without dynamic support just fine. I never saw the dead lock detection code fire. Now I did leave the test suite running yesterday night, and when looking through the test suite results, there were quite a few failure. Curiously a lot of them were due to ghc missing dynamic support (doh!). But also quite a few that failed due to the deadlock detection. T12485, hs_try_putmvar003, ds-wildcard, ds001, read029, T2817, tc011, tc021, T4524 So, my question then is this: are we fine with ghc generating this code? Or, if we are not, do we want to figure out if we can eliminate it? The issue 367 goes into quite a bit of detail why this is tricky to handle generally. Or should we add -fno-omit-yields to the test-cases? The ultimate option is to just turn of the detection, and I'm fine with doing so. However I'd rather ask if anyone sees value in detecting this or not. Cheers, Moritz -- [0]: https://gitlab.haskell.org/ghc/ghc/-/issues/367 [1]: https://gitlab.haskell.org/ghc/ghc/-/blob/46fba2c91e1c4d23d46fa2d9b18dcd000…

2 6

ApplicativeDo needs a bit of love
by Simon Marlow 12 Aug '20

12 Aug '20

Folks, ApplicativeDo does what it does quite well, indeed it's in heavy use doing what we built it for, and I was pleasantly surprised to see that 12% of people would even like it to be enabled by default <https://taylor.fausak.me/2019/11/16/haskell-survey-results/>. But a number of issues have arisen around it, some stemming from limitations in the original implementation and others due to interactions with other features that we didn't consider at the time. This message is just to point out that I'd be delighted if anyone wanted to jump on any of these tickets and work on them, as it's unlikely I'll find the bandwidth to work on them myself. All the tickets are tagged with ApplicativeDo: https://gitlab.haskell.org/ghc/ghc/-/issues?scope=all&utf8=%E2%9C%93&state=… Have at it! Cheers Simon

1 0

[ANNOUNCE] Glasgow Haskell Compiler 8.10.2 released
by Ben Gamari 08 Aug '20

08 Aug '20

Hello all, The GHC team is happy to announce the release of GHC 8.10.2. Source and binary distributions are available at the usual place: https://downloads.haskell.org/ghc/8.10.2/ GHC 8.10.2 fixes a number of issues in present in GHC 8.10.1 including: * Fixes a bug in process creation on Windows (#17926). * Fixes a code generation bug resulting in incorrect code generation for foreign calls with complex arguments (#18527) * Fixes a bug causing object merging to fail when the lld linker is in use (#17962) * Introduces a workaround for a Linux kernel bug in the implementation of the timerfd mechanism (#18033). * Fixes a few specialiser regressions (#17810, #18120) as well introduces a variety of miscellaneous specialiser improvements (#16473, #17930, #17966) * Fixes a potential loss of sharing due to left operator sections (#18151). * Fix bootstrapping of GHC with the LLVM backend on x86-64 (#17920). * A few important correctness fixes for the low-latency garbage collector. Users of `--nonmoving-gc` are strongly encouraged to upgrade promptly. Note that at the moment we still require that macOS Catalina users exempt the binary distribution from the notarization requirement by running `xattr -cr .` on the unpacked tree before running `make install`. This situation will hopefully be improved for GHC 9.0.1 with the resolution of #17418 [1]. Cheers, - Ben [1] https://gitlab.haskell.org/ghc/ghc/issues/17418

1 0

Question about binary distributions
by Mathieu Boespflug 07 Aug '20

07 Aug '20

Hi all, GHC currently has 3 tier-1 platforms: Linux, macOS and Windows. I'll focus the dicussion below on these three platforms. The binary distributions for Linux and macOS are designed to be unpacked, then the user types ./configure && make install. This is not the case for Windows. On all platforms it's possible to create "relocatable" installations, such that GHC doesn't really care where it's installed, and commands will still work if the install directory changes location on the filesystem. So my question is, why do we have a ./configure step on Linux and macOS? Why could we not have bindists for all platforms that work like the Windows one? I.e. a binary distribution that you just unpack, in any directory of your choice, without any configuration or installation step.

5 4

Re: [Haskell-cafe] STM friendly TreeMap (or similar with range scan api) ? WAS: Best ways to achieve throughput, for large M:N ratio of STM threads, with hot TVar updates?
by Compl Yue 06 Aug '20

06 Aug '20

Hi Devs & Cafe, I would report back my progress on it, actually I've got a rough conclusion that TL;DR: > For data-intensive workloads, x86_64 ISA has its cache of CPU chips being a hardware bottleneck, it's very hard to scale up with added number of cores, so long as they share the cache as being in a single chip. For the details - I developed a minimal script interpreter for diagnostic purpose, dependent only on libraries bundled with GHC, the source repository is at: https://github.com/complyue/txs <https://github.com/complyue/txs> I benchmarked on my machine with a single 6-core Xeon E5 CPU chip, for contention-free read/write performance scaling, got numbers at: https://github.com/complyue/txs/blob/master/results/baseline.csv <https://github.com/complyue/txs/blob/master/results/baseline.csv> conc thread avg tps scale eff populate 1 1741 1.00 1.00 2 1285 1.48 0.74 3 1028 1.77 0.59 4 843 1.94 0.48 5 696 2.00 0.40 6 600 2.07 0.34 scan 1 1565 1.00 1.00 2 1285 1.64 0.82 3 1018 1.95 0.65 4 843 2.15 0.54 5 696 2.22 0.44 6 586 2.25 0.37 The script is at: https://github.com/complyue/txs/blob/master/scripts/scan.txs <https://github.com/complyue/txs/blob/master/scripts/scan.txs> GHC cmdl is at: https://github.com/complyue/txs/blob/master/metric.bash <https://github.com/complyue/txs/blob/master/metric.bash> ghc --make -Wall -threaded -rtsopts -prof -o txs -outputdir . -stubdir . -i../src ../src/Main.hs && ( ./txs +RTS -N10 -A32m -H256m -qg -I0 -M5g -T -s <../scripts/"${SCRIPT}".txs ) I intended to use a single Haskell based process to handle meta data about many ndarrays being crunched, acting as a centralized graph database, as it turned out, many clients queued to query/insert meta data against a single database node, will create such high data throughput that just few CPU chips can't handle well, we didn't expect this but apparently we'll have to deploy more machines as for such a database instance, with data partitioned and distributed to more nodes for load balancing. (A single machine with many sockets for CPU thus many NUMA nodes is neither an option for us.) While the flexibility a central graph database would provide, is not currently a crucial requirement of our business, so we are not interested to further develop this database system. We currently have CPU intensive workloads handled by some cluster of machines running Python processes (crunching numbers with Numpy and C++ tensors), while some Haskell based number crunching software are still under development, it may turn out some day in the future, that some heavier computation be bound with the db access, effectively creating some CPU intensive workloads for the database functionality, then we'll have the opportunity to dive deeper into the database implementation. And in case more flexibility required in near future, I think I'll tend to implement embedded database instances in those worker processes, in contrast to centralized db servers. I wonder if ARM servers will have up scaling of data intensive workloads easier, though that's neither a near feasible option for us. Thanks for everyone that have been helpful! Best regards, Compl > On 2020-07-31, at 22:35, YueCompl via Haskell-Cafe <haskell-cafe(a)haskell.org> wrote: > > Hi Ben, > > Thanks as always for your great support! And at the moment I'm working on a minimum working example to reproduce the symptoms, I intend to work out a program depends only on libraries bundled with GHC, so it can be easily diagnosed without my complex env, but so far no reprod yet. I'll come with some piece of code once it can reproduce something. > > Thanks in advance. > > Sincerely, > Compl > > >> On 2020-07-31, at 21:36, Ben Gamari <ben(a)well-typed.com> wrote: >> >> Simon Peyton Jones via Haskell-Cafe <haskell-cafe(a)haskell.org> writes: >> >>>> Compl’s problem is (apparently) that execution becomes dominated by >>>> GC. That doesn’t sound like a constant-factor overhead from TVars, no >>>> matter how efficient (or otherwise) they are. It sounds more like a >>>> space leak to me; perhaps you need some strict evaluation or >>>> something. >>> >>> My point is only: before re-engineering STM it would make sense to get >>> a much more detailed insight into what is actually happening, and >>> where the space and time is going. We have tools to do this (heap >>> profiling, Threadscope, …) but I know they need some skill and insight >>> to use well. But we don’t have nearly enough insight to draw >>> meaningful conclusions yet. >>> >>> Maybe someone with experience of performance debugging might feel able >>> to help Compl? >>> >> Compl, >> >> If you want to discuss the issue feel free to get in touch on IRC. I >> would be happy to help. >> >> It would be great if we had something of a decision tree for performance >> tuning of Haskell code in the users guide or Wiki. We have so many tools >> yet there isn't a comprehensive overview of >> >> 1. what factors might affect which runtime characteristics of your >> program >> 2. which tools can be used to measure which factors >> 3. how these factors can be improved >> >> Cheers, >> >> - Ben >> _______________________________________________ >> Haskell-Cafe mailing list >> To (un)subscribe, modify options or view archives go to: >> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe >> Only members subscribed via the mailman list are allowed to post. > > _______________________________________________ > Haskell-Cafe mailing list > To (un)subscribe, modify options or view archives go to: > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe > Only members subscribed via the mailman list are allowed to post.

1 0

Hi. I'm new to this mailing list and have a few questions.
by Anselm Schüler (conversations subemail) 04 Aug '20

04 Aug '20

8 13

How should we treat changes to the GHC API?
by Simon Peyton Jones 31 Jul '20

31 Jul '20

A recent MR for GHC<https://gitlab.haskell.org/ghc/ghc/-/merge_requests/3758> (adding machinery for plugins to write data to extensible interface files) made me wonder: how we should treat significant changes to the GHC API? Changes to the GHC API, especially to bits used by plugins or by IDEs, are clearly user-visible to an important class of users - they are not just internal to GHC itself. So, how should we review them? Should they perhaps be part of the GHC proposals process? Or some other similar process? (The collection of experts on the GHC API, plugins, IDEs etc, is rather different to the membership of the GHC steering group.) I'm asking, not to be obstructive, but because the GHC API deserves to be thought of as a whole; in the past it has grown incrementally, without much discussion, and that has not served us well. But at the moment there is no process, no group to consult. Any views? Simon

11 17

Fwd: [Haskell-cafe] For STM to practically serve large in-mem datasets with cyclic structures WAS: STM friendly TreeMap (or similar with range scan api) ? WAS: Best ways to achieve throughput, for large M:N ratio of STM threads, with hot TVar updates?
by YueCompl 30 Jul '20

30 Jul '20

Dear GHC Devs, I realize I should seek your helps regarding my current situation. TL;DR the most serious suffering is: After the heap loaded with many TVars with cyclic structures, GC will dominate my CPU utilization with little business progressing. Nonmoving GC `-xn` with 8.10.1 helps, but the ceiling is not high enough for my case, obvious performance degrade starts at about ~350MB RSS, and falls unusable as RSS approaching 1GB. While I expect a GHC compiled process to serve 20~30GB in-mem data practically okay. https://mail.haskell.org/pipermail/haskell-cafe/2020-July/132556.html <https://mail.haskell.org/pipermail/haskell-cafe/2020-July/132556.html> contains most interesting conversations. I found https://tech.channable.com/posts/2020-04-07-lessons-in-managing-haskell-mem… <https://tech.channable.com/posts/2020-04-07-lessons-in-managing-haskell-mem…> much relevant, they managed to keep 100GB per instance, but switching to immutable data structures to reside within compact regions is not immediately feasible for my case, as the schema has to be re-designed, though I have my colleges started evaluating that option. Best regards, Compl > Begin forwarded message: > > From: YueCompl via Haskell-Cafe <haskell-cafe(a)haskell.org> > Subject: [Haskell-cafe] For STM to practically serve large in-mem datasets with cyclic structures WAS: STM friendly TreeMap (or similar with range scan api) ? WAS: Best ways to achieve throughput, for large M:N ratio of STM threads, with hot TVar updates? > Date: 2020-07-30 at 21:28:31 GMT+8 > To: Haskell Cafe <haskell-cafe(a)haskell.org> > Reply-To: YueCompl <compl.yue(a)icloud.com> > > For the record, overhead of STM over IO (or other means where manual composition of transactions needed) based concurrency control, is a price I'm willing to pay in my use case, as it's not machine-performance critical in distributing input data + parameters to a cluster of worker nodes, and collecting their results into permanent storage or a data pipeline. But to keep professionally crafting well synced, race-free scheduling code is barely affordable by my org, as shape of datasets, relationship between them and algorithms processing them are varying at fast paces, we have difficulty, or lack the willingness, to hire some workforce specifically to keep each new data pipeline race free, it has to be, but better at cost of machine-hours instead of human head counts. > > While easily compositing stm code, wrapped in scriptable procedures, will enable our analysts to author the scheduling scripts without too much concerns. Then our programmers can focus on performance critical parts of the data processing code, like optimization of tight-loops. > > Only if not in the tight loops, I think it's acceptable by us, that up to 2~3 order of magnitude slower for an stm solution compared to its best rivals, as long as it's scalable. For a (maybe cheating) example, if fully optimized code can return result in 10 ms after an analyst clicked a button, we don't bother if unoptimized stm script needs 10 second, so long as the result is correct. > > In a philosophic thinking, I heard that AT&T had UNIX specifically designed for their Control panel, while their Data panel runs separate software (and on separate hardware obviously), while modern systems have powerful CPUs tempting us to squeeze more performance out of it, and SIMD instructions make it even more tempting, I think we'd better resist it when programming something belong to the Control panel per se, but do it in programming something belong to the Data panel. And appears Data panel programs are being shifted to GPUs nowadays, which feels right. > > Regards, > Compl > > >> On 2020-07-30, at 20:10, YueCompl via Haskell-Cafe <haskell-cafe(a)haskell.org <mailto:haskell-cafe@haskell.org>> wrote: >> >> Hi Peter, >> >> Great to hear from you! >> >> For the record tskiplist (and stm-containers together) did improve my situation a great lot with respect to scalability at concurrency/parallelism! >> >> I'm still far from the stage to squeeze last drops of performance, currently I'm just making sure performance wise concerns be reasonable during my PoC in correctness and ergonomics of my HPC architecture (an in-memory graph + out-of-core (mmap) array DBMS powered computation cluster, with shared storage), and after parallelism appears acceptable, I seemingly suffer from serious GC issue at up scaling on process working memory size. I'm suspecting it's because of the added more TVars and/or aggressive circular structures of them in my case, and can not find a way to overcome it by far. >> >> Thanks for your detailed information! >> >> Best regards, >> Compl >> >> >>> On 2020-07-30, at 19:19, Peter Robinson <pwr(a)lowerbound.io <mailto:pwr@lowerbound.io>> wrote: >>> >>> Hi Compl, >>> >+ This package provides a proof-of-concept implementation of a skip list in STM >>> >>> This has to mean something but I can't figure out yet. >>> >>> Dear Peter Robinson, I hope you can see this message and get in the loop of discussion. >>> >>> >>> The reason for adding this sentence was that tskiplist hasn't been optimized for production use. Later on, I wrote an implementation of a concurrent skip list with atomic operations that performs significantly better, but it's operations work in the IO monad. >>> >>> I'm surprised to hear that you're getting poor performance even when using the stm-container package, which I believe was meant to be used in production. A while ago, I ran some benchmarks comparing concurrent dictionary data structures (such as stm-container) under various workloads. While STMContainers.Map wasn't as fast as the concurrent-hashtable package, the results indicate that the performance doesn't degrade too much under larger workloads. >>> >>> You can find these benchmark results here (10^6 randomly generated insertion/deletion/lookup requests distributed among 32 threads): >>> https://lowerbound.io/blog/bench2-32.html <https://lowerbound.io/blog/bench2-32.html> >>> And some explanations about the benchmarks are here: >>> https://lowerbound.io/blog/2019-10-24_concurrent_hash_table_performance.html <https://lowerbound.io/blog/2019-10-24_concurrent_hash_table_performance.html> >>> >>> One issue that I came across when implementing the tskiplist package was this: If a thread wants to insert some item into the skip list, it needs to search for the entry point by performing readTVar operations starting at the list head. So, on average, a thread will read O(log n) TVars (assuming a skip list of n items) and, if any of these O(log n) TVars are modified by a simultaneously running thread, the STM runtime will observe a (false) conflict and rerun the transaction. It's not clear to me how to resolve this issue without access to something like unreadTVar (see [1]). >>> >>> Best, >>> Peter >>> >>> [1] UnreadTVar: Extending Haskell Software Transactional Memory for Performance (2007) by Nehir Sonmez , Cristian Perfumo , Srdjan Stipic , Adrian Cristal , Osman S. Unsal , Mateo Valero. >>> >>> _______________________________________________ >>> Haskell-Cafe mailing list >>> To (un)subscribe, modify options or view archives go to: >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe <http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe> >>> Only members subscribed via the mailman list are allowed to post. >> >> _______________________________________________ >> Haskell-Cafe mailing list >> To (un)subscribe, modify options or view archives go to: >> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe <http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe> >> Only members subscribed via the mailman list are allowed to post. > > _______________________________________________ > Haskell-Cafe mailing list > To (un)subscribe, modify options or view archives go to: > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe > Only members subscribed via the mailman list are allowed to post.

1 1

GHC 9.0?
by Krzysztof Gogolewski 25 Jul '20

25 Jul '20

Hi, There is an exceptional number of changes stated for the next release. * Better pattern matching coverage detection * New windows IO manager * Linear types * Large-scale typechecker changes - Taming the Kind Inference Monster, simplified subsumption * Better register allocation, improving runtime by 0.8% according to release notes * ghc-bignum * Explicit specificity and eager instantiation * Qualified do * Lexical negation * Perhaps Quick Look will manage to land Should we call it GHC 9.0? I think the name would be deserved.

12 14