- ghc-devs - Haskell.org

Re: [Haskell-cafe] STM friendly TreeMap (or similar with range scan api) ? WAS: Best ways to achieve throughput, for large M:N ratio of STM threads, with hot TVar updates?
by Compl Yue 06 Aug '20

06 Aug '20

Hi Devs & Cafe, I would report back my progress on it, actually I've got a rough conclusion that TL;DR: > For data-intensive workloads, x86_64 ISA has its cache of CPU chips being a hardware bottleneck, it's very hard to scale up with added number of cores, so long as they share the cache as being in a single chip. For the details - I developed a minimal script interpreter for diagnostic purpose, dependent only on libraries bundled with GHC, the source repository is at: https://github.com/complyue/txs <https://github.com/complyue/txs> I benchmarked on my machine with a single 6-core Xeon E5 CPU chip, for contention-free read/write performance scaling, got numbers at: https://github.com/complyue/txs/blob/master/results/baseline.csv <https://github.com/complyue/txs/blob/master/results/baseline.csv> conc thread avg tps scale eff populate 1 1741 1.00 1.00 2 1285 1.48 0.74 3 1028 1.77 0.59 4 843 1.94 0.48 5 696 2.00 0.40 6 600 2.07 0.34 scan 1 1565 1.00 1.00 2 1285 1.64 0.82 3 1018 1.95 0.65 4 843 2.15 0.54 5 696 2.22 0.44 6 586 2.25 0.37 The script is at: https://github.com/complyue/txs/blob/master/scripts/scan.txs <https://github.com/complyue/txs/blob/master/scripts/scan.txs> GHC cmdl is at: https://github.com/complyue/txs/blob/master/metric.bash <https://github.com/complyue/txs/blob/master/metric.bash> ghc --make -Wall -threaded -rtsopts -prof -o txs -outputdir . -stubdir . -i../src ../src/Main.hs && ( ./txs +RTS -N10 -A32m -H256m -qg -I0 -M5g -T -s <../scripts/"${SCRIPT}".txs ) I intended to use a single Haskell based process to handle meta data about many ndarrays being crunched, acting as a centralized graph database, as it turned out, many clients queued to query/insert meta data against a single database node, will create such high data throughput that just few CPU chips can't handle well, we didn't expect this but apparently we'll have to deploy more machines as for such a database instance, with data partitioned and distributed to more nodes for load balancing. (A single machine with many sockets for CPU thus many NUMA nodes is neither an option for us.) While the flexibility a central graph database would provide, is not currently a crucial requirement of our business, so we are not interested to further develop this database system. We currently have CPU intensive workloads handled by some cluster of machines running Python processes (crunching numbers with Numpy and C++ tensors), while some Haskell based number crunching software are still under development, it may turn out some day in the future, that some heavier computation be bound with the db access, effectively creating some CPU intensive workloads for the database functionality, then we'll have the opportunity to dive deeper into the database implementation. And in case more flexibility required in near future, I think I'll tend to implement embedded database instances in those worker processes, in contrast to centralized db servers. I wonder if ARM servers will have up scaling of data intensive workloads easier, though that's neither a near feasible option for us. Thanks for everyone that have been helpful! Best regards, Compl > On 2020-07-31, at 22:35, YueCompl via Haskell-Cafe <haskell-cafe(a)haskell.org> wrote: > > Hi Ben, > > Thanks as always for your great support! And at the moment I'm working on a minimum working example to reproduce the symptoms, I intend to work out a program depends only on libraries bundled with GHC, so it can be easily diagnosed without my complex env, but so far no reprod yet. I'll come with some piece of code once it can reproduce something. > > Thanks in advance. > > Sincerely, > Compl > > >> On 2020-07-31, at 21:36, Ben Gamari <ben(a)well-typed.com> wrote: >> >> Simon Peyton Jones via Haskell-Cafe <haskell-cafe(a)haskell.org> writes: >> >>>> Compl’s problem is (apparently) that execution becomes dominated by >>>> GC. That doesn’t sound like a constant-factor overhead from TVars, no >>>> matter how efficient (or otherwise) they are. It sounds more like a >>>> space leak to me; perhaps you need some strict evaluation or >>>> something. >>> >>> My point is only: before re-engineering STM it would make sense to get >>> a much more detailed insight into what is actually happening, and >>> where the space and time is going. We have tools to do this (heap >>> profiling, Threadscope, …) but I know they need some skill and insight >>> to use well. But we don’t have nearly enough insight to draw >>> meaningful conclusions yet. >>> >>> Maybe someone with experience of performance debugging might feel able >>> to help Compl? >>> >> Compl, >> >> If you want to discuss the issue feel free to get in touch on IRC. I >> would be happy to help. >> >> It would be great if we had something of a decision tree for performance >> tuning of Haskell code in the users guide or Wiki. We have so many tools >> yet there isn't a comprehensive overview of >> >> 1. what factors might affect which runtime characteristics of your >> program >> 2. which tools can be used to measure which factors >> 3. how these factors can be improved >> >> Cheers, >> >> - Ben >> _______________________________________________ >> Haskell-Cafe mailing list >> To (un)subscribe, modify options or view archives go to: >> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe >> Only members subscribed via the mailman list are allowed to post. > > _______________________________________________ > Haskell-Cafe mailing list > To (un)subscribe, modify options or view archives go to: > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe > Only members subscribed via the mailman list are allowed to post.

1 0

Hi. I'm new to this mailing list and have a few questions.
by Anselm Schüler (conversations subemail) 04 Aug '20

04 Aug '20

8 13

How should we treat changes to the GHC API?
by Simon Peyton Jones 31 Jul '20

31 Jul '20

A recent MR for GHC<https://gitlab.haskell.org/ghc/ghc/-/merge_requests/3758> (adding machinery for plugins to write data to extensible interface files) made me wonder: how we should treat significant changes to the GHC API? Changes to the GHC API, especially to bits used by plugins or by IDEs, are clearly user-visible to an important class of users - they are not just internal to GHC itself. So, how should we review them? Should they perhaps be part of the GHC proposals process? Or some other similar process? (The collection of experts on the GHC API, plugins, IDEs etc, is rather different to the membership of the GHC steering group.) I'm asking, not to be obstructive, but because the GHC API deserves to be thought of as a whole; in the past it has grown incrementally, without much discussion, and that has not served us well. But at the moment there is no process, no group to consult. Any views? Simon

11 17

Fwd: [Haskell-cafe] For STM to practically serve large in-mem datasets with cyclic structures WAS: STM friendly TreeMap (or similar with range scan api) ? WAS: Best ways to achieve throughput, for large M:N ratio of STM threads, with hot TVar updates?
by YueCompl 30 Jul '20

30 Jul '20

Dear GHC Devs, I realize I should seek your helps regarding my current situation. TL;DR the most serious suffering is: After the heap loaded with many TVars with cyclic structures, GC will dominate my CPU utilization with little business progressing. Nonmoving GC `-xn` with 8.10.1 helps, but the ceiling is not high enough for my case, obvious performance degrade starts at about ~350MB RSS, and falls unusable as RSS approaching 1GB. While I expect a GHC compiled process to serve 20~30GB in-mem data practically okay. https://mail.haskell.org/pipermail/haskell-cafe/2020-July/132556.html <https://mail.haskell.org/pipermail/haskell-cafe/2020-July/132556.html> contains most interesting conversations. I found https://tech.channable.com/posts/2020-04-07-lessons-in-managing-haskell-mem… <https://tech.channable.com/posts/2020-04-07-lessons-in-managing-haskell-mem…> much relevant, they managed to keep 100GB per instance, but switching to immutable data structures to reside within compact regions is not immediately feasible for my case, as the schema has to be re-designed, though I have my colleges started evaluating that option. Best regards, Compl > Begin forwarded message: > > From: YueCompl via Haskell-Cafe <haskell-cafe(a)haskell.org> > Subject: [Haskell-cafe] For STM to practically serve large in-mem datasets with cyclic structures WAS: STM friendly TreeMap (or similar with range scan api) ? WAS: Best ways to achieve throughput, for large M:N ratio of STM threads, with hot TVar updates? > Date: 2020-07-30 at 21:28:31 GMT+8 > To: Haskell Cafe <haskell-cafe(a)haskell.org> > Reply-To: YueCompl <compl.yue(a)icloud.com> > > For the record, overhead of STM over IO (or other means where manual composition of transactions needed) based concurrency control, is a price I'm willing to pay in my use case, as it's not machine-performance critical in distributing input data + parameters to a cluster of worker nodes, and collecting their results into permanent storage or a data pipeline. But to keep professionally crafting well synced, race-free scheduling code is barely affordable by my org, as shape of datasets, relationship between them and algorithms processing them are varying at fast paces, we have difficulty, or lack the willingness, to hire some workforce specifically to keep each new data pipeline race free, it has to be, but better at cost of machine-hours instead of human head counts. > > While easily compositing stm code, wrapped in scriptable procedures, will enable our analysts to author the scheduling scripts without too much concerns. Then our programmers can focus on performance critical parts of the data processing code, like optimization of tight-loops. > > Only if not in the tight loops, I think it's acceptable by us, that up to 2~3 order of magnitude slower for an stm solution compared to its best rivals, as long as it's scalable. For a (maybe cheating) example, if fully optimized code can return result in 10 ms after an analyst clicked a button, we don't bother if unoptimized stm script needs 10 second, so long as the result is correct. > > In a philosophic thinking, I heard that AT&T had UNIX specifically designed for their Control panel, while their Data panel runs separate software (and on separate hardware obviously), while modern systems have powerful CPUs tempting us to squeeze more performance out of it, and SIMD instructions make it even more tempting, I think we'd better resist it when programming something belong to the Control panel per se, but do it in programming something belong to the Data panel. And appears Data panel programs are being shifted to GPUs nowadays, which feels right. > > Regards, > Compl > > >> On 2020-07-30, at 20:10, YueCompl via Haskell-Cafe <haskell-cafe(a)haskell.org <mailto:haskell-cafe@haskell.org>> wrote: >> >> Hi Peter, >> >> Great to hear from you! >> >> For the record tskiplist (and stm-containers together) did improve my situation a great lot with respect to scalability at concurrency/parallelism! >> >> I'm still far from the stage to squeeze last drops of performance, currently I'm just making sure performance wise concerns be reasonable during my PoC in correctness and ergonomics of my HPC architecture (an in-memory graph + out-of-core (mmap) array DBMS powered computation cluster, with shared storage), and after parallelism appears acceptable, I seemingly suffer from serious GC issue at up scaling on process working memory size. I'm suspecting it's because of the added more TVars and/or aggressive circular structures of them in my case, and can not find a way to overcome it by far. >> >> Thanks for your detailed information! >> >> Best regards, >> Compl >> >> >>> On 2020-07-30, at 19:19, Peter Robinson <pwr(a)lowerbound.io <mailto:pwr@lowerbound.io>> wrote: >>> >>> Hi Compl, >>> >+ This package provides a proof-of-concept implementation of a skip list in STM >>> >>> This has to mean something but I can't figure out yet. >>> >>> Dear Peter Robinson, I hope you can see this message and get in the loop of discussion. >>> >>> >>> The reason for adding this sentence was that tskiplist hasn't been optimized for production use. Later on, I wrote an implementation of a concurrent skip list with atomic operations that performs significantly better, but it's operations work in the IO monad. >>> >>> I'm surprised to hear that you're getting poor performance even when using the stm-container package, which I believe was meant to be used in production. A while ago, I ran some benchmarks comparing concurrent dictionary data structures (such as stm-container) under various workloads. While STMContainers.Map wasn't as fast as the concurrent-hashtable package, the results indicate that the performance doesn't degrade too much under larger workloads. >>> >>> You can find these benchmark results here (10^6 randomly generated insertion/deletion/lookup requests distributed among 32 threads): >>> https://lowerbound.io/blog/bench2-32.html <https://lowerbound.io/blog/bench2-32.html> >>> And some explanations about the benchmarks are here: >>> https://lowerbound.io/blog/2019-10-24_concurrent_hash_table_performance.html <https://lowerbound.io/blog/2019-10-24_concurrent_hash_table_performance.html> >>> >>> One issue that I came across when implementing the tskiplist package was this: If a thread wants to insert some item into the skip list, it needs to search for the entry point by performing readTVar operations starting at the list head. So, on average, a thread will read O(log n) TVars (assuming a skip list of n items) and, if any of these O(log n) TVars are modified by a simultaneously running thread, the STM runtime will observe a (false) conflict and rerun the transaction. It's not clear to me how to resolve this issue without access to something like unreadTVar (see [1]). >>> >>> Best, >>> Peter >>> >>> [1] UnreadTVar: Extending Haskell Software Transactional Memory for Performance (2007) by Nehir Sonmez , Cristian Perfumo , Srdjan Stipic , Adrian Cristal , Osman S. Unsal , Mateo Valero. >>> >>> _______________________________________________ >>> Haskell-Cafe mailing list >>> To (un)subscribe, modify options or view archives go to: >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe <http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe> >>> Only members subscribed via the mailman list are allowed to post. >> >> _______________________________________________ >> Haskell-Cafe mailing list >> To (un)subscribe, modify options or view archives go to: >> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe <http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe> >> Only members subscribed via the mailman list are allowed to post. > > _______________________________________________ > Haskell-Cafe mailing list > To (un)subscribe, modify options or view archives go to: > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe > Only members subscribed via the mailman list are allowed to post.

1 1

GHC 9.0?
by Krzysztof Gogolewski 25 Jul '20

25 Jul '20

Hi, There is an exceptional number of changes stated for the next release. * Better pattern matching coverage detection * New windows IO manager * Linear types * Large-scale typechecker changes - Taming the Kind Inference Monster, simplified subsumption * Better register allocation, improving runtime by 0.8% according to release notes * ghc-bignum * Explicit specificity and eager instantiation * Qualified do * Lexical negation * Perhaps Quick Look will manage to land Should we call it GHC 9.0? I think the name would be deserved.

12 14

Unmerged Patch: 3358
by Matthew Pickering 21 Jul '20

21 Jul '20

Hi, My patch 3358 needs to get merged before the 8.12 fork. When I finished it (in May), it passed CI and after this point I lacked time to work on it further. Now I have asked 4 times for this patch to get merged and it is still open. The GHC proposal for this patch already took an extortionate amount of time to get accepted. Please can we close this chapter by merging the patch. Cheers, Matt

5 4

New Windows I/O manager in GHC 8.12
by Phyx 20 Jul '20

20 Jul '20

Hi All, In case you've missed it, about 150 or so commits were committed to master yesterday. These commits add WinIO (Windows I/O) to GHC. This is a new I/O manager that is designed for the native Windows I/O subsystem instead of relying on the broken posix-ish compatibility layer that MIO used. This is one of 3 big patches I have been working on for years now.. So before I continue on why WinIO was made I'll add a TL;DR; WinIO adds an internal API break compared to previous GHC releases. That is the internal code was modified to support a completely asynchronous I/O system. What this means is that we have to keep track of the file pointer offset which previously was done by the C runtime. This is because in async I/O you cannot assume the offset to be at any given location. What does this mean for you? Very little. If you did not use internal GHC I/O code. In particular if you haven't used Buffer, BufferIO and RawIO. If you have you will to explicitly add support for GHC 8.12+. Because FDs are a Unix concept and don't behave as you would expect on Windows, the new I/O manager also uses HANDLE instead of FD. This means that any library that has used the internal GHC Fd type won't work with WinIO. Luckily the number of libraries that have seems quite low. If you can please stick to the external Handle interface for I/O functions. The boot libraries have been updated, and in particular process *requires* the version that is shipped with GHC. Please respect the version bounds here! I will be writing a migration guide for those that need to migrate code. The amount of work is usually trivial as Base provides shims to do most of the common things you would have used Fd for. Also if I may make a plea to GHC developers.. Do not add non-trivial implementations in the external exposed modules (e.g. System.xxx, Data.xxx) but rather add them to internal modules (GHC.xxx) and re-export them from the external modules. This allows us to avoid import cycles inside the internal modules :) -- So why WinIO? Over the years a number of hard to fix issues popped up on Windows, including proper Unicode console I/O, cooked inputs, ability to cancel I/O requests. This also allows libraries like Brick to work on Windows without re-inventing the wheel or have to hide their I/O from the I/O manager. In order to attempt to do some of these with MIO layer upon layers of hacks were added. This means that things sometimes worked.., but when it didn't was rather unpredictable. Some of the issues were simply unfixable with MIO. I will be making some posts about how WinIO works (and also archiving them on the wiki don't worry :)) but for now some highlights: WinIO is 3 years of work, First started by Joey Hess, then picked up by Mikhail Glushenkov before landing at my feet. While the majority has been rewritten their work did provide a great jumping off point so thanks! Also thanks to Ben and AndreasK for helping me get it over the line.. As you can imagine I was exhausted by this point :). Some stats: ~8000 new lines and ~1100 removed ones spread over 130+ commits (sorry this was the smallest we could get it while not losing some historical context) and with over 153 files changed not counting the changes to boot libraries. It Fixes #18307, #17035, #16917, #15366, #14530, #13516, #13396, #13359, #12873, #12869, #11394, #10542, #10484, #10477, #9940, #7593, #7353, #5797, #5305, #4471, #3937, #3081, #12117, #2408, #10956, #2189 (but only on native windows consoles, so no msys shells) and #806 which is 14 years old! WinIO is a dynamic choice, so you can switch between I/O managers using the RTS flag --io-manager=[native|posix]. On non-Windows native is the same as posix. The chosen Async interface for this implementation is using Completion Ports. The I/O manager uses a new interface added in Windows Vista called GetQueuedCompletionStatusEx which allows us to service multiple request interrupts in one go. Some highlights: * Drops Windows Vista support Vista is out of extended support as of 2017. The new minimum is Windows 7. This allows us to use much more efficient OS provided abstractions. * Replace Events and Monitor locks with much faster and efficient Conditional Variables and SlimReaderWriterLocks. * Change GHC's Buffer and I/O structs to support asynchronous operation by not relying on the OS managing File Offset. * Implement a new command line flag +RTS --io-manager=[native|posix] to control which I/O manager is used. * Implement a new Console I/O interface supporting much faster reads/writes and unicode output correctly. Also supports things like cooked input etc. * In new I/O manager if the user still has their code-page set to OEM, then we use UTF-8 by default. This allows Unicode to work correctly out of the box. * Add Atomic Exchange PrimOp and implement Atomic Ptr exchanges. * Flush event logs eagerly as to not rely on finalizers running. * A lot of refactoring and more use of hsc2hs to share constants * Control aborts Ctrl+C should be a bit more reliable. * Add a new IOPort primitive that should be only used for these I/O operations. Essentially an IOPort is based on an MVar with the following major differences: - Does not allow multiple pending writes. If the port is full a second write is just discarded. - There is no deadlock avoidance guarantee. If you block on an IOPort and your Haskell application does not have any work left to do the whole application is stalled. In the threaded RTS we just continue idling, in the non-threaded rts the scheduler is blocked. * Support various optimizations in the Windows I/O manager such as skipping I/O Completion if the request finished synchronously etc. * The I/O manager is now agnostic to the handle type. i.e. There is no socket specific code in the manager. This is now all pushed to the network library. Completely de-coupling these. * Unified threaded and non-threaded I/O code. The only major difference is where event loop is driven from and that the non-threaded rts will always use a single OS thread to service requests. We cannot use more as there are no rts locks to make concurrent modifications safe. Cheers, Tamar

4 7

HEAD doesn't build. Totally stalled.
by Simon Peyton Jones 20 Jul '20

20 Jul '20

I'm getting this failure in a clean HEAD build. Any ideas? I'm totally stalled because I can't build GHC any more. I'm using Windows Subsystem for Linux (WSL). Help help! Thanks Simon /home/simonpj/code/HEAD-9/rts/dist/build/libHSrts_thr_p.a(RtsSymbols.thr_p_o): RtsSymbols.c:rtsSyms: error: undefined reference to '__stack_chk_guard' collect2: error: ld returned 1 exit status `cc' failed in phase `Linker'. (Exit code: 1) utils/iserv/ghc.mk:105: recipe for target 'utils/iserv/stage2_p/build/tmp/ghc-iserv-prof' failed make[1]: *** [utils/iserv/stage2_p/build/tmp/ghc-iserv-prof] Error 1 make[1]: *** Waiting for unfinished jobs....

5 12

DreamObjects outage
by Ben Gamari 18 Jul '20

18 Jul '20

Hi everyone, It seems like our block storage provider is currently experiencing a service interruption. Consequently you may find that accessing CI logs and CI artifacts fails. I have contact the provider and hopefully the issue will be resolved shortly. Cheers, - Ben

1 0

Error messages
by Erik de Castro Lopo 17 Jul '20

17 Jul '20

Hi all, Although it was many years ago I did spend soem time working on GHC and I do know what a thankless task it is. I made a compliant about GHC error messages on an internal Slack channel and Mortiz encouraged me to repeat it here. I am incredibly happy about the quality of error messges for older more standard parts of Haskell, probably most things in Haskell98 and Haskell2010. By way of contrast, error messages for some newer parts of Haskell are incredibly poor. In fact, for the new parts, the error rmessages are often wrong, just defaulting to error messages for older parts of Haskell. As an example (open source code in a public GH repo): src/Cardano/DbSync/StateQuery.hs:87:44: error: • Data constructor not in scope: QueryAnytime :: QueryHardFork xs0 (Interpreter xs0) -> Query (CardanoBlock TPraosStandardCrypto) (Interpreter (CardanoEras TPraosStandardCrypto)) • Perhaps you want to add ‘QueryAnytime’ to the import list in the import of ‘Ouroboros.Consensus.HardFork.Combinator.Ledger.Query’ (src/Cardano/DbSync/StateQuery.hs:49:1-116). | 87 | queryHistoryInterpreter connInfo (point, QueryAnytime GetInterpreter) The suggestion is that I need to import `QueryAnytime` but just 20 line above I have: import Ouroboros.....Query (QueryAnytime (..), QueryHardFork (GetInterpreter)) The problem is that `QueryAnytime` is defined as a pattern synonym. I have only the tinest amount of experience using pattern synonyms and that error message is not really useful. I would like to suggest that a prerequesite for merging of new features would be that it provides good error messages and more importantly does not provide wrong or misleading error messages like the one above. Erik -- ---------------------------------------------------------------------- Erik de Castro Lopo http://www.mega-nerd.com/

5 4