- ghc-devs - Haskell.org

What's the benefit of taking "do" blocks apart? Is there a way to turn that off?
by Erdi, Gergo 29 Dec '21

29 Dec '21

PUBLIC Hi, I'm seeing 'do' blocks getting taking apart into top-level definitions, so e.g. main = do some complicated expression 1 some complicated expression 2 is compiled into sat_sKv = some complicated expression 1 sat_sKw = \_ -> some complicated expression 2 main = bindIO sat_sKv sat_sKw This seems to happen regardless of any common subexpressions, i.e. it is not the case that sat_sKv or sat_sKw are used anywhere else. What is the intended benefit of this floating-out? Is there a particular Core-to-Core pass that causes this? Is it possible to turn it off? Thanks, Gergo This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries at https: //www.sc.com/en/our-locations Where you have a Financial Markets relationship with Standard Chartered PLC, Standard Chartered Bank and their subsidiaries (the "Group"), information on the regulatory standards we adhere to and how it may affect you can be found in our Regulatory Compliance Statement at https: //www.sc.com/rcs/ and Regulatory Compliance Disclosures at http: //www.sc.com/rcs/fm Insofar as this communication is not sent by the Global Research team and contains any market commentary, the market commentary has been prepared by the sales and/or trading desk of Standard Chartered Bank or its affiliate. It is not and does not constitute research material, independent research, recommendation or financial advice. Any market commentary is for information purpose only and shall not be relied on for any other purpose and is subject to the relevant disclaimers available at https: //www.sc.com/en/regulatory-disclosures/#market-disclaimer. Insofar as this communication is sent by the Global Research team and contains any research materials prepared by members of the team, the research material is for information purpose only and shall not be relied on for any other purpose, and is subject to the relevant disclaimers available at https: //research.sc.com/research/api/application/static/terms-and-conditions. Insofar as this e-mail contains the term sheet for a proposed transaction, by responding affirmatively to this e-mail, you agree that you have understood the terms and conditions in the attached term sheet and evaluated the merits and risks of the transaction. We may at times also request you to sign the term sheet to acknowledge the same. Please visit https: //www.sc.com/en/regulatory-disclosures/dodd-frank/ for important information with respect to derivative products.

2 1

Output language of typechecking pass?
by Benjamin Redelings 28 Dec '21

28 Dec '21

Hi, I have been looking for info on what actually comes out of the type-checking pass in GHC. This is mostly because it seems like the "Type classes in Haskell" paper implements both type checking and translation to dictionary-passing in one pass, whereas it seems like GHC separates this into (i) type checking and (ii) desugaring. Questions: 1. It seems like this separation is actually necessary, in order to apply generalization only to let arguments written by the programmer, and not to let bindings introduced during desugaring. Is that right? 2. Does the output of type checking contain type lambdas? 3. Does the type checking pass determine where to add dictionary arguments? 4. Are there any other resources I should be looking at? I am confused about #3, because the `-ddump-tc` output doesn't seem to show any type dictionaries in function bodies themselves, but it does seem to contain some kind of info about dictionaries as "evidence" -- but I am not sure what "evidence" is, or how it links into the AST for a function body. I did briefly look at `-ddump-tc-ast`, but not in detail yet. -BenRI

4 7

Measuring thread cpu time (using bound threads)
by Harendra Kumar 28 Dec '21

28 Dec '21

Hi GHC devs, I want to measure the CPU time spent in a particular Haskell thread across thread yield points. I am assuming Haskell threads do not keep track of per thread cpu time. Therefore, to measure it I have to use the thread cpu clock (CLOCK_THREAD_CPUTIME_ID) provided by the OS (Linux). But Haskell threads can keep jumping across multiple OS threads so the OS thread's CPU time cannot be used reliably. To solve that issue I thought I could use bound threads. From the documentation it sounded like bound threads are exclusively bound to a particular OS thread so we can get thread cpu time from the OS and use it. However, if I print the thread id for a bound thread using "gettid" (OS thread id on Linux) or using "pthread_self" (pthread id), both these thread ids are changing for the same Haskell thread which is a bound thread. Is that expected, or am I doing something wrong? My questions: * Are bound threads really bound to a particular OS thread or they can use multiple OS threads for executing Haskell code, and only use a fixed OS thread for FFI code? * Is there a way to make this work? Can we obtain the thread cpu time reliably across thread yield points? Is there a way to really bind Haskell threads to OS threads? * Is there a better or alternative way to get the thread CPU time? Thanks, Harendra

1 0

Source locations from Core
by Erdi, Gergo 28 Dec '21

28 Dec '21

PUBLIC Hi, I'm looking for ways to map Core fragments back to source locations. I see there is an annotated version of Core in `GHC/Core.hs` called `AnnExpr`, which I could see being useful for this if I set the annotation type to `SrcSpan`, but that's not what I get out of GHC's desugarer, simplifier or tidier. If there's no built-in mechanism for this, my only idea would be to create a HsExpr-to-HsExpr transformation that wraps every node in a call that is opaque enough to be persisted through Core-to-Core transformations but still transparent enough that it doesn't block optimization opportunities. Is that even possible? Alternatively, would it make it easer if I was content with only getting source locations for variable occurrences? Thanks, Gergo This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries at https: //www.sc.com/en/our-locations Where you have a Financial Markets relationship with Standard Chartered PLC, Standard Chartered Bank and their subsidiaries (the "Group"), information on the regulatory standards we adhere to and how it may affect you can be found in our Regulatory Compliance Statement at https: //www.sc.com/rcs/ and Regulatory Compliance Disclosures at http: //www.sc.com/rcs/fm Insofar as this communication is not sent by the Global Research team and contains any market commentary, the market commentary has been prepared by the sales and/or trading desk of Standard Chartered Bank or its affiliate. It is not and does not constitute research material, independent research, recommendation or financial advice. Any market commentary is for information purpose only and shall not be relied on for any other purpose and is subject to the relevant disclaimers available at https: //www.sc.com/en/regulatory-disclosures/#market-disclaimer. Insofar as this communication is sent by the Global Research team and contains any research materials prepared by members of the team, the research material is for information purpose only and shall not be relied on for any other purpose, and is subject to the relevant disclaimers available at https: //research.sc.com/research/api/application/static/terms-and-conditions. Insofar as this e-mail contains the term sheet for a proposed transaction, by responding affirmatively to this e-mail, you agree that you have understood the terms and conditions in the attached term sheet and evaluated the merits and risks of the transaction. We may at times also request you to sign the term sheet to acknowledge the same. Please visit https: //www.sc.com/en/regulatory-disclosures/dodd-frank/ for important information with respect to derivative products.

2 2

CI: Choice of base commit for perf comparisons
by Joachim Breitner 22 Dec '21

22 Dec '21

Hi, the new (or “new”?) handling of perf numbers, where CI just magically records and compares them, without us having to manually edit the `all.T` files, is a big improvement, thanks! However, I found the choice of the base commit to compare against unhelpful. Assume master is at commit M, and I start a feature branch and MR with commit A. CI runs, and tells me about a performance regressions, and CI is red. I now fix the issue and push commit B to the branch. CI runs, but it picks A to compare against, and now it is red because of an seemingly unexpected performance improvement! I would have expected that all CI runs for this MR to compare the performance against the base branch on master, and to look for perf change notices in all commit messages in between. I see these advantages: * The reported perf changes correspond to the changes shown on the MR page * Green CI = the MR is ready (after squashing) * CI will have numbers for the base commit more reliably (else, if I push commit C quickly after B, then the job for B might be cancelled and Ci will report changes of C against A instead of B, which is unexpected). I have used this logic of reporting perf changes (or any other “differential CI”) against the base branch in the Motoko project and it was quite natural. Would it be desirable and possible for us here, too? (A possible rebuttal might be: we don’t push new commits to feature branches, but always squash and rebase, as that’s what we have to do before merging anyways. If that’s the case then ok, although I generally lean to having chronological commits on feature branches and a nice squashed commit on master.) Cheers, Joachim -- Joachim Breitner mail(a)joachim-breitner.de http://www.joachim-breitner.de/

2 2

Merge request merge timing
by Ben Gamari 21 Dec '21

21 Dec '21

Hello all, Recently there have been a few instances where merge requests were merged extremely quickly (e.g. within an hour of the MR being opened). While prompt review is a good thing, we also want to make sure that all potentially interested reviewers have an opportunity to comment. In this vein, I propose that we wait at least 24 hours from the time an MR is opened before sending it to Marge. I have documented this to the merge-request review guidelines on the Wiki [1]. Cheers, - Ben [1] https://gitlab.haskell.org/ghc/ghc/-/wikis/gitlab/merge-requests

1 0

[PSA] Patches to Haddock regarding GHC development move to GitLab!
by Hécate 17 Dec '21

17 Dec '21

Hi everyone, just a PSA from the Haddock team and GHC release managers to warn everyone that, effective immediately, patches to Haddock needed for GHC merge requests are not to be directed to GitHub anymore. Those patches are to be sent to the GitLab mirror: https://gitlab.haskell.org/ghc/haddock. I will try and warn folks who currently have PRs opened on GitHub targeting ghc-head, but please know that they'll be closed in the next few days. Have a great end of the week, Hécate, for the Haddock Team. -- Hécate ✨ 🐦: @TechnoEmpress IRC: Hecate WWW: https://glitchbra.in RUN: BSD

2 1

Transparently hooking into the STG stack to validate an escape analysis
by Sebastian Graf 17 Dec '21

17 Dec '21

Hi Devs, my master's student Sebastian and I (also Sebastian :)) are working on an escape analysis in STG, see https://gitlab.haskell.org/ghc/ghc/-/issues/16891#note_347903. We have a prototype for the escape analysis that we want to validate/exploit now. The original plan was to write the transformation that allocates non-escaping objects on the STG stack. But that is quite tricky for many reasons, one of them being treatment of the GC. This mail is rather lengthy, so I suggest you skip to "What we hope could work" and see if you can answer it without the context I provide below. If you can't, I would be very grateful if you were willing to suffer through the exposition. # Instrumentation So instead we thought about doing a (easily changed and thus versatile) instrumentation-based approach: Assign a sequence number to every instantiation (a term I will use to mean "allocation of g's closure") of things that we our analysis determines as escaping or non-escaping, such as STG's let bindings (focusing on non-let-no-escape functions for now). (One sequence number *per allocation* of the let binding's closure, not based on its syntactic identity.) Then, we set a bit in a (dynamically growing) global bit vector whenever the let "RHS is entered" and then unset it when we "leave the let-body". Example: f = \x y -> let { g = [y] \z -> y + z; } in g x Here, our analysis could see that no instantiation (which I say instead of "allocation of g's closure") of g will ever escape its scope within f. Our validation would give a fresh sequence number to the instantiation of g whenever f is called and store it in g's closure (which we arrange by piggy-backing on -prof and adding an additional field to the profiling header). Then, when g's RHS is entered, we set the bit in the global bit vector, indicating "this instantiation of g might escape". After leaving the RHS of g, we also leave the body of the defining let, which means we unset the bit in the bit vector, meaning "every use so far wasn't in an escaping scenario". So far so good. Modifying the code upon entering g takes a bit of tinkering but can be done by building on TickyTicky in StgToCmm. But what is not done so easily is inserting the continuation upon entering the let that will unset the bit! # What doesn't work: Modifying the Sequel At first, we tried to modify the sequel <https://gitlab.haskell.org/ghc/ghc/-/blob/4c6985cc91b9cf89b393f69c9a603e8df…> of the let-body to an `AssignTo`. That requires us to know the registers in which the let-body will return its results, which in turn means we have to know the representation of those results, so we have to write a function `stgExprPrimRep :: GenStgExpr p -> [PrimRep]`. Urgh! We were very surprised that there was no such function. And while we tested our function, we very soon knew why. Consider the following pattern synonym matcher: GHC.Natural.$mNatJ# :: forall {rep :: GHC.Types.RuntimeRep} {r :: TYPE rep}. GHC.Num.Natural.Natural -> (GHC.Num.BigNat.BigNat -> r) -> ((# #) -> r) -> r = {} \r [scrut_sBE cont_sBF fail_sBG] case scrut_sBE of { GHC.Num.Natural.NS _ -> fail_sBG GHC.Prim.(##); GHC.Num.Natural.NB ds_sBJ -> let { sat_sBK :: GHC.Num.BigNat.BigNat = CCCS GHC.Num.BigNat.BN#! [ds_sBJ]; } in cont_sBF sat_sBK; }; Note how its result is representation-polymorphic! It only works because our machine implementation allows tail-calls. It's obvious in hindsight that we could never write `stgExprPrimRep` in such a way that it will work on the expression `cont_sBF sat_sBK`. So the sequel approach seems infeasible. # What we hope could work: A special stack frame The other alternative would be to insert a special continuation frame on the stack when we enter the let-body (inspired by stg_restore_cccs). This continuation frame would simply push all registers (FP regs, GP regs, Vec regs, ...) to the C stack, do its work (unsetting the bit), then pop all registers again and jump to the topmost continuation on the STG stack. Example: f :: forall rep (r :: TYPE rep). Int# -> (Int# -> r) -> r f = \x g -> let { h = [x] \a -> x + a; } in case h x of b { __DEFAULT -> g b } We are only interested in unsetting the bit for h here. Consider the stack when entering the body of h. caller_of_f_cont_info <- Sp Now push our special continuation frame: caller_of_f_cont_info seq_h unset_bit_stk_info <- Sp E.g., the stack frame contains the info pointer and the sequence number. (Btw., I hope I got the stack layout about right and this is even possible) Then, after we entered the continuation of the __DEFAULT alt, we do a jump to g. Plot twist: g returns an unboxed 8-tuple of `Int#`s (as caller_of_f_cont_info knows, but f certainly doesn't!), so before it returns it will push two args on the stack (correct?): caller_of_f_cont_info seq_h unset_bit_stk_info unboxed tuple component 7 unboxed tuple component 8 <- Sp And then `g` jumps directly to the entry code for `unset_bit_stk_info` (which does the register saving I mentioned), which absolutely can't figure out from Sp alone where seq_h is. Darn! I think Luite's recent work on the StgToByteCode had to work around similar issues, I found this wiki page <https://gitlab.haskell.org/ghc/ghc-wiki-mirror/-/blob/master/Unboxed-Tuples…>. But we aren't in a position where we know the representation of `r` *at all*! So our idea was to scan the stack, beginning from `Sp`, until I find `unset_bit_stk_info`, giving us the following situation: caller_of_f_cont_info seq_h unset_bit_stk_info <- Bp unboxed tuple component 7 unboxed tuple component 8 <- Sp I suggestively named the register in which we store the result Bp in analogy to the traditional base pointer. This information would be enough to unset the bit at index seq_h and then copy the unboxed tuple components 7 and 8 up by two words: caller_of_f_cont_info unboxed tuple component 7 unboxed tuple component 8 <- Sp Then jump to caller_of_f_cont_info, which knows what to make of the stack and the register state. The stack scanning is horrible and too brittle and slow for a production setting, as any of the unboxed tuple components could have the same bit pattern as `unset_bit_stk_info`. We simply hope that is never the case and it's fine for the purposes of a quick-and-dirty instrumentation. QUESTION 1: What do you think? Could this work? Can you anticipate pit falls of this approach? # What about Thunks and StgRhsCon? QUESTION 2: The instrumentation as described won't work for Thunks (which are only entered once) and constructor applications (like sat_sBK in the BigNat matcher above). Not sure how to change that without incurring huge performance hits (by deactivating memoisation). Ideas are welcome here. Thanks for reading this far. I hope you could follow and are now fizzing with excitement because you have a much better idea of how to do this and will let me know :) Sebastian

2 4

Re: Thoughts on async RTS API?
by Cheng Shao 16 Dec '21

16 Dec '21

Hi Alex, Thanks for reminding. hs_try_put_mvar() wouldn't work for our use case. If the C function finishes work and calls hs_try_put_mvar() synchronously, in Haskell takeMVar wouldn't block at all, which is all fine. However, if the C function is expected to call hs_try_put_mvar() asynchronously, the non-threaded RTS will hang! Here's a minimal repro. It works with non-threaded RTS at first, but if you change scheduleCallback() in C so hs_try_put_mvar() is only invoked in a detached pthread, then the program hangs. The proposed async RTS API and related scheduler refactorings can't be avoided, if the MVar is intended to be fulfilled in an async manner, using the non-threaded RTS, on a platform with extremely limited syscall capabilities. ```haskell import Control.Concurrent import Control.Exception import Foreign import Foreign.C import GHC.Conc main :: IO () main = makeExternalCall >>= print makeExternalCall :: IO CInt makeExternalCall = mask_ $ do mvar <- newEmptyMVar sp <- newStablePtrPrimMVar mvar fp <- mallocForeignPtr withForeignPtr fp $ \presult -> do (cap,_) <- threadCapability =<< myThreadId scheduleCallback sp cap presult takeMVar mvar peek presult foreign import ccall "scheduleCallback" scheduleCallback :: StablePtr PrimMVar -> Int -> Ptr CInt -> IO () ``` ```c #include "HsFFI.h" #include "Rts.h" #include "RtsAPI.h" #include <unistd.h> #include <pthread.h> struct callback { HsStablePtr mvar; int cap; int *presult; }; void* callback(struct callback *p) { usleep(1000); *p->presult = 42; hs_try_putmvar(p->cap,p->mvar); free(p); return NULL; } void scheduleCallback(HsStablePtr mvar, HsInt cap, int *presult) { pthread_t t; struct callback *p = malloc(sizeof(struct callback)); p->mvar = mvar; p->cap = cap; p->presult = presult; // pthread_create(&t, NULL, callback, p); // pthread_detach(t); callback(p); } ``` On Thu, Dec 16, 2021 at 12:10 PM Alexander V Vershilov <alexander.vershilov(a)gmail.com> wrote: > > Hello, replying off-the thread as it would be basically an offtopic. > > But you can achieve the solution using MVars only. > The idea is that you can call mkStablePtr on the MVar that way it will > not be marked as dead, so RTS will not exit. > Then you can use hs_try_put_mvar in C thread to call the thread back. > > On Wed, 15 Dec 2021 at 05:07, Cheng Shao <cheng.shao(a)tweag.io> wrote: > > > > Hi devs, > > > > To invoke Haskell computation in C, we need to call one of rts_eval* > > functions, which enters the scheduler loop, and returns only when the > > specified Haskell thread is finished or killed. We'd like to enhance > > the scheduler and add async variants of the rts_eval* functions, which > > take C callbacks to consume the Haskell thread result, kick off the > > scheduler loop, and the loop is allowed to exit when the Haskell > > thread is blocked. Sync variants of RTS API will continue to work with > > unchanged behavior. > > > > The main intended use case is async foreign calls for the WebAssembly > > target. When an async foreign call is made, the Haskell thread will > > block on an MVar to be fulfilled with the call result. But the > > scheduler will eventually fail to find work due to empty run queue and > > exit with error! We need a way to gracefully exit the scheduler, so > > the RTS API caller can process the async foreign call, fulfill that > > MVar and resume Haskell computation later. > > > > Question I: does the idea of adding async RTS API sound acceptable by > > GHC HQ? To be honest, it's not impossible to workaround lack of async > > RTS API: reuse the awaitEvent() logic in non-threaded RTS, pretend > > each async foreign call reads from a file descriptor and can be > > handled by the POSIX select() function in awaitEvent(). But it'd > > surely be nice to avoid such hacks and do things the principled way. > > > > Question II: how to modify the scheduler loop to implement this > > feature? Straightforward answer seems to be: check some RTS API > > non-blocking flag, if present, allow early exit due to empty run > > queue. > > > > Thanks a lot for reading this, I appreciate any suggestions or > > questions :) > > > > Best regards, > > Cheng > > _______________________________________________ > > ghc-devs mailing list > > ghc-devs(a)haskell.org > > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > > > -- > -- > Alexander

1 0

Thoughts on async RTS API?
by Cheng Shao 15 Dec '21

15 Dec '21

Hi devs, To invoke Haskell computation in C, we need to call one of rts_eval* functions, which enters the scheduler loop, and returns only when the specified Haskell thread is finished or killed. We'd like to enhance the scheduler and add async variants of the rts_eval* functions, which take C callbacks to consume the Haskell thread result, kick off the scheduler loop, and the loop is allowed to exit when the Haskell thread is blocked. Sync variants of RTS API will continue to work with unchanged behavior. The main intended use case is async foreign calls for the WebAssembly target. When an async foreign call is made, the Haskell thread will block on an MVar to be fulfilled with the call result. But the scheduler will eventually fail to find work due to empty run queue and exit with error! We need a way to gracefully exit the scheduler, so the RTS API caller can process the async foreign call, fulfill that MVar and resume Haskell computation later. Question I: does the idea of adding async RTS API sound acceptable by GHC HQ? To be honest, it's not impossible to workaround lack of async RTS API: reuse the awaitEvent() logic in non-threaded RTS, pretend each async foreign call reads from a file descriptor and can be handled by the POSIX select() function in awaitEvent(). But it'd surely be nice to avoid such hacks and do things the principled way. Question II: how to modify the scheduler loop to implement this feature? Straightforward answer seems to be: check some RTS API non-blocking flag, if present, allow early exit due to empty run queue. Thanks a lot for reading this, I appreciate any suggestions or questions :) Best regards, Cheng

2 2