- Libraries - Haskell.org

RE: [GHC DevOps Group] Making GHC's fast release cadence work
by Simon Peyton Jones 09 Jul '19

09 Jul '19

It seems to me that both questions are sensible ones: 1. What should GHC’s release cadence be? For example, would annual be better than semi-annual? Herbert and Michael both think so, and they know a thing or two. I’d love to hear from more people on this question. 1. When we do make a release, whenever that is, how we can work most productively with library authors, so that the release comes out in a timely way? This was Ben’s original question, and it applies regardless of cadence. Simon From: Ghc-devops-group <ghc-devops-group-bounces(a)haskell.org> On Behalf Of Michael Snoyman Sent: 02 July 2019 05:54 To: Herbert Valerio Riedel <hvriedel(a)gmail.com> Cc: David Feuer <david.feuer(a)gmail.com>; ghc-devops-group(a)haskell.org; Ashley Yakeley <ashley(a)semantic.org>; Judah Jacobson <judah.jacobson(a)gmail.com>; David Terei <code(a)davidterei.com>; Herbert Valerio Riedel <hvr(a)gmail.com>; Haskell Libraries (libraries(a)haskell.org) <libraries(a)haskell.org> Subject: Re: [GHC DevOps Group] Making GHC's fast release cadence work On Tue, Jul 2, 2019 at 12:12 AM Herbert Valerio Riedel <hvriedel(a)gmail.com<mailto:hvriedel@gmail.com>> wrote: On Mon, Jul 1, 2019 at 8:09 PM Ben Gamari <ben(a)well-typed.com<mailto:ben@well-typed.com>> wrote: > Naturally, delays like this make it hard for GHC to maintain its faster release cycle > ... > How do you think we might speed up this process? IMO You're asking the wrong question. This seems based on a premise that everyone agreed with a faster release cycle... to me the downsides on the ecosystem and infrastructure of a faster release churn significantly outweight the modest benefit some people might perceive; and the issue's we've been observing (not only boot libraries, but also the 10k packages on Hackage) with the overspeeded release cycle are IMO a sign that we're moving faster than the ecosystem can accommodate. Just because GHC HQ managed to optimize their release processes (and I have to say, at the expense of the release quality -- GHC 8.6.5 was the first major release since GHC's beginning to require five attempts -- and I have to note that this results in annoying busy work for GHC packagers like myself) doesn't mean that everyone else has the time and energy to adapt to this new order as well. +1

10 15

Proposal: add HasCallStack for all partial functions in base
by LuoChen 08 Jul '19

08 Jul '19

Motivation Partial functions in base (especially Prelude) often cause runtime errors and is hard to locate. (here <https://gist.github.com/luochen1990/6fcbb0fa4c0c7caa1b242188eb586d1f> is a document about the concept of totality and partial function, ignore this if you are familiar with them) For example, consider the following piece of code: import GHC.Stack foo :: HasCallStack => [Int] -> Int foo xs = last xs + 1 xs :: [Int] xs = [] main :: IO () main = do print $ foo xs In this case, the error message will tell nothing about foo, and the HasCallStack constraint is totally helpless, because the call stack is cut off by the call to last which without HasCallStack constraint. My current workaround is define my own wrapper functions with HasCallStack constraint for some mostly used partial functions to make them traceable, and use the wrapper (the traceable version) whenever I need them. e.g. last' :: HasCallStack => [a] -> a last' xs = case xs of [] -> error "abuse last"; _ -> last xs So, IMHO, if our goal is to make errors in haskell traceable, then *only providing HasCallStack mechanism is not enough, we have to provide traceable base package and prelude at the same time*. Further more, all untraceable partial functions are considered to be harmful, and should not be exported by any package. Because an improper call to an untraceable partial function will cut off the call stack, and here is a demonstration about that <https://gist.github.com/luochen1990/94179a2492ff7d1ca45153645f1bb449>. On the other hand, is it ever necessary for us to add HasCallStack for a total function? Or we can ask, is it possible that a call to a total function cause runtime error? Maybe it’s a NO, since a total function will also crash when the machine is Out Of Memory, but that is also the only situation I can find out. So I suggest that we add HasCallStack constraint only for partial functions, and IMHO this could be a good balance for better debugging experience and less runtime overhead. Proposal 1. add HasCallStack constraint for all partial functions in base package 2. suggest all programmers to add HasCallStack constraint for their exported partial functions when they release a package 3. provide a compiler option -fignore-hascallstack to toggle off the effect of HasCallStack constraint in case somebody need best performance Other Considerations How to get a full list of partial functions provided by the base package? I wanted to provide a full list of partial functions exported by the base package in this post, but I find it is hard, since Haskell have no totality checking mechanism like Idris have, and there are no consistent keyword like “total” or “partial” in document, so it takes a lot of work to list all the partial functions of a package by check every item manually. Maybe we can work on this list later — when it’s turned out to be worth after some discussion. Here <https://gist.github.com/luochen1990/5b7a0844701413486da595b9b997f3c2> is part of the list that I have tidied for several modules of the base package. How to encourage all package contributors to obey the rule (see proposal #2)? I don’t know, but I think there may be some other rules to obey when contributing packages. Maybe we can just add this into the list. Obviously, the final perfect solution should be let the compiler to check the totality of functions, and automatically add HasCallStack for the ones which the compiler cannot confirm it’s totality. But this seems too far away from us, since we still doesn’t have dependent type haskell yet. How to deal with recursive partial functions? Since the HasCallStack constraint affects the performance (not only because of the runtime overhead, but also it’s influence on optimization strategy), It is best not to add HasCallStack on recursive functions. In most of the cases, we can just check the input shallowly before everything start, just like how we deal with the non-recursive ones. But in some other cases, we need to go deep to recognize the invalidity of the input. A trivial solution is just perform a deep check before everything start, but the checking phase seems expensive. The best solution, IMHO, is to make the recursive part a total function, wrap the return value into Maybe or some similar things, and the partial version is just the total version combined fromJust. In this way, we avoid the single input checking phase and left the error awareing logic where it was before this translation.

9 18

Proposal: composition util for Maps
by Alexandre Esteves 07 Jul '19

07 Jul '19

In https://github.com/haskell/containers/issues/647 I proposed the following util: composition :: Ord b => Map b c -> Map a b -> Map a c composition bc ab = flip mapMaybe ab $ flip lookup bc Which has O(|ab| * log |bc|) performance. It's not particularly hard to write, but it does feel like a primitive-ish operation, and it's not obvious to me whether there's a faster implementation. Other name suggestions - (.) - chain - compose

6 6

Install GHC from source with profiling build of base
by Andrew Martin 02 Jul '19

02 Jul '19

When I build GHC from source, I can see a bunch of .p_hi files under libraries/base. This makes sense given that the default for GhcLibWays is "v p". But when I run "make install" all the profiling stuff gets stripped out: amartin@thadtop:~/.local/lib/ghc-8.9.0.20190628$ find . | grep -o '\.[^./0-9]*$' | sort -u . .a .cache .conf .d .dyn_hi .h .hi .lock .o .so .txt Not a profiling-related artifact in sight. And this local build rightfully fails to build anything with profiling turned on. So, how do I actually install the profiling builds of the wired in libraries? -- -Andrew Thaddeus Martin

1 0

Re: [GHC DevOps Group] Making GHC's fast release cadence work
by Boespflug, Mathieu 02 Jul '19

02 Jul '19

A related discussion came up previously: https://mail.haskell.org/pipermail/ghc-devops-group/2017-December/thread.ht…. >From the moment that GHC accepts unreleased dependencies on its release branch, GHC developers lose full control over their own release timeline. Because a new GHC release can only ship when the dependencies have been released, the release process could block on upstream for indefinite periods of time, like is currently happening with at least one GHC dependency that has been blocked on the maintainer for at least 27 days (and counting). It was suggested then that GHC should only use release versions of its dependencies on the master or release branch, and that if this isn't practical due to co-dependency, then that suggests said dependencies should have the same release cycle as GHC (i.e. same people can cut releases of both) or even be merged in GHC itself. -- Mathieu Boespflug Founder at http://tweag.io. On Mon, 1 Jul 2019 at 21:03, David Feuer <david.feuer(a)gmail.com> wrote: > I realize that my response came off as rather unfriendly. I'm sorry about > that. Thinking a bit more about it, I think GHC HQ can help Core Library > maintainers in several unintrusive ways. Here are a couple I would > appreciate myself; others may disagree. > > 1. Work with Herbert, Ryan, etc., to make it easy to use -Werror in Travis > scripts. See https://github.com/haskell/containers/pull/645 > > 2. Give us a nudge when you have a release timeframe in mind. "We're > hoping to have the core libraries ready for release by December 5" (several > weeks beforehand, repeated a couple times thereafter with appropriate > adjustments) is much less stressful than "We're basically ready for release > but we're waiting on core libraries." > > On Mon, Jul 1, 2019, 1:30 PM David Feuer <david.feuer(a)gmail.com> wrote: > >> I think it would be nice to start by letting the core library maintainers >> know your preferred release deadlines. >> >> On Mon, Jul 1, 2019, 1:22 PM Ben Gamari <ben(a)well-typed.com> wrote: >> >>> Hi everyone, >>> >>> GHC's core libraries are a critical part of the Haskell ecosystem. I >>> want to thank you for overseeing the maintenance of this infrastructure. >>> >>> However, for the last three weeks the release candidate for GHC 8.8.1 >>> has been ready aside from releases of a couple of our core libraries. >>> >>> Naturally, delays like this make it hard for GHC to maintain its faster >>> release cycle. At the same time, we do not want this cadence to impose >>> an undue burden on our core library maintainers. >>> >>> How do you think we might speed up this process? >>> >>> For instance, perhaps the GHC release manager could pick up >>> some of the "boring parts" of core library maintenance limited to: >>> >>> * Version bound bumps >>> >>> * Changes of CPP conditionals to accommodate changes in the >>> compiler and other core libraries >>> >>> * Changelog entries to describe these releases >>> >>> * Uploading these releases or revisions to Hackage >>> >>> Of course, this would merely be an offer to maintainers; this would be >>> GHC's way of carrying some of the burden that our release process >>> imposes. >>> >>> In general, I am interested in a discussion on how to make this faster >>> release pace work. Ideas? >>> >>> Cheers, >>> >>> - Ben >>> >>> _______________________________________________ > Ghc-devops-group mailing list > Ghc-devops-group(a)haskell.org > https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devops-group >

2 2

(no subject)
by Andrew Martin 01 Jul '19

01 Jul '19

To get GHC to raise an exception from an inline primop, I presume that I'd need to jump to stg_raisezh. None of the existing inline primops do anything quite like this. I see things like emitMemcmpCall and emitMemsetCall, but these ultimately just wrap emitForeignCall, which wraps mkUnsafeCall, which wraps the data constructor CmmUnsafeForeignCall. I think I want CmmCall instead, which is described by the comments as being used for "a native call or tail call". Hmm... emitRtsCall might be what I want. I'll try pursuing this route further. -- -Andrew Thaddeus Martin

2 1

Making Int{Map,Set} polymorphic over the key type
by Georg Rudoy 29 Jun '19

29 Jun '19

Hi folks, What do you think about making IntMap and IntSet polymorphic over the keys as long as the key type k is Coercible to Int? The motivation is that oftentimes one needs to have a collection of newtype wrappers around Int, and then they have to either unpack/repack it around the relevant containers (which is not nice), or just use HashMap/HashSet (which is perhaps suboptimal). If this is a direction the community might deem reasonable, I'd be happy to work on this and make a PR to the containers repo (perhaps after discussing the specific details about naming, etc). -- Georg Rudoy

4 7

Fwd: lazier writer monad
by Tristan Wibberley 26 Jun '19

26 Jun '19

Dear Haskell Libraries mailing list members, I've submitted a merge request for an increase in the laziness of the writer monad: https://gitlab.haskell.org/ghc/ghc/merge_requests/1262 and it requires discussion via this mailing list. The link above includes an example of the motivation for this change and I hope you can all extrapolate to see the importance of this change, sufficient to not only accept this but to be moved to add a language flag, available within the content of a module's source file, to match unique constructors lazily throughout a module that uses it, lifting _|_ to the next least-defined value. The core problem is that for the most handy and practical library interfaces, we need function results to be as well-defined as possible before a strict pattern match is required to choose between two constructors/values. There are several parts to providing that but one important part is that elements of patterns that cannot be unmatched should be lazily matched so the function's definition can be used without evaluating the function's arguments first. This avoids having to train users to always avoid patterns except after a let binding which would otherwise be necessary to establish a standard habit to lazily pattern match so the user base encourages each other to make their functions maximally useful. -- Tristan

3 2

transformers mirroring to GitHub
by chessai . 25 Jun '19

25 Jun '19

Could we mirror transformers from darcshub to GitHub? This would make browsing the current source easier for those of us who do not frequent darcshub. It would also help with the uniformity of having core libs on GitHub. Of course this would just be a mirror, and users would be advised to not open PRs there. Not sure how easy mirroring from DarcsHub to GitHub is. Thoughts? Thanks

2 2

Re: Data.Graph transitive closure
by Johannes Waldmann 24 Jun '19

24 Jun '19

> ... reasonably efficient transitive closure for Data.Graph ... well, since we have type Graph = Array Vertex [Vertex] efficiency is already questionable? We cannot quickly check whether an edge is present, we cannot quickly get all predecessors of a vertex. But perhaps we don't need to. The underlying assumptions (cost of elementary operations) of http://www.cs.hut.fi/~enu/thesis.html are not immediately clear. The cited Agda library has newtype Graph n e = Graph (Map n (Map n e)) Another option is to store back edges as well, as in https://github.com/haskell-works/relation/blob/master/src/Data/Relation/Int… FGL fuses these two maps type GraphRep a b = IntMap (Context' a b) type Context' a b = (IntMap [b], a, IntMap [b]) https://hackage.haskell.org/package/fgl-5.7.0.1/docs/src/Data.Graph.Inducti… this saves some look-ups (if you need successors and predecessors of the very same node) but this can't be used for relations, where start and end of an edge have different types. But for transitive closure, these types would agree anyway. What to make of this? Perhaps Data.Graph should be moved out of containers. The design space is just too large to single out one specific point? If we keep it, it would be very nice to document the complexity of algorithms. Section 7 of the King/Launchbury paper (cited in the API docs) claims "DFS in O(V+E)", backed up by experiments. This seems to be the main motivation for this library (DFS, with application: SCC). It's not clear whether the underlying design should be recommended for a general graph library. - Johannes

6 12