RE: Hadrian

Hi Mathieu, Yes, in principle we can merge right now, as long as it's clear that Hadrian still requires more work before taking over. My only concern is that merging will make it more difficult for us to quickly iterate on Hadrian: the GitHub workflow is more convenient (at least for me) than the Phabricator one. Perhaps, we can keep Hadrian on GitHub as a submodule? This also has the advantage that we can keep all existing references to GitHub issues/PRs without migrating everything to GHC Trac. It would be very unfortunate to lose all history during the merge. Cheers, Andrey -----Original Message----- From: Boespflug, Mathieu [mailto:m@tweag.io] Sent: 19 October 2017 19:21 To: Andrey Mokhov Cc: Ben Gamari ; Manuel M T Chakravarty ; Moritz Angermann ; Jonas Pfenniger Chevalier ; ghc-devs Subject: Re: Hadrian Hi all, As a user who tried to be an early adopter of Hadrian, I can attest to Andrey's remarks about GHC progress sometimes (frequently?) breaking Hadrian. Ben, Andrey - how about this strawman proposal: we merge Hadrian into GHC *now*, not because ./validate with Hadrian works (it doesn't yet), but because the build of GHC succeeds with Hadrian. The resulting compiler might well be garbage. But that's fine - we can iterate in the official ghc repo, all the while knowing that CI has our back if ever some change makes Hadrian no longer even build a compiler. Once ./validate passes with Hadrian, the guarantee that CI gives will become stronger still: we'll know if any change would make the Hadrian-produced compiler garbage again. Maybe that does sound totally bonkers to you? :) Maybe it's radical, but sounds to me like the best way to get the benefit *now* of ensuring Make and Hadrian move forward in lock step thanks to CI. The above all assumes that we're committed to Hadrian being the future of GHC's build system. But I take it that's a given by now. Best, Mathieu On 19 October 2017 at 19:05, Andrey Mokhov wrote:

Hi Ben, Manuel and all,

Ben has already covered most questions in his answer, but let me add a couple of comments.

...

So, Mathieu had the clever idea that having the two build system in GHC side-by-side and then build in CI both ways might be a good way of making sure that keep constant tabs on Hadian and get a clear (and continuous picture) of where it is and what is missing.

It would be great to build every GHC commit both by Make and Hadrian. Not only this would allow to monitor the readiness of Hadrian, it would also make it easier for us to catch up with changes in GHC and the Make build system. At the moment, Hadrian is frequently broken by changes in GHC/Make and we need to do detective work of figuring out which commit broke what and fix Hadrian accordingly. These fixes are trivial in many cases (e.g. adding a new flag to one of the builders) so GHC developers would probably be able to easily mirror these changes in Hadrian if only their commits were marked as Hadrian-breaking by the common CI.

...

In other words, why can’t we have Hadrian *today*?

I'd say the biggest issue is #299 -- i.e. making sure that the GHC built by Hadrian passes validation. At the moment we still have 100-200 unexpected failures, most of which probably have to do with outdated command line flags (GHC's build system continuously evolves and it wasn't possible to keep track of all flag changes over time, so there are certainly some differences in Hadrian that lead to validation failures). This is where help is currently needed.

Cheers, Andrey

-----Original Message----- From: Ben Gamari [mailto:ben@well-typed.com] Sent: 19 October 2017 15:08 To: Manuel M T Chakravarty Cc: Mathieu Boespflug ; Moritz Angermann ; Jonas Pfenniger Chevalier ; Andrey Mokhov Subject: Re: Hadrian

CCing Andrey and Zhen, who are the principle authors of Hadrian.

Manuel M T Chakravarty writes:

...

Hi Ben,

I our exploration of cross-compilation for GHC and the CI integration, we talked with Moritz and got to the topic of Hadrian. It seems that there are few interdependent issues here, and I am really interested in getting your take on them.

* Hadrian is slated for inclusion for 8.4, but I couldn’t find any tickets except #12599. Is this because Hadrian has its own issue tracker on GitHub? How can I get an overview over what’s missing in getting Hadrian into GHC?

Right, Hadrian tickets are currently tracked in its own issue tracker.

...

* For Moritz’ work on ARM and also for CI cross-compiling is important, but Moritz pointed out rightly that it seems wasteful put much time into improving this in the make-based build system if that is going to go soon anyway. However, because Hadrian is not in yet, doing the work on Hadrian now also doesn’t really make sense.

Indeed. Hadrian is one of the things I am currently working on for precisely this reason.

...

* Obviously, one issue with Hadrian is stability and feature completeness. This has ramifications on our desire to really get 8.4 out of the door in February and not delay the release and also get the CI done, because good CI is the best way of making sure the Hadrian integration goes smoothly. (There is obviously a cyclic dependence between this point and the previous.)

So, Mathieu had the clever idea that having the two build system in GHC side-by-side and then build in CI both ways might be a good way of making sure that keep constant tabs on Hadian and get a clear (and continuous picture) of where it is and what is missing. It would also allow Moritz to do cross-compilation work right there on Hadrian.

In other words, why can’t we have Hadrian *today*?

At this point the plan is for Hadrian and the Make-based build system to coexist in 8.4, with the make system serving as a backup.

Well, we could (and potentially should) merge Hadrian

...

If the limiting factor is resources, maybe we can do something about that, but it would require a good solid plan that we can execute perfectly when given the opportunity.

The blocking issues in Hadrian are tagged with the "GHC Merge" project in the issue tracker. The overall themes currently are,

* Fixing validation support (finishing #187)

* Support the remaining test targets (#360)

* Documentation, so we don't overwhelm Andrey with questions when things break. (#308)

* Finish and test install rules (#318, #373, #379)

* bindist install (the last part of #219). I started working on this, although for the moment bugs and 8.2.2 have taken priority.

Andrey, do correct me if I've missed something.

Help is of course always welcome!

Cheers,

- Ben