The most friendly option to developers would be not building the perf

build by default but this has been repeatedly shot down in the past.

I agree. But I also find it reasonable to make it so one won't accidentally build a dev build when trying to make a release. (that being said, in the Ocaml world, things have been going the other way, where, if I understand correctly, `dune build` builds development version, while `dune build @install` builds release artifact). I can't say I have a strong opinion on this. 1. -c doesn't work very well compared to ./configure for me and picks

up the wrong versions of alex/happy as installed by hadrian rather than the ones provisioned by ghc.nix/. I have tried to avoid it anyway after encountering some problems.

If anybody else have such issues with `-c`, please share your experience in the thread. I haven't had any problem so far, myself.

On Mar 15, 2019, at 2:19 AM, Herbert Valerio Riedel wrote: On Thu, Mar 14, 2019 at 4:20 PM Spiwack, Arnaud wrote: • The -c option should be the default. Very strong -1 from me on this one; I've been quite vocal on the Hadrian issue tracker early on and multiple times against having Hadrian invoke ./configure at all, even more so against having it do so by default. I don't have the ticket number at my fingertips but it should be fairly easy to find.

I'm with Herbert here. I think that the `-c` flag should be mandatory if you want hadrian to invoke autoconf magic. I believe the confusion might stem from the newcomers guide[1]? I'd rather see the newcomers guide *not* use `-c`, and instead make it obvious to call `boot` and `configure`. These are essential steps and hiding them makes them less obvious. `boot` does - (1) checks that url rewrites are in place. - (2) checks that all bootpackages are available - (3) run autoreconf as needed - (4) and generates a bunch of make files for the make based build system. (4) can be disabled by passing `--hadrian` to `boot`. (1) is needed due to the relative submodules I beleive. `configure` generates the necessary configuration files based on the configure flags passed. And hadrian does the actual build step that `make` used to do. It's magically conflating two different phases with `-c`. The configure phase and the build phase. Making this the default means it's always magic. I don't like magic! Cheers, Moritz -- [1]: https://github.com/tdammers/ghc-wiki/blob/wip/newcomers/newcomers-tutorial.m...

Hi Arnaud,

> On Mar 15, 2019, at 8:32 PM, Spiwack, Arnaud  wrote:
> 
> On Thu, Mar 14, 2019 at 7:20 PM Herbert Valerio Riedel  wrote:
> I don't have the ticket number at my fingertips but it should be fairly easy to find.
> 
> I'm afraid it doesn't appear to be. Could you share your arguments in this thread?
This was the last one that lead to the current `-c` state:
- https://github.com/snowleopard/hadrian/issues/457
There is also
- https://github.com/snowleopard/hadrian/issues/655

if you look through the issues on snowleopard/hadrian and sort by comment frequency
you'll likely find quite a lot of further discussion about not making configure and
boot the default.

> 
> On Fri, Mar 15, 2019 at 3:10 AM Moritz Angermann  wrote:
> It's magically conflating two different phases with `-c`. The configure phase and
> the build phase. Making this the default means it's always magic. I don't like magic!
> 
> Unfortunately, I really don't understand what you are saying. What's magic about combining the phases?

We have two phases:

Phase 1: autoconf

  This phase is essentially a code-generation phase, where specific templates are
  instantiated to configure time value.  Which again can be split into two specific
  subproblems:

  - Generation of the configure script from the configure.ac and aclocal.m4 files
    using autoconf.
  - Generating code using the configure script by computing configure time calues
    and filling those into the `.in` files producing the files that lack the `.in`
    extension.

Phase 2: building

  This has been traditionally the job of make, and this is what hadrian should
  replace.


By subsuming the configure phase (by invoking ./configure) from hadrian we loose
the phase distinction and if the `-c` flag is optional, users will *not even see*
a flag that indicates that the system will run `./configure` for them. This is the
magic I'm referring to and to which I strongly object.  If we can retire autoconf
and do the whole configuration in hadrian, that story may change.  But as long as
we are using an autoconf based configuration we should *not* run that magically.
The `-c` flag is at least there to show that hadrian is explicilty instructed to
run configure.

./configure supports its own set of flags, if hadrian subsumes those, we'd need
some generic way of passing flags to ./configure, at which point I have to ask
why do we do this in the first place and try to call ./configure from within hadrian?

Unless you want to reconfigure ghc, or hack on it's autoconf part, you are likely
going to run the following only:

./boot --hadrian
./configure 
./hadrian/build.sh -j ...
./hadrian/build.sh -j ...
./hadrian/build.sh -j ...
./hadrian/build.sh -j ...
...

the configure step is required, and should be explicit. That is where you configure
your ghc build. Set host/build/target values, and other configure flags that
influence how you want your ghc to be configure. Hadrian is there to build that
configuration.  Mixing both may be convenient but hides the fact that there is a
./configure step.  I consider this hiding to be magic which is meant to benefit the
user but hides what's really going on.  And again I don't like magic.

Cheers,
 Moritz

PS: we also don't hide the `./configure` step in the usual `./configure  && make -j`
    instructions when building other software, even though you could surely hack that into
    your Makefile if you so wanted to.  Why start with ghc now?

Agreed, I was also an opponent of -c as the default, and as you pointed out it only works for the case where the default is used. But even if the defaults are used it is still harmful to do it automatically as the user's environment could have changed resulting in different configure output if you automatically rerun it. Not only isn't there a sane UX for this, there isn't even a safe way to do this. Sent from my Mobile On Fri, Mar 15, 2019, 16:51 Alp Mestanogullari wrote:

I have to admit I sympathize with Moritz's view. Since `-c` only "subsumes" the case where we call 'configure' with no extra env var or argument, and in the absence of a generic way to pass options to 'configure' when using -c, I'd quite like to keep -c as a "cherry on top", for users who just need to boot and configure with the default arguments and for whom hadrian provides a way to save a few keystrokes. Just like Moritz, I'm not even sure it would make all that much sense to provide a way to pass configure arguments through hadrian, I have a hard time seeing a better UX than just running configure with the extra arguments directly and _then_ calling hadrian to start the build. On 15/03/2019 14:35, Moritz Angermann wrote:

Hi Arnaud,

On Mar 15, 2019, at 8:32 PM, Spiwack, Arnaud wrote:

On Thu, Mar 14, 2019 at 7:20 PM Herbert Valerio Riedel wrote: I don't have the ticket number at my fingertips but it should be fairly easy to find.

I'm afraid it doesn't appear to be. Could you share your arguments in this thread?

This was the last one that lead to the current `-c` state: - https://github.com/snowleopard/hadrian/issues/457 There is also - https://github.com/snowleopard/hadrian/issues/655

if you look through the issues on snowleopard/hadrian and sort by comment frequency you'll likely find quite a lot of further discussion about not making configure and boot the default.

On Fri, Mar 15, 2019 at 3:10 AM Moritz Angermann wrote: It's magically conflating two different phases with `-c`. The configure phase and the build phase. Making this the default means it's always magic. I don't like magic!

Unfortunately, I really don't understand what you are saying. What's magic about combining the phases?

We have two phases:

Phase 1: autoconf

This phase is essentially a code-generation phase, where specific templates are instantiated to configure time value. Which again can be split into two specific subproblems:

- Generation of the configure script from the configure.ac and aclocal.m4 files using autoconf. - Generating code using the configure script by computing configure time calues and filling those into the `.in` files producing the files that lack the `.in` extension.

Phase 2: building

This has been traditionally the job of make, and this is what hadrian should replace.

By subsuming the configure phase (by invoking ./configure) from hadrian we loose the phase distinction and if the `-c` flag is optional, users will *not even see* a flag that indicates that the system will run `./configure` for them. This is the magic I'm referring to and to which I strongly object. If we can retire autoconf and do the whole configuration in hadrian, that story may change. But as long as we are using an autoconf based configuration we should *not* run that magically. The `-c` flag is at least there to show that hadrian is explicilty instructed to run configure.

./configure supports its own set of flags, if hadrian subsumes those, we'd need some generic way of passing flags to ./configure, at which point I have to ask why do we do this in the first place and try to call ./configure from within hadrian?

Unless you want to reconfigure ghc, or hack on it's autoconf part, you are likely going to run the following only:

./boot --hadrian ./configure <your flags of choice> ./hadrian/build.sh -j<N> ... ./hadrian/build.sh -j<N> ... ./hadrian/build.sh -j<N> ... ./hadrian/build.sh -j<N> ... ...

the configure step is required, and should be explicit. That is where you configure your ghc build. Set host/build/target values, and other configure flags that influence how you want your ghc to be configure. Hadrian is there to build that configuration. Mixing both may be convenient but hides the fact that there is a ./configure step. I consider this hiding to be magic which is meant to benefit the user but hides what's really going on. And again I don't like magic.

Cheers, Moritz

PS: we also don't hide the `./configure` step in the usual `./configure <args> && make -j` instructions when building other software, even though you could surely hack that into your Makefile if you so wanted to. Why start with ghc now?

_______________________________________________ ghc-devs mailing listghc-devs@haskell.orghttp://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

-- Alp Mestanogullari, Haskell Consultant Well-Typed LLP, https://www.well-typed.com/

Registered in England and Wales, OC335890 118 Wymering Mansions, Wymering Road, London, W9 2NF, England

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

On Fri, Mar 15, 2019 at 4:52 PM Brandon Allbery wrote:

Newcomers to autoconf-based ecosystems often add a rule to run configure to their Makefiles, as a "shortcut". At some time thereafter, they discover that there's a problem when configure alters the build plan make is executing. There's no sane way to integrate this properly if the build plan changes enough, even if configure is always the very first thing executed as part of the original build plan. This generally results in forcing make to abort after running autoconf, since trying to be more clever results in looping over configure. The only other way that works is arranging for 2-stage Makefiles where the outer one amounts to "configure && make -f Makefile.real". Which makes it noticeably more error-prone to tweak something and do a fast rebuild, since configure will often cause it to be a full rebuild instead. If you try to give configure dependencies so that it doesn't get re-run, you discover that it's dependent in some sense on its own outputs and again can cause loops.

Trying to integrate configure with hadrian runs the same risk, heightened by hadrian not using anything like Makefiles, so it would be even easier for it to either miss changes made by configure or end up looping on it. Or worse, the same with autoconf — or for ghc, the boot script which runs autoconf, among other things, and thus has even nastier risks in terms of missing important changes made thereby or causing loops.

Aha! But is Hadrian doesn't use Make. And the shortcoming of Make which make autoconf necessary as a separate tools are ones that Shake fix. Making it a breeze to integrate a configure script without hassle. I'll be honest, so far, so far, the thread doesn't have much convincing arguments in favour of keeping `-c`. The argument which I understand is Ben's in [ https://github.com/snowleopard/hadrian/issues/457 ], who saw the arguments he passed to ./configure overwritten by Hadrian. This is surprising and unnecessary. But you have to realise what you are advocating for: the most typical workflow (not passing special arguments to ./configure) is the one which is special cased. We should rather take this as a challenge: how do we fix the limitations of `-c` so that it can be made the default; rather than giving up. A possibility would be to record the arguments passed to the configure script in a file, and make Hadrian pass these arguments to `./configure`. For instance. I'd also like to impart a bit of urgency to this: we are breaking everybody's habit now, so now is the time to discuss changes. When we're used to the new build system and it isn't new anymore, it will become much harder to justify change. By the way, on the default flavour: in addition to Ocaml's Dune, it appears that Bazel (Google's newish multi-language build system) also has the developers' build be the default. It's not enough data to establish that this default is the consensus. But I'm quite warming up to the position myself. Best, Arnaud

I feel like you didn't read this completely. I used make as the example, but hadrian is also a build system and will have some version of the same problem: configure's dependencies are also its outputs. Hadrian might be able to handle this, if it recomputes the entire dependency tree after every operation — but any mistakes in the dependencies will lead to loops. And none of this addresses the point that if configure is in any way automatic, including as a separate initialization operation before dependencies are computed, it's going to make life difficult for developers who want to restart a build. Granting the viewpoint that says that end-builder ease is more important than developer ease, but even there if you make things too difficult for developers, a project that is already short on developers may become even more short on developers and/or developer time. (I also think decisions like whether and how to run configure and/or the build tooling should be part of my local configuration, because my own needs will differ from others'. We already see this with devs who prefer e.g. quick vs. perf builds.) By far the safest approach is to keep configure separate. On Tue, Mar 19, 2019 at 4:42 AM Spiwack, Arnaud wrote:

On Fri, Mar 15, 2019 at 4:52 PM Brandon Allbery wrote:

...

Newcomers to autoconf-based ecosystems often add a rule to run configure to their Makefiles, as a "shortcut". At some time thereafter, they discover that there's a problem when configure alters the build plan make is executing. There's no sane way to integrate this properly if the build plan changes enough, even if configure is always the very first thing executed as part of the original build plan. This generally results in forcing make to abort after running autoconf, since trying to be more clever results in looping over configure. The only other way that works is arranging for 2-stage Makefiles where the outer one amounts to "configure && make -f Makefile.real". Which makes it noticeably more error-prone to tweak something and do a fast rebuild, since configure will often cause it to be a full rebuild instead. If you try to give configure dependencies so that it doesn't get re-run, you discover that it's dependent in some sense on its own outputs and again can cause loops.

Trying to integrate configure with hadrian runs the same risk, heightened by hadrian not using anything like Makefiles, so it would be even easier for it to either miss changes made by configure or end up looping on it. Or worse, the same with autoconf — or for ghc, the boot script which runs autoconf, among other things, and thus has even nastier risks in terms of missing important changes made thereby or causing loops.

Aha! But is Hadrian doesn't use Make. And the shortcoming of Make which make autoconf necessary as a separate tools are ones that Shake fix. Making it a breeze to integrate a configure script without hassle.

I'll be honest, so far, so far, the thread doesn't have much convincing arguments in favour of keeping `-c`. The argument which I understand is Ben's in [ https://github.com/snowleopard/hadrian/issues/457 ], who saw the arguments he passed to ./configure overwritten by Hadrian. This is surprising and unnecessary.

But you have to realise what you are advocating for: the most typical workflow (not passing special arguments to ./configure) is the one which is special cased. We should rather take this as a challenge: how do we fix the limitations of `-c` so that it can be made the default; rather than giving up. A possibility would be to record the arguments passed to the configure script in a file, and make Hadrian pass these arguments to `./configure`. For instance.

I'd also like to impart a bit of urgency to this: we are breaking everybody's habit now, so now is the time to discuss changes. When we're used to the new build system and it isn't new anymore, it will become much harder to justify change.

By the way, on the default flavour: in addition to Ocaml's Dune, it appears that Bazel (Google's newish multi-language build system) also has the developers' build be the default. It's not enough data to establish that this default is the consensus. But I'm quite warming up to the position myself.

Best, Arnaud

-- brandon s allbery kf8nh allbery.b@gmail.com

On 2019-03-15 at 13:28:21 -0400, Ben Gamari wrote: [...]

We should keep in mind that users and packagers, not GHC developers, are the primary audience of GHC's build system. For their sake we should strive to keep GHC's build system as close to the usual configure/make/make install workflow as possible.

Actually, even as a GHC developer, I strongly prefer the IMO quite intuitive configure/make phase-separated workflow I've been used to for over 2 decades. It's deeply ingrained into my muscle memory and my tooling automatically knows what to do when it discovers a `configure` script and/or a `GNUmakefile`/`Makefile` file in the filesystem and I'm not excited to have to adapt all my dev scripts/integrations to make them Hadrian-aware which is effectively a bespoken build-system used by exactly one project and starting with a specific version of the project while the previous ones still require Makefiles.

In fact, I have at times wondered whether when we switch to Hadrian as our primary build system we should ship our source distributions with a stub makefile to build and run hadrian, allowing downstreams to treat the distribution as a normal autotools-based package.

Definitely, I've been wanting this as well! It's perfectly fine IMO to have it be a best-effort leaky abstraction. It'd be great if the following standard invocations kept working, which represents the usage pattern I use both in packaging scripts as well as in my dev-workflow scripts. The less I have to be aware of Hadrian pulling the strings behind the curtains for my 80% use-cases the better. And if I need to do something from the remaining 20% of use-cases that's when I'd be willing to invoke Hadrian directly, as that's also when I'd be doing something non-standard/non-routine anyway and I'd be consciously paying attention anyway... ,---- | # an example build.mk with some of the keys I typically set one way or | # another; just support a flat simple key/value grammar as text-file | # config input to Hadrian | | cat > mk/build.mk <