My understanding of this backwards compat logic is that it's only there to allow you to do stuff like: build with hadrian, and then continue using make with the artifacts (partially) built by hadrian. I think this is a horrible idea in and onto itself, even if I can somewhat see the appeal as a gateway drug; in which you'd slowly have hadrian take over parts that make used to do, and use make for the stuff that doesn't work (yet) in hadrian. However, I don't think the benefit of constraining hadrian to work in the make framework makes much sense. We should be permitted to explore new (and better) solutions, that do not align with how the make based build system did things if it allows for a less complex build system or faster builds or ... Cheers, Moritz On Wed, Feb 10, 2021 at 9:28 PM Richard Eisenberg wrote:

This sounds very reasonable on the surface, but I don't understand the consequences of this proposal. What are these consequences? Will this break `make`? (It sounds like it won't, given that the change is to Hadrian.) Does this mean horrible things will happen if I use `make` and `hadrian` in the same tree? (I have never done this, other than with hadrian/ghci, which seems to have its own working directory.) Basically: for someone who uses the build system but does not work on it, how does this affect me? (Maybe not at all!)

I would explicitly like to endorse the direction of travel toward Hadrian and away from `make`.

Richard

...

On Feb 10, 2021, at 8:05 AM, Moritz Angermann < moritz.angermann@gmail.com> wrote:

Hi,

so we've finally run into a case where we need to bump the rts version. This has a great ripple effect. There is some implicit assumption that rts-1.0 will always be true. Of course that was a lie, but a lie we lived with for a long time.

Now, hadrian tries *really* hard to replicate some of the Make based build systems idiosyncrasies, this includes creating versionless symlinks for the rts. E.g. libHSrts<X> -> libHSrts-1.0<X>. There is a great deal of logic just to achieve this, and of course it all crumbles now.

I'd therefore like to float and propose the idea that we agree to *not* bother (too?) much with make based build systems backwards compatibility and warts that grew over the years in the make based build system with hadrian going forward.

Yes, I can probably fix this, and add even more code to this burning pile of complexity, but why? The next person will assume libHSrts does not need to be versioned and continue with this mess.

Let's have Hadrian be a clean cut in some areas (it already is, it does away with the horrible abomination that ghc-cabal is--which only serves the purpose of translating cabal descriptions into make readable files), and not be bogged down by backwards compatibility.

This is thus my call for voicing concern or the upkeep of legacy support, or I'll take silence as the collective support of making hadrian *not* be held back by backwards compatibility. (This would mean in this case, that I'd just delete the backwards compat code instead of adding even more to it).

I hope we all still want Hadrian to replace Make, if not and we want to keep Make, why are we concerning ourselves with Hadrian in the first place. If we are intending to ditch Make, let's not be held back by it.

Cheers, Moritz _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

On Feb 10, 2021, at 8:50 AM, Simon Peyton Jones wrote:

build with hadrian, and then continue using make with the artifacts (partially) built by Hadrian

I almost suggested that this had to be the reason for the back-compat design, but I assumed I had to be wrong. I also agree this is a non-goal; I'm quite happy to be forced to pick one or the other and stick with that choice until blasting away all build products. Richard

Tamar, thanks so much for the backstory and the tickets. I’ll go dig down this path a bit more. Cheers, Moritz On Thu, 11 Feb 2021 at 5:31 PM, Phyx wrote:

Hi, Just leaving my two cents feel free to ignore..

...

I almost suggested that this had to be the reason for the back-compat design

You're right, but not for backwards compat of Hadrian vs Make, but for compat with RTS versions. I could be wrong, but my understanding is the current design in Make is just an artifact of getting something that works on all OSes without much pain, but has proven to be suboptimal in a very important use case (slight detour time):

You have to make a choice of which RTS to use at compile time. Which is quite bad. Because it means that you can't swap between two RTS flavors with the same ABI. It also means building presents a problem, you want your compiler at the end of stage1 to use your new rts, not the one of the stage0 compiler.

You can't have multiple versions of the RTS in one library, but if you have the full name as a dependency the dynamic loader happily loads you multiple copies.

To solve this issue the design was made to not declare the RTS as a dependency on any haskell library. i.e. there's not DT_NEEDED entry for it on ELF operating systems. Which means before you load a Haskell produced dynamic library on Linux you need to LD_PRELOAD an rts. It's clunky, but it works, it allows you to switch between debug and non-debug rts at initialization time.

On Windows, this problem was punted, because everything is statically linked. But the problem exists that you can have multiple DLLs with different RTS and ABIs. This is fine as long as the DLLs have no dependencies on each other. Once they do... you have a big problem. This is one of the primary blockers of shared library support on Windows.

I.. don't know whatever wacky solution MacOS uses so can't comment there.

Now back to the original question about version 1.0, this has nothing to do with Make at all. Make based system only implemented the scheme that was wanted. It's not like any Make system design issues forced this scheme. Now over the years, assumptions that the RTS is always version 1.0 could have krept into the build system. But I don't believe this to have been design, just convenience. Right now, the design only requires you to know the GHC version, which is available in all makefiles. Knowing the RTS version would be difficult, but the point is that in a proper design you don't need to know the version.

Almost half a decade ago a plan was made to replace this scheme with one that would work on all OSes and would allow us to solve these issues. The design was made and debated here https://gitlab.haskell.org/ghc/ghc/-/issues/10352

The actual solution isn't as simple as just adding the rts version to the library name or add it only to the build system, in fact this would be the wrong approach as it makes it impossible to observe backwards compatibility between GHC releases. i.e. without it, you'd need to have GHC 9.0.1 installed to run GHC 9.0.1 programs, you can't run using GHC 9.2.x rts if the version changed.

Typically ELF based platforms solve this by a combination of SONAME and symbol versioning. Windows solves this by a combination of SxS Assembly versioning or mingw style SONAME.

All of which require you to have the same filename for the libraries, but use a different path to disambiguate:

lib/ghc-${ver}/rts-1.0/libHSrts-ghc${ver}.so

lib/ghc-${ver}/rts-1.0/thr/libHSrts-ghc${ver}.so

lib/ghc-${ver}/rts-1.0/debug/libHSrts-ghc${ver}.so

lib/ghc-${ver}/rts-1.0/l/libHSrts-ghc${ver}.so

lib/ghc-${ver}/rts-1.0/thr_l/libHSrts-ghc${ver}.so

for each RTS with the same ABI. profiling libs for instance have a different ABI and can't use this scheme.

So what has taken so long to implement this? Well.. time. As it turns out, getting this scheme to work required a lot of foundational work in GHC (Particularly on Windows where dynamic linking design wasn't optimal, but both GHC and the dynamic linker are happy now).

On Linux it took a while to get SONAME support in cabal https://github.com/haskell/cabal/issues/4052 so we don't have to hack around in the build system.

But anyway this is why the current scheme exists, and why just adding an rts version isn't really sufficient, especially if the name propagates to the shared lib.

TL;DR;

If we are going to change the build system, we should do it properly.

The current scheme exists because GHC does not observe any mechanism to support multiple runtimes with the same ABI and does not really have a backwards compatibility story.

Kind Regards,

Tamar

On Wed, Feb 10, 2021 at 11:00 PM Richard Eisenberg wrote:

...

On Feb 10, 2021, at 8:50 AM, Simon Peyton Jones wrote:

build with hadrian, and then continue using make with the artifacts (partially) built by Hadrian

I almost suggested that this had to be the reason for the back-compat design, but I assumed I had to be wrong. I also agree this is a non-goal; I'm quite happy to be forced to pick one or the other and stick with that choice until blasting away all build products.

Richard _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs