*BSD support in 6.8.3

Simon Marlow

28 May 2008 28 May '08

5:05 a.m.

Hi Folks, 6.8.3 is nearing release, and we have an outstanding bug affecting the GHCi on the BSDs: http://hackage.haskell.org/trac/ghc/ticket/2013 We need someone to help out with this. The patch in the ticket apparently works, but can't be committed as is because it isn't correctly #ifdef'd and will presumably break other platforms. Also it needs to be tested on OpenBSD/NetBSD in addition to FreeBSD. I don't have access to a *BSD machine right now, and I don't have the time available to set one up. If someone can donate a temporary account then that would be helpful, but most helpful would be if someone could work with us to get this bug fixed in time for 6.8.3 (i.e. the next few days). Otherwise, we have to release with the bug still in, which would be bad. Cheers, Simon

Show replies by date

Gregory Wright

28 May 28 May

12:25 p.m.

Hi Simon, On May 28, 2008, at 5:05 AM, Simon Marlow wrote:

...

Hi Folks,

6.8.3 is nearing release, and we have an outstanding bug affecting the GHCi on the BSDs:

http://hackage.haskell.org/trac/ghc/ticket/2013

We need someone to help out with this. The patch in the ticket apparently works, but can't be committed as is because it isn't correctly #ifdef'd and will presumably break other platforms. Also it needs to be tested on OpenBSD/NetBSD in addition to FreeBSD.

I don't have access to a *BSD machine right now, and I don't have the time available to set one up. If someone can donate a temporary account then that would be helpful, but most helpful would be if someone could work with us to get this bug fixed in time for 6.8.3 (i.e. the next few days). Otherwise, we have to release with the bug still in, which would be bad.

Cheers, Simon

I have a FreeBSD 7.0/x86_64 machine I can test this on. Should the patch be applied to the 6.8.3 release candidate tarball, or just the head of the 6.8.x branch? Greg

Ian Lynagh

1:14 p.m.

On Wed, May 28, 2008 at 12:25:48PM -0400, Gregory Wright wrote:

...

I have a FreeBSD 7.0/x86_64 machine I can test this on. Should the patch be applied to the 6.8.3 release candidate tarball, or just the head of the 6.8.x branch?

Either, they're the same at the moment - although if you want to make a darcs patch of the result then you'll need to apply it to the darcs branch. It's possible there will be some fuzz, as the patch is 5 months old. Thanks Ian

Don Stewart

1:46 p.m.

marlowsd:

...

Hi Folks,

6.8.3 is nearing release, and we have an outstanding bug affecting the GHCi on the BSDs:

http://hackage.haskell.org/trac/ghc/ticket/2013

We need someone to help out with this. The patch in the ticket apparently works, but can't be committed as is because it isn't correctly #ifdef'd and will presumably break other platforms. Also it needs to be tested on OpenBSD/NetBSD in addition to FreeBSD.

I don't have access to a *BSD machine right now, and I don't have the time available to set one up. If someone can donate a temporary account then that would be helpful, but most helpful would be if someone could work with us to get this bug fixed in time for 6.8.3 (i.e. the next few days). Otherwise, we have to release with the bug still in, which would be bad.

So, try the patch to get the patch to compile against ghc 6.8.3-snapshot, and see if it works on OpenBSD? -- Don

Simon Marlow

29 May 29 May

7:19 a.m.

Don Stewart wrote:

...

marlowsd:

...
Hi Folks,

6.8.3 is nearing release, and we have an outstanding bug affecting the GHCi on the BSDs:

http://hackage.haskell.org/trac/ghc/ticket/2013

We need someone to help out with this. The patch in the ticket apparently works, but can't be committed as is because it isn't correctly #ifdef'd and will presumably break other platforms. Also it needs to be tested on OpenBSD/NetBSD in addition to FreeBSD.

I don't have access to a *BSD machine right now, and I don't have the time available to set one up. If someone can donate a temporary account then that would be helpful, but most helpful would be if someone could work with us to get this bug fixed in time for 6.8.3 (i.e. the next few days). Otherwise, we have to release with the bug still in, which would be bad.

So, try the patch to get the patch to compile against ghc 6.8.3-snapshot, and see if it works on OpenBSD?

It's not just a case of testing the patch, there are a couple of issues to address: - it has a couple of wired-in addresses: one place to load object files, another to put jump tables at. This is necessary because *BSD doesn't have the MAP_32BIT flag for mmap(). However, the particular wired-in addresses needed will probably vary on the different *BSDs. Someone needs to look at the memory map and make sure we're picking sensible addresses. - the patch doesn't #ifdef its changes, so it'll break other platforms (easy to fix). Also the code has changed in HEAD, and we need a completely different patch there (although the same idea applies, pick an address and use MAP_FIXED). Cheers, Simon

Wilhelm B. Kloke

7:56 a.m.

Simon Marlow schrieb:

...

...
So, try the patch to get the patch to compile against ghc 6.8.3-snapshot, and see if it works on OpenBSD?

It's not just a case of testing the patch, there are a couple of issues to address:

- it has a couple of wired-in addresses: one place to load object files, another to put jump tables at. This is necessary because *BSD doesn't have the MAP_32BIT flag for mmap(). However, the particular wired-in addresses needed will probably vary on the different *BSDs. Someone needs to look at the memory map and make sure we're picking sensible addresses.

It is not very likely that the adresses change between *BSDs because they are resulting from simple hardware-related constraints. Though, a test is necessary. OTOH, has anybody from the GHC team asked the *BSD developers to add a MAP_32BIT flag already? I know that these people are very reluctant to change requests, this one may find their mercy if the right persons ask for it.

...

- the patch doesn't #ifdef its changes, so it'll break other platforms (easy to fix).

I looked into the patch, and it doesn't look as if it would break other platforms, as the changed code is inside proper ifdefs mostly. Some parts may be superfluous. The easiest check is to compile the FreeBSD-patched code under one of the other platforms (are there other than Linux and OSX?). In any case, the ifdef'ing can be automated using diff with -D__FreeBSD__ option.

...

Also the code has changed in HEAD, and we need a completely different patch there (although the same idea applies, pick an address and use MAP_FIXED).

This is, of course, the most significant problem for GHC, but probably not relevant to this special release process. It is better to include the patch than to leave it out (provided that I am right resp. to the breaking of otherplatforms). -- Dipl.-Math. Wilhelm Bernhard Kloke Institut fuer Arbeitsphysiologie an der Universitaet Dortmund Ardeystrasse 67, D-44139 Dortmund, Tel. 0231-1084-373 PGP: http://vestein.arb-phys.uni-dortmund.de/~wb/mypublic.key

Donn Cave

11:15 a.m.

On Thu, 29 May 2008 11:56:59 +0000 (UTC) "Wilhelm B. Kloke" wrote:

...

Simon Marlow schrieb:

...

...
- it has a couple of wired-in addresses: one place to load object files, another to put jump tables at. This is necessary because *BSD doesn't have the MAP_32BIT flag for mmap(). However, the particular wired-in addresses needed will probably vary on the different *BSDs. Someone needs to look at the memory map and make sure we're picking sensible addresses.

It is not very likely that the adresses change between *BSDs because they are resulting from simple hardware-related constraints. Though, a test is necessary.

I have made some progress with NetBSD-amd64, now I'm working on stage2 of 6.8.2. So there's some hope that I may be able to look at this issue on that platform. (It's going pretty slow, though - lucky I got laid off last week and I have a 1.8 Ghz computer with nearly empty hard disk to work with, or building GHC would be just about out of the question!) But I noticed that, in Linker.c, NetBSD doesn't configure USE_MMAP - whatever the hardware platform. I don't know if that's just by neglect, but if I don't use mmap(), then support for MAP_32BIT is no concern? -- Donn Cave

Simon Marlow

11:19 a.m.

Ok, I've now modified the patch and attached a new version to the ticket: http://hackage.haskell.org/trac/ghc/attachment/ticket/2013/2013.patch *BSD folks please test. Wilhelm B. Kloke wrote:

...

Simon Marlow schrieb:

...
...
So, try the patch to get the patch to compile against ghc 6.8.3-snapshot, and see if it works on OpenBSD? It's not just a case of testing the patch, there are a couple of issues to address:

- it has a couple of wired-in addresses: one place to load object files, another to put jump tables at. This is necessary because *BSD doesn't have the MAP_32BIT flag for mmap(). However, the particular wired-in addresses needed will probably vary on the different *BSDs. Someone needs to look at the memory map and make sure we're picking sensible addresses.

It is not very likely that the adresses change between *BSDs because they are resulting from simple hardware-related constraints. Though, a test is necessary.

These are virtual addresses, so hardware doesn't really come into it, right? I remember seeing memory maps change between versions of Linux in the past (i.e. where shared libraries are loaded, where the binary is loaded, etc.). We need to avoid clashing with anything else, otherwise the OS might allocate us an address outside the 2Gb range.

...

OTOH, has anybody from the GHC team asked the *BSD developers to add a MAP_32BIT flag already? I know that these people are very reluctant to change requests, this one may find their mercy if the right persons ask for it.

That's a good idea, I might try it.

...

I looked into the patch, and it doesn't look as if it would break other platforms, as the changed code is inside proper ifdefs mostly. Some parts may be superfluous. The easiest check is to compile the FreeBSD-patched code under one of the other platforms (are there other than Linux and OSX?).

So the patch as it stands always passes 1Gb as the address to mmap(). This might have undesirable effects on other systems. I fixed that in the new version of the patch (see above). Cheers, Simon

Gregory Wright

30 May 30 May

10:43 a.m.

Hi Simon, On May 29, 2008, at 11:19 AM, Simon Marlow wrote:

...

Ok, I've now modified the patch and attached a new version to the ticket:

http://hackage.haskell.org/trac/ghc/attachment/ticket/2013/2013.patch

*BSD folks please test.

I built the 20080529 snapshot with this patch and my light testing of ghci showed no problems (FreeBSD 7.0/x86_64). I will built it again tonight in a darcs checkout and run the test suite. I should be able to send back any tweaks to the patch in the next two days. -Greg

Donn Cave

1 Jun 1 Jun

4:24 a.m.

On Fri, 30 May 2008 10:43:22 -0400 Gregory Wright wrote:

...

...
http://hackage.haskell.org/trac/ghc/attachment/ticket/2013/2013.patch

*BSD folks please test.

I built the 20080529 snapshot with this patch and my light testing of ghci showed no problems (FreeBSD 7.0/x86_64).

20080531 and NetBSD-amd64 4.0, not so good: GHCi, version 6.8.2.20080531: http://www.haskell.org/ghc/ :? for help Loading package base ... linking ... done. Bad system call (core dumped) SIGSYS appears to happen in select(), is all I could get out of gdb. Prior to the patch, it was the usual error. Otherwise, 20080531 seems to work as well as 6.8.2 release on NetBSD-amd64. NetBSD defines ELF64_ST_TYPE and ELF64_ST_BIND. Unfortunately, it won't build with the GNU "ar" that's standard on this platform. It can't index archives as big as libHSbase.a: apparently, it allocates too many moderately large hash tables for the many small modules in that archive, and runs out of memory. I worked around this by reducing the default hash table size in libbfd.so. I'm not saying any of this makes sense, it's just what I eventually managed to get working. Otherwise, it builds smoothly. -- Donn Cave

Simon Marlow

2 Jun 2 Jun

4:36 a.m.

Donn Cave wrote:

...

On Fri, 30 May 2008 10:43:22 -0400 Gregory Wright wrote:

...
...
http://hackage.haskell.org/trac/ghc/attachment/ticket/2013/2013.patch

*BSD folks please test.

I built the 20080529 snapshot with this patch and my light testing of ghci showed no problems (FreeBSD 7.0/x86_64).

20080531 and NetBSD-amd64 4.0, not so good:

GHCi, version 6.8.2.20080531: http://www.haskell.org/ghc/ :? for help Loading package base ... linking ... done. Bad system call (core dumped)

SIGSYS appears to happen in select(), is all I could get out of gdb. Prior to the patch, it was the usual error.

I guess the error in the linker was masking the SIGSYS in select(), because it occurred first. So now we need to figure out what is causing this SIGSYS.

...

Otherwise, 20080531 seems to work as well as 6.8.2 release on NetBSD-amd64. NetBSD defines ELF64_ST_TYPE and ELF64_ST_BIND.

Unfortunately, it won't build with the GNU "ar" that's standard on this platform. It can't index archives as big as libHSbase.a: apparently, it allocates too many moderately large hash tables for the many small modules in that archive, and runs out of memory. I worked around this by reducing the default hash table size in libbfd.so. I'm not saying any of this makes sense, it's just what I eventually managed to get working. Otherwise, it builds smoothly.

Perhaps turning off -split-objs would also work around this problem? Cheers, Simon

Donn Cave

11:31 a.m.

On Mon, 02 Jun 2008 09:36:41 +0100 Simon Marlow wrote:

...

...
Unfortunately, it won't build with the GNU "ar" that's standard on this platform. It can't index archives as big as libHSbase.a: apparently, it allocates too many moderately large hash tables for the many small modules in that archive, and runs out of memory. I worked around this by reducing the default hash table size in libbfd.so. I'm not saying any of this makes sense, it's just what I eventually managed to get working. Otherwise, it builds smoothly.

Perhaps turning off -split-objs would also work around this problem?

Yes, so the NetBSD package maintainer suggested, after I wrote to him about this. He expressed some interest in getting the amd64 build into the package system, as there have been requests (currently supports only the i386 platform), so I'm going to pursue that. -- Donn Cave

Donn Cave

3 Jun 3 Jun

4:31 p.m.

On Mon, 02 Jun 2008 09:36:41 +0100 Simon Marlow wrote:

...

Donn Cave wrote: ...

...
20080531 and NetBSD-amd64 4.0, not so good:

GHCi, version 6.8.2.20080531: http://www.haskell.org/ghc/ :? for help Loading package base ... linking ... done. Bad system call (core dumped)

SIGSYS appears to happen in select(), is all I could get out of gdb. Prior to the patch, it was the usual error.

I guess the error in the linker was masking the SIGSYS in select(), because it occurred first. So now we need to figure out what is causing this SIGSYS.

Well, good news and bad news. The patch works - this was indeed a different problem, and with that out of the way ghci comes up and works well enough to pass some randomly selected tests from a testsuite - ghci, rts. This is 20080531 plus patch 2013. The SIGSYS problem is an awkward issue between GHC and NetBSD, which I see has already been reported, #2305. C code compiled from /usr/include/signal.h links to __sigprocmask14, but libraries/base/System/Posix/Internals.hs links to sigprocmask because it misses out on the __RENAME magic in signal.h. That's a compatibility function that doesn't work out so well for ghci. I will take this up with the NetBSD ghc package maintainer. At worst, we can just hard-code the (incompatibility?) function names. thanks, -- Donn Cave

Simon Marlow

5 Jun 5 Jun

4:49 a.m.

Donn Cave wrote:

...

On Mon, 02 Jun 2008 09:36:41 +0100 Simon Marlow wrote:

...
Donn Cave wrote: ...

...
20080531 and NetBSD-amd64 4.0, not so good:

GHCi, version 6.8.2.20080531: http://www.haskell.org/ghc/ :? for help Loading package base ... linking ... done. Bad system call (core dumped)

SIGSYS appears to happen in select(), is all I could get out of gdb. Prior to the patch, it was the usual error. I guess the error in the linker was masking the SIGSYS in select(), because it occurred first. So now we need to figure out what is causing this SIGSYS.

Well, good news and bad news. The patch works - this was indeed a different problem, and with that out of the way ghci comes up and works well enough to pass some randomly selected tests from a testsuite - ghci, rts. This is 20080531 plus patch 2013.

The SIGSYS problem is an awkward issue between GHC and NetBSD, which I see has already been reported, #2305. C code compiled from /usr/include/signal.h links to __sigprocmask14, but libraries/base/System/Posix/Internals.hs links to sigprocmask because it misses out on the __RENAME magic in signal.h. That's a compatibility function that doesn't work out so well for ghci. I will take this up with the NetBSD ghc package maintainer. At worst, we can just hard-code the (incompatibility?) function names.

See my update to #2305 - it looks like we need a C wrapper for sigprocmask() in the base package. Would you like to make a ticket for this, or better still a patch (it's not too hard, there are plenty of other examples of this in base already)? Cheers, Simon

Donn Cave

12:23 p.m.

On Thu, 05 Jun 2008 09:49:22 +0100 Simon Marlow wrote:

...

See my update to #2305 - it looks like we need a C wrapper for sigprocmask() in the base package. Would you like to make a ticket for this, or better still a patch (it's not too hard, there are plenty of other examples of this in base already)?

OK. I can find 7 of these renamed externals, in base, network and unix, out of 88 in NetBSD 4.0. This will obviously need to be revisited occasionally, but I will be happy to make a patch. Unified context diff OK? It looks like, in base anyway, this actually goes in a .h file, to a pattern like INLINE int __hscore_xyz( ... ) { return xyz( ... ); } thanks, -- Donn Cave

Donn Cave

1:55 p.m.

On Thu, 5 Jun 2008 09:23:36 -0700 Donn Cave wrote:

...

... It looks like, in base anyway, this actually goes in a .h file, to a pattern like

... ah, never mind that, I see those wrappers are not the sort I need. -- Donn Cave

Simon Marlow

7 Jun 7 Jun

12:14 a.m.

Donn Cave wrote:

...

On Thu, 5 Jun 2008 09:23:36 -0700 Donn Cave wrote:

...
... It looks like, in base anyway, this actually goes in a .h file, to a pattern like

... ah, never mind that, I see those wrappers are not the sort I need.

Yes, those are the kind of wrappers I mean. For example, base has a wrapper for lseek() (__hsccore_lseek()) for exactly the same reason. Cheers, Simon

Donn Cave

1:09 p.m.

On Sat, 07 Jun 2008 05:14:02 +0100 Simon Marlow wrote:

...

Donn Cave wrote:

...
On Thu, 5 Jun 2008 09:23:36 -0700 Donn Cave wrote:

...
... It looks like, in base anyway, this actually goes in a .h file, to a pattern like

... ah, never mind that, I see those wrappers are not the sort I need.

Yes, those are the kind of wrappers I mean. For example, base has a wrapper for lseek() (__hsccore_lseek()) for exactly the same reason.

Thanks, I submitted one of those yesterday (unsetenv, where the type of the function has changed in POSIX.1 from the older spec that GHC follows.) I've done the rest of them too, but since this is a general platform issue, I wanted to get some input from the NetBSD package maintainer, who is interested in the issue but I believe is off at some conference. Other renamed functions, in NetBSD 4.0, are sigprocmask, opendir, socket, times, sigpending, and sigsuspend. -- Donn Cave

Gregory Wright

1 Jun 1 Jun

10:08 p.m.

Hi,

...

On May 29, 2008, at 11:19 AM, Simon Marlow wrote:

...
Ok, I've now modified the patch and attached a new version to the ticket:

http://hackage.haskell.org/trac/ghc/attachment/ticket/2013/2013.patch

*BSD folks please test.

I built the 20080529 snapshot with this patch and my light testing of ghci showed no problems (FreeBSD 7.0/x86_64).

I will built it again tonight in a darcs checkout and run the test suite. I should be able to send back any tweaks to the patch in the next two days.

-Greg

I've built ghc 6.8.x from the darcs repository (checked out on 30 May) with the 2013.patch on FreeBSD 7.0/x86_64. As far as ghci is concerned, it seems to work and passes almost all of the tests. However, there are many unexpected failures in the testsuite, almost all apparently coming from failures to find mkstemp, mknod and S_IFDIR in header files. Either some conditional includes are broken in ghc or FreeBSD 7.0 moved some things. I'll have a look tomorrow. -Greg

Simon Marlow

2 Jun 2 Jun

4:37 a.m.

Gregory Wright wrote:

...

Hi,

...
On May 29, 2008, at 11:19 AM, Simon Marlow wrote:

...
Ok, I've now modified the patch and attached a new version to the ticket:

http://hackage.haskell.org/trac/ghc/attachment/ticket/2013/2013.patch

*BSD folks please test.

I built the 20080529 snapshot with this patch and my light testing of ghci showed no problems (FreeBSD 7.0/x86_64).

I will built it again tonight in a darcs checkout and run the test suite. I should be able to send back any tweaks to the patch in the next two days.

-Greg

I've built ghc 6.8.x from the darcs repository (checked out on 30 May) with the 2013.patch on FreeBSD 7.0/x86_64. As far as ghci is concerned, it seems to work and passes almost all of the tests. However, there are many unexpected failures in the testsuite, almost all apparently coming from failures to find mkstemp, mknod and S_IFDIR in header files. Either some conditional includes are broken in ghc or FreeBSD 7.0 moved some things. I'll have a look tomorrow.

Great, thanks for the feedback Greg. I'll push this patch. Cheers, Simon

6191

Age (days ago)

6201

Last active (days ago)

List overview

Download

19 comments

6 participants

participants (6)

Don Stewart
Donn Cave
Gregory Wright
Ian Lynagh
Simon Marlow
Wilhelm B. Kloke