Offering GHC builder build slaves

Alain O'Dea

7 Apr 2014 7 Apr '14

9:06 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi folks: I have successfully built Ian Lynagh's GHC builder on Ubuntu and SmartOS and I would like to offer build slaves based on Carter's mirroring of the code to https://github.com/cartazio/ghc-builder. I can offer several build slaves, but I'm not sure what the process is. How do I run multiple build slaves? Do I need a separate username for each? Is there a username convention? The suggestion at https://ghc.haskell.org/trac/ghc/wiki/Builder is that I post a username and password to ghc@. There are two issues with this: 1. ghc@ does not exist as far as I can tell 2. Posting a password to a mailing list will make it publicly accessible and allow others to impersonate my build slaves I am happy to send this information if the admins of the GHC builder infrastructure are comfortable with the risks. Alternatively, I am able to send this via GPG encrypted email given the public key and email of an admin. Feel free to contact me on or off list about this. Best, Alain -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJTQ0uTAAoJEP0rIXJNjNSA+fMH/2p9yeWgpfDeaTHilur5qoZ6 6DZiktJgvMpVeqFwHyaFMk+ubEp8/xU/VUrOEztr01jmzNLS2UoqDVH/7EZPZPtV 0ONh/bcYYA8EBa9SCqaVXVb9I8EdDE6w2cQGDqnwHqaFerAUqv8OiQEWyJCuJSdr 7EL/vcacNfr5Ix4oG2pkESvEJBHHEN4ZTXoKeJnyQT93o2RaC1fgImm6F6tgpdQE Jh+QInIX1dRuSl+pDlHg6kbLZqFG8Mnyz0bcWuUL2ekGQBp3HtT0MwonM9HUTmmG 0baRYUaK/moO5Q6lUXhI2eRT/UFl4qnzwJw/sIVuaL7h1Ikj0ZMOuPs0GjfRbY8= =/Rwk -----END PGP SIGNATURE-----

Show replies by date

Mateusz Kowalczyk

7 Apr 7 Apr

9:32 p.m.

On 08/04/14 02:06, Alain O'Dea wrote:

...

Hi folks:

I have successfully built Ian Lynagh's GHC builder on Ubuntu and SmartOS and I would like to offer build slaves based on Carter's mirroring of the code to https://github.com/cartazio/ghc-builder.

I can offer several build slaves, but I'm not sure what the process is.

How do I run multiple build slaves?

Do I need a separate username for each?

Is there a username convention?

The suggestion at https://ghc.haskell.org/trac/ghc/wiki/Builder is that I post a username and password to ghc@. There are two issues with this:

1. ghc@ does not exist as far as I can tell 2. Posting a password to a mailing list will make it publicly accessible and allow others to impersonate my build slaves

I am happy to send this information if the admins of the GHC builder infrastructure are comfortable with the risks. Alternatively, I am able to send this via GPG encrypted email given the public key and email of an admin.

Feel free to contact me on or off list about this.

Best, Alain _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

I can offer a 32-bit Linux slave (or 2) myself. -- Mateusz K.

Páli Gábor János

8 Apr 8 Apr

3:25 a.m.

2014-04-08 3:32 GMT+02:00 Mateusz Kowalczyk :

...

I can offer a 32-bit Linux slave (or 2) myself.

Please mail me with the specification of the host system (Linux distribution used, version, etc.) off-list and I could allocate a builder account for you.

Páli Gábor János

3:23 a.m.

2014-04-08 3:06 GMT+02:00 Alain O'Dea :

...

I can offer several build slaves, but I'm not sure what the process is.

As far as I know, the infrastructure has become a volunteer-run effort, and it looks like I am the volunteer who runs it... :-)

...

How do I run multiple build slaves?

Ideally, each of the slaves should run in their own isolated (mostly virtualized) environment.

...

Do I need a separate username for each?

Yes.

...

Is there a username convention?

So far I assigned names to the clients by the "${os}-${arch}-${branch}" scheme, such as linux-ppc64-head.

...

The suggestion at https://ghc.haskell.org/trac/ghc/wiki/Builder is that I post a username and password to ghc@. There are two issues with this:

Actually, I think this wiki page is not valid any more -- the original builder infrastructure was abandoned last year. I have started to replicate the whole thing for my own clients, and it works well (for me, and nowadays for Karel) since then. However, my efforts has not been "blessed" and I am not sure if there are at least plans to make the builders part of the official haskell.org infrastructure. Either way it goes, I can update the corresponding wiki page with my contact information and start accommodation further clients until the fate of the service is decided, if there will not be any objections in the next few days.

...

I am happy to send this information if the admins of the GHC builder infrastructure are comfortable with the risks.

All right, I can follow up you with the details off-list.

Simon Peyton Jones

4:03 a.m.

| > The suggestion at https://ghc.haskell.org/trac/ghc/wiki/Builder is | > that I post a username and password to ghc@. There are two issues | > with this: | | Actually, I think this wiki page is not valid any more -- the original | builder infrastructure was abandoned last year. I have started to | replicate the whole thing for my own clients, and it works well (for | me, and nowadays for Karel) since then. | | However, my efforts has not been "blessed" and I am not sure if there | are at least plans to make the builders part of the official | haskell.org infrastructure. Either way it goes, I can update the | corresponding wiki page with my contact information and start | accommodation further clients until the fate of the service is decided, | if there will not be any objections in the next few days. Bless you, my son! Seriously, I advertised a couple of weeks ago for help with our nightly-build infrastructure. Quite a few people responded -- thank you very much. So we have willing horsepower. But the moment we lack leadership. Alain rightly says "I don't know what the process is" because we don't *have* a process. We need a mechanism for creating a process, taking decisions, etc. I think what is needed is: * A group of people willing to act as a kind of committee. That could be everyone who replied. You could create a mailing list, or (initially better) just chat on ghc-devs. But it would be useful to have a list of who is involved. * Someone (or a couple of people) to play the role of chair. That doesn't mean an autocrat... it means someone who gently pushes discussions to a conclusion, and says "I propose that we do X". * Then the group can formulate a plan and proceed with it. For example, should Pali's efforts be "blessed"? I don't know enough to know, but you guys do. In my experience, people are often unwilling to put themselves forward as chair, not because they are unwilling, but because they feel it'd be "pushy". So I suggest this: if you think (based on the traffic you've seen) that X would be a chair you'd trust, suggest them. In short: power to the people! GHC is your compiler. Thanks! Simon

Joachim Breitner

4:30 a.m.

Hi, Am Dienstag, den 08.04.2014, 08:03 +0000 schrieb Simon Peyton Jones:

...

So we have willing horsepower. But the moment we lack leadership. Alain rightly says "I don't know what the process is" because we don't *have* a process. We need a mechanism for creating a process, taking decisions, etc.

we also need a culture of just doing stuff, and less asking for it. In Debian it works likes this: Someone has an idea (continuous integration), does it and tells the mailing list about it. Then people tell you that it’s great, or how it could be greater. If it works out, eventually it comes an official service, whatever that means. So in this case (independent of any committee or process): Páli, you have starting running some builder infrastructure. Great! Just keep doing it! And if you want people to join their builders, tell them what information you need from them and add them. Feel free to modify the wiki so that people find you. Make up some rules (about usernames etc.) as you go, if necessary. In essence what you said in

...

However, my efforts has not been "blessed" and I am not sure if there are at least plans to make the builders part of the official haskell.org infrastructure. Either way it goes, I can update the corresponding wiki page with my contact information and start accommodation further clients until the fate of the service is decided, if there will not be any objections in the next few days.

but without waiting for objections. Just do it. And tell us about your achievements. Also worry less about official or not. The Travis setup is not official, but (IMHO) has been useful quite a few times. I’d _like_ it to be official, i.e. hosted on git.haskell.org, but that is not important. If your service becomes “critical” in some sense it is still time to move it some official infrastructure... but that can come second, and should not hinder anyone from contributing. Greetings, and thanks for your contributions, Joachim -- Joachim “nomeata” Breitner mail@joachim-breitner.de • http://www.joachim-breitner.de/ Jabber: nomeata@joachim-breitner.de • GPG-Key: 0x4743206C Debian Developer: nomeata@debian.org

Simon Peyton Jones

4:36 a.m.

| we also need a culture of just doing stuff, and less asking for it. I agree with that -- hence "power to the people". You absolutely don't need the approval of a committee or of GHC HQ to just get on with something. But it also makes sense to work together, to achieve more as a group than a single individual can. Simon | -----Original Message----- | From: ghc-devs [mailto:ghc-devs-bounces@haskell.org] On Behalf Of | Joachim Breitner | Sent: 08 April 2014 09:31 | To: ghc-devs@haskell.org | Subject: Re: Offering GHC builder build slaves | | Hi, | | Am Dienstag, den 08.04.2014, 08:03 +0000 schrieb Simon Peyton Jones: | > So we have willing horsepower. But the moment we lack leadership. | > Alain rightly says "I don't know what the process is" because we don't | > *have* a process. We need a mechanism for creating a process, taking | > decisions, etc. | | we also need a culture of just doing stuff, and less asking for it. | | In Debian it works likes this: Someone has an idea (continuous | integration), does it and tells the mailing list about it. Then people | tell you that it’s great, or how it could be greater. If it works out, | eventually it comes an official service, whatever that means. | | So in this case (independent of any committee or process): Páli, you | have starting running some builder infrastructure. Great! Just keep | doing it! | And if you want people to join their builders, tell them what | information you need from them and add them. Feel free to modify the | wiki so that people find you. Make up some rules (about usernames etc.) | as you go, if necessary. In essence what you said in | | > However, my efforts has not been "blessed" and I am not sure if there | > are at least plans to make the builders part of the official | > haskell.org infrastructure. Either way it goes, I can update the | > corresponding wiki page with my contact information and start | > accommodation further clients until the fate of the service is | > decided, if there will not be any objections in the next few days. | | but without waiting for objections. Just do it. And tell us about your | achievements. | | Also worry less about official or not. The Travis setup is not official, | but (IMHO) has been useful quite a few times. I’d _like_ it to be | official, i.e. hosted on git.haskell.org, but that is not important. | | If your service becomes “critical” in some sense it is still time to | move it some official infrastructure... but that can come second, and | should not hinder anyone from contributing. | | Greetings, and thanks for your contributions, Joachim | | -- | Joachim “nomeata” Breitner | mail@joachim-breitner.de • http://www.joachim-breitner.de/ | Jabber: nomeata@joachim-breitner.de • GPG-Key: 0x4743206C | Debian Developer: nomeata@debian.org

Páli Gábor János

4:50 a.m.

2014-04-08 10:30 GMT+02:00 Joachim Breitner :

...

we also need a culture of just doing stuff, and less asking for it.

Yes, I was educated in the same spirit but in the FreeBSD Project. I did not ask much for it when I replicated the whole setup just for myself last year.

...

if you want people to join their builders, tell them what information you need from them and add them. Feel free to modify the wiki so that people find you. Make up some rules (about usernames etc.) as you go, if necessary.

That is good to hear. First, I had the impression from the previous discussions that Ian's solution is not proven enough so you want to go for some other solution. Second, I do not want to duplicate anybody else's efforts. Although I have already stated that I am willing to let others connect to my server and replied the related mails, but I felt that the offer was still ignored or lost. I do not want to be pushy, I do not like stepping on other's toes. But actually I can if that is what you want -- that is how I did eight BSD workshops and developer summits in the last four years and eventually become the secretary of the FreeBSD Core Team.

...

Just do it. And tell us about your achievements.

I guess the ghc-builds mailing list speaks for itself.

...

Also worry less about official or not. The Travis setup is not official, but (IMHO) has been useful quite a few times. I'd _like_ it to be official, i.e. hosted on git.haskell.org, but that is not important.

All right, if the rules of game are like that, let it be so...

...

If your service becomes "critical" in some sense it is still time to move it some official infrastructure... but that can come second, and should not hinder anyone from contributing.

Okay, thanks for the clarification!

Johan Tibell

5:22 a.m.

I'll mirror what Joachim said: just get something working. Having an email sent to ghc-devs@ every time something breaks (and getting down false positives here is very important) is already a huge win, no matter what kind of build system sits behind that email.

Páli Gábor János

5:31 a.m.

2014-04-08 11:22 GMT+02:00 Johan Tibell :

...

I'll mirror what Joachim said: just get something working. Having an email sent to ghc-devs@ every time something breaks (and getting down false positives here is very important) is already a huge win, no matter what kind of build system sits behind that email.

The ghc-builds mailing list has the all the results -- however, earlier there was the concern that folks do not want to watch it every day. I still do, just to know everything is all right. Filtering the false positives out might be solved automatically, I think, hopefully I can come up with a solution soon. Anyway, for starter, here is a nice core dump for linux-ppc64: chmod +x inplace/bin/dll-split inplace/bin/dll-split compiler/stage2/build/.depend-v-p-dyn.haskell "DynFlags" "Annotations Avail Bag BasicTypes BinIface Binary Bitmap BlockId BooleanFormula BreakArray BufWrite BuildTyCl ByteCodeAsm ByteCodeInstr ByteCodeItbls CLabel Class CmdLineParser Cmm CmmCallConv CmmExpr CmmInfo CmmMachOp CmmNode CmmType CmmUtils CoAxiom ConLike CodeGen.Platform CodeGen.Platform.ARM CodeGen.Platform.NoRegs CodeGen.Platform.PPC CodeGen.Platform.PPC_Darwin CodeGen.Platform.SPARC CodeGen.Platform.X86 CodeGen.Platform.X86_64 Coercion Config Constants CoreArity CoreFVs CoreLint CoreSubst CoreSyn CoreTidy CoreUnfold CoreUtils CostCentre DataCon Demand Digraph DriverPhases DsMonad DynFlags Encoding ErrUtils Exception ExtsCompat46 FamInstEnv FastBool FastFunctions FastMutInt FastString FastTypes Finder Fingerprint FiniteMap ForeignCall Hooks Hoopl Hoopl.Dataflow HsBinds HsDecls HsDoc HsExpr HsImpExp HsLit HsPat HsSyn HsTypes HsUtils HscTypes IOEnv Id IdInfo IfaceEnv IfaceSyn IfaceType InstEnv InteractiveEvalTypes Kind ListSetOps Literal LoadIface Maybes MkCore MkGraph MkId Module MonadUtils Name NameEnv NameSet OccName OccurAnal OptCoercion OrdList Outputable PackageConfig Packages Pair Panic PatSyn PipelineMonad Platform PlatformConstants PprCmm PprCmmDecl PprCmmExpr PprCore PrelInfo PrelNames PrelRules Pretty PrimOp RdrName Reg RegClass Rules SMRep Serialized SrcLoc StaticFlags StgCmmArgRep StgCmmClosure StgCmmEnv StgCmmLayout StgCmmMonad StgCmmProf StgCmmTicky StgCmmUtils StgSyn Stream StringBuffer TcEvidence TcIface TcRnMonad TcRnTypes TcType TcTypeNats TrieMap TyCon Type TypeRep TysPrim TysWiredIn Unify UniqFM UniqSet UniqSupply Unique Util Var VarEnv VarSet" make[1]: *** [compiler/stage2/dll-split.stamp] Segmentation fault make: *** [all] Error 2 For the details, please consult: http://haskell.inf.elte.hu/builders/linux-ppc64-head/6.html Karel (see CC'ed) runs the client, feel free to prod him for more.

Johan Tibell

5:34 a.m.

On Tue, Apr 8, 2014 at 11:31 AM, Páli Gábor János wrote:

...

2014-04-08 11:22 GMT+02:00 Johan Tibell :

...
I'll mirror what Joachim said: just get something working. Having an email sent to ghc-devs@ every time something breaks (and getting down false positives here is very important) is already a huge win, no matter what kind of build system sits behind that email.

The ghc-builds mailing list has the all the results -- however, earlier there was the concern that folks do not want to watch it every day. I still do, just to know everything is all right. Filtering the false positives out might be solved automatically, I think, hopefully I can come up with a solution soon.

I don't read ghc-builds at all because I don't want to see emails that say "everything is fine" in my inbox. I assume everything is fine until something tells me it isn't. :)

Páli Gábor János

5:46 a.m.

2014-04-08 11:34 GMT+02:00 Johan Tibell :

...

I don't read ghc-builds at all because I don't want to see emails that say "everything is fine" in my inbox. I assume everything is fine until something tells me it isn't. :)

For what it is worth, you do not have subscribe to any mailing list just to see a brief overview of the builders. The builder server maintains a nice web site for the results, e.g. [1]. But I see what you mean: I will work something out for forwarding valid breakage notifications directly to ghc-devs. [1] http://haskell.inf.elte.hu/builders/

Alain O'Dea

7:59 a.m.

This is great. I'm excited about this. Páli, I am happy to do troubleshooting, recruiting, and assistance for new build slave volunteers. I will gladly support your leadership on this. I will work to ensure that you don't carry an undue share of the effort. I have four (one in slings due to a busted PSU) reasonably powerful (if old) Dell Precision T3500s that can provide a variety of x86 and x86_64 OS builds on Xeon. They currently run SmartOS, but they can run other x86 and x86_64 guests. They are purpose-built for isolated virtualization. The biggest limitation for these right now is RAM (they have 6GB each), but I'm considering more soon. I'll send my first build slave username and password off list to you later today. Once I have that working I'll document the current process for volunteering and setting up build slaves. Best, Alain

...

On Apr 8, 2014, at 8:50, Páli Gábor János wrote:

2014-04-08 10:30 GMT+02:00 Joachim Breitner :

...
we also need a culture of just doing stuff, and less asking for it.

Yes, I was educated in the same spirit but in the FreeBSD Project. I did not ask much for it when I replicated the whole setup just for myself last year.

...
if you want people to join their builders, tell them what information you need from them and add them. Feel free to modify the wiki so that people find you. Make up some rules (about usernames etc.) as you go, if necessary.

That is good to hear. First, I had the impression from the previous discussions that Ian's solution is not proven enough so you want to go for some other solution. Second, I do not want to duplicate anybody else's efforts. Although I have already stated that I am willing to let others connect to my server and replied the related mails, but I felt that the offer was still ignored or lost.

I do not want to be pushy, I do not like stepping on other's toes. But actually I can if that is what you want -- that is how I did eight BSD workshops and developer summits in the last four years and eventually become the secretary of the FreeBSD Core Team.

...
Just do it. And tell us about your achievements.

I guess the ghc-builds mailing list speaks for itself.

...
Also worry less about official or not. The Travis setup is not official, but (IMHO) has been useful quite a few times. I'd _like_ it to be official, i.e. hosted on git.haskell.org, but that is not important.

All right, if the rules of game are like that, let it be so...

...
If your service becomes "critical" in some sense it is still time to move it some official infrastructure... but that can come second, and should not hinder anyone from contributing.

Okay, thanks for the clarification! _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

Austin Seipp

8:16 a.m.

Hello all, I was actually going to send an email about this as I'm wrapping up 7.8.1, but since it got started early... Yes, I would really like if someone would be willing to help manage infrastructure for build slaves. If people are willing to support and run with what we've got now, that's great! Simon and I talked about this recently - the decision about what infrastructure to use was somewhat in the air (custom infrastructure has the downside it has less overall support of course), but we figured we'd bring it to the people to discuss, but you beat me to it! Here's a few extra notes: - I can absolutely provide infrastructure, especially for Windows virtual machines, using Rackspace. This is one thing that's missing, and it's how I'm building binaries now. In addition, we can also offer a lot of Linux (and FreeBSD systems) through this. The prices for very low-powered builders are very cheap, and we could easily add several of them for either CI or nightly builds, in 32 and 64bit configurations. This can effectively be free for a lot of supported platforms. - I can also absolutely make any infrastructure 'official' by giving it a domain name, for example, and we can lean on this free infrastructure for what we can, in addition to any machines people are willing to offer. We'll need these extra machines for more platforms! - Whatever we do, I'd really like better visibility. I don't think ghc-builds is enough, personally - historically people seem to ignore it unless someone manages to eye a problem or alert someone else, and I think this was because in the past there was a lot of noise which lead to this sort of perception. But really, it would be great if we could offer a web interface or something. An IRC bot would be useful too - more and more 'regular' users hang out there, and it provides a nice form of passive but responsive feedback. There are certainly fairly pre-canned solutions to this I'm sure. Anyway, whatever we do, I'm happy to support people with the resources, access and visibility they need. Pali, since you seem to leading this - what are your thoughts? I'm more than willing to give you some hardware and put it under haskell.org domain, and just get out of your way if you'd like. :) On Tue, Apr 8, 2014 at 6:59 AM, Alain O'Dea wrote:

...

This is great. I'm excited about this.

Páli, I am happy to do troubleshooting, recruiting, and assistance for new build slave volunteers. I will gladly support your leadership on this. I will work to ensure that you don't carry an undue share of the effort.

I have four (one in slings due to a busted PSU) reasonably powerful (if old) Dell Precision T3500s that can provide a variety of x86 and x86_64 OS builds on Xeon. They currently run SmartOS, but they can run other x86 and x86_64 guests. They are purpose-built for isolated virtualization. The biggest limitation for these right now is RAM (they have 6GB each), but I'm considering more soon.

I'll send my first build slave username and password off list to you later today. Once I have that working I'll document the current process for volunteering and setting up build slaves.

Best, Alain

...
On Apr 8, 2014, at 8:50, Páli Gábor János wrote:

2014-04-08 10:30 GMT+02:00 Joachim Breitner :

...
we also need a culture of just doing stuff, and less asking for it.

Yes, I was educated in the same spirit but in the FreeBSD Project. I did not ask much for it when I replicated the whole setup just for myself last year.

...
if you want people to join their builders, tell them what information you need from them and add them. Feel free to modify the wiki so that people find you. Make up some rules (about usernames etc.) as you go, if necessary.

That is good to hear. First, I had the impression from the previous discussions that Ian's solution is not proven enough so you want to go for some other solution. Second, I do not want to duplicate anybody else's efforts. Although I have already stated that I am willing to let others connect to my server and replied the related mails, but I felt that the offer was still ignored or lost.

I do not want to be pushy, I do not like stepping on other's toes. But actually I can if that is what you want -- that is how I did eight BSD workshops and developer summits in the last four years and eventually become the secretary of the FreeBSD Core Team.

...
Just do it. And tell us about your achievements.

I guess the ghc-builds mailing list speaks for itself.

...
Also worry less about official or not. The Travis setup is not official, but (IMHO) has been useful quite a few times. I'd _like_ it to be official, i.e. hosted on git.haskell.org, but that is not important.

All right, if the rules of game are like that, let it be so...

...
If your service becomes "critical" in some sense it is still time to move it some official infrastructure... but that can come second, and should not hinder anyone from contributing.

Okay, thanks for the clarification! _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

-- Regards, Austin Seipp, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/

Páli Gábor János

8:30 a.m.

2014-04-08 14:16 GMT+02:00 Austin Seipp :

...

Pali, since you seem to leading this - what are your thoughts? I'm more than willing to give you some hardware and put it under haskell.org domain, and just get out of your way if you'd like. :)

I have summarized some of my thoughts in my previous mail to Alain:

...

I think the best would be if we could cover all the possible and probable combination of architectures and platforms and avoid the redundancies.

Earlier, there was the concept of "sponsor" for the given platform, I guess it would still make sense to re-introduce it. The sponsor is somebody who is willing to pay attention to the given platform, run a builder client for it, therefore maintaining it. For Tier-1 platforms, such support was a requirement back then. So, we would need to set up at least the following builders: {Linux, Windows, Mac OS X} {x86, x86_64}, and then I could migrate all the others I have on my machine to there as well.

Austin Seipp

9:51 p.m.

Right, I can provide right now, with minimal fuss:: - 64 bit Linux (32bit as well, if you just use a chroot - not hard. Rackspace just doesn't provide 32bit VMs by default these days). 24/7 365 availability, roughty. - 32 bit Windows and 64bit Windows. 24/7 365 availability, roughly. - I do have a dedicated OS X Lion machine that can be a 64bit Mac build slave. But no 32bit machine. And occasionally the network sometimes cuts out, but this is normally fixable and/or transient. So I think I can already cobble together all the necessary Tier 1 platforms to get us going, I just need to point the machines somewhere to get them started. In lieu of anyone else, I can monitor the Windows client. I can monitor the Linux and OS X builds of course (and do so regularly anyway), since it seems FreeBSD and Solaris are well cared for (hooray!) But ideally someone else can do so too, and I'd really like that! Anyone willing? I'll look into getting these running with builder clients as soon as possible and follow up. On Tue, Apr 8, 2014 at 7:30 AM, Páli Gábor János wrote:

...

2014-04-08 14:16 GMT+02:00 Austin Seipp :

...
Pali, since you seem to leading this - what are your thoughts? I'm more than willing to give you some hardware and put it under haskell.org domain, and just get out of your way if you'd like. :)

I have summarized some of my thoughts in my previous mail to Alain:

...
I think the best would be if we could cover all the possible and probable combination of architectures and platforms and avoid the redundancies.

Earlier, there was the concept of "sponsor" for the given platform, I guess it would still make sense to re-introduce it. The sponsor is somebody who is willing to pay attention to the given platform, run a builder client for it, therefore maintaining it. For Tier-1 platforms, such support was a requirement back then.

So, we would need to set up at least the following builders: {Linux, Windows, Mac OS X} {x86, x86_64}, and then I could migrate all the others I have on my machine to there as well.

-- Regards, Austin Seipp, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/

Páli Gábor János

8:17 a.m.

2014-04-08 13:59 GMT+02:00 Alain O'Dea :

...

I have four (one in slings due to a busted PSU) reasonably powerful (if old) Dell Precision T3500s that can provide a variety of x86 and x86_64 OS builds on Xeon. They currently run SmartOS, but they can run other x86 and x86_64 guests.

I think the best would be if we could cover all the possible and probable combination of architectures and platforms and avoid the redundancies. So far, we have the following: Solaris x86, Linux/arm and Linux/ppc64 (by Karel), FreeBSD/i386 and FreeBSD/amd64 (by myself), and Mateusz has written me about setting up a Linux x86 builder on Gentoo. We could certainly use Linux x86_64 builders, so as 32-bit and 64-bit Mac OS X and Windows ones. Regarding Linux, there might be builders for each of the distributions, however, I am not sure if need to store downloadable snapshots for all of them.

...

I'll send my first build slave username and password off list to you later today.

You do not have to, I can generate both for you, once the OS and the architecture is known.

...

Once I have that working I'll document the current process for volunteering and setting up build slaves.

I think this has been already covered on the GHC Developers' wiki, there: https://ghc.haskell.org/trac/ghc/wiki/Builder Note that this page is also linked from the front page of the wiki.

Alain O'Dea

9:16 a.m.

Hi,

...

On Apr 8, 2014, at 12:17, Páli Gábor János wrote:

2014-04-08 13:59 GMT+02:00 Alain O'Dea :

...
I have four (one in slings due to a busted PSU) reasonably powerful (if old) Dell Precision T3500s that can provide a variety of x86 and x86_64 OS builds on Xeon. They currently run SmartOS, but they can run other x86 and x86_64 guests.

I think the best would be if we could cover all the possible and probable combination of architectures and platforms and avoid the redundancies.

So far, we have the following: Solaris x86,

I think SmartOS can reasonably stand in for Illumos and Solaris x86_64 and can run Ubuntu x86_64 as a KVM guest.

...

Linux/arm and Linux/ppc64 (by Karel), FreeBSD/i386 and FreeBSD/amd64 (by myself), and Mateusz has written me about setting up a Linux x86 builder on Gentoo. We could certainly use Linux x86_64 builders, so as 32-bit and 64-bit Mac OS X and Windows ones. Regarding Linux, there might be builders for each of the distributions, however, I am not sure if need to store downloadable snapshots for all of them.

...
I'll send my first build slave username and password off list to you later today.

You do not have to, I can generate both for you, once the OS and the architecture is known.

Cool, that's easier. I usually use AlainODea or alainodea as my base username, but that's immaterial. The two build slaves I intend to run are: - x86_64 Solaris (on SmartOS) - x86_64 Linux (on Ubuntu as a SmartOS KVM guest)

...

...
Once I have that working I'll document the current process for volunteering and setting up build slaves.

I think this has been already covered on the GHC Developers' wiki, there:

https://ghc.haskell.org/trac/ghc/wiki/Builder

Good point. It needs some minor tweaks since the volunteers need to request a username and password by providing their OS and architecture to the list rather than sending their username and password to the list.

...

Note that this page is also linked from the front page of the wiki.

Karel Gardas

4:17 p.m.

On 04/ 8/14 03:16 PM, Alain O'Dea wrote:

...

The two build slaves I intend to run are: - x86_64 Solaris (on SmartOS)

Please name it smartos-x86 then. I'm running real Solaris 11.1 as a builder here so let's make it different name builder and see if there are any incompatibilities between those two (I hope still very close) platforms. BTW: what's your platform triple detected by config.guess? Thanks! Karel

Alain O'Dea

9:41 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 14-04-08 08:17 PM, Karel Gardas wrote:

...

On 04/ 8/14 03:16 PM, Alain O'Dea wrote:

...
The two build slaves I intend to run are: - x86_64 Solaris (on SmartOS)

Please name it smartos-x86 then. I'm running real Solaris 11.1 as a builder here so let's make it different name builder and see if there are any incompatibilities between those two (I hope still very close) platforms.

BTW: what's your platform triple detected by config.guess?

Thanks! Karel

Hi Karel, My platform triple detected by config.guess is: x86_64-unknown-solaris2 I got that by running `cat config.status | grep TargetPlatformFull`. Best, Alain -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJTRKVFAAoJEP0rIXJNjNSAYAsH/16CiNTRvrx2aTEekMw/Rf19 2XgNLliGzqmMZn8tub2+CN6p2eeaYUWbxm3C4Hczca3h3HPpThMvqMc7vZ1v6Dav gGyC6OJpXHhlGQUxmAzXrYVJ/MZOwC8o+rXCulEv9xZJWZMsxffMN57/3azNdMwm SAj9qmeLieyK1rOLoA+vWcnobkOGfVtjFLxej1xs8tRZrdq3aLaEl65k737COT7p q24LUwgSo3YOC+uf3/8SQdsbtvAroB5+IYoi9nuBDmHImRmw58Mu8n3Cn7wDqfaB jCHTQQoVsweSAZuxDGNBn07eQXB4gq8j0X4SL/mYNDLA7/MiCdcKtawJKY1Zw7o= =iK6S -----END PGP SIGNATURE-----

Karel Gardas

9 Apr 9 Apr

4:36 a.m.

On 04/ 9/14 03:41 AM, Alain O'Dea wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 14-04-08 08:17 PM, Karel Gardas wrote:

...
On 04/ 8/14 03:16 PM, Alain O'Dea wrote:

...
The two build slaves I intend to run are: - x86_64 Solaris (on SmartOS)

Please name it smartos-x86 then. I'm running real Solaris 11.1 as a builder here so let's make it different name builder and see if there are any incompatibilities between those two (I hope still very close) platforms.

BTW: what's your platform triple detected by config.guess?

Thanks! Karel

Hi Karel,

My platform triple detected by config.guess is: x86_64-unknown-solaris2

I got that by running `cat config.status | grep TargetPlatformFull`.

Hi Alain, hmm, that's after GHC own platform processing is done, but what does sh config.guess tells you? E.g. on my two Solaris 11 I get: $ sh config.guess i386-pc-solaris2.11 $ sh config.guess sparc-sun-solaris2.11 BTW: also if you are building GHC on AMD64/Solaris/err/SmartOS, then I think Christian (cced) may be interested as he is currently working on cross-compiling from i386/Solaris to AMD64/Solaris and he is hit by some bugs along the way. So if you do have already binary for GHC on that, then perhaps it'll work on Solaris too... You may wonder why I'm so picky about Solaris, but I've just been hit on Solaris 10 by bugs (in linker) which are not presented on Solaris 11 so in the past I needed to switch off shared libs on Solaris 10 and keep this functionality on Solaris 11 so I'm curious what third member to Solaris family will bring here. I hope SmartOS is still more closer to Solaris 11 than Solaris 10 thanks to its roots in OpenSolaris project... Thanks, Karel

Alain O'Dea

7:49 a.m.

...

On Apr 9, 2014, at 8:36, Karel Gardas wrote:

...
On 04/ 9/14 03:41 AM, Alain O'Dea wrote: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

...
On 14-04-08 08:17 PM, Karel Gardas wrote:

...
On 04/ 8/14 03:16 PM, Alain O'Dea wrote: The two build slaves I intend to run are: - x86_64 Solaris (on SmartOS)

Please name it smartos-x86 then. I'm running real Solaris 11.1 as a builder here so let's make it different name builder and see if there are any incompatibilities between those two (I hope still very close) platforms.

BTW: what's your platform triple detected by config.guess?

Thanks! Karel

Hi Karel,

My platform triple detected by config.guess is: x86_64-unknown-solaris2

I got that by running `cat config.status | grep TargetPlatformFull`.

Hi Alain,

hmm, that's after GHC own platform processing is done, but what does

sh config.guess

tells you? E.g. on my two Solaris 11 I get:

$ sh config.guess i386-pc-solaris2.11

$ sh config.guess sparc-sun-solaris2.11

Ah. I'll have to send that later today.

...

BTW: also if you are building GHC on AMD64/Solaris/err/SmartOS, then I think Christian (cced) may be interested as he is currently working on cross-compiling from i386/Solaris to AMD64/Solaris and he is hit by some bugs along the way. So if you do have already binary for GHC on that, then perhaps it'll work on Solaris too...

Christian, the SmartOS GHC binaries should work unmodified on Solaris 10 and 11. Both 32- and 64-bit GHC 7.6.3 are available in SmartOS PKGSRC. I have successfully used them as bootstrap for GHC 7.8. The binaries should be separable. I can get the build details if desired.

...

You may wonder why I'm so picky about Solaris, but I've just been hit on Solaris 10 by bugs (in linker) which are not presented on Solaris 11 so in the past I needed to switch off shared libs on Solaris 10 and keep this functionality on Solaris 11 so I'm curious what third member to Solaris family will bring here.

It seems SmartOS builds GHC 7.8 cleanly without patches. There are some test failures which I'm working on.

...

I hope SmartOS is still more closer to Solaris 11 than Solaris 10 thanks to its roots in OpenSolaris project...

SmartOS is an Illumos distro (like Ubuntu is to Linux). Illumos is descended directly from the last public commit of OpenSolaris 10 (grr Oracle).

...

Thanks, Karel

Alain O'Dea

8:52 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 14-04-09 11:49 AM, Alain O'Dea wrote:

...

...
On Apr 9, 2014, at 8:36, Karel Gardas wrote:

...
On 04/ 9/14 03:41 AM, Alain O'Dea wrote: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

...
On 14-04-08 08:17 PM, Karel Gardas wrote:

...
On 04/ 8/14 03:16 PM, Alain O'Dea wrote: The two build slaves I intend to run are: - x86_64 Solaris (on SmartOS)

Please name it smartos-x86 then. I'm running real Solaris 11.1 as a builder here so let's make it different name builder and see if there are any incompatibilities between those two (I hope still very close) platforms.

BTW: what's your platform triple detected by config.guess?

Thanks! Karel

Hi Karel,

My platform triple detected by config.guess is: x86_64-unknown-solaris2

I got that by running `cat config.status | grep TargetPlatformFull`.

Hi Alain,

hmm, that's after GHC own platform processing is done, but what does

sh config.guess

tells you? E.g. on my two Solaris 11 I get:

$ sh config.guess i386-pc-solaris2.11

$ sh config.guess sparc-sun-solaris2.11

Karel, I think I finally have what you were looking for: x86_64-pc-solaris2.11 i386-pc-solaris2.11 Gábor sent me usernames and passwords for my builders and I'm in the process of setting those up now.

...

Ah. I'll have to send that later today.

...
BTW: also if you are building GHC on AMD64/Solaris/err/SmartOS, then I think Christian (cced) may be interested as he is currently working on cross-compiling from i386/Solaris to AMD64/Solaris and he is hit by some bugs along the way. So if you do have already binary for GHC on that, then perhaps it'll work on Solaris too...

Christian, the SmartOS GHC binaries should work unmodified on Solaris 10 and 11. Both 32- and 64-bit GHC 7.6.3 are available in SmartOS PKGSRC. I have successfully used them as bootstrap for GHC 7.8. The binaries should be separable. I can get the build details if desired.

...
You may wonder why I'm so picky about Solaris, but I've just been hit on Solaris 10 by bugs (in linker) which are not presented on Solaris 11 so in the past I needed to switch off shared libs on Solaris 10 and keep this functionality on Solaris 11 so I'm curious what third member to Solaris family will bring here.

It seems SmartOS builds GHC 7.8 cleanly without patches. There are some test failures which I'm working on.

...
I hope SmartOS is still more closer to Solaris 11 than Solaris 10 thanks to its roots in OpenSolaris project...

SmartOS is an Illumos distro (like Ubuntu is to Linux). Illumos is descended directly from the last public commit of OpenSolaris 10 (grr Oracle).

...
Thanks, Karel

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJTRetbAAoJEP0rIXJNjNSAGpIH/Rd0Kp4K3OV5jIdwiT3x1GNI jbed0L48bOB9u4ChasoLsq+12iOhwYkb4eBd+hAWGnjW+/EDsX7F19qfuBugsr9a GiYyGRprRaMFMUQ0DR1pFPqeLKe5EUaNw5Al4KVW5i9W3LDCwGL3pQI8D0dRYzlN 4QlG7OcDG9DN8mTyiFAtWOjFloqkQN1G6EQG1GbwkHSdjKrXVRfeatAxMl9QfS7H I1o9sLDKJWQTL38XmnMuXWKqLmvxundO0UUJNmvmKoJdSnRnBvD5m4BvnpIN1Cl2 xVnIvPef5PuPE5I1EXqZu61zUD2qgqmyVCHZui5D+ltZoEUS0Hh94JSzb2YOSGs= =mu2x -----END PGP SIGNATURE-----

Karel Gardas

10 Apr 10 Apr

3:57 a.m.

On 04/10/14 02:52 AM, Alain O'Dea wrote:

...

...
...
hmm, that's after GHC own platform processing is done, but what does

sh config.guess

tells you? E.g. on my two Solaris 11 I get:

$ sh config.guess i386-pc-solaris2.11

$ sh config.guess sparc-sun-solaris2.11

Karel, I think I finally have what you were looking for:

x86_64-pc-solaris2.11 i386-pc-solaris2.11

Great, so this means SmartOS is just another member of Solaris family. I've installed it and verified that `ld' and `nm' are really what we expect on real Solaris. In fact thanks for your advice I've been able to use its packaged GHC on my Solaris 11 to compile some code and even attempted the bootstrap of HEAD for Solaris/AMD64 platform. There are some outstanding issues with bootstrap so this needs to wait till my weekend ghc hobby time...(if someone does not solved it faster of course...) Thanks! Karel

Alain O'Dea

7:45 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thu 10 Apr 2014 07:57:16 AM UTC, Karel Gardas wrote:

...

On 04/10/14 02:52 AM, Alain O'Dea wrote:

...
...
...
hmm, that's after GHC own platform processing is done, but what does

sh config.guess

tells you? E.g. on my two Solaris 11 I get:

$ sh config.guess i386-pc-solaris2.11

$ sh config.guess sparc-sun-solaris2.11

Karel, I think I finally have what you were looking for:

x86_64-pc-solaris2.11 i386-pc-solaris2.11

Great, so this means SmartOS is just another member of Solaris family. I've installed it and verified that `ld' and `nm' are really what we expect on real Solaris.

Them's fightin' words ;) There are hard feelings (with good reason) between Illumos and Solaris. Oracle betrayed the OpenSolaris community and particularly their Open Source contributors when they closed Solaris. It was a very unethical thing to do. Bryan Cantrill spoke well on the reasons for and nature of the Illumos fork at LISA11: http://smartos.org/2011/12/15/fork-yeah-the-rise-and-development-of-illumos-... I am very happy that they remain binary compatible. I can immediately use Solaris 10 binaries on Illumos and in many cases Solaris 10 and 11 can run Illumos binaries.

...

In fact thanks for your advice I've been able to use its packaged GHC on my Solaris 11 to compile some code and even attempted the bootstrap of HEAD for Solaris/AMD64 platform. There are some outstanding issues with bootstrap so this needs to wait till my weekend ghc hobby time...(if someone does not solved it faster of course...)

Thanks! Karel -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJTRy0bAAoJEP0rIXJNjNSA4oUH/irlhd1xztoUA1yTDB/y5eDs jygqjp5cymU9jELfMGTJqTzuIyw/whvpDcqmiPqEaDrWdTgCUIHraZrxk0UTT/BF w6jtY1dBCqECUkQhT5Pdr/T/GtnRxVItiMGxn6fJ8c4yWb0HDcMFmXmCkyrwQQi6 ZwiLqpWu8P1zAHplbaOeEihusRaKtllOEm07eIajZdYcyjCoszgQrZyORaBVllkh Czwyk9WCkkh9u9GWhYZu7p1p8Z7ToJ/lrv/VgGWxbaCZcS1q+j4a7Z9r41L6HJ8v mgpEqBNtKgVZ1cRzVN8sapinXWt6PoNR38dHHUEGeW76z9TrqD5pPvOab9ROfGc= =5KpS -----END PGP SIGNATURE-----

Simon Marlow

11 Apr 11 Apr

4:40 a.m.

On 08/04/2014 09:30, Joachim Breitner wrote:

...

we also need a culture of just doing stuff, and less asking for it.

Where is the "like" button on this line? +1 Simon

Simon Peyton Jones

18 Jun 18 Jun

6:47 p.m.

Back in April I said: | Seriously, I advertised a couple of weeks ago for help with our nightly- | build infrastructure. Quite a few people responded -- thank you very | much. | | So we have willing horsepower. But the moment we lack leadership. Alain | rightly says "I don't know what the process is" because we don't *have* a | process. We need a mechanism for creating a process, taking decisions, | etc. | | I think what is needed is: | | * A group of people willing to act as a kind of committee. That | could be everyone who replied. You could create a mailing list, | or (initially better) just chat on ghc-devs. But it would be | useful to have a list of who is involved. | | * Someone (or a couple of people) to play the role of chair. | That doesn't mean an autocrat... it means someone who gently pushes | discussions to a conclusion, and says "I propose that we do X". | | * Then the group can formulate a plan and proceed with it. | For example, should Pali's efforts be "blessed"? I don't | know enough to know, but you guys do. | | In my experience, people are often unwilling to put themselves forward as | chair, not because they are unwilling, but because they feel it'd be | "pushy". So I suggest this: if you think (based on the traffic you've | seen) that X would be a chair you'd trust, suggest them. | | In short: power to the people! GHC is your compiler. Since then various people have done various things, but so far as I know we don't have any of the three "*" items above. The people who seem in principle willing to help include Joachim Breitner Herbert Valerio Riedel Páli Gábor János Karel Gardas Alain O'Dea William Knop Austin Seipp There may well be others! I sense that the problem is not willingness but simply that no one feels accredited to take the lead. Please, I would love someone to do so! I was reminded of this by William Knop's recent message below, in which he implicitly offers to help (thanks William). But his offer will fall on deaf ears unless that little group exists to welcome him in. In hope, and with thanks, Simon | -----Original Message----- | From: ghc-devs [mailto:ghc-devs-bounces@haskell.org] On Behalf Of William | Knop | Sent: 18 June 2014 21:10 | To: ghc-devs@haskell.org | Subject: Continuous Integration and Cross Compilation | | Hello all, | | I've seen quite a few comments on the list and elsewhere lamenting the | time it takes to compile and validate ghc. It's troublesome not only | because it's inconvenient, but, more seriously, people are holding off on | sending patches in which stifles development. I would like to propose a | solution: | | 1. Implement proper cross-compilation, such that build and host may be | different- e.g. a linux x86_64 machine can build ghc that runs on Windows | x86. What sort of work would this entail? | | 2. Batch cross-compiled builds for all OSs/archs on a continuous | integration service (e.g. Travis CI) or cloud service, then package up | the binaries with the test suite. | | 3. Send the package to our buildbots, and run the test suite. | | 4. (optional) If using a CI service, have the buildbots send results back | to the CI. This could be useful if we'd use GitHub for pulls in the | future *. | | Cheers, | Will | | | * I realize vanilla GitHub currently has certain annoying limitations, | though some of them are pretty easy to solve via the github-services | and/or webhooks. I don't think this conflicts with the desire to use | Phabricator, either, so I'll send details and motivations to that thread. |

3989

Age (days ago)

4060

Last active (days ago)

List overview

Download

26 comments

9 participants

participants (9)

Alain O'Dea
Austin Seipp
Joachim Breitner
Johan Tibell
Karel Gardas
Mateusz Kowalczyk
Páli Gábor János
Simon Marlow
Simon Peyton Jones