Hi Joachim: From what I understand Travis CI limits running time for each build. We may be able to create binaries of stage1 and/or stage2 in one build and test them in another. We could also fan out the test process using a Build Matrix to let GHC's full suite fit into the time limit as fragments. That would require some changes to the testsuite or some lengthy build scripts for each segment that explicitly run many "make TEST=sometest" commands. I had to do something similar at Verafin because the whole test suite was taking too long as a whole unit. I split it into builds for each separate module and used more build agents so they could run in parallel. Verafin is using TeamCity, but I think the concepts are achievable in Buildbot or Travis CI. In my opinion, the best build and CI system is the one you can get working. Best, Alain

On Apr 1, 2014, at 10:46, Joachim Breitner wrote:

Hi,

Am Dienstag, den 01.04.2014, 10:25 +0000 schrieb Simon Peyton Jones:

...

Joachim Breitner has set up Travis-CI. (I don't know exactly what that is, but it sounds useful.)

Travis is a free cloud service that runs arbitrary tests (in our case, a stripped version of validate) upon pushes to git repositories on github. I set it up to validate our master, so we get a nice history of successes and failures on https://travis-ci.org/nomeata/ghc-complete/builds and I get mails when things fail; that is when I send hopefully polite enough mails to ghc-dev, asking people to fix their commits.

(Unless I broke it myself; then I silently fix it and hide.)

It is a makeshift solution until we get our own infrastructure working.

...

An early question would be: to continue to use a DIY system (Builder), or to move to some other better-supported (but perhaps less malleable) system. I don't even know what the options are.

Sigh, test infrastructure are like content management systems: There are plenty out there to choose from, all can do lots of things one does not need, but none can do all, so one starts writing something selfmade, which eventually evolves in yet another of these beasts, just with fewer users.

I’d recommend a move to existing, proven tools. Unfortunately, I cannot give advice as to what tool to move to. But if all these¹ projects are happy with buildbot, it might not be the worst choice.

¹ http://trac.buildbot.net/wiki/SuccessStories

Greetings, Joachim -- Joachim “nomeata” Breitner mail@joachim-breitner.de • http://www.joachim-breitner.de/ Jabber: nomeata@joachim-breitner.de • GPG-Key: 0x4743206C Debian Developer: nomeata@debian.org _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

Hi,

On Apr 1, 2014, at 11:11, Joachim Breitner wrote:

Hi,

Am Dienstag, den 01.04.2014, 11:08 +0000 schrieb Alain O'Dea:

...

From what I understand Travis CI limits running time for each build. We may be able to create binaries of stage1 and/or stage2 in one build and test them in another. We could also fan out the test process using a Build Matrix to let GHC's full suite fit into the time limit as fragments. That would require some changes to the testsuite or some lengthy build scripts for each segment that explicitly run many "make TEST=sometest" commands.

The main problem with Travis is that it only tests on one architecture. There are more (e.g. very undetailed reporting compared to a buildbot waterfall/grid)

Good points.

Hence: Travis is _not_ going to be a solution for us; we will want our own infrastructure.

I'm happy to assist with getting whatever system fits the bill working. It seems like Buildbot should. Where can we get infrastructure on multiple architectures easily? I imagine we may need to mix and match.

Greetings, Joachim

-- Joachim “nomeata” Breitner mail@joachim-breitner.de • http://www.joachim-breitner.de/ Jabber: nomeata@joachim-breitner.de • GPG-Key: 0x4743206C Debian Developer: nomeata@debian.org

On Tue, Apr 1, 2014 at 1:11 PM, Joachim Breitner wrote:

Hi,

Am Dienstag, den 01.04.2014, 11:08 +0000 schrieb Alain O'Dea:

...

From what I understand Travis CI limits running time for each build. We may be able to create binaries of stage1 and/or stage2 in one build and test them in another. We could also fan out the test process using a Build Matrix to let GHC's full suite fit into the time limit as fragments. That would require some changes to the testsuite or some lengthy build scripts for each segment that explicitly run many "make TEST=sometest" commands.

The main problem with Travis is that it only tests on one architecture. There are more (e.g. very undetailed reporting compared to a buildbot waterfall/grid)

Hence: Travis is _not_ going to be a solution for us; we will want our own infrastructure.

I do agree, but if anybody wants to look more closely into using Travis-CI, I suggest to also consider drone.io. It appears to fix some of the shortcomings and annoyances of Travis-CI.

On Tue, Apr 1, 2014 at 1:48 PM, Joachim Breitner wrote:

Hi Tuncer,

Am Dienstag, den 01.04.2014, 13:41 +0200 schrieb Tuncer Ayaz:

...

...

Hence: Travis is _not_ going to be a solution for us; we will want our own infrastructure.

I do agree, but if anybody wants to look more closely into using Travis-CI, I suggest to also consider drone.io. It appears to fix some of the shortcomings and annoyances of Travis-CI.

but not the timelimit: Limits

The following limits are placed upon all builds:

* a build cannot exceed 15 minutes (free tier only)

Expect https://drone.io/github.com/nomeata/ghc-complete/1 to fail in 10 minutes...

Yeah, but if you're serious about using their service, I wouldn't be surprised for them to make an exception for some open source projects. BTW, a reasonable hard timeout makes a lot of sense to me in case a build can be stuck. With that being said, I also agree with Karel wrt reinstating the "old" builder infrastructure, especially considering coverage of different hardware architectures. In fact, I recall an email from Ian some time ago with details on what would be required to restore that.

Indeed, there is no reason not to use Ian et al's Builder stuff. It's one of the options. But it depends on a critical evaluation of what the advantages and disadvantages of different approaches are Simon | -----Original Message----- | From: Glasgow-haskell-users [mailto:glasgow-haskell-users- | bounces@haskell.org] On Behalf Of Karel Gardas | Sent: 01 April 2014 12:46 | To: Joachim Breitner; ghc-devs@haskell.org; Páli Gábor János | Cc: glasgow-haskell-users@haskell.org | Subject: Re: Buildbots | | | Hi, | | I'm curious why not to use what's already written by Ian and others and | which is currently running again? E.g. Janos Gabor Pali was so nice to | start and keep builder server running on | http://haskell.inf.elte.hu/builders/ | | Just few are there, but others may be added. Just send email to Janos to | get the credentials and more information about what to get from where | (recent builder client is needed). | | Perhaps we shall more advertise this option on GHC's builder wiki page? | | Thanks, | Karel | | | On 04/ 1/14 01:11 PM, Joachim Breitner wrote: | > Hence: Travis is _not_ going to be a solution for us; we will want our | > own infrastructure. | _______________________________________________ | Glasgow-haskell-users mailing list | Glasgow-haskell-users@haskell.org | http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

2014-04-01 14:03 GMT+02:00 Simon Peyton Jones :

Indeed, there is no reason not to use Ian et al's Builder stuff. It's one of the options. But it depends on a critical evaluation of what the advantages and disadvantages of different approaches are

I found Ian's buildbot an appealing alternative as it does a full build, including testing, and uploads the resulting binaries to a common place where anybody can access them (but it can be configured to do almost anything). The builders may be configured individually from a single (Haskell-language) configuration file and they are run on various volunteer-supplied systems so it is also distributed. I use this to keep track of the status of the FreeBSD builds to make my work easier on building the releases and maintaining the associated ports in the FreeBSD Ports Collection, while offering regular developer snapshots for the users. This approach also allows me to control and maintain the builder environment too as it may require minor or major changes and fixes over time that I can do myself as a FreeBSD developer. In the past, there were cases where the build was failing due to bugs in the kernel or the userland, so this is not purely about GHC itself (unfortunately). In my humble opinon, there are merits for the Travis-based Continuous Integration, so as for the daily snapshot building on each supported platform. I do not care if it is not Haskell-based or it is hosted at a central place with individual Virtual Machines for each platform -- until I can keep doing what I have been doing for 4 years now.

Thank you Carter. I think it's reasonable to incubate it on your Github profile for now until we are certain it is fully working again. Either way works though :) Best, Alain

On Apr 1, 2014, at 17:45, Carter Schonwald wrote:

hey all, I just exported the igloo builder code from darcs to git, and put it here https://github.com/cartazio/ghc-builder would this be something worth adding to github.com/haskell ? (i can easily add it if other folks it should be surfaced more visibly)

...

On Tue, Apr 1, 2014 at 11:04 AM, Páli Gábor János wrote: 2014-04-01 14:03 GMT+02:00 Simon Peyton Jones :

...

Indeed, there is no reason not to use Ian et al's Builder stuff. It's one of the options. But it depends on a critical evaluation of what the advantages and disadvantages of different approaches are

I found Ian's buildbot an appealing alternative as it does a full build, including testing, and uploads the resulting binaries to a common place where anybody can access them (but it can be configured to do almost anything). The builders may be configured individually from a single (Haskell-language) configuration file and they are run on various volunteer-supplied systems so it is also distributed.

I use this to keep track of the status of the FreeBSD builds to make my work easier on building the releases and maintaining the associated ports in the FreeBSD Ports Collection, while offering regular developer snapshots for the users. This approach also allows me to control and maintain the builder environment too as it may require minor or major changes and fixes over time that I can do myself as a FreeBSD developer. In the past, there were cases where the build was failing due to bugs in the kernel or the userland, so this is not purely about GHC itself (unfortunately).

In my humble opinon, there are merits for the Travis-based Continuous Integration, so as for the daily snapshot building on each supported platform. I do not care if it is not Haskell-based or it is hosted at a central place with individual Virtual Machines for each platform -- until I can keep doing what I have been doing for 4 years now. _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

good to know (i assumed it was in working order from your remarks) I think making it more surfaced / discoverable might enable a lot more volunteer build bots (which is an issue aside from maintaining it) of course, officially moving it to github should be with ian's blessing, its mostly his work afaik :) On Tue, Apr 1, 2014 at 2:45 PM, Páli Gábor János wrote:

2014-04-01 20:15 GMT+02:00 Alain O'Dea :

...

until we are certain it is fully working again.

In what sense? I have been using the latest checkout from the darcs repository for both the server and the clients, I seldom experienced any serious problems. Of course, there is place for improvements and there are some rough edges, but I think it is quite usable, even in its current state. _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

On 2014-04-01 at 12:46:05 +0200, Joachim Breitner wrote:

Hi,

Am Dienstag, den 01.04.2014, 10:25 +0000 schrieb Simon Peyton Jones:

...

Joachim Breitner has set up Travis-CI. (I don't know exactly what that is, but it sounds useful.)

Travis is a free cloud service that runs arbitrary tests (in our case, a stripped version of validate) upon pushes to git repositories on github. I set it up to validate our master, so we get a nice history of successes and failures on https://travis-ci.org/nomeata/ghc-complete/builds and I get mails when things fail; that is when I send hopefully polite enough mails to ghc-dev, asking people to fix their commits.

What's more, we also have TravisCI set-up for the various libraries/* sub-packages, to make sure they work as announced by their respective .cabal files with older GHC/base-version configurations (so if anybody pushes a commit that breaks that contract, email notifications are sent out), here's a non-exhaustive list of examples: - https://travis-ci.org/ghc/packages-process - https://travis-ci.org/ghc/packages-unix - https://travis-ci.org/ghc/packages-hpc - https://travis-ci.org/ghc/packages-parallel - https://travis-ci.org/ghc/packages-deepseq Cheers, hvr

Hi, Am Dienstag, den 01.04.2014, 21:07 +0200 schrieb Johan Tibell:

If the false positive rate is low, feel free to automatically have the emails sent to ghc-devs@. We want to know when we broke stuff ASAP.

unfortunately, it is not as low as it should be, partially because of hitting the 50 minute limit, partially errors while cloning or other non-GHC-related errors. So unless you insist that we should try it I’ll rather do the manual filtering. Or we could have a "ghc-bots" mailinglist where any kind of machine-generated GHC-related mails can be sent to, then interested parties can subscribe. I can also manually add people to receive the travis mails, so if any of you feel QAish today, let me know. (I was about to suggest to add the statuses and links to build logs as git notes, so that "git log" would show that, but git notes are not pullable in a convenient way, so I don’t suggest that.) Greetings, Joachim -- Joachim “nomeata” Breitner mail@joachim-breitner.de • http://www.joachim-breitner.de/ Jabber: nomeata@joachim-breitner.de • GPG-Key: 0x4743206C Debian Developer: nomeata@debian.org

Hi, Am Dienstag, den 01.04.2014, 23:38 +0200 schrieb Páli Gábor János:

2014-04-01 23:30 GMT+02:00 Joachim Breitner :

...

Or we could have a "ghc-bots" mailinglist where any kind of machine-generated GHC-related mails can be sent to, then interested parties can subscribe.

There is ghc-builds [1] for the daily snapshot builders, would not that be good for this purpose as well?

[1] http://www.haskell.org/mailman/listinfo/ghc-builds

heh, I should check if things exist before suggesting them. Travis should now also send its notifications to that address. Let me know if the false positives annoy anyone. Greetings, Joachim -- Joachim “nomeata” Breitner mail@joachim-breitner.de • http://www.joachim-breitner.de/ Jabber: nomeata@joachim-breitner.de • GPG-Key: 0x4743206C Debian Developer: nomeata@debian.org