Simon Marlow wrote:

On 21 October 2005 21:12, Jacques Carette wrote:

...

I would strongly recommend against 'allow everyone to just commit' without the presence of a large automated test suite which is used to (automatically) reject code that breaks a test.

pre-commit testing using GHC isn't really practical; it would be a huge bottleneck. The asynchronous testing we have right now (full testsuite every night) is a better compromise.

And for check-ins of changesets that affect a *lot* of tests, this is also the compromise that was chosen at Maplesoft. The upshot of that is that it 1) complicated the automated rejection of code a lot, enough so that 2) it necessitates having someone monitor nightly test failures every day to manually figure out what managed to make things fail #2 can vary from very little work to quite onerous, depending on the level of activity. It only gets bad when things are allowed to degenerate. Interestingly cases of many tests failures seem to be more people-related than anything else. So a social process (ie making them feel really guilty by letting everyone know who screwed up the build, again) seems to reach a workable middle ground quite efficiently.

However, online testing of libraries using Hugs might be do-able. It would make a nice little project if anyone's interested. I would happily run a bunch of Hugs tests before committing a library change if it was done either with a single command or automatically by the checkin.

I can't *do* this, but I can certainly help with the design. I helped write parts of Maplesoft's "rfindtest" (script to find which tests to run given a changeset) and "robocop" (script to find which changeset most likely is the source of a particular test failure). The hardest part is to ensure that one has the right data available in the right format, the rest is easy database query stuff coupled with lots of scripting. Being able to run the appropriate 100 tests (out of several million) for a small change certainly is very convenient! Jacques