2014-07-07 3:40 GMT+02:00 William Knop
I think using Jenkins may be a step in the right direction for a few reasons: [..] Now, I don’t have much experience with buildbots, so I may be unfairly elevating Jenkins here. If buildbots can be easily extended to do exactly what we need, I’m all for it, and in that case I’d volunteer to help in that regard.
I do not see any problem if you decide to go with Jenkins. I volunteered to maintain the buildbots because I felt it useful for maintaining the FreeBSD port, and because it did not require more than a working Haskell software stack which would the compilation require anyway. To be honest, I do not really want to fiddle with Jenkins and cloud services, and I would feel overkill to turn the buildbots into a fully-fledged CI service. This a home-brew solution and probably has no chance to compete with Jenkins, and it does not want to. I like it is implemented in the functional programming domain, that is all. For what it is worth, I am planning to extend the buildbots with more long-term testing instead. For example, it would be nice to add steps for building cabal-install and then build Stackage on every available platforms to provide some more real-world load. As a side-effect, we could also provide up-to-date snapshots for the users if they do not want to build the sources themselves, which may help with testing. I also want to add clang-based validators to see if everything works with Clang as well. And, of course, working out heuristics for spotting valid errors from the logs without much human intervention. Note that the aforementioned 80 minutes build time and 20 minutes build time is due to the single-threaded build and testing done from scratch. Obviously, with incremental builds and on more multiple threads, things would get quicker, but -- as I wrote previously -- they are disabled for clarity/correctness. Although, what we could do is launching builds for every commit, so they could preserve this invariant while utilizing the underlying hardware more. But that is where I feel this would be just in vain; Jenkins is probably a lot better solution, especially that it integrates nicely with the recently introduced Phabricator. Unfortunately, I cannot offer any experience in that regard. I see the diversity of testing implementations as an advantage, I naively believe different solutions can peacefully co-exist at the same time, helping each other.