[GHC] #13716: Move CI to Jenkins
#13716: Move CI to Jenkins -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: task | Status: new Priority: normal | Milestone: Component: None | Version: 8.0.1 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- Currently we use Harbormaster to build Differentials and commits. While this works, it leaves much to be desired, * job scheduling is (literally) random: this means that some patch authors end up waiting literally ways for their patches to be built * there is little support for automatic provisioning of builders: this means we can't scale to meet demand and make poor use of our computational resources * periodic builds (e.g. nightlies) are not supported. Ideally we would not only provide * it doesn't have a sensible interface for integration with external tools: this means that efforts like CI-before-merge have been pushed off * status reporting is poor: even answering the question of what a given builder is currently working on is surprisingly difficult * maintenance is a byzantine: Adding a new builder requires adding at least six different objects (a Harbormaster Build Plan, a Drydock Working Copy, a Drydock Blueprint, an Almanac Service, an Almanac Device, and an Almanac Binding) in various Phabricator applications. None of this configuration can be tracked under version control nor can most of it be cloned from an existing builder's configuration. * Design assumptions don't match GHC's constraints: Harbormaster was designed under the assumption that builds are cheap and computation plentiful. The maintainers have stated that they have little interest in supporting environments where this doesn't hold. Jenkins, while far from perfect, seems a bit better suited to our needs, more mature, and far more flexible. This ticket will serve as the checklist for our move to Jenkins. In the end we want, * Builders (static or dynamically-provisioned, as appropriate) for * x86-64, i386 Linux * x86-64, i386 Windows * x86-64 Darwin * x86-64 OpenBSD * Cross compile from x86-64 to ARM * Native ARM * Differential builds with sensible scheduling (e.g. first build on 64-bit Linux where machines are cheap, then build on the others) * Per-commit builds on all * Nightly builds on all, including * Collection of binary distribution for user download * Update of `master` documentation mirror on downloads.haskell.org * Slow validation (including tests requiring Hackage packages) * nofib run on some platforms? * Test-before-merge-to-master -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13716 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13716: Move CI to Jenkins
-------------------------------------+-------------------------------------
Reporter: bgamari | Owner: (none)
Type: task | Status: new
Priority: normal | Milestone:
Component: None | Version: 8.0.1
Resolution: | Keywords:
Operating System: Unknown/Multiple | Architecture:
| Unknown/Multiple
Type of failure: None/Unknown | Test Case:
Blocked By: | Blocking:
Related Tickets: | Differential Rev(s):
Wiki Page: |
-------------------------------------+-------------------------------------
Comment (by Ben Gamari
#13716: Move CI to Jenkins -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: task | Status: new Priority: normal | Milestone: Component: None | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: 13897 | Blocking: Related Tickets: #11958 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * related: => #11958 * blockedby: => 13897 Comment: #13897 needs to be resolved before we can move forward on this. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13716#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13716: Move CI to Jenkins -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: task | Status: new Priority: normal | Milestone: Component: None | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: 13897 | Blocking: Related Tickets: #11958 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Description changed by bgamari: Old description:
Currently we use Harbormaster to build Differentials and commits. While this works, it leaves much to be desired,
* job scheduling is (literally) random: this means that some patch authors end up waiting literally ways for their patches to be built * there is little support for automatic provisioning of builders: this means we can't scale to meet demand and make poor use of our computational resources * periodic builds (e.g. nightlies) are not supported. Ideally we would not only provide * it doesn't have a sensible interface for integration with external tools: this means that efforts like CI-before-merge have been pushed off * status reporting is poor: even answering the question of what a given builder is currently working on is surprisingly difficult * maintenance is a byzantine: Adding a new builder requires adding at least six different objects (a Harbormaster Build Plan, a Drydock Working Copy, a Drydock Blueprint, an Almanac Service, an Almanac Device, and an Almanac Binding) in various Phabricator applications. None of this configuration can be tracked under version control nor can most of it be cloned from an existing builder's configuration. * Design assumptions don't match GHC's constraints: Harbormaster was designed under the assumption that builds are cheap and computation plentiful. The maintainers have stated that they have little interest in supporting environments where this doesn't hold.
Jenkins, while far from perfect, seems a bit better suited to our needs, more mature, and far more flexible. This ticket will serve as the checklist for our move to Jenkins.
In the end we want,
* Builders (static or dynamically-provisioned, as appropriate) for * x86-64, i386 Linux * x86-64, i386 Windows * x86-64 Darwin * x86-64 OpenBSD * Cross compile from x86-64 to ARM * Native ARM
* Differential builds with sensible scheduling (e.g. first build on 64-bit Linux where machines are cheap, then build on the others) * Per-commit builds on all * Nightly builds on all, including * Collection of binary distribution for user download * Update of `master` documentation mirror on downloads.haskell.org * Slow validation (including tests requiring Hackage packages) * nofib run on some platforms? * Test-before-merge-to-master
New description: Currently we use Harbormaster to build Differentials and commits. While this works, it leaves much to be desired, * job scheduling is (literally) random: this means that some patch authors end up waiting literally ways for their patches to be built * there is little support for automatic provisioning of builders: this means we can't scale to meet demand and make poor use of our computational resources * periodic builds (e.g. nightlies) are not supported. * it lacks a sensible interface for integration with external tools: this means that efforts like CI-before-merge have been pushed off * status reporting is poor: even answering the question of what a given builder is currently working on is surprisingly difficult * maintenance is a byzantine: Adding a new builder requires adding at least six different objects (a Harbormaster Build Plan, a Drydock Working Copy, a Drydock Blueprint, an Almanac Service, an Almanac Device, and an Almanac Binding) in various Phabricator applications. None of this configuration can be tracked under version control nor can most of it be cloned from an existing builder's configuration. * Design assumptions don't match GHC's constraints: Harbormaster was designed under the assumption that builds are cheap and computation plentiful. The maintainers have stated that they have little interest in supporting environments where this doesn't hold. Jenkins, while far from perfect, seems a bit better suited to our needs, more mature, and far more flexible. This ticket will serve as the checklist for our move to Jenkins. In the end we want, * Builders (static or dynamically-provisioned, as appropriate) for * x86-64, i386 Linux * x86-64, i386 Windows * x86-64 Darwin * x86-64 OpenBSD * Cross compile from x86-64 to ARM * Native ARM * Differential builds with sensible scheduling (e.g. first build on 64-bit Linux where machines are cheap, then build on the others) * Per-commit builds on all * Nightly builds on all, including * Collection of binary distribution for user download * Update of `master` documentation mirror on downloads.haskell.org * Slow validation (including tests requiring Hackage packages) * nofib run on some platforms? * Test-before-merge-to-master -- -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13716#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13716: Move CI to Jenkins -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: task | Status: new Priority: normal | Milestone: Component: None | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: 13897 | Blocking: Related Tickets: #11958 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Description changed by bgamari: Old description:
Currently we use Harbormaster to build Differentials and commits. While this works, it leaves much to be desired,
* job scheduling is (literally) random: this means that some patch authors end up waiting literally ways for their patches to be built * there is little support for automatic provisioning of builders: this means we can't scale to meet demand and make poor use of our computational resources * periodic builds (e.g. nightlies) are not supported. * it lacks a sensible interface for integration with external tools: this means that efforts like CI-before-merge have been pushed off * status reporting is poor: even answering the question of what a given builder is currently working on is surprisingly difficult * maintenance is a byzantine: Adding a new builder requires adding at least six different objects (a Harbormaster Build Plan, a Drydock Working Copy, a Drydock Blueprint, an Almanac Service, an Almanac Device, and an Almanac Binding) in various Phabricator applications. None of this configuration can be tracked under version control nor can most of it be cloned from an existing builder's configuration. * Design assumptions don't match GHC's constraints: Harbormaster was designed under the assumption that builds are cheap and computation plentiful. The maintainers have stated that they have little interest in supporting environments where this doesn't hold.
Jenkins, while far from perfect, seems a bit better suited to our needs, more mature, and far more flexible. This ticket will serve as the checklist for our move to Jenkins.
In the end we want,
* Builders (static or dynamically-provisioned, as appropriate) for * x86-64, i386 Linux * x86-64, i386 Windows * x86-64 Darwin * x86-64 OpenBSD * Cross compile from x86-64 to ARM * Native ARM
* Differential builds with sensible scheduling (e.g. first build on 64-bit Linux where machines are cheap, then build on the others) * Per-commit builds on all * Nightly builds on all, including * Collection of binary distribution for user download * Update of `master` documentation mirror on downloads.haskell.org * Slow validation (including tests requiring Hackage packages) * nofib run on some platforms? * Test-before-merge-to-master
New description: Currently we use Harbormaster to build Differentials and commits. While this works, it leaves much to be desired, * job scheduling is (literally) random: this means that some patch authors end up waiting literally ways for their patches to be built * there is little support for automatic provisioning of builders: this means we can't scale to meet demand and make poor use of our computational resources * periodic builds (e.g. nightlies) are not supported. * it lacks a sensible interface for integration with external tools: this means that efforts like CI-before-merge have been pushed off * status reporting is poor: even answering the question of what a given builder is currently working on is surprisingly difficult * common maintenance tasks are byzantine: Adding a new builder requires adding at least six different objects (a Harbormaster Build Plan, a Drydock Working Copy, a Drydock Blueprint, an Almanac Service, an Almanac Device, and an Almanac Binding) in various Phabricator applications. None of this configuration can be tracked under version control nor can most of it be cloned from an existing builder's configuration. * Design assumptions don't match GHC's constraints: Harbormaster was designed under the assumption that builds are cheap and computation plentiful. The maintainers have stated that they have little interest in supporting environments where this doesn't hold. Jenkins, while far from perfect, seems a bit better suited to our needs, more mature, and far more flexible. This ticket will serve as the checklist for our move to Jenkins. In the end we want, * Builders (static or dynamically-provisioned, as appropriate) for * x86-64, i386 Linux * x86-64, i386 Windows * x86-64 Darwin * x86-64 OpenBSD * Cross compile from x86-64 to ARM * Native ARM * Differential builds with sensible scheduling (e.g. first build on 64-bit Linux where machines are cheap, then build on the others) * Per-commit builds on all * Nightly builds on all, including * Collection of binary distribution for user download * Update of `master` documentation mirror on downloads.haskell.org * Slow validation (including tests requiring Hackage packages) * nofib run on some platforms? * Test-before-merge-to-master -- -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13716#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13716: Move CI to Jenkins -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: task | Status: new Priority: normal | Milestone: Component: None | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: 13897 | Blocking: Related Tickets: #11958 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Description changed by simonpj: Old description:
Currently we use Harbormaster to build Differentials and commits. While this works, it leaves much to be desired,
* job scheduling is (literally) random: this means that some patch authors end up waiting literally ways for their patches to be built * there is little support for automatic provisioning of builders: this means we can't scale to meet demand and make poor use of our computational resources * periodic builds (e.g. nightlies) are not supported. * it lacks a sensible interface for integration with external tools: this means that efforts like CI-before-merge have been pushed off * status reporting is poor: even answering the question of what a given builder is currently working on is surprisingly difficult * common maintenance tasks are byzantine: Adding a new builder requires adding at least six different objects (a Harbormaster Build Plan, a Drydock Working Copy, a Drydock Blueprint, an Almanac Service, an Almanac Device, and an Almanac Binding) in various Phabricator applications. None of this configuration can be tracked under version control nor can most of it be cloned from an existing builder's configuration. * Design assumptions don't match GHC's constraints: Harbormaster was designed under the assumption that builds are cheap and computation plentiful. The maintainers have stated that they have little interest in supporting environments where this doesn't hold.
Jenkins, while far from perfect, seems a bit better suited to our needs, more mature, and far more flexible. This ticket will serve as the checklist for our move to Jenkins.
In the end we want,
* Builders (static or dynamically-provisioned, as appropriate) for * x86-64, i386 Linux * x86-64, i386 Windows * x86-64 Darwin * x86-64 OpenBSD * Cross compile from x86-64 to ARM * Native ARM
* Differential builds with sensible scheduling (e.g. first build on 64-bit Linux where machines are cheap, then build on the others) * Per-commit builds on all * Nightly builds on all, including * Collection of binary distribution for user download * Update of `master` documentation mirror on downloads.haskell.org * Slow validation (including tests requiring Hackage packages) * nofib run on some platforms? * Test-before-merge-to-master
New description: Currently we use Harbormaster to build Differentials and commits. While this works, it leaves much to be desired: * job scheduling is (literally) random: this means that some patch authors end up waiting literally ways for their patches to be built * there is little support for automatic provisioning of builders: this means we can't scale to meet demand and make poor use of our computational resources * periodic builds (e.g. nightlies) are not supported. * it lacks a sensible interface for integration with external tools: this means that efforts like CI-before-merge have been pushed off * status reporting is poor: even answering the question of what a given builder is currently working on is surprisingly difficult * common maintenance tasks are byzantine: Adding a new builder requires adding at least six different objects (a Harbormaster Build Plan, a Drydock Working Copy, a Drydock Blueprint, an Almanac Service, an Almanac Device, and an Almanac Binding) in various Phabricator applications. None of this configuration can be tracked under version control nor can most of it be cloned from an existing builder's configuration. * Design assumptions don't match GHC's constraints: Harbormaster was designed under the assumption that builds are cheap and computation plentiful. The maintainers have stated that they have little interest in supporting environments where this doesn't hold. Jenkins, while far from perfect, seems a bit better suited to our needs, more mature, and far more flexible. This ticket will serve as the checklist for our move to Jenkins. See [https://ghc.haskell.org/trac/ghc/blog/jenkins-ci Ben's blog post (Aug 17)]. In the end we want, * Builders (static or dynamically-provisioned, as appropriate) for * x86-64, i386 Linux * x86-64, i386 Windows * x86-64 Darwin * x86-64 OpenBSD * Cross compile from x86-64 to ARM * Native ARM * Differential builds with sensible scheduling (e.g. first build on 64-bit Linux where machines are cheap, then build on the others) * Per-commit builds on all * Nightly builds on all, including * Collection of binary distribution for user download * Update of `master` documentation mirror on downloads.haskell.org * Slow validation (including tests requiring Hackage packages) * nofib run on some platforms? * Test-before-merge-to-master -- -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13716#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13716: Move CI to Jenkins -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: task | Status: new Priority: normal | Milestone: Component: None | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: 13897 | Blocking: Related Tickets: #11958 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by refold): Is there any chance Cabal/cabal-install could piggyback on this effort? We've outgrown free services like Travis and AppVeyor and could use support for additional platforms (OpenBSD and maybe ARM) too. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13716#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13716: Move CI to Jenkins -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: task | Status: new Priority: normal | Milestone: Component: None | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: 13897 | Blocking: Related Tickets: #11958, #13205 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * related: #11958 => #11958, #13205 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13716#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13716: Move CI to Jenkins -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: task | Status: new Priority: normal | Milestone: Component: None | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: 13897, 14291 | Blocking: Related Tickets: #11958, #13205 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * blockedby: 13897 => 13897, 14291 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13716#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13716: Move CI to Jenkins -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: task | Status: closed Priority: normal | Milestone: Component: None | Version: 8.0.1 Resolution: wontfix | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: 13897, 14291 | Blocking: Related Tickets: #11958, #13205 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * status: new => closed * resolution: => wontfix Comment: Well, it's more-or-less official at this point. GHC will be moving not to Jenkins but rather to CircleCI and Appveyor. See the rather lengthy thread on the [[https://mail.haskell.org/pipermail/ghc-devops- group/2017-October/000037.html|ghc-devops]] list for details. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13716#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13716: Move CI to Jenkins -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: task | Status: closed Priority: normal | Milestone: Component: Continuous | Version: 8.0.1 Integration | Resolution: wontfix | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: 13897, 14291 | Blocking: Related Tickets: #11958, #13205 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * component: None => Continuous Integration -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13716#comment:10 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
participants (1)
-
GHC