Hello again, Thanks to everyone who pointed out spurious failures over the last few weeks. Here's the current state of affairs and some discussion on next steps. * * *Dashboard *** I made a dashboard for tracking spurious failures: https://grafana.gitlab.haskell.org/d/167r9v6nk/ci-spurious-failures?orgId=2 I created this for three reasons: 1. Keep tabs on new occurrences of spurious failures 2. Understand which problems are causing the most issues 3. Measure the effectiveness of any intervention The dashboard still needs development, but it can already be used to show that the number of "Cannot connect to Docker daemon" failures has been reduced. * * *Characterizing and Fixing Failures* I have preliminary results on a few failure types. For instance, I used the "docker" type of failure to bootstrap the dashboard. Along with "Killed with signal 9", it seems to indicate a problem with the CI runner, itself. To look more deeply into these types of runner-system failures, *I will need more access*. If you are responsible for some runners and you're comfortable giving me shell access, you can find my public ssh key at https://gitlab.haskell.org/-/snippets/5546. (Posted as a snippet so at least you know the key comes from somebody who can access my GitLab account. Other secure means of communication are listed at https://keybase.io/chreekat.) Send me a message if you do so. Besides runner problems, there are spurious failures that may have more to do with the CI code, itself. They include some problem with environment variables and (probably) some issue with console buffering. Neither of these are being tracked on the dashboard yet. Many other problems are yet to be explored at all. *Next Steps* The theme for the next steps is finalizing the dashboard and characterizing more failures. * Track more failure types on the dashboard * Improve the process of backfilling failure data on the dashboard * Include more metadata (like project id!) on the dashboard so it's easier to zoom on failures * Document the dashboard and the processes that populate it for posterity * Diagnose runner-system failures (if accessible) * Continue exploring other failure types * Fix failures omg!? The list of next steps is currently heavy on finalizing the dashboard and light on fixing spurious failures. I know that might be frustrating. My justification is that CI is a complex hardware/software/human system under continuous operation where most the low-hanging fruit have already been plucked. It's time to get serious. :) My goal is to make spurious failures surprising rather than commonplace. This is the best way I know to achieve that. Thanks again for helping me with this goal. :) -Bryan P.S. If you're interested, I've been posting updates like this one on Discourse: https://discourse.haskell.org/search?q=DevOps%20Weekly%20Log%20%23haskell-fo... On 18/05/2022 13:25, Bryan wrote:
Hi all,
I'd like to get some data on weird CI failures. Before clicking "retry" on a spurious failure, please paste the url for your job into the spreadsheet you'll find linked at https://gitlab.haskell.org/ghc/ghc/-/issues/21591.
Sorry for the slight misdirection. I wanted the spreadsheet to be world-writable, which means I don't want its url floating around in too many places. Maybe you can bookmark it if CI is causing you too much trouble. :)
-Bryan