#11501: Building nofib/fibon returns permission denied -------------------------------------+------------------------------------- Reporter: rem | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: NoFib benchmark | Version: 7.10.3 suite | Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 | (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by thomie): Thanks for reporting. Maybe you want to help fix it? Once `fibon` is at least building again, we can make it part of validate. See #9572. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11501#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11501: Building nofib/fibon returns permission denied -------------------------------------+------------------------------------- Reporter: rem | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: NoFib benchmark | Version: 7.10.3 suite | Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 | (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by thomie): A rewrite of the build system is happening here: https://github.com/snowleopard/shaking-up-ghc. I don't know what the plans for nofib are. Some code was written 4 years ago, see #5793. If you want, you can also pick this project up. The bug in this ticket is probably unrelated to `make` vs `shake`, but I might be wrong. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11501#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11501: Building nofib/fibon returns permission denied -------------------------------------+------------------------------------- Reporter: rem | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: NoFib benchmark | Version: 7.10.3 suite | Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 | (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by rem): How does one take on a ticket, or has to be assigned? Haven't done this before. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11501#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11501: Building nofib/fibon returns permission denied -------------------------------------+------------------------------------- Reporter: rem | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: NoFib benchmark | Version: 7.10.3 suite | Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 | (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by rem): * owner: rem => -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11501#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11501: Building nofib/fibon returns permission denied -------------------------------------+------------------------------------- Reporter: rem | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: NoFib benchmark | Version: 7.10.3 suite | Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 | (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by simonpj): * cc: gracjanpolak@… (added) Comment: Gracjan says (email to ghc-devs): I went into the rabbit hole that starts here So far I know that: 1. fibon is not built regularly by any CI and it bitrotted significantly. 2. Makefile's for fibon use undefined variables like INPLACE_HSC2HS_PGM. Probably those were sourced from ghc/mk/*.mk, but not now. 3. Haskell courses are pre-AMP (this is easiest to fix). 4. Makefile's build object files one by one and that fails in case modules depend on each other. ghc-paths.mk sorts objects and that gives wrong compile order. Does anybody know a commit-hash or date when it did compile last time? Knowing that this module (fibon) wasn't used for so long does it make sense to fix it? How about removing it? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11501#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11501: Building nofib/fibon returns permission denied -------------------------------------+------------------------------------- Reporter: rem | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: NoFib benchmark | Version: 7.10.3 suite | Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 | (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by simonpj): * cc: gracjan (removed) * cc: gracjanpolak@… (added) Comment: I have no idea when it last worked. Below is a log of the most commits.. mostly back in 2014. The `nofib/fibon/shootout` programs are (I think) also in `nofib/shootout`. Worth checking. The `nofib/fibon/Repa` and `Hackage` programs look quite useful. `nofib` is old, and we could really do with more. So the ones in there might be jolly useful. Let's resurrect it not kill it! THANK YOU! Simon {{{ git log -8 fibon commit 35fc121fc8cc501ea2713c579a053be7ea65b16e Author: Seraphime Kirkovski Date: Sun May 22 17:47:26 2016 +0200 Fix: #12084 deprecate old profiling flags commit f9a577973edf6976e4c0703bf2afac900d7b6fd8 Author: Gabor Greif Date: Fri Jul 31 00:43:38 2015 +0200 Typos in comments commit c9c20d477088a8a7d5747f16afdf0652fba6dadf Author: Joachim Breitner Date: Tue Sep 2 17:53:52 2014 +0200 Hide Word from Prelude in benchmarks where Word is used. This fixes fall-out from #9531. commit 77a87f18166aa4c806e33b0adf4c2fbe40551a4f Author: Gabor Greif Date: Fri Jan 10 12:02:06 2014 +0100 fix some typos commit b9d8bcdd929d664539843f96e56b50800f623891 Author: Krzysztof Gogolewski Date: Tue Sep 3 21:23:09 2013 +0200 Remove deprecated _scc_ (#8170) commit eb9f4a3ff7f3b24896534011717099c9d61e146d Author: Johan Tibell Date: Tue Feb 5 15:50:23 2013 -0800 Removed some shootout benchmarks from under fibon Newer versions (corresponding to what's currently on the shootout page) have been added under nofib/shootout in an attempt to make these easier to run. }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11501#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11501: Building nofib/fibon returns permission denied -------------------------------------+------------------------------------- Reporter: rem | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: NoFib benchmark | Version: 7.10.3 suite | Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 | (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): As far as `fibon` is concerned, I get the impression that it may be easier and perhaps more useful to start with a clean slate or concentrate on `nofib` itself. The `fibon` [[https://hackage.haskell.org/package/fibon|tools]] themselves haven't been touched in nearly half a decade and it looks like the tests are similarly on life support. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11501#comment:13 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11501: Building nofib/fibon returns permission denied -------------------------------------+------------------------------------- Reporter: rem | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: NoFib benchmark | Version: 7.10.3 suite | Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 | (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by gracjan): After some research I have some insights to share. Looks like nofib is used for two distinct purposes: 1. To test speed of generated code. 2. To test compilation speed. `nofib` setup is geared towards interactive use, that is to be able to run speed experiments and compare them pair-wise. Running `make` runs both compilation speed and execution speed tests at the same time. There is also a way to run `valgrind`s `cachegrind` at the same time. Major problems with the test suite I see now are: 1. Only a handful of speed tests are enabled, everything else bitrots. 2. Build system compiles objects and links them, does not use `--make`. 3. Build system is ill suited for tests extracted from cabal projects. 4. There is a confusion between compilation speed tests and execution speed tests. The way forward as I see: 1. Enable all tests by default, but make it easy to select a subset for interactive work. 2. Use `ghc --make` for builds of simpler cases (no dependencies besiades what is shipped with ghc). 3. Use `cabal install` for more complicated cases (with more dependencies). 4. Clearly separate compilation speed tests from execution speed tests. And in distant future: 1. Run execution speed tests on TravisCI in multiple combinations of os/ghc/flags/test. 2. Run compilation speed tests on TravisCI in multiple combinations of os/ghc/flags/test. 3. Run incremental compilation speed tests on TravisCI in multiple combinations of os/ghc/flags/test. 4. Publish metrics to some kind of timeseries database. 5. Let Grafana show some nice graphs. Graphana is here: http://grafana.org/ I'd like to hear from people that use nofib in daily work how do they do this so that we do not skip any important use cases. Can you guys chime in and describe your workflow? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11501#comment:15 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11501: Building nofib/fibon returns permission denied -------------------------------------+------------------------------------- Reporter: rem | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: NoFib benchmark | Version: 7.10.3 suite | Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 | (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): I have also used `nofib` a great deal to locate past regressions in compiler performance regressions. Your observations are generally correct. I think it would be great to enable more tests by default. However, it's not at all clear to me that we necessarily always want to use `--make`. The current use of one-shot mode makes it significantly easier to narrow down what code in particular GHC has regressed on. Moving to `--make` may better reflect `Cabal`'s usage, but I think it would make the common case of locating compiler performance regressions a bit harder. In general I think there is certainly room for a broader performance testsuite consisting of a sampling of packages from Hackage. However, I'm not sure that `nofib` is that place. Keeping `nofib` a set of standalone tests without dependencies makes it a significantly smaller maintenance burden that the Cabal-centric approach that you describe. We as a community (myself included!) have had a poor record of following through with maintaining our performance infrastructure. Consequently I think there is value in keeping at least one testsuite which "just works" with minimal maintenance. I gave a talk at HIW this year describing a very rough tool that I developed while tracking down performance regressions in nofib using the performance build bot which nomeata stood up to server [[http://perf.haskell.org/|gipeda]]. It essentially provides the nice(?) graphs which you describe. Sadly, I've not had a chance to put it up in a publicly accessible location. In the meantime it can be found at http://home.smart-cactus.org/~ben/ghc-perf/. The source is available [[https://github.com/bgamari/ghc-perf-import/|here]]. The general workflow is, 1. navigate to http://home.smart-cactus.org/~ben/ghc-perf/ 2. stand in awe of my poor front-end design aesthetic 3. select a test environment on the right-hand pane (e.g. my own build bot or nomeata's; the latter has better coverage). 4. select a set of tests to plot from the left-hand pane. The tests list can be filtered by substring match using the text field at the top; I usually filter by `compile-allocs`. Loading the results may take a while, sadly. 5. select a change threshold percentage such that commits which exhibit large changes in the selected tests show up in the commit list at the bottom This is how I typically find a starting point to dive into performance work. There are several types of tests captured, * Characterizing GHC's performance * `compile-allocs`: These are the compiler allocations for compiling each module of each test * `compile-time`: These are the compiler runtimes for compiling each module of each test * Compiled code characteristics * `binary-size`: This is the size of the produced executable * `module-size`: The size of each module's object file * Characterizing compiled code performance * `allocs`: This is the runtime allocations of each test * `gc-time`: The time spent in garbage collection while running the test * `elapsed-time`: The wall-clock time of the test run * `mut-elapsed-time`: The wall-clock time spent in the mutator * `mut-time`: CPU time spent in the mutator * `run-time`: The overall CPU time spent in the test Note that in general runtimes are very unstable. I find it most helpful to first identify potential regressions by looking at allocations, then look at runtime to see which allocations changes actually move real runtime. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11501#comment:17 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11501: Building nofib/fibon returns permission denied -------------------------------------+------------------------------------- Reporter: rem | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: NoFib benchmark | Version: 7.10.3 suite | Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 | (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by gracjan): @simonpj: Do you also use nofib to test speed of compilation or just efficiency of generated code? @bgamari: `ghc --make` will only be used to test generated code. Compilation speed will have own tests that compile single files. Unless we will want to test compilation speed for many modules at once. We can also go full high level and see if compiling programs with latest complier and latest package set got suddenly slower than previous compiler and previous package set. This is very high level, but this is what constitutes the perception 'GHC compiles slower than it used to'.

However, I'm not sure that nofib is that place.

This is what `fibon` is and I'm not sure either if this a good place. I like gipeda very much, this is more or less what I had in mind. I would advise using off-the-shelf tools for this purpose, as it is a lot of work to create one from scratch. There is a relevant industry that is very concerned with performance and performance variations in time. They have developed a lot of tools to collect performance data points and then be able to dig into the issues by finding exceptional value, high variation, correlating causes and effects and so on. This industry is network monitoring and operations in general. And tools I like are Graphana and a backend database for data points, could be ElasticSearch or InfluxDB. Those more or less form gipeda, well, rather more than gipeda :) Anyway I was thinking of a datapoint that will be a tuple: (operating system, machine arch, ghc version, test name, date of last merged commit, and METRIC) where METRIC is one of metrics you listed. Can you share database of results? I'd like to try to import data into InfluxDB, connect Graphana, see if this is useful. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11501#comment:19 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11501: Building nofib/fibon returns permission denied -------------------------------------+------------------------------------- Reporter: rem | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: NoFib benchmark | Version: 7.10.3 suite | Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 | (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonpj):

@simonpj: Do you also use nofib to test speed of compilation or just efficiency of generated code?

Primarily the latter; but the former might be good too, esp if it was a new column in the table at the top of `nofib-analyse`'s output. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11501#comment:21 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11501: Building nofib/fibon returns permission denied -------------------------------------+------------------------------------- Reporter: rem | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: NoFib benchmark | Version: 7.10.3 suite | Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 | (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by gracjan): @bgamari: Your database contains many duplicate results: {{{ ghc_perf=# select * from results where commit_id = 1481 and test_id = 14005; result_id | commit_id | test_env_id | test_id | result_value -----------+-----------+-------------+---------+-------------- 2666081 | 1481 | 1 | 14005 | 68981392 16414734 | 1481 | 1 | 14005 | 68981392 27735337 | 1481 | 1 | 14005 | 68981392 29330138 | 1481 | 1 | 14005 | 68981392 29950528 | 1481 | 1 | 14005 | 68981392 30987336 | 1481 | 1 | 14005 | 68981392 32588450 | 1481 | 1 | 14005 | 68981392 35533576 | 1481 | 1 | 14005 | 68981392 37080771 | 1481 | 1 | 14005 | 68981392 38801911 | 1481 | 1 | 14005 | 68981392 38869658 | 1481 | 1 | 14005 | 68981392 38871489 | 1481 | 1 | 14005 | 68981392 38873320 | 1481 | 1 | 14005 | 68981392 (13 rows) }}} I'm not sure why is this. Have you been testing same (commit,env,test) configuration multiple times? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11501#comment:23 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11501: Building nofib/fibon returns permission denied -------------------------------------+------------------------------------- Reporter: rem | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: NoFib benchmark | Version: 7.10.3 suite | Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 | (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by gracjan): If you cared to deduplicate: {{{ begin; delete from results where exists (select true from results s2 where s2.result_id < results.result_id and s2.commit_id = results.commit_id and s2.test_env_id = results.test_env_id and s2.test_id = results.test_id and s2.result_value = results.result_value); -- DELETE 24126761 alter table results add constraint results_unique unique (test_env_id,commit_id,test_id); commit; }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11501#comment:25 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11501: Building nofib/fibon returns permission denied -------------------------------------+------------------------------------- Reporter: rem | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: NoFib benchmark | Version: 7.10.3 suite | Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 | (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by gracjan): * Attachment "grafana-ghc-perf.png" added. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11501 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11501: Building nofib/fibon returns permission denied -------------------------------------+------------------------------------- Reporter: rem | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: NoFib benchmark | Version: 7.10.3 suite | Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 | (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by gracjan): * Attachment "get_all_as_line_protocol.sql" added. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11501 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11501: Building nofib/fibon returns permission denied -------------------------------------+------------------------------------- Reporter: rem | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: NoFib benchmark | Version: 7.10.3 suite | Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 | (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by Michal Terepeta ): In [changeset:"50812b141f1a6c372c85cceb3541e9c1c7bab926/nofib" 50812b1/nofib]: {{{ #!CommitTicketReference repository="nofib" revision="50812b141f1a6c372c85cceb3541e9c1c7bab926" Remove fibon Summary: It doesn't compile, nobody is running it and upstream seems to have abandoned it too. IMHO, at this point we should simply remove it and focus on improving the benchmarks that do work & adding new ones. See also the discussion in #11501 Signed-off-by: Michal Terepeta Test Plan: compile & run nofib Reviewers: bgamari Subscribers: }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11501#comment:29 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11501: Building nofib/fibon returns permission denied -------------------------------------+------------------------------------- Reporter: rem | Owner: (none) Type: bug | Status: closed Priority: normal | Milestone: 8.2.1 Component: NoFib benchmark | Version: 7.10.3 suite | Resolution: fixed | Keywords: Operating System: Linux | Architecture: x86_64 | (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * status: new => closed * resolution: => fixed * milestone: => 8.2.1 Comment: Let's close is and open a new ticket for `nofib` improvements. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11501#comment:31 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler