I have some suggestions for low hanging fruits in this effort. 1. Make ghc print more statistics on what it spending time on When I did the linking investigation recently (https://www.reddit.com/r/haskell/comments/63y43y/liked_linking_3x_faster_wit...) I noticed (with strace) that there are lots of interesting syscalls being made that you might not expect. For example, each time TH is used, shared libraries are loaded, and to determine the shared library paths, ghc shells out to `gcc --print-file-name`. Each such invocation takes 20 ms on my system, and I have 1000 invocations in my build. That's 20 seconds (out of 2 minutes build time) just asking gcc for paths. I recommend that for every call to an external GHC measures how long that call took, so that it can be asked to print a summary when it's done. That might give us lots of interesting things to optimize. For example, This would have made the long linker times totally obvious. At the end, I would love to know for each compilation (both one-shot as used in ghc's build system, and `ghc --make`): * What programs did it invoke and how long did they take * What files did it read and how long did that take * How long did it take to read all the `.hi` files in `ghc --make` * High level time summary (parsing, typechecking, codegen, .hi files, etc) That way we'll know at least what is slow, and don't have to resort to strace every time in order to obtain this basic answer. 2. Investigate if idiotic syscalls are being done and how much There's this concept I call "idiotic syscalls", which are syscalls of which you know from before that they won't contribute anything productive. For example, if you give a linker N many `-L` flags (library dirs) and M many `-l` flags (library names to link), it will try to `stat()` or `open()` N*M many files, out of which most are total rubbish, because we typically know what library is in what dir. Example: You pass `-L/usr/lib/opencv -L/usr/lib/imagemagick -L/usr/lib/blas -lopencv -limagemagick -lblas`. Then you you will get things like `open("/usr/lib/opencv/libimagemagick.so") = ENOENT` which makes no sense and obviously that file doesn't exist. This is a problem with the general "search path" concept; same happens for running executables searching through $PATH. Yes, nonexistent file opens fail fast, but in my typical ghc invocation I get millions of them (and we should at least measure how much time is wasted on them), and they clutter the strace output and make the real problems harder to investigate. We should check if we can create ways to give pass those files that do exist. 3. Add pure TemplateHaskell It is well known that TH is a problem for incremental compilation because it can have side effects and we must therefore be more conservative about when to recompile; when you see a `[TH]` in your `ghc --make` output, it's likely that time again. I believe this could be avoided by adding a variant of TH that forbids the use of the `runIO` function, and can thus not have side effects. Most TH does not need side effects, for example any form of code generation based on other data types (lenses, instances for whatever). If that was made "pure TH", we would not have to recompile when inputs to our TH functions change. Potentially this could even be determined automatically instead of adding a new variant of TH like was done for typed TH `$$()`, simply by inspecting what's in the TH and if we can decide there's no `runIO` in there, mark it as clean, otherwise as tainted. 4. Build ghc with `ghc --make` if possible This one might be controversial or impossible (others can likely tell us). Most Haskell code is built with `ghc --make`, not with the one-shot compilation system + make or Hadrian as as done in GHC's build system. Weirdly, often `ghc --make` scales much worse and has much worse incremental recompilation times than the one-shot mode, which doesn't make sense given that it has no process creation overhead, can do much better caching etc. I believe that if ghc or large parts of it (e.g. stage2) itself was built with `--make`, we would magically see --make become very good, simply we make the right people (GHC devs) suffer through it daily :D. I expect from this the solution of the `-j` slowness, GHC overhead reduction, faster interface file loads and so on. These are some ideas. Niklas