To have a single-process ghc --make -j you first of all need internal thread-safety: GHC internally keeps a number of global caches that need to be made thread-safe: - table of interned strings (this is actually written in C and accessed via FFI) - cache of interface files loaded, these are actually loaded lazily using unsafeInterleaveIO magic (yuck) - cache of packages descriptions (I think) - the NameCache: a cache of string -> magic number. This is used to implement fast comparisons between symbols. The magic numbers are generated non-deterministically (more unsafeInterleaveIO) so you need to keep this cache around. - HomeModules: These are the modules that have been compiled in this --make run. The NameCache is used when loading interface files and also by the Parser. Making these things thread-safe basically involves updating these caches via atomicModifyIORef instead of just modifyIORef. I made those changes a few years ago, but at least one of them was rolled-back. I forgot the details, but I think it was one use of unsafePerformIO that caused the issues. unsafePerformIO needs to traverse the stack to look for thunks that are potentially evaluated by multiple threads. If you have a deep stack that can be expensive. SimonM since added stack chunks which should reduce the overhead of this. Could be worthwhile re-evaluating the patch. To have a multi-process ghc --make you don't need thread-safety. However, without sharing the caches -- in particular the interface file caches -- the time to read data from the disk may outweigh any advantages from parallel execution. Evan's approach of using a long-running worker process avoids issues with reloading the most of the caches for each module, but it probably couldn't take advantage of the HomeModule cache. It would be interesting to see if that was the issues. Then, it would be interesting if the disk access or the serialisation overhead is the issue; if it's the former, some clever use of mmap could help. HTH, / Thomas On 13 May 2013 17:35, Evan Laforge wrote:

I wrote a ghc-server that starts a persistent process for each cpu. Then a 'ghc' frontend wrapper sticks each job in a queue. It seemed to be working, but timing tests didn't reveal any speed-up. Then I got a faster computer and lost motivation. I didn't investigate very deeply why it didn't speed up as I hoped. It's possible the approach is still valid, but I made some mistake in the implementation.

So I can stop writing this little blurb I put it on github:

https://github.com/elaforge/ghc-server

On Mon, May 13, 2013 at 8:40 PM, Niklas Hambüchen wrote:

...

I know this has been talked about before and also a bit in the recent GSoC discussion.

I would like to know what prevents ghc --make from working in parallel, who worked at that in the past, what their findings were and a general estimation of the difficulty of the problem.

Afterwards, I would update http://hackage.haskell.org/trac/ghc/ticket/910 with a short summary of what the current situation is.

Thanks to those who know more!

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

It's also useful to note that the disk cache might do a surprisingly good job at caching those .hi files for you. That, and a lot of people (like me!) use SSDs, where the parallel compilation takes the vast majority of time. I'd be really excited to see parallel ghc --make. By the way, totally unrelated, but why does cabal support -j when cabal-dev doesn't? - Clark On Wednesday, May 15, 2013, Niklas Hambüchen wrote:

Hello Thomas,

thanks for your detailed answer.

...

Could be worthwhile re-evaluating the patch.

Does your patch still apply somewhat cleanly? And does it address all the caches in your list already or only some subset of them?

...

To have a multi-process ghc --make you don't need thread-safety. However, without sharing the caches -- in particular the interface file caches -- the time to read data from the disk may outweigh any advantages from parallel execution.

That might be a big step already - I've never seen a project where I'd care about parallel compilation that is not totally CPU-bound.

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org javascript:; http://www.haskell.org/mailman/listinfo/haskell-cafe

Oh! That works quite nicely. It's not supported for install-deps. I assumed that it just wasn't implemented. I should open a ticket. Thanks! - Clark On Wednesday, May 15, 2013, Clark Gaebel wrote:

It's also useful to note that the disk cache might do a surprisingly good job at caching those .hi files for you. That, and a lot of people (like me!) use SSDs, where the parallel compilation takes the vast majority of time.

I'd be really excited to see parallel ghc --make.

By the way, totally unrelated, but why does cabal support -j when cabal-dev doesn't?

- Clark

On Wednesday, May 15, 2013, Niklas Hambüchen wrote:

...

Hello Thomas,

thanks for your detailed answer.

...

Could be worthwhile re-evaluating the patch.

Does your patch still apply somewhat cleanly? And does it address all the caches in your list already or only some subset of them?

...

To have a multi-process ghc --make you don't need thread-safety. However, without sharing the caches -- in particular the interface file caches -- the time to read data from the disk may outweigh any advantages from parallel execution.

That might be a big step already - I've never seen a project where I'd care about parallel compilation that is not totally CPU-bound.

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe