#14697: Redundant computation in fingerprintDynFlags when compiling many modules -------------------------------------+------------------------------------- Reporter: niteria | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by niteria): * cc: simonmar (added) Old description:

I profiled a build of a production code base with thousands of modules and computing `fingerprintDynFlags` is `7%` of time and `14%` of allocations.

Here's a synthetic test case inspired by what I observed: {{{ SIZE=1000

for i in $(seq -w 1 $SIZE); do echo "module A$i where" > A$i.hs echo "data A$i = A$i" >> A$i.hs done }}}

This generates a 1000 modules each with one datatype. Compiling them with: {{{ inplace/bin/ghc-stage2 A*.hs -optP-D__F{1..10000}__ }}} results in `fingerprintDynFlags` being the top cost centre in the profile. AFAICT there's only one module dependent piece that goes into computing `fingerprintDynFlags` and the rest is the same between those 1000 modules.

Now, why would I have so many preprocessor flags? This is how the Buck build system currently works. If a Haskell library depends on a C++ library then the GHC invocation gets the C++ library's directory as include path (`-optP-I -optP-I some/library/path`). This can grow quite big.

New description: I profiled a build of a production code base with thousands of modules and computing `fingerprintDynFlags` is `7%` of time and `14%` of allocations. Here's a synthetic test case inspired by what I observed: {{{ SIZE=1000 for i in $(seq -w 1 $SIZE); do echo "module A$i where" > A$i.hs echo "data A$i = A$i" >> A$i.hs done }}} This generates a 1000 modules each with one datatype. Compiling them with: {{{ inplace/bin/ghc-stage2 A*.hs -optP-D__F{1..10000}__ }}} results in `fingerprintDynFlags` being the top cost centre in the profile. AFAICT there's only one module dependent piece that goes into computing `fingerprintDynFlags` and the rest is the same between those 1000 modules. Now, why would I have so many preprocessor flags? This is how the Buck build system currently works. If a Haskell library depends on a C++ library then the GHC invocation gets the C++ library's directory as include path (`-optP -I -optP some/library/path`). This can grow quite big. -- -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14697#comment:1> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

#14697: Redundant computation in fingerprintDynFlags when compiling many modules -------------------------------------+------------------------------------- Reporter: niteria | Owner: (none) Type: bug | Status: patch Priority: normal | Milestone: Component: Compiler | Version: Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D4445 Wiki Page: | -------------------------------------+------------------------------------- Comment (by Bartosz Nitka <niteria@…>): In [changeset:"b8f03bbe16af7a09b494a33fbbe523ecd82f1a50/ghc" b8f03bbe/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="b8f03bbe16af7a09b494a33fbbe523ecd82f1a50" Cache the fingerprint of sOpt_P Before this change we would compute a hash of all the command line -optP flags once per file. With a lot of files and many -optP flags, that's a lot of repeated work. I added a new Note that explains the approach and rationale. Test Plan: new test Reviewers: simonmar, simonpj, bgamari Reviewed By: simonpj Subscribers: rwbarton, thomie, carter GHC Trac Issues: #14697 Differential Revision: https://phabricator.haskell.org/D4445 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14697#comment:3> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

#14697: Redundant computation in fingerprintDynFlags when compiling many modules -------------------------------------+------------------------------------- Reporter: niteria | Owner: (none) Type: bug | Status: closed Priority: normal | Milestone: Component: Compiler | Version: Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): phab:D4445 Wiki Page: | -------------------------------------+------------------------------------- Comment (by Ben Gamari <ben@…>): In [changeset:"af986f9dfb49db5544e4d670cde9a9a70a9a1223/ghc" af986f9d/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="af986f9dfb49db5544e4d670cde9a9a70a9a1223" testsuite: Disable T14697 on Windows Test Plan: Validate on Windows Subscribers: thomie, carter GHC Trac Issues: #14697, #15072 Differential Revision: https://phabricator.haskell.org/D4619 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14697#comment:5> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler