#10728: Add e.g. "-N<=4" in addition to the fixed "-N4" and variable "-N" RTS options -------------------------------------+------------------------------------- Reporter: rrnewton | Owner: simonmar Type: feature request | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Description changed by rrnewton: Old description:

As discussed in this issue:

https://github.com/commercialhaskell/stack/issues/680

stack is one example of a program that optimistically turned on "-N". Indeed, it feels like a reasonable and even safe option for a multithreaded program. Unfortunately, -N currently guarantees bad performance on large machines and especially on large machines with hyperthreading. Pretty much one should ship an executable with -N by default as of GHC 7.8 and 7.10 IMHO.

Unfortunately, even if stack did get a speedup at, say, 4 or 8 cores, it would not be good to ship it with "-N8" either. This would be an unreasonable choice on small, one or two core machines.

What we need is a way to say that the program can productively use parallelism up to a certain upper bound, but that fewer threads should be used if there are not enough cores available. I propose "-N<=8" as a potential syntax.

Currently, this behavior can be achieved with `setNumCapabilities`/`getNumProcessors`, but I think it's worth a command line RTS option.

New description: As discussed in this issue: https://github.com/commercialhaskell/stack/issues/680 stack is one example of a program that optimistically turned on "-N". Indeed, it feels like a reasonable and even safe option for a multithreaded program. Unfortunately, -N currently guarantees bad performance on large machines and especially on large machines with hyperthreading. No one should ship an executable with -N by default as of GHC 7.8 and 7.10 IMHO. Unfortunately, even if stack did get a speedup at, say, 4 or 8 cores, it would not be good to ship it with "-N8" either. This would be an unreasonable choice on small, one or two core machines. What we need is a way to say that the program can productively use parallelism up to a certain upper bound, but that fewer threads should be used if there are not enough cores available. I propose "-N<=8" as a potential syntax. Currently, this behavior can be achieved with `setNumCapabilities`/`getNumProcessors`, but I think it's worth a command line RTS option. -- -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10728#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10728: Add e.g. "-N<=4" in addition to the fixed "-N4" and variable "-N" RTS options -------------------------------------+------------------------------------- Reporter: rrnewton | Owner: simonmar Type: feature request | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #9221, #8224 | Differential Revisions: -------------------------------------+------------------------------------- Changes (by thomie): * related: => #9221, #8224 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10728#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10728: Add e.g. "-N<=4" in addition to the fixed "-N4" and variable "-N" RTS options -------------------------------------+------------------------------------- Reporter: rrnewton | Owner: simonmar Type: feature request | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 7.10.2 Resolution: | Keywords: newcomer Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #9221, #8224 | Differential Revisions: -------------------------------------+------------------------------------- Changes (by thomie): * keywords: => newcomer -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10728#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10728: Add e.g. "-N<=4" in addition to the fixed "-N4" and variable "-N" RTS options -------------------------------------+------------------------------------- Reporter: rrnewton | Owner: Type: feature request | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 7.10.2 Resolution: | Keywords: newcomer Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #9221, #8224 | Differential Revisions: -------------------------------------+------------------------------------- Changes (by thomie): * cc: simonmar (added) * owner: simonmar => -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10728#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10728: Add e.g. "-N<=4" in addition to the fixed "-N4" and variable "-N" RTS options -------------------------------------+------------------------------------- Reporter: rrnewton | Owner: MarcelineVQ Type: feature request | Status: patch Priority: normal | Milestone: Component: Runtime System | Version: 7.10.2 Resolution: | Keywords: newcomer Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #9221, #8224 | Differential Rev(s): Phab:D1650 Wiki Page: | -------------------------------------+------------------------------------- Changes (by MarcelineVQ): * status: new => patch * differential: => Phab:D1650 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10728#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10728: Add e.g. "-N<=4" in addition to the fixed "-N4" and variable "-N" RTS options -------------------------------------+------------------------------------- Reporter: rrnewton | Owner: MarcelineVQ Type: feature request | Status: patch Priority: normal | Milestone: Component: Runtime System | Version: 7.10.2 Resolution: | Keywords: newcomer Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #9221, #8224 | Differential Rev(s): Phab:D1650 Wiki Page: | -------------------------------------+------------------------------------- Comment (by thomie): Congratulations with your first GHC patch, Matthew. Still todo: * `-Ndiv<n>`: use the number of processors divided by <n> * `-Nsub<n>`: use the number of processors minus <n> * release notes * tests (use `getNumCapabilities` from `GHC.Conc.Sync`) Maybe you could take care of those as well? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10728#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10728: Add e.g. "-N<=4" in addition to the fixed "-N4" and variable "-N" RTS options -------------------------------------+------------------------------------- Reporter: rrnewton | Owner: MarcelineVQ Type: feature request | Status: patch Priority: normal | Milestone: Component: Runtime System | Version: 7.10.2 Resolution: | Keywords: newcomer Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #9221, #8224 | Differential Rev(s): Phab:D1650 Wiki Page: | -------------------------------------+------------------------------------- Comment (by hvr): First, the single-dash introducing a long-opt is confusing. But let's ignore that one for a moment. Moreover, in the context of `-Ndiv<n>` and `-Nsub<n>` I consider `-Nmax<n>` wrongly named: It should rather be named `-Nmin<n>` to avoid confusion: Intuitively, `-N<infixop><n>` denotes {{{N `infixop` <n>}}}, with the binary operators {{{#!hs min, max, add, sub, div, mul :: Int -> Int -> Int }}} and the obvious definitions, this would imply that `-Nmax4` means {{{N `max` 4}}} which is not what's implemented here. The description of the flag alone should rise red flags already: | -N'''max'''<x> chooses the '''min'''imum of processor count or x -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10728#comment:12 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10728: Add e.g. "-N<=4" in addition to the fixed "-N4" and variable "-N" RTS options -------------------------------------+------------------------------------- Reporter: rrnewton | Owner: Type: feature request | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 7.10.2 Resolution: | Keywords: newcomer Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #9221, #8224 | Differential Rev(s): Phab:D1650 Wiki Page: | -------------------------------------+------------------------------------- Comment (by hvr): Here's two less controversial variants to `-Nmax` I wouldn't object to: - `-maxN` (this follows common conventions for `max` used for upper- bounding, c.f. `find . -mindepth <n> -maxdepth <n>` - Use a single letter option, e.g. `-n <n>`; this follows current `+RTS --help` convention, c.f. {{{ -M<size> Sets the maximum heap size (default unlimited) Egs: -M256k -M1G -H<size> Sets the minimum heap size (default 0M) Egs: -H24m -H1G }}} For reference, here's the current output of `ghc +RTS --help` for GHC HEAD, where `-Nmax` clearly deviates from the existing naming conventions: {{{ ghc: unknown RTS option: --help ghc: ghc: Usage: <prog> <args> [+RTS <rtsopts> | -RTS <args>] ... --RTS <args> ghc: ghc: +RTS Indicates run time system options follow ghc: -RTS Indicates program arguments follow ghc: --RTS Indicates that ALL subsequent arguments will be given to the ghc: program (including any of these RTS flags) ghc: ghc: The following run time system options are available: ghc: ghc: -? Prints this message and exits; the program is not executed ghc: --info Print information about the RTS used by this program ghc: ghc: -K<size> Sets the maximum stack size (default: 80% of the heap) ghc: Egs: -K32k -K512k -K8M ghc: -ki<size> Sets the initial thread stack size (default 1k) Egs: -ki4k -ki2m ghc: -kc<size> Sets the stack chunk size (default 32k) ghc: -kb<size> Sets the stack chunk buffer size (default 1k) ghc: ghc: -A<size> Sets the minimum allocation area size (default 512k) Egs: -A1m -A10k ghc: -n<size> Allocation area chunk size (0 = disabled, default: 0) ghc: -O<size> Sets the minimum size of the old generation (default 1M) ghc: -M<size> Sets the maximum heap size (default unlimited) Egs: -M256k -M1G ghc: -H<size> Sets the minimum heap size (default 0M) Egs: -H24m -H1G ghc: -m<n> Minimum % of heap which must be available (default 3%) ghc: -G<n> Number of generations (default: 2) ghc: -c<n> Use in-place compaction instead of copying in the oldest generation ghc: when live data is at least <n>% of the maximum heap size set with ghc: -M (default: 30%) ghc: -c Use in-place compaction for all oldest generation collections ghc: (the default is to use copying) ghc: -w Use mark-region for the oldest generation (experimental) ghc: -I<sec> Perform full GC after <sec> idle time (default: 0.3, 0 == off) ghc: ghc: -T Collect GC statistics (useful for in-program statistics access) ghc: -t[<file>] One-line GC statistics (if <file> omitted, uses stderr) ghc: -s[<file>] Summary GC statistics (if <file> omitted, uses stderr) ghc: -S[<file>] Detailed GC statistics (if <file> omitted, uses stderr) ghc: ghc: ghc: -Z Don't squeeze out update frames on stack overflow ghc: -B Sound the bell at the start of each garbage collection ghc: ghc: -h Heap residency profile (output file <program>.hp) ghc: -i<sec> Time between heap profile samples (seconds, default: 0.1) ghc: ghc: -C<secs> Context-switch interval in seconds. ghc: 0 or no argument means switch as often as possible. ghc: Default: 0.02 sec. ghc: -V<secs> Master tick interval in seconds (0 == disable timer). ghc: This sets the resolution for -C and the heap profile timer -i, ghc: and is the frequence of time profile samples. ghc: Default: 0.01 sec. ghc: ghc: -N[<n>] Use <n> processors (default: 1, -N alone determines ghc: the number of processors to use automatically) ghc: -qg[<n>] Use parallel GC only for generations >= <n> ghc: (default: 0, -qg alone turns off parallel GC) ghc: -qb[<n>] Use load-balancing in the parallel GC only for generations >= <n> ghc: (default: 1, -qb alone turns off load-balancing) ghc: -qa Use the OS to set thread affinity (experimental) ghc: -qm Don't automatically migrate threads between CPUs ghc: -qi<n> If a processor has been idle for the last <n> GCs, do not ghc: wake it up for a non-load-balancing parallel GC. ghc: (0 disables, default: 0) ghc: --install-signal-handlers= ghc: Install signal handlers (default: yes) ghc: -e<n> Maximum number of outstanding local sparks (default: 4096) ghc: -xm Base address to mmap memory in the GHCi linker ghc: (hex; must be <80000000) ghc: -xq The allocation limit given to a thread after it receives ghc: an AllocationLimitExceeded exception. (default: 100k) ghc: ghc: RTS options may also be specified using the GHCRTS environment variable. }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10728#comment:15 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10728: Add e.g. "-N<=4" in addition to the fixed "-N4" and variable "-N" RTS options -------------------------------------+------------------------------------- Reporter: rrnewton | Owner: MarcelineVQ Type: feature request | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 7.10.2 Resolution: | Keywords: newcomer Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #9221, #8224 | Differential Rev(s): Phab:D1650, Wiki Page: | Phab:D1677 -------------------------------------+------------------------------------- Changes (by MarcelineVQ): * owner: => MarcelineVQ * differential: Phab:D1650 => Phab:D1650, Phab:D1677 Comment: New revision to check out, settled on -maxN and also discovered possible -m bug where it silently ignores bad opts that begin with m. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10728#comment:17 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10728: Add e.g. "-N<=4" in addition to the fixed "-N4" and variable "-N" RTS options -------------------------------------+------------------------------------- Reporter: rrnewton | Owner: MarcelineVQ Type: feature request | Status: closed Priority: normal | Milestone: Component: Runtime System | Version: 7.10.2 Resolution: fixed | Keywords: newcomer Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #9221, #8224 | Differential Rev(s): Phab:D1650, Wiki Page: | Phab:D1677 -------------------------------------+------------------------------------- Changes (by bgamari): * status: new => closed * resolution: => fixed -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10728#comment:19 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10728: Add e.g. "-N<=4" in addition to the fixed "-N4" and variable "-N" RTS options -------------------------------------+------------------------------------- Reporter: rrnewton | Owner: MarcelineVQ Type: feature request | Status: closed Priority: normal | Milestone: 8.0.1 Component: Runtime System | Version: 7.10.2 Resolution: fixed | Keywords: newcomer Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #9221, #8224 | Differential Rev(s): Phab:D1650, Wiki Page: | Phab:D1677 -------------------------------------+------------------------------------- Comment (by simonmar): Ugh, I think `-maxN` is worse than `-Nmax` because of the weird overlap with `-m`. I think `-Nmax<n>` was completely fine (but I was too late!). The documentation could read "use at most <n> processors". -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10728#comment:21 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10728: Add e.g. "-N<=4" in addition to the fixed "-N4" and variable "-N" RTS options -------------------------------------+------------------------------------- Reporter: rrnewton | Owner: MarcelineVQ Type: feature request | Status: closed Priority: normal | Milestone: 8.0.1 Component: Runtime System | Version: 7.10.2 Resolution: fixed | Keywords: newcomer Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #9221, #8224 | Differential Rev(s): Phab:D1650, Wiki Page: | Phab:D1677 -------------------------------------+------------------------------------- Comment (by rwbarton): Probably won't help, but I agree that the original `-Nmax` is clearly the best name. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10728#comment:23 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10728: Add e.g. "-N<=4" in addition to the fixed "-N4" and variable "-N" RTS options -------------------------------------+------------------------------------- Reporter: rrnewton | Owner: MarcelineVQ Type: feature request | Status: closed Priority: normal | Milestone: 8.0.1 Component: Runtime System | Version: 7.10.2 Resolution: fixed | Keywords: newcomer Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #9221, #8224 | Differential Rev(s): Phab:D1650, Wiki Page: | Phab:D1677 -------------------------------------+------------------------------------- Comment (by Thomas Miedema ): In [changeset:"1b72534b99ea17012746ef97b4892a7c9c3450dd/ghc" 1b72534/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="1b72534b99ea17012746ef97b4892a7c9c3450dd" Fixup test for #10728 It was failing for WAY=ghci. }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10728#comment:25 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler