#10560: -f and -O options interact in non-obvious, order dependent ways -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Incorrect | Unknown/Multiple warning at compile-time | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Description changed by bgamari: Old description:

The `-O[012]` and `-f*` flags are both implemented by manipulating a set of flags in `DynFlags`. These manipulations occur in the order that the flags occur in the command line. This leads to some rather surprising behavior.

For instance, if the user wants to compile with optimization but specifically wants to disable specialisation, they might enter `-fno- specialise -O1`. This, however, would not do what the user expects as `-O1` implies `-fspecialise`, overriding the first flag.

This is surprising and poorly documented behavior at best.

New description: The `-O[012]` and `-f*` flags are both implemented by manipulating a set of flags in `DynFlags`. These manipulations occur in the order that the flags occur in the command line. This leads to some rather surprising behavior. For instance, if the user wants to compile with optimization but specifically wants to disable specialisation, they might enter `-fno- specialise -O1`. This, however, would not do what the user expects as `-O1` implies `-fspecialise`, overriding the first flag. This is surprising and poorly documented behavior at best. See the later comments of #10491 which lead to this ticket. -- -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10560#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10560: -f and -O options interact in non-obvious, order dependent ways -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Incorrect | Unknown/Multiple warning at compile-time | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by nomeata): I don’t find that too surprising, and the order dependency is often used in build tools, which assume that you can always override earlier choices by ''appending'' to the command line. But it could be documented better, of course. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10560#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10560: -f and -O options interact in non-obvious, order dependent ways -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Incorrect | Unknown/Multiple warning at compile-time | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by bgamari): I have opened Phab:D1008 to document the current behavior. nomeata, I find the fact that the `-f` flags are order dependent to be perfectly acceptable. Moreover, I don't know how else one would handle this without losing expressive power. I think the real issue is the fact that `-O` and `-f` can silently conflict and one has know way of knowing this other than looking at `optLevelFlags` in the source. This tripped up both me and George in #10491. I think gidyn's suggestion of warning is a nice solution. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10560#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10560: -f and -O options interact in non-obvious, order dependent ways -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Incorrect | Unknown/Multiple warning at compile-time | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by gidyn): Overriding -fx with a subsequent -Ox is arguably bad practice, as it is not clear which flags (if any) are being overridden. It should be overridden with the corresponding -fnot-x. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10560#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10560: -f and -O options interact in non-obvious, order dependent ways -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Incorrect | Unknown/Multiple warning at compile-time | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by svenpanne): I think we should follow the principle of least surprise, and IMHO if the order is relevant, it *is* least surprising to all people who know various tools/compilers/etc. I would strongly object any change regarding this, even adding warnings (will break builds, see above) or handling meta flags differently. If it helps, think of the command line as a list of functions modifying the flag state and GHC as doing a foldl over it, starting with its default state. :-) Making the documentation more explicit/clearer is perfectly fine, though. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10560#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10560: -f and -O options interact in non-obvious, order dependent ways -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Incorrect | Unknown/Multiple warning at compile-time | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by svenpanne): It's totally irrelevant if it's easy to fix: The proposed change is completely surprising, unnatural and ad hoc. When you intend to break things, you'll better have a very good reason for doing so, and I don't think that this is the case here. It is the first time that I've heard that order-dependent flags are considered a problem, quite the opposite: Making them behave differently would confuse people and cause useless work. Later options override/modify previous options, that's how most commandline tools work. By all means, improve the documentation, but leave the flag processing as it is... -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10560#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10560: -f and -O options interact in non-obvious, order dependent ways -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Incorrect | Unknown/Multiple warning at compile-time | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by bgamari): George, it would be great if you could have a quick look at D1009 and confirm that the proposed language it addresses your concerns (suggestions for improvement would also be appreciated). Sven, I think we agree that the left-to-right parsing is standard and expected. The problem here is the fact that there is no sign to the user that `-f` and `-O` should interact. This spooky action at a distance has the very real potential to trip up users and has already masked the effectiveness of a workaround to a known bug. However, I can also see that it's not always possible to ensure that `-O` is placed first in the command line and making users endure a warning in this case is unfortunate. However, I think this is outweighed by the danger of a user not knowing which optimizations the compiler is actually performing. Trusting that they we see and remember a note in the documentation does not seem like a strong enough measure here. If the warning really is a problem we could issue it only if the last flag to set a given option was `-O`. This would allow the user to silence the warning by adding an additional `-f`, reaffirming the value set by `-O`. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10560#comment:13 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10560: -f and -O options interact in non-obvious, order dependent ways -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Incorrect | Unknown/Multiple warning at compile-time | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by rwbarton): The parts of this I feel strongly about are * GHC should never warn about a command line that is valid. It will just cause too many headaches for build systems. The effects of options should be clearly documented and then GHC should do what the user asks without any second-guessing. If a combination of options really doesn't make sense, then it should be an error and not a warning. * `-ffoo -fno-foo` and `-fno-foo -ffoo` (and `-O1 -O0`, etc.) should be valid with the last flag taking priority, as is the case today. I'm not really qualified to comment on how obvious the current behavior of `-fno-specialise -O1` is (it's obvious to me, but I've had the dubious pleasure of reading through DynFlags.hs on many occasions). I also can't think of a use case where the rule suggested in comment:2 "process all `-O` flags before all `-f` flags" would prevent one from doing something useful. So it sounds OK to me. But no warnings please! -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10560#comment:15 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10560: -f and -O options interact in non-obvious, order dependent ways -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Incorrect | Unknown/Multiple warning at compile-time | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by svenpanne): Hmmm, GCC's behavior is a bit weird, and to be honest: It's the first time that I've heard of the fact that e.g. -Ofoo is processed before -fblah, regardless of the order on the command line. Re-reading GCC's documentation, it is explicitly stated that order is important, but the docs introduce the notion of "option kind", without actually specifying what those kinds are and in which order those kinds are processed (https://gcc.gnu.org/onlinedocs/gcc-5.1.0/gcc/Invoking-GCC.html). Obviously, the -Ofoo options are of a different kind than the -fblah options, and the -Ofoo ones are processed before the -fblah ones. *But* within these kinds, order matters: {{{ $ gcc -c -Q --help=optimizers -O3 -O0 | grep tree-vectorize -ftree-vectorize [disabled] $ gcc -c -Q --help=optimizers -O0 -O3 | grep tree-vectorize -ftree-vectorize [enabled] }}} OTOH, clang doesn't seem to have this notion of "kinds" and simply processes options in order: {{{ $ echo 'int x;' | clang -xc -fno-vectorize -O2 - -o /dev/null -\#\#\# 2>&1 | sed 's/\s/\n/g' | grep vectorize-loops "-vectorize-loops" $ echo 'int x;' | clang -xc -O2 -fno-vectorize - -o /dev/null -\#\#\# 2>&1 | sed 's/\s/\n/g' | grep vectorize-loops <no output> }}} Same for most other tools, including e.g. git: {{{ $ git log --pretty=short --oneline ab3926f Fix PR13851: Preserve metadata for the unswitched branch 9e7539a Remove broken banner. ... $ git log --oneline --pretty=short commit ab3926f Author: Weiming Zhao Fix PR13851: Preserve metadata for the unswitched branch commit 9e7539a Author: Rafael Espindola Remove broken banner. ... }}} So in a nutshell: If we really want to mirror GCC's behavior (which is not 100% clear to me), we have to introduce "option kinds", partition our flags into those kinds, and impose an order on those kinds. Furthermore, one can't simply expand "meta" flags and process those expanded flags afterwards (see the -O3 -O0 vs. -O0 -O3 example above), the expansion can only be done after the corresponding "kind phase" has been processed. Much fun documenting all this stuff. :-) IMHO this is far too complicated, and GCC seems to be an outlier regarding this, just processing things in order (as we currently do?) is the easiest thing. But I'll leave the final decision up to those implementing and documenting any change, as long as we don't introduce any ad hoc warnings. Final thought: clang's optimizer options for -O0, -O1, ... are *not* subsets of the previous optimization level, so one has to think about meta options removing flags. o_O I hope we won't do this... -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10560#comment:17 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10560: -f and -O options interact in non-obvious, order dependent ways -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Incorrect | Unknown/Multiple warning at compile-time | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by George): Replying to [comment:18 slyfox]:

I don't have any expectations on the effect of -f / -O ordering. Would be nice to have an option to dump final option effect on GHC as gcc does it with: {{{ gcc --help=optimizers -Q <options> }}} I agree that it would be nice to see "what you said" on the command line, maybe the -v option could give that output?

-- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10560#comment:19 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#10560: -f and -O options interact in non-obvious, order dependent ways -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Incorrect | Unknown/Multiple warning at compile-time | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by thomie): A similar [https://mail.haskell.org/pipermail/haskell- cafe/2015-June/120047.html discussion] for language pragmas: `{-# LANGUAGE GADTs, NoMonoLocalBinds #-}` vs `{-# LANGUAGE NoMonoLocalBinds, GADTs #-}`. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10560#comment:21 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler