[GHC] #10951: HPC program has poor error reporting / strange CLI in general
#10951: HPC program has poor error reporting / strange CLI in general -------------------------------------+------------------------------------- Reporter: mgsloan | Owner: Type: bug | Status: new Priority: low | Milestone: Component: Code Coverage | Version: 7.10.2 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | -------------------------------------+------------------------------------- 1. Many erroneous usages of the hpc program result in reporting 100% (0/0) coverage. Proposed resolution: * Report an error and exit with failure. I can't think of a legitimate case where you would want a report that doesn't consider any expressions / declarations. * If a --include flag is specified, I think it would be valuable to output all the module names, so that it's clearer why the --include flag didn't work. * At the very least, (0/0) coverage shouldn't be reported as "100%". 2. Positional arguments after the initial tix file get interpreted as --include filters, filtering which modules / packages are considered. By default, HPC includes all modules in the report, unless include filters are supplied. If you accidentally provide a positional argument which isn't a valid thing for "--include", then your reports will always be (0/0) coverage. Proposed resolution: * Deprecate using positional arguments for include filters. This way it will complain about this instead of yielding trivial (0/0) coverage reports 3. Ideally, you'd be able to pass in multiple tix files and combine them in memory instead of using "hpc combine" and generating intermediate tix files. This would be a better usage of positional arguments than include filters. For now, the additional tix files would need to be flag arguments. This also isn't all that useful until https://ghc.haskell.org/trac/ghc/ticket/1853 is fixed. 4. `--srcdir`, `--hpcdir`, and `--reset-hpcdirs` interface has strange semantics for searching for mix files. It literally does something roughly like `[sd hd | sd <- srcDirs, hd <- hpcDirs]`, and uses this when looking for mix files. This would be OK, except that the `srcdir` must be the work dir used for compilation, in order for source markup to be generated. This means that you are tied to having your hpcdir be a sub-directory of the compilation work dir. Proposed resolution: * Add `--mixdir`, specifying a path to a mix file directory. Can be supplied multiple times. If `--mixdir` is provided, then it's an error to also use `--hpcdir` / `--reset-hpcdirs`. Instead of fixing the hpc program, I have considered forking it and giving it a new executable name, so that the CLI and behavior can be entirely changed. Thoughts on this alternative? It is appealing because it seems tricky to fix the hpc program's CLI while ensuring that backwards compatibility is maintained. I am keen on working on fixing this situation, but I want consensus from y'all about the approach that should be taken. Maybe it makes sense for this hpc-ng program to be developed outside of the GHC repo? This way it can use more recent libraries for generating HTML and parsing command line arguments. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10951 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10951: HPC program has poor error reporting / strange CLI in general -------------------------------------+------------------------------------- Reporter: mgsloan | Owner: Type: bug | Status: new Priority: low | Milestone: Component: Code Coverage | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): -------------------------------------+------------------------------------- Description changed by mgsloan: Old description:
1. Many erroneous usages of the hpc program result in reporting 100% (0/0) coverage. Proposed resolution:
* Report an error and exit with failure. I can't think of a legitimate case where you would want a report that doesn't consider any expressions / declarations.
* If a --include flag is specified, I think it would be valuable to output all the module names, so that it's clearer why the --include flag didn't work.
* At the very least, (0/0) coverage shouldn't be reported as "100%".
2. Positional arguments after the initial tix file get interpreted as --include filters, filtering which modules / packages are considered. By default, HPC includes all modules in the report, unless include filters are supplied. If you accidentally provide a positional argument which isn't a valid thing for "--include", then your reports will always be (0/0) coverage. Proposed resolution:
* Deprecate using positional arguments for include filters. This way it will complain about this instead of yielding trivial (0/0) coverage reports
3. Ideally, you'd be able to pass in multiple tix files and combine them in memory instead of using "hpc combine" and generating intermediate tix files. This would be a better usage of positional arguments than include filters. For now, the additional tix files would need to be flag arguments. This also isn't all that useful until https://ghc.haskell.org/trac/ghc/ticket/1853 is fixed.
4. `--srcdir`, `--hpcdir`, and `--reset-hpcdirs` interface has strange semantics for searching for mix files. It literally does something roughly like `[sd hd | sd <- srcDirs, hd <- hpcDirs]`, and uses this when looking for mix files. This would be OK, except that the `srcdir` must be the work dir used for compilation, in order for source markup to be generated. This means that you are tied to having your hpcdir be a sub-directory of the compilation work dir. Proposed resolution:
* Add `--mixdir`, specifying a path to a mix file directory. Can be supplied multiple times. If `--mixdir` is provided, then it's an error to also use `--hpcdir` / `--reset-hpcdirs`.
Instead of fixing the hpc program, I have considered forking it and giving it a new executable name, so that the CLI and behavior can be entirely changed. Thoughts on this alternative? It is appealing because it seems tricky to fix the hpc program's CLI while ensuring that backwards compatibility is maintained.
I am keen on working on fixing this situation, but I want consensus from y'all about the approach that should be taken.
Maybe it makes sense for this hpc-ng program to be developed outside of the GHC repo? This way it can use more recent libraries for generating HTML and parsing command line arguments.
New description: 1. Many erroneous usages of the hpc program result in reporting 100% (0/0) coverage. Proposed resolution: * Report an error and exit with failure. I can't think of a legitimate case where you would want a report that doesn't consider any expressions / declarations. * If a --include flag is specified, I think it would be valuable to output all the module names, so that it's clearer why the --include flag didn't work. * At the very least, (0/0) coverage shouldn't be reported as "100%". 2. Positional arguments after the initial tix file get interpreted as --include filters, filtering which modules / packages are considered. By default, HPC includes all modules in the report, unless include filters are supplied. If you accidentally provide a positional argument which isn't a valid thing for "--include", then your reports will always be (0/0) coverage. Proposed resolution: * Deprecate using positional arguments for include filters. This way it will complain about this instead of yielding trivial (0/0) coverage reports 3. Ideally, you'd be able to pass in multiple tix files and combine them in memory instead of using "hpc combine" and generating intermediate tix files. This would be a better usage of positional arguments than include filters. For now, the additional tix files would need to be flag arguments. This also isn't all that useful until https://ghc.haskell.org/trac/ghc/ticket/1853 is fixed. 4. `--srcdir`, `--hpcdir`, and `--reset-hpcdirs` interface has strange semantics for searching for mix files. It literally does something roughly like `[sd hd | sd <- srcDirs, hd <- hpcDirs]`, and uses this when looking for mix files. This would be OK, except that the `srcdir` must be the work dir used for compilation, in order for source markup to be generated. This means that you are tied to having your hpcdir be a sub-directory of the compilation work dir. Proposed resolution: * Add `--mixdir`, specifying a path to a mix file directory. Can be supplied multiple times. If `--mixdir` is provided, then it's an error to also use `--hpcdir` / `--reset-hpcdirs`. * If GHC is provided an absolute `-hpcdir`, then use it rather than appending to the CWD. Instead of fixing the hpc program, I have considered forking it and giving it a new executable name, so that the CLI and behavior can be entirely changed. Thoughts on this alternative? It is appealing because it seems tricky to fix the hpc program's CLI while ensuring that backwards compatibility is maintained. I am keen on working on fixing this situation, but I want consensus from y'all about the approach that should be taken. Maybe it makes sense for this hpc-ng program to be developed outside of the GHC repo? This way it can use more recent libraries for generating HTML and parsing command line arguments. -- -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10951#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10951: HPC program has poor error reporting / strange CLI in general -------------------------------------+------------------------------------- Reporter: mgsloan | Owner: Type: bug | Status: new Priority: low | Milestone: Component: Code Coverage | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): -------------------------------------+------------------------------------- Description changed by mgsloan: Old description:
1. Many erroneous usages of the hpc program result in reporting 100% (0/0) coverage. Proposed resolution:
* Report an error and exit with failure. I can't think of a legitimate case where you would want a report that doesn't consider any expressions / declarations.
* If a --include flag is specified, I think it would be valuable to output all the module names, so that it's clearer why the --include flag didn't work.
* At the very least, (0/0) coverage shouldn't be reported as "100%".
2. Positional arguments after the initial tix file get interpreted as --include filters, filtering which modules / packages are considered. By default, HPC includes all modules in the report, unless include filters are supplied. If you accidentally provide a positional argument which isn't a valid thing for "--include", then your reports will always be (0/0) coverage. Proposed resolution:
* Deprecate using positional arguments for include filters. This way it will complain about this instead of yielding trivial (0/0) coverage reports
3. Ideally, you'd be able to pass in multiple tix files and combine them in memory instead of using "hpc combine" and generating intermediate tix files. This would be a better usage of positional arguments than include filters. For now, the additional tix files would need to be flag arguments. This also isn't all that useful until https://ghc.haskell.org/trac/ghc/ticket/1853 is fixed.
4. `--srcdir`, `--hpcdir`, and `--reset-hpcdirs` interface has strange semantics for searching for mix files. It literally does something roughly like `[sd hd | sd <- srcDirs, hd <- hpcDirs]`, and uses this when looking for mix files. This would be OK, except that the `srcdir` must be the work dir used for compilation, in order for source markup to be generated. This means that you are tied to having your hpcdir be a sub-directory of the compilation work dir. Proposed resolution:
* Add `--mixdir`, specifying a path to a mix file directory. Can be supplied multiple times. If `--mixdir` is provided, then it's an error to also use `--hpcdir` / `--reset-hpcdirs`.
* If GHC is provided an absolute `-hpcdir`, then use it rather than appending to the CWD.
Instead of fixing the hpc program, I have considered forking it and giving it a new executable name, so that the CLI and behavior can be entirely changed. Thoughts on this alternative? It is appealing because it seems tricky to fix the hpc program's CLI while ensuring that backwards compatibility is maintained.
I am keen on working on fixing this situation, but I want consensus from y'all about the approach that should be taken.
Maybe it makes sense for this hpc-ng program to be developed outside of the GHC repo? This way it can use more recent libraries for generating HTML and parsing command line arguments.
New description: 1. Many erroneous usages of the hpc program result in reporting 100% (0/0) coverage. Proposed resolution: * Report an error and exit with failure. I can't think of a legitimate case where you would want a report that doesn't consider any expressions / declarations. * If a --include flag is specified, I think it would be valuable to output all the module names, so that it's clearer why the --include flag didn't work. * At the very least, (0/0) coverage shouldn't be reported as "100%". 2. Positional arguments after the initial tix file get interpreted as --include filters, filtering which modules / packages are considered. By default, HPC includes all modules in the report, unless include filters are supplied. If you accidentally provide a positional argument which isn't a valid thing for "--include", then your reports will always be (0/0) coverage. Proposed resolution: * Deprecate using positional arguments for include filters. This way it will complain about this instead of yielding trivial (0/0) coverage reports 3. Ideally, you'd be able to pass in multiple tix files and combine them in memory instead of using "hpc combine" and generating intermediate tix files. This would be a better usage of positional arguments than include filters. For now, the additional tix files would need to be flag arguments. This also isn't all that useful until https://ghc.haskell.org/trac/ghc/ticket/1853 is fixed. 4. `--srcdir`, `--hpcdir`, and `--reset-hpcdirs` interface has strange semantics for searching for mix files. It literally does something roughly like `[sd hd | sd <- srcDirs, hd <- hpcDirs]`, and uses this when looking for mix files. This would be OK, except that the `srcdir` must be the work dir used for compilation, in order for source markup to be generated. This means that you are tied to having your hpcdir be a sub-directory of the compilation work dir. Proposed resolution: * Add `--mixdir`, specifying a path to a mix file directory. Can be supplied multiple times. If `--mixdir` is provided, then it's an error to also use `--hpcdir` / `--reset-hpcdirs`. * If GHC is provided an absolute `-hpcdir`, then use it rather than appending to the CWD. 5. If anything goes wrong when matching up a module in a tix file with mix metadata, the error is "can not find MODNAME in DIRS". (see the code here: https://github.com/ghc/packages- hpc/blob/fb14d3428ba24d36e779736989dae3092a50a957/Trace/Hpc/Mix.hs#L87) This is quite confusing because this same message is used for all of the following cases: * There's a parse error of the mix file * Any IO error happens * The module hash mismatches Instead of fixing the hpc program, I have considered forking it and giving it a new executable name, so that the CLI and behavior can be entirely changed. Thoughts on this alternative? It is appealing because it seems tricky to fix the hpc program's CLI while ensuring that backwards compatibility is maintained. I am keen on working on fixing this situation, but I want consensus from y'all about the approach that should be taken. Maybe it makes sense for this hpc-ng program to be developed outside of the GHC repo? This way it can use more recent libraries for generating HTML and parsing command line arguments. -- -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10951#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10951: HPC program has poor error reporting / strange CLI in general -------------------------------------+------------------------------------- Reporter: mgsloan | Owner: Type: bug | Status: new Priority: low | Milestone: Component: Code Coverage | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): -------------------------------------+------------------------------------- Changes (by thomie): * cc: thoughtpolice (added) Comment: @mgsloan: It would be great if you could work on this. Asking @thoughtpolice to comment on the pros/cons of developing hpc outside of the GHC repository.
"can not find MODNAME in DIRS" This should be already fixed in HEAD (#10529).
I fixed some other issues as well, not sure if you ran into them: https://ghc.haskell.org/trac/ghc/query?status=closed&component=Code+Coverage&milestone=8.0.1. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10951#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10951: HPC program has poor error reporting / strange CLI in general -------------------------------------+------------------------------------- Reporter: mgsloan | Owner: Type: bug | Status: new Priority: low | Milestone: Component: Code Coverage | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): -------------------------------------+------------------------------------- Comment (by mgsloan): @thomie Cool! I did indeed run into the current work directory / absolute paths handling. Glad that's fixed! I've thought about it a little more, and I think it probably doesn't make sense at this time to work on the hpc program outside of the GHC repo. So I guess the main thing is: Do my proposed resolutions seem like the right approach to resolving these issues? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10951#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10951: HPC program has poor error reporting / strange CLI in general -------------------------------------+------------------------------------- Reporter: mgsloan | Owner: Type: bug | Status: new Priority: low | Milestone: Component: Code Coverage | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by thomie): * cc: andygill@… (added) Comment: 1. yes 2. is it worth breaking backward compatibility here, after 1. is fixed? Maybe there are scripts in existence that make use of positional arguments. 3. seems like a less pressing need, but why not. This effectively folds `hpc combine` into `hpc report` and `hpc markup`, using a new `--tixfile` or so flag. Might be difficult to explain why `hpc sum` and `hpc combine` still exist after this. 4. maybe not needed anymore, now that `-hpcdir` takes absolute paths, but why not. 5. should be fixed, but please make changes as you see fit. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10951#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10951: HPC program has poor error reporting / strange CLI in general -------------------------------------+------------------------------------- Reporter: mgsloan | Owner: Type: bug | Status: new Priority: low | Milestone: Component: Code Coverage | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by mgsloan): 1. Cool :D 2. Good point. Maybe the solution to #1 can just notice if there's more than one positional argument and explain to the user what their meaning is. 3. Now I'm thinking it likely makes sense to take a conservative approach with these changes. How about just letting `hpc combine` take more than two tix files? 4. Yeah, I don't think this is needed anymore after thomie's changes 5. Cool, sounds good! I hope to get around to these code changes soonish, but no promises - it might languish on the backburner for a little while. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10951#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
participants (1)
-
GHC