[GHC] #15093: Testsuite should output JUnit XML for better CircleCI support
#15093: Testsuite should output JUnit XML for better CircleCI support -------------------------------------+------------------------------------- Reporter: mpickering | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.2.2 Keywords: CI | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- At the moment when builds are run on CircleCI, failures are not reported in any special way. This means you have to download the build log to see the bottom of the testing phase as it is too long to see in the browser. Instead, you can output the test results as JUnit XML which then CircleCI can slurp up and present nicely. https://circleci.com/docs/2.0 /configuration-reference/#store_test_results There is a python library which makes it easy to generate this output. https://github.com/kyrus/python-junit-xml Even better, an actual python testing library could be used which could automatically output this information. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15093 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15093: Testsuite should output JUnit XML for better CircleCI support -------------------------------------+------------------------------------- Reporter: mpickering | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.2.2 Resolution: | Keywords: CI Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by mboes): There are also ways to do this in Haskell. But not without incurring extra dependencies. See e.g. http://hackage.haskell.org/package/tasty-jenkins- xml and http://hackage.haskell.org/package/hspec-jenkins. Perhaps inspiration for generating the requisite XML from the current test harness. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15093#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15093: Testsuite should output JUnit XML for better CircleCI support -------------------------------------+------------------------------------- Reporter: mpickering | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.2.2 Resolution: | Keywords: CI Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by mboes): * cc: mboes (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15093#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15093: Testsuite should output JUnit XML for better CircleCI support -------------------------------------+------------------------------------- Reporter: mpickering | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.2.2 Resolution: | Keywords: CI Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): The testsuite driver indeed already has support for this. See 54d3a1fdeb7044a1d9bb025d4880d08c708b4cd0. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15093#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15093: Testsuite should output JUnit XML for better CircleCI support -------------------------------------+------------------------------------- Reporter: mpickering | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.2.2 Resolution: | Keywords: CI Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by mpickering): This was easy to integrate but the XML was malformed somehow. {{{ ParseError at [row,col]:[1,316] Message: The value of attribute "type" associated with an element type "failure" must not contain the '<' character. }}} https://circleci.com/gh/tweag/ghc/904#tests/containers/0 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15093#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15093: Testsuite should output JUnit XML for better CircleCI support -------------------------------------+------------------------------------- Reporter: mpickering | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.2.2 Resolution: | Keywords: CI Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by mpickering): In particular, the XML produced doesn't look like it follows the schema. https://www.ibm.com/support/knowledgecenter/en/SSQ2R2_14.1.0/com.ibm.rsar.an... There appear to be a number of missing fields that are required. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15093#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15093: Testsuite should output JUnit XML for better CircleCI support -------------------------------------+------------------------------------- Reporter: mpickering | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.2.2 Resolution: | Keywords: CI Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by mpickering): Here is the generated XML. https://911-16347359-gh.circle-artifacts.com/0/home/ghc/project/test- results/junit.xml I think I can see where the problem is coming from.. Will submit a patch later. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15093#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15093: Testsuite should output JUnit XML for better CircleCI support -------------------------------------+------------------------------------- Reporter: mpickering | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.2.2 Resolution: | Keywords: CI Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by mpickering): See https://phabricator.haskell.org/D4645 and https://phabricator.haskell.org/D4646 and for an example of the output - https://circleci.com/gh/tweag/ghc/920 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15093#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15093: Testsuite should output JUnit XML for better CircleCI support
-------------------------------------+-------------------------------------
Reporter: mpickering | Owner: (none)
Type: bug | Status: new
Priority: normal | Milestone: 8.6.1
Component: Compiler | Version: 8.2.2
Resolution: | Keywords: CI
Operating System: Unknown/Multiple | Architecture:
| Unknown/Multiple
Type of failure: None/Unknown | Test Case:
Blocked By: | Blocking:
Related Tickets: | Differential Rev(s):
Wiki Page: |
-------------------------------------+-------------------------------------
Comment (by Ben Gamari
#15093: Testsuite should output JUnit XML for better CircleCI support
-------------------------------------+-------------------------------------
Reporter: mpickering | Owner: (none)
Type: bug | Status: new
Priority: normal | Milestone: 8.6.1
Component: Compiler | Version: 8.2.2
Resolution: | Keywords: CI
Operating System: Unknown/Multiple | Architecture:
| Unknown/Multiple
Type of failure: None/Unknown | Test Case:
Blocked By: | Blocking:
Related Tickets: | Differential Rev(s):
Wiki Page: |
-------------------------------------+-------------------------------------
Comment (by Ben Gamari
#15093: Testsuite should output JUnit XML for better CircleCI support -------------------------------------+------------------------------------- Reporter: mpickering | Owner: (none) Type: bug | Status: closed Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.2.2 Resolution: fixed | Keywords: CI Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * status: new => closed * resolution: => fixed Comment: Thanks mpickering! -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15093#comment:10 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
participants (1)
-
GHC