Cabal's detailed test interface

27 Jul 2011

      Greetings interested Cabal users and developers!

I am pleased to present another revision of Cabal's detailed test
interface for your scrutiny. I have attached a patch for the Cabal
repository to this message so you can try the new interface and test
runner for yourselves, but the interface from Distribution.TestSuite
is included below with a few of my comments.

This interface is based very heavily on Duncan's proposal from our
last round of discussion. I indicated deviations by comments in the
source, but the commit log is a good summary of the changes:

* The constructors of 'Tests' have been renamed. I think 'TestGroup'
and 'ExtraTestOptions' are redundant: it is clear from context what
sort of Group and what sort of ExtraOptions are being considered and
qualified imports can resolve any name conflicts. Group and
ExtraOptions have the advantage of improving the brevity of pattern
matches on Tests, which are used very often in D.S.Test.

* The 'concurrentSafe :: Bool' field of TestInstance has become the
'concurrently' field of Group, allowing package and test framework
authors greater control over concurrency.

* The 'Finished' constructor of 'Progress' now contains the options
used to run the test in addition to the test result. Without returning
the options, it would be difficult to extract the RNG seed used to run
a test.

* A detailed test suite module now exports the symbol 'test :: IO
Tests'. This enables the use of IO to enumerate the tests in a group,
suggested by Duncan as a way to accommodate the GHC test suite.
...
module Distribution.TestSuite
    ( TestInstance(..)
    , OptionDescr(..)
    , OptionType(..)
    , Tests(..)
    , Options
    , Progress(..)
    , Result(..)
    ) where
data TestInstance = TestInstance
    { run       :: IO Progress      -- ^ Perform the test.
    , name      :: String           -- ^ A name for the test, unique within a
                                    -- test suite.
    , tags      :: [String]         -- ^ Users can select groups of tests by
                                    -- their tags.
    , options   :: [OptionDescr]    -- ^ Descriptions of the options recognized
                                    -- by this test.
    , setOption :: String -> String -> Either String TestInstance
        -- ^ Try to set the named option to the given value. Returns an error
        -- message if the option is not supported or the value could not be
        -- correctly parsed; otherwise, a 'TestInstance' with the option set to
        -- the given value is returned.
    }
data OptionDescr = OptionDescr
    { optionName        :: String
    , optionDescription :: String       -- ^ A human-readable description of the
                                        -- option to guide the user setting it.
    , optionType        :: OptionType
    , optionDefault     :: Maybe String
    }
  deriving (Eq, Read, Show)
data OptionType
    = OptionFile
        { optionFileMustExist   :: Bool
        , optionFileIsDir       :: Bool
        , optionFileExtensions  :: [String]
        }
    | OptionString
        { optionStringMultiline :: Bool
        , optionStringDescription :: String
        }
    | OptionNumber
        { optionNumberIsInt     :: Bool
        , optionNumberBounds    :: (Maybe String, Maybe String)
        }
    | OptionBool
    | OptionEnum [String]
    | OptionSet [String]
    | OptionRngSeed
  deriving (Eq, Read, Show)
data Tests
    = Test TestInstance
    | Group
        { groupName     :: String
        , concurrently  :: Bool     -- ^ If true, then the child 'Test's or
                                    -- 'Group's can safely be run concurrently;
                                    -- otherwise, they must be run in series.
        , groupTests    :: [Tests]
        }
    | ExtraOptions [OptionDescr] Tests
type Options = [(String, String)]
data Progress = Finished Options Result
                -- ^ The options used must be returned when the test is finished,
                -- or it will not be possible to recreate tests taking a random
                -- seed.
              | Progress String (IO Progress)
data Result = Pass
            | Fail String
            | Error String
  deriving (Eq, Read, Show)
As always, your comments are appreciated. Thanks!

-- 
Thomas Tuegel