Problems using tasty-golden
Hi *, I've been trying to write a testsuite using tasty-golden and I've run into two technical problems. I'll start with some background. I'm working on singletons library that provides Template Haskell functionality. I would like to tests whether some particular TH splices compile at all and if they compile whether they produce expected output. My plan is to attempt compilation of a file with GHC. If compilation fails then the test fails. If it succeedes then output is compared with a golden file. I am directly calling goldenTest function from Test.Tasty.Golden.Internal. I wrote this function to produce the test output: compileWithGHC :: IO (Either BSL.ByteString BSL.ByteString) It runs GHC with some specified options and returns Left stderr (if compilation fails) or Right stdout (if it succeedes). This function is passed as third parameter to goldenTest (using liftIO). Then I have a comparing function: cmp :: Either BSL.ByteString BSL.ByteString -> Either BSL.ByteString BSL.ByteString -> IO (Maybe String) If it gets two Rights then it compares them. If the second param is left then the function returns Just stderr. I've run into following problems: 1. When I try to run my test I get: message threw an exception: MY_GOLDEN_FILE: hGetBufSome: illegal operation (handle is closed) I suspect that lazy IO is getting in the way but I have no idea why exactly things go wrong. 2. I had to hardcode the path to my golden file. Otherwise the file could not be found. I suspect the reason is that test are run somewhere in dist/ subdirectory whereas my golden files are located alongside my test files. What can I do to have the golden files located without giving specific subdirectories in the source code? A bonus question. I redefined vgReadFile from Test.Tasty.Golden.Internal to return Either: vgReadFileE :: FilePath -> ValueGetter r (Either BSL.ByteString BSL.ByteString) vgReadFileE path = (fmap Right . liftIO . BSL.hGetContents =<<) $ ValueGetter $ ContT $ \k -> bracket (openBinaryFile path ReadMode) hClose k The only modification is the "fmap Right .". How can I return Left if opening the file fails? Janek
* Jan Stolarek
Hi *,
I've been trying to write a testsuite using tasty-golden and I've run into two technical problems.
I'll start with some background. I'm working on singletons library that provides Template Haskell functionality. I would like to tests whether some particular TH splices compile at all and if they compile whether they produce expected output. My plan is to attempt compilation of a file with GHC. If compilation fails then the test fails. If it succeedes then output is compared with a golden file. I am directly calling goldenTest function from Test.Tasty.Golden.Internal. I wrote this function to produce the test output:
compileWithGHC :: IO (Either BSL.ByteString BSL.ByteString)
It runs GHC with some specified options and returns Left stderr (if compilation fails) or Right stdout (if it succeedes). This function is passed as third parameter to goldenTest (using liftIO). Then I have a comparing function:
cmp :: Either BSL.ByteString BSL.ByteString -> Either BSL.ByteString BSL.ByteString -> IO (Maybe String)
If it gets two Rights then it compares them. If the second param is left then the function returns Just stderr.
I've run into following problems:
1. When I try to run my test I get:
message threw an exception: MY_GOLDEN_FILE: hGetBufSome: illegal operation (handle is closed)
I suspect that lazy IO is getting in the way but I have no idea why exactly things go wrong.
That's a plausible explanations. File handles are closed when the ValueGetter monad is run. Try to add something like Control.Exception.evaluate $ Control.DeepSeq.force cmpResult to your comparison function. (I think I should build this forcing behaviour right into tasty-golden itself.) (Filed as https://github.com/feuerbach/tasty-golden/issues/7)
2. I had to hardcode the path to my golden file. Otherwise the file could not be found. I suspect the reason is that test are run somewhere in dist/ subdirectory whereas my golden files are located alongside my test files. What can I do to have the golden files located without giving specific subdirectories in the source code?
Golden files are looked up in an ordinary way — i.e. relative paths are interpreted relative to the test program's CWD. So, for example, if your test files are under 'test/' subdirectory of your project's main directory, and you want to run the test suite from your project's main directory, then your paths must look like "test/...". (Filed as https://github.com/feuerbach/tasty-golden/issues/8)
A bonus question. I redefined vgReadFile from Test.Tasty.Golden.Internal to return Either:
vgReadFileE :: FilePath -> ValueGetter r (Either BSL.ByteString BSL.ByteString) vgReadFileE path = (fmap Right . liftIO . BSL.hGetContents =<<) $ ValueGetter $ ContT $ \k -> bracket (openBinaryFile path ReadMode) hClose k
The only modification is the "fmap Right .". How can I return Left if opening the file fails?
First off, you don't have to do that (unless you have some special needs — if so, what are they?). If the file doesn't exist, tasty will catch the exception and handle it gracefully. If you really want to do it (or are just curious), the easiest way is to revert your addition of 'fmap Right' and replace the 'bracket' code with 'try' code. Roman
* Roman Cheplyaka
A bonus question. I redefined vgReadFile from Test.Tasty.Golden.Internal to return Either:
vgReadFileE :: FilePath -> ValueGetter r (Either BSL.ByteString BSL.ByteString) vgReadFileE path = (fmap Right . liftIO . BSL.hGetContents =<<) $ ValueGetter $ ContT $ \k -> bracket (openBinaryFile path ReadMode) hClose k
The only modification is the "fmap Right .". How can I return Left if opening the file fails?
First off, you don't have to do that (unless you have some special needs — if so, what are they?). If the file doesn't exist, tasty will catch the exception and handle it gracefully.
If you really want to do it (or are just curious), the easiest way is to revert your addition of 'fmap Right' and replace the 'bracket' code with 'try' code.
On a second thought, you can't *replace* 'bracket' with 'try', because of possible exceptions arising from 'k' (also, because of possible async exceptions). Instead, you need to use both. Something like (untested) bracket (try $ openBinaryFile path ReadMode :: IO (Either IOException Handle)) (either (const $ pure ()) hClose) k Roman
Thanks for quick reply Roman (and sorry for my slow reply - I was AFK for the weekend).
That's a plausible explanations. File handles are closed when the ValueGetter monad is run. Try to add something like
Control.Exception.evaluate $ Control.DeepSeq.force cmpResult
to your comparison function. Yes, that works. I wonder why no one stumbled upon this problem before.
if your test files are under 'test/' subdirectory of your project's main directory, and you want to run the test suite from your project's main directory, then your paths must look like "test/...". Oh, right. I couldn't get this right because I set wrong CWD for the GHC process.
First off, you don't have to do that (unless you have some special needs — if so, what are they?). No, I don't have any special needs. It's just that my cmp function has to cover all possible cases - including when the file loading function returns Left - so I figured out I can handle the potential error there and have my own error message. But that's not really necessary, which means I can get rid of my custom cgReadFile altogether and use (fmap Right $ vgReadFile path).
Once again thank you. Janek
participants (2)
-
Jan Stolarek -
Roman Cheplyaka