Re: Windows test failures

Hi Tamar, WINDOWS=======On Windows I did the following changes before running the 'plugin09' test: 1.) In the compiler (HscMain.hs), just before calling the plugin function 'parsedPlugin', I set BufferMode of the file stdout to LineBuffering. 2.) At the same place, I write a message to stdout with the text "COMPILER About to call plugin parse" and the result of the buffer-mode query. 3.) In the plugin, in the function parsedPlugin, I query and print (to stdout) the buffer mode. 4.) I added the heading PLUGIN to the normal parse message issued by the parsedPlugin function 5.) In the compiler (HscMain.hs) just after returning from the plugin, I print the line "COMPILER Returning from plugin parse" to stdout. 6.) In the  plugin function interfaceLoadPlugin' that is called much later, I flush stdout, and add the heading "PLUGIN". This gives the following interesting result:  COMPILER About to call plugin parse: LineBuffering COMPILER Returning from plugin parse PLUGIN Buffermode: BlockBuffering Nothing PLUGIN parsePlugin(a,b) PLUGIN interfacePlugin: Prelude ... The output lines do not appear in the sequence they were produced!!The plugin doesn't see/inherit the BlockBuffer mode (LineBuffering) set by the compiler!! This is a strong indication, that there are two different buffers for stdout. One in the compiler and another one in the plugin.At the end of the processing, the buffer in the compiler is automatically flushed, however the buffer in the plugin never gets flushed! LINUX=====I did a similar test in Linux, however, here I set the buffer mode to 'Blockmode Nothing' and I didn't do a manual flush in the plugin. I got the following result:  COMPILER About to call plugin parse: Buffering mode: BlockBuffering Nothing PLUGIN Buffering: BlockBuffering Nothing PLUGIN parsePlugin(a,b) COMPILER Returning from plugin parse PLUGIN interfacePlugin: Prelude ... Here the lines are in the same order as they were produced.The setting of the Buffering mode is inherited by the plugin. I think, on Linux the compiler and the plugin share the same buffer. To fix the issue on Windows, the compiler and the plugin should use the same buffer for stdout.However I don't know whether this is possible / difficult / easy?What's your opinion? Many thanks and kind regards   Roland Here are my changes for Windows in code: Change in HscMain the line "import System.IO (fixIO)" to "import System.IO" Last lines of function HscMain.hs:hscParse'             -- apply parse transformation of plugins            let applyPluginAction p opts                  = parsedResultAction p opts mod_summary            liftIO $ hSetBuffering stdout LineBuffering            mode <- liftIO $ hGetBuffering stdout            liftIO $ putStrLn ("COMPILER About to call plugin parse: " ++ show mode)            rsxresult <- withPlugins dflags applyPluginAction res            liftIO $ putStrLn "COMPILER Returning from plugin parse"            return rsxresult New code for function SourcePlugin.hs:parsedPlugin parsedPlugin opts _ pm  = do       mode <- liftIO $ hGetBuffering stdout       liftIO $ putStrLn $ "PLUGIN Buffermode: " ++ show mode       liftIO $ putStrLn $ "PLUGIN parsePlugin(" ++ intercalate "," opts ++ ")"       return pm New code for function SourcePlugin.hs:interfaceLoadPlugin' interfaceLoadPlugin' :: [CommandLineOption] -> ModIface -> IfM lcl ModIfaceinterfaceLoadPlugin' _ iface  = do liftIO $ putStrLn $ "PLUGIN interfacePlugin: "                              ++ (showSDocUnsafe $ ppr $ mi_module iface)       liftIO $ hFlush stdout       return iface Am Dienstag, den 04.12.2018, 00:02 +0000 schrieb Phyx:

Hi Roland,

Thanks for looking into these.

...

  I looked into the testcases 'plugins09', 'plugins10' and 'plugins11' and  found the following: GHC-Windows uses BufferMode 'BlockBuffering  Nothing', however, GHC-Linux uses 'LineBuffering'. 

Ah, yes, this isn't technically a Linux vs Windows thing, GHC will always default to LineBuffering for terminals and BlockBuffering for anything else. The issue is that msys2 terminals have std handles that are backed by files instead of pipes. This is an artifact of how they try to emulate posix shells, See https://github.com/msys2/msys2/ wiki/Porting#standard-streams-in-mintty.   This means that when GHC runs inside an msys2 program such as bash it'll always default to BlockBuffering.

...

  I don't know anything about the "Why" and "Where" in the GHC IO module on Windows, so I'm unable to come up with a patch. 

The above said the handles should be getting flushed when the finalizers are run, but these aren't 100% guaranteed so if the tests rely on this then your solution (to force buffer mode to LineBuffering) sounds like perfectly adequate.  Could you put a patch up with that?

My new I/O manager takes a different approach to determine the buffer mode, but I still have some kinks to work out before posting it.

...

  PS: I can't say anything about the tests 'plugin-recomp-pure' and  'plugin-recomp-impure' as these tests run successfully on my (slow)  Windows box. 

These don't fail for me or harbormaster either, so if Simon is able to consistently reproduce these then I'll have to ask him for a core dump so I can take a look.

Thanks again, I appreciate the help!

Regards, Tamar

On Mon, Dec 3, 2018 at 3:34 PM Roland Senn wrote:

...

Hi Tamar,

I looked into the testcases 'plugins09', 'plugins10' and 'plugins11' and found the following: GHC-Windows uses BufferMode 'BlockBuffering Nothing', however, GHC-Linux uses 'LineBuffering'.

If I add an 'import System.IO' and the line 'liftIO $ hSetBuffering stdout LineBuffer' as first line in the do block of the function '.../testuite/tests/plugins/simple- plugin/Simple/SourcePlugin.hs:parsedPlugin' then all 3 tests pass successfully on Windows!

I don't know anything about the "Why" and "Where" in the GHC IO module on Windows, so I'm unable to come up with a patch.

Regards    Roland

PS: I can't say anything about the tests 'plugin-recomp-pure' and 'plugin-recomp-impure' as these tests run successfully on my (slow) Windows box.