#13112: Windows 64-bit GHC HEAD segfaults on the code with a lot of TH stuff. ---------------------------------+-------------------------------------- Reporter: awson | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by awson): Sorry for the noise, I've deleted my previous comment since it was quite misleading. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13112#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13112: Windows 64-bit GHC HEAD segfaults on the code with a lot of TH stuff. ---------------------------------+-------------------------------------- Reporter: awson | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by awson): I've managed to create quick and dirty solution to make GHC work properly (basically we have to add `MEM_TOP_DOWN` flag to `VirtualAlloc` in our main `allocNew` allocation routine (see `OSMem.c`). In such a way we allocate all memory except executable (which uses `allocateExec` and `allocateImageAndTrampolines` allocation routines) from top to down leaving enough room for executable memory in low address area. OTOH, while this solves the problem for GHC itself, it can create problems for programs linked against such modified RTS: 1. MS warns that allocation from top to down may be slower; 2. If a client makes its own dynamic data execution using these data, its code will likely crash. Thus if we ever would go this way, we should modify the build system, using different allocation strategies for GHC itself and for RTS libraries distributed along with GHC. I have another couple of patches, which may or may not improve Windows allocation matters in RTS. E.g. `HeapAlloc` works well for `allocateImageAndTrampolines` saving quite a lot of memory comparing to the quite wasteful `VirtualAlloc`. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13112#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13112: Windows 64-bit GHC HEAD segfaults on the code with a lot of TH stuff. ---------------------------------+-------------------------------------- Reporter: awson | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by rwbarton): Is it possible on Windows to allocate a pool of memory addresses, below 2GB and of size, say, 1GB, without actually committing pages? I see comments in the two-step allocator about "VirtualAlloc MEM_RESERVE/MEM_COMMIT" and why it doesn't work for reserving 1TB worth of address space, but perhaps for 1GB it would be okay. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13112#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

Yeah, I thought about this also. It's '''definitely''' possible, but

#13112: Windows 64-bit GHC HEAD segfaults on the code with a lot of TH stuff. ---------------------------------+-------------------------------------- Reporter: awson | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by awson): Replying to [comment:7 awson]: this way we need to write a custom allocator which would commit/decommit pages inside this reserved space. This is surely doable, but not so much easily. Ah, we, perhaps, could reuse parts of `USE_LARGE_ADDRESS_SPACE` code for this (not using any large address space in fact). -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13112#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13112: Windows 64-bit GHC HEAD segfaults on the code with a lot of TH stuff. ---------------------------------+-------------------------------------- Reporter: awson | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by quyse): Replying to [comment:9 awson]:

FYI, I've built (slightly patched) GHC HEAD (170118) with `USE_LARGE_ADDRESS_SPACE` enabled.

Now it automatically allocates main heap at 2GB mark (no need to give any extra options, since Windows already can't find 1TB or contiguous address space starting below 2GB mark) and things work '''perfectly'''.

Apparently, also there are no problems with `USE_LARGE_ADDRESS_SPACE` on W10 1607. Woohoo! Hi, I'm seeing these segfaults on Windows too with GHC 8.0.2 building my own code using TH. Is there a GHC patch I can apply to fix this? I tried using `-fexternal-interpreter`, it works on development machine, but frequently hangs indefinitely on CI machine.

-- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13112#comment:10 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13112: Windows 64-bit GHC HEAD segfaults on the code with a lot of TH stuff. ---------------------------------+-------------------------------------- Reporter: awson | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by awson): I can't. No segfault on 8.2.1 (and never was on 8.0.2). -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13112#comment:12 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13112: Windows 64-bit GHC HEAD segfaults on the code with a lot of TH stuff. ---------------------------------+-------------------------------------- Reporter: awson | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by jbetz): I can reproduce it on 64-bit Windows, with or without TH. The bindings to PostreSQL are completely broken because of it, presumably do to C code generated in postgresql-libpq. It's been identified in multiple database projects: * https://github.com/yesodweb/persistent/issues/697 * https://github.com/tomjaguarpaw/haskell-opaleye/issues/338 Probably not much help in narrowing it down, but this really needs to be looked into. Especially since WSL still isn't a viable alternative at this point. It's unbearably slow. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13112#comment:14 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13112: Windows 64-bit GHC HEAD segfaults on the code with a lot of TH stuff. ---------------------------------+-------------------------------------- Reporter: awson | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by RyanGlScott): I'm aware of the `persistent` issue, see https://ghc.haskell.org/trac/ghc/ticket/14089#comment:8. But alas, as I've documented at that link, I can no longer reliably reproduce the segfault on GHC 8.2.1! Coming up with a minimal example (with preferably no external dependencies) would go a long way in diagnosing this issue. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13112#comment:16 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13112: Windows 64-bit GHC HEAD segfaults on the code with a lot of TH stuff. ---------------------------------+-------------------------------------- Reporter: awson | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by RyanGlScott): jbetz, which code are you referring to? This is the code snippet at the top of https://github.com/tomjaguarpaw/haskell-opaleye/issues/338 at the time of writing: {{{#!hs module Main where import Data.Int import Data.Profunctor.Product (p1) import Database.PostgreSQL.Simple (Connection, connect, connectDatabase, connectPassword, defaultConnectInfo) import Opaleye (Column, PGText, Query, Table (Table), queryTable, required, runInsertMany, runQuery) import qualified Opaleye.PGTypes as P main :: IO () main = do connection <- connect $ defaultConnectInfo { connectDatabase = "postgres", connectPassword = "changeme"} r1 <- doTestInsertQuery connection print r1 r2 <- doTestSelectQuery connection print r2 doTestInsertQuery :: Connection -> IO Int64 doTestInsertQuery connection = runInsertMany connection testTable [P.pgString "ok"] doTestSelectQuery :: Connection -> IO [String] doTestSelectQuery connection = runQuery connection testQuery testQuery :: Query (Column PGText) testQuery = queryTable testTable testTable :: Table (Column PGText) (Column PGText) testTable = Table "test" (p1 (required "message")) }}} But compiling this with GHC 8.2.1 on my 64-bit Windows machine seems to work! {{{ $ ghc -fforce-recomp Bug.hs [1 of 1] Compiling Main ( Bug.hs, Bug.o ) Linking Bug.exe ... $ ./Bug.exe Bug.exe: SqlError {sqlState = "", sqlExecStatus = FatalError, sqlErrorMsg = "could not connect to server: Connection refused (0x0000274D/10061)\n\tIs the server running on host \"127.0.0.1\" and accepting\n\tTCP/IP connections on port 5432?\n", sqlErrorDetail = "", sqlErrorHint = ""} }}} The same thing happens if I compile with `-O2`. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13112#comment:18 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13112: Windows 64-bit GHC HEAD segfaults on the code with a lot of TH stuff. ---------------------------------+-------------------------------------- Reporter: awson | Owner: (none) Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by mgsloan): Not exactly a minimal example ;) But it looks like this is affecting people building recent stack development versions on windows - https://github.com/commercialhaskell/stack/issues/3797#issuecomment-36160107... -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13112#comment:20 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13112: Windows 64-bit GHC HEAD segfaults on the code with a lot of TH stuff. ---------------------------------+-------------------------------------- Reporter: awson | Owner: (none) Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by RyanGlScott): tysonzero, I am unable to reproduce the segfault you're experiencing. Is this the program you're using from https://github.com/yesodweb/persistent/issues/794#issuecomment-370539509 ? (I had to guess some details you left out.) {{{#!hs {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE PackageImports #-} {-# LANGUAGE TypeApplications #-} module Main where import "monad-logger" Control.Monad.Logger import "transformers" Control.Monad.Trans.Reader import "persistent-postgresql" Database.Persist.Postgresql main :: IO () main = runNoLoggingT . withPostgresqlConn "<connection string>" . runReaderT @SqlBackend $ pure () }}} {{{ $ ghc --version The Glorious Glasgow Haskell Compilation System, version 8.2.2 $ ghc -- Bug.hs -O2 -fforce-recomp [1 of 1] Compiling Main ( Bug.hs, Bug.o ) Linking Bug.exe ... $ ./Bug.exe Bug.exe: SqlError {sqlState = "", sqlExecStatus = FatalError, sqlErrorMsg = "missing \"=\" after \"http://ghc.haskell.org/trac/ghc/ticket/13112#comment:22 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13112: Windows 64-bit GHC HEAD segfaults on the code with a lot of TH stuff. ---------------------------------+-------------------------------------- Reporter: awson | Owner: (none) Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by RyanGlScott): OK. I'm completely unfamiliar with `persistent-postgresql`—can you walk me through the exact steps I'd need to take to set up such a connection string? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13112#comment:24 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13112: Windows 64-bit GHC HEAD segfaults on the code with a lot of TH stuff. ---------------------------------+-------------------------------------- Reporter: awson | Owner: (none) Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by RyanGlScott): As far as the original ticket goes, I'm stumped. I tried to construct a minimal Template Haskell program which should eat up memory by causing a space leak: {{{#!hs {-# LANGUAGE TemplateHaskell #-} module Bug where $(let f :: Int -> a f x = f (x + 1) in f 0) }}} I tried compiling this on GHC 8.2.2 with and without `-O2`, but despite it using all of my 8GB of memory available, this didn't trigger a segfault. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13112#comment:26 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13112: Windows 64-bit GHC HEAD segfaults on the code with a lot of TH stuff. ---------------------------------+-------------------------------------- Reporter: awson | Owner: (none) Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by RyanGlScott): I think you misinterpreted my comment. I'm //deliberately// writing a program with a space leak, in hopes of triggering the segfault that was originally reported. (I wasn't successful, but that's beside the point.) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13112#comment:28 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13112: Windows 64-bit GHC HEAD segfaults on the code with a lot of TH stuff. ---------------------------------+-------------------------------------- Reporter: awson | Owner: (none) Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by jbransen): With GHC 8.4.3 I am also experiencing segfaults on my 64-bit Windows 10 machine, and I'd like to add some observations that can hopefully help pinpoint the exact problem: - I am working on a relatively big package with many dependencies, amongst which is `persistent`, and the segfault happens when compiling the first module containing TH code calling persistent generation. - With GHC 8.2 on the same codebase, the segfaults seemed to appear only when compiling a module with TH for the first time. Building again then worked. With GHC 8.4 the segfault is consistent. - Using `-fexternal-interpreter` solves the problem, but since that is not compatible with intero, it breaks my editor integration and does not help me so much. I've also tried haskell-ide-backend which segfaults in the same way... - I am using Stack, and `stack build` fails with a segfault, but `stack repl` (so using ghci) does work! - I tried to create a minimal example, but I can't. I started to strip down as much as possible, and I now have a strange project which does consistently generate a segfault. However, when I remove some unused (!) import, or some unused (!) dependency from the .cabal file, it does compile. It does not seem to matter so much what I remove, so it really seems like it is related to the amount of dependencies that are in scope. My guess is that somehow it is related to how much is loaded into memory, and that after some threshold we end up in large address space. Hence, I have the feeling there is no "minimal example" triggering this bug... -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13112#comment:30 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13112: Windows 64-bit GHC HEAD segfaults on the code with a lot of TH stuff. ---------------------------------+-------------------------------------- Reporter: awson | Owner: (none) Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by Phyx-): @jbransen, yeah we pretty much know that this happens because TH is exceeding the memory model. That's a sort of design issue with TH. But the issue is made worse on Windows because of the extreme waste the runtime linker has when allocating memory. I have a patch that addresses this but I haven't gotten reviews yet. Hopefully it'll be in 8.6 which should make this a bit harder to trigger. I also have further linker changes planned to make this more rare too. But we'll see if I get the time to finish them for 8.6 (and get them reviewed which will likely be the sticking point) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13112#comment:32 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler