[GHC] #14533: Make GHC more robust against PC crashes by using atomic writes
#14533: Make GHC more robust against PC crashes by using atomic writes -------------------------------------+------------------------------------- Reporter: nh2 | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.2.1 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: Compile-time Unknown/Multiple | crash or panic Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- Yesterday my Windows machine crashed when ghc was building, left me with a lot of corrupted .hi files (and probably .o files as well). That made me think if we should change ghc to write its output with atomic moves. Discussion on `#ghc`: {{{ bgamari: nh2: well, perhaps I'm not sure it's terribly common for compilers to take such precautions though and if we were to do it for interface files then presumably we would also want to do it for object files siarheit_: that would be very nice geekosaur: compilers in general are happy to write out incomplete/garbage object files bgamari: it seems that way nh2: bgamari: right, if we wanted to do it we should do it for all files ghc writes. Possible that other compilers can also write garbage, but maybe ghc can do better -- one less thing the developer has to think about Phyx-: most compilers also don't do the aggressive recompilation avoidance things we do either.. corrupt hi files wouldn't be an issue if the next time it would override them :) }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14533 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14533: Make GHC more robust against PC crashes by using atomic writes -------------------------------------+------------------------------------- Reporter: nh2 | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Build System | Version: 8.2.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash or panic | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by nh2): * component: Compiler => Build System -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14533#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14533: Make GHC more robust against PC crashes by using atomic writes -------------------------------------+------------------------------------- Reporter: nh2 | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Build System | Version: 8.2.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash or panic | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by svenpanne): I'm not sure if I understand "atomic writes" here correctly, but e.g. on POSIX systems there is not really much "atomic" regarding I/O. One of the few things is renaming a file, but even that is not atomic depending on the underlying file system (e.g. NFS and/or SMB IIRC). So I think the best a compiler can do is to write its file under a fresh name next to its destination, and rename it when it completed writing it. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14533#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14533: Make GHC more robust against PC crashes by using atomic writes -------------------------------------+------------------------------------- Reporter: nh2 | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Build System | Version: 8.2.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash or panic | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari):
So I think the best a compiler can do is to write its file under a fresh name next to its destination, and rename it when it completed writing it.
Indeed, I believe this is what nh2 was suggesting. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14533#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14533: Make GHC more robust against PC crashes by using atomic writes -------------------------------------+------------------------------------- Reporter: nh2 | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Build System | Version: 8.2.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash or panic | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by nh2): Yes, I had the rename method in mind. If the underlying FS can't do atomic renames, then all bets are off and we've done the best we could. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14533#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14533: Make GHC more robust against PC crashes by using atomic writes -------------------------------------+------------------------------------- Reporter: nh2 | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Build System | Version: 8.2.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash or panic | Test Case: Blocked By: | Blocking: Related Tickets: #14788 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by nh2): * related: => #14788 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14533#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14533: Make GHC more robust against PC crashes by using atomic writes -------------------------------------+------------------------------------- Reporter: nh2 | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.2.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash or panic | Test Case: Blocked By: | Blocking: Related Tickets: #14788 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by nh2): * component: Build System (make) => Compiler Comment: Uh, with `Build system` I meant the `ghc --make` compile driver, not the build system that builds GHC. Can we have a `Component:` Trac category for that? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14533#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14533: Make GHC more robust against PC crashes by using atomic writes -------------------------------------+------------------------------------- Reporter: nh2 | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.2.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash or panic | Test Case: Blocked By: | Blocking: Related Tickets: #14788 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by nh2): I've started a new initiative to rid the Haskell build ecosystem of persistent build errors due to truncated files: https://github.com/commercialhaskell/stack/issues/4559 ---- First PR for this in: https://gitlab.haskell.org/ghc/ghc/merge_requests/391 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14533#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14533: Make GHC more robust against PC crashes by using atomic writes
-------------------------------------+-------------------------------------
Reporter: nh2 | Owner: (none)
Type: bug | Status: new
Priority: normal | Milestone:
Component: Compiler | Version: 8.2.1
Resolution: | Keywords:
Operating System: Unknown/Multiple | Architecture:
Type of failure: Compile-time | Unknown/Multiple
crash or panic | Test Case:
Blocked By: | Blocking:
Related Tickets: #14788 | Differential Rev(s):
Wiki Page: |
-------------------------------------+-------------------------------------
Comment (by Marge Bot
#14533: Make GHC more robust against PC crashes by using atomic writes
-------------------------------------+-------------------------------------
Reporter: nh2 | Owner: (none)
Type: bug | Status: new
Priority: normal | Milestone:
Component: Compiler | Version: 8.2.1
Resolution: | Keywords:
Operating System: Unknown/Multiple | Architecture:
Type of failure: Compile-time | Unknown/Multiple
crash or panic | Test Case:
Blocked By: | Blocking:
Related Tickets: #14788 | Differential Rev(s):
Wiki Page: |
-------------------------------------+-------------------------------------
Comment (by Ben Gamari
#14533: Make GHC more robust against PC crashes by using atomic writes -------------------------------------+------------------------------------- Reporter: nh2 | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.2.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash or panic | Test Case: Blocked By: | Blocking: Related Tickets: #14788 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by nh2): This was reverted due to a (apparently nondeterministic) build failure on Windows in later builds even though my Winodws validate succeeded. The build that failed for bgamari is https://gitlab.haskell.org/ghc/ghc/-/jobs/37771 The only thing I can imagine could result in this is when in `withAtomicRename targetFile f`, the action `f` doesn't actually write the file. But why would the assembler ''sometimes'' not do that? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14533#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14533: Make GHC more robust against PC crashes by using atomic writes
-------------------------------------+-------------------------------------
Reporter: nh2 | Owner: (none)
Type: bug | Status: new
Priority: normal | Milestone:
Component: Compiler | Version: 8.2.1
Resolution: | Keywords:
Operating System: Unknown/Multiple | Architecture:
Type of failure: Compile-time | Unknown/Multiple
crash or panic | Test Case:
Blocked By: | Blocking:
Related Tickets: #14788 | Differential Rev(s):
Wiki Page: |
-------------------------------------+-------------------------------------
Comment (by Marge Bot
participants (1)
-
GHC