[GHC] #9058: System.IO.openTempFile does not scale
#9058: System.IO.openTempFile does not scale ------------------------------------+------------------------------------- Reporter: slyfox | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.8.2 Keywords: | Operating System: Unknown/Multiple Architecture: Unknown/Multiple | Type of failure: None/Unknown Difficulty: Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | ------------------------------------+------------------------------------- In search of a bug in darcs http://bugs.darcs.net/issue2364 i've notice very bad property of openTempFile: it's pattern is very predictable and has O(n^2) of already created temp files. Predictability allows very fun bugs survive in buggy programs, like: {{{ thread1: (fn, fh) <- openTempFile "." "hello" renameFile fn "something" -- some time after when (some_rare_buggy_condition) $ -- oops, reused temp name, but too late, other thread killed it writeFileFile fn thread2: (fn, fh) <- openTempFile "." "hello" workWithFn fn -- nobody should touch it, right? }}} It's _very_ hard to debug data corruption when all temp files are named "foo${pid}" and sometimes "foo${pid+1}". And more serious bug: the more threads you have trying to create similar temps performance drops significantly: Attached program shows the following numbers: {{{ $ time ./bench-temps same 2000 real 0m2.795s user 0m1.516s sys 0m1.190s $ time ./bench-temps diff 2000 real 0m0.161s user 0m0.043s sys 0m0.115s }}} It's O(N^2) growing open() storm. https://github.com/ghc/ghc/blob/master/libraries/base/System/IO.hs#L465 {{{ FileExists -> findTempName (x + 1) }}} This is the source of the problem. I'd suggest always using random name for it. For portability reasons I suggest adding at least insecure random '''rand()''' value from C library. That way we will succeed in opening temp file at the first attempt. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9058 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9058: System.IO.openTempFile does not scale -------------------------------------+------------------------------------ Reporter: slyfox | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.8.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: None/Unknown | Difficulty: Unknown Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by slyfox): Here is one simple solution. it basically changes prefix ++ getpid() ++ suffix for prefix ++ rand() ++ rand() ++ suffix Technically rand() might be thread-unsafe (but not on glibc, there random_r() is used silently) but i don't think it's a troblem at all. Otherwise we could use time-based timestamp, but picking precise time source (to use it more, than 1000 times a second) might be a problem. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9058#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9058: System.IO.openTempFile does not scale -------------------------------------+------------------------------------ Reporter: slyfox | Owner: Type: bug | Status: patch Priority: normal | Milestone: Component: Compiler | Version: 7.8.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: None/Unknown | Difficulty: Unknown Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Changes (by slyfox): * status: new => patch -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9058#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9058: System.IO.openTempFile does not scale -------------------------------------+------------------------------------- Reporter: slyfox | Owner: Type: bug | Status: closed Priority: normal | Milestone: 7.10.1 Component: Compiler | Version: 7.8.2 Resolution: fixed | Keywords: Operating System: | Architecture: Unknown/Multiple Unknown/Multiple | Difficulty: Unknown Type of failure: | Blocked By: None/Unknown | Related Tickets: Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Changes (by thoughtpolice): * status: patch => closed * resolution: => fixed * milestone: => 7.10.1 Comment: Merged, thanks! -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9058#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9058: System.IO.openTempFile does not scale -------------------------------------+------------------------------------- Reporter: slyfox | Owner: Type: bug | Status: closed Priority: normal | Milestone: 7.10.1 Component: Compiler | Version: 7.8.2 Resolution: fixed | Keywords: Operating System: | Architecture: Unknown/Multiple Unknown/Multiple | Difficulty: Unknown Type of failure: | Blocked By: None/Unknown | Related Tickets: Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Comment (by hvr): For the record, here's the commit that failed to use the proper ticket- reference syntax: f510c7cac5b2e9afe0ebde2766a671c59137f3cc -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9058#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9058: System.IO.openTempFile does not scale -------------------------------------+------------------------------------- Reporter: slyfox | Owner: Type: bug | Status: new Priority: normal | Milestone: 7.10.1 Component: Compiler | Version: 7.8.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: None/Unknown | Unknown/Multiple Blocked By: | Test Case: Related Tickets: | Blocking: | Differential Revisions: -------------------------------------+------------------------------------- Changes (by rwbarton): * status: closed => new * resolution: fixed => Comment: This does now mean that the sequence of filenames is completely fixed for a given program (since we don't seed the random number generator) which has some obvious downsides. It would be nice to just use `mkstemps`, but maybe that is not portable enough? I wonder what other languages' standard libraries do here? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9058#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9058: System.IO.openTempFile does not scale -------------------------------------+------------------------------------- Reporter: slyfox | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.1 Component: Compiler | Version: 7.8.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Changes (by thoughtpolice): * milestone: 7.10.1 => 8.0.1 Comment: Moving to 8.0.1 (since the semantics aren't going to change in a minor point release). -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9058#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9058: System.IO.openTempFile does not scale -------------------------------------+------------------------------------- Reporter: slyfox | Owner: Type: bug | Status: closed Priority: normal | Milestone: 8.0.1 Component: Compiler | Version: 7.8.2 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #10940 | Differential Rev(s): -------------------------------------+------------------------------------- Changes (by thomie): * status: new => closed * resolution: => fixed * related: => #10940 Comment: Closing this in favor of #10940, which has an example showing the problem mentined in comment:5. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9058#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
participants (1)
-
GHC