darcs patch: XMonad.Core: rw recompilation

1 patch for repository http://code.haskell.org/xmonad: Sat Apr 10 13:45:52 EDT 2010 gwern0@gmail.com * XMonad.Core: rw recompilation This diverts the intermediate *.o *.hi files to /tmp, so they don't clutter up ~/.xmonad/

On Sat, Apr 10, 2010 at 1:50 PM,
1 patch for repository http://code.haskell.org/xmonad:
Sat Apr 10 13:45:52 EDT 2010 gwern0@gmail.com * XMonad.Core: rw recompilation This diverts the intermediate *.o *.hi files to /tmp, so they don't clutter up ~/.xmonad/
Ping. -- gwern

gwern0:
On Sat, Apr 10, 2010 at 1:50 PM,
wrote: 1 patch for repository http://code.haskell.org/xmonad:
Sat Apr 10 13:45:52 EDT 2010 gwern0@gmail.com * XMonad.Core: rw recompilation This diverts the intermediate *.o *.hi files to /tmp, so they don't clutter up ~/.xmonad/
Ping.
Rather than send an email for each patch that has been submitted but not applied, which risks spamming and annoying people, I would suggest a more constructive mechanism: write a single status report email with all the ones that you think are ready to go, and do that maybe once a month. This is a worthy task, and I'm glad you're taking it on, but adding more email to my inbox doesn't help me get to the work. -- Don

On Sat, May 1, 2010 at 2:58 PM, Don Stewart
gwern0:
On Sat, Apr 10, 2010 at 1:50 PM,
wrote: 1 patch for repository http://code.haskell.org/xmonad:
Sat Apr 10 13:45:52 EDT 2010 gwern0@gmail.com * XMonad.Core: rw recompilation This diverts the intermediate *.o *.hi files to /tmp, so they don't clutter up ~/.xmonad/
Ping.
Rather than send an email for each patch that has been submitted but not applied, which risks spamming and annoying people, I would suggest a more constructive mechanism: write a single status report email with all the ones that you think are ready to go, and do that maybe once a month.
This is a worthy task, and I'm glad you're taking it on, but adding more email to my inbox doesn't help me get to the work.
-- Don
I did try that for a year or so. It got minimal results - persistent backlogs, few comments or replies. So, I switched to individual patches, which also let me set ultimatums for XMC; that has seemed to work much better. (I would like to run the numbers, but I don't know any way to ask Darcs for when a patch was applied to a given repo, as opposed to when a patch was recorded; with the former, I could check on things like 'did the average time before a patch was dealt with definitively shrink or expand when gwern switched email styles?') As I've suggested before: if you and Spencer are so overloaded as to be unable to review modest patches given half a year, why not give a third person the core commit bit? I wouldn't trust myself with the core, but is there no one else? Joachim Breitner, for example, since he's already patching Xmonad-core under his Debian maintainer hat. -- gwern

Hi, Am Samstag, den 01.05.2010, 16:07 -0400 schrieb Gwern Branwen:
(I would like to run the numbers, but I don't know any way to ask Darcs for when a patch was applied to a given repo, as opposed to when a patch was recorded; with the former, I could check on things like 'did the average time before a patch was dealt with definitively shrink or expand when gwern switched email styles?')
At least for patches tracked by DarcsWatch and applied after the Vienna Darcs hacking sprint (Nov 2009), this information is stored and present on http://darcswatch.nomeata.de/repo_http:__code.haskell.org_xmonad.html#bundle...
Joachim Breitner, for example, since he's already patching Xmonad-core under his Debian maintainer hat.
Thanks for the kudos, but I am not up-to-date with xmonad’s progress in Darcs and wouldn’t have much time to spend on it either. Greetings, Joachim -- Joachim Breitner e-Mail: mail@joachim-breitner.de Homepage: http://www.joachim-breitner.de ICQ#: 74513189 Jabber-ID: nomeata@joachim-breitner.de

On Sat, May 1, 2010 at 4:07 PM, Gwern Branwen
(I would like to run the numbers, but I don't know any way to ask Darcs for when a patch was applied to a given repo, as opposed to when a patch was recorded; with the former, I could check on things like 'did the average time before a patch was dealt with definitively shrink or expand when gwern switched email styles?')
As I've suggested before: if you and Spencer are so overloaded as to be unable to review modest patches given half a year, why not give a third person the core commit bit? I wouldn't trust myself with the core, but is there no one else? Joachim Breitner, for example, since he's already patching Xmonad-core under his Debian maintainer hat.
Here is a graph to compare the kinds of delays core and contrib patches take before being applied: http://code.haskell.org/~aavogt/darcsVersions/applyDelays.svg http://code.haskell.org/~aavogt/darcsVersions/applyDelays.png (much smaller file) Note that you need a copy of the relevant repos that preserves the mtime of the patches, which darcs get doesn't preserve, in order to generate that graph with: http://code.haskell.org/~aavogt/darcsVersions/cmpDates.hs Going from what's in the repos, core and contrib situations are not much different. -- Adam

vogt.adam:
On Sat, May 1, 2010 at 4:07 PM, Gwern Branwen
wrote: (I would like to run the numbers, but I don't know any way to ask Darcs for when a patch was applied to a given repo, as opposed to when a patch was recorded; with the former, I could check on things like 'did the average time before a patch was dealt with definitively shrink or expand when gwern switched email styles?')
As I've suggested before: if you and Spencer are so overloaded as to be unable to review modest patches given half a year, why not give a third person the core commit bit? I wouldn't trust myself with the core, but is there no one else? Joachim Breitner, for example, since he's already patching Xmonad-core under his Debian maintainer hat.
Here is a graph to compare the kinds of delays core and contrib patches take before being applied:
http://code.haskell.org/~aavogt/darcsVersions/applyDelays.svg http://code.haskell.org/~aavogt/darcsVersions/applyDelays.png (much smaller file)
Note that you need a copy of the relevant repos that preserves the mtime of the patches, which darcs get doesn't preserve, in order to generate that graph with: http://code.haskell.org/~aavogt/darcsVersions/cmpDates.hs
Going from what's in the repos, core and contrib situations are not much different.
Beautiful data.

On Sat, May 1, 2010 at 10:45 PM, adam vogt
On Sat, May 1, 2010 at 4:07 PM, Gwern Branwen
wrote: (I would like to run the numbers, but I don't know any way to ask Darcs for when a patch was applied to a given repo, as opposed to when a patch was recorded; with the former, I could check on things like 'did the average time before a patch was dealt with definitively shrink or expand when gwern switched email styles?')
As I've suggested before: if you and Spencer are so overloaded as to be unable to review modest patches given half a year, why not give a third person the core commit bit? I wouldn't trust myself with the core, but is there no one else? Joachim Breitner, for example, since he's already patching Xmonad-core under his Debian maintainer hat.
Here is a graph to compare the kinds of delays core and contrib patches take before being applied:
http://code.haskell.org/~aavogt/darcsVersions/applyDelays.svg http://code.haskell.org/~aavogt/darcsVersions/applyDelays.png (much smaller file)
Note that you need a copy of the relevant repos that preserves the mtime of the patches, which darcs get doesn't preserve, in order to generate that graph with: http://code.haskell.org/~aavogt/darcsVersions/cmpDates.hs
Going from what's in the repos, core and contrib situations are not much different.
Well, I would point out that right now, XMC has only one patch outstanding, from 2 or 3 months ago. So a 'complete' chart would show one more red point just under the lowest dotted line - not a significant difference. But XMonad has ~10 patches outstanding: http://darcswatch.nomeata.de/repo_http:__code.haskell.org_xmonad.html 5 patches are from 2009; so that means 5 blue dots are at least 125 days old. The oldest 2 are from July 2009, so that means they are ~339 days old - the second box from the top, older than the oldest patch - which is also an XMonad patch BTW. Toss in the other 3 2009 patches in the 200-300 box, and then the remaining 5 somewhere in the lower 2 boxes. If those were plotted, the charts would not look so comparable. -- gwern

On Sat, Apr 10, 2010 at 1:50 PM,
1 patch for repository http://code.haskell.org/xmonad:
Sat Apr 10 13:45:52 EDT 2010 gwern0@gmail.com * XMonad.Core: rw recompilation This diverts the intermediate *.o *.hi files to /tmp, so they don't clutter up ~/.xmonad/
Actually, I'm rethinking this one. dons suggests (http://stackoverflow.com/questions/1411089/how-to-stop-ghc-from-generating-i...) that one could specify as the redirect target /dev/null, and looking at the GHC manual again (http://www.haskell.org/ghc/docs/6.12.2/html/users_guide/separate-compilation...), I see that there's a shortcut for redirecting all the intermediates: "-outputdir dir The -outputdir option is shorthand for the combination of -odir, -hidir, and -stubdir." So if we used "-outputdir /dev/null", we don't have to worry about cluttering up /tmp (as I think someone complained about in connection to mueval), we don't have to do a call to getTempDir (saving a line & import), and the compilation line itself increase only minimally. I think /dev/null exists on any system Xmonad is likely to run on, and so we can hardcode it. Thoughts? -- gwern

On May 7, 2010, at 17:00 , Gwern Branwen wrote:
"-outputdir dir
The -outputdir option is shorthand for the combination of -odir, -hidir, and -stubdir."
So if we used "-outputdir /dev/null", we don't have to worry about
I expect you'll get a "Not a directory" diagnostic from GHC about that one, unless GHC itself special-cases /dev/null. -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH

On Fri, May 7, 2010 at 11:42 PM, Brandon S. Allbery KF8NH
On May 7, 2010, at 17:00 , Gwern Branwen wrote:
"-outputdir dir
The -outputdir option is shorthand for the combination of -odir, -hidir, and -stubdir."
So if we used "-outputdir /dev/null", we don't have to worry about
I expect you'll get a "Not a directory" diagnostic from GHC about that one, unless GHC itself special-cases /dev/null.
Bleh, unfortunately it doesn't seem to. '/dev/null/Main.o: getModificationTime: inappropriate type (Not a directory)' Do you know of any similar directories we could use in place of /dev/null? -- gwern

On May 8, 2010, at 08:49 , Gwern Branwen wrote:
On Fri, May 7, 2010 at 11:42 PM, Brandon S. Allbery KF8NH
wrote: On May 7, 2010, at 17:00 , Gwern Branwen wrote:
"-outputdir dir
The -outputdir option is shorthand for the combination of -odir, -hidir, and -stubdir."
So if we used "-outputdir /dev/null", we don't have to worry about
I expect you'll get a "Not a directory" diagnostic from GHC about that one, unless GHC itself special-cases /dev/null.
Bleh, unfortunately it doesn't seem to.
'/dev/null/Main.o: getModificationTime: inappropriate type (Not a directory)'
Do you know of any similar directories we could use in place of /dev/ null?
No; and in any case, I don't think you could get away with it because ghc is using those .hi and .o files internally (including passing the .o files to ld), so they have to go *somewhere*. And Unix doesn't have the notion of a temporary directory that goes away when the creating process exits, as it has for files (the open-and-unlink idiom); and there are problems with providing one. I think what you really want is for ghc to have a treat-intermediate- files-in-this-session-as-ephemeral flag, such that the .hi and .o files created during a ghc invocation are removed after the link step. -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH

On Sat, May 8, 2010 at 3:42 PM, Brandon S. Allbery KF8NH
On May 8, 2010, at 08:49 , Gwern Branwen wrote:
On Fri, May 7, 2010 at 11:42 PM, Brandon S. Allbery KF8NH
wrote: On May 7, 2010, at 17:00 , Gwern Branwen wrote:
"-outputdir dir
The -outputdir option is shorthand for the combination of -odir, -hidir, and -stubdir."
So if we used "-outputdir /dev/null", we don't have to worry about
I expect you'll get a "Not a directory" diagnostic from GHC about that one, unless GHC itself special-cases /dev/null.
Bleh, unfortunately it doesn't seem to.
'/dev/null/Main.o: getModificationTime: inappropriate type (Not a directory)'
Do you know of any similar directories we could use in place of /dev/null?
No; and in any case, I don't think you could get away with it because ghc is using those .hi and .o files internally (including passing the .o files to ld), so they have to go *somewhere*. And Unix doesn't have the notion of a temporary directory that goes away when the creating process exits, as it has for files (the open-and-unlink idiom); and there are problems with providing one.
I think what you really want is for ghc to have a treat-intermediate-files-in-this-session-as-ephemeral flag, such that the .hi and .o files created during a ghc invocation are removed after the link step.
Yes, I'm giving up on this one and punting it to GHC HQ. There just doesn't seem to be any satisfactory way for us to do it, short of shelling out to 'find'. The bug report is http://hackage.haskell.org/trac/ghc/ticket/4114 if anyone wants to cc themselves (remember, CCs are like votes! except they count even less). -- gwern

On Wed, Jun 02, 2010 at 05:40:50PM -0400, Gwern Branwen wrote:
On Sat, May 8, 2010 at 3:42 PM, Brandon S. Allbery KF8NH
wrote: No; and in any case, I don't think you could get away with it because ghc is using those .hi and .o files internally (including passing the .o files to ld), so they have to go *somewhere*. And Unix doesn't have the notion of a temporary directory that goes away when the creating process exits, as it has for files (the open-and-unlink idiom); and there are problems with providing one.
I think what you really want is for ghc to have a treat-intermediate-files-in-this-session-as-ephemeral flag, such that the .hi and .o files created during a ghc invocation are removed after the link step.
Yes, I'm giving up on this one and punting it to GHC HQ. There just doesn't seem to be any satisfactory way for us to do it, short of shelling out to 'find'.
The bug report is http://hackage.haskell.org/trac/ghc/ticket/4114 if anyone wants to cc themselves (remember, CCs are like votes! except they count even less).
Another stab at this: Indeed putting stuff into /tmp can be a security risk. I think one solution is to use something like mkdtemp to create a temporary directory in a secure way and pass that to GHC. But according to this thread http://www.mail-archive.com/darcs-devel@darcs.net/msg03101.html even mkdtemp can be a problem in combination with tmp cleaners. On top of that, there doesn't seem to be an easily available mkdtemp implementation for Haskell. Maybe Unixutils on Hackage would fit the bill, but I guess we don't really want another package just for that. Because of all the security headache, it seems to me that most people just give up on /tmp and instead put stuff into directories somewhere below the user's home directory. So my suggestion: Redirect the intermediate files to ~/.xmonad/.ghc_temporary_outputdir and just delete that directory afterwards. This achieves: * less ways for GHC to break (after a GHC upgrade), Joachim's initial reason for the patch * less clutter in ~/.xmonad, as mentioned before as well * should work for modular configs too * has non of the /tmp security concerns Patch is attached! :-) Comments? Regards, Jan

On Fri, Jun 11, 2010 at 7:42 PM, Jan Vornberger
Another stab at this: Indeed putting stuff into /tmp can be a security risk. I think one solution is to use something like mkdtemp to create a temporary directory in a secure way and pass that to GHC.
But according to this thread http://www.mail-archive.com/darcs-devel@darcs.net/msg03101.html even mkdtemp can be a problem in combination with tmp cleaners. On top of that, there doesn't seem to be an easily available mkdtemp implementation for Haskell. Maybe Unixutils on Hackage would fit the bill, but I guess we don't really want another package just for that.
Because of all the security headache, it seems to me that most people just give up on /tmp and instead put stuff into directories somewhere below the user's home directory.
So my suggestion: Redirect the intermediate files to ~/.xmonad/.ghc_temporary_outputdir and just delete that directory afterwards.
This achieves: * less ways for GHC to break (after a GHC upgrade), Joachim's initial reason for the patch * less clutter in ~/.xmonad, as mentioned before as well * should work for modular configs too * has non of the /tmp security concerns
Patch is attached! :-) Comments?
Regards, Jan
Well, that does look like it would work. (Didn't know we *had* a rm -rf in the libraries.) It's something of a hack to make our own temporary directory, but I doubt anyone will ever create such a dot-dir deliberately. If there really are no other downsides, then I guess this is worth applying. -- gwern
participants (7)
-
adam vogt
-
Brandon S. Allbery KF8NH
-
Don Stewart
-
Gwern Branwen
-
gwern0@gmail.com
-
Jan Vornberger
-
Joachim Breitner