All,<br><br>Referencing this thread (<a href="http://www.haskell.org/pipermail/haskell-cafe/2007-May/025769.html">http://www.haskell.org/pipermail/haskell-cafe/2007-May/025769.html</a>), I have prepared a small update to robots.txt
. It doesn&#39;t do much, but should exclude all dynamic pages from being spidered. Please vet it and let me know of any concerns. Note I&#39;m not really interested in reconfiguring the wiki to have nicer URLs - what is there is what I&#39;m working with right now.
<br><br>Here&#39;s the updated robots.txt (availabe as <a href="http://www.haskell.org/robots2.txt">http://www.haskell.org/robots2.txt</a> right now):<br><br>&nbsp; User-agent: *<br>&nbsp; Disallow: /haskellwiki/Special:<br>&nbsp; Disallow: /haskellwiki/Special%3A
<br>&nbsp; Disallow: /haskellwiki/?<br>&nbsp; Disallow: /haskellwiki/%3f<br><br>Any feedback, let me know. Thanks!<br><br>Justin<br>