I notice that getDirectoryContents appears to return its results in alphabetical order. Is this behaviour actually guaranteed? Related: Is there a way to get rid of "." and ".." in the results? (Obviously this causes directory recusion to malfunction.) I can of course manually filter them out, but it's annoying. Finally, why do doesDirectoryExist "C:" doesDirectoryExist "C:\" both yield False? (This frequently trips up programs that try to check that a destination folder exists.)
All these questions are actually Windows-centric; the answers are different on Unix. On Oct 12, 2007, at 13:21 , Andrew Coppin wrote:
I notice that getDirectoryContents appears to return its results in alphabetical order. Is this behaviour actually guaranteed?
There is no guarantee, however on Windows this is done by invoking Win32 directory globbing routines which happen to return things in alpha order. (On Unix it might be alpha order if the underlying filesystem is a btree, but there are no guarantees at all except perhaps that '.' and '..' come first.)
Related: Is there a way to get rid of "." and ".." in the results? (Obviously this causes directory recusion to malfunction.) I can of course manually filter them out, but it's annoying.
Manual filtering is always required, whether C, Perl, Haskell, etc. I dunno, maybe python filters them for you or something. Every language I've ever used makes you deal.
doesDirectoryExist "C:" doesDirectoryExist "C:\"
both yield False? (This frequently trips up programs that try to check that a destination folder exists.)
Ask Microsoft. Drive top level handling has been weird since MS-DOS 2.0. -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH
Andrew Coppin wrote:
Is there a way to get rid of "." and ".." in the results?
Brandon S. Allbery wrote:
Manual filtering is always required, whether C, Perl, Haskell, etc. I dunno, maybe python filters them for you or something.
Correct, Python filters them out. This is clearly the correct behavior. That is what is needed in the vast majority of cases, and it is still reasonably easy to deal with the unusual cases. It is too bad that Haskell is among the many languages that get this wrong. Python also has os.walk, a very convenient functional (sort of) tool for recursing through directories. (It sounds trivial, but it is not, there are enough annoying details that this function saves huge amounts of time.) Very embarrassing that Haskell is missing this. How about a built-in function that represents a directory tree as a lazy Data.Tree? -Yitz
On Sat, 13 Oct 2007, Yitzchak Gale wrote:
Andrew Coppin wrote:
Is there a way to get rid of "." and ".." in the results?
Brandon S. Allbery wrote:
Manual filtering is always required, whether C, Perl, Haskell, etc. I dunno, maybe python filters them for you or something.
Correct, Python filters them out. This is clearly the correct behavior. That is what is needed in the vast majority of cases, and it is still reasonably easy to deal with the unusual cases.
It is too bad that Haskell is among the many languages that get this wrong.
me too
Python also has os.walk, a very convenient functional (sort of) tool for recursing through directories. (It sounds trivial, but it is not, there are enough annoying details that this function saves huge amounts of time.) Very embarrassing that Haskell is missing this.
Maybe it is already in one of the "Haskell for scripting" packages?
On Sat, Oct 13, 2007 at 23:27:13 +0200, Yitzchak Gale wrote:
Andrew Coppin wrote:
Is there a way to get rid of "." and ".." in the results?
Brandon S. Allbery wrote:
Manual filtering is always required, whether C, Perl, Haskell, etc. I dunno, maybe python filters them for you or something.
Correct, Python filters them out. This is clearly the correct behavior. That is what is needed in the vast majority of cases, and it is still reasonably easy to deal with the unusual cases.
It is too bad that Haskell is among the many languages that get this wrong.
Python also has os.walk, a very convenient functional (sort of) tool for recursing through directories. (It sounds trivial, but it is not, there are enough annoying details that this function saves huge amounts of time.) Very embarrassing that Haskell is missing this.
How about a built-in function that represents a directory tree as a lazy Data.Tree?
http://therning.org/magnus/index.php?tag=haskell&paged=3 Not really what you're looking for, but hopefully it's a good place to start. /M -- Magnus Therning (OpenPGP: 0xAB4DFBA4) magnus＠therning．org Jabber: magnus．therning＠gmail．com http://therning.org/magnus
Yitzchak Gale wrote:
Python also has os.walk, a very convenient functional (sort of) tool for recursing through directories. (It sounds trivial, but it is not, there are enough annoying details that this function saves huge amounts of time.) Very embarrassing that Haskell is missing this.
See System.FilePath.Find in http://hackage.haskell.org/cgi-bin/hackage-scripts/package/FileManip-0.2
How about a built-in function that represents a directory tree as a lazy Data.Tree?
Not a very good idea. Representing a directory structure as a tree
makes people think they can manipulate it as if it were a tree, which
leads to all kinds of nasty bugs when the real world bleeds through the
holes in the abstraction.
I wrote:
...a tool for recursing through directories... How about a built-in function that represents a directory tree as a lazy Data.Tree?
Bryan O'Sullivan wrote:
See System.FilePath.Find in http://hackage.haskell.org/cgi-bin/hackage-scripts/package/FileManip-0.2
Not a very good idea. Representing a directory structure as a tree makes people think they can manipulate it as if it were a tree, which leads to all kinds of nasty bugs when the real world bleeds through the holes in the abstraction.
You are right. Well, I didn't say what I meant by a "lazy Data.Tree" :). Your library is very nice. But - it suffers from the same problem. You use unsafe IO operations to build a "lazy" IO list, and we all know what grief that can lead to. Especially in this application, using that type of "laziness" is really unsafe. Any user must be explicitly aware of potential side effect complications, such as directory structure shifting under your feet as you work, possibly induced by your own traversal if you are not careful. (In my opinion, the documention for Python's os.walk http://docs.python.org/lib/os-file-dir.html#l2h-2715 gives a nice, clear description of what must be avoided during such a traversal.) So I think that what is needed here is something like the following. Based on my experience, this would be a really convenient API: data TraversalDirection = TopDown | BottomUp data Directory = Directory { dirPath :: FilePath, fileNames :: [FilePath], subdirNames :: [FilePath]} -- List all directories in the tree rooted at the given path traverseDirectories :: MonadIO m => TraversalDirection -> FilePath -> ListT m Directory -- List all non-directory files in the tree rooted at the given path traverseFiles :: MonadIO m => TraversalDirection -> FilePath -> ListT m FilePath traverseFiles = join . fmap (liftList . fileNames) . traverseDirectories -- List the paths to all subdirectories in the tree rooted at the given path traverseSubdirs :: MonadIO m => TraversalDirection -> FilePath -> ListT m FilePath traverseSubdirs = join . fmap (liftList . subdirNames) . traverseDirectories Here it is critical that ListT be taken from http://haskell.org/haskellwiki/ListT_done_right and not the broken implementation that comes with mtl. (Finally fixed in 6.8? Please?) You could plug the above into your machinery for "recursion predicates" and all the other nice stuff. -Yitz
I wrote:
...a tool for recursing through directories... How about a built-in function that represents a directory tree as a lazy Data.Tree?
Bryan O'Sullivan wrote:
See System.FilePath.Find in http://hackage.haskell.org/cgi-bin/hackage-scripts/package/FileManip-0.2
-- List all directories in the tree rooted at the given path traverseDirectories :: MonadIO m => TraversalDirection -> FilePath -> ListT m Directory
You could plug the above into your machinery for "recursion predicates" and all the other nice stuff.
Or - getting back to the "lazy Data.Tree" idea - we could define TreeT, analgous to ListT: newtype TreeT m a = NodeT (m (a, TreeT (ListT m) a)) and give it a nice Monad instance. Then you can prune and filter trees in a natural way, inside the monad. -Yitz
Yitzchak Gale wrote:
I wrote:
...a tool for recursing through directories... How about a built-in function that represents a directory tree as a lazy Data.Tree?
Bryan O'Sullivan wrote:
See System.FilePath.Find in http://hackage.haskell.org/cgi-bin/hackage-scripts/package/FileManip-0.2
-- List all directories in the tree rooted at the given path traverseDirectories :: MonadIO m => TraversalDirection -> FilePath -> ListT m Directory
You could plug the above into your machinery for "recursion predicates" and all the other nice stuff.
Or - getting back to the "lazy Data.Tree" idea - we could define TreeT, analgous to ListT:
newtype TreeT m a = NodeT (m (a, TreeT (ListT m) a))
and give it a nice Monad instance. Then you can prune and filter trees in a natural way, inside the monad.
Eh, isn't that what ListT is already for? I mean, type Directory = FilePath contents :: MonadIO m => Directory -> m [FilePath] liftList :: Monad m => m [a] -> ListT m a runList :: Monad m => ListT m a -> m [a] allows you to prune the directory tree in whatever way you like it. Here's an example top-down traversal that lists all non-directories: allFiles :: MonadIO m => Directory -> m [FilePath] allFiles d = runList $ do f <- liftList $ contents d if isDirectory f then allFiles f else return f In other words, ListT is like one of those iterators I keep hearing from the Python/Java/.../imperative world (except that you can't suspend a traversal once started). Regards, apfelmus
Hi apfelmus, I wrote:
...a tool for recursing through directories... How about a built-in function that represents a directory tree as a lazy Data.Tree?
-- List all directories in the tree rooted at the given path traverseDirectories :: MonadIO m => TraversalDirection -> FilePath -> ListT m Directory
Or - getting back to the "lazy Data.Tree" idea - we could define TreeT, analgous to ListT: newtype TreeT m a = NodeT (m (a, TreeT (ListT m) a)) and give it a nice Monad instance. Then you can prune and filter trees in a natural way, inside the monad.
apfelmus wrote:
Eh, isn't that what ListT is already for? ...allows you to prune the directory tree in whatever way you like it. Here's an example top-down traversal that lists all non-directories:
allFiles :: MonadIO m => Directory -> m [FilePath] allFiles d = runList $ do f <- liftList $ contents d if isDirectory f then allFiles f else return f
In other words, ListT is like one of those iterators I keep hearing from the Python/Java/.../imperative world
Well, yes, you could also use it that way. And yes, years ago they used to do it that way in Python. Where every user has to build in the logic to discover the tree structure manually, as they iterate along. But it turns out that it is much more natural and convenient to be given the tree structure pre-built, and then just traverse it. You need some laziness to avoid visiting directories that you don't care about and might be costly to visit. Python currently uses generators to get the laziness. Bryan uses unsafe IO, together with his "recursion predicates and filter predicates". My first proposal was just a minor modification of Bryan's ideas. I presented the tree pre-traversed - i.e., flattened, like Bryan, and also used his predicates. But I used ListT to provide the laziness instead of unsafe IO. My second proposal is to present the entire tree inside a lazy monad transformer. While the internal plumbing is slightly more complex, I think it will provide the most natural and convenient interface for the end user (who needs not understand all that monad stuff). -Yitz
Hi Yitzchak, On Sun, Oct 14, 2007 at 11:33:38AM +0200, Yitzchak Gale wrote:
Here it is critical that ListT be taken from
http://haskell.org/haskellwiki/ListT_done_right
and not the broken implementation that comes with mtl. (Finally fixed in 6.8? Please?)
mtl is not part of GHC (although it is currently included in some distributions as an "extra lib"). If you want to propose that mtl switches to a different definition of ListT then please see http://www.haskell.org/haskellwiki/Library_submissions Thanks Ian
Yitzchak Gale wrote:
Your library is very nice. But - it suffers from the same problem. You use unsafe IO operations to build a "lazy" IO list, and we all know what grief that can lead to.
This is little different from the approach taken by Python's os.walk,
which lazily yields the contents of a directory tree as it traverses it.
I'm a little unclear on why one appears good in your eyes, while the
other is not, beyond perhaps the depth/breadth knob and differences in
amount of documentation. Maybe you could expand on that a bit?
Hi Bryan, You wrote:
This is little different from the approach taken by Python's os.walk, which lazily yields the contents of a directory tree as it traverses it. I'm a little unclear on why one appears good in your eyes, while the other is not, beyond perhaps the depth/breadth knob and differences in amount of documentation. Maybe you could expand on that a bit?
Well firstly, I never said that your approach is not good. I was very pleased to see your work on this, and I was inspired by it. I do think that it is much better to provide IO laziness using monad transformers (or whatever) rather than unsafe IO. The basic difference is that with unsafe IO, you relinquish control over the order in which things happen, and leave it to the whims of the compiler/runtime. Besides the fact that it can always lead to bugs (viz. the constant problems people have with readFile and getContents), in this case the exact order of the side effects happens to be a critical and delicate issue, so you really don't want to give up that control here. My second proposal introduces another deviation from your approach in that it presents a full tree view rather than a pre-flattened tree view. I think that is more natural and more flexible - just my own taste. Regards, Yitz
On Sun, 14 Oct 2007, Bryan O'Sullivan wrote:
Yitzchak Gale wrote:
I do think that it is much better to provide IO laziness using monad transformers (or whatever) rather than unsafe IO.
That's fair enough. I think it would be great if you were to turn your ideas into a library and provide a few examples of its use.
I'm also interested in a more structured approach to lazy reading of data. I used unsafeInterleaveIO several times (including readFile) and got several problems including too many open files. Problems, that are certainly solved easily if you have more control over the IO laziness.
Jules Bean wrote:
Yitzchak Gale wrote:
How about a built-in function that represents a directory tree as a lazy Data.Tree?
Please no.
The last thing haskell needs is more dangerous semantically broken non-referentially-transparent lazy IO structures.
Woah! Well, I understood *one* of those adjectives...
I wrote:
How about a built-in function that represents a directory tree as a lazy Data.Tree?
Jules Bean wrote:
Please no. The last thing haskell needs is more dangerous semantically broken non-referentially-transparent lazy IO structures.
Agreed. I would definitely not want it to be a dangerous semantically broken non-referentially-transparent lazy IO structure. -Yitz
On Fri, 12 Oct 2007 18:21:07 +0100, you wrote:
I notice that getDirectoryContents appears to return its results in alphabetical order. Is this behaviour actually guaranteed?
This is a Windows thing. All of the NT-based operating systems list files in alphabetical order by default. You see the same thing if you use the DIR command from a command-line prompt. Steve Schafer Fenestra Technologies Corp. http://www.fenestra.com/
Hello Andrew, Friday, October 12, 2007, 9:21:07 PM, you wrote:
I notice that getDirectoryContents appears to return its results in alphabetical order. Is this behaviour actually guaranteed?
on NTFS filesystem, files are stored in directory alphabetically sorted. on FAT disks the order may be arbitrary -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com
participants (11)
-
Andrew Coppin -
apfelmus -
Brandon S. Allbery KF8NH -
Bryan O'Sullivan -
Bulat Ziganshin -
Henning Thielemann -
Ian Lynagh -
Jules Bean -
Magnus Therning -
Steve Schafer -
Yitzchak Gale