On Mon, Jul 17, 2006 at 03:07:51AM +0100, Neil Mitchell wrote:

Hi Brian,

...

I kind of expect that a Haskell library for file paths will use the type system to ensure some useful properties about the paths.

I am specificially concentrating on type FilePath = String, because that is how it is defined by Haskell 98. And consequently that's how it works with readFile/writeFile/appendFile etc.

Perhaps a far better solution to this would not be to hack these kind of guarantees in at the filepath level, but have a restricted IO monad that only lets you perform operations inside certain directories, or only lets you read/write files. I know that both House and Halfs use these techniques. Without too much effort Yhc (for example) could be modified to perform restricted IO operations (only on certain directories etc).

You seem to want to distinguish between relative, relative down only and absolute paths. By putting this in the filepath, and having different types for each, you pretty much guarantee that all standard functions will operate on all 3 types of path, so you don't gain any security that way, since mistakes will still slip through. How about adding something like "restrictFilePaths :: FilePath -> IO ()" which will restrict the area that can be played with to that beneath the given FilePath?

Darcs also does something similar (typeclasses for control of IO actions), and this is certainly the way to go. However, I also agree that type distinctions between paths would be nice. My preference has long been that the "FilePath" should be a class rather than a type. Then one could have single IO functions that accept restricted and unrestricted file paths, and other ones that accept only restricted file paths, so you could get compile-time checking that your "safe" chroot monad won't die at runtime. -- David Roundy

On 18/07/06, Stephane Bortzmeyer wrote:

For instance, many lazy (not in the Haskell meaning) programmers believe that the path is safe if it does not include .. but it is false (hint: ."."/foo/bar is a legal path on Unix).

I believe this does not cause trouble. If it is a shell expression, it will go one level up. However, when treated as a filesystem path alone it will stay beneath. After all, the filesystem does not interpret quotation marks. Regards, Piotr Kalinowski -- Intelligence is like a river: the deeper it is, the less noise it makes

On Mon, Jul 17, 2006 at 03:07:51AM +0100, Neil Mitchell wrote a message of 64 lines which said:

I am specificially concentrating on type FilePath = String, because that is how it is defined by Haskell 98.

And if someone wants to define a "new" and "better" FilePath type, I would prefer something more abstract, such as a list of Path components, with functions to serialize it as a String and to parse it from a String.

Neil Mitchell wrote:

...

And if someone wants to define a "new" and "better" FilePath type, I would prefer something more abstract, such as a list of Path components, with functions to serialize it as a String and to parse it from a String.

A list of path components is just not enough, I'm afraid. What about extensions? What about drives? If you want an abstract type it will probably need to be entirely abstract, rather than with some exposed structure.

Why not just delete Unix and Windows from the equation altogether, and define a simple Haskell file system with something like: newtype Path a = Path [a] newtype Filename a = Filename a data Origin a -- some abstract type deriving Eq -- this would be nice if it is possible to implement data IString a => FileSpecifier a = FileSpecifier !(Origin a) !(Path a) !(Filename a) instance IString ByteString.Char8 ... instance IString String ... Origins could be created by a factory appropriate to the underlying operating system (they would represent drives or volumes or mount points) - in any case a drive can't be mentioned in a program or the program wouldn't be portable! Athough even with a nice rational reconstruction the monstrously unfortunate fact remains that Windows is case insensitive (how impossibly moronic!!!) and Unix isn't so it is not possible to write code that will work the same for both OS's if one is required to use filenames that will look the same in other OS apps (ie the trick of encoding the complete Unicode char set in terms of legal filename chars is probably not acceptable). Anyway this is probably straying too far from what you are trying to do at the moment. Regards, Brian. -- Logic empowers us and Love gives us purpose. Yet still phantoms restless for eras long past, congealed in the present in unthought forms, strive mightily unseen to destroy us. http://www.metamilk.com

Bulat Ziganshin wrote:

Hello Brian,

Sunday, July 23, 2006, 1:20:36 AM, you wrote:

...

instance IString ByteString.Char8 ... instance IString String ...

i think that we should ask Donald Stewart who is patronized SoC project involving development of such type class. If he will say that such type class is not developed, i feel himself enough interested to start developing such class. i can add this module to ByteString lib, if there is no better variants

i propose something like this:

class ListLike ce e | ce->e instance ListLike [a] a instance ListLike Data.ByteString.ByteString Word8 instance ListLike Data.ByteString.Lazy.ByteString Word8 instance ListLike Data.ByteString.Char8.ByteString Char instance ListLike Data.ByteString.Lazy.Char8.ByteString Char

Hi Bulat - I've been thinking of a sequence class for the project I'm working on at the moment, something like: class ISeq seq_a a | seq_a -> a where -- (*) empty :: seq_a single :: a -> seq_a length :: seq_a -> Int append :: seq_a -> seq_a -> seq_a pushL :: a -> seq_a -> seq_a -- (**) pushR :: seq_a -> a -> seq_a at :: seq_a -> Int -> a atL :: seq_a -> a -- (***) atR :: seq_a -> a viewL :: seq_a -> ViewL -- plus lots of other ops toList :: seq_a -> [a] fromList :: [a] -> seq_a data ViewL seq_a a = EmptyL | PushL !a !seq_a -- this is strict so we don't pay an extra laziness penalty (meaningful names such as pushL, pushR etc inspired by the absolutely brilliant C++ STL library as opposed to odd lispy names like cons and snoc (do we really want to have to start reading lexemes backwards then apply a quirky historical reference to understand code?)) then the IString class would be something like: class IChar c where toCChar :: c -> CChar fromCChar :: CChar -> c class (IChar c, ISeq s c) => IString s where withCString :: MonadIO m => s -> (Ptr CChar -> IO a) -> m a withCStringLen :: MonadIO m => s -> (Ptr CChar -> Int -> IO a) -> m a withCAString :: MonadIO m => s -> (Ptr CChar -> IO a) -> m a withCAStringLen :: MonadIO m => s -> (Ptr CChar -> Int -> IO a) -> m a -- possibly also withCWString etc peekCString :: Ptr CChar -> s peekCAString :: Ptr CChar -> s ie the IString class deals with the complexity of marshalling character strings which may or may not be in Unicode. (*) I assume that the reason for putting the collection type first is because usually you want to map a collection of elements to a collection of some different element type rather than mapping between different collection types. (**) I think conventional names like "foldr" should be replaced by "foldR" so that camel case is followed consistently and so that confusing names like "reducer", which is a word by itself in English, cannot arise when "reduceR" was meant. (***) "atL" and "atR" are more visual than the conventional names "head" and "last". I think names should be chosen so that the syntactic differences between lexemes indicate similarity of meaning, thus it's clear that "atL" "atR" and "at" all do something similar, whereas "head", "last", and "index" are just 3 random English words whose perceived commonality depends on many years programming experience and is therefore vague and loose. Anyway - it's just a rough idea at the moment, Regards, Brian. -- Logic empowers us and Love gives us purpose. Yet still phantoms restless for eras long past, congealed in the present in unthought forms, strive mightily unseen to destroy us. http://www.metamilk.com

I've been writing a Stringable class for my SoC project. You can check out the code at http://darcs.haskell.org/SoC/fps-soc/Data/Stringable.hs. Spencer Janssen On 7/23/06, Bulat Ziganshin wrote:

Hello Brian,

Sunday, July 23, 2006, 1:20:36 AM, you wrote:

...

instance IString ByteString.Char8 ... instance IString String ...

i think that we should ask Donald Stewart who is patronized SoC project involving development of such type class. If he will say that such type class is not developed, i feel himself enough interested to start developing such class. i can add this module to ByteString lib, if there is no better variants

i propose something like this:

class ListLike ce e | ce->e instance ListLike [a] a instance ListLike Data.ByteString.ByteString Word8 instance ListLike Data.ByteString.Lazy.ByteString Word8 instance ListLike Data.ByteString.Char8.ByteString Char instance ListLike Data.ByteString.Lazy.Char8.ByteString Char

-- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Hi,

...

In this library proposal, there are a bunch of "xxxDrive" functions .. [remove them] I strongly agree about this.

I have decided you are right, on Windows getDrive x can be written simply as: getDrive x | isRelative x = "" | otherwise = head (getDirectories x) And given that people probably shouldn't be playing with drives anyway, if they do want to, they can do a bit more work. All the drive related functions and therefore removed from the interface. I have also added a canonicalPath function, support for spotting file\con as invalid and fixing it, support for \\?\ paths (if you don't know what they are, don't look it up, they are quite painful!) and a few very obscure corner cases which broke some of the properties. Anyone have another other thoughts or comments? Thanks Neil

Hi,

I haven't been following this discussion very closely, but this caught my eye. Has anyone pointed out yet that eliminating ".." in a FilePath isn't valid in the presence of symbolic links? I vaguely recall that this is why Krasimir's System.FilePath library doesn't include normalisation. Normalisation that just changes separators, removes redundant separators and removes "." components is probably ok. Ok, I'll remove this behaviour

I don't really see the point of fullPathWith. Isn't it just combine? It also applies normalise to the result, although I'm not sure thats a particularly useful thing to do. I'll think about this.

I think shortPath is on dodgy ground, given the difficulty with normalising FilePaths, and the fact that paths are not unique in the presence of symlinks. I think this is still a useful thing, but I'll think harder about it in relation to symlinks, and figure out what properties it should have.

The temporary file stuff is wrong - see System.IO.openTemporaryFile. The only way to reliably create a temporary file is to open it at the same time, otherwise there's a race condition. Looking at it again, I'm not sure that the temporary operations belong in this module. Ditto for the directory operations. It would be handy if the directory operations (particularly ensureDirectory) were somewhere else in the standard libraries though.

I have some other issues with naming, and the fact that the library mixes IO and non-IO functions. I think this is unavoidable, I tried to make as many functions as possible pure, but where the filesystem must be consulted they have to be IO based. For example canonicalPath probably belongs in this module, but has to be IO based.

We should avoid referring to $PATH as the "path", since we already have FilePath. Agreed, but I couldn't come up with a better name, if anyone has any suggestions.

Where you use the term Filename, I think it should probably be FilePath. I think we should consistently use a single term, preferably FilePath. I have tried to consistently use FilePath as a path to a file, and filename as the actual name once you get there. For example:

/usr/bin/ghc Has a FilePath of "/usr/bin/ghc" and a FileName of "ghc". That distinction seems more logical to me, but if everyone thinks otherwise I'm happy to change it. Thanks Neil

Neil Mitchell wrote:

...

The temporary file stuff is wrong - see System.IO.openTemporaryFile. The only way to reliably create a temporary file is to open it at the same time, otherwise there's a race condition.

Looking at it again, I'm not sure that the temporary operations belong in this module. Ditto for the directory operations. It would be handy if the directory operations (particularly ensureDirectory) were somewhere else in the standard libraries though.

It's already there - see System.Directory.createDirectoryIfMIssing.

...

I have some other issues with naming, and the fact that the library mixes IO and non-IO functions.

I think this is unavoidable, I tried to make as many functions as possible pure, but where the filesystem must be consulted they have to be IO based. For example canonicalPath probably belongs in this module, but has to be IO based.

There are very few IO functions left: the directory operations are already provided by System.Directory, we have openTemporaryFile in System.IO which subsumes the temporary file operations in here. fullPath is just getCurrentDirectory and combine - it could go in System.Directory as a variant of getCurrentDirectory, or just leave it out. If we have shortPath at all, perhaps it would be better named relativeToCurrentDIrectory? I'm not sure about canonicalPath. This interface seems to suffer from the same race conditions as the temporary file interface: the answer is immediately invalid, so you can't rely on it.

...

We should avoid referring to $PATH as the "path", since we already have FilePath.

Agreed, but I couldn't come up with a better name, if anyone has any suggestions.

"search path" seems the best option, I think.

...

Where you use the term Filename, I think it should probably be FilePath. I think we should consistently use a single term, preferably FilePath.

I have tried to consistently use FilePath as a path to a file, and filename as the actual name once you get there. For example:

/usr/bin/ghc

Has a FilePath of "/usr/bin/ghc" and a FileName of "ghc".

Then this seems inconsistent with the naming of splitFileName, which should be splitFilePath. Isn't joinFilename the same as combine? get/setFilename are ok, but I prefer to reserve get/set for statelike things. Perhaps instead: directoryOf :: FilePath -> String filenameOf :: FilePath -> String extensionOf :: FilePath -> String basenaneOf :: FilePath -> String replaceFilename = joinFilePath . directoryOf replaceDirectory = flip joinFilePath . filenameOf etc. (unfortunately this use of basename is inconsistent with the Unix command, perhaps there's a better name?). Cheers, Simon

On 7/26/06, Neil Mitchell wrote:

The main purpose of canoncialPath is to fix the case on Windows, so "c:\my documents\file.doc" becomes "C:\My Documents\file.doc" if that is the case correct version of the file. I think this function will not actually change the file with relation to the underying file system, so should be race free. (I will document more to make the operation clearer)

Hi Neil, It seems like your canoncialPath function is already in the base package. Look at System.Directory.canonicalizePath. I have added it when I was working on the FilePath module for Cabal. The FilePath abstraction was discussed a number of times and it seems that people prefer an ADT representation instead of plain String. I tend to agree. Maybe such ADT based library can be integrated with some new IO library like the Streams library. Cheers, Krasimir

Neil Mitchell wrote:

...

I'm not sure about canonicalPath. This interface seems to suffer from the same race conditions as the temporary file interface: the answer is immediately invalid, so you can't rely on it.

The main purpose of canoncialPath is to fix the case on Windows, so "c:\my documents\file.doc" becomes "C:\My Documents\file.doc" if that is the case correct version of the file. I think this function will not actually change the file with relation to the underying file system, so should be race free. (I will document more to make the operation clearer)

Right, as Krasimir pointed out we already have System.Directory.canonicalizePath. Incedentally, this version removes "/../" in a symlink-friendly way on Unix, because it uses realpath() underneath. It also works with relative pathnames, prepending the current directory.

...

Then this seems inconsistent with the naming of splitFileName, which should be splitFilePath.

No, since its splits a FilePath into a FileName and the rest left over. In the same way that splitExtension takes a FilePath, but splits into an extension. I have named the operations by the result, rather than the input (which is almost always a FilePath)

It seems a bit odd because splitFileName follows the usual verb-noun convention, but in this case the verb doesn't refer to the noun. Not all the oeprations are named by the result: eg. dropFileName. That follows the verb-noun convention in the normal way.

...

Isn't joinFilename the same as combine?

Yes, I hadn't spotted thats (the code is duplicated, and written in a different style). However I feel its useful to have both of them in there, since having the join/split duality is quite handy, and combine "conceptually" operates on a lot more than just filenames.

I'd rather not have duplication because that can lead to confusion too - but at least it should be clearly documented. Certainly we should not have all of joinFileName, combine, addFileName since they all do the same thing. Also, getDirectory, dropFileName do the same thing. (btw, don't take all this as criticism, it's great that you're spending time on this and I think it's worth sanding down the rough edges so we can incorporate it in the base package). Cheers, Simon

Hi,

Perhaps instead:

directoryOf :: FilePath -> String filenameOf :: FilePath -> String extensionOf :: FilePath -> String basenaneOf :: FilePath -> String

replaceFilename = joinFilePath . directoryOf replaceDirectory = flip joinFilePath . filenameOf

Trying to design a consistent naming system, it helps if we all agree on what the various parts of a filepath are called, this is my draft of that: http://www-users.cs.york.ac.uk/~ndm/temp/filepath.png With a better name for basename, if anyone can think of one. Once we have that, how about takeElement :: FilePath -> String dropElement :: FilePath -> String replaceElement :: FilePath -> String -> FilePath addElement :: FilePath -> String -> FilePath splitElement :: FilePath -> (String, String) joinElement :: String -> String -> FilePath With the restriction that not all of these are provided. Some don't make sense (splitBaseName, dropBaseName), some are implemented via combine (addFileName, joinFileName), some are redundant (addExtensions == addExtension) I'm also debating whether split/join should be exported, since they are less likely to be used and can easily be written as a take/drop pair. And of course, a bigger interface is harder to understand. Opinions on this? It's easier to tweak a specification than the actual code :) Thanks Neil

Trying to design a consistent naming system, it helps if we all agree on what the various parts of a filepath are called, this is my draft of that:

http://www-users.cs.york.ac.uk/~ndm/temp/filepath.png

For what it's worth, the FSF's coding conventions uses "path" only for lists of directories (typically used for a search, e.g. MANPATH, PATH, LD_LIBRARY_PATH, load-path, ...), not for file names. In their convention, "/foo/bar/baz" is a file name, where "bar" is a file name element IIRC. Stefan

[Sorry for the late reply.] On Wed, Jul 19, 2006 at 03:16:48AM +0100, Neil Mitchell wrote:

...

...

I want to make sure a filename is valid. For example, "prn" and "con" This is another rat's nest, so I suggest that it be dealt with separately from the basic filepath module. The notion of "valid" is squishy: It depends entirely on what you intend to do with the path.

Its a rats nest to do it properly, but some very basic idea of "does this path have things which there is no way could possibly be in a file" - for example c:\|file is a useful thing to have.

This seems to encourage the classic mistake of checking "not known bad" rather than "known good". "known bad" is rarely useful in my experience. What use case do you have in mind?

...

...

In this library proposal, there are a bunch of "xxxDrive" functions that many Unix-oriented programmers are sure to ignore because they are no-ops on Unixy systems. Even on Windows, they are not very useful:

I strongly agree about this. The temptation in path modules seems to be to throw in everything you can think of (without specifying any of it precisely), just in case someone finds it useful.

The drive functions stand on their own as a chunk, and are possibly not well suited to a Posix system, but are critical for a Windows system.

Why are they critical for portable code? I am fine with Windows-specific functions, but I think it's a mistake to bundle them portable functions. (In my design, I have separate types for Windows and Unix paths, and imagine full support for Windows-specific operations, but only on the Windows type.)

I have tried to specify the functions precisely, and I use this specification as a test suite. Currently there are 114 properties in this test suite, all can be seen on the haddock documentation. If you consider any function to be ambiguously specified, please say which one and I'll add extra tests until it gives you no suprises at all.

My criticism is that your properties are all specified in terms of string manipulation. The whole point of paths is that they are interpreted by the system, so if you neglect to say what your operations mean to the system, what have you specified? Here are some specific cases I take issue with. (Quotes are from your generated docs.) Sorry if I seem to be piling it on, but I think these matters are important for a good path library.

pathSeparator :: Char The character that seperates directories.

So what do I do with this? If I need it, it seems like the module has failed.

splitFileName :: FilePath -> (String, String) Split a filename into directory and file.

Which directory and which file?

splitFileName "bob" == ("", "bob")

"" is not a directory.

Windows: splitFileName "c:" == ("c:","")

"c:" is arguably not a directory. (Consider that "dir c:" lists the current directory on c:, not c:\)

getFileName "test/" == ""

"" is not a filename. Also, it looks from this that you treat paths differently depending on whether they end in a separator. Yet this makes no difference to the system. That seems wrong to me.

setFileName :: FilePath -> String -> FilePath Set the filename.

This is vague to me. Eg, what does it do with "/", which has no filename?

getDirectory :: FilePath -> FilePath Get the directory name, move up one level.

What does this mean, in the presence of dots and symlinks?

normalise :: FilePath -> FilePath Normalise a file

As Simon asked, when is this safe to use?

equalFilePath :: FilePath -> FilePath -> Bool Equality of two FilePaths. If you call fullPath first this has a much better chance of working. Note that this doesn't follow symlinks or DOSNAM~1s.

As you acknowledge, it's a crap-shoot. So what's the point?

isValid :: FilePath -> Bool Is a FilePath valid, i.e. could you create a file like it?

There are a whole host of reasons you might not be able to create a file. Which ones does this address?

...

I tried to export a minimal set of operations that seem to me sufficient for everything not very platform-specific (though I am interested in counterexamples):

Anything to do with file extensions? Its also important (I feel) for people to have easy access to common operations, but I guess that is a design decision.

I think of that as a separate module, because extensions have no meaning to the system and can be done with portable, functional code, as far as I understand. Andrew

On Wed, Jul 26, 2006 at 03:36:13AM +0100, Neil Mitchell wrote:

...

...

pathSeparator :: Char The character that seperates directories. So what do I do with this? If I need it, it seems like the module has failed. Hopefully no one will ever use it. Its part of the low level functions that the FilePath module builds on. However, pragmatically, someone somewhere will have a use for it, and the second they do they'll just write '/', and at that point we've lost.

I'd just point out that I'm not aware of an operating system that GHC runs on that doesn't accept '/' as a path separator. It may be that you could fine an OS where you could compile with jhc or run with hugs that doesn't use '/' (e.g. MacOS 9), but support for MacOS 9 at this stage I wouldn't consider a high priority. Since noone ought to need the path separator, and since they can currently assume '/' without loss of portability, it seems like adding in an extra function to protect us from the introduction of an operating system some time in the future that doesn't allow '/' as a path separator is a bit much. Of course, I may be wrong. Does windows disallow mixing of '/' and '\\' as path separators? In darcs we always just use '/' as all the path separators, and it works fine... -- David Roundy

On Wed, Jul 26, 2006 at 03:36:13AM +0100, Neil Mitchell wrote:

...

...

Its a rats nest to do it properly, but some very basic idea of "does this path have things which there is no way could possibly be in a file" - for example c:\|file is a useful thing to have.

This seems to encourage the classic mistake of checking "not known bad" rather than "known good". "known bad" is rarely useful in my experience. What use case do you have in mind?

wget on windows saves web pages such as "http://www.google.com/index.html?q=haskell" to the file "index.html?q=haskell". This just doesn't work, and is the main reason I added this in. I don't think it will be a commonly used operation.

Ok, this is a good use case. What should wget do if "isValid" fails? Certainly not abort the download. So isValid alone is no help. Well, you have makeValid as well, but this is even more of a rats' nest. There are a zillion different ways you might want this function to work, depending on your purposes. Should makeValid be system-dependent? Should it be reversible? How hard should it try to preserve the name verbatim? Should it "prettify" legal but unprintable characters? The answers are application-dependent. Also, wget has to worry about not just whether the filename is valid, but whether it currently exists, and in that case modify it. So attempting to provide a generic makeValid is quixotic and will only lead to misuse.

...

My criticism is that your properties are all specified in terms of string manipulation. The whole point of paths is that they are interpreted by the system, so if you neglect to say what your operations mean to the system, what have you specified? True, but at the same time specifying what something means with respect to a filesystem is very hard :) If you had any insight how this could be done I'd be interested.

The first step is to think carefully about what operations to provide, and be conservative. I think the operations I included in my library all have pretty clear meanings, though I don't claim to have nailed them down all the way. Criticism welcome. http://haskell.org/pipermail/libraries/2006-February/004890.html

Hopefully no one will ever use it. Its part of the low level functions that the FilePath module builds on. However, pragmatically, someone somewhere will have a use for it, and the second they do they'll just write '/', and at that point we've lost.

Yes, on one hand you want to be pragmatic. But IMO this way of thinking--expose the guts just in case--is the path to madness. Not to mention, it clutters the API and makes it less clear how the module is supposed to be used. Maybe the "guts" could go into a separate module?

...

...

splitFileName :: FilePath -> (String, String) Split a filename into directory and file. Which directory and which file? Ok, thats probably the wrong description. Splits off the last filename would be a better description, leaving the rest.

Ok, but now what is "the rest" good for? And what is the "last filename" in cases like "/" or "..". The conclusion I come to is that this operation is unsound to begin with, and should not be part of the API in any form.

...

Also, it looks from this that you treat paths differently depending on whether they end in a separator. Yet this makes no difference to the system. That seems wrong to me. That was something I thought over quite a while. If the user enters "directory/" then they do not mean the file called directory, they mean the directory called directory. And in Windows certainly you can't open a file called "file/"

Ok, fair, but "dir" and "dir/" are treated identically if dir is a directory, so it is still confusing for your library to distinguish them. Maybe the user needs to indicate whether a path represents a file or directory? These matters confuse your specification. I made the simplifying approximation that "foo" and "foo/" should considered equivalent. This may not turn out to be the right decision, but at least it helped me keep the semantics clear.

...

...

getDirectory :: FilePath -> FilePath Get the directory name, move up one level. What does this mean, in the presence of dots and symlinks? It gets a parent directory, there may be one, but the one returned will be a parent.

Is "/a" a parent of "/a/.."? That seems dubious.

...

...

equalFilePath :: FilePath -> FilePath -> Bool Equality of two FilePaths. If you call fullPath first this has a much better chance of working. Note that this doesn't follow symlinks or DOSNAM~1s. As you acknowledge, it's a crap-shoot. So what's the point? Its a case of reality, at the moment people use == to test if two file paths are equal, at least this is a better test.

Why is it better?

...

I think of that as a separate module, because extensions have no meaning to the system and can be done with portable, functional code, as far as I understand. Not really, what about getExtension "file.ext\lump" - the answer is "" on windows and ".ext\lump" on Posix.

You would only call the extension functions on a segment name. Andrew

Hi

...

re: isValid Perhaps we should be more specific and make it talk about illegal file name characters if that is indeed the use case. Perhaps we should provide a system-dependent list of characters that are not allowed in file names. For example, on windows that would include '?'.

Then an application can decide for itself what to do about that depending on the context. It might be able to tell the user to pick a different name, or in the wget case replace it with a different character or remove it or something.

Unfortunately thats too much work for the user of the API. Since it tends to work on Posix, people probably won't go to the hassle of fixing it up. If they have a "simple fix" then there is at least a chance that they'll accept a patch that fixes the behaviour on Windows. The only options that I can see are either you want it fixed, or you want to get the user to fix it manually - both are catered for. And on Windows its more complex, LPT1.txt is also an invalid file, but LPT1.txt.txt isn't. Trying to express the weirdness of Windows is probably beyond the chances of an API :)

So maybe we should keep isValid but specify exactly what it checks. I'm happy to specify things in more detail, at the moment its pretty much a no-op on Posix, but if any Posix user suggests thats wrong I'll happily fix it up.

Sorry, I meant to say what I think wget should do. IMO, it should have a conservative set of allowed characters, encode the filename into that

Not enough, because of the LPT1 issue - unless you add L as a disallowed letter :)

...

Haskell makes the difference, runInteractiveCommand vs runInteractiveProcess I'm not following.

Having some considerations towards a real path, one that can be used on the command is reasonable, I think, because Haskell has functions within it that distinguish between firing something at the underlying filepath vs at the console. I don't however think its worth having a special type for working with emacs, unless you have System.FilePath.Emacs, given that Emacs is almost an operating system :)

But the trouble is that if you silently drop the '/', then the only way for me to implement a correct cp(1) in Haskell is to not use your proposed interface for pathname handling, which drops this information. Ok, now I remember the reasons I kept the trailing slash, I'll leave it in. Esp the risers4 issue.

Thanks Neil

On Wed, Jul 26, 2006 at 04:02:31PM -0700, Andrew Pimlott wrote:

I admit I don't know enough to say how the lpt1 issue should be handled. Is there any Win32 call I can make that will help me avoid accidentally opening these magic files? Say, if I call open with O_CREAT | O_EXCL? Unfortunately, I can find very little information on how one should handle this issue.

Thanks to a suggestion from Bulat to use c_open, I was able to test O_WRONLY | O_CREAT | O_EXCL on Windows. In fact, Windows does allow files like "nul" to be opened (as many times as you like) with these flags, which I find dismaying. So I still don't know the proper way to handle them. Andrew

On Thu, 2006-07-27 at 11:07 -0700, Andrew Pimlott wrote:

On Wed, Jul 26, 2006 at 04:02:31PM -0700, Andrew Pimlott wrote:

...

I admit I don't know enough to say how the lpt1 issue should be handled. Is there any Win32 call I can make that will help me avoid accidentally opening these magic files? Say, if I call open with O_CREAT | O_EXCL? Unfortunately, I can find very little information on how one should handle this issue.

Thanks to a suggestion from Bulat to use c_open, I was able to test O_WRONLY | O_CREAT | O_EXCL on Windows. In fact, Windows does allow files like "nul" to be opened (as many times as you like) with these flags, which I find dismaying. So I still don't know the proper way to handle them.

You can open the file and test the file type with GetFileType. If it's type FILE_TYPE_CHAR then it's probably not what you wanted. http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/g... Duncan

Andrew Pimlott wrote:

...

The drive functions stand on their own as a chunk, and are possibly not well suited to a Posix system, but are critical for a Windows system.

Why are they critical for portable code? I am fine with Windows-specific functions, but I think it's a mistake to bundle them [with] portable functions.

I couldn't agree more. In fact, why can't we pretend the world is sane at least within Haskell and just put away those drive letters?

My criticism is that your properties are all specified in terms of string manipulation.

Exactly. I believe, a FilePath should be an algebraic datatype. Most operations on that don't have to be specified, because they are simple and have an obvious effect. Add a system specific parser and a system specific renderer, maybe also define a canonical format, and the headaches stop. What's wrong with this? data FilePath = Absolute RelFilePath | Relative RelFilePath data RelFilePath = ThisDirectory | File String | ParentOf RelFilePath | String :|: RelFilePath parseSystemPath :: String -> Maybe FilePath renderSystemPath :: FilePath -> String We can even clearly distiguish between the name of a directory in its parent and the directory itself. On Windows, the root directory just contains the drive letters and is read-only, drive-absolute-but-directory-relative paths are simply ignored (they are a dumb idea anyway). Seperator characters are never exposed, all we need now is a mapping from Unicode to whatever the system wants.

...

pathSeparator :: Char The character that seperates directories.

So what do I do with this? If I need it, it seems like the module has failed.

Indeed.

...

splitFileName "bob" == ("", "bob")

"" is not a directory.

Some problems just vanish: parseSystemPath "bob" == Just (Relative (File "bob")) splitFileName (Relative (File "bob")) = (Relative ThisDirectory, File "bob")

...

Windows: splitFileName "c:" == ("c:","")

"c:" is arguably not a directory.

parseSystemPath "c:" == Nothing parseSystemPath "c:\" == Absolute ("C:" :|: ThisDirectory)

(Consider that "dir c:" lists the current directory on c:, not c:\)

I'd rather ignore that altogether. Multiple roots with associated "current directories" are just a needless headache. Even a "current directory" is somewhat ill-fitted for a functional language like Haskell.

...

getFileName "test/" == ""

"" is not a filename.

getFileName (Relative ("test" :|: ThisDirectory)) == error "pattern match failure"

Also, it looks from this that you treat paths differently depending on whether they end in a separator. Yet this makes no difference to the system. That seems wrong to me.

Not to the system, but some programs like to make a difference. If you give rsync a path that doesn't end in a slash, it will take that to mean the directory. With a slash, it means the contents of the directory. The difference is an additional path component that ends up on the target file system or doesn't.

...

getDirectory :: FilePath -> FilePath Get the directory name, move up one level.

What does this mean, in the presence of dots and symlinks?

You're right, this has to be ill-defined. Instead it should be moveUp :: FilePath -> IO FilePath which would end up in the parent of the linked-to directory after following a symlink. Cutting of a component is done by simple pattern matching, no special functions needed. Sorry for the rant, but this is Haskell, not Perl. We have true data types, not just strings... Udo. -- A politician is someone who calls a spade a portable, hand-operated digging implement. -- author unknown

Duncan Coutts wrote:

On Wed, 2006-07-26 at 15:29 +0200, Udo Stenzel wrote:

...

Exactly. I believe, a FilePath should be an algebraic datatype.

We've had this discussion before. The main problem is that all the current IO functions (readFile, etc) use the FilePath type, which is just a String.

So what's better? - use an ADT (correct and portable by construction), convert to String when calling the IO library - fumble with Strings, use an unholy mix of specialized and general functions, trip over a corner case

So a new path ADT is fine if at the same time we provide a new IO library.

We should just wrap the old API, filePathToString any parameters and liftIO the function while we're at it.

That's another portability headache - file name string encodings. Windows and OSX use encodings of Unicode. Unix uses strings of bytes.

Indeed. There are two ways out: - declare that Unix uses Unicode too, take the appropriate conversion from the locale - parameterize the FilePath ADT on the character type, you get (FilePath Word16) on Windows (which uses UCS-2, not UCS-4 and not UTF-16) and (FilePath Word8) on Unix; provide conversions from/to (FilePath String). I tend towards the second option. It at least doesn't make anything worse than it already is. It's also irrelevant, since pretending the issue doesn't exist works equally well with an ADT.

My point is it's not quite as simple as "just making an ADT".

Mine is that it is :) Moreover, a path already has internal structure. Those string manipulating functions either reconstruct the structure, then operate on that, then encode it back into a string or implement an approximation to that. The latter leads to surprises and making the former explicit can never hurt. Heck, NO library fumbles with strings, neither parsers nor pretty printers nor Network... why should a FilePath be different? Udo.

Neil Mitchell wrote:

How about

class FilePathLike a where getRealFilePath :: a -> String

Then convert readFile etc. to take a FilePathLike, rather than a filepath?

Uhm, just a minute ago the argument was that we can't change the IO library... Anyway, my gut says, it dislikes 'class FilePathLike'. Gives us even less well defined semantics than a plain string.

Windows doesn't use UTF-16, NTFS does.

I was under the impression that NT's Unicode support was conceived when it meant UCS-2. So it uses UCS-2 and not UTF-16, which would mean that you could in principle encounter lone surrogate characters or something equally nonsensical.

FAT doesn't. And what about the Samba drive I have mounted under Windows?

Windows has two sets of file system related functions, one for legacy 8-bit character sets, one for Unicode. What happens if I call the Unicode API on a FAT system that doesn't support it? Does it do a half-assed version of the locale specific encoding that we deem impossible and wrong here? Ah, never mind, I get the strong feeling I really don't want to know all this. When even Windows 98 has been end-of-lifed we should rely on the Unicode API, if anything. Udo. -- "There are three ways to make money. You can inherit it. You can marry it. You can steal it." -- conventional wisdom in Italy

Hello Udo, Thursday, July 27, 2006, 8:53:10 PM, you wrote:

Ah, never mind, I get the strong feeling I really don't want to know all this. When even Windows 98 has been end-of-lifed we should rely on the Unicode API, if anything.

i don't think we can do this in _compiler's system libraries_ otherwise, we should write in GHC manual: "programs, developed with ghc, can't work on win 98/me systems" -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

On Wed, 2006-07-26 at 19:41 +0200, Udo Stenzel wrote:

Duncan Coutts wrote:

...

On Wed, 2006-07-26 at 15:29 +0200, Udo Stenzel wrote:

...

Exactly. I believe, a FilePath should be an algebraic datatype.

We've had this discussion before. The main problem is that all the current IO functions (readFile, etc) use the FilePath type, which is just a String.

So what's better?

- use an ADT (correct and portable by construction), convert to String when calling the IO library

- fumble with Strings, use an unholy mix of specialized and general functions, trip over a corner case

In practise in the short term, the choice is between each application fumbling with strings in different incorrect ways or a library that fumbles with strings in a rather more considered and portable way.

...

So a new path ADT is fine if at the same time we provide a new IO library.

We should just wrap the old API, filePathToString any parameters and liftIO the function while we're at it.

Try proposing something concrete and see if you can get it generally accepted. Perhaps you can get it accepted for the next major release of various Haskell implementations or for Haskell-prime.

...

That's another portability headache - file name string encodings. Windows and OSX use encodings of Unicode. Unix uses strings of bytes.

Indeed. There are two ways out:

- declare that Unix uses Unicode too, take the appropriate conversion from the locale

Sadly this does not work. For one thing you don't know that the locale you're using now was the locale of the program that wrote the file. This happens on multi-user systems where different users use different languages. Then there is the fact that converting from Unicode back to the file name is not guaranteed to give the same sequence of bytes. For example, see the section "File Name Encodings" in the glib api: http://developer.gnome.org/doc/API/2.0/glib/glib-Character-Set-Conversion.ht...

- parameterize the FilePath ADT on the character type, you get (FilePath Word16) on Windows (which uses UCS-2, not UCS-4 and not UTF-16) and (FilePath Word8) on Unix; provide conversions from/to (FilePath String).

I tend towards the second option. It at least doesn't make anything worse than it already is. It's also irrelevant, since pretending the issue doesn't exist works equally well with an ADT.

Yeah, keeping it in the native format and doing no change of encoding is almost certainly the way to go. It doesn't address the issue of converting file names to/from displayable strings, but perhaps that's reasonable.

...

My point is it's not quite as simple as "just making an ADT".

Mine is that it is :) Moreover, a path already has internal structure. Those string manipulating functions either reconstruct the structure, then operate on that, then encode it back into a string or implement an approximation to that. The latter leads to surprises and making the former explicit can never hurt. Heck, NO library fumbles with strings, neither parsers nor pretty printers nor Network... why should a FilePath be different?

For compatibility with the Haskell98 IO library. There's also the issue here that adding in lots of conversions ADT <-> String means that people will not bother to use it and will continue to do things like: readFile (path ++ "/" ++ file) If anyone can actually design and implement an ADT that addresses most of these problems and can get it to work nicely with whatever is the popular IO system of the time then that'd be great. I think you'll find that it's not quite as simple as it looks. There was a discussion on a path ADT on the libraries list a while ago that's probably worth reading. I don't think it reached consensus. Duncan

On Wed, Jul 26, 2006 at 04:34:50PM +0100, Duncan Coutts wrote:

On Wed, 2006-07-26 at 15:29 +0200, Udo Stenzel wrote:

...

Exactly. I believe, a FilePath should be an algebraic datatype. Most operations on that don't have to be specified, because they are simple and have an obvious effect. Add a system specific parser and a system specific renderer, maybe also define a canonical format, and the headaches stop. What's wrong with this?

We've had this discussion before. The main problem is that all the current IO functions (readFile, etc) use the FilePath type, which is just a String.

Geesh, just provide a few wrappers. If we make this a show-stopper, we'll never get there.

That's another portability headache - file name string encodings. Windows and OSX use encodings of Unicode. Unix uses strings of bytes. They are not fully inter-convertible. On Unix the traditional technique is to keep a system file name in the original encoding and convert to Unicode to display to the user, but the Unicode version is never converted back to a system file name because it doesn't necessarily convert back to the same sequence of bytes.

The only solution for this, IMO, is to provide different types for different systems. Hence my typeclass approach. (Which I'm not saying is good enough to cover for all these differences yet.) Andrew

On Wed, Jul 26, 2006 at 03:29:01PM +0200, Udo Stenzel wrote:

Andrew Pimlott wrote:

...

Also, it looks from this that you treat paths differently depending on whether they end in a separator. Yet this makes no difference to the system. That seems wrong to me.

Not to the system, but some programs like to make a difference.

How does it make a difference? Do you have an answer that applies uniformly to all programs? If not, aren't we just walking down a blind alley? I've heard that Emacs treats double-separators specially. Do we account for that too? Maybe the trailing slash is important enough to take into account. But it complicates things, and the problem is hard enough with out it. So I say, leave it out. (In my design, with a different type for different systems, it would be possible to create types for rsync paths, or Emacs paths, etc. That might be a better approach to the problem.) Andrew

On Wed, Jul 26, 2006 at 08:19:47PM +0100, Neil Mitchell wrote:

...

Ok, this is a good use case. What should wget do if "isValid" fails?

makeValid is system dependant, and unspecified in its behaviour, although obviously some kind of closeness to the original would be ideal. So what if isValid fails and we don't have this?

Sorry, I meant to say what I think wget should do. IMO, it should have a conservative set of allowed characters, encode the filename into that set using an escaping mechanism it specifies, attempt to open the file O_EXCL, modify the name if it fails. The allowed characters set could perhaps come from the filepath module, though I suspect this is overkill. Simpler just to hard-code the set so that the name mangling is platform-independent and can be fully documented.

...

...

...

As you acknowledge, it's a crap-shoot. So what's the point? Its a case of reality, at the moment people use == to test if two file paths are equal, at least this is a better test. Why is it better?

Because its right more often, so pragmatically, is better

To me, that answer is unsatisfactory.

...

...

...

I think of that as a separate module, because extensions have no meaning to the system and can be done with portable, functional code, as far as I understand. Not really, what about getExtension "file.ext\lump" - the answer is "" on windows and ".ext\lump" on Posix. You would only call the extension functions on a segment name.

So this system independant extension module is dependant on a platform specific FilePath module? Or do you demand people make two function calls to get the extension?

One or the other, it seems a minor detail to me. There's nothing wrong with having the extension module use the filepath module.

...

...

Not to the system, but some programs like to make a difference.

How does it make a difference? Do you have an answer that applies uniformly to all programs? If not, aren't we just walking down a blind alley? I've heard that Emacs treats double-separators specially. Do we account for that too?

Haskell makes the difference, runInteractiveCommand vs runInteractiveProcess

I'm not following. Andrew

On Wed, Jul 26, 2006 at 11:39:40AM -0700, Andrew Pimlott wrote:

On Wed, Jul 26, 2006 at 03:29:01PM +0200, Udo Stenzel wrote:

...

Andrew Pimlott wrote:

...

Also, it looks from this that you treat paths differently depending on whether they end in a separator. Yet this makes no difference to the system. That seems wrong to me.

Not to the system, but some programs like to make a difference.

How does it make a difference? Do you have an answer that applies uniformly to all programs? If not, aren't we just walking down a blind alley? I've heard that Emacs treats double-separators specially. Do we account for that too?

cp(1), for example, treats paths with trailing separators differently from paths without. rm -rf foo bar echo test > foo echo othertest > bar cp foo bar/ cp foo bar It's part of the user interface, that allows the user to specify that he or she intends to use a path to describe a directory. This doesn't apply uniformly to all programs--except that we can say that any path with a trailing '/' is intended to be a directory, and if it's not, then that's an error. But the trouble is that if you silently drop the '/', then the only way for me to implement a correct cp(1) in Haskell is to not use your proposed interface for pathname handling, which drops this information. I'd also point out that rieser4, for instance, treats paths with a trailing slash differently even for files. True, it's probably not a good idea, but if we're talking about a portable library we might want it to work even on systems running an "interesting" filesystem like rieser4. -- David Roundy

Andrew Pimlott wrote:

On Wed, Jul 26, 2006 at 05:06:41PM -0400, David Roundy wrote:

...

This doesn't apply uniformly to all programs--except that we can say that any path with a trailing '/' is intended to be a directory, and if it's not, then that's an error.

I thought some more about this, and I think the right way to handle this is on parsing and printing.

Amen.

After all, the trailing slash has no real meaning for any intermediate processing you might do.

Here I beg to differ. I'd expect: *> setFileName "foo" "bar" == "bar" *> setFileName "foo/" "bar" == "foo/bar" In fact, that's consistent with the current documentation, because *> getFileName "foo" == "foo" *> getFileName "foo/" == "" No matter whether I'm correct, whether my expectation is natural or practical and whether others agree, the bahaviour has to be clearly specified and the final slash certainly isn't unimportant.

readPath :: String -> (Path, Bool {- trailing delimiter -}) showPath :: Path -> String showPathTrailingSlash :: Path -> String

This is far simpler than trying to figure out what the slash means for every path operation.

It's also far uglier... besides, it isn't about the slash, it is about the difference between file and directory. Udo. -- If you cannot in the long run tell everyone what you have been doing, your doing was worthless. -- Erwin Schrödinger

Andrew Pimlott wrote:

On Thu, Jul 27, 2006 at 09:59:37PM +0200, Udo Stenzel wrote:

...

In fact, that's consistent with the current documentation, because

*> getFileName "foo" == "foo" *> getFileName "foo/" == ""

I have to disagree with that.

No, you don't. That's the current behaviour of Neil Mitchell's System.FilePath 0.9 according to the haddockumentation. There isn't much point in disagreeing about observable facts, is there?

First of all, "" is not a filename;

Most certainly it isn't. Which is all the more reason not to like the current design. An empty filename just isn't the same as no filename.

if you mean that "foo/" has no filename, it makes much more sense to use something like a Maybe type.

It does very much. In fact, I don't deem getFileName to be an essential function when a simple pattern match would do the same thing. "foo/" really doesn't have a file name, as it very explicitly names a directory.

Second, "foo" is just as good a directory as "foo/" to the system

...unless you have both (think Reiser4) or you want to create the file (I think, but I'm not sure). However, what's the point in being ambiguous when we can be explicit? Sometimes there is a difference, libraries and tools shouldn't gloss over that without consideration.

But if you wish to make the distinction, at least provide an operation that lets me force a path to be treated "file-wise" or "directory-wise".

WTF?! A path names either a directory or a file. We might have some operations that accept file names instead of path names. What's there to be treated? Being explicit about the distinction makes any ambiguity go away.

Filesystems are ugly. :-)

So are microprocessors. We can still have a nice programming language, and we can also have a nice filesystem language.

And it is about the slash: "foo" can be a directory.

No, it still isn't. We can distinguish between "Directory (but not file, fifo, character or block special)" and "anything (if in doubt, not directory)", which is an essential semantic distinction and not just the accidental presence of a slash (or backslash or colon or whatever $EXOTIC_OS uses). Also, parsing paths _once_ and printing them _once_ but doing everything else by operating on their logical structure makes specifying any intermediate operation a lot easier, if nothing else. If this thread shows anything, then it is that specifying path operations is harder than expected. Udo. -- Structure is _nothing_ if it is all you got. Skeletons _spook_ people if they try to walk around on their own. I really wonder why XML does not. -- Erik Naggum

On Sat, Jul 29, 2006 at 05:35:30PM -0700, Andrew Pimlott wrote:

On Sat, Jul 29, 2006 at 02:59:06AM +0200, Udo Stenzel wrote:

...

Andrew Pimlott wrote:

...

Second, "foo" is just as good a directory as "foo/" to the system

...unless you have both (think Reiser4) or you want to create the file (I think, but I'm not sure). However, what's the point in being ambiguous when we can be explicit? Sometimes there is a difference, libraries and tools shouldn't gloss over that without consideration.

As I said, it's one of those line-drawing exercises. But your points are well taken, and maybe the trailing delimiter should be part of the model. (My criterion has been whether any filesystem operations require the trailing delimiter. It sounds like with reiser4fs they might.)

Actually, I just read in LWN that that part of reiser4 has been dropped. On the other hand, it was only dropped after considerable debate, and people using an older version of reiser4 still have the strange file-as-directory semantics. -- David Roundy http://www.darcs.net

Udo Stenzel writes:

No, not the trailing slash. The difference between a directory and its contents is important enough. This is ususally encoded using a trailing slash, but I'd rather not worry about that detail in a program.

What does Emacs do with double separators? I'm at a loss thinking of anything they could denote, but it could be useful.

Double separators invoke ange-ftp mode from find-file.

On 27/07/06, Jeremy Shaw wrote:

You mean like,

/path/to/somewhere//with/double/seperator

If so, it treats it as if you had typed in:

/with/double/seperator

This is actually false. There's nothing weird about Emacs and double seperators. It _is_ true that if you use C-x C-f (find-file) from *inside* Emacs, then Emacs will treat a double-slash as starting again from root, and a tilde anywhere as starting again from ~, but this isn't true when you start Emacs from the command line (try it!). As I guess the only time Haskell programs will be dealing with Emacs will be invoking from the shell, rather than calling inner Emacs Lisp functions, I doubt this is a problem. -- -David House, dmhouse@gmail.com