Hello John, Saturday, February 04, 2006, 3:30:49 AM, you wrote: JM> It would be nice if a 'FastByteArray' could be pulled out of JM> FastPackedString since it is a very useful thing in its own right. are UArray/IOUArray/StorableArray is not appropriate? -- Best regards, Bulat mailto:bulatz@HotPOP.com

Hello John, Saturday, February 04, 2006, 10:54:35 PM, you wrote:

...

JM> It would be nice if a 'FastByteArray' could be pulled out of JM> FastPackedString since it is a very useful thing in its own right.

are UArray/IOUArray/StorableArray is not appropriate?

JM> appropriate building blocks, but their interfaces are rather austere JM> compared to FastPackedString, and it would nice to have all of its JM> options available in something not dedicated just to strings, since we JM> have already implemented them in any case. so it seems logical to add these interfaces to all arrays or at least all unboxed arrays? this also corresponds with problem of having common collections interface, what is a goal of J.P. Bernardy -- Best regards, Bulat mailto:bulatz@HotPOP.com

Hello John, Sunday, February 05, 2006, 5:58:20 PM, you wrote:

...

so it seems logical to add these interfaces to all arrays or at least all unboxed arrays? this also corresponds with problem of having

JM> Though, independent of that whole ball of wax, the FastByteString JM> concrete data type would be nice to have. how it is different from UArray Int Word8? i think it will be better to add 0-based unboxed arrays -- Best regards, Bulat mailto:bulatz@HotPOP.com

On Mon, Feb 06, 2006 at 12:08:12PM +0300, Bulat Ziganshin wrote:

Hello Einar,

Saturday, February 04, 2006, 12:00:18 AM, you wrote:

...

...

KA> * Do you think that we have to use unboxed arrays of Word8/Word16 KA> for file path?

well, if FilePath should be ADT, then:

1) all operations that returns filenames (getFileContents, anything more?) should return this ADT instead

I agree. In System.Directory: You mean getDirectoryContents? getCurrentDirectory, getHomeDirectory, getTemporaryDirectory, etc. canonicalizePath. All sorts of Cabal stuff. Some of these could just return String. Others could return the ADT... Actually couldn't they all be made polymorphic? getDirectoryContents :: FilePath p => IO p class FilePath p where fromADT :: ADT -> p toADT :: p -> ADT (where I have declined naming the ADT)

2) this ADT should support conversion to/from String (for interaction with users) and to/from [Word8] (to send these values across storage/network). Of course, these conversions should be as exact and revertible as possible, i think that at least we can make guarantee for ascii chars

Probably we'd also want a withCString equivalent that works with the ADT, for use when working with the FFI. True, you could do the conversion via [Word8], but that's ugly (and potentially inefficient, depending how the ADT is stored internally).

3) all operations that receives filenames should accept a _class_ which includes String/[Word8]/this ADT

That would be nice, but I suspect not strictly necessary.

of course, this means incompatible changes in the APIs, so it's better to introduce new fucntions that returns ADT/accepts class with a new names

I don't know. I'd lean towards not introducing new function names. The point of accepting the class would be that existing code would continue to work. The only code that would be broken would be code that uses getFileContents, which is a sufficiently small minority of code that I would say just let it break. You'd leave the Haskell 98 Directory module returning String, but I'd vote to make System.Directory. -- David Roundy http://www.darcs.net

On Mon, Feb 06, 2006 at 04:47:51PM +0200, Krasimir Angelov wrote:

2006/2/6, David Roundy :

...

Actually couldn't they all be made polymorphic?

getDirectoryContents :: FilePath p => IO p

class FilePath p where fromADT :: ADT -> p toADT :: p -> ADT

Would you mean that we should have two instances:

instance FilePath String where .... instance FilePath ADT where ....

It will work but I think that it is unnecessary complication. I prefer to use type classes only when they are really necessary i.e. when I have more than 2-3 instances or when I expect to have a lot of generic code that should work on both types.

The advantage would be that one could continue to use code that doesn't use the ADT for file paths, which would be convenient. And you'd need the conversion routines anyways, in order to display filepaths. True, it's just a convenience issue, but being able to write writeFile "output" contents would seem to me to be worth the complexity of having a class. In fact, I would think that all filepath-related code would work with either type. Also, this would allow users to define FastPackedStrings or whatever else to be in class FilePath, which is certainly something I've wanted to do with darcs, just to save the memory associated with all those Strings without the cost of converting to a String and then back to a CString (which the IO library must do internally) on every file operation.

I think that if we should switch from plain String to ADT then this should be made for a new i/o library (the Handle(s) replacement) which will change the thing a lot in any way. For the current library I think that the plain String is more reasonable. If the general consensus is that ADT is the way to go then I think some effort on a new efficient IO library be forced. It should have:

- FilePath ADT - Packed string - Fast I/O with unicode support

I know some pieces already exists but they need integration and polishing.

Indeed, it would be great to have a shiny new IO library, but even then I'd lean towards a FilePath class just for convenience. It'd be a royal pain for quickly written programs to have to always use (toADT "filename") when doing file IO with string literals. -- David Roundy http://www.darcs.net

Hello David, Monday, February 06, 2006, 4:12:41 PM, you wrote:

...

well, if FilePath should be ADT, then: 1) all operations that returns filenames (getFileContents, anything more?) should return this ADT instead

DR> You mean getDirectoryContents? DR> getCurrentDirectory, getHomeDirectory, getTemporaryDirectory, etc. DR> canonicalizePath. DR> Actually couldn't they all be made polymorphic? yes, of course they should! now i myself experiment with that sort of polymorphic interfaces. for example, "getFileSize" can return any Integral. it's a user decision, whether he needs to receive Int, Integer or say Word64 in answer. this have some limitations, namely that we need to add type signatures sometimes (although this can be addressed in future Haskell by extending the "default" directive) and that this can be slower/generate more code DR> getDirectoryContents :: FilePath p => IO p DR> class FilePath p where DR> fromADT :: ADT -> p DR> toADT :: p -> ADT DR> (where I have declined naming the ADT) yes, we can define all interfaces using this class, while internally all these functions will convert parameters to the type they need: convertADT = fromADT . toADT -- like "fromIntegral" definition getDirectoryContents :: (FilePath dir, FilePath file) => dir -> IO [file] getDirectoryContents dir = do let (dirStr :: String) = convertADT dir .... return (map convertADT filelist)

...

2) this ADT should support conversion to/from String (for interaction with users) and to/from [Word8] (to send these values across storage/network). Of course, these conversions should be as exact and revertible as possible, i think that at least we can make guarantee for ascii chars

DR> Probably we'd also want a withCString equivalent that works with the ADT, DR> for use when working with the FFI. True, you could do the conversion via DR> [Word8], but that's ugly (and potentially inefficient, depending how the DR> ADT is stored internally). probably, this ADT should be different for different OSes and just be the same as filenames representation native for this OS. that guarantee us that any filenames possible in this OS will be representable and minimal overhead of OS calls. if someone need another concrete representation - he would convert this ADT to the concrete datatype he need, be it String, [Word8], "UArray Int Word8" or FastPackedString

...

3) all operations that receives filenames should accept a _class_ which includes String/[Word8]/this ADT

DR> That would be nice, but I suspect not strictly necessary. i mean "all operations in standard library". it's easy to do, at least while we don't count efficiency problems

...

of course, this means incompatible changes in the APIs, so it's better to introduce new fucntions that returns ADT/accepts class with a new names

DR> I don't know. I'd lean towards not introducing new function names. The DR> point of accepting the class would be that existing code would continue to DR> work. The only code that would be broken would be code that uses DR> getFileContents, which is a sufficiently small minority of code that I DR> would say just let it break. You'd leave the Haskell 98 Directory module DR> returning String, but I'd vote to make System.Directory. really, promoting all operations to using class will be enough to not break compatibility with any existing code -- Best regards, Bulat mailto:bulatz@HotPOP.com

Ben Rudiak-Gould wrote:

Also, getArgs should return [ADT].

I would rather this be left as [String]. I don't want to have to do back-conversion from ADT to get my option flags. -- Ashley Yakeley

Ashley Yakeley wrote:

I would rather this be left as [String]. I don't want to have to do back-conversion from ADT to get my option flags.

It's not actually a back-conversion -- see my other post. -- Ben

Hello Ben, Tuesday, February 07, 2006, 1:33:44 AM, you wrote:

...

1) all operations that returns filenames (getFileContents, anything more?) should return this ADT instead

BRG> Also, getArgs should return [ADT]. getLine also shoukd return ADT - just for case when user supplied a filename? :) and "++" operation should return ADT too? :) String is filename representation for dialogue with user and this ADT should just support conversion to/from Strings. while getArgs, unlike the current atate of affairs, should return just proper Unicode strings -- Best regards, Bulat mailto:bulatz@HotPOP.com

On Tue, Feb 07, 2006 at 04:25:35PM +0000, Ben Rudiak-Gould wrote:

The point is that different things are natively handled in different formats under different OSes, e.g.

Posix NT Win9x

pathnames bytes UTF-16 locale command line bytes UTF-16 locale file contents bytes bytes bytes pipes/sockets bytes bytes bytes

actually, Posix systems should be the following

pathnames locale UTF-16 locale command line locale UTF-16 locale file contents * bytes bytes pipes/sockets * bytes bytes

Although the Posix interface is in terms of bytes, the strings should always be interpreted via the locale specified in $LANG or $LC_CTYPE also, for file contents and pipes/sockets, if you are passing text, and in the absence of some overriding standard or protocol, you should be using the encoding specified in the locale too. John -- John Meacham - ⑆repetae.net⑆john⑈

On Wed, Feb 08, 2006 at 09:10:37PM +0000, Ben Rudiak-Gould wrote:

John Meacham wrote:

...

On Tue, Feb 07, 2006 at 04:25:35PM +0000, Ben Rudiak-Gould wrote:

...

Posix NT Win9x

pathnames bytes UTF-16 locale command line bytes UTF-16 locale file contents bytes bytes bytes pipes/sockets bytes bytes bytes

actually, Posix systems should be the following

...

pathnames locale UTF-16 locale command line locale UTF-16 locale file contents * bytes bytes pipes/sockets * bytes bytes

Although the Posix interface is in terms of bytes, the strings should always be interpreted via the locale specified in $LANG or $LC_CTYPE also, for file contents and pipes/sockets, if you are passing text, and in the absence of some overriding standard or protocol, you should be using the encoding specified in the locale too.

But that's an application-level convention; the kernel only knows about bytes. On Windows the encoding of pathnames and the command line is a requirement imposed by the kernel. I think assuming the locale encoding for the command line on Posix is a bad idea. Users are unlikely to pass a misencoded command line explicitly, but I want my-haskell-util `find .` to work even on a mounted volume that uses the wrong encoding. (And I also want your-haskell-util to work, even if you didn't write it with this situation in mind.)

when the command line is to be interpreted as a string, then interpreting it in the current locale is definitly the right thing to do. This is why we need two varieties of getArgs, one which returns [String] and one which returns [[Word8]]. though, I doubt the second form will be needed much since in general you usually think of command line arguments as strings, but it should be provided since it can't really be worked around. John -- John Meacham - ⑆repetae.net⑆john⑈

On 08.02 14:03, Wolfgang Thaller wrote:

1) Widely used languages and libraries like Java and GTK+ assume that all file names and command lines are encoded in the system locale, or at least that they can all be converted to unicode strings.

Which causes much annoyance to users having to define various environment variables just to get them to open a file.

2) Command lines are usually entered as TEXT on a terminal and are therefore encoded in whatever encoding the terminal uses.

Actually I like the ablity to delete/copy files even if they happen to have filenames in weird chinese encodings too. Users just use wildcards or tab completion to get around filenames that are hard to type.

3) None of the recent linux distributions I have installed did anything but set up a UTF-8 based system.

Very many people needing to use their own language still use other things and will continue so for the foreseeable future.

So I think we should try hard to avoid introducing any additional complexity, like filename ADTs used for program arguments, to deal with the small minority of systems where file names cannot be converted to unicode. Maybe it's possible to use some user-defined unicode code points to achieve a lossless conversion of arbitrary byte strings to unicode? I mean, byte strings that are valid in the system encoding would get transcoded correctly, and invalid bytes would get mapped to some extra code points so that they can be converted back if necessary.

What would happen if you tried to output such a String? The raw bytes or the escaped versions? Also this would mean that Haskell unicode != unicode (isn't Java's broken handling enough). - Einar Karttunen

On 07.02 10:48, Bulat Ziganshin wrote:

Hello Ben,

Tuesday, February 07, 2006, 1:33:44 AM, you wrote:

...

...

1) all operations that returns filenames (getFileContents, anything more?) should return this ADT instead

BRG> Also, getArgs should return [ADT].

Just define: getArgsAdt :: IO [ADT] getArgsAdt = ... getArgs :: IO [String] getArgs = fmap ADT.toString getArgsAdt - Einar Karttunen