Sven Moritz Hallberg wrote:

decodeOne :: [Word8] -> (Either Error Char, Int, [Word8]) -- 2nd. component: number of bytes consumed, -- 3rd. component: rest of bytes decode :: [Word8] -> (String, [(Error,Int)]) -- 2nd. component: list of errors and their index in the byte stream -- Maybe we should reverse the order of error/index -- so it looks like any association list?

Comments welcome.

IMHO: 1. Decoders should present a consistent interface; possibly by means of a class. Modal encodings would need some kind of state. 2. "decode" should return the remaining bytes, in case the input ends with a partial character (or contains errors; see next). 3. The basic decoder interface shouldn't attempt to recover from errors. Rather, it should return the list of complete characters, the list of remaining octets, and the final state. Any error recovery should be an optional add-on. -- Glynn Clements

Ketil Z. Malde wrote:

...

3. The basic decoder interface shouldn't attempt to recover from errors. Rather, it should return the list of complete characters, the list of remaining octets, and the final state. Any error recovery should be an optional add-on.

Could error handling be passed as a parameter to the encoder, perhaps? E.g. if I'm not really interested in debugging the code, just extracting what's possible, I could pass an error handler that tries to skip errors and keep going, without having to pollute my higher level code with it?

I would prefer not to see the base decoders cluttered with error recovery functionality. IMHO, a better alternative would be to provide functions to generate fault-tolerant decoders from an existing decoder, e.g. by repeatedly calling the underlying decoder until all octets have been consumed, handling errors in either a predefined or user-defined manner. Individual encodings could also provide custom fault-tolerant decoders; in some cases, it may be desirable to have a choice of several alternatives. The nature of the problem differs substantially between different types of encoding. ISO-8859-* is trivial; one octet corresponds to one character. The only possible error is an undefined codepoint (e.g. 0x80-0x9F); there are no synchronisation issues. UTF-8 is almost as simple; character boundaries are unambiguous, even for invalid streams. However, there exists some variation between existing decoders. Over-long sequences (e.g. using a two byte sequence to represent a 7-bit character) are technically invalid, but many decoders allow this; some applications even encourage it (e.g. using 0xC0,0x80 to represent an "embedded" NUL). Other encodings may be more problematic, and error recovery may involve some guesswork. This can be helped by knowledge of the relative likelihood of certain classes of error and/or the nature of the text (e.g. language). My main concern is that less knowledgable users don't end up being "steered" into using dubious semantics (e.g. fault tolerance, especially when involving somewhat arbitrary heuristics) by way of using the "default" interface. Of particular concern are the potential security implications. The most obvious[1] example is the use of invalid or ambiguous encoded forms to circumvent access controls or input validation. [1] Obvious to regular readers of BugTraq, at least; this specific issue has been identified as a security problem in a wide range of products. Basically, my view is that if the user is required to explicitly choose fault tolerance, there's more chance that they will consider some of the issues involved than if some form of fault tolerance is "bundled". -- Glynn Clements

tis 2002-09-03 klockan 01.06 skrev Sven Moritz Hallberg:

With the ICFP contest finally over, I have just committed mine to CVS (thanks for setting it up Ashley!). I hope it is of reasonable quality, I've not performance-tested it. I'm looking forward to all feed-back.

My quickCheck tests found errors in three and four byte encodings: *Text.Encoding.TestUTF8> encodeOne (toEnum 51177) [236,159,140] *Text.Encoding.TestUTF8> decodeOne [236,159,140] (Right '\51148',3,[]) *Text.Encoding.TestUTF8> encodeOne (toEnum 845466) [243,206,154,154] *Text.Encoding.TestUTF8> decodeOne [243,206,154,154] (Left (InvalidLaterByte 1),1,[206,154,154]) 51177 should encode into [236,159,169] 845466 should encode into [243,142,154,154] I see you use hugs, you could consider using ghci, it's really nice and work exactly like hugs. Regards, Martin -- Martin Norbäck d95mback@dtek.chalmers.se Kapplandsgatan 40 +46 (0)708 26 33 60 S-414 78 GÖTEBORG http://www.dtek.chalmers.se/~d95mback/ SWEDEN OpenPGP ID: 3FA8580B

On Tue, 2002-09-03 at 10:46, Martin Norbäck wrote:

My quickCheck tests found errors in three and four byte encodings:

*Text.Encoding.TestUTF8> encodeOne (toEnum 51177) [236,159,140] *Text.Encoding.TestUTF8> decodeOne [236,159,140] (Right '\51148',3,[]) *Text.Encoding.TestUTF8> encodeOne (toEnum 845466) [243,206,154,154] *Text.Encoding.TestUTF8> decodeOne [243,206,154,154] (Left (InvalidLaterByte 1),1,[206,154,154])

51177 should encode into [236,159,169] 845466 should encode into [243,142,154,154]

Fixed. All three properties seem to be satisfied in GHCi now. Beware of prop_DecEnc, though, quickCheck erroneously claims to falsify it. I guess this is caused by a bug in GHC 5.04's handling of strings that hit the list some time ago (try "\666"==['\666']). Manually applying prop_DecEnc to the alleged counter example has always yielded True for me. Regards, Sven Moritz

G'day all. On Thu, Aug 29, 2002 at 11:31:16AM +0200, Martin Norbäck wrote:

hfl (Andrew Bromage) Haskell Foundation Library "The future of the Haskell standard library is here." CVS contains edison and some Monads

That would be me. HFL currently contains my personally hacked-over version of Edison, featuring fundeps and "not just maybe" methods, plus one or two of the gaps filled in. The other part is the Monad Template Library with one addition (MonadNondet) and one more in development, not yet checked in. My thinking behind hfl is similar to Boost for C++. In particular, I want to produce libraries which are suitable for future standardisation. That is, it has to be general enough, flexible enough and generally useful enough to be in a future incarnation of the standard library. To some extent, this is closely aligned with the philosophy behind haskell-i18n, but perhaps not with the other two projects, which look to me like more like CPAN, that is, a repository of useful libraries most of which most programs would not need. The line between the two ideas is somewhat fuzzy, but intuitively, an awful lot of programs need FiniteMap or something like it, but not so many need an IMAP library, even though an IMAP library is a very good thing to have for those times you need it.

Perhaps we should try and merge these, is there any point in having multiple projects and repositories when we have hierarchical libraries?

Semi-random half-baked thoughts follow. The major problem with Haskell libraries at the moment, and one of the reasons why I started HFL, is that there are a lot of existing libraries "out there" which have nothing in common. Naming schemes are inconsistent, error/exception handling is inconsistent, iterator support is handled in a dozen different ways and so on. This is partly a symptom of the fact that there's not a lot of common engineering experience with Haskell "out there". Those few places which do use Haskell (or similar languages) don't cross- pollinate with each other nearly as much as, say, C++ programmers. As a result, there are as many styles as there are programmers. Plus, to be honest, most of the people who write Haskell libraries are scientists. When you have a research quota to fill, integration with existing libraries isn't high on your agenda, and rightly so. Now I'm not opposed to merging in principle, however, if my intuition on the relative philosophies behind the four current projects is correct, then it seems to me that we really want is two (sub-)projects: the "core" libraries, and the "add on" libraries. Naturally we'd want these (sub)projects to work as symbiotically as possible. I'd also like, eventually, to institute some proper unit testing, code auditing and peer review, particularly on anything we're proposing for standardisation, but this can be discussed later. Cheers, Andrew Bromage

G'day all. On Fri, Aug 30, 2002 at 10:09:25AM +0100, Simon Marlow wrote:

Agreed that there are lots of inconsistent libraries out there, but why start a new project when there's already libraries@haskell.org? Surely this is the right point of focus for developing new libraries, and we also have a CVS repository for the code: fptools/libraries on cvs.haskell.org. We also have the beginnings of guidelines for naming conventions and coding style.

The short answer is that, speaking as a user of GHC and GHCi, I'd prefer to have libraries with mature interfaces "out of the box". Integrating experimental libraries too early inevitably creates a body of legacy code that I don't want to be responsible for. For example, for my part I've made a lot of changes in my version of Edison which are not backwards-compatible, which I believe are an improvement, but I still don't know if I've got it "right" yet. I don't want to break everyone's Edison code only to have it break again next release. Of course if the changes turn out not to require incompatibilities, there's nothing stopping me from submitting them, but had I been hacking fptools/libraries to begin with, I might have been more hesitant about playing with existing libraries in the first place. The long answer I won't go into in detail, but part of the problem is that being a fptools/libraries developer basically means having a GHC development environment. That requires an investment which I'm personally not able to make at the moment.

Really, it's not that hard - the only reasons I would actively argue against something going into fptools/libraries are: if there is duplication of functionality between libraries that will only serve to confuse users, or if there is substantial disagreement about whether a particular API is the "right thing".

...both of which describe exactly the situation that I'm in at the moment!

I see the process of standardisation as separate; at some point after the libraries have matured in fptools/libraries for some time, we will standardise some of them in a Haskell 98 addendum. This will probably be an ongoing process, with more libraries becoming standardised as they mature.

This raises an interesting question, because almost all newer libraries (certainly the ones I write) all use non-98 language features. I suspect that this is true of many new libraries, and certainly most of the "interesting" ones. Cheers, Andrew Bromage