I'd like to request that utf8-string be added to the haskell platform, so that HP users can work with non-ascii text. Thanks a lot!
On Thu, May 14, 2009 at 4:23 PM, Simon Michael <simon@joyful.com> wrote:
I'd like to request that utf8-string be added to the haskell platform, so that HP users can work with non-ascii text.
I'd rather this wasn't added. It's an acceptable crutch for the short term, but we shouldn't be using String for text manipulation, and bundling utf8-string implicitly blesses that approach. The text library needs a few weeks of polish and some more testing work for QA, but it'll be the right answer well before the end of this year.
paolo.losi:
Bryan O'Sullivan wrote:
but we shouldn't be using String for text manipulation, and bundling utf8-string implicitly blesses that approach.
... sorry for the naive question. If we should not be using String, what should we be using instead?
For *text manipulation* (i.e. large bodies of text) a packed representation supporting unicode is appropriate. The 'text' package is the main candidate here.
On Thu, 2009-05-14 at 19:07 -0700, Bryan O'Sullivan wrote:
On Thu, May 14, 2009 at 4:23 PM, Simon Michael <simon@joyful.com> wrote: I'd like to request that utf8-string be added to the haskell platform, so that HP users can work with non-ascii text.
I'd rather this wasn't added. It's an acceptable crutch for the short term, but we shouldn't be using String for text manipulation, and bundling utf8-string implicitly blesses that approach. The text library needs a few weeks of polish and some more testing work for QA, but it'll be the right answer well before the end of this year.
I agree with Bryan. The utf8-string is a fine intermediate workaround for the lack of decent Unicode IO in the base library and the lack of a decent packed text representation. However both these things are on their way and we should not standardise on a second rate solution (despite the fact that many packages do already depend on utf8-string). On the topic of adding things (once we've resolved the issue of API stability in minor releases) I'd like us to get on with discussing what we should require for new packages entering the platform. I think it's best to decide that first, before looking at too many concrete examples and having to reverse engineer our standards from our previous decisions. Duncan
On 15/05/2009 03:07, Bryan O'Sullivan wrote:
On Thu, May 14, 2009 at 4:23 PM, Simon Michael <simon@joyful.com <mailto:simon@joyful.com>> wrote:
I'd like to request that utf8-string be added to the haskell platform, so that HP users can work with non-ascii text.
I'd rather this wasn't added. It's an acceptable crutch for the short term, but we shouldn't be using String for text manipulation, and bundling utf8-string implicitly blesses that approach. The text library needs a few weeks of polish and some more testing work for QA, but it'll be the right answer well before the end of this year.
We ought to think about the interaction between text (and bytestring) and the new Unicode IO library. What does text have in the way of IO operations? I've been wondering about what bytestring's hGetLine should do. Right now I have it doing decoding and then taking the low 8 bits, but that's not right. OTOH, looking for '\n' in a stream of bytes doesn't seem right. Maybe it should just be deprecated. Cheers, Simon
Fortunately, the bytewise encoding of '\n' is sufficient to recognize a newline, any other attempted representation in UTF8 (i.e. as a 2-byte symbol starting with 0xc0) would be non-canonical and per RFC 3629 should be rejected anyways. So if you view ByteString as a stream of bytes that may or may not be utf8 encoded, scanning for 0x0a gives you the correct behavior for both scenarios. -Edward Kmett On Fri, May 15, 2009 at 7:02 AM, Simon Marlow <marlowsd@gmail.com> wrote:
On 15/05/2009 03:07, Bryan O'Sullivan wrote:
On Thu, May 14, 2009 at 4:23 PM, Simon Michael <simon@joyful.com <mailto:simon@joyful.com>> wrote:
I'd like to request that utf8-string be added to the haskell platform, so that HP users can work with non-ascii text.
I'd rather this wasn't added. It's an acceptable crutch for the short term, but we shouldn't be using String for text manipulation, and bundling utf8-string implicitly blesses that approach. The text library needs a few weeks of polish and some more testing work for QA, but it'll be the right answer well before the end of this year.
We ought to think about the interaction between text (and bytestring) and the new Unicode IO library. What does text have in the way of IO operations?
I've been wondering about what bytestring's hGetLine should do. Right now I have it doing decoding and then taking the low 8 bits, but that's not right. OTOH, looking for '\n' in a stream of bytes doesn't seem right. Maybe it should just be deprecated.
Cheers, Simon
_______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries
On 15/05/2009 14:55, Edward Kmett wrote:
Fortunately, the bytewise encoding of '\n' is sufficient to recognize a newline, any other attempted representation in UTF8 (i.e. as a 2-byte symbol starting with 0xc0) would be non-canonical and per RFC 3629 should be rejected anyways. So if you view ByteString as a stream of bytes that may or may not be utf8 encoded, scanning for 0x0a gives you the correct behavior for both scenarios.
The byte string can be in any encoding, not just UTF-8. Cheers, Simon
-Edward Kmett On Fri, May 15, 2009 at 7:02 AM, Simon Marlow <marlowsd@gmail.com <mailto:marlowsd@gmail.com>> wrote:
On 15/05/2009 03:07, Bryan O'Sullivan wrote:
On Thu, May 14, 2009 at 4:23 PM, Simon Michael <simon@joyful.com <mailto:simon@joyful.com> <mailto:simon@joyful.com <mailto:simon@joyful.com>>> wrote:
I'd like to request that utf8-string be added to the haskell platform, so that HP users can work with non-ascii text.
I'd rather this wasn't added. It's an acceptable crutch for the short term, but we shouldn't be using String for text manipulation, and bundling utf8-string implicitly blesses that approach. The text library needs a few weeks of polish and some more testing work for QA, but it'll be the right answer well before the end of this year.
We ought to think about the interaction between text (and bytestring) and the new Unicode IO library. What does text have in the way of IO operations?
I've been wondering about what bytestring's hGetLine should do. Right now I have it doing decoding and then taking the low 8 bits, but that's not right. OTOH, looking for '\n' in a stream of bytes doesn't seem right. Maybe it should just be deprecated.
Cheers, Simon
_______________________________________________ Libraries mailing list Libraries@haskell.org <mailto:Libraries@haskell.org> http://www.haskell.org/mailman/listinfo/libraries
On Fri, 2009-05-15 at 12:02 +0100, Simon Marlow wrote:
On 15/05/2009 03:07, Bryan O'Sullivan wrote:
On Thu, May 14, 2009 at 4:23 PM, Simon Michael <simon@joyful.com <mailto:simon@joyful.com>> wrote:
I'd like to request that utf8-string be added to the haskell platform, so that HP users can work with non-ascii text.
I'd rather this wasn't added. It's an acceptable crutch for the short term, but we shouldn't be using String for text manipulation, and bundling utf8-string implicitly blesses that approach. The text library needs a few weeks of polish and some more testing work for QA, but it'll be the right answer well before the end of this year.
We ought to think about the interaction between text (and bytestring) and the new Unicode IO library.
Yes absolutely. We should (re-)design the IO functions for text and bytestring in parallel with the new IO stuff.
What does text have in the way of IO operations?
It doesn't have any at the moment. The plan I think is to have it work with the Unicode stuff in the new IO system. So yes, we designing System.IO properly and with consideration for types like Text and ByteString and not standardising on interim solutions like utf8-string.
I've been wondering about what bytestring's hGetLine should do.
ByteString is always binary so hGetLine for ByteString only makes sense on binary handles. Going via any Unicode decoding would be wrong. When we consider ByteString via the Char8 view it's still binary but we assume there is some ASCII mixed in so we can look for the ASCII encodings of '\n' etc. On the other hand hGetLine for the String or Text type only makes sense for text handles where we're decoding and translating into Unicode. Duncan
On Fri, May 15, 2009 at 4:02 AM, Simon Marlow <marlowsd@gmail.com> wrote:
We ought to think about the interaction between text (and bytestring) and the new Unicode IO library. What does text have in the way of IO operations?
Nothing. It has pure functions for transcoding to/from ByteString. Those functions aren't general enough, since they throw errors if an encoding problem is encountered. We'll need to provide alternatives that require an error handler as a parameter, so that you can do nothing, throw an error, or provide a substitution character instead.
I've been wondering about what bytestring's hGetLine should do. Right now I have it doing decoding and then taking the low 8 bits, but that's not right. OTOH, looking for '\n' in a stream of bytes doesn't seem right. Maybe it should just be deprecated.
It should be deprecated once there's something sensible to replace it with. Note that "something sensible" isn't necessarily straightforward; Unicode specifies 7 different code point sequences as newline equivalents (LF, CR, CR+LF, NEL, FF, PS, LS).
Tracked as ticket: http://trac.haskell.org/haskell-platform/ticket/48 Thanks. simon:
I'd like to request that utf8-string be added to the haskell platform, so that HP users can work with non-ascii text.
Thanks a lot!
_______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries
Don Stewart wrote:
Is that really supposed to have milestone 2009.4.0? I thought the first release would just be extralibs. Ganesh =============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ===============================================================================
ganesh.sittampalam:
Don Stewart wrote:
Is that really supposed to have milestone 2009.4.0? I thought the first release would just be extralibs.
2009.4.0 is the next major release. We should decide by then if a utf8 lib is in. -- Don
participants (8)
-
Bryan O'Sullivan -
Don Stewart -
Duncan Coutts -
Edward Kmett -
Paolo Losi -
Simon Marlow -
Simon Michael -
Sittampalam, Ganesh