On 19 October 2010 22:08, Roman Leshchinskiy wrote:

On 19/10/2010, at 15:22, John Lato wrote:

...

I think there's a significant difference between vector and text, namely a Vector is conceptually the same as a list/1D array, while a Text is not.  I think this difference is enough to warrant a break from the list API.

Are you sure? From its interface Text looks exactly like a list of Chars to me.

Right, that's a very common misunderstanding of Unicode. A Unicode code point (type Char) does not correspond 1:1 with the human notion of a character. It would be nice if it did, but unfortunately it is not something we can ignore. Because of this it is better not to think of operations on individual Chars but on short sequences of Chars. In any case, when processing text (even ASCII where Chars do match characters) many of the most common operations that you want are substring not element based. So this is why a list/vector API is not necessarily appropriate for text. Duncan

On 10/19/10 7:59 PM, Ross Paterson wrote:

On Wed, Oct 20, 2010 at 12:35:33AM +0100, Duncan Coutts wrote:

...

On 19 October 2010 22:08, Roman Leshchinskiy wrote:

...

On 19/10/2010, at 15:22, John Lato wrote:

...

I think there's a significant difference between vector and text, namely a Vector is conceptually the same as a list/1D array, while a Text is not. I think this difference is enough to warrant a break from the list API.

Are you sure? From its interface Text looks exactly like a list of Chars to me.

Right, that's a very common misunderstanding of Unicode. A Unicode code point (type Char) does not correspond 1:1 with the human notion of a character. It would be nice if it did, but unfortunately it is not something we can ignore. Because of this it is better not to think of operations on individual Chars but on short sequences of Chars. In any case, when processing text (even ASCII where Chars do match characters) many of the most common operations that you want are substring not element based.

I believe Roman is referring to the Text API, which does indeed look a lot like the list API specialized to Char, with relatively few exceptions. The above would be an argument against including any of the functions with Char parameters, but a high proportion of them do.

<musing> I almost wonder if it would be worth it to define a new type, Character, which does correspond 1:1 to the human notion of a "character" (being intentionally vague about what exactly that means). Then we could have that Text is a vector/list/sequence of Characters, and give it the appropriate interface for being thought of that way. Of course, under the covers, Character would just be a newtype of Text[1] and so the bulk of text/text-icu implementation would need no changes. At least, it seems like that might make it possible for us to get out of this impasse about the text library matching vector/list/sequence APIs when Text is not a vector/list/array of Char. Also, it helps to codify what we mean by "a short sequence of Chars", which could possibly allow for some simplifying assumptions for the algorithms being used (since often there are better (X,X)->Y algos available when we know one of the X is much smaller than the other). </musing> [1] Using a type alias seems like it'd be too easy to break the API idealization. -- Live well, ~wren

On 20/10/2010, at 00:35, Duncan Coutts wrote:

On 19 October 2010 22:08, Roman Leshchinskiy wrote:

...

On 19/10/2010, at 15:22, John Lato wrote:

...

I think there's a significant difference between vector and text, namely a Vector is conceptually the same as a list/1D array, while a Text is not. I think this difference is enough to warrant a break from the list API.

Are you sure? From its interface Text looks exactly like a list of Chars to me.

Right, that's a very common misunderstanding of Unicode. A Unicode code point (type Char) does not correspond 1:1 with the human notion of a character.

True. But I didn't say it's a list of characters. It's a list of Chars, i.e., of Unicode code points.

It would be nice if it did, but unfortunately it is not something we can ignore. Because of this it is better not to think of operations on individual Chars but on short sequences of Chars. In any case, when processing text (even ASCII where Chars do match characters) many of the most common operations that you want are substring not element based.

So this is why a list/vector API is not necessarily appropriate for text.

But text *does* have a list API. It doesn't get any "listier" than uncons, foldr and unfoldr. From my point of view, if these functions make sense for the data type then it's a list. If it's not a list then these functions don't make sense and shouldn't be provided. Roman

On October 19, 2010 19:35:33 Duncan Coutts wrote:

Right, that's a very common misunderstanding of Unicode. A Unicode code point (type Char) does not correspond 1:1 with the human notion of a character. It would be nice if it did, but unfortunately it is not something we can ignore. Because of this it is better not to think of operations on individual Chars but on short sequences of Chars. In any case, when processing text (even ASCII where Chars do match characters) many of the most common operations that you want are substring not element based.

I read the wikipedia article on code points, but still do not feel I have a firm grasp as to what exactly you are referring to. If you have a few minutes, would you mind providing a short example to clarify this with a specific example (e.g., a specific code point that gives issues with a 1:1 model and what those issues are). Thanks! -Tyson

On Wed, Oct 20, 2010 at 05:28:15PM +0200, Johan Tibell wrote:

If you process a string code point by code point you might mistakenly confuse a plain "a" (A) with a "å" (A-RING *or* A + COMBINING RING).

But isn't that what text does here?: Data.Text Data.Text.IO> let t = pack "z\x0061\x030A\x0061z" Data.Text Data.Text.IO> putStrLn t zåaz Data.Text Data.Text.IO> putStrLn (replace (pack "a") (pack "y") t) zẙyz Thanks Ian

On Wed, Oct 20, 2010 at 9:52 AM, Johan Tibell wrote:

I think the right thing to do here is to perform normalization first but I'm not sure.

Hi, friendly neighbourhood Unicode expert here. Yes, in the case Ian cites, you want to perform normalization before doing the replacement. The behaviour he demonstrates is normal, expected, and consistent with the standard. For more details, see http://unicode.org/reports/tr15

On Oct 20, 2010, at 19:44, Ian Lynagh wrote:

Johan wrote:

...

If you process a string code point by code point you might mistakenly confuse a plain "a" (A) with a "å" (A-RING *or* A + COMBINING RING).

But when characters and codepoints are 1:1, you /can/ process code point by code point.

Am I missing something?

AFAIK there are scripts that have so many combinations that Unicode does not have a single codepoints for each character. In Arabic you can have one of 5 vowel signs on each of the 28 letters. But Unicode does not provide 5*28 codepoints for the combinations. That is probably the reason for have these combined characters. Mac OS tries to take all the characters into as many codepoints as possible whereas Windows tries to merge them as much as possible. I don't think there is a good semantics for replace without knowing what (normal) form you're working on. Normally, search/replace and sorting on Unicode are specialized algorithms that cannot be reduces to simple substitutions or permutations. So I suggest to just provide functions on codepoints and let the user struggle with the rest. Cheers, Axel

On October 20, 2010 15:45:44 Axel Simon wrote:

AFAIK there are scripts that have so many combinations that Unicode does not have a single codepoints for each character. In Arabic you can have one of 5 vowel signs on each of the 28 letters. But Unicode does not provide 5*28 codepoints for the combinations. That is probably the reason for have these combined characters.

Mac OS tries to take all the characters into as many codepoints as possible whereas Windows tries to merge them as much as possible. I don't think there is a good semantics for replace without knowing what (normal) form you're working on. Normally, search/replace and sorting on Unicode are specialized algorithms that cannot be reduces to simple substitutions or permutations.

Thanks to everyone for the examples. Given that not all combined characters can be reduced to a single code point (from your first paragraph), it would seem that MacOS normalization has a conceptual advantage over Windows normalization. Specifically, it is appealing that the normalized string is in some sense less complex in that it only contains elementary codepoints (ones that can't be further decomposed) and compositions. The other would still contain a mix. Am I correct then in understanding that, from the view of strings as a vector/list of elementary chars, the elementary chars would actually have to be a codepoint plus an arbitrary number of additional composition codepoints in order to correspond well to the human notion of a character. This then doesn't map well onto the existing vector/list style interfaces because this elementary char type is not a simple enumeration to be treated atomically. Operations would actually need to frequently look inside it (e.g., replace base codepoints irrespective of the compositional codepoints). Cheers! -Tyson

On Wed, 2010-10-20 at 11:11 -0400, Tyson Whitehead wrote:

On October 19, 2010 19:35:33 Duncan Coutts wrote:

...

Right, that's a very common misunderstanding of Unicode. A Unicode code point (type Char) does not correspond 1:1 with the human notion of a character. It would be nice if it did, but unfortunately it is not something we can ignore. Because of this it is better not to think of operations on individual Chars but on short sequences of Chars. In any case, when processing text (even ASCII where Chars do match characters) many of the most common operations that you want are substring not element based.

I read the wikipedia article on code points, but still do not feel I have a firm grasp as to what exactly you are referring to.

If you have a few minutes, would you mind providing a short example to clarify this with a specific example (e.g., a specific code point that gives issues with a 1:1 model and what those issues are).

Combining characters are the major one. These are things like accents, but there are many more of them in some other languages. For most of the European languages there are both all-in-one code points that combine the base character with the extra mark (because those already existed in previous character sets), but for many other languages the canonical form is made up of multiple code points (and not necessarily just 2). So if you're searching for a particular "character" then searching for a single Char is not sufficient, you need to search for a short sequence of Chars. Duncan