On Tue, Aug 17, 2010 at 10:08 AM, Ketil Malde <ketil@malde.org> wrote:

Benedikt Huber <benjovi@gmx.net> writes:

> Despite of all this, I think the performance of the text
> package is very promising, and hope it will improve further!

I agree, Data.Text is great. Unfortunately, its internal use of UTF-16
makes it inefficient for many purposes.

[..]

>From a quick glance, it appears that utf8-string is the most complete
and well maintained of the crowd, but I could be wrong. It'd be nice if
a similar effort as Data.Text has seen could be applied to
e.g. utf8-string, to produce a similarly efficient and effective library
and allow the deprecation of the others. IMO, this could in time
replace .Char8 as the default ByteString string representation.
Hackathon, anyone?

Let me ask the question a different way: what are the motivations for having the text package use UTF-16 internaly? I know that some system APIs in Windows use it (at least, I think they do), and perhaps it's more efficient for certain types of processing, but overall do those benefits outweigh all of the reasons for UTF-8 pointed out in this thread?

Michael