Chris Kuklewicz wrote:

Can I be among the first to ask that any Unicode variant of ByteString use a recognized encoding?

<snip>

In reading all the poke/peek function I did not see anything that your tag bits accomplish that the tag bits in utf-8 do not, except that you want to write only a single routine for the poke/peek forwards and backwards operations instead of two routines. It is definitely more compact in the worst case, and more "Once And Only Once", but at a very high cost of incompatibility.

The reason for inventing my own encoding is that it is easier to use and takes less space than UTF-8. The only advantage UTF-8 has is that it can be read and written directly. I guess this is a trade off, faster manipulation and smaller storage compared to simpler and faster io. I have not benchmarked it either way, so it is just guesswork for now. Fortunately the entire library can be easily converted to use a different encoding by just changing the peekChar/pokeChar functions.

One of the biggest wins with with a Unicode ByteString will be the ability to transfer the buffer directly to and from the disk and network. Your code will always need the data to be rewritten both incoming and outgoing.

The most ideal case would be the ability to load different encodings via import statements while using the same API.

I was hoping that there would be only a single string type, with different encodings handled by functions:

encode :: CompactString -> ByteString decode :: ByteString -> CompactString

This is important if it is not know beforehand how a file is encoded. For example on windows Unicode files are either UTF-8 or UTF-16, identified by a byte order mark. Twan

John Meacham wrote:

I would highly highly recommend using utf8. inventing new formats without very clear and pervasive benefits is just not good practice and I wouldn't want to see it in standard libraries.

I still think it should not matter what the library uses *internally*. The only way the user can see the encoding is through unsafeTo/FromByteString functions. However, I have decided to make a UTF-8 version of the library. Fortunately the only things that change are the encoding functions, which means that the library could potentially be used for all sorts of variable length encodings. This new version is available from: http://twan.home.fmf.nl/compact-string/ By the way, many more functions are implemented now, and there are also QuickCheck tests for everything.

not to mention that utf8 was designed so things like sorting a raw bytestring with utf8 in it produces the exact same result as decoding it, then sorting it. a _very_ large win for the 'Ord' instance for CompactString.

You are right, using UTF-8 here makes some things a lot easier. It is quite possible that UTF-8 is a net win over my custom encoding in most common use cases. Twan

shelarcy wrote:

Hello Twan,

On Mon, 05 Feb 2007 08:46:35 +0900, Twan van Laarhoven <twanvl@gmail.com> wrote:

...

I would like to announce my attempt at making a Unicode version of Data.ByteString. The library is named Data.CompactString to avoid conflict with other (Fast)PackedString libraries.

How about add abstract layer?

Spencer Janssen tried to provied abstract layer for Unicode ByteString, last year's summer of code project. It has no Unicode support. But it supplied a good layer, Stringable class.

http://code.google.com/soc/haskell/appinfo.html?csaid=B934AEBE95120AB2 http://darcs.haskell.org/SoC/fps-soc/ http://darcs.haskell.org/SoC/fps-soc-aug21/

...

The library uses a variable length encoding (1 to 3 bytes) of Chars into Word8s, which are then stored in a ByteString. The structure is very much based on Data.ByteString, most of the implementation is copied from there. Hopefully this means that fusion rules could be copied as well.

UTF-8 also uses 4 to 6 byte encodings now. CJK Unified Ideographs Extension B, Tai Xuan Jing Symbol and Music Symbol, etc ... use 4 byte encoding.

Looking at several sources, it seems you are incorrect. Haskell Char go up to Unicode 1114111 (decimal) or 0x10ffff Hexidecimal). These are encoded by UTF-8 in 1,2,3,or 4 bytes. CJK Unified Ideographs Extension B starts at 131072 or 0x20000 Tai Xuan Jing Symbols start at 119552 or 0x1d300 These are all within the official utf-8 encoding scheme.

Many Hasekll UTF-8 libraries doesn't support over 3 byte encodings.

UTF-8 uses 1,2,3, or 4 bytes. Anything that does not support 4 bytes does not support UTF-8

But Takusen's implementation support it correctly.

The Takusen does have unreachable dead code to serialize Char as (ord c :: Int) up to 31 bits into as many as 6 bytes. But it does decode up to 6 bytes to 31 bits and try to "chr" this from Int to Char. Decoding that many bits is not consistent with the UTF-8 standard.

http://darcs.haskell.org/takusen/Foreign/C/UTF8.hs http://www.haskell.org/pipermail/libraries/2007-February/006841.html

How about support 4 to 6 byte encodings?

UTF-8 is a 4 byte encoding. There is no valid UTF-8 5 or 6 byte encoding.

Best Regards,

Alistair Bayley writes:

On 05/02/07, Chris Kuklewicz <haskell@list.mightyreason.com> wrote:

...

UTF-8 is a 4 byte encoding. There is no valid UTF-8 5 or 6 byte encoding.

Chris is right here, in that Takusen's decoder is incorrect w.r.t. the standard, in allowing up to 6 bytes to encode a single char.

<snip>

There's nothing stopping the Unicode consortium from expanding the range of codepoints, is there? Or have they said that'll never happen?

I believe they have. In particular, UTF-16 only supports code points up to 10FFFF. From <http://en.wikipedia.org/wiki/Universal_Character_Set>:

the UCS stops at 10FFFF and ISO/IEC 10646 has stated that all future assignments of characters will also take place in that range [...] ISO 10646 was limited to contain as many characters as could be encoded by UTF-16 and no more, that is, a little over a million characters instead of over 2,000 million -- David Menendez <zednenem@psualum.com> | "In this house, we obey the laws <http://www.eyrie.org/~zednenem> | of thermodynamics!"

On Tue, Feb 06, 2007 at 03:16:17PM +0900, shelarcy wrote:

I'm afraid that its fantasy is broken again, as no surrogate pair UCS-2 cover all language that is trusted before Europe and America people.

UCS-2 is a disaster in every way. someone had to say it. :) everything should be ascii, utf8 or ucs-4 or migrating to it. John -- John Meacham - ⑆repetae.net⑆john⑈

On Thu, 2007-02-08 at 17:01 -0800, John Meacham wrote:

UCS-2 is a disaster in every way. someone had to say it. :)

UCS-2 has been deprecated for many years.

everything should be ascii, utf8 or ucs-4 or migrating to it.

UCS-4 has also been deprecated for many years. The main forms of Unicode in use are UTF-16, UTF-8, and (less frequently) UTF-32. On Feb 9, 2007, at 6:02 AM, Duncan Coutts wrote:

Apparently UTF-16 (which is like UCS-2 but covers all code points) is a good internal format. It is more compact than UTF-32 in almost all cases and a less complex encoding than UTF-8. So it's faster than either UTF-32 (because of data-density) or UTF-8 (because of the encoding complexity). The downside compared to UTF-32 is that it is a more complex encoding so the code is harder to write (but apparently it doesn't affect performance much because characters outside the BMP are very rare).

UTF-16 is never less compact than UTF-32. The worst case of UTF-16 is that it is the same size as UTF-32. This only happens when a string consists entirely of characters from the supplementary planes.

The ICU lib uses UTF-16 internally I believe, though I can't at the moment find on their website the bit where they explain why the use UTF-16 rather than -8 or -32.

http://icu.sourceforge.net/userguide/unicodeBasics.html UTF-16 is the native Unicode encoding for ICU, Microsoft Windows, and Mac OS X.

Btw, when it comes to all these encoding names, I find it helpful to maintain the fiction that there's no such thing (any more) as UCS-N, there's only UTF-8, 16 and 32. This is also what the Unicode consortium tries to encourage.

It's not a fiction. :-) UCS-2 and UCS-4 are *deprecated*, by the merger between Unicode and ISO 10646 that limited the code point space to [0..0x10FFFF]. In addition to UTF-8, UTF-16, and UTF-32, there's SCSU, a compressed form used in some applications. See: http://www.unicode.org/reports/tr17/

My view is that we should just provide all three: Data.PackedString.UTF8 Data.PackedString.UTF16 Data.PackedString.UTF32

that all provide the same interface. This wouldn't actually be too much code to write since most of it can re-use the streams code, so the only difference is the single implementation per-encoding of: stream :: PackedString -> Stream Char unstream :: Stream Char -> PackedString

and then get fusion for free of course.

I agree that all three should be supported. UTF-16 is used in Windows and Mac OS X, and UTF-8 is widely used on Unix platforms (and at the BSD level of Mac OS X). UTF-32 matches the Char type in Haskell, and is used for wchar_t on some platforms. SCSU can be handled the same as a non-Unicode encoding (e.g., like GB2312 or Shift JIS). Note that some Unicode algorithms require the ability to back up in a stream of code points, so that may be a consideration in the design (maybe they could be implemented in Haskell in a way that doesn't require that; I'm still learning, so I'm not sure yet). And regular expression processing requires essentially random access (same Haskell-fu considerations apply).

I have proposed this task as an MSc project in my department. Hopefully we'll get a student to pick this up.

I hope so! ICU has a BSD/MIT-style license, so feel free to steal whatever is appropriate from there. I would love to see Haskell support Unicode operations like locale-sensitive collation, text boundary analysis, and more on both [Char] and packed strings. Deborah