John Meacham wrote:
I would highly highly recommend using utf8. inventing new formats without very clear and pervasive benefits is just not good practice and I wouldn't want to see it in standard libraries.
I still think it should not matter what the library uses *internally*. The only way the user can see the encoding is through unsafeTo/FromByteString functions. However, I have decided to make a UTF-8 version of the library. Fortunately the only things that change are the encoding functions, which means that the library could potentially be used for all sorts of variable length encodings. This new version is available from: http://twan.home.fmf.nl/compact-string/ By the way, many more functions are implemented now, and there are also QuickCheck tests for everything.
not to mention that utf8 was designed so things like sorting a raw bytestring with utf8 in it produces the exact same result as decoding it, then sorting it. a _very_ large win for the 'Ord' instance for CompactString.
You are right, using UTF-8 here makes some things a lot easier. It is quite possible that UTF-8 is a net win over my custom encoding in most common use cases. Twan