On Wed, Feb 04, 2009 at 01:46:16PM +0000, Duncan Coutts wrote:
On Wed, 2009-02-04 at 13:31 +0000, Simon Marlow wrote:
Yes; the utf16 and utf32 encodings accept a BOM (and generate a BOM in write mode). This caused interesting bugs when doing re-decoding after switching encodings, because the BOM constitutes state in the decoder, which means that decoding is not necessarily repeatable unless you save the state (which iconv doesn't provide a way to do).
Are there other encodings that have this kind of state? If so, then they might be restricted to NoBuffering at least when switching encodings.
Yes, I believe there are some Asian encodings that are stateful.
Yes, ISO-2022 encodings, like ISO-2022-JP, have a number of states and switch between them with escape sequences. They're a nightmare.