This confusion is common enough that the mistaken encoding has actually been standardized so that at least broken programs can be made to interoperate. An example of this confusion can be seen in the encoding CESU-8, which is what you get if you convert UTF-16 text into UTF-8 by just encoding each half of a surrogate pair as a separate character (using 6 bytes per character three bytes to encode each half of the surrogate pair in UTF-8), instead of decoding the pair to its codepoint and encoding that into UTF-8. In UTF-16, however, 4 byte characters are fairly uncommon, so it's a lot easier to make fixed width assumptions and have everything work until you run into a corner case that you didn't catch. Both of them are variable width encodings, and so have the complexity that entails. UTF-8 is better in almost every way than UTF-16. $ iconv -f UTF-8 -t UTF-16 nhk.html > nhk.16.html If I transcode it to UTF-16, it grows to almost twice its original size: Even for a lot of text in that range, UTF-8 winds up being comparable, because the markup of that text (HTML, XML, RTF, or what have you) is all in the ASCII range, for which UTF-8 is half the size of UTF-16.įor example, if I pick a random web page in Japanese, the home page of nhk.or.jp, it is encoded in UTF-8. The only range in which UTF-16 is more efficient than UTF-8 is for characters from U+07FF to U+FFFF, which includes Indic alphabets and CJK. For characters outside the Basic Multilingual Plane, both UTF-8 and UTF-16 use 4 bytes per character, so it's a wash there. For characters between the ASCII range and U+07FF (which includes Latin Extended, Cyrillic, Greek, Arabic, and Hebrew), UTF-8 also uses two bytes per character, so it's a wash. The following is more of an expanded response to Liv's answer than an answer on its own it's a description of why UTF-8 is preferable to UTF-16 even for CJK content.įor characters in the ASCII range, UTF-8 is more compact (1 byte vs 2) than UTF-16. UTF-8 is thus the only common encoding that everything is required to support.
UTF-8 and Windows-1252 are the only encodings required to be supported by browsers, and UTF-8 and UTF-16 are the only encodings required to be supported by XML parsers. Īuthoring tools should default to using UTF-8 for newly-created documents. Conformance checkers may advise authors against using legacy encodings. UTF-8 is the preferred encoding for the web from the HTML5 draft standard:Īuthors are encouraged to use UTF-8. The best choice for this purpose is UTF-8. If you want to support a variety of languages for web content, you should use an encoding that covers the entire Unicode range.
0 kommentar(er)
