5. Charsets and encodings¶
5.1. Encodings¶
There are many encodings around the world. Before Unicode, each manufacturer invented its own encoding to fit its client market and its usage. Most encodings are incompatible on at least one code, with some exceptions. A document stored in ASCII can be read using ISO 8859-1 or UTF-8, because ISO-8859-1 and UTF-8 are supersets of ASCII. Each encoding can have multiple aliases, examples:
ASCII: US-ASCII, ISO 646, ANSI_X3.4-1968, …
ISO-8859-1: Latin-1, iso88591, …
UTF-8: utf8, UTF_8, …
Unicode is a charset and it requires a encoding. Only encodings of the UTF family are able to encode and decode all Unicode code points. Other encodings only support a subset of Unicode codespace. For example, ISO-8859-1 are the first 256 Unicode code points (U+0000—U+00FF).
This book presents the following encodings: ASCII, cp1252, GBK, ISO 8859-1, ISO 8859-15, JIS, UCS-2, UCS-4, UTF-8, UTF-16 and UTF-32.
5.2. Popularity¶
The three most common encodings are, in chronological order of their creation: ASCII (1968), ISO 8859-1 (1987) and UTF-8 (1996).
Google posted an interesting graph of the usage of different encodings on the web: Unicode nearing 50% of the web (Mark Davis, january 2010). Because Google crawls a huge part of the web, these numbers should be reliable. In 2001, the most used encodings were:
1st (56%): ASCII
2nd (23%): Western Europe encodings (ISO 8859-1, ISO 8859-15 and cp1252)
3rd (8%): Chinese encodings (GB2312, …)
and then come Korean (EUC-KR), Cyrillic (cp1251, KOI8-R, …), East Europe (cp1250, ISO-8859-2), Arabic (cp1256, ISO-8859-6), etc.
(UTF-8 was not used on the web in 2001)
In december 2007, for the first time: UTF-8 becomes the most used encoding (near 25%). In january 2010, UTF-8 was close to 50%, and ASCII and Western Europe encodings were near 20%. The usage of other encodings doesn’t change.
5.3. Encodings performances¶
Complexity of getting the n th character in a string, and of getting the length in character of a string:
\(O(1)\) for 7 and 8 bit encodings (ASCII, ISO 8859 family, …), UCS-2 and UCS-4
\(O(n)\) for variable length encodings (e.g. the UTF family)
5.4. Examples¶
Encoding |
A (U+0041) |
é (U+00E9) |
€ (U+20AC) |
U+10FFFF |
---|---|---|---|---|
ASCII |
|
— |
— |
— |
ISO-8859-1 |
|
|
— |
— |
UTF-8 |
|
|
|
|
UTF-16-LE |
|
|
|
|
UTF-32-BE |
|
|
|
|
— indicates that the character cannot be encoded.
5.5. Handle undecodable bytes and unencodable characters¶
5.5.1. Undecodable byte sequences¶
When a byte string is decoded, the decoder may
fail to decode a specific byte sequence. For example, 0x61 0x62 0x63 0xE9
is not decodable from ASCII nor UTF-8, but it is decodable from
ISO 8859-1.
Some encodings are able to decode any byte sequences. All encodings of the ISO-8859 family have this property, because all of the 256 code points of these 8 bits encodings are assigned.
5.5.2. Unencodable characters¶
When a character string is encoded to a character set smaller than the Unicode character set (UCS), a character may not be encodable. For example, € (U+20AC) is not encodable to ISO 8859-1, but it is encodable to ISO 8859-15 and UTF-8.
5.5.3. Error handlers¶
There are different choices to handle undecodable byte sequences and unencodable characters:
strict: raise an error
ignore
replace by ? (U+003F) or � (U+FFFD)
replace by a similar glyph
escape: format its code point
etc.
Example of the “abcdé” string encoded to ASCII, é (U+00E9) is not encodable to ASCII:
Error handler |
Output |
---|---|
strict |
raise an error |
ignore |
|
replace by ? |
|
replace by a similar glyph |
|
escape as hexadecimal |
|
escape as XML entities |
|
5.5.4. Replace unencodable characters by a similar glyph¶
By default, WideCharToMultiByte()
replaces unencodable characters by
similarly looking characters. The normalization to NFKC
and NFKD does also such operation. Examples:
Character |
Replaced by |
||
---|---|---|---|
U+0141, latin capital letter l with stroke |
Ł |
L |
U+004C, latin capital letter l |
U+00B5, micro sign |
µ |
μ |
U+03BC, greek small letter mu |
U+221E, infinity |
∞ |
8 |
U+0038, digit eight |
U+0133, latin small ligature ij |
ij |
ij |
{U+0069, U+006A} |
U+20AC, euro sign |
€ |
EUR |
{U+0045, U+0055, U+0052} |
∞ (U+221E) replaced by 8 (U+0038) is the worst example of the method: these two characters have completely different meanings.
5.5.5. Escape the character¶
Python “backslashreplace” error handler uses \xHH
, \uHHHH
or
\UHHHHHHHH
where HHH…H is the code point formatted in hexadecimal. PHP
“long” error handler uses U+HH
, U+HHHH
or encoding+HHHH
(e.g.
JIS+7E7E
).
PHP “entity” and Python “xmlcharrefreplace” error handlers escape
the code point as an HTML/XML entity. For example, when U+00E9 is encoded to
ASCII: it is replaced by é
in PHP and é
in Python.
5.6. Other charsets and encodings¶
There are much more charsets and encodings, but it is not useful to know them. The knowledge of a good conversion library, like iconv, is enough.