|
The Plan 9 character set and representation are based on the Unicode
Standard and on the ISO multibyte UTF–8 encoding (Universal Character
Set Transformation Format, 8 bits wide). The Unicode Standard
represents its characters in 20 bits; UTF–8 represents such values
in an 8–bit byte stream. Throughout this
manual, UTF–8 is shortened to UTF.
In Plan 9, a rune is a 32–bit quantity representing a Unicode character.
Internally, programs may store characters as runes. However, any
external manifestation of textual information, in files or at
the interface between programs, uses a machine–independent, byte–stream
encoding called UTF.
UTF is designed so the 7–bit ASCII set (values hexadecimal 00 to
7F), appear only as themselves in the encoding. Runes with values
above 7F appear as sequences of two or more bytes with values
only from 80 to FF.
The UTF encoding of the Unicode Standard is backward compatible
with ASCII: programs presented only with ASCII work on Plan 9
even if not written to deal with UTF, as do programs that deal
with uninterpreted byte streams. However, programs that perform
semantic processing on ASCII graphic characters must convert
from UTF to runes in order to work properly with non–ASCII input.
See rune(2).
Letting numbers be binary, a rune x is converted to a multibyte
UTF sequence as follows:
001. x in [00000000.00000000.0bbbbbbb] → 0bbbbbbb
010. x in [00000000.00000bbb.bbbbbbbb] → 110bbbbb, 10bbbbbb
011. x in [00000000.bbbbbbbb.bbbbbbbb] → 1110bbbb, 10bbbbbb, 10bbbbbb
100. x in [0000bbbb.bbbbbbbb.bbbbbbbb] → 11110bbb, 10bbbbbb, 10bbbbbb
10bbbbbb
Conversion 001 provides a one–byte sequence that spans the ASCII
character set in a compatible way. Conversions 010, 011 and 100
represent higher–valued characters as sequences of two to four
bytes with the high bit set. When there are multiple ways to encode
a value, for example rune 0, the shortest encoding is
used.
In the inverse mapping, any sequence except those described above
is incorrect and is converted to rune hexadecimal FFFD.
|