Why can one character take several bytes in UTF-8?
Quick answer
UTF-8 encodes Unicode code points using a variable number of bytes: code points below 128 use one byte, while other code points use two, three, or four. That means a string’s byte size depends on the code points it contains, not just their number. To work it out, add the UTF-8 byte counts for each code point in the text. [S1]
The question, the details, the next step.
Why isn’t a character always one byte? I thought a byte was the basic unit for storing text.
A byte is a unit of storage, but it does not have to hold one complete character. UTF-8 encodes Unicode code points as bytes. A code point below 128 uses one byte; other code points use sequences of two, three, or four bytes. This variable-length scheme means the byte size depends on the text being encoded. [S1]
So “one character equals one byte” is not a general rule for UTF-8. Some code points take more than one byte.
Can you show how the byte counts compare and add up for a short string?
Here is an illustrative comparison using one code point from each UTF-8 byte-length group:
| Code point’s UTF-8 length | Bytes |
|---|---|
| Below 128 | 1 |
| In a two-byte sequence | 2 |
| In a three-byte sequence | 3 |
| In a four-byte sequence | 4 |
For a string containing one code point from each group, the total is 1 + 2 + 3 + 4 = 10 bytes. That string has four code points, but its UTF-8 encoding occupies ten bytes. The count depends on the particular code points, not just the number of them. [S1]
These are byte-length categories, rather than names of particular characters; the exact text determines which category applies to each code point.
So how do I count the bytes for text I actually have?
Start with the exact text and confirm that UTF-8 is the encoding you need. Then determine the UTF-8 byte length of each code point and add those lengths. Each contributes one, two, three, or four bytes, according to its encoding. [S1]
For instance, if your text has three code points with UTF-8 lengths of one, one, and three bytes, its total is 1 + 1 + 3 = 5 bytes. This is a worked example of the addition method; use the actual text’s code points to get its byte count.
Why can’t I just multiply the number of characters by one byte? And what should I check when a limit is involved?
Multiplying by one assumes every code point uses one byte. That assumption fails whenever the text includes code points that UTF-8 encodes in two, three, or four bytes. [S1]
For a storage-size question, make sure you have the exact text and the required encoding, then count its encoded bytes. If you are checking a limit, read how that limit is defined: a byte limit is not the same measure as a count of code points. Don’t infer the byte size from the visible length of the text.
A practical checklist
Use the exact text you need to measure.
Confirm that UTF-8 is the required encoding.
Count bytes in the encoded text, rather than assuming one byte per code point.
Add the byte lengths for the text’s code points.
Check whether a stated limit is measured in bytes or another unit.
A few more useful answers
How many bytes can a UTF-8 code point use?
One, two, three, or four bytes. Code points below 128 use one byte; other code points use sequences of two, three, or four bytes. [S1]
Will two strings with the same number of code points always use the same number of UTF-8 bytes?
No. Their code points may have different UTF-8 byte lengths, so the totals can differ even when the code-point counts are the same. [S1]
What is the quickest way to estimate a string’s UTF-8 size?
Identify the UTF-8 byte length of each code point in the exact text and add the lengths. The result depends on the text, not only its code-point count. [S1]
Sources
References used for this guide.
- [S1] Unicode HOWTO — Python documentation docs.python.org · Checked