What happens when a text decoder encounters bytes that are not valid UTF-8?

It means the decoder encountered a byte sequence it could not interpret as valid UTF-8 under its current settings. That tells you something about the input and decoding operation, but not by itself why the bytes are there. For example, the data might have been altered, cut off, or interpreted using an encoding that does not match how it was written; those are possibilities to investigate, not conclusions you can draw from the error alone.

In Python’s standard codecs, decoding defaults to strict, which raises a UnicodeError when it encounters malformed data. Other handlers change the response: replace inserts U+FFFD, ignore discards the problematic data, and backslashreplace displays byte escapes. [S1] The specific decoder and its configured handler matter, so don’t assume every program responds the same way.

Usually, treat � as a warning marker, not as a recovered character. In Python, the replace handler inserts U+FFFD for malformed data. [S1] That replacement does not identify the original intended character: multiple possible inputs or causes may be consistent with the same marker, and the marker alone does not preserve the missing bytes.

If you still have the original byte data, keep it unchanged while investigating. Make a separate copy for experiments. Compare how that copy behaves with the decoder’s current settings and, if you know how the data was created, check whether the chosen encoding matches that process. Avoid editing the replacement marker into a guessed character unless you can verify the intended text from another reliable copy.

Yes—separate inspection from repair. Start with the original bytes, and use a decoding mode that makes problems visible rather than silently dropping data. In Python, strict raises an error, while backslashreplace displays byte escapes. Python’s surrogateescape can preserve certain undecodable bytes for a later round trip if you use the same handler when encoding again. [S1] That is a preservation technique, not a way to determine the intended text.

For example, assuming data is the original byte string you are investigating, you could compare these two attempts:

data.decode("utf-8", errors="strict")

data.decode("utf-8", errors="backslashreplace")

The first raises an error if it finds malformed data; the second displays escapes for it. [S1] Treat the second result as a diagnostic view, not proof that the bytes have been repaired. This example describes Python behavior only; it does not establish what another application will do or what the original author meant.

Record the evidence and the goal separately. Note where the bytes came from, which decoder and encoding you tried, the exact error or visible marker, and whether you still have an untouched copy. Then decide whether your goal is to display readable text, retain the bytes for a later round trip, or produce a corrected document. Those goals may call for different handling.

A practical sequence is: preserve the original; reproduce the issue on a copy; record the decoder’s settings; inspect errors without silently discarding data; and verify any proposed correction against a trusted version or the source that produced the file. If there is no independent evidence for a missing character, leave it marked as unresolved rather than presenting a guess as recovered text.