Recent chats
ASKGEEZ Q&A GUIDETechnology & devices

What is a Unicode surrogate, and why might it appear in a programming-language string?

Quick answer

A Unicode surrogate is one of two UTF-16 code units used together to represent a code point outside the Basic Multilingual Plane. In JavaScript, string indexes and length count those 16-bit units, so one such code point can take two indexes even though it is one code point. Iteration can expose code points, but that still does not necessarily count visual grapheme clusters. Other languages may use different string models. [S1]

LET’S WORK THROUGH IT

The question, the details, the next step.

Question

What is a Unicode surrogate, and why might it appear in a programming-language string? I’m confused because a symbol on screen seems to be one character, but a string index or length says two.

Geez

A surrogate is a UTF-16 code unit used as part of a pair: a high surrogate and a low surrogate together represent one Unicode code point outside the Basic Multilingual Plane. In JavaScript, strings are sequences of 16-bit UTF-16 code units, and indexes and length count those units. So a supplementary code point can occupy two string indexes while representing one code point. [S1]

A simplified diagram:

One code point outside the BMP:
        ┌─────────────────────┐
        │     one code point  │
        └──────────┬──────────┘
                   │ represented in UTF-16 by
          ┌────────┴────────┐
          │                 │
   high-surrogate     low-surrogate
   code unit 1        code unit 2
          │                 │
        index 0           index 1

JavaScript length: 2 code units
Code-point count:  1

That’s why “string position,” “code point,” and “what looks like one character” can produce different counts. This explanation is specifically about JavaScript’s string model; it should not be assumed to describe every language. [S1]

Follow-up question

Can I see the mismatch with a short example, and what should I count when I need a position?

Geez

For a JavaScript string containing one supplementary symbol, inspect both the UTF-16 length and the number of values produced by string iteration:

const s = "\u{1F600}";

console.log(s.length);       // 2: UTF-16 code units
console.log([...s].length);  // 1: values yielded by iteration

The example illustrates the distinction: JavaScript length counts code units, while the string iterator yields code points. [S1] Choose the measurement that matches your task. If you need a JavaScript string index, work in code-unit positions; if you need a count of code points, iterate the string. Don’t treat either result as an automatic count of visible symbols: code-point iteration does not necessarily yield visual grapheme clusters. [S1]

Also be precise about the word “character.” It might mean a UTF-16 code unit, a Unicode code point, or a user-perceived grapheme cluster. State which one you mean before comparing counts or positions.

Follow-up question

What does it mean if I encounter a lone surrogate instead of a pair?

Geez

A lone surrogate is an unpaired UTF-16 code unit. In JavaScript, it does not represent a Unicode character, unlike the high-and-low pair that represents one supplementary code point. [S1]

If you find one, first record the exact string and the operation that exposed it. Check whether your code is viewing individual UTF-16 positions or iterating values, and whether the unit has a matching partner. Don’t infer from a displayed mark or a count alone that the underlying string contains a valid pair. How a particular library, file format, or other programming language handles a lone surrogate depends on that context; the JavaScript description here does not establish those behaviors. [S1]

Follow-up question

Can you give me a reusable way to explain this mismatch in a bug report or code review?

Geez

Use a short note that names the representation and the measurement, then include a minimal example. For instance:

The value is one code point outside the Basic Multilingual Plane, represented in JavaScript as two UTF-16 code units. Therefore `length` and string indexes count 2 here, while string iteration yields 1 code point. This is not necessarily a count of displayed grapheme clusters.

When reporting a real issue, replace “the value” with a safe placeholder or an appropriate test string, and say which operation produced the surprising count. If the code is not JavaScript, verify that language’s string model before using this explanation; JavaScript behavior does not establish how other languages count or index strings. [S1]

A practical checklist

  • Name the environment before applying JavaScript’s UTF-16 explanation.

  • Decide whether you need a code-unit index, a code-point count, or a visual grapheme count.

  • For JavaScript, compare length with string iteration when investigating a supplementary code point.

  • When inspecting a suspicious unit, check whether it has a matching surrogate partner.

  • In bug reports, include the operation, the measurement used, and a minimal example.

A few more useful answers

Does a JavaScript string length of 2 always mean there is one supplementary code point?

No. length counts UTF-16 code units, not code points, so the number alone does not tell you how those units are grouped. Inspect the string and the operation you are using.

Does iterating a JavaScript string count what a person sees as one character?

Not necessarily. The string iterator yields code points, but code points do not necessarily correspond one-to-one with visual grapheme clusters. [S1]

Do surrogate pairs explain string indexing in every programming language?

No. The cited behavior describes JavaScript’s UTF-16 strings, indexes, and length; it does not establish the string model used by other languages. [S1]

Sources

References used for this guide.

  1. [S1] JavaScript String: UTF-16 code units, surrogate pairs, and indexing
    developer.mozilla.org · Checked

YOUR QUESTION. YOUR DETAILS.

Let’s work through your situation.

Get a tailored answer or help with your next task. Starts at $1, with a quote before you pay.

Ask your own question
← Back to Geez at Work