Optimize tokenizer string scanning with pre-computed escape-end positions

[alanzabihi]

Hypothesis

When the tokenizer encounters a quoted string, it scans forward character by character checking for the closing quote and handling backslash escapes. The escape handling involves checking the next character and potentially skipping hex sequences via RE_HEX_ESCAPE regex.

For typical CSS, most strings are short (class names, font names, URLs) and contain no escapes. Add a fast path that uses indexOf to find the closing quote first. If no backslash exists between the opening and closing quote, accept the string immediately without character-by-character scanning:

let closeQuote = css.indexOf(quote === SINGLE_QUOTE ? "'" : '"', pos + 1)
if (closeQuote !== -1) {
  let backslash = css.indexOf('\\', pos + 1)
  if (backslash === -1 || backslash > closeQuote) {
    // No escape in this string — fast path
    currentToken = ['string', css.slice(pos, closeQuote + 1), pos, closeQuote]
    pos = closeQuote + 1
    return
  }
}
// Fall through to character-by-character scanning

This avoids per-character charCodeAt checks for the common no-escape case.

Editable surface

• lib/tokenize.js — add fast-path string scanning for no-escape strings

What's different from prior work

• Optimize tokenizer comment scanning to use direct charCode loop instead of indexOf #59 proposed replacing indexOf with charCode loops for comment scanning. This thesis uses indexOf as a fast-path BEFORE character scanning, not as a replacement.
• Optimize tokenizer character scanning with lookup table #6/Optimize tokenizer regex usage and reduce indexOf calls #9/Optimize tokenizer word scanning with direct charCode comparison #37 tried lookup tables for character classification. This targets string literal scanning specifically.
• The key insight: most CSS strings have no escapes, so we can use indexOf (which V8 SIMD-optimizes) to skip the entire string in one call.

Expected impact

METRIC_A improvement of 1-3ms (large file with string values). METRIC_B improvement of 1-2ms. Combined 2-3ms.

[alanzabihi]

Polyresearch approval: thesis #68.