emojis/nonstandard characters mess up alignment

[yo1dog]

This is due to an underlying issue with vscode: microsoft/vscode#100730

There is no way to know how wide a character will be when displayed in vscode. It is completely up to the render engine with many environmental variables involved (font, ligatures, grapheme clustering, etc.). A character may even have a fractional width! Thus, there are some scenarios that are simply impossible to align correctly.

For example, this emoji 👨‍👩‍👦‍👦 is actually 7 characters: U+1F468 U+200D U+1F469 U+200D U+1F466 U+200D U+1F466. In vscode, placing the cursor after this emoji advances the column position by 7, indicating that it would take up the width of 7 characters. However, visually the emoji spans about 2.2 characters.

On the flip side, the Japanese Kanji for man 男 is the single character U+7537, advances the cursor column position by 1, yet visually spans about 1.9 columns.

As such, there is nothing this extension can do to solve this problem. This issue will be left open for informational purposes.

[1nkwell]

You could consider adding a configuration option to set the width of non-standard characters to double. This is because many monospace fonts used for CJK adopt this practice.

[yo1dog]

Ability for the user to define the display width of characters is an interesting idea.

Can you provide some examples of popular CJK monospace fonts? In environments that use these fonts, are the spaces single width while all the typically used characters are wide? Or is the space typically wide as well? I would have guessed that all glyphs in a monospace font would be... monospace/equal width, but I am ignorant on this topic.

---

Using the upcoming Cascadia Next font for JP as an example, here is a break down of glyph widths: https://gist.github.com/yo1dog/93a6f78bc069444753e5063aa8dc06d0

I guess it's a question of accuracy, but my gut says it's not so simple to bifurcate "standard" vs "non-standard" characters. At least, I would expect fonts to differ on which characters they consider "non-standard" and thus differ on which glyphs are wide.

For example, there are a few wide glyphs §±¶ mixed with single width in the Latin-1 Supplement block and General Punctuation block.

But to your point, ignoring the explicit Halfwidth characters, the vast majority of glyphs after the General Punctuation block are indeed wide. So perhaps a toggle could treat all characters U+2070 and beyond as wide and be mostly accurate, but it seems like a toss up for the characters before.

Of course, this is only looking at the glyphs the font actually defines. For any character your primary default font does not define a glyph for, vscode will attempt to use the next font in your list of fallbacks, which may have it's own set of wide characters. For example, the ½ vulgar fraction character is not defined by this font.

---

Rather than a toggle, we could allow users to define character ranges to treat as wide. For the above font it could look something like:

{"wideCharacterRanges":[
  "A7","B1","B6",
  "2C9",
  "2003","2010","2015","2016","2020","2021","2025","2027-2051",
  "2100-FFFFF"
]}

Or, less accurate but more simply - treat all non-ascii characters as wide:

{"wideCharacterRanges":["80-FFFFF"]}

Or... we could provide a tool for generating the ranges of wide characters. I used opentype.js in a simple script to create the above CSV table. If the user provides their font (or even their list of font fallbacks), we could do the same to generate the exact list of wide characters.

[yo1dog]

Taking a step back here, if there is not some fundamental agreement between CJK monospace fonts on which characters are wide vs single width, then simply changing display fonts would break vertical alignment, and would make me question the usefulness of vertical alignment in such environments.

Again, I am ignorant on this subject, but if the majority of fonts define the same set of wide characters then a toggle could work.

Unicode does define East Asian Width, but of the many characters defined as "ambiguous" or "neutral", the Cascadia glyphs are a mix of single width and wide. I suspect glyph widths will vary between fonts for these characters.

It looks like the implementation for many best-effort column width calculators and wcwidth use this unicode database. For example, here is Node.JS' and one in python:

• https://github.com/nodejs/node/blob/04a086a1f415491c7aae0d300afe08b9ed831109/lib/internal/util/inspect.js#L2971
• https://github.com/jquast/wcwidth/blob/master/wcwidth/wcwidth.py

Maybe this is good enough for a toggle. Though it fails the Cascadia font, so I would guess not:

> console.table(['111','§§§'])
┌─────────┬────────┐
│ (index) │ Values │
├─────────┼────────┤
│ 0       │ '111'  │
│ 1       │ '§§§'  │
└─────────┴────────┘

[1nkwell]

I think allowing users to define character ranges to treat as wide is a better choice.

Can you provide some examples of popular CJK monospace fonts?

• Source Han Mono
• Sarasa Mono
• Maple Mono

Maybe this is good enough for a toggle. Though it fails the Cascadia font, so I would guess not:

> console.table(['111','§§§'])
┌─────────┬────────┐
│ (index) │ Values │
├─────────┼────────┤
│ 0       │ '111'  │
│ 1       │ '§§§'  │
└─────────┴────────┘

I think that is because Cascadia font is not a CJK monospace font. It displayed as below with JetBrains Maple Mono font:

[ganlvtech]

I'm using Inconsolata font together with CJK font Source Han Sans SC. CJK characters will be exactly 2 * ascii char width.

Maybe there can be some configurable settings that specify some range of codeUnit => float point number char width map. and then measure the character by rounded integer?

{
  "codeUnitRange": [
    {"from":"4e00","to":"9fff","width":2}
    // or matcher
    {"pattern":"\\u{1F468}\\u{200D}\\u{1F469}\\u{200D}\\u{1F466}\\u{200D}\\u{1F466}","flags":"u","width":2.506}
  ]
}

vscode-cursor-align/extension.js

Lines 251 to 266 in 372c213

	if (codeUnit === 9) {
	// Tab character.
	colSpan += tabSize - ((textCol + colSpan) % tabSize);
	}
	else if (
	codeUnit >= 0xDC00 && codeUnit <= 0xDFFF && // current is low surrogate
	prevCodeUnit >= 0xD800 && prevCodeUnit <= 0xDBFF // previous was high surrogate
	) {
	// Skip the low surrogate in a VALID surrogate pair.
	// NOTE: We are ignoring an edge case: Technically, it is possible for a selection to split an
	// otherwise valid surrogate pair. Practically, this shouldn't be possible from vscode UI but
	// might be possible if the selection range was set programmatically.
	}
	else {
	colSpan += 1;
	}