TEDIT: What to do about Alto/Ascii fonts and codes

[rmkaplan]

          I pushed a version with the page region fix described above.  Seems to work on the lispcourse/ files that I tried.

But there also seems to be a character encoding problem in these files. Some of them have character code 156 (=234Q) that shows up on the display as an em-dash. In the pdf it shows up as z with hacek/caron.

This is in the TIMESROMAN font. 234Q is in the undefined (upper control) region of XCCS Cset 0. So maybe these are not XCCS files? ISO8859/1 ? Or something funky in the original Alto fonts?

Or maybe it is just that the Postscript stream doesn't have the proper mapping here?

Originally posted by @rmkaplan in #1352 (comment)

[MattHeffron]

I agree with @rmkaplan:

The easiest solution is to use the mappings upfront, to recode the Ascii-code/font characters to their NS-code/font equivalents when the files are read in, after which there is nothing more to do. (Comment in #1352 )

I do not see any value in preserving the old encoding in files when they're edited. It is not like when Microsoft Word changed from .doc to .docx files, where there was a huge installed base which required a path for backwards compatibility. It seems that our objective should be to make them work.

[masinter]

There is code in the INTERPRESS code to maintain a mapping from "ascii" (Alto) fonts to XNS character codes. The "legacy" files like "lispcourse" were made for printing to PRESS I thought, with Interpress later. Interpress assumed character sequences were sequences of 16 bit.
But I don't see anything like that ini the POSTSCRIPTSTREAM code.

[rmkaplan]

I created this issue to capture some of the complications that surround the use of Alto (or often called Ascii fonts, even though they have code assignments above 128) in Tedit, and perhaps in the system more broadly.

More broadly, it is the case that Alto fonts, particularly GACHA are specified in various places in the system. For example, the STANDARD font profile comes in with GACHA specified as the default font. That is why we see more black boxes in the exec window for NS characters than we would if TERMINAL was the default (although TERMINAL is itself pretty empty). Is there any reason not to switch the occurrences of GACHA, here and elsewhere to TERMINAL?

This set up has an impact on TEDIT, because when you create an empty TEDIT window to start a document, it picks a font for display (the value of the variable TEDIT.DEFAULT.CHARLOOKS). I found that that variable is initialized from the value of DEFAULTFONT at the time that TEDIT is loaded (e.g. loadup-full.sh). Thus it always comes in set for GACHA, and stays that way unless the user explicitly resets that variable. Changing the font profile, e.g. (FONTPROFILE 'NS) has no effect. The result, currently, is that by default we continue to propagate a reliance on GACHA in Tedit files.

It may be too much of a disturbance to fiddle with the font profiles. So I propose to break this connection, at least for Tedit, by allowing FONT as a property that can be given to TEDIT or OPENTEXTSTREAM when a new document is created, and to add (FONT (TERMINAL 10)) to the initial value of TEDIT.DEFAULT.PROPS. (Although I don't know why the default has always been fixed pitch, why not CLASSIC ?) At least that pushes the default into the XCCS world.

[rmkaplan]

Then there's the question that has been discussed, what should happen to existing Tedit files that have characters in Alto fonts. For TIMESROMAN, HELVETICA, and GACHA, many of the code assignments are the same as prescribed by the XCCS standard, but we have noticed that there are some differences.

Thinking of this as just a printing problem means that it is pushed off for Postscript and Interpress to do the right code and font substitutions to make things look right. Interpress has a set of mappings that can be used (Although Interpress doesn't include the MATH mappping. I propose to move that mapping from MATHTONS to INTERPRESS, and then move MATHTONS to obsolete.)

But then there are incompatibilities between the representation in Tedit and other tools and interfaces that deal with character codes. Virtual keyboards, for example, is set up to produce NS codes. If you type a math or Greek character into a Timesroman line, you will see a black box. If such a code displays properly in a Timesroman line and you switch that line to Classic, the intended glyph will no longer appear. Moreover, unless there is a Alto/Ascii version of a virtual keyboard, it would be quite difficult to enter one of these characters in a Timesroman run--you would have to switch to Classic to see what you intend.

The most intuitive arrangement, I think, is to translate the Alto codes and fonts to their intended NS equivalents when a an existing file is opened, according to the mappings as specified in INTERPRESS. After the translation, Timesorman runs will be converted to Classic, and needed code translations would have been applied. Writing the file will produce a version going forward that is free of Alto fonts and codes.

This is trivial to do. A weaker approach (and a little more complicated) would be to only do the translation for individual unaligned codes (e.g. just find all the 156's and move them to Classic and XCCS em-dash. The unmapped characters would be displayed in Timesroman instead of Classic--the substitution of Classic would be left to the printing software. So you get the same printed output either way, the difference is better consistency with what is displayed, and how you type. I think it is better to purge the Alto fonts on open and be done with them.

If making such a change automatically is too aggressive, it could be controlled by a property at open (PURGE ALTOFONTS YES or NO), and I would make YES the default. If NO, then you presumably would not see em-dashes where you expect.

[nbriggs]

Re (2 comments back) - "Is there any reason not to switch the occurrences of GACHA, here and elsewhere to TERMINAL?"

because TERMINAL has _ instead of ← attached to the underscore key, and ^ instead of ↑ attached to shift-6 ?

[MattHeffron]

In line with our stated goal of modernization, ultimately the correct solution would be to change everything to Unicode mapping of character codes to glyphs.
Unicode exists exactly to resolve the issue of unambiguously mapping character codes to glyphs.
All of the encodings that we're dealing with pre-date Unicode.
Printing to PostScript or Interpress would need to deal with mapping only from Unicode to their respective encoding.
Would this be hard? Yes
Would it be intractable? No
Would it be "the right thing"? Probably yes.

Just my 2¢

[rmkaplan]

That is more of a bug than a feature for the long run. A key labeled caret produces a code (caret in XCCS and Unicode) that shows up as an uparrow, but only if you are seeing it in Gacha, Timesroman, Helvetica. If you really want the arrow code, you have to go through some other special mechanism (virtualkeyboard instead of typing ^, use a TEDIT.ABBREVS/EXPAND sequence after you’ve typed something (but, where is the EXPAND key?). As an example, arrows in particular are significant in the LFG formalism, they appear in every line of every rule. But LFG also wants to display multilingual characters. So its edting windows are set up in Terminal/Classic, not Gacha, and the user sees the _ and ^ as they type. When the edit is done, the rule is internalized and the _ and ^ codes are replaced by the XCCS arrow codes. The rules can then be exported to the corresponding C implementation of LFG, which uses standard codings (ISO8859/1, Unicode…). The rule is also redisplayed in the Tedit window, if it is still open, so that it looks the way it should. But of course, once it has been converted to the uparrow, you can’t search just by typing ^, you have to get an uparrow code one way or another. So, in LFG there’s a lot of compensation to make it look the way a grammar writer would expect and to be compatible with external applicatinos. If CLISP or the record package hadn’t picked left arrow for assignment and uparrow for exponentiation (presumably to look somewhat like Algol or other languages of the day), then we would be less anchored to this early font-based encoding. But that’s where we are. And I agree, that’s a reason to move very cautiously.

…

On Nov 9, 2023, at 10:28 PM, Nick Briggs ***@***.***> wrote: Re (2 comments back) - "Is there any reason not to switch the occurrences of GACHA, here and elsewhere to TERMINAL?" because TERMINAL has _ instead of ← attached to the underscore key, and ^ instead of ↑ attached to shift-6 ? — Reply to this email directly, view it on GitHub <#1403 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQSTUJPVTQUTO2UOSJ4PTOTYDXCQ5AVCNFSM6AAAAAA7FBE2FSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBVGE3TKNRVGU>. You are receiving this because you were mentioned.

[masinter]

The way I've been thinking of the _ ^ issue is to talk about a character encoding which is very similar to the Xerox Character Code Standard except it has _ and ^ swappedd with left and up arrow.
Call this XCCS+ and just declare that :XCCS really means XCCS+. There might be a few files around that really want _ to be an underbar and ^ to be a circumflex because they are being printed with XCCS fonts ?

There is some ambiguity now; pick the option that will cause the least amount of trouble.

[MattHeffron]

↑ and ← were the glyphs in the initial ASCII standard (1963) based heavily on the Teletype Model 33, and were changed to ^ and _ in the 1965 ASCII revision.

[rmkaplan]

I’m reminded of Sarah Palin’s phrase: lipstick on a pig.

…

On Nov 10, 2023, at 9:51 AM, Larry Masinter ***@***.***> wrote: The way I've been thinking of the _ ^ issue is to talk about a character encoding which is very similar to the Xerox Character Code Standard except it has _ and ^ swappedd with left and up arrow. Call this XCCS+ and just declare that :XCCS really means XCCS+. There might be a few files around that really want _ to be an underbar and ^ to be a circumflex because they are being printed with XCCS fonts ? There is some ambiguity now; pick the option that will cause the least amount of trouble. — Reply to this email directly, view it on GitHub <#1403 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQSTUJNZGI7PJQ5OPZEEKYLYDZSRDAVCNFSM6AAAAAA7FBE2FSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBWGE3DMNZUGQ>. You are receiving this because you were mentioned.

[masinter]

Loops is pretty impossible to read without left-arrow.

This might be analogous to the EOL situation -- import has to guess, based on the age of the file and where it came from.

Early normalization has a lot of problems -- it's particularly bad if donstream processors "know better": you transform something and you can't say (in a standad way at least) that this is "UTF-8 converted from ASCII 1965"

[rmkaplan]

The way I've been thinking of the _ ^ issue is to talk about a character encoding which is very similar to the Xerox Character Code Standard except it has _ and ^ swappedd with left and up arrow. Call this XCCS+ and just declare that :XCCS really means XCCS+. There might be a few files around that really want _ to be an underbar and ^ to be a circumflex because they are being printed with XCCS fonts ?

There is some ambiguity now; pick the option that will cause the least amount of trouble.

Internally, our challenge is to make the internal character code representation consistent with the fonts that we happen to have now. If we had a reliable set of Unicode fonts, we would want to move away from XCCS to Unicode, internally, and avoid a bunch of (not always well defined) recodings. Once we have the fonts, that will take some work.

But following your suggestion, what if we defined our current internal encoding as XCCS+, with those 2 code switches, and not XCCS. Then the consistent thing to do would be to change our current XCCS font families (Classic, Modern, Terminal) from XCCS to XCCS+? I.e. go through and swap the character bitmaps so they line up like the Ascii text fonts. (We might still want to coerce characters for the arbitrarily different fonts Hippo, Cyrillic, Math, Symbol.)

There might still be some conversion when reading/writing a plain text file, but that would just follow by the fact that external formats map from back and forth from internal to whatever the file specifies. Currently the XCCS format doesn't have to do any code translations (unlike UTF-8), but that's no biggie.