Errors loading LOOPS sources

[nbriggs]

Loading LOOPS into a newly built sysout, where it started with a reasonable DEFAULTFILETYPES list, but something appears to have smashed the list in a way that causes OPENSTREAM to choke when it tries to CL:ASSOC an entry on the DEFAULTFILETYPES alist because there's a non-string/non-atom object as a key. No idea what did this.

[masinter]

save the LISP and FULL and "mid" sysouts -- loading POSTSCRIPTSTREAM adds (PS . TEXT) there; let's track down the smash if we can. I have a commit that reworks the initialization

[nbriggs]

I've saved all of "tmp" after the loadup. The value of DEFAULTFILETYPELIST is OK at the beginning, OK after loading LOADLOOPS (the source), but when I execute (LOADLOOPS NIL 'SOURCE) it loads a couple of files and then dies as above with the smashed atom on the a-list.

[nbriggs]

Current loadups produce this result instead...

[rmkaplan]

If this is coming from deep inside the UFS implementation, that begs the question (apart from this particular failure): what is the point of trying to guess the type of a file? Does it matter in any way?

…

On Mar 10, 2022, at 12:44 PM, Nick Briggs ***@***.***> wrote: Current loadups produce this result instead... <https://user-images.githubusercontent.com/8138841/157751014-4a861159-5e9a-446f-adb4-a55c55e3367a.png> — Reply to this email directly, view it on GitHub <#733 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQSTUJMTYR4JPOMAZBWJ4BLU7JNJ7ANCNFSM5QFFGKUA>. You are receiving this because you are subscribed to this thread.

[nbriggs]

I don't believe this is coming from the UFS implementation. I don't think the other problem was caused by the UFS implementation either -- just the symptoms showed up there. Also, I'm not sure that I agree with the DEFAULTFILETYPELIST that a PDF file is TEXT (they almost always contain binary data). It's unclear to me, either, why it needs to know whether a file is TEXT or BINARY...

[orcmid]

@nbriggs Also, I'm not sure that I agree with the DEFAULTFILETYPELIST that a PDF file is TEXT (they almost always contain binary data). It's unclear to me, either, why it needs to know whether a file is TEXT or BINARY...

I may have this out of context but PDF files should always be treated as binary and on GIT, there should be a .gitattributes that establishes that.

The problem stems from at least the suppression of CR/LF and LF translation when text files are moved between different platforms and/or git/GitHub (a problem that FTP clients also had to deal with). Also, there should not be DIFFs between binaries and no prospect of Unicode mishaps by mistreatment of files as platform-specific text.

If this is relevant in the context of this discussion :).

[masinter]

the DEFAULTFILETYPELIST variable was an innocent bystander. I don't think it's used for much

grep -lri defaultfiletypelist

shows

docs/Sun Users Guide/5-MEDLEY-FILES.TEDIT
docs/Sun Users Guide/INDEX.TEDIT
docs/html-sunguide/Sun-Users-Guide_files/content.htm
loadups/full.sysout
loadups/lisp.venuesysout
loadups/starter.sysout
loadups/full.venuesysout
loadups/whereis.hash
loadups/lisp.sysout
tmp/full.sysout
tmp/init.sysout
tmp/init-mid.sysout
tmp/init.dlinit
tmp/whereis.hash
tmp/lisp.sysout
library/POSTSCRIPTSTREAM
library/POSTSCRIPTSTREAM.LCOM

seems it's contained to UFS.
EDITCALLERS(DEFAULTFILETYPELIST UFS)
makes it looks like it has some kind of effect on EOL handling.
This is all before MIME so the expectations about file "type" based on file extension might be a leftover.
The recent PR changing DEFAULTFILETYPELIST initiation was to get rid of code inside LOADUP-FULL that mucked with it.

[rmkaplan]

DEFAULTFILETYPELIST appears only to be used internal to UFS. We inherited the text/binary distinction from the fact that ftp in those days did EOL conversion, but in hindsite it was a mistake to import that into our file handling. Because whether or not we want to treat a file as binary or not depends on the Lisp function that we are using to operate on it—in reading with a character function, all EOL varaitions should map to our internal EOL, in writing it is only when we are applying a character printing function that we would worry about the EOL convention on the file, whether or not the file is text or not (it could have text regions and binary regions). It may be helpful to tell git that a given file is binary, so it knows not to bother to do a meaningless compare. I don’t know about that. In GITFNS, I look at inside the file to see whether it is a lisp sourcefile, a compiled file, a tedit file, to choose a useful comparison function. If unrecognized, I then look at PRINTFILETYPES, which typically does look at extensions, and I tend to avoid comparing on files that it thinks are binary. Sort of an attempt to behave like git in that situation. But maybe that is a fool's errand. Unlike git, the GITFNS comparison is menu driven—you have to select the files that you want to compare. If you happen to select a binary file you won’t get a meaningful result, but otherwise, no harm no foul.

…

On Mar 10, 2022, at 7:26 PM, Larry Masinter ***@***.***> wrote: the DEFAULTFILETYPELIST variable was an innocent bystander. I don't think it's used for much grep -lri defaultfiletypelist shows docs/Sun Users Guide/5-MEDLEY-FILES.TEDIT docs/Sun Users Guide/INDEX.TEDIT docs/html-sunguide/Sun-Users-Guide_files/content.htm loadups/full.sysout loadups/lisp.venuesysout loadups/starter.sysout loadups/full.venuesysout loadups/whereis.hash loadups/lisp.sysout tmp/full.sysout tmp/init.sysout tmp/init-mid.sysout tmp/init.dlinit tmp/whereis.hash tmp/lisp.sysout library/POSTSCRIPTSTREAM library/POSTSCRIPTSTREAM.LCOM seems it's contained to UFS. EDITCALLERS(DEFAULTFILETYPELIST UFS) makes it looks like it has some kind of effect on EOL handling. This is all before MIME so the expectations about file "type" based on file extension might be a leftover. The recent PR changing DEFAULTFILETYPELIST initiation was to get rid of code inside LOADUP-FULL that mucked with it. — Reply to this email directly, view it on GitHub <#733 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQSTUJJGR7CRIN5CSIENSELU7K4MZANCNFSM5QFFGKUA>. You are receiving this because you commented.

[masinter]

The Git software and settings for text/binary decisions is completely separate from the determination in Lisp.

https://docs.github.com/en/get-started/getting-started-with-git/configuring-git-to-handle-line-endings

separate issue---the per-repo .gitattributes can't be used because lisp source files don't have a filename extension.
But without that the heuristic (for deciding text/binary) doesn't work with some TEDIT files with long ascii initial sections.

[nbriggs]

I think we may be getting burned by the LOOPSSPEEDUPS implementation of Make-Not-Reference-Counted

[rmkaplan]

Interesting. I wonder what their strategy is, whether it is better than what we did in the LFG and finite state implementations. There we added BLCONS (backlink CONS—reference count only the CDR) and XCONS (Xpointers for both) datatypes surrounded by a substantial suite of the natural functions, i.s.oprs etc. to operate on them.

…

On Mar 11, 2022, at 12:41 PM, Nick Briggs ***@***.***> wrote: I think we may be getting burned by the LOOPSSPEEDUPS implementation of Make-Not-Reference-Counted — Reply to this email directly, view it on GitHub <#733 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQSTUJIXJDKGDHOHJGJ6L4LU7OVXDANCNFSM5QFFGKUA>. You are receiving this because you commented.

[masinter]

This LOOPS code came from the envos 2.0 release. Before the "new" CDR-coding change. So if loops is mucking about with things at that level, the behavior is sure to be wrong. It's unlikely to be just a matter of recompiling or running interpreted.

[rmkaplan]

In our BACKLINK module we use XPOINTER and FULLXPOINTER datatype field declarations. Would those be vulnerable to this system-level change? Seems to work.

[nbriggs]

It looks as though it's a non-reference-counted block of 100 POINTERs, they're not doing anything with list cells.
Looking closer, though, I'm suspecting this:
LLDATATYPE:

(PUTPROPS \GETDTD  MACRO
   ((typeNum)
    (ADDBASE \DTDSpaceBase (ITIMES typeNum 18))))
)

and LOOPSSPEEDUP:

(PUTPROPS \GETDTD MACRO ((typeNum)
                                 (ADDBASE \DTDSpaceBase (LLSH typeNum 4))))
)

Something about the allocation changed to make it * 18 instead of * 16 to get the datatype.

[nbriggs]

Fixing that improved things a lot. I suppose that the definition is there at all so that the loops code can be compiled without needing to load EXPORTS.ALL to get the \GETDTD macro definition...?

Now I'm stuck on an apparently missing definition at load time for function $AV which appears to be a reader for ActiveValues.