2026 refactor

[digikar99]

Credits to @ajberkley's Pull Request: #36

[ajberkley]

Hi, note also ajberkley@0385f33 which fixes errors that occur during lisp callbacks.

(the change to the hash-table is not related, but probably correct, but is sbcl specific)

[ajberkley]

Somewhere you had asked why we have a raw-py lock and an interaction lock. It was a result of bad design. I reworked the multi-threading stuff for simplicity in my fork so the blocking on the python-input stream can be done independently of grabbing an interaction lock on the system. I did this in a bit of a rush while I was debugging the issue I describe below so expect some clean-ups as time goes on.

More importantly, though, I discovered that there is a fundamental issue in the protocol between python and lisp in the presence of either multiple threads in python or asynchronous callbacks. Identifying what response goes with which eval request is indeterminate. Python and lisp can disagree on the ordering of things.

In general we assume a stack like procession of calls --- lisp wants to eval something so sends an eval message to python, python in response calls back into lisp to get something, then calls back into python to evaluate something, gets the response to that, then unwinds to the original eval and returns a response to that. But if that second callback actually was due to an asynchronous callback, not in response to the lisp eval request, then this assigns the responses to the wrong requests! After much head scratching over this I finally just added ids to the messages from lisp which solves this problem for now so we can match them up without assuming the two sides agree on an ordering.

This has been broken since the beginning of time I think, just not tested for (and probably not relevant for anyone since threading on the python side or asynchronous callbacks weren't a real thing).

Anyhow, my fork or py4cl2 is under heavy use at work so these sorts of bugs are coming up :) We've built a full matplotlib replacement for our use of matlab at work and it's quite nice... minus matplotlib being very slow at some things.