Add basic semantic highlighting

[Gustaf-C]

Fixes mathworks/MATLAB-extension-for-vscode#45

This adds basic support for semantic highlighting. Currently only variables is implemented, but I plan to do some more work before it is ready to be merged. I am opening a draft PR to allow for some initial feedback.

Results can be seen below where indexed vectors are now correctly identified as variables:

Changes

• Added SemanticTokensProvider.ts which uses the pre-existing indexer to find semantic tokens
• Updated server.ts to register the provider

Note
Depends on plumbing in mathworks/MATLAB-extension-for-vscode#321

[Gustaf-C]

I have identified a few issues that I am looking into. It seems to be possible to correctly identify variables, but class and function references, as well as methods and properties, are hard to differentiate, where I’d say that the latter is more annoying.

Since the current parser is in P-code, I can’t make any adjustments there on my own. If the current parser would be updated, the information could easily be used in the highlighter. Otherwise, the best option could be to update the textmate scopes so the highlighting is at least not actively misleading.

[dklilley]

Hi @Gustaf-C , thanks for working on this!

Overall, this approach looks good to me. I agree that class, function, method, and property references are going to be difficult to differentiate at this time.

That being said, I think adding simple variable semantic highlighting is a good start which adds definite value! I will take a closer look at the code itself when this is transferred out of draft state.

The parsing logic currently uses some undocumented MATLAB APIs which are very difficult to work with, so we are trying to move away from their usage. We are currently waiting for a new API to become available in MATLAB which should provide us easier-to-use and more detailed info about the file contents. At that point, it should be possible to add more semantic token types.