Fix #503: Improve caching

CHANGELOG.md

@@ -3,9 +3,63 @@
 All notable changes to the "vscode-rpgle" extension can be found in the
 [Releases](https://github.com/codefori/vscode-rpgle/releases) section of the GitHub repository.
 
-### Added
+## [Unreleased]
 
-- **Input specification (`I` spec) parsing** (fixed-format RPG IV): the parser now recognises and processes all four I spec sub-types — `programRecord`, `programField`, `externalRecord`, and `externalField`.
+### Added - OPM RPG Language Support
+
+- **OPM (Original Program Model) RPG Parser**: Full support for legacy OPM RPG language
+  - New [`OpmParser`](language/opm/parser.ts) class for fixed-format specification parsing
+  - Complete specification type support: Control (H), File (F), Extension (E), Input (I), Calculation (C), and Output (O) specs
+  - [`parseSpecification()`](language/opm/specs.ts) function with typed specification objects for all OPM spec types
+  - Symbol extraction for: files, data structures, variables, constants, subroutines, PLISTs, KLISTs, and CALL statements
+  - External file format resolution via table fetch (EXTNAME support)
+  - Include file processing (`/COPY` directive support)
+  - Embedded SQL recognition and aggregation
+  - Local Data Area (LDA) marker detection (`**`) to stop parsing at compile-time data
+
+- **Dual Parser Architecture**
+  - [`ParserFactory`](language/parserFactory.ts) class for intelligent parser routing based on RPG language variant
+  - Reorganized ILE parser to [`language/ile/`](language/ile/) subdirectory for clean separation
+  - Common [`IParser`](language/parserFactory.ts:12-18) interface implemented by both parsers
+  - Shared table fetch and include file resolution between OPM and ILE parsers
+  - Dynamic parser selection in language server based on RPG language variant
+
+- **Language Server Integration**
+  - VS Code language activation for OPM RPG via `onLanguage:rpg` event
+  - All providers updated to use appropriate parser:
+    - Completions, hover, definitions, references, rename, signature help
+    - Document symbols (outline view)
+    - Code actions and linting
+  - Unified cache model shared between both parsers
+
+- **Comprehensive Test Suite**
+  - [`tests/suite/opm/scope.test.ts`](tests/suite/opm/scope.test.ts) - 8 parser integration tests covering real-world OPM scenarios
+  - [`tests/suite/opm/specs.test.ts`](tests/suite/opm/specs.test.ts) - Specification parsing validation tests
+  - 7 OPM test fixtures covering various patterns: data structures, file operations, subroutines, PLISTs, KLISTs, edge cases
+  - Test coverage for: symbol resolution, external formats, multi-line C-specs, constants, LDA boundaries
+
+### Changed - OPM RPG Language Support
+
+- **Column Assistant and Fixed-Format Tools now support both RPG language variants**:
+  - All commands (`Shift+F4`, `Ctrl+Shift+F4`, `Ctrl+[`, `Ctrl+]`) now work with both ILE RPG and OPM RPG
+  - Added **OPM-specific spec definitions** (`opmSpecs` and `opmSpecRulers`) in [`specs.ts`](extension/client/src/schemas/specs.ts) with correct RPG III column positions
+  - Column Assistant automatically uses correct spec definitions based on RPG language variant
+  - **OPM specs supported**: H-spec (Control), E-spec (Extension), F-spec (File), I-spec (Input), C-spec (Calculation), O-spec (Output)
+  - **Critical fix**: OPM and ILE have **different column positions** for specs (e.g., C-spec Factor1 is 18-27 in OPM vs 12-25 in ILE)
+  - Updated `documentIsFree()` to recognize OPM as always fixed-format
+  - Language ID checks updated throughout [`columnAssist.ts`](extension/client/src/language/columnAssist.ts) and [`package.json`](package.json)
+- Folder structure reorganized for dual-parser architecture:
+  - ILE parser moved from `language/*.ts` to `language/ile/*.ts`
+  - OPM parser added in `language/opm/` directory
+  - Shared models remain in `language/models/`
+- All language server providers now use `getParser(uri)` for dynamic parser selection
+- Extension now supports both ILE RPG (`.rpgle`/`.sqlrpgle`) and OPM RPG (`.rpg`/`.sqlrpg`) language variants
+
+
+
+### Added - Previous ILE RPG Enhancements
+
+- **Input specification (`I` spec) parsing** (fixed-format ILE RPG): the parser now recognises and processes all four I spec sub-types — `programRecord`, `programField`, `externalRecord`, and `externalField`.
 - New `parseISpec()` and `prettyTypeFromISpecTokens()` functions in `language/models/fixed.ts` for decoding fixed-format I spec column layout.
 - New `trimQuotes()` utility exported from `language/tokens.ts`.
 - `cache.inputs` getter — returns all `Declaration` objects whose type is `"input"`, mirroring the existing `cache.structs`, `cache.files`, etc. accessors.

README.md

@@ -2,12 +2,12 @@
 
 <img src="https://github.com/codefori/vscode-rpgle/blob/main/media/logo.png?raw=true" height="180px" align="right">
 
-Adds functionality to assist in writing accurate, readable and consistent RPGLE, including:
+Adds functionality to assist in writing accurate, readable and consistent RPG language code, including:
 
 - Content assist
 - Outline view
 - Linter, including indentation checking and reformatting (`**FREE` only)
-- Column assist for fixed-format RPGLE.
+- Column assist for fixed-format RPGLE and OPM RPG
 
 Depends on the Code for IBM i extension due to source code living on the remote system when developing with source members.
 
@@ -33,6 +33,16 @@ To run debug the extension and server, from the VS Code debugger:
 1. Debug 'Launch Client'
 2. Debug 'Attach to Server'
 
+## Testing
+
+The test suite covers both RPG language variants:
+
+- **ILE RPG tests**: `tests/suite/*.test.ts`
+- **OPM RPG tests**: `tests/suite/opm/*.test.ts`
+  - Specification parsing (`specs.test.ts`)
+  - Symbol resolution and scoping (`scope.test.ts`)
+  - Real-world OPM fixtures (`tests/fixtures/opm/*.rpg`)
+
 # Previous contributors
 
 Thanks so much to everyone [who has contributed](https://github.com/codefori/vscode-rpgle/graphs/contributors).
@@ -46,4 +56,5 @@ Thanks so much to everyone [who has contributed](https://github.com/codefori/vsc
 - [@richardm90](https://github.com/richardm90)
 - [@wright4i](https://github.com/wright4i)
 - [@SanjulaGanepola](https://github.com/SanjulaGanepola)
-- [@bobcozzi](https://github.com/bobcozzi)
+- [@bobcozzi](https://github.com/bobcozzi)
+- [@Mohammed-Yaseen-Ali-2081](https://github.com/Mohammed-Yaseen-Ali-2081)

cli/rpglint/readme.md

@@ -2,6 +2,8 @@
 
 This is a command-line interface (CLI) for the RPG Linter, derived from the vscode-rpgle extension. It allows you to lint your RPG code from the command line, using the same rules and configuration as the vscode-rpgle extension.
 
+**Note**: The CLI currently supports ILE RPG (`.rpgle`, `.sqlrpgle`) files only. OPM RPG (`.rpg`) support is not yet available in the CLI version.
+
 ## Installation
 
 `rpglint` can be installed through npm. You can see the package on npmjs.com! `rpglint` is intended to be installed globally and not at a project level. To do that, you can simply run:

extension/client/src/commands.ts

@@ -1,11 +1,30 @@
 import { commands, ExtensionContext, Uri, window } from "vscode";
-import { clearTableCache, getCache } from "./requests";
+import { clearAllCache, clearTableCache, getCache, getCacheMetrics } from "./requests";
 import { LanguageClient } from "vscode-languageclient/node";
 
+function formatStats(label: string, hits: number, misses: number): string {
+  const total = hits + misses;
+  const hitRate = total > 0 ? `${Math.round((hits / total) * 100)}%` : `n/a`;
+  return `${label}: ${hits} hits, ${misses} misses (hit rate ${hitRate})`;
+}
+
 export function registerCommands(context: ExtensionContext, client: LanguageClient) {
   context.subscriptions.push(
-    commands.registerCommand(`vscode-rpgle.server.reloadCache`, () => {
-      clearTableCache(client);
+
+    commands.registerCommand(`vscode-rpgle.server.clearCache`, () => {
+      clearAllCache(client);
+      window.showInformationMessage(`RPGLE caches cleared.`);
+    }),
+
+    commands.registerCommand(`vscode-rpgle.server.viewCacheStats`, async () => {
+      const stats = await getCacheMetrics(client);
+      const message = [
+        formatStats(`Parsed`, stats.parsed.hits, stats.parsed.misses),
+        formatStats(`Table`, stats.table.hits, stats.table.misses),
+        formatStats(`Include`, stats.include.hits, stats.include.misses),
+      ].join(` | `);
+
+      window.showInformationMessage(`RPGLE cache stats - ${message}`);
     }),
 
     commands.registerCommand(`vscode-rpgle.server.getCache`, (uri: Uri) => {

extension/client/src/configuration.ts

@@ -6,4 +6,21 @@ export function get<T>(prop: string) {
 }
 
 export const RULER_ENABLED_BY_DEFAULT = `rulerEnabledByDefault`;
-export const projectFilesGlob = `**/*.{rpgle,RPGLE,sqlrpgle,SQLRPGLE,rpgleinc,RPGLEINC}`;
+export const projectFilesGlob = `**/*.{rpgle,RPGLE,sqlrpgle,SQLRPGLE,rpgleinc,RPGLEINC}`;
+
+export const CACHE_FILE_TTL_SECONDS = `cache.fileTTLSeconds`;
+export const CACHE_FILE_MAX_ENTRIES = `cache.fileMaxEntries`;
+
+export const CACHE_FILE_TTL_SECONDS_DEFAULT = 300;
+export const CACHE_FILE_MAX_ENTRIES_DEFAULT = 200;
+
+export interface CacheSettings {
+	fileTTLSeconds: number;
+	fileMaxEntries: number;
+}
+
+export function getCacheSettings(): CacheSettings {
+	const fileTTLSeconds = get<number>(CACHE_FILE_TTL_SECONDS) ?? CACHE_FILE_TTL_SECONDS_DEFAULT;
+	const fileMaxEntries = get<number>(CACHE_FILE_MAX_ENTRIES) ?? CACHE_FILE_MAX_ENTRIES_DEFAULT;
+	return { fileTTLSeconds, fileMaxEntries };
+}

extension/client/src/extension.ts

@@ -17,7 +17,7 @@ import {
 	TransportKind
 } from 'vscode-languageclient/node';
 
-import { projectFilesGlob } from './configuration';
+import { projectFilesGlob, getCacheSettings } from './configuration';
 import { clearTableCache, buildRequestHandlers } from './requests';
 import { getServerImplementationProvider, getServerSymbolProvider } from './language/serverReferences';
 import { checkAndWait, loadBase, onCodeForIBMiConfigurationChange } from './base';
@@ -50,17 +50,19 @@ export function activate(context: ExtensionContext) {
 
 	// Options to control the language client
 	const clientOptions: LanguageClientOptions = {
-		// Register the server for plain text documents
+		// Register the server for both ILE and OPM RPG documents.
 		documentSelector: [
 			{ language: 'rpgle' },
+			{ language: 'rpg' },
 		],
 		synchronize: {
 			fileEvents: [
 				workspace.createFileSystemWatcher('**/iproj.json'),
 				workspace.createFileSystemWatcher('**/rpglint.json'),
 				workspace.createFileSystemWatcher(projectFilesGlob),
 			]
-		}
+		},
+		initializationOptions: getCacheSettings()
 	};
 
 	// Create the language client and start the client.
@@ -92,21 +94,27 @@ export function activate(context: ExtensionContext) {
 		}
 	});
 
+	// Restart the language server when cache settings change so it picks up the new values
+	context.subscriptions.push(
+		workspace.onDidChangeConfiguration(e => {
+			if (e.affectsConfiguration(`vscode-rpgle.cache`)) {
+				client.stop().then(() => client.start());
+			}
+		})
+	);
+
 	// Start the client. This will also launch the server
 	client.start();
 
 	Linter.initialise(context);
 	columnAssist.registerColumnAssist(context);
-	
+
 	registerCommands(context, client);
-	
+
 	context.subscriptions.push(getServerSymbolProvider());
 	context.subscriptions.push(getServerImplementationProvider());
 	context.subscriptions.push(setLanguageSettings());
 	// context.subscriptions.push(...initBuilder(client));
-
-
-	console.log(`started`);
 }
 
 export function deactivate(): Thenable<void> | undefined {

extension/client/src/language/columnAssist.ts

@@ -18,28 +18,32 @@ const outlineBar = window.createTextEditorDecorationType({});
 let rulerEnabled = Configuration.get(Configuration.RULER_ENABLED_BY_DEFAULT) || false
 let currentEditorLine = -1;
 
-import { SpecFieldDef, SpecFieldValue, SpecRulers, specs } from '../schemas/specs';
+import { SpecFieldDef, SpecFieldValue, SpecRulers, specs, opmSpecs, opmSpecRulers } from '../schemas/specs';
 
-const getAreasForLine = (line: string, index: number) => {
+const getAreasForLine = (line: string, index: number, languageId: string = 'rpgle') => {
   if (line.length < 6) return undefined;
   if (line[6] === `*` || line[6] === `/`) return undefined;
 
+  // Use OPM specs for .rpg files, ILE specs for .rpgle files
+  const specDefinitions = languageId === 'rpg' ? opmSpecs : specs;
+  const rulerDefinitions = languageId === 'rpg' ? opmSpecRulers : SpecRulers;
+
   const specLetter = line[5].toUpperCase();
-  if (specs[specLetter]) {
-    const specification = specs[specLetter];
+  if (specDefinitions[specLetter]) {
+    const specification = specDefinitions[specLetter];
 
     const active = specification.findIndex((box: any) => index >= box.start && index <= box.end);
 
     return {
       specification,
       active,
-      outline: SpecRulers[specLetter]
+      outline: rulerDefinitions[specLetter]
     };
-  } else if (SpecRulers[specLetter]) {
+  } else if (rulerDefinitions[specLetter]) {
     return {
       specification: [] as SpecFieldDef[],
       active: -1,
-      outline: SpecRulers[specLetter]
+      outline: rulerDefinitions[specLetter]
     };
   }
 }
@@ -48,6 +52,9 @@ function documentIsFree(document: TextDocument) {
   if (document.languageId === `rpgle`) {
     const line = document.getText(new Range(0, 0, 0, 6)).toUpperCase();
     return line === `**FREE`;
+  } else if (document.languageId === `rpg`) {
+    // OPM RPG is always fixed-format
+    return false;
   }
 
   return false;
@@ -60,14 +67,15 @@ export function registerColumnAssist(context: ExtensionContext) {
       if (editor) {
         const document = editor.document;
 
-        if (document.languageId === `rpgle`) {
+        if (document.languageId === `rpgle` || document.languageId === `rpg`) {
           if (!documentIsFree(document)) {
             const lineNumber = editor.selection.start.line;
             const positionIndex = editor.selection.start.character;
 
             const positionsData = await promptLine(
               document.getText(new Range(lineNumber, 0, lineNumber, 100)),
-              positionIndex
+              positionIndex,
+              document.languageId
             );
 
             if (positionsData) {
@@ -111,14 +119,15 @@ export function registerColumnAssist(context: ExtensionContext) {
 }
 
 function moveFromPosition(direction: "left"|"right", editor = window.activeTextEditor) {
-  if (editor && editor.document.languageId === `rpgle` && !documentIsFree(editor.document)) {
+  if (editor && (editor.document.languageId === `rpgle` || editor.document.languageId === `rpg`) && !documentIsFree(editor.document)) {
     const document = editor.document;
     const lineNumber = editor.selection.start.line;
     const positionIndex = editor.selection.start.character;
 
     const positionsData = getAreasForLine(
       document.getText(new Range(lineNumber, 0, lineNumber, 100)),
-      positionIndex
+      positionIndex,
+      document.languageId
     );
 
     if (positionsData) {
@@ -145,14 +154,15 @@ function updateRuler(editor = window.activeTextEditor) {
 
   if (editor) {
     const document = editor.document;
-    if (document.languageId === `rpgle`) {
+    if (document.languageId === `rpgle` || document.languageId === `rpg`) {
       if (!documentIsFree(document)) {
         const lineNumber = editor.selection.start.line;
         const positionIndex = editor.selection.start.character;
 
         const positionsData = getAreasForLine(
           document.getText(new Range(lineNumber, 0, lineNumber, 100)),
-          positionIndex
+          positionIndex,
+          document.languageId
         );
 
         if (positionsData) {
@@ -218,7 +228,7 @@ interface FieldBox {
   maxLength?: number
 }
 
-async function promptLine (line: string, _index: number): Promise<string|undefined> {
+async function promptLine (line: string, _index: number, languageId: string = 'rpgle'): Promise<string|undefined> {
   const base = loadBase();
 
   if (!base) {
@@ -231,9 +241,12 @@ async function promptLine (line: string, _index: number): Promise<string|undefin
   if (line[6] === `*`) return undefined;
   line = line.padEnd(80);
 
+  // Use OPM specs for .rpg files, ILE specs for .rpgle files
+  const specDefinitions = languageId === 'rpg' ? opmSpecs : specs;
+
   const specLetter = line[5].toUpperCase();
-  if (specs[specLetter]) {
-    const specification = specs[specLetter];
+  if (specDefinitions[specLetter]) {
+    const specification = specDefinitions[specLetter];
 
     let parts: FieldBox[] = [];