add --verify flag to audit research coverage

[IovetsNikolay]

User description

Add auditResearch() method that compares research output against ARIA tree and classifies missed elements by root cause (code filter, prompt issue, or site-specific). Adds --verify flag to /research TUI command and CLI. Includes research-audit skill for manual analysis.

---

CodeAnt-AI Description

Add --verify audit for research to report ARIA coverage

What Changed

• /research (CLI and TUI) accepts a new --verify flag that runs an audit after research and prints a plain-text report comparing research output to the page's ARIA tree
• Researcher now produces a structured audit report that lists total interactive ARIA elements, elements found, and missing elements grouped by root cause: code filter issues (unnamed elements, non-clickable roles, stop-word matches) and prompt issues
• Included a user skill document ("Research Audit") with steps to run the audit, interpret results, and triage fixes (code vs prompt vs site-specific)

Impact

✅ Clearer research coverage reports
✅ Faster root-cause identification for missed interactive elements
✅ Easier in-session auditing of research results

💡 Usage Guide

Checking Your Pull Request

Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.

Talking to CodeAnt AI

Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:

@codeant-ai ask: Your question here

This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.

Example

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

Preserve Org Learnings with CodeAnt

You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:

@codeant-ai: Your feedback here

This helps CodeAnt AI learn and adapt to your team's coding style and standards.

Example

@codeant-ai: Do not flag unused imports.

Retrigger review

Ask CodeAnt AI to review the PR again, by typing:

@codeant-ai: review

Check Your Repository Health

To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.

[codeant-ai]

CodeAnt AI is reviewing your PR.

[codeant-ai]

Nitpicks 🔍

🔒 No security issues identified

⚡ Recommended areas for review Matching Logic The audit currently checks whether an element's accessible name appears in the research output using a simple substring check. This can produce false positives (matching partial words) and false negatives (punctuation, casing, or small variations). Consider using escaped word-boundary regex matching, normalization (strip punctuation/extra whitespace), or fuzzy matching to improve accuracy. Report Accuracy Audit report entries include hardcoded file paths and line numbers (e.g., 'src/utils/aria.ts:81') and only the element name/role. Those hardcoded references may become stale or unhelpful. Also the report lacks element locators (CSS/XPath) or other identifiers that would make issues actionable.

[codeant-ai]

Suggestion: The audit routine classifying missed elements as code-filter or prompt issues hardcodes DEFAULT_STOP_WORDS and omits the name-length filter used in interactive exploration, so elements excluded by configurable stop words or by the length filter are incorrectly reported as prompt issues; it should reuse the researcher config stopWords and apply the same length filter to keep classifications accurate. [logic error]

Severity Level: Major ⚠️

- ⚠️ Research audit mislabels code-filtered elements as prompt issues.
- ⚠️ Misleads users tuning prompts versus code filters.
- ⚠️ Reduces reliability of `--verify` research coverage reports.

Suggested change

	const codeFilterIssues: Array<{ element: string; role: string; reason: string; file: string }> = [];
	const promptIssues: Array<{ element: string; role: string }> = [];
	const found: Array<{ element: string; role: string }> = [];
	for (const node of ariaNodes) {
	const role = String(node.role || '');
	const name = String(node.name || '').trim();
	const isUnnamed = !!node.unnamed;
	const inResearch = name ? researchLower.includes(name.toLowerCase()) : false;
	if (inResearch) {
	found.push({ element: name || `unnamed ${role}`, role });
	continue;
	}
	if (isUnnamed) {
	codeFilterIssues.push({
	element: `unnamed ${role}`,
	role,
	reason: 'Unnamed button/link (icon-only element without aria-label)',
	file: 'src/utils/aria.ts:81',
	});
	continue;
	}
	if (!CLICKABLE_ROLES.has(role.toLowerCase())) {
	codeFilterIssues.push({
	element: name,
	role,
	reason: `Role "${role}" not in CLICKABLE_ROLES`,
	file: 'src/ai/researcher.ts:40',
	});
	continue;
	}
	if (this.matchesStopWord(name, DEFAULT_STOP_WORDS)) {
	codeFilterIssues.push({
	element: name,
	role,
	reason: `Name "${name}" matches DEFAULT_STOP_WORDS`,
	const config = this.getResearcherConfig();
	const stopWords = config?.stopWords ?? DEFAULT_STOP_WORDS;
	const codeFilterIssues: Array<{ element: string; role: string; reason: string; file: string }> = [];
	const promptIssues: Array<{ element: string; role: string }> = [];
	const found: Array<{ element: string; role: string }> = [];
	for (const node of ariaNodes) {
	const role = String(node.role || '');
	const name = String(node.name || '').trim();
	const isUnnamed = !!node.unnamed;
	const inResearch = name ? researchLower.includes(name.toLowerCase()) : false;
	if (inResearch) {
	found.push({ element: name || `unnamed ${role}`, role });
	continue;
	}
	if (isUnnamed) {
	codeFilterIssues.push({
	element: `unnamed ${role}`,
	role,
	reason: 'Unnamed button/link (icon-only element without aria-label)',
	file: 'src/utils/aria.ts:81',
	});
	continue;
	}
	if (!CLICKABLE_ROLES.has(role.toLowerCase())) {
	codeFilterIssues.push({
	element: name,
	role,
	reason: `Role "${role}" not in CLICKABLE_ROLES`,
	file: 'src/ai/researcher.ts:40',
	});
	continue;
	}
	if (name.length > 50) {
	codeFilterIssues.push({
	element: name,
	role,
	reason: 'Name too long for interactive exploration filter (>50 characters)',
	file: 'src/ai/researcher.ts:359',
	});
	continue;
	}
	if (this.matchesStopWord(name, stopWords)) {
	codeFilterIssues.push({
	element: name,
	role,
	reason: `Name "${name}" matches researcher stopWords`,

Steps of Reproduction ✅

1. Run the normal research flow on any page whose ARIA snapshot (stored on
`WebPageState.ariaSnapshot` and parsed by `collectInteractiveNodes` in
`src/ai/researcher.ts`) contains a clickable element (role `button`, `link`, etc.) with
either: (a) an accessible name longer than 50 characters, or (b) a name that matches a
custom stop word configured via `ai.agents.researcher.stopWords` (used in
`performInteractiveExploration()` in `src/ai/researcher.ts`).

2. During interactive exploration, `performInteractiveExploration()` in
`src/ai/researcher.ts` filters candidates by role and name; it skips elements whose
`name.length > 50` and elements where `this.matchesStopWord(name, stopWords)` holds, where
`stopWords` comes from `this.getResearcherConfig().stopWords ?? DEFAULT_STOP_WORDS`. As a
result, the long-name or custom-stop-word element is never clicked or explored.

3. After research completes, invoke the verification/audit path that calls
`Researcher.auditResearch()` (the method defined at `src/ai/researcher.ts:1119`), passing
the same `WebPageState` and the markdown `researchOutput` produced earlier (which does not
contain the skipped element's text).

4. Inside `auditResearch()`, the loop over `ariaNodes` classifies this element: it is not
`inResearch`, it is named, it has a clickable role, and it is only checked against
`DEFAULT_STOP_WORDS` with no `name.length > 50` filter, so it is treated as having "passed
all code filters" and is pushed into `promptIssues` instead of `codeFilterIssues`,
producing a "PROMPT ISSUES" entry in the generated "Research Audit Report" even though the
real root cause was the code-level exploration filters.

Prompt for AI Agent 🤖

This is a comment left during a code review.

**Path:** src/ai/researcher.ts
**Line:** 1122:1163
**Comment:**
	*Logic Error: The audit routine classifying missed elements as code-filter or prompt issues hardcodes DEFAULT_STOP_WORDS and omits the name-length filter used in interactive exploration, so elements excluded by configurable stop words or by the length filter are incorrectly reported as prompt issues; it should reuse the researcher config stopWords and apply the same length filter to keep classifications accurate.

Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.

👍 | 👎

[codeant-ai]

CodeAnt AI finished reviewing your PR.