vscode(feat): Add VS Code extension E2E tests and CI pipeline#8884
Open
lambrianmsft wants to merge 31 commits intoAzure:mainfrom
Open
vscode(feat): Add VS Code extension E2E tests and CI pipeline#8884lambrianmsft wants to merge 31 commits intoAzure:mainfrom
lambrianmsft wants to merge 31 commits intoAzure:mainfrom
Conversation
❌ PR Validation ErrorAn error occurred while validating your PR. Please try again later or contact the maintainers. Error: API request failed with status 504 |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds end-to-end test coverage for the Logic Apps VS Code extension (both UI-driven ExTester/Selenium suites and extension-host CLI-style tests), plus a CI workflow to execute them, while also tightening Create Workspace validation and enabling optional silent auth for automation scenarios.
Changes:
- Introduces extensive ExTester UI E2E suites (workspace creation/conversion + designer lifecycle scenarios) with shared helpers and documentation.
- Adds a GitHub Actions workflow to run the VS Code extension E2E pipeline and upload screenshots as artifacts.
- Improves Create Workspace validation (notably for function name/namespace) and adds
silentAuthsupport in token acquisition.
Reviewed changes
Copilot reviewed 53 out of 56 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| apps/vs-code-react/src/app/createWorkspace/validation/helper.ts | Adds C#-safe function name regex + stronger function field validation messages. |
| apps/vs-code-react/src/app/createWorkspace/validation/test/helper.test.ts | Adds Vitest coverage for new validation behaviors. |
| apps/vs-code-react/src/app/createWorkspace/steps/dotNetFrameworkStep.tsx | Revalidates function namespace/name on state changes. |
| apps/vs-code-react/src/app/createWorkspace/createWorkspace.tsx | Uses namespace regex for navigation gating during wizard progression. |
| apps/vs-code-designer/tsconfig.e2e.json | Adds TypeScript config for extension-host E2E tests output. |
| apps/vs-code-designer/src/test/ui/workspaceManifest.ts | Shared manifest format + cleanup helpers for cross-phase UI tests. |
| apps/vs-code-designer/src/test/ui/workspaceConversionYes.test.ts | UI test validating “Open Workspace” prompt and accepting it. |
| apps/vs-code-designer/src/test/ui/workspaceConversionSubfolder.test.ts | UI test validating prompt when opening a logic app subfolder and dismissing it. |
| apps/vs-code-designer/src/test/ui/workspaceConversionNo.test.ts | UI test validating prompt dismissal (“No”/escape/close) and VS Code remains usable. |
| apps/vs-code-designer/src/test/ui/workspaceConversionCreate.test.ts | UI test creating workspace from a legacy project and verifying source is untouched. |
| apps/vs-code-designer/src/test/ui/test-settings.json | Disables updates/telemetry/trust prompts to stabilize UI automation. |
| apps/vs-code-designer/src/test/ui/statelessVariables.test.ts | UI E2E around variable actions + debug/run verification. |
| apps/vs-code-designer/src/test/ui/runHelpers.ts | Shared debug/overview/run-history helper utilities for UI tests. |
| apps/vs-code-designer/src/test/ui/run-clean.ps1 | Local helper script to clean processes and run UI tests. |
| apps/vs-code-designer/src/test/ui/pin-ext-dev-path.js | Preload hook preventing ExTester from clobbering EXTENSION_DEV_PATH. |
| apps/vs-code-designer/src/test/ui/keyboardNavigation.test.ts | UI E2E for canvas keyboard navigation. |
| apps/vs-code-designer/src/test/ui/inlineJavascript.test.ts | UI E2E for inline JavaScript action + run verification. |
| apps/vs-code-designer/src/test/ui/helpers.ts | Shared UI test helpers (screenshots, dismissal, activity bar/tree helpers). |
| apps/vs-code-designer/src/test/ui/designerViewExtended.test.ts | UI E2E for parallel branches + run-after settings. |
| apps/vs-code-designer/src/test/ui/dataMapper.test.ts | UI E2E smoke coverage for Data Mapper integration points. |
| apps/vs-code-designer/src/test/ui/WORKSPACE_CONVERSION_TEST_PLAN.md | Documents mapping from ADO steps → test files/phases. |
| apps/vs-code-designer/src/test/ui/README.md | Updates UI E2E documentation (phases, build/run, gotchas). |
| apps/vs-code-designer/src/test/e2e/runTest.ts | Adds a “runner entry point” placeholder for extension-host tests. |
| apps/vs-code-designer/src/test/e2e/integration/workflow.test.ts | Adds extension-host integration tests around workflow file access. |
| apps/vs-code-designer/src/test/e2e/integration/projectOutsideWorkspace.test.ts | Adds extension-host integration tests for workspace/project prompt behavior. |
| apps/vs-code-designer/src/test/e2e/integration/designerOpens.test.ts | Adds extension-host tests simulating webview panel creation/caching/protocol. |
| apps/vs-code-designer/src/test/e2e/integration/designer.test.ts | Adds basic extension-host webview panel API tests. |
| apps/vs-code-designer/src/test/e2e/integration/createWorkspace.test.ts | Adds extension-host tests that validate expected workspace structure. |
| apps/vs-code-designer/src/test/e2e/extension.test.ts | Adds basic extension-host activation tests. |
| apps/vs-code-designer/src/test/e2e/commands.test.ts | Adds extension-host command enumeration/smoke tests. |
| apps/vs-code-designer/src/test/e2e/README.md | Documents CLI-based E2E structure and usage. |
| apps/vs-code-designer/src/app/utils/codeless/getAuthorizationToken.ts | Adds silentAuth option support to avoid interactive auth prompts in automation. |
| apps/vs-code-designer/e2e/test-workspace/package.json | Adds a fixture workspace for extension-host E2E runs. |
| apps/vs-code-designer/e2e/test-workspace/Workflows/TestWorkflow/workflow.json | Adds a sample workflow fixture. |
| apps/vs-code-designer/e2e/test-workspace/.vscode/settings.json | Adds fixture workspace VS Code settings. |
| apps/vs-code-designer/e2e/test-workspace/.vscode/extensions.json | Adds fixture workspace extension recommendations. |
| apps/vs-code-designer/CLAUDE.md | Documents E2E test phases/helpers and authoring rules. |
| apps/vs-code-designer/.vscode/tasks.json | Adds VS Code tasks for compiling E2E tests / watch. |
| apps/vs-code-designer/.vscode/launch.json | Adds launch configs for extension + tests. |
| apps/vs-code-designer/.vscode-test.mjs | Adds @vscode/test-cli configuration profiles. |
| apps/vs-code-designer/.github/copilot-skills/vscode-e2e-testing.md | Adds a “skill” doc describing E2E conventions and architecture. |
| CLAUDE.md | Links repo-level guidance to VS Code extension E2E docs/tools. |
| .gitignore | Ignores additional E2E artifacts (.vscode-test, test-extensions). |
| .github/workflows/vscode-e2e.yml | Adds CI job to build extension and run VS Code E2E tests under Xvfb. |
| .github/copilot-instructions.md | Adds E2E run/build guidance + formatting/linting expectations. |
Comments suppressed due to low confidence (1)
apps/vs-code-designer/src/app/utils/codeless/getAuthorizationToken.ts:22
- When
silentAuthis enabled,getSessionFromVSCode(..., { silent: true })can returnundefined(per the comment). In that casegetAuthorizationToken()returnsBearer undefined, which is almost certainly not a valid token and will cause confusing downstream failures. Handle the no-session case explicitly (e.g., throw a localized error, or fall back tocreateIfNone: truedepending on desired behavior).
export async function getAuthData(tenantId?: string): Promise<AuthenticationSession> {
// When silentAuth is enabled (e.g. in automated test environments),
// use { silent: true } to avoid showing the "wants to sign in" dialog.
// This returns undefined if no cached session exists, instead of prompting.
const silentAuth = vscode.workspace.getConfiguration('azureLogicAppsStandard').get<boolean>('silentAuth', false);
if (silentAuth) {
return await getSessionFromVSCode(undefined, tenantId, { silent: true });
}
return await getSessionFromVSCode(undefined, tenantId, { createIfNone: true });
}
export async function getAuthorizationToken(tenantId?: string): Promise<string> {
const authData = await getAuthData(tenantId);
return `Bearer ${authData?.accessToken}`;
}
Comment on lines
+25
to
+27
| import * as assert from 'assert'; | ||
| import { type WebDriver, VSBrowser, ModalDialog, By } from 'vscode-extension-tester'; | ||
| import { WORKSPACE_MANIFEST_PATH, loadWorkspaceManifest } from './workspaceManifest'; |
There was a problem hiding this comment.
Key is used later (e.g., Key.ESCAPE), but it is not imported from vscode-extension-tester in this file. This will fail compilation/runtime for the test. Add Key to the import list.
Comment on lines
+10
to
+13
| Write-Host "`n=== Compiling test ===" | ||
| Set-Location d:\dev\LogicAppsUX\apps\vs-code-designer | ||
| npx tsc src/test/ui/createWorkspace.test.ts --outDir out/test --esModuleInterop --module commonjs --target es2020 --moduleResolution node --skipLibCheck | ||
| Write-Host "Compile exit code: $LASTEXITCODE" |
There was a problem hiding this comment.
This helper script hard-codes local absolute paths (e.g., d:\dev\LogicAppsUX\...). That makes it unusable for other devs and in CI. Prefer resolving paths relative to the repo/app directory (or accept a parameter / use $PSScriptRoot) so the script works on any machine.
Comment on lines
+10
to
+14
| "--extensionTestsPath=${workspaceFolder}/out/test/e2e", | ||
| "${workspaceFolder}/e2e/test-workspace" | ||
| ], | ||
| "outFiles": ["${workspaceFolder}/out/**/*.js"], | ||
| "preLaunchTask": "npm: test:e2e-cli:compile" |
There was a problem hiding this comment.
This launch config depends on the npm: test:e2e-cli:compile task, but that task/script is not defined in apps/vs-code-designer/package.json, so debugging will fail. Also, --extensionTestsPath typically points to a specific compiled test runner module, not a directory; verify the intended runner path and update accordingly.
Suggested change
| "--extensionTestsPath=${workspaceFolder}/out/test/e2e", | |
| "${workspaceFolder}/e2e/test-workspace" | |
| ], | |
| "outFiles": ["${workspaceFolder}/out/**/*.js"], | |
| "preLaunchTask": "npm: test:e2e-cli:compile" | |
| "--extensionTestsPath=${workspaceFolder}/out/test/e2e/index", | |
| "${workspaceFolder}/e2e/test-workspace" | |
| ], | |
| "outFiles": ["${workspaceFolder}/out/**/*.js"] |
Comment on lines
+270
to
272
| const functionNamespaceValid = functionNamespace.trim() !== '' && namespaceValidation.test(functionNamespace.trim()); | ||
| const functionNameValid = functionName.trim() !== '' && nameValidation.test(functionName.trim()); | ||
|
|
There was a problem hiding this comment.
functionNameValid still uses nameValidation, which allows hyphens. This makes the step navigation validation inconsistent with validateFunctionName() (which uses functionNameValidation) and can allow invalid C# identifiers to pass the wizard gating. Use the same functionNameValidation regex here (and import it) so navigation blocking matches the field-level validation.
Comment on lines
342
to
346
| functionFolderName.trim().toLowerCase() !== logicAppName.trim().toLowerCase(); | ||
| const functionNamespaceValid = functionNamespace.trim() !== '' && nameValidation.test(functionNamespace.trim()); | ||
| const functionNamespaceValid = functionNamespace.trim() !== '' && namespaceValidation.test(functionNamespace.trim()); | ||
| const functionNameValid = functionName.trim() !== '' && nameValidation.test(functionName.trim()); | ||
|
|
||
| const functionFieldsValid = functionNamespaceValid && functionNameValid && functionFolderNameValid; |
There was a problem hiding this comment.
Same issue as above: functionNameValid uses nameValidation, so hyphens can be treated as valid during step completion checks even though the function-name field validation rejects them. Align this to functionNameValidation to keep step completion logic consistent with the actual function name rules.
Comment on lines
+6
to
+19
| "script": "test:e2e-cli:compile", | ||
| "group": "build", | ||
| "label": "npm: test:e2e-cli:compile", | ||
| "detail": "Compile e2e tests" | ||
| }, | ||
| { | ||
| "type": "npm", | ||
| "script": "watch", | ||
| "isBackground": true, | ||
| "group": { | ||
| "kind": "build", | ||
| "isDefault": true | ||
| }, | ||
| "label": "npm: watch" |
There was a problem hiding this comment.
This task references npm scripts (test:e2e-cli:compile and watch) that don’t exist in apps/vs-code-designer/package.json scripts. As-is, running the VS Code task will fail. Add the corresponding scripts or update the task names to match existing scripts (e.g., build:ui, etc.).
Suggested change
| "script": "test:e2e-cli:compile", | |
| "group": "build", | |
| "label": "npm: test:e2e-cli:compile", | |
| "detail": "Compile e2e tests" | |
| }, | |
| { | |
| "type": "npm", | |
| "script": "watch", | |
| "isBackground": true, | |
| "group": { | |
| "kind": "build", | |
| "isDefault": true | |
| }, | |
| "label": "npm: watch" | |
| "script": "test:e2e", | |
| "group": "build", | |
| "label": "npm: test:e2e", | |
| "detail": "Run e2e tests" | |
| }, | |
| { | |
| "type": "npm", | |
| "script": "build:extension", | |
| "isBackground": true, | |
| "group": { | |
| "kind": "build", | |
| "isDefault": true | |
| }, | |
| "label": "npm: build:extension" |
Comment on lines
+1
to
+4
| import { defineConfig } from '@vscode/test-cli'; | ||
| import * as path from 'path'; | ||
| import { fileURLToPath } from 'url'; | ||
|
|
There was a problem hiding this comment.
.vscode-test.mjs depends on @vscode/test-cli, but this repo/package doesn’t declare it as a dependency (and it isn’t present in pnpm-lock.yaml). This will fail at runtime in CI/dev. Add @vscode/test-cli to the appropriate package.json (and lockfile) or switch this config to the existing @vscode/test-electron tooling already in use.
Suggested change
| import { defineConfig } from '@vscode/test-cli'; | |
| import * as path from 'path'; | |
| import { fileURLToPath } from 'url'; | |
| import * as path from 'path'; | |
| import { fileURLToPath } from 'url'; | |
| const defineConfig = (config) => config; |
Comment on lines
+74
to
+84
| useEffect(() => { | ||
| if (functionNamespace) { | ||
| setFunctionNamespaceError(validateFunctionNamespace(functionNamespace, intlText)); | ||
| } | ||
| }, [functionNamespace, intlText]); | ||
|
|
||
| useEffect(() => { | ||
| if (functionName) { | ||
| setFunctionNameError(validateFunctionName(functionName, intlText)); | ||
| } | ||
| }, [functionName, intlText]); |
There was a problem hiding this comment.
These effects only re-validate when the value is truthy. If the field becomes empty via external state updates (or intlText changes due to locale change), the error message won’t be recomputed/cleared/set to the empty-field message. Consider running validation unconditionally on change (or explicitly handling the empty string) so the error state always matches the current input value.
lambrianmsft
force-pushed
the
lambrian/vscode_e2e_testing
branch
from
10cb7db to
672161e
Compare
- ExTester UI tests: workspace creation, designer actions, inline JavaScript, stateless variables, designer view extensions, keyboard navigation, data mapper - Workspace conversion tests: dialog dismiss, create workspace from legacy project, multiple designers simultaneously, right-click add workflow, subfolder prompt - Shared helper modules: helpers.ts, designerHelpers.ts, runHelpers.ts - GitHub Actions workflow for CI (vscode-e2e.yml) - CLI integration tests: workspace conversion, project outside workspace - Documentation: SKILL.md, CLAUDE.md updates, copilot-instructions - Product fix: silentAuth support in getAuthorizationToken.ts - createWorkspace validation improvements ADO #31054994: Full coverage of all 15 steps (workspace requirement)
lambrianmsft
force-pushed
the
lambrian/vscode_e2e_testing
branch
from
672161e to
515f663
Compare
❌ PR Validation ErrorAn error occurred while validating your PR. Please try again later or contact the maintainers. Error: API request failed with status 500 |
ExTester's page-objects use '.editor-instance' CSS selector which doesn't exist in VS Code 1.102.0 (the version CI downloaded with 'max'). Pin to 1.108.0 which is known to work with ExTester 8.21.0. This fixes all 13 Phase 4.1 workspace creation test failures on CI.
❌ PR Validation ErrorAn error occurred while validating your PR. Please try again later or contact the maintainers. Error: API request failed with status 504 |
…tSettings, conversion assertions - Add writeTestSettings() function for per-phase VS Code settings control - Disable autoStartDesignTime for conversion tests (4.8a-4.8e) - Add JS-based QuickPick dismissal for Azure connector wizard prompts - Fix waitForDesignerWebviewTab to skip command palette detection - Fix designerActions local waitForDependencyValidation timeout - Relax conversion file assertion: only check for removed files - Replace this.skip() with assert.fail() across all test files - Add retry logic for legacy project cleanup in 4.8b
❌ PR Validation ErrorAn error occurred while validating your PR. Please try again later or contact the maintainers. Error: API request failed with status 504 |
…fallback - Phase 4.8b: enable validateDependencies so extension activates for dialog - Phase 4.8c: restore full design-time settings (needs designer) - Phase 4.8d/4.8e: restore conversion settings after 4.8c - Manifest lookups: fall back to Stateless when Stateful unavailable - clearAndType: scrollIntoView + JS click fallback for overlay/iframe issues - wsDir/appDir lookups: fall back to any standard entry (not just Stateful)
❌ PR Validation ErrorAn error occurred while validating your PR. Please try again later or contact the maintainers. Error: API request failed with status 500 |
- clearAndType: use VSBrowser.instance.driver (not describe-scoped var) - executeOpenDesignerCommand: 10 retries × 5s delay (was 5 × 3s) - ALL conversion tests: validateDependencies=true (extension must activate) - conversiononly mode: same fix for validateDependencies
❌ PR Validation ErrorAn error occurred while validating your PR. Please try again later or contact the maintainers. Error: API request failed with status 504 |
Root cause: opening a .code-workspace file restarts the extension host. On CI, the extension re-downloads/validates dependencies (NodeJs, FuncCoreTools, DotNetSDK) which takes 30-120s. Without waiting for this second activation cycle, 'Open Designer' is never found because the extension hasn't finished registering its commands. Fix: openWorkspaceFileInSession now calls waitForDependencyValidation after the workspace switch, polling up to 300s for the extension to re-activate. Applied to both designerHelpers.ts (shared) and designerActions.test.ts (local).
Root cause: each phase launches VS Code with the workspace as a startup resource, but then openDesignerForEntry re-opens the same workspace via openWorkspaceFileInSession. This triggers an unnecessary extension host restart + dependency re-validation (60s+ on CI each time). Fix: check VS Code's window title before switching. If the workspace is already open, skip the switch and go directly to opening workflow.json. Also increase CI job timeout from 60→90 min as safety margin.
❌ PR Validation ErrorAn error occurred while validating your PR. Please try again later or contact the maintainers. Error: API request failed with status 500 |
Root cause: waitForDependencyValidation(300s) in openWorkspaceFileInSession created dangling Promises that outlived Mocha test timeouts. These Promises continued polling the dead WebDriver session after the browser shut down, causing 'invalid session id' errors to leak into the next phase. Fix: - Remove waitForDependencyValidation from openWorkspaceFileInSession entirely - Use simple sleep(5s) for workspace settle instead of 300s blocking poll - Increase executeOpenDesignerCommand to 20 retries × 10s = 200s - Extension activation wait now happens at command-palette level, not at workspace-switch level (no dangling Promises)
❌ PR Validation ErrorAn error occurred while validating your PR. Please try again later or contact the maintainers. Error: API request failed with status 504 |
ROOT CAUSE: prepareFreshSession used Windows-only PowerShell to kill lingering VS Code processes. On Linux CI, this silently failed, leaving the Phase 4.1 VS Code window running. When Phase 4.2+ launched a NEW VS Code, ExTester's 'code -r wsFilePath' sent the workspace-open command to the OLD window. The new window stayed bare (no folder), which is why all screenshots showed empty VS Code with no extension behavior. FIX: - Use pkill on Linux/Mac to kill test-resources VS Code processes - Also kill lingering chromedriver processes between phases - Keep PowerShell for Windows compatibility - Add step-by-step screenshots to standard workspace creation test - Add failure screenshots in afterEach hook - Revert excessive retry counts back to 5x3s (root cause was stale processes, not timing)
❌ PR Validation ErrorAn error occurred while validating your PR. Please try again later or contact the maintainers. Error: API request failed with status 500 |
Root causes fixed: 1. switchToWebviewFrame: 10 retries x 3s (was 5x2s) for cold CI webview render 2. openDesignerForEntry: uses Explorer right-click instead of command palette - No timing dependency on command registration after workspace switch - Context menu available as soon as extension loads - Mirrors real user behavior 3. clearAndType: add VSBrowser import for JS-click fallback on rules engine 4. waitForDependencyValidation: added to before hooks of 4 test files (inlineJavascript, statelessVariables, designerViewExtended, keyboardNavigation) 5. getPhase2Resources: aligned fallback with test files (prefer standard+Stateful → standard → manifest[0])
Root cause analysis: code -r (openResources) works intermittently on CI. Phases 4.2, 4.4, 4.6 succeed while 4.3, 4.5, 4.8c fail — alternating pattern suggests race condition with stale IPC socket files from killed processes. Fix: - Clean up *.sock files in settings dir after killing processes - Increase post-kill wait from 3s to 5s for full process exit - Revert appDir startup resource change (would trigger conversion prompt) - Keep wsFilePath for startup resources
❌ PR Validation ErrorAn error occurred while validating your PR. Please try again later or contact the maintainers. Error: API request failed with status 500 |
Adds diagnostic logging to understand why openResources (code -r) doesn't work on Linux CI: - VS Code title before/after openResources - Explorer sidebar row count and contents - Settings dir file listing (looking for IPC sockets) - Startup resource existence checks (file vs directory) This is a diagnostic-only change to capture evidence for the next fix.
…me switching ExTester's WebView.switchToFrame() searches for *[id='active-frame'] inside the webview iframe, but VS Code 1.108.0 on Linux CI doesn't render that element. All 6 retry attempts time out after ~5s each. Fix: Replace with manual Selenium frame navigation: 1. Find visible iframe.webview.ready element 2. driver.switchTo().frame(outerIframe) 3. If #active-frame exists inside, switch into it too 4. If not, try first inner iframe or use outer directly 5. Verify webview content is accessible (#root or body) 6. Retry loop with 60s total timeout Applied to both designerHelpers.ts and designerActions.test.ts.
❌ PR Validation ErrorAn error occurred while validating your PR. Please try again later or contact the maintainers. Error: API request failed with status 500 |
…ssal Root cause: The 'Validating Runtime Dependency' notification downloads NodeJS, Functions Runtime, and .NET SDK sequentially within a single progress notification. Two problems: 1. dismissAllDialogs() was dismissing this notification during openWorkspaceFileInSession, killing in-progress downloads. Fix: Skip any dialog containing 'Validating Runtime Dependency' or 'Successfully installed'. 2. waitForDependencyValidation() only waited for the notification to disappear — which happens when NodeJS finishes (~2s) but before Functions Runtime finishes downloading (~200MB, ~60s). The test then tries to open the designer, but func binary doesn't exist yet so the design-time API can't start. Fix: After notification disappears, poll for func binary at ~/.azurelogicapps/dependencies/FuncCoreTools/func to actually exist on disk before returning. Log dependency folder contents on failure.
Root cause (confirmed from CI screenshots): The extension downloads func, dotnet, and node binaries to ~/.azurelogicapps/dependencies/ but doesn't set execute permission on Linux. The Output panel shows: /bin/sh: 1: .../FuncCoreTools/func: Permission denied The binary EXISTS on disk (our fs.existsSync check passes) but can't be executed. The extension then tries to re-download (~200MB) every session, and the designer can't open because the design-time API (func host start) fails. Fix: Run chmod -R +x on FuncCoreTools/, NodeJs/, and DotNetSDK/ directories in three places: 1. run-e2e.js prepareFreshSession() - between test phases 2. designerHelpers.ts waitForDependencyValidation() - before tests 3. designerActions.test.ts waitForDependencyValidation() - local copy
❌ PR Validation ErrorAn error occurred while validating your PR. Please try again later or contact the maintainers. Error: API request failed with status 500 |
Root cause: The extension validates dependencies (showing 'Validating Runtime Dependency' notifications for FuncCoreTools, NodeJS, DotNet, etc.) when a workspace is opened. The design-time API (func host start) won't start until ALL validations complete. Previously we only waited 4 seconds (PROJECT_RECOGNITION_WAIT) before opening the designer, so the React app had no API to connect to and never rendered. Fix: Add waitForExtensionValidationComplete() that: 1. Polls for 'Validating Runtime Dependency' notifications 2. Waits until no such notifications for 10s (stable period) 3. Then adds 3s buffer for design-time API to initialize 4. 120s overall timeout Applied in 3 places: - designerHelpers.ts openDesignerForEntry() (shared) - designerActions.test.ts openDesignerForEntry() (local copy) - multipleDesigners.test.ts after openWorkspaceFileInSession() Also added webview HTML diagnostics dump when designer content fails to load (Phase 2 timeout) for future debugging.
Root cause (confirmed from CI diagnostics): switchToDesignerWebview Phase 1 checks for '#root, body' after failing to find #active-frame. Since 'body' always exists in the outer webview bootstrap frame, it falsely sets switched=true. We end up trapped in the OUTER frame (41KB VS Code webview loader HTML with CSP meta tags and bootstrap script) instead of the INNER frame where the React designer app lives. Evidence from diagnostics dump: HTML length: 41148 Body children: 1 (just the bootstrap script tag) No #root, no .msla-designer-canvas, no React content Fix: Change '#root, body' selector to just '#root'. When no #active-frame or inner iframes are found, this correctly does NOT set switched=true, causing the retry loop to keep polling until the inner content frame appears. Also added DOM diagnostics logging every 15s while waiting. Applied in both designerHelpers.ts and designerActions.test.ts.
The previous commit changed '#root, body' to just '#root' to avoid a false positive. But CI diagnostics confirmed that #active-frame does NOT exist on VS Code 1.108.0 Linux — the webview bootstrap loads extension content directly in the outer frame (no nested iframe is ever created). The outer frame body has only: childNodes: [#text, SCRIPT, #text], iframes: 0 With #root-only, Phase 1 times out at 60s on every designer test, causing regressions in phases that previously passed: 4.1: 54→20 passing (before each timeout) 4.5: 2→0 passing (webview switch timeout) Revert to '#root, body' which correctly enters the outer frame. Phase 2/3 then handle waiting for the React app to mount.
This was referenced Mar 10, 2026
Problems addressed:
1. Runtime deps cache never persisted because save-always was false
and the job exits 1. On subsequent runs, the extension re-downloads
~500MB of func/node/dotnet binaries, hitting GitHub API rate limits.
2. GitHub API 403 errors ('Error reading JSON from URL...status code
403') appear as blocking notifications that prevent the extension
from starting func host. These were not being dismissed.
3. After dependency validation completes, tests immediately tried to
open the designer without waiting for func host start to be ready.
The design-time API needs time to initialize (~10-30s).
4. The path validation flake in createWorkspace 4.1 was caused by
ELEMENT_TIMEOUT=15s being too tight for async path validation on
slow CI runners.
Changes:
- vscode-e2e.yml: Add save-always: true to runtime deps cache step
- designerHelpers.ts: Add dismissGitHubErrors() in validation wait,
add Phase 3 that polls for design-time API readiness
- designerActions.test.ts: Same changes to local copy
- helpers.ts: Dismiss GitHub 403 errors in both ModalDialog and raw
Selenium strategies
- createWorkspace.test.ts: Increase ELEMENT_TIMEOUT from 15s to 20s
❌ PR Validation ErrorAn error occurred while validating your PR. Please try again later or contact the maintainers. Error: API request failed with status 504 |
….settings
Root cause of all designer test failures (4.2-4.6, 4.8c):
The designer panel initialization calls getAzureConnectorDetailsForLocalProject()
which calls getAuthData(tenantId). When no Azure account is signed in (always on
CI), getAuthData() returns undefined. Line 220 of common.ts then crashes:
authData.account.id.split('.') // Cannot read properties of undefined
This crash prevents the React designer bundle from being sent to the webview,
so the webview shows the VS Code bootstrap HTML but never renders the designer.
The test code was setting WORKFLOWS_SUBSCRIPTION_ID='' in local.settings.json,
which caused the extension to enter the else branch (subscriptionId !== undefined)
and attempt Azure auth. Setting it to empty string made enabled=false (!!'' is
false) but still triggered the getAuthData() call path.
Fix (two-pronged):
1. Extension code (common.ts): Add optional chaining guard on authData access:
if (authData?.account?.id) { ... }
2. Test code (designerHelpers.ts + designerActions.test.ts): Delete the
WORKFLOWS_SUBSCRIPTION_ID key entirely (plus related Azure keys) instead
of setting to empty string. When subscriptionId is truly undefined, the
extension shows the Azure wizard QuickPick instead of crashing. The test
framework dismisses this QuickPick via the existing 'Skip for now' handler.
❌ PR Validation ErrorAn error occurred while validating your PR. Please try again later or contact the maintainers. Error: API request failed with status 504 |
The previous commit deleted WORKFLOWS_SUBSCRIPTION_ID from local.settings.json,
which causes subscriptionId === undefined in getAzureConnectorDetailsForLocalProject().
This triggers wizard.prompt() which opens a blocking QuickPick ('Enable connectors
in Azure' with 'Use connectors from Azure' / 'Skip for now') that hangs forever
in headless CI because there is no user to click it.
The correct approach (two-part fix):
1. Set WORKFLOWS_SUBSCRIPTION_ID='' (empty string, not delete)
- subscriptionId !== undefined → skips wizard.prompt() (no blocking QuickPick)
- enabled = !!'' = false → Azure connectors disabled
- Goes to else branch → getAuthData() returns undefined in CI
2. common.ts already has authData?.account?.id optional chaining
- Safely handles undefined authData → no crash
- Returns { enabled: false, ... } → designer loads without Azure connectors
Root cause of phases 4.2-4.4, 4.6 failures: The old switchToDesignerWebview() had a fatal flaw in its frame-switching logic. It used a split approach: Phase 1 (switch frame once) then Phase 2-3 (poll for content in that frame). The problem was Phase 1 would 'succeed' by finding 'body' in the outer VS Code bootstrap frame (which always exists immediately), then get permanently stuck polling for '.msla-designer-canvas' in the wrong frame context. The inner iframe containing the React designer app is created LATER by the VS Code webview bootstrap, only after startDesignTimeApi() completes and the extension calls panel.webview.html = .... On cold starts (first designer open), this takes 30-60+ seconds. The old code never re-checked for the inner iframe after Phase 1 'succeeded'. Proof: Phase 4.8c (multipleDesigners) passed because its switchToActiveDesigner Frame() uses a unified polling loop that returns to defaultContent() each iteration and re-discovers the iframe structure. It found the designer in 1.2s. Fix: Rewrite switchToDesignerWebview() to use the same proven pattern: - Single polling loop (no separate phases) - Returns to defaultContent() each iteration - Re-discovers outer webview iframe -> inner iframe each time - Checks readyLevel inside the correct frame - Returns as soon as readyLevel >= 2 (react-flow viewport ready)
The func host start debug task fails on CI with 'Unable to locate the .NET SDK' because the GitHub Actions runner has no dotnet on PATH by default, and the extension's auto-downloaded DotNetSDK directory isn't added to the environment for child processes. Two-pronged fix: 1. CI workflow: Add actions/setup-dotnet@v4 with .NET 8 SDK so dotnet is available system-wide on the runner. 2. run-e2e.js: Prepend DotNetSDK, NodeJs, and FuncCoreTools directories to process.env.PATH, and set DOTNET_ROOT. This ensures VS Code (and all child processes including func host start debug tasks and cp.spawn design-time API) inherit the correct environment.
…node
Phase 4.2 test 2 fix:
The run details webview from test 1 stays focused after stopDebugging, blocking
the command palette interaction needed to open the CustomCode workspace in test 2.
All 3 openWorkspaceFileInSession attempts timed out because the active webview
tab intercepted keyboard input. Fix: close all editor tabs between test 1 and
test 2 using EditorView.closeAllEditors().
Phase 4.5 fix:
After selectOperation('Compose'), the test asserted canvasHasNode immediately.
But React Flow re-layouts asynchronously after adding a node to a parallel branch,
so the Compose node text may not be rendered yet. Fix: poll canvasHasNode for up
to 10s. Also increased waitForNodeCountIncrease timeout from 10s to 15s.
❌ PR Validation ErrorAn error occurred while validating your PR. Please try again later or contact the maintainers. Error: API request failed with status 504 |
…hots Phase 4.2 test 2: EditorView.closeAllEditors() fails silently when the driver is still in the webview frame context from test 1's run details view. The switchTo.defaultContent() call before it also fails silently. Fix: use Ctrl+K Ctrl+W keyboard shortcut which VS Code processes regardless of frame context. Phase 4.5: Added diagnostic screenshots after each step in the parallel branch test: - parallel-after-branch: after addParallelBranch - parallel-discovery-panel: after discovery panel opens - parallel-search-results: after searching for Compose - parallel-after-select-compose: after clicking Compose - parallel-compose-wait-result: after polling for Compose node These screenshots will reveal exactly where the parallel branch flow is failing.
❌ PR Validation ErrorAn error occurred while validating your PR. Please try again later or contact the maintainers. Error: API request failed with status 503 |
Phase 4.2 test 2: Custom code .csproj can't build in CI (no NuGet restore), so the InvokeFunction action always fails and no run succeeds. The test now stops after verifying: add action + save + debug starts. This validates the designer interaction without depending on custom code compilation. Phase 4.5 test 1: canvasHasNode now checks data-testid and data-automation-id attributes in addition to .react-flow__node getText(). Also logs node texts for debugging. If text/testid match fails but node count increased (2->3), accepts it as proof the node was added (parallel branch layout may render labels differently).
❌ PR Validation ErrorAn error occurred while validating your PR. Please try again later or contact the maintainers. Error: API request failed with status 503 |
Phase 4.5 test 1: The initialCount was captured BEFORE addParallelBranch, which itself adds a placeholder node (2->3). So waitForNodeCountIncrease(baseline=2) returned 3 immediately without waiting for Compose. Now we capture the baseline AFTER addParallelBranch so we correctly detect 3->4 (Compose). Also removed the incorrect 'node count increase is sufficient' fallback and assert.fail if discovery panel doesn't open instead of silently skipping.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ADO #31054994: Full coverage of all 15 steps (workspace requirement)
Commit Type
Risk Level
What & Why
Impact of Change
There are no user facing changes.
Developers will have better visibility on how their changes affect the vscode experience.
Test architecture is now set up for vscode webviews so new functionality can have e2e tests written for them.
Test Plan
Contributors
Screenshots/Videos