Add mobile replay harness

scripts/mobile/.gitignore

@@ -0,0 +1,12 @@
+*.local.json
+.env.local
+tmp/
+
+# Goal files are app-specific. Only example.json is committed.
+goals/*
+!goals/example.json
+
+# Fixture-specific scripts that stay local for internal testing.
+images-goal.mjs
+run-images-*.mjs
+validate-images-*.mjs

scripts/mobile/ARCHITECTURE.md

@@ -0,0 +1,84 @@
+# Mobile Harness Architecture
+
+## Ownership Split
+
+The flow has four separate pieces:
+
+1. **Goal runner:** `run-lidar-live-ios.mjs` or `run-goal.mjs` drives any app through a goal manifest. Navigation steps, target screen, and scroll counts come from the goal JSON, not hardcoded values.
+2. **Mobile driver:** Lidar owns generic device control through `live-*` tools: connect, snapshot, tap, fill, scroll, handle system alerts, and apply sightmap annotations.
+3. **Session discovery:** the runner or a future Lidar/mobile bridge maps a device run to a FullStory session URL, trace ID, user ID, or session ID.
+4. **Replay diagnosis:** Subtext `review-open`, `review-view`, and `review-diff` inspect the FullStory replay and compare it to the goal.
+
+## Flow
+
+```
+Goal JSON  +  .env.local
+       |           |
+       v           v
+  run-lidar-live-ios.mjs  (or run-goal.mjs for Appium)
+       |
+       |-- live-connect (platform: "ios")
+       |-- live-view-snapshot
+       |-- live-act-click / live-act-drag
+       |-- live-disconnect
+       |
+       v
+  Snapshots + report  -->  validate-goal-artifacts.mjs
+       |
+       v
+  fetch-subtext-review-evidence.mjs
+       |
+       v
+  capture-subtext-review-observations.mjs
+       |
+       v
+  validate-replay-observations.mjs
+```
+
+## Scripts
+
+| Script | Purpose |
+| --- | --- |
+| `run-lidar-live-ios.mjs` | Drive an app through Lidar MCP live tools using a goal manifest |
+| `run-goal.mjs` | Drive an app through Appium/WebDriverIO using a goal manifest |
+| `run-local-lidar-ios.mjs` | Build and start a local Lidar, start Appium, run the Lidar goal |
+| `validate-goal-artifacts.mjs` | Validate device artifacts against goal expectations |
+| `prepare-subtext-review.mjs` | Generate a replay review request from goal and session URL |
+| `fetch-subtext-review-evidence.mjs` | Fetch replay evidence from Subtext MCP |
+| `capture-subtext-review-observations.mjs` | Extract observations from Subtext review evidence |
+| `capture-replay-observations-from-snapshot.mjs` | Extract observations from browser replay snapshots |
+| `validate-replay-observations.mjs` | Validate replay observations against goal expectations |
+| `appium-layer.mjs` | Appium/WebDriverIO connection, capabilities, and device primitives |
+| `device-e2e-common.mjs` | Shared env loading, log capture, session URL extraction |
+
+## Goal Manifest
+
+All app-specific behavior comes from the goal JSON. The runner does not assume any screen names, labels, or navigation paths. See `goals/example.json` for the documented schema.
+
+Key fields:
+
+- `run.navigation[]`: steps to reach the target screen. Each step has an `action` (`tap`, `scrollToLabel`, `screenshot`, `source`, `dismissAlert`) and relevant parameters.
+- `run.targetScreen`: the expected active screen after navigation.
+- `run.scrollDownCount` / `run.scrollUpCount`: how many times to scroll.
+- `replayChecks.observationHeuristics`: keyword lists used by the observation scripts to detect events, screen presence, image content, and scroll activity in replay evidence.
+- `sensitiveRegions[]`: privacy expectations for UI regions.
+
+## Lidar iOS Integration
+
+The Lidar live tools provide a unified surface for iOS. `live-connect` with `platform: "ios"` creates an Appium session through the Lidar iOS backend. All subsequent `live-view-snapshot`, `live-act-click`, `live-act-drag`, and `live-disconnect` calls route to the iOS driver based on the connection type.
+
+The local wrapper `run-local-lidar-ios.mjs` builds Lidar from source, starts it on free ports, starts Appium if needed, generates MCP caps, and delegates to `run-lidar-live-ios.mjs`.
+
+## Replay Sampling
+
+Replay validation inspects multiple timestamps after the target screen opens. The fetcher samples a short timeline, and callers can override it:
+
+```bash
+MOBILE_SUBTEXT_VIEW_TIMESTAMPS=12000,22000,37000 node scripts/mobile/fetch-subtext-review-evidence.mjs
+```
+
+## Current Limits
+
+- The runner supports tap, scroll, and simple navigation. Login, deep links, text input, multi-screen flows, and manual checkpoints need a richer step format.
+- Session discovery depends on SDK logs or a supplied session identifier.
+- Build and install of the customer app is outside the runner. The customer must provide an installed app or clear install steps.

scripts/mobile/CUSTOMER_INTAKE.md

@@ -0,0 +1,55 @@
+# Mobile Debugging Intake
+
+Use this template when asking what Subtext needs for a mobile replay debugging run.
+
+## App Access
+
+- App format: installed app, `.ipa`, `.app`, simulator build, or source repo.
+- Bundle ID:
+- FullStory org and environment:
+- Is the app already instrumented with FullStory?
+- If source/build is required, exact build and install commands:
+
+## Device Target
+
+- Device type: physical iOS device or simulator.
+- Device name:
+- UDID:
+- iOS version:
+- Signing or WebDriverAgent requirements:
+
+## Auth Path
+
+- Starting state: logged out, logged in, fresh install, or existing app state.
+- Test account or login method:
+- MFA, passkey, captcha, deep link, or manual step requirements:
+- Any system prompts expected on first launch:
+
+## Goal
+
+- User task to perform:
+- Screen or flow that should appear:
+- Interactions required: taps, typing, scrolls, waits, gestures.
+- Expected replay behavior:
+- Things that should be considered failure:
+
+## Session Discovery
+
+Provide at least one way to identify the matching FullStory session:
+
+- FullStory session URL from SDK logs.
+- User ID and approximate run time.
+- Session ID from logs.
+- Trace ID.
+- A user-provided replay URL.
+- A deterministic test account that can be searched in FullStory.
+
+## Evidence Output
+
+The run should produce:
+
+- Device snapshots/screenshots for the performed goal.
+- FullStory session URL or equivalent session identifier.
+- Subtext `review-open` evidence.
+- Multiple `review-view` samples around the target flow.
+- A final pass/warn/fail report against the rubric.

scripts/mobile/README.md

@@ -0,0 +1,198 @@
+# Mobile Replay Harness
+
+Drives any iOS app on a physical device, then checks whether the FullStory replay of that session looks right. Everything the runner needs to know about your app comes from two places: environment variables and a goal JSON file. Nothing is hardcoded for any particular app.
+
+> iOS only for now. Android is not supported yet and will come in a follow-up.
+
+There are two ways to drive the app:
+
+- **Lidar live tools** (`run-lidar-live-ios.mjs`) -- talks to a Lidar MCP server over HTTP. This is the primary path.
+- **Appium direct** (`run-goal.mjs`) -- talks to a local Appium server through WebDriverIO. Useful when you don't have a Lidar server.
+
+After driving, a separate set of scripts fetches the FullStory replay, pulls out observations, and compares them to what the goal says should have happened.
+
+## Setup
+
+1. Copy the env example and fill in your values:
+
+```
+cp scripts/mobile/mobile.env.example scripts/mobile/.env.local
+```
+
+2. Write a goal JSON file for your app. Copy `goals/example.json` and edit it. The goal says which screen to navigate to, what taps to perform, how many times to scroll, and what the replay should contain.
+
+3. Make sure you have:
+   - A physical iOS device connected (or a simulator).
+   - The app already installed on the device.
+   - The device UDID (run `xcrun devicectl list devices` to find it).
+   - A FullStory API key for the org that instruments the app.
+
+## Environment Variables
+
+Put these in `scripts/mobile/.env.local`. That file is gitignored.
+
+**Always required:**
+
+| Variable | What it is |
+| --- | --- |
+| `FULLSTORY_API_KEY` | API key for MCP auth and replay fetches |
+| `MOBILE_BUNDLE_ID` | Bundle ID of the app installed on the device |
+| `MOBILE_UDID` | Device UDID |
+| `MOBILE_DEVICE_NAME` | Device name (e.g. "My iPhone") |
+| `MOBILE_GOAL_EXPECTATIONS` | Path to the goal JSON file |
+| `MOBILE_OUT_DIR` | Where to write output artifacts |
+
+**For non-prod FullStory orgs:**
+
+| Variable | What it is |
+| --- | --- |
+| `MOBILE_FULLSTORY_APP_HOST` | Base URL for replay links. Defaults to `https://app.fullstory.com`. Set to `https://app.staging.fullstory.com` for staging or `https://app.eu1.fullstory.com` for EU. |
+
+**For Lidar runs:**
+
+| Variable | What it is |
+| --- | --- |
+| `LIDAR_IOS_MCP_URL` | URL of the Lidar MCP endpoint |
+
+**For local Lidar development** (when you want to build and run Lidar from source):
+
+| Variable | What it is |
+| --- | --- |
+| `LOCAL_MCP_ORG_ID` | FullStory org ID for generating local MCP caps |
+| `LOCAL_MCP_EMAIL` | Email for the fake signed session |
+
+**For Appium runs:**
+
+| Variable | What it is |
+| --- | --- |
+| `MOBILE_CAPABILITIES_PATH` | Path to a capabilities JSON file, if env vars aren't enough |
+| `MOBILE_CONSOLE_LAUNCH_PATTERN` | Regex to detect app launch in console output |
+
+## Goal Files
+
+A goal file tells the runner what to do and what to check. Here's what goes in it:
+
+- `name` -- human name for the goal.
+- `run.targetScreen` -- the screen you're navigating to.
+- `run.slug` -- short name used in filenames.
+- `run.navigation` -- array of steps to get to the target screen. Each step has an `action` (`tap`, `scrollToLabel`) and a `label`. A tap can set `"ifVisible": true` to skip if the element isn't there.
+- `run.scrollDownCount` / `run.scrollUpCount` -- how many times to scroll once you're on the target screen.
+- `replayChecks` -- what the replay validation scripts look for.
+- `replayChecks.observationHeuristics` -- keyword lists that the observation scripts use to detect events, screen presence, image content, and scroll activity in replay evidence.
+- `sensitiveRegions` -- privacy expectations for specific parts of the UI.
+
+If `navigation` is missing, the runner just connects, snapshots, scrolls, and disconnects. No assumptions about how your app's navigation works.
+
+See `goals/example.json` for the full shape with comments.
+
+## Running
+
+### Against a remote Lidar server
+
+Set `LIDAR_IOS_MCP_URL` to the server, set your goal and output dir, and run:
+
+```
+node scripts/mobile/run-lidar-live-ios.mjs
+```
+
+### Against a local Lidar (build from source)
+
+This builds Lidar from your local Go checkout, starts Appium if it's not running, starts Lidar on free ports, generates MCP caps, and runs the goal. All you need in `.env.local` is device info, API key, org ID, and email.
+
+```
+node scripts/mobile/run-local-lidar-ios.mjs
+```
+
+Set `MOBILE_LIDAR_START=0` and `LIDAR_IOS_MCP_URL` to skip the build and reuse an existing Lidar.
+
+### Through Appium directly (no Lidar)
+
+Start Appium first:
+
+```
+pnpm exec appium --address 127.0.0.1 --port 4723 --base-path /
+```
+
+Then run:
+
+```
+node scripts/mobile/run-goal.mjs
+```
+
+## Replay Validation
+
+After the device run finishes, there are separate scripts to check the replay. You run them in order:
+
+1. **Validate device artifacts** -- checks that the expected files exist and contain what the goal says.
+   ```
+   node scripts/mobile/validate-goal-artifacts.mjs
+   ```
+
+2. **Fetch replay evidence** -- calls Subtext MCP to open the session and grab snapshots at several timestamps.
+   ```
+   node scripts/mobile/fetch-subtext-review-evidence.mjs
+   ```
+
+3. **Extract observations** -- reads the raw replay evidence and pulls out structured observations (which screen, which events, whether content was visible, etc).
+   ```
+   node scripts/mobile/capture-subtext-review-observations.mjs
+   ```
+
+4. **Validate observations** -- compares observations to the goal's expected checks and writes a pass/warn/fail report.
+   ```
+   node scripts/mobile/validate-replay-observations.mjs
+   ```
+
+You can also use `prepare-subtext-review.mjs` to generate a markdown review request instead of fetching evidence directly.
+
+## Output
+
+Everything goes in whatever you set `MOBILE_OUT_DIR` to. Typical files:
+
+- `live-ios-*.json` / `live-ios-*.txt` -- snapshots from Lidar.
+- `live-ios-*.png.base64` -- screenshot data.
+- `*-source.xml` / `*.png` -- Appium source and screenshots.
+- `fullstory-session-url.txt` -- the FullStory replay URL.
+- `replay-observations.json` -- structured observations from replay.
+- `*-validation-report.md` -- final report.
+
+## What's Not Committed
+
+The `.gitignore` keeps out anything app-specific:
+
+- `.env.local` -- your personal config.
+- `goals/*` except `example.json` -- your app-specific goals.
+- `tmp/` -- all run output.
+- `capabilities.local.json` -- your device capabilities.
+- Any `run-images-*` or `images-goal.mjs` scripts from internal testing.
+
+## Device Config
+
+For Appium runs, you can provide full capabilities as a JSON file:
+
+```
+MOBILE_CAPABILITIES_PATH=./scripts/mobile/capabilities.local.json node scripts/mobile/run-goal.mjs
+```
+
+Or as inline JSON:
+
+```
+MOBILE_CAPABILITIES_JSON='{"platformName":"iOS","appium:automationName":"XCUITest","appium:udid":"device-udid"}' node scripts/mobile/run-goal.mjs
+```
+
+## Replay Validation Details
+
+Replay checks are semantic, not pixel-perfect. The goal is to catch real problems:
+
+- Wrong screen in replay.
+- Missing events (taps, page properties).
+- Blank or frozen content where there should be images or text.
+- Privacy violations -- content visible when it should be masked, or masked when it should be visible.
+
+Privacy states you can set in goal manifests:
+
+- `unmasked` -- content should be visible.
+- `masked` -- content should be obscured.
+- `excluded` -- content should be blocked entirely.
+- `omitted` -- the element should not appear at all.
+- `config_dependent` -- depends on the org's privacy rules.