diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 97b8d42..ed15c8e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -67,13 +67,16 @@ If you only need the production bundle without the lint/format gate, use: npm run build:app ``` -If local port `3015` is already occupied, run Playwright on another isolated port: +If local port `3015` is already occupied, run the stable Playwright smoke on another isolated port: ```bash -PLAYWRIGHT_TEST_PORT=3016 npm run test:e2e +PLAYWRIGHT_TEST_PORT=3016 npm run test:e2e:ci ``` -The Playwright suite starts an isolated local app per worker under `.tmp-playwright/workers/` and should not reuse your normal local dashboard data. `npm run verify:package` builds the real tarball and verifies that the packaged CLI can start outside the repo checkout. +The Playwright suite starts an isolated local app per worker under `.tmp-playwright/workers/` and +should not reuse your normal local dashboard data. Use `npm run test:e2e` only when you intentionally +want the fresh app build plus the default local worker count. `npm run verify:package` builds the +real tarball and verifies that the packaged CLI can start outside the repo checkout. Then manually verify the main user flows touched by your change: diff --git a/README.md b/README.md index a17014a..8a93a0a 100644 --- a/README.md +++ b/README.md @@ -98,8 +98,8 @@ Then either: The auto-import path prefers: 1. local `toktrack` -2. `bunx toktrack@` -3. `npx --yes toktrack@` +2. `bunx` with the exact `toktrack` package spec pinned by this TTDash release +3. `npx --yes` with the exact `toktrack` package spec pinned by this TTDash release ## Common Commands @@ -177,21 +177,29 @@ Commands: Environment variables: -| Variable | Description | -| ----------------------- | --------------------------------------------------------- | -| `PORT` | Override the start port | -| `NO_OPEN_BROWSER=1` | Disable browser auto-open | -| `HOST` | Override the bind host, for example `HOST=0.0.0.0 ttdash` | -| `TTDASH_ALLOW_REMOTE=1` | Explicitly allow binding to a non-loopback host | +| Variable | Description | +| ----------------------------------------- | ------------------------------------------------------ | +| `PORT` | Override the start port | +| `NO_OPEN_BROWSER=1` | Disable browser auto-open | +| `HOST` | Override the bind host | +| `TTDASH_ALLOW_REMOTE=1` | Explicitly allow binding to a non-loopback host | +| `TTDASH_REMOTE_TOKEN=` | Required for non-loopback binds; use at least 24 chars | -Binding to a non-loopback host such as `0.0.0.0` exposes the local dashboard API to your network, including destructive routes for local data and settings resets. TTDash now refuses that bind unless you also set `TTDASH_ALLOW_REMOTE=1`. Only use this on trusted networks. +Binding to a non-loopback host such as `0.0.0.0` exposes the local dashboard API to your network, including destructive routes for local data and settings resets. TTDash refuses that bind unless you set both `TTDASH_ALLOW_REMOTE=1` and a `TTDASH_REMOTE_TOKEN` with at least 24 characters. Only use remote token access over a trusted LAN, VPN, or SSH tunnel; for any public hostname, put TTDash behind an HTTPS reverse proxy with valid TLS termination before sending the bearer token. Example: ```bash -TTDASH_ALLOW_REMOTE=1 HOST=0.0.0.0 ttdash +TTDASH_ALLOW_REMOTE=1 TTDASH_REMOTE_TOKEN= HOST=0.0.0.0 ttdash +curl -H "Authorization: Bearer $TTDASH_REMOTE_TOKEN" http://127.0.0.1:3000/api/usage ``` +When calling the server from another device, replace `127.0.0.1` with the server's LAN, VPN, or +SSH-tunneled host. For public hostnames, call an HTTPS reverse proxy URL instead; do not send the +bearer token over public HTTP. + +Remote API requests can authenticate with the `Authorization: Bearer $TTDASH_REMOTE_TOKEN` header or the equivalent `X-TTDash-Remote-Token: $TTDASH_REMOTE_TOKEN` header. + ## Features - Provider and model filtering across OpenAI, Anthropic, Google, and other imported providers @@ -371,12 +379,16 @@ To inspect the slowest suites and test cases after a Vitest run: npm run test:timings ``` -The Playwright suite starts an isolated local app per worker under `.tmp-playwright/workers/`. If the base port `3015` is already occupied locally, run it from another base port: +The Playwright suite starts an isolated local app per worker under `.tmp-playwright/workers/`. For +stable Playwright-only validation from another base port, use the CI-style worker cap: ```bash -PLAYWRIGHT_TEST_PORT=3016 npm run test:e2e +PLAYWRIGHT_TEST_PORT=3016 npm run test:e2e:ci ``` +Use `npm run test:e2e` when you intentionally want the fresh app build plus the default local worker +count. + Refresh the README screenshots: ```bash diff --git a/SECURITY.md b/SECURITY.md index 92f86fe..5c8c456 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -31,10 +31,19 @@ This project is maintained on a best-effort basis by a single maintainer. Report `TTDash` is intended to run as a local-first app on loopback by default. Binding it to a non-loopback host exposes local API routes for uploads, imports, resets, and report generation to your network. -Non-loopback binding therefore requires an explicit opt-in: +Non-loopback binding therefore requires an explicit opt-in and a remote token with at least 24 +characters. Only use remote token access over a trusted LAN, VPN, or SSH tunnel; for any public +hostname, put TTDash behind an HTTPS reverse proxy with valid TLS termination before sending the +bearer token. ```bash -TTDASH_ALLOW_REMOTE=1 HOST=0.0.0.0 ttdash +TTDASH_ALLOW_REMOTE=1 TTDASH_REMOTE_TOKEN= HOST=0.0.0.0 ttdash +curl -H "Authorization: Bearer $TTDASH_REMOTE_TOKEN" http://127.0.0.1:3000/api/usage ``` -Only use that mode on trusted networks. +When calling the server from another device, replace `127.0.0.1` with the server's LAN, VPN, or +SSH-tunneled host. For public hostnames, call an HTTPS reverse proxy URL instead; do not send the +bearer token over public HTTP. + +Remote API requests can authenticate with `Authorization: Bearer $TTDASH_REMOTE_TOKEN` or the +equivalent `X-TTDash-Remote-Token` header. Keep the token secret. diff --git a/docs/ttdash-dashboard-analytics.png b/docs/ttdash-dashboard-analytics.png index c8ae602..ae88237 100644 Binary files a/docs/ttdash-dashboard-analytics.png and b/docs/ttdash-dashboard-analytics.png differ diff --git a/docs/ttdash-dashboard-settings.png b/docs/ttdash-dashboard-settings.png index 9745dcb..92da138 100644 Binary files a/docs/ttdash-dashboard-settings.png and b/docs/ttdash-dashboard-settings.png differ diff --git a/docs/ttdash-dashboard.png b/docs/ttdash-dashboard.png index 3266b8c..ef491d4 100644 Binary files a/docs/ttdash-dashboard.png and b/docs/ttdash-dashboard.png differ diff --git a/scripts/capture-readme-screenshots.js b/scripts/capture-readme-screenshots.js index 366ca72..5553e48 100644 --- a/scripts/capture-readme-screenshots.js +++ b/scripts/capture-readme-screenshots.js @@ -4,44 +4,199 @@ const fs = require('fs'); const path = require('path'); const { execSync, spawn } = require('child_process'); const { chromium } = require('@playwright/test'); +const { createDefaultPersistedAppSettings } = require('../shared/app-settings.js'); +const { normalizeIncomingData } = require('../usage-normalizer.js'); +const { waitForRenderedChartData } = require('./rendered-chart-data.js'); const root = path.resolve(__dirname, '..'); const docsDir = path.join(root, 'docs'); const sampleUsagePath = path.join(root, 'examples', 'sample-usage.json'); +const screenshotRuntimeRoot = path.join(root, '.tmp-playwright', 'readme-screenshots'); +const screenshotServerRuntimeRoot = path.join(screenshotRuntimeRoot, 'app'); +const screenshotLocalAuthToken = 'ttdash-readme-screenshots-local-auth-token'; +const screenshotSeedLoadedAt = '2026-04-01T12:30:00.000Z'; const host = process.env.PLAYWRIGHT_TEST_HOST || '127.0.0.1'; const port = process.env.PLAYWRIGHT_TEST_PORT || '3017'; const baseUrl = `http://${host}:${port}`; +const secureDirMode = 0o700; +const secureFileMode = 0o600; function sleep(ms) { return new Promise((resolve) => setTimeout(resolve, ms)); } -async function waitForServer(url, timeoutMs = 30_000) { +function createScreenshotAuthSession(url = baseUrl, token = screenshotLocalAuthToken) { + const normalizedToken = String(token || '').trim(); + if (!normalizedToken) { + throw new Error('README screenshot local auth token is required.'); + } + + const bootstrapUrl = new URL(url); + bootstrapUrl.searchParams.set('ttdash_token', normalizedToken); + + return { + authorizationHeader: `Bearer ${normalizedToken}`, + bootstrapUrl: bootstrapUrl.href, + }; +} + +function createAuthHeaders(authSession) { + return authSession?.authorizationHeader + ? { Authorization: authSession.authorizationHeader } + : undefined; +} + +function getErrorMessage(error) { + return error instanceof Error ? error.message : String(error); +} + +function writeJsonAtomic(filePath, data) { + fs.mkdirSync(path.dirname(filePath), { recursive: true, mode: secureDirMode }); + const tempPath = `${filePath}.${process.pid}.${Date.now()}.tmp`; + + try { + fs.writeFileSync(tempPath, JSON.stringify(data, null, 2), { + mode: secureFileMode, + }); + if (process.platform !== 'win32') { + fs.chmodSync(tempPath, secureFileMode); + } + fs.renameSync(tempPath, filePath); + if (process.platform !== 'win32') { + fs.chmodSync(filePath, secureFileMode); + } + } catch (error) { + try { + fs.unlinkSync(tempPath); + } catch (cleanupError) { + if (cleanupError?.code !== 'ENOENT') { + throw new AggregateError( + [error, cleanupError], + `Failed atomic JSON write and temp-file cleanup for ${path.basename(filePath)}.`, + ); + } + } + throw error; + } +} + +function readSampleUsage(filePath = sampleUsagePath) { + try { + return JSON.parse(fs.readFileSync(filePath, 'utf8')); + } catch (error) { + throw new Error( + `Failed to read README screenshot sample usage data from ${filePath}: ${getErrorMessage( + error, + )}`, + ); + } +} + +function normalizeSampleUsage(usagePayload) { + try { + return normalizeIncomingData(usagePayload); + } catch (error) { + throw new Error( + `Failed to normalize README screenshot sample usage data: ${getErrorMessage(error)}`, + ); + } +} + +function seedSampleUsageFile({ + runtimeRoot = screenshotServerRuntimeRoot, + loadedAt = screenshotSeedLoadedAt, + sampleUsage, + sampleUsageFile = sampleUsagePath, +} = {}) { + const usageData = normalizeSampleUsage( + sampleUsage === undefined ? readSampleUsage(sampleUsageFile) : sampleUsage, + ); + const settings = { + ...createDefaultPersistedAppSettings(), + lastLoadedAt: loadedAt, + lastLoadSource: 'file', + }; + const dataFile = path.join(runtimeRoot, 'data', 'data.json'); + const settingsFile = path.join(runtimeRoot, 'config', 'settings.json'); + + writeJsonAtomic(dataFile, usageData); + writeJsonAtomic(settingsFile, settings); + + return { + dataFile, + settings, + settingsFile, + usageData, + }; +} + +function createWaitForServerTimeoutError(url) { + return new Error(`Timed out waiting for screenshot server: ${url}`); +} + +function getRemainingTimeoutMs(startedAt, timeoutMs) { + return Math.max(0, timeoutMs - (Date.now() - startedAt)); +} + +async function fetchUsageWithTimeout(url, { authSession, fetchImpl, timeoutMs }) { + const controller = new AbortController(); + let timeoutId; + const timeoutPromise = new Promise((_, reject) => { + timeoutId = setTimeout(() => { + controller.abort(); + reject(createWaitForServerTimeoutError(url)); + }, timeoutMs); + }); + + try { + return await Promise.race([ + fetchImpl(`${url}/api/usage`, { + headers: createAuthHeaders(authSession), + signal: controller.signal, + }), + timeoutPromise, + ]); + } finally { + clearTimeout(timeoutId); + } +} + +async function waitForServer( + url, + { + timeoutMs = 30_000, + pollMs = 250, + fetchImpl = fetch, + sleepImpl = sleep, + authSession = createScreenshotAuthSession(url), + } = {}, +) { const startedAt = Date.now(); - while (Date.now() - startedAt < timeoutMs) { + while (true) { + const remainingTimeoutMs = getRemainingTimeoutMs(startedAt, timeoutMs); + if (remainingTimeoutMs <= 0) { + throw createWaitForServerTimeoutError(url); + } + try { - const response = await fetch(`${url}/api/usage`); + const response = await fetchUsageWithTimeout(url, { + authSession, + fetchImpl, + timeoutMs: remainingTimeoutMs, + }); if (response.ok) { - return; + return authSession; } } catch { // Keep polling until the local server is reachable. } - await sleep(250); + const sleepMs = Math.min(pollMs, getRemainingTimeoutMs(startedAt, timeoutMs)); + if (sleepMs > 0) { + await sleepImpl(sleepMs); + } } - - throw new Error(`Timed out waiting for screenshot server: ${url}`); -} - -async function uploadSampleUsage(page) { - await page.locator('[data-testid="usage-upload-input"]').setInputFiles(sampleUsagePath); - await page - .getByText( - /^(Datei sample-usage\.json erfolgreich geladen|File sample-usage\.json loaded successfully)$/, - ) - .waitFor(); } async function switchToEnglish(page) { @@ -49,8 +204,72 @@ async function switchToEnglish(page) { await page.getByText('Filter status').waitFor(); } +function isChildProcessRunning(childProcess) { + return ( + childProcess && + childProcess.killed !== true && + childProcess.exitCode === null && + childProcess.signalCode === null + ); +} + +async function closeBrowserResources(context, browser) { + await context?.close().catch(() => {}); + await browser?.close().catch(() => {}); +} + +async function stopServer(server, { shutdownGraceMs = 5_000 } = {}) { + if (!isChildProcessRunning(server)) { + return; + } + + await new Promise((resolve) => { + let timeoutId; + let settled = false; + const cleanup = () => { + if (timeoutId) { + clearTimeout(timeoutId); + } + server.off('close', finish); + server.off('error', finish); + }; + const finish = () => { + if (settled) { + return; + } + settled = true; + cleanup(); + resolve(); + }; + + server.once('close', finish); + server.once('error', finish); + timeoutId = setTimeout(() => { + try { + if (isChildProcessRunning(server)) { + server.kill('SIGKILL'); + } + } catch { + // Ignore shutdown escalation failures during cleanup. + } + finish(); + }, shutdownGraceMs); + timeoutId.unref?.(); + + try { + if (!server.kill('SIGTERM')) { + finish(); + } + } catch { + finish(); + } + }); +} + async function captureScreenshots() { fs.mkdirSync(docsDir, { recursive: true }); + fs.rmSync(screenshotRuntimeRoot, { recursive: true, force: true }); + fs.mkdirSync(screenshotRuntimeRoot, { recursive: true }); execSync('npm run build:app', { cwd: root, @@ -64,57 +283,77 @@ async function captureScreenshots() { NO_OPEN_BROWSER: '1', PLAYWRIGHT_TEST_HOST: host, PLAYWRIGHT_TEST_PORT: String(port), + PLAYWRIGHT_TEST_RUNTIME_ROOT: screenshotServerRuntimeRoot, + TTDASH_LOCAL_AUTH_TOKEN: screenshotLocalAuthToken, }, stdio: 'inherit', }); try { - await waitForServer(baseUrl); - - const browser = await chromium.launch(); - const context = await browser.newContext({ - viewport: { width: 1600, height: 1400 }, - colorScheme: 'dark', - }); - const page = await context.newPage(); + let browser; + let context; + const authSession = createScreenshotAuthSession(baseUrl); + await waitForServer(baseUrl, { authSession }); + seedSampleUsageFile(); - await page.addInitScript(() => { - globalThis.__TTDASH_TEST_HOOKS__ = {}; - }); + browser = await chromium.launch(); + try { + context = await browser.newContext({ + viewport: { width: 1600, height: 1400 }, + colorScheme: 'dark', + reducedMotion: 'reduce', + }); + const page = await context.newPage(); - await page.goto(baseUrl); - await uploadSampleUsage(page); - await switchToEnglish(page); + await page.addInitScript(() => { + globalThis.__TTDASH_TEST_HOOKS__ = {}; + }); - await page.evaluate(() => globalThis.scrollTo(0, 0)); - await page.screenshot({ - path: path.join(docsDir, 'ttdash-dashboard.png'), - }); + await page.goto(authSession?.bootstrapUrl || baseUrl); + await switchToEnglish(page); - await page.locator('#charts').scrollIntoViewIfNeeded(); - await sleep(500); - await page.locator('#charts').screenshot({ - path: path.join(docsDir, 'ttdash-dashboard-analytics.png'), - }); + await page.evaluate(() => globalThis.scrollTo(0, 0)); + await page.screenshot({ + path: path.join(docsDir, 'ttdash-dashboard.png'), + }); - await page.evaluate(() => { - globalThis.__TTDASH_TEST_HOOKS__?.openSettings?.(); - }); - await page.getByRole('dialog').waitFor(); - await sleep(300); - await page.getByRole('dialog').screenshot({ - path: path.join(docsDir, 'ttdash-dashboard-settings.png'), - }); + await page.locator('#charts').scrollIntoViewIfNeeded(); + await waitForRenderedChartData(page, { sectionSelector: '#charts' }); + await page.locator('#charts').screenshot({ + path: path.join(docsDir, 'ttdash-dashboard-analytics.png'), + }); - await context.close(); - await browser.close(); + await page.evaluate(() => { + globalThis.__TTDASH_TEST_HOOKS__?.openSettings?.(); + }); + const dialog = page.getByRole('dialog'); + await dialog.waitFor({ state: 'visible' }); + await dialog.screenshot({ + path: path.join(docsDir, 'ttdash-dashboard-settings.png'), + }); + } finally { + await closeBrowserResources(context, browser); + } } finally { - server.kill('SIGTERM'); - await new Promise((resolve) => server.once('close', resolve)); + await stopServer(server); } } -captureScreenshots().catch((error) => { - console.error(error); - process.exitCode = 1; -}); +if (require.main === module) { + captureScreenshots().catch((error) => { + console.error(error); + process.exitCode = 1; + }); +} + +module.exports = { + closeBrowserResources, + createAuthHeaders, + createScreenshotAuthSession, + isChildProcessRunning, + seedSampleUsageFile, + screenshotLocalAuthToken, + screenshotSeedLoadedAt, + stopServer, + waitForServer, +}; diff --git a/scripts/rendered-chart-data.js b/scripts/rendered-chart-data.js new file mode 100644 index 0000000..bf60110 --- /dev/null +++ b/scripts/rendered-chart-data.js @@ -0,0 +1,78 @@ +const renderedChartDataSelector = [ + 'path.recharts-line-curve', + 'path.recharts-area-area', + 'path.recharts-sector', + '.recharts-bar-rectangle path', + 'path.recharts-rectangle', +].join(','); + +async function countRenderedChartDataShapes(section) { + return section.evaluate((element, selector) => { + return Array.from(element.querySelectorAll(selector)).filter((node) => { + const style = globalThis.getComputedStyle(node); + const box = node.getBoundingClientRect(); + const totalLength = typeof node.getTotalLength === 'function' ? node.getTotalLength() : 0; + + return ( + style.display !== 'none' && + style.visibility !== 'hidden' && + Number(style.opacity || '1') > 0 && + (totalLength > 10 || (box.width > 2 && box.height > 2)) + ); + }).length; + }, renderedChartDataSelector); +} + +function isTimeoutError(error) { + return ( + error instanceof Error && (error.name === 'TimeoutError' || /timeout/i.test(error.message)) + ); +} + +async function waitForRenderedChartData( + page, + { + minShapes = 1, + pollMs = 100, + sectionSelector = '#charts', + sleepImpl = (ms) => new Promise((resolve) => setTimeout(resolve, ms)), + timeoutMs = 10_000, + } = {}, +) { + const section = page.locator(sectionSelector); + const startedAt = Date.now(); + const timeoutError = () => + new Error(`Timed out waiting for rendered chart data in ${sectionSelector}`); + const remainingTimeoutMs = () => Math.max(0, timeoutMs - (Date.now() - startedAt)); + + try { + await section.waitFor({ timeout: remainingTimeoutMs() }); + } catch (error) { + if (remainingTimeoutMs() <= 0 && isTimeoutError(error)) { + throw timeoutError(); + } + + throw error; + } + + while (remainingTimeoutMs() > 0) { + if ((await countRenderedChartDataShapes(section)) >= minShapes) { + return; + } + + const remainingBeforeSleepMs = remainingTimeoutMs(); + if (remainingBeforeSleepMs <= 0) { + break; + } + + await sleepImpl(Math.min(pollMs, remainingBeforeSleepMs)); + } + + throw timeoutError(); +} + +module.exports = { + countRenderedChartDataShapes, + renderedChartDataSelector, + waitForRenderedChartData, +}; diff --git a/tests/e2e/dashboard-load-upload.spec.ts b/tests/e2e/dashboard-load-upload.spec.ts index 34d1c3e..a385767 100644 --- a/tests/e2e/dashboard-load-upload.spec.ts +++ b/tests/e2e/dashboard-load-upload.spec.ts @@ -1,9 +1,16 @@ -import { expect, test } from './fixtures' +import { chartCardByTitle, expect, test, type Page } from './fixtures' import { gotoDashboard, resetAppState, uploadSampleUsage } from './helpers' +import renderedChartDataHelpers from '../../scripts/rendered-chart-data.js' + +const { countRenderedChartDataShapes } = renderedChartDataHelpers as { + countRenderedChartDataShapes: (section: ReturnType) => Promise +} const importEntryButtonPattern = /^(Auto-Import|Auto import|Import)$/ const uploadEntryButtonPattern = /^(Datei hochladen|Upload file|Upload)$/ const csvButtonPattern = /^(CSV|CSV exportieren|Export CSV)$/ +const cumulativeProviderCostPattern = /Cumulative cost per provider|Kumulative Kosten pro Anbieter/ +const costByModelOverTimePattern = /Cost by model over time|Kosten nach Modell im Zeitverlauf/ test('uploads sample usage data and renders the dashboard without browser errors', async ({ page, @@ -58,10 +65,19 @@ test('shows cumulative provider cost next to model cost trends in cost analysis' await costAnalysisSection.scrollIntoViewIfNeeded() await expect(costAnalysisSection.getByText(/Cost analysis|Kostenanalyse/)).toBeVisible() - await expect( - costAnalysisSection.getByText(/Cumulative cost per provider|Kumulative Kosten pro Anbieter/), - ).toBeVisible() - await expect( - costAnalysisSection.getByText(/Cost by model over time|Kosten nach Modell im Zeitverlauf/), - ).toBeVisible() + await expect(costAnalysisSection.getByText(cumulativeProviderCostPattern)).toBeVisible() + await expect(costAnalysisSection.getByText(costByModelOverTimePattern)).toBeVisible() + + const cumulativeProviderCostChart = chartCardByTitle( + costAnalysisSection, + cumulativeProviderCostPattern, + ) + const costByModelOverTimeChart = chartCardByTitle(costAnalysisSection, costByModelOverTimePattern) + + await expect + .poll(async () => countRenderedChartDataShapes(cumulativeProviderCostChart)) + .toBeGreaterThan(0) + await expect + .poll(async () => countRenderedChartDataShapes(costByModelOverTimeChart)) + .toBeGreaterThan(0) }) diff --git a/tests/e2e/fixtures.ts b/tests/e2e/fixtures.ts index 220f3d4..ad22a39 100644 --- a/tests/e2e/fixtures.ts +++ b/tests/e2e/fixtures.ts @@ -23,6 +23,15 @@ function sleep(ms: number) { return new Promise((resolve) => setTimeout(resolve, ms)) } +export function chartCardByTitle(section: ReturnType, titlePattern: RegExp) { + return section + .getByText(titlePattern) + .first() + .locator( + 'xpath=ancestor::*[contains(concat(" ", normalize-space(@class), " "), " group ") and contains(concat(" ", normalize-space(@class), " "), " relative ")][1]', + ) +} + function buildWorkerServer(workerIndex: number) { const port = basePort + workerIndex const authToken = `ttdash-playwright-local-auth-token-worker-${workerIndex}-port-${port}` diff --git a/tests/unit/documentation-contract.test.ts b/tests/unit/documentation-contract.test.ts new file mode 100644 index 0000000..b9e4156 --- /dev/null +++ b/tests/unit/documentation-contract.test.ts @@ -0,0 +1,66 @@ +import { readFile } from 'node:fs/promises' +import { describe, expect, it } from 'vitest' + +async function readRepoFile(path: string) { + return readFile(path, 'utf8') +} + +describe('documentation contracts', () => { + it('keeps remote binding examples aligned with token authentication', async () => { + const docs = { + 'README.md': await readRepoFile('README.md'), + 'SECURITY.md': await readRepoFile('SECURITY.md'), + } + const remoteBindWithoutTokenPattern = + /^(?![^\n`]*TTDASH_REMOTE_TOKEN)(?=[^\n`]*TTDASH_ALLOW_REMOTE=1)(?=[^\n`]*HOST=0\.0\.0\.0)[^\n`]*/m + const bindAddressApiUrlPattern = /http:\/\/0\.0\.0\.0(?::\d+)?\/api\/usage/ + + for (const [path, content] of Object.entries(docs)) { + expect(content, path).not.toMatch(remoteBindWithoutTokenPattern) + expect(content, path).not.toMatch(bindAddressApiUrlPattern) + expect(content, path).toContain('TTDASH_REMOTE_TOKEN=') + expect(content, path).toContain('trusted LAN, VPN, or SSH tunnel') + expect(content, path).toContain('HTTPS reverse proxy') + expect(content, path).toContain('do not send the') + } + + expect(docs['README.md']).toContain('Authorization: Bearer $TTDASH_REMOTE_TOKEN') + expect(docs['README.md']).toContain('X-TTDash-Remote-Token') + expect(docs['SECURITY.md']).toContain('Authorization: Bearer $TTDASH_REMOTE_TOKEN') + expect(docs['SECURITY.md']).toContain('X-TTDash-Remote-Token') + }) + + it('keeps README toktrack fallback docs version-agnostic', async () => { + const readme = await readRepoFile('README.md') + const firstRunStart = readme.indexOf('## First Run') + const commonCommandsStart = readme.indexOf('## Common Commands') + + expect(firstRunStart, 'README must have a First Run section').toBeGreaterThanOrEqual(0) + expect(commonCommandsStart, 'README must have Common Commands after First Run').toBeGreaterThan( + firstRunStart, + ) + + const firstRunSection = readme.slice(firstRunStart, commonCommandsStart) + + expect(firstRunSection).toContain('exact `toktrack` package spec pinned by this TTDash release') + expect(firstRunSection).toContain('`bunx`') + expect(firstRunSection).toContain('`npx --yes`') + expect(firstRunSection).not.toMatch(/toktrack@\d+\.\d+\.\d+/) + expect(firstRunSection).not.toContain('toktrack@') + }) + + it('keeps contributor Playwright docs on the stable local command path', async () => { + const docs = { + 'README.md': await readRepoFile('README.md'), + 'CONTRIBUTING.md': await readRepoFile('CONTRIBUTING.md'), + } + const uncappedPortOverridePattern = + /PLAYWRIGHT_TEST_PORT=3016 npm run test:e2e(?!:(?:ci|parallel|smoke))/ + + for (const [path, content] of Object.entries(docs)) { + expect(content, path).toContain('npm run verify:full') + expect(content, path).toContain('PLAYWRIGHT_TEST_PORT=3016 npm run test:e2e:ci') + expect(content, path).not.toMatch(uncappedPortOverridePattern) + } + }) +}) diff --git a/tests/unit/readme-screenshots-script.test.ts b/tests/unit/readme-screenshots-script.test.ts new file mode 100644 index 0000000..207da3e --- /dev/null +++ b/tests/unit/readme-screenshots-script.test.ts @@ -0,0 +1,347 @@ +import { EventEmitter } from 'node:events' +import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises' +import { createRequire } from 'node:module' +import { tmpdir } from 'node:os' +import path from 'node:path' +import { afterEach, describe, expect, it, vi } from 'vitest' + +const require = createRequire(import.meta.url) +const { + closeBrowserResources, + createAuthHeaders, + createScreenshotAuthSession, + isChildProcessRunning, + seedSampleUsageFile, + stopServer, + waitForServer, +} = require('../../scripts/capture-readme-screenshots.js') as { + closeBrowserResources: ( + context?: { close: () => Promise }, + browser?: { close: () => Promise }, + ) => Promise + createAuthHeaders: (authSession: { authorizationHeader?: string } | null) => HeadersInit + createScreenshotAuthSession: ( + url?: string, + token?: string, + ) => { + authorizationHeader: string + bootstrapUrl: string + } + seedSampleUsageFile: (options?: { + loadedAt?: string + sampleUsage?: unknown + sampleUsageFile?: string + runtimeRoot?: string + }) => { + dataFile: string + settings: { + language: string + lastLoadedAt: string | null + lastLoadSource: string | null + theme: string + } + settingsFile: string + usageData: { + daily: Array<{ date: string; modelsUsed: string[] }> + totals: { + requestCount: number + totalCost: number + totalTokens: number + } + } + } + isChildProcessRunning: (childProcess: { + exitCode: number | null + killed?: boolean + signalCode: string | null + }) => boolean + waitForServer: ( + url: string, + options: { + authSession?: { authorizationHeader?: string } | null + fetchImpl?: typeof fetch + pollMs?: number + sleepImpl?: (ms: number) => Promise + timeoutMs?: number + }, + ) => Promise<{ authorizationHeader: string; bootstrapUrl: string } | null> + stopServer: ( + server: EventEmitter & { + exitCode: number | null + kill: (signal: string) => boolean + killed?: boolean + signalCode: string | null + }, + options?: { shutdownGraceMs?: number }, + ) => Promise +} + +const tempDirs: string[] = [] + +afterEach(async () => { + vi.restoreAllMocks() + vi.useRealTimers() + + await Promise.all(tempDirs.splice(0).map((dir) => rm(dir, { recursive: true, force: true }))) +}) + +async function createTempDir() { + const dir = await mkdtemp(path.join(tmpdir(), 'ttdash-screenshots-')) + tempDirs.push(dir) + return dir +} + +function createSampleUsage() { + return { + daily: [ + { + date: '2026-04-01', + inputTokens: 1, + outputTokens: 2, + cacheCreationTokens: 3, + cacheReadTokens: 4, + thinkingTokens: 5, + totalCost: 1.25, + requestCount: 2, + modelBreakdowns: [ + { + modelName: 'GPT-Test', + inputTokens: 1, + outputTokens: 2, + cacheCreationTokens: 3, + cacheReadTokens: 4, + thinkingTokens: 5, + cost: 1.25, + requestCount: 2, + }, + ], + }, + ], + } +} + +async function readJson(filePath: string) { + return JSON.parse(await readFile(filePath, 'utf8')) as unknown +} + +describe('README screenshot script helpers', () => { + it('creates deterministic local auth session metadata for the screenshot server', () => { + const token = 'local-token-12345678901234567890' + + expect(createScreenshotAuthSession('http://127.0.0.1:3018', token)).toEqual({ + authorizationHeader: `Bearer ${token}`, + bootstrapUrl: `http://127.0.0.1:3018/?ttdash_token=${token}`, + }) + }) + + it('uses the deterministic auth header while polling the protected usage API', async () => { + const fetchImpl = vi.fn(async (_url: string, options?: RequestInit) => { + const authorization = new Headers(options?.headers).get('Authorization') + return new Response('{}', { + status: authorization === 'Bearer local-token' ? 200 : 401, + }) + }) as typeof fetch + const authSession = { + authorizationHeader: 'Bearer local-token', + bootstrapUrl: 'http://127.0.0.1:3018/?ttdash_token=local-token', + } + + await expect( + waitForServer('http://127.0.0.1:3018', { + authSession, + fetchImpl, + pollMs: 0, + sleepImpl: async () => {}, + timeoutMs: 1000, + }), + ).resolves.toEqual(authSession) + + expect(fetchImpl).toHaveBeenCalledWith('http://127.0.0.1:3018/api/usage', { + headers: { Authorization: 'Bearer local-token' }, + signal: expect.any(AbortSignal), + }) + }) + + it('caps polling sleeps to the remaining server wait budget', async () => { + let currentTimeMs = 0 + vi.spyOn(Date, 'now').mockImplementation(() => currentTimeMs) + + const fetchImpl = vi.fn(async (_url: string, options?: RequestInit) => { + expect(options?.signal).toBeInstanceOf(AbortSignal) + currentTimeMs = 900 + return new Response('{}', { status: 401 }) + }) as typeof fetch + const sleepImpl = vi.fn(async (ms: number) => { + currentTimeMs += ms + }) + + await expect( + waitForServer('http://127.0.0.1:3018', { + authSession: { + authorizationHeader: 'Bearer local-token', + bootstrapUrl: 'http://127.0.0.1:3018/?ttdash_token=local-token', + }, + fetchImpl, + pollMs: 250, + sleepImpl, + timeoutMs: 1000, + }), + ).rejects.toThrow('Timed out waiting for screenshot server: http://127.0.0.1:3018') + + expect(sleepImpl).toHaveBeenCalledWith(100) + }) + + it('aborts in-flight readiness requests when the server wait budget expires', async () => { + vi.useFakeTimers() + let signal: AbortSignal | undefined + const fetchImpl = vi.fn((_url: string, options?: RequestInit) => { + signal = options?.signal + return new Promise((_resolve, reject) => { + signal?.addEventListener('abort', () => reject(new DOMException('Aborted', 'AbortError')), { + once: true, + }) + }) + }) as unknown as typeof fetch + + const waitPromise = waitForServer('http://127.0.0.1:3018', { + authSession: { + authorizationHeader: 'Bearer local-token', + bootstrapUrl: 'http://127.0.0.1:3018/?ttdash_token=local-token', + }, + fetchImpl, + pollMs: 10, + sleepImpl: async () => {}, + timeoutMs: 50, + }) + const assertion = expect(waitPromise).rejects.toThrow( + 'Timed out waiting for screenshot server: http://127.0.0.1:3018', + ) + + await vi.advanceTimersByTimeAsync(50) + + await assertion + expect(signal?.aborted).toBe(true) + }) + + it('omits auth headers until a local auth session exists', () => { + expect(createAuthHeaders(null)).toBeUndefined() + expect(createAuthHeaders({ authorizationHeader: '' })).toBeUndefined() + }) + + it('seeds normalized sample usage into the isolated screenshot runtime', async () => { + const runtimeRoot = await createTempDir() + const sampleUsageFile = path.join(runtimeRoot, 'sample-usage.json') + await writeFile(sampleUsageFile, JSON.stringify(createSampleUsage())) + + const result = seedSampleUsageFile({ + loadedAt: '2026-05-05T12:00:00.000Z', + runtimeRoot, + sampleUsageFile, + }) + + const usageData = (await readJson(path.join(runtimeRoot, 'data', 'data.json'))) as { + daily: Array<{ modelsUsed: string[] }> + totals: { requestCount: number; totalCost: number; totalTokens: number } + } + const settings = (await readJson(path.join(runtimeRoot, 'config', 'settings.json'))) as { + language: string + lastLoadedAt: string | null + lastLoadSource: string | null + theme: string + } + + expect(result.dataFile).toBe(path.join(runtimeRoot, 'data', 'data.json')) + expect(result.settingsFile).toBe(path.join(runtimeRoot, 'config', 'settings.json')) + expect(usageData.daily[0]?.modelsUsed).toEqual(['GPT-Test']) + expect(usageData.totals).toMatchObject({ + requestCount: 2, + totalCost: 1.25, + totalTokens: 15, + }) + expect(settings).toMatchObject({ + language: 'de', + lastLoadedAt: '2026-05-05T12:00:00.000Z', + lastLoadSource: 'file', + theme: 'dark', + }) + }) + + it('rejects invalid sample usage before writing screenshot runtime data', async () => { + const runtimeRoot = await createTempDir() + + expect(() => + seedSampleUsageFile({ + runtimeRoot, + sampleUsage: { invalid: true }, + }), + ).toThrow('Failed to normalize README screenshot sample usage data') + }) + + it('closes browser resources even when one close operation fails', async () => { + const context = { + close: vi.fn(async () => { + throw new Error('context close failed') + }), + } + const browser = { + close: vi.fn(async () => {}), + } + + await expect(closeBrowserResources(context, browser)).resolves.toBeUndefined() + + expect(context.close).toHaveBeenCalledTimes(1) + expect(browser.close).toHaveBeenCalledTimes(1) + }) + + it('stops only live screenshot server child processes', async () => { + const runningServer = Object.assign(new EventEmitter(), { + exitCode: null, + kill: vi.fn(() => { + runningServer.emit('close') + return true + }), + killed: false, + signalCode: null, + }) + const closedServer = Object.assign(new EventEmitter(), { + exitCode: 0, + kill: vi.fn(() => true), + killed: false, + signalCode: null, + }) + + expect(isChildProcessRunning(runningServer)).toBe(true) + expect(isChildProcessRunning(closedServer)).toBe(false) + + await stopServer(runningServer) + await stopServer(closedServer) + + expect(runningServer.kill).toHaveBeenCalledWith('SIGTERM') + expect(closedServer.kill).not.toHaveBeenCalled() + }) + + it('escalates screenshot server shutdown after the grace period', async () => { + vi.useFakeTimers() + const runningServer = Object.assign(new EventEmitter(), { + exitCode: null, + kill: vi.fn((signal: string) => { + if (signal === 'SIGKILL') { + runningServer.signalCode = 'SIGKILL' + } + return true + }), + killed: false, + signalCode: null as string | null, + }) + + const stopPromise = stopServer(runningServer, { shutdownGraceMs: 50 }) + + expect(runningServer.kill).toHaveBeenCalledWith('SIGTERM') + await vi.advanceTimersByTimeAsync(50) + await expect(stopPromise).resolves.toBeUndefined() + + expect(runningServer.kill).toHaveBeenCalledWith('SIGKILL') + expect(runningServer.listenerCount('close')).toBe(0) + expect(runningServer.listenerCount('error')).toBe(0) + }) +}) diff --git a/tests/unit/rendered-chart-data.test.ts b/tests/unit/rendered-chart-data.test.ts new file mode 100644 index 0000000..1c51ea9 --- /dev/null +++ b/tests/unit/rendered-chart-data.test.ts @@ -0,0 +1,139 @@ +import { createRequire } from 'node:module' +import { afterEach, describe, expect, it, vi } from 'vitest' + +const require = createRequire(import.meta.url) +const { renderedChartDataSelector, waitForRenderedChartData } = + require('../../scripts/rendered-chart-data.js') as { + renderedChartDataSelector: string + waitForRenderedChartData: ( + page: { + locator: (selector: string) => { + evaluate: () => Promise + waitFor: (options?: { timeout?: number }) => Promise + } + }, + options?: { + minShapes?: number + pollMs?: number + sectionSelector?: string + sleepImpl?: (ms: number) => Promise + timeoutMs?: number + }, + ) => Promise + } + +afterEach(() => { + vi.restoreAllMocks() + vi.useRealTimers() +}) + +describe('rendered chart data helpers', () => { + it('waits for rendered chart data shapes before capturing analytics screenshots', async () => { + const evaluate = vi + .fn<() => Promise>() + .mockResolvedValueOnce(0) + .mockResolvedValueOnce(4) + const waitFor = vi.fn(async () => {}) + const locator = vi.fn(() => ({ evaluate, waitFor })) + + await waitForRenderedChartData( + { locator }, + { + minShapes: 3, + pollMs: 0, + sectionSelector: '#charts', + sleepImpl: async () => {}, + timeoutMs: 1000, + }, + ) + + expect(renderedChartDataSelector).toContain('recharts-line-curve') + expect(locator).toHaveBeenCalledWith('#charts') + const waitForTimeout = waitFor.mock.calls[0]?.[0]?.timeout + expect(waitForTimeout).toBeGreaterThan(0) + expect(waitForTimeout).toBeLessThanOrEqual(1000) + expect(evaluate).toHaveBeenCalledTimes(2) + }) + + it('normalizes section wait timeout errors when the chart budget expires', async () => { + let currentTimeMs = 0 + vi.spyOn(Date, 'now').mockImplementation(() => currentTimeMs) + + const timeoutError = new Error('Timeout 1000ms exceeded') + timeoutError.name = 'TimeoutError' + const evaluate = vi.fn<() => Promise>() + const waitFor = vi.fn(async () => { + currentTimeMs = 1000 + throw timeoutError + }) + const locator = vi.fn(() => ({ evaluate, waitFor })) + + await expect( + waitForRenderedChartData( + { locator }, + { + sectionSelector: '#charts', + timeoutMs: 1000, + }, + ), + ).rejects.toThrow('Timed out waiting for rendered chart data in #charts') + + expect(evaluate).not.toHaveBeenCalled() + }) + + it('preserves non-timeout section wait errors after the chart budget expires', async () => { + let currentTimeMs = 0 + vi.spyOn(Date, 'now').mockImplementation(() => currentTimeMs) + + const sectionError = new Error('Locator resolved to multiple chart sections') + const evaluate = vi.fn<() => Promise>() + const waitFor = vi.fn(async () => { + currentTimeMs = 1000 + throw sectionError + }) + const locator = vi.fn(() => ({ evaluate, waitFor })) + + await expect( + waitForRenderedChartData( + { locator }, + { + sectionSelector: '#charts', + timeoutMs: 1000, + }, + ), + ).rejects.toBe(sectionError) + + expect(evaluate).not.toHaveBeenCalled() + }) + + it('uses one shared timeout budget while waiting for chart data', async () => { + let currentTimeMs = 0 + vi.spyOn(Date, 'now').mockImplementation(() => currentTimeMs) + + const evaluate = vi.fn<() => Promise>().mockResolvedValue(0) + const waitFor = vi.fn(async () => { + currentTimeMs = 800 + }) + const locator = vi.fn(() => ({ evaluate, waitFor })) + const sleepImpl = vi.fn(async (ms: number) => { + currentTimeMs += ms + }) + + await expect( + waitForRenderedChartData( + { locator }, + { + minShapes: 3, + pollMs: 250, + sectionSelector: '#charts', + sleepImpl, + timeoutMs: 1000, + }, + ), + ).rejects.toThrow('Timed out waiting for rendered chart data in #charts') + + expect(waitFor).toHaveBeenCalledWith({ timeout: 1000 }) + expect(evaluate).toHaveBeenCalledTimes(1) + expect(sleepImpl).toHaveBeenCalledWith(200) + }) +})