pdf-server: annotations, interact tool, page extraction & prompt engineering#506
Merged
pdf-server: annotations, interact tool, page extraction & prompt engineering#506
Conversation
@modelcontextprotocol/ext-apps
@modelcontextprotocol/server-basic-preact
@modelcontextprotocol/server-basic-react
@modelcontextprotocol/server-basic-solid
@modelcontextprotocol/server-basic-svelte
@modelcontextprotocol/server-basic-vanillajs
@modelcontextprotocol/server-basic-vue
@modelcontextprotocol/server-budget-allocator
@modelcontextprotocol/server-cohort-heatmap
@modelcontextprotocol/server-customer-segmentation
@modelcontextprotocol/server-debug
@modelcontextprotocol/server-map
@modelcontextprotocol/server-pdf
@modelcontextprotocol/server-scenario-modeler
@modelcontextprotocol/server-shadertoy
@modelcontextprotocol/server-sheet-music
@modelcontextprotocol/server-system-monitor
@modelcontextprotocol/server-threejs
@modelcontextprotocol/server-transcript
@modelcontextprotocol/server-video-resource
@modelcontextprotocol/server-wiki-explorer
commit: |
Add PDF annotation system with 7 annotation types (highlight, underline, strikethrough, note, rectangle, freetext, stamp), text-based highlighting, form filling, and annotated PDF download using pdf-lib. - Server: annotation Zod schemas, extended interact tool with add/update/remove annotations, highlight_text, and fill_form actions - Client: annotation layer rendering with PDF coordinate conversion, persistence via localStorage (using toolInfo.id key), pdf-lib-based download with embedded annotations and form fills, uses app.downloadFile() SDK with <a> fallback - Model context includes annotation summary Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New tool `get_pages` lets the model get text and/or screenshots from arbitrary page ranges without navigating the visible viewer. - Server: `get_pages` tool with interval-based page ranges (optional start/end, open ranges supported), `getText`/`getScreenshots` flags, request-response bridge via `submit_page_data` app-only tool - Client: offscreen rendering (hidden canvas, no visual interference), text from cache or on-demand extraction, screenshots scaled to 768px max dimension, results submitted back to server - Max 20 pages per request, 60s timeout Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fold get_pages into the interact tool to minimize tools requiring approval. Now accessed via `interact(action: "get_pages", ...)`. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add concrete per-type schema docs with field names in tool description - Add JSON example showing add_annotations with highlight + stamp - Replace opaque z.record(z.string(), z.unknown()) with typed union of all annotation schemas (full + partial forms) so the model sees exact field names and types - Remove redundant manual safeParse since Zod inputSchema validates Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- display_pdf result text now explicitly lists annotation capabilities (highlights, stamps, notes, etc.) instead of vague "navigate, search, zoom, etc." - Restructured interact tool description: annotations promoted to top, with clear type reference, JSON example, and bold section headers - Added pdf-annotations.spec.ts with 6 E2E tests covering: - Result text mentions annotation capabilities - interact tool available in dropdown - add_annotations renders highlight - Multiple annotation types render (highlight, note, stamp, freetext, rectangle) - remove_annotations removes from DOM - highlight_text finds and highlights text Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests that Claude can discover and use PDF annotation capabilities by calling the Anthropic Messages API with the tool schemas and simulated display_pdf result. Disabled by default — skipped unless ANTHROPIC_API_KEY is set: ANTHROPIC_API_KEY=sk-... npx playwright test tests/e2e/pdf-annotations-api.spec.ts 3 scenarios tested: - Model uses highlight_text when asked to highlight the title - Model discovers annotation capabilities when asked "can you annotate?" - Model uses interact (add_annotations or get_pages) when asked to add notes Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…e to README - Example prompts for annotations, navigation, page extraction, stamps, forms - Documents how to run E2E tests and API prompt discovery tests - Updated tools table to include interact tool - Updated key patterns table with annotations, command queue, file download - Added pdf-lib to dependencies list Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The typed Zod union (14 anyOf variants: 7 full + 7 partial annotation types) produced a 5,817-char JSON schema for the annotations field alone. This bloated the interact tool schema to 7,802 chars, which may cause the model to struggle with or skip the tool. Replace with z.record(z.string(), z.any()) — annotation types are already fully documented in the tool description. Schema drops to 2,239 chars (71% reduction), annotations field to 254 chars (96% reduction). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The display_pdf result text now lists every action by name (navigate, search, find, search_navigate, zoom, add_annotations, update_annotations, remove_annotations, highlight_text, fill_form, get_pages) so the model knows exactly what commands are available without needing to inspect the interact tool schema. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The model was passing "pdf-viewer" instead of the actual UUID, causing get_pages to timeout (commands queued under wrong key, client never picks them up). - Add activeViewUUIDs set tracking UUIDs issued by display_pdf - Validate viewUUID at the top of interact handler with clear error - Add "IMPORTANT: viewUUID must be the exact UUID returned by display_pdf" to the interact tool description Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ochafik
changed the title
- Raise annotation-layer z-index above text-layer so note annotations receive hover/click events (was z-index: 1, now 3; text-layer is 2) - Replace memo emoji (data-icon attr) with CSS mask SVG document icon that respects currentColor for consistent cross-platform rendering
Right-side panel (250px) shows all annotations grouped by page with expand/collapse cards. Clicking a card navigates to the page and pulses the annotation; clicking a note icon on the PDF highlights its card in the panel. Panel auto-shows on first annotation, remembers user toggle preference via localStorage, and shows a badge count when collapsed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Server now holds poll_pdf_commands requests open (up to 30s) until commands arrive, waking waiters via enqueueCommand. Client loops sequentially instead of using setInterval, with 2s backoff on errors. Reduces idle RPC traffic from ~3 calls/sec to ~2 calls/min. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Render PDF.js AnnotationLayer with renderForms:true so form fields appear as interactive HTML inputs. Build a fieldName→annotationID map via getFieldObjects() to bridge fill_form (which uses field names) with annotationStorage (which uses annotation IDs). Sync user input back to formFieldValues for persistence and PDF download. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…lling When enabled and the client supports elicitation, extracts form fields from the PDF via pdf-lib and prompts the user to fill them before the viewer loads. Elicited values are returned in content/structuredContent and enqueued as a fill_form command for the viewer. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…d extraction Use pdfjs-dist's getDocument() + getFieldObjects() instead of pdf-lib's PDFDocument.load() + getForm().getFields() in extractFormSchema(). This removes the pdf-lib import from the server bundle (pdf-lib is still used client-side for PDF modification in downloadAnnotatedPdf). Uses the legacy pdfjs-dist build to avoid DOMMatrix dependency in Node.js. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Set --scale-factor/--total-scale-factor CSS variables on the form layer so AnnotationLayer font-size rules resolve correctly instead of falling back to browser defaults. Also update live DOM elements directly in fill_form handler so values appear immediately without waiting for a full re-render. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
After rendering the annotation layer, shrink select[size] font to fit within the PDF rect height. The default AnnotationLayer CSS uses a fixed 9px * scale-factor which overflows when many options share a small rect. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Show a delete button on each annotation card that appears on hover. Clicking it removes the annotation from the PDF and persists the change. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Skip keyboard navigation shortcuts (space, arrows, +/-) when any input, textarea, or select element is focused, not just the search/page inputs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Form field values now appear in the sidebar under a "Form Fields" group, with trash icons to clear individual values. The badge count and auto-show logic include form fields. The panel open/close now calls requestFitToContent with width to avoid overflow in inline layout mode. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Explicitly state that Y=0 is the bottom edge and Y=792 is the top for US Letter, with concrete guidance on typical values for top/bottom placement to prevent models from using top-down screen coordinates. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extract form field names during display_pdf and cache per viewer UUID. fill_form now soft-fails on unknown field names (applies valid ones, reports skipped ones). Both display_pdf and fill_form results include the list of valid field names so the model can self-correct. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
In inline mode, replace the 250px side panel with a slim bottom strip that shows one annotation/form-field at a time with prev/next navigation. The side panel is still used in fullscreen mode. - Strip shows item swatch, label, preview + counter (N of M · Page P) - Click item to navigate to its page; delete single or clear all - requestFitToContent includes strip height in size calculation - fieldNameToPage map built during init for form field page context - Display mode toggle switches between strip and panel automatically
…ases Add explicit guidance on when to use the tool: filling out forms (tax forms, applications), annotating PDFs, and interactive review. This helps models route user requests like 'help me fill out this form' to the display_pdf tool.
Fixes CodeQL js/xss + js/client-side-unvalidated-url-redirection on line 1994 (renderImageAnnotation) and the canvas-paint fallback. The server resolves imageUrl to imageData before enqueueing, so def.imageUrl only survives to the client if the server-side fetch fails. safeImageSrc whitelists https/http/data/blob and returns undefined for javascript: et al.
get_pages round-trip (interact → enqueue → poll → render → submit):
- GET_PAGES_TIMEOUT_MS 60s → 45s. The MCP SDK also defaults to 60s, so
if the round-trip got close, the client timed out first and our error
went nowhere. Server rejects first now.
- waitForPageData wires extra.signal. If the client cancels interact
(timeout/user abort), we clean up the pendingPageRequests entry
immediately instead of leaking until our own timeout. Also simplified
to a single settle callback instead of {resolve, reject, timer}.
- handleGetPages .catch() → submit empty pages. Was fire-and-forget
with no error path; any rejection before the inner try/catch meant
no submit_page_data → server waited the full timeout.
fill_form response: dropped the full field listing that was appended
on every call. display_pdf already returns it; echoing it back after
every successful fill is noise. Only list valid field names when the
model got one wrong.
The previous attempt returned the raw input string after a scheme check, which CodeQL's taint tracker doesn't recognise as a sanitiser (it sees tainted in → tainted out). Now parse with the URL constructor and return the .href property — this is a canonical sanitiser pattern CodeQL models as a barrier for js/xss and unvalidated-url-redirection. Also folded the data: URI branch into safeImageSrc for a single call site responsible for producing the <img src> string.
# Conflicts: # examples/pdf-server/src/mcp-app.css # examples/pdf-server/src/mcp-app.ts
Dir roots whose basename is 'uploads' (e.g. Claude Desktop's attachment drop folder) are now treated as read-only unless --writeable-uploads-root is passed. This prevents the save button from appearing for attached PDFs that the client doesn't expect to be overwritten. Also gates the _debug diagnostic block on --debug and adds save/restore of writeFlags.allowUploadsRoot in test beforeEach/afterEach.
…t for read-only cmds - Wire --debug CLI flag through to createServer; _debug diagnostic block in display_pdf _meta is now only emitted when --debug is set. - Viewer: showDebugBubble renders _debug payload as a fixed overlay. - Fix interact get_pages timeout: await handleGetPages instead of fire-and-forget, so submit_page_data doesn't queue behind the next 30s long-poll on serialized host connections. - Skip persistAnnotations for read-only commands (get_pages, file_changed).
After a successful save_pdf, update pdfBaselineAnnotations and pdfBaselineFormValues to reflect what was written to disk. Without this, removing all annotations after a save produced an empty diff (compared to the stale pre-save baseline), incorrectly disabling the save button even though the file on disk still had annotations.
…-index Clicking an annotation on the canvas now re-renders the panel to expand the correct page section and scrolls the card into view. Previously only the selection class toggled without updating the accordion. Raise tooltip z-index to 100 within the annotation layer and promote the annotation layer above the form layer on note hover (:has selector) so tooltips aren't clipped by the form overlay.
…otations Adds window.__pdfDebug() that dumps annotationMap, baseline, layer children, and localStorage diff to diagnose ghost annotations (visible on canvas but not in panel, not selectable). Also logs when importPdfjsAnnotation returns null for non-widget annotations, and surfaces any thrown errors during baseline import instead of silently swallowing them.
…f internals
__pdfDebug() now also dumps PDF.js's annotationStorage contents
(editor stamps live there, invisible to our tracking) and all
localStorage keys matching pdf-annot pattern.
After running __pdfDebug(), internals are exposed as
window.__pdf.{pdfDocument, annotationMap, annotationLayerEl, formLayerEl}
for interactive console poking.
…ons panel - display_pdf result text no longer leaks viewUUID when the interact tool is not registered (model has no use for it). - Annotations panel default width: 250px -> 500px (max-width:50vw still caps on narrow viewports).
PdfCommand was defined independently in server.ts and mcp-app.ts with no compiler check to keep them in sync. Extract to src/commands.ts as the single source of truth; both sides import the type. Uses the TS interfaces from pdf-annotations.ts (not Zod-inferred types) since those already describe the post-resolveImageAnnotation wire shape (imageData present, x/y/width/height non-optional). Also deletes ~150 lines of dead Zod schemas (PdfAnnotationDef and the 10 variant schemas that built it) -- they were never used as runtime validators, only for z.infer<typeof> type inference. Input remains z.record(z.any()) for model-API forgiveness.
- findTextRectsFromCache: return [] instead of a hardcoded placeholder rect when text-layer DOM isn't rendered. The placeholder was persisted as real coordinates; caller already guards on empty. - onteardown: bump loadGeneration to stop the preloader (reuses the reload-abort check) and clear searchDebounceTimer. - startPreloading: re-check loadGeneration inside the pause-wait loop so teardown mid-pause exits instead of spinning. - app.sendMessage calls: wrap in .catch(log.error) so host-unsupported doesn't throw an unhandled rejection. - Remove dead ternary (Highlight:Highlight), stale TODO and commented downloadBtn line.
…lags - Example prompts referenced get_pages with getText/getScreenshots params; actual model-facing actions are get_text (intervals) and get_screenshot (single page). Fixed 3 prompts. - Document --debug, --enable-interact, --writeable-uploads-root flags. - Add image annotation type to the Annotation Types table. - Note that interact is disabled in HTTP mode unless --enable-interact. - Add Deployment section (stdio vs HTTP single-instance vs stateless).
New describe("interact tool") block with 7 tests covering:
- enqueue -> poll_pdf_commands roundtrip
- missing-arg error paths for navigate, fill_form, add_annotations
- command queue isolation across distinct viewUUIDs
- fill_form passthrough when viewFieldNames not registered
- (skip) unknown-UUID poll (LONG_POLL_TIMEOUT_MS not exported, can't
bypass the 30s wait without changing server.ts)
Surprises found:
- interact never validates viewUUID exists; enqueues to any string
- batch-mode early-exits on first error, silently dropping later commands
…l.ts Split ~930 lines from mcp-app.ts into two new modules: - src/viewer-state.ts (50 lines): shared Maps/Sets/Arrays + types (annotationMap, formFieldValues, selectedAnnotationIds, fieldNameTo*, undoStack/redoStack, TrackedAnnotation, EditEntry). Only containers with stable bindings — both modules mutate contents, never reassign. - src/annotation-panel.ts (1084 lines): floating panel positioning, accordion rendering, annotation/form-field cards, reset/clear-all. Coupling to mcp-app flows through PanelDeps injected at init — scalars like currentPage/pdfDocument reach the panel via a state() getter so the ~100 use sites in mcp-app stay unwrapped. mcp-app.ts: 5666 → 4721 lines. panelState.open replaces annotationPanelOpen; panelState.openAccordionSection is written by selectAnnotation. syncSidebarSelection moved to the panel module.
Two new tests: - User-added annotation modification: captured in diff.added (id not in baseline, so latest content persists correctly). - Baseline annotation in-place modification: KNOWN LIMITATION. Same-id edit produces empty diff (id-set based), so the change vanishes on reload. Viewer's addAnnotation() works around via remove+add, but updateAnnotation() (interact tool's update_annotations) does not.
Pre-commit hook runs build:all which regenerates all example screenshots. These 24 PNGs have no semantic change; reverting to keep the PR diff focused on pdf-server.
…y file read)
resolveImageAnnotation() did fs.readFile on whatever imageUrl the model
sent, bypassing the allowedLocalFiles/allowedLocalDirs machinery that
protects display_pdf and save_pdf. Model could request
{imageUrl:"/Users/x/.ssh/id_rsa"}, server would base64 the bytes into
the add_annotations command, iframe stores it, get_screenshot reads it
back.
Also: fetch() branch accepted plain http:// (SSRF to local network).
Fix: call validateUrl(imageUrl) before touching the filesystem or
network. Throws on rejection; caller converts to {isError:true} so the
model sees a clear error instead of silent skip.
3 tests: reject local path outside roots, reject http://, accept path
under allowed dir (with real readFile + imageData population).
…ht-only coord shift
Three bugs from PR review, all in viewer state/coord handling:
1. Deleted baseline annotations zombied back on reload (mcp-app.ts:2512).
restoreAnnotations' apply loop was add-only. loadBaselineAnnotations
runs between the two restore calls and re-seeds annotationMap with
every baseline id, including the ones in diff.removed. The zombie
survives, and the next persistAnnotations sees it in currentIds ->
computeDiff produces removed=[] -> deletion permanently lost.
Fix: delete diff.removed ids after the merge loop.
2. Clear All didn't strip form values from the saved PDF (mcp-app.ts:2756).
clearAllItems() empties formFieldValues, but buildAnnotatedPdfBytes
gates on formFields.size > 0 and only writes entries present in the
map. Empty map -> zero setText/uncheck calls -> pdf-lib keeps the
original /V values. Fix: at getAnnotatedPdfBytes time, inject an
explicit clearing sentinel ("" or false by baseline type) for every
baseline field absent from formFieldValues.
3. update_annotations shifted rect/circle/image when patching height
without y (mcp-app.ts:4161). The old code spread existing.def
(internal coords, y = bottom edge) with update (model coords, y =
top-left), then key-filtered back to only update keys. But internal
y = pageHeight - modelY - height; patching height with top fixed
requires rewriting internal y, which the key-filter discarded.
Fix: convert existing to model coords, merge in model space, convert
back, pass the full merged def. Also uses the target page height
when update.page differs (fixes the low-severity page-move bug too).
…annotations sort Two lower-severity findings from review: Leak (server.ts:265): pruneStaleQueues iterated commandQueues to clean up viewFieldNames/viewFieldInfo/viewFileWatches, but display_pdf populates those maps without creating a commandQueues entry (only enqueueCommand does), and dequeueCommands deletes the entry on every poll. Net: the sweep found nothing; aux maps leaked every display_pdf. viewFileWatches entries hold an fs.StatWatcher -> slow FD exhaustion on HTTP --enable-interact. Fix: new viewLastActivity heartbeat map, touched at display_pdf/enqueue/dequeue, swept on TTL. Dead second loop (entry.commands.length===0 was unreachable) removed. Line-annotation sort (annotation-panel.ts:400): getAnnotationY checked 'y' in def and 'rects' in def, but LineAnnotation has only x1/y1/x2/y2. Fell through to return 0 -> all line annotations sorted to panel bottom. Added y1 branch returning max(y1, y2).
…nteract The model was calling display_pdf again to add annotations instead of using interact with the existing viewUUID. Root cause: the description's first two sentences invited exactly that -- 'Use this to display, annotate, edit, and fill form fields' 'when the user wants to ... annotate, edit, sign, stamp' -- then the CRITICAL warning arrived three paragraphs later, after the model had already pattern-matched 'user wants to sign' -> display_pdf. Three changes: - First sentence is now narrow: 'Open a PDF. Call this ONCE per PDF.' - Follow-up-actions-go-through-interact is sentence 2, not paragraph 3. - Result text is directive at the decision point: tells the model to call interact with the viewUUID, warns against re-calling display_pdf.
…ompts Both viewers coexist; nothing is discarded. The actual failure mode is the model now holds a new viewUUID and addresses the wrong viewer -- interact calls land on the fresh empty one, not the one with the user's existing annotations. Rewording to say that.
antonpk1
approved these changes
Mar 19, 2026
Contributor
Author
|
/update-snapshots |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds full annotation, interaction, and page extraction capabilities to the PDF server:
navigate,search,find,search_navigate,zoomadd_annotations(7 types: highlight, underline, strikethrough, note, rectangle, freetext, stamp),update_annotations,remove_annotationshighlight_text— auto-find and highlight text by queryget_pages— batch text and/or screenshot extraction from page ranges without visual navigation (offscreen rendering)fill_form— fill PDF form fieldspdf-lib(client-side) +app.downloadFile()SDK supporttoolInfo.idNew dependency
pdf-lib(^1.17.1) — client-side PDF modification for annotated downloadFiles changed
examples/pdf-server/server.tsexamples/pdf-server/src/mcp-app.tsexamples/pdf-server/mcp-app.htmlexamples/pdf-server/src/mcp-app.cssexamples/pdf-server/README.mdtests/e2e/pdf-annotations.spec.tstests/e2e/pdf-annotations-api.spec.tsANTHROPIC_API_KEY)Test plan
npx playwright test tests/e2e/pdf-annotations.spec.ts— 6 tests pass (annotation CRUD, highlight_text)npx playwright test -g "PDF Server"— existing screenshot tests passANTHROPIC_API_KEY=... npx playwright test tests/e2e/pdf-annotations-api.spec.ts— 3/3 pass (model discovers annotations)npm run --workspace examples/pdf-server build— compiles cleanly🤖 Generated with Claude Code