Show WebTV description and related documents; fix scraper bugs#10
Closed
davidpomerenke wants to merge 2 commits into
Closed
Show WebTV description and related documents; fix scraper bugs#10davidpomerenke wants to merge 2 commits into
davidpomerenke wants to merge 2 commits into
Conversation
…ture Three defects in the WebTV asset-page scraper: 1. extractRelatedDocuments matched `<a href="...">` with the closing bracket directly after the href. Real markup carries `target="_blank"`, so the inner regex matched zero links on *every* asset — related_documents has been silently empty site-wide, including in the public JSON API. 2. extractTextContent strips all tags and collapses whitespace, fusing adjacent block elements: two <li> items rendered as "...Production Patterns SDG 9 and interlinkages...". Descriptions now go through extractRichText, which keeps block boundaries as newlines. Consumers render with `whitespace-pre-line`. 3. getVideoMetadata swallowed every failure into empty metadata, making a network error, a 404, and "this asset has no metadata" indistinguishable. Failures now reach Sentry. Assets with no metadata block at all (media stakeouts, b-roll) legitimately parse empty and carry no `field__label`, so that marker separates them from markup drift. Splits the parser out as the pure `parseVideoMetadata(html)` so it can be tested against real markup without a network round-trip. Also documents why the `/en/` in the fetch URL is load-bearing: WebTV localizes the very labels this parser keys on, and not every asset has a translated page. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both fields were already scraped and passed to the client, but only `summary` was ever rendered — the rest reached users only through the JSON API. Adds the long-form description (collapsed to three lines behind a Show more / Show less toggle, mirroring WebTV's own control) and the list of related sites and documents, which includes ODS document links. The toggle measures real overflow rather than guessing from string length, so short descriptions get no pointless control. Sections render only when non-empty, so assets with no metadata block show nothing at all. Headings are translated in all six locales, using the UN's own wording taken from WebTV's localized pages. The scraped content itself stays English (see getVideoMetadata), so translated headings wrap English text for now. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+64
to
+65
| const text = html | ||
| .replace(/<[^>]*>/g, "") |
| const withBreaks = html | ||
| .replace(/<br\s*\/?>/gi, "\n") | ||
| .replace(/<\/(?:p|li|h[1-6]|div|tr)>/gi, "\n"); | ||
| return decodeEntities(withBreaks.replace(/<[^>]*>/g, "")) |
Collaborator
Author
|
Closing: opened without an explicit request from the developer. The work is preserved on the local branch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Comparing our asset page against the WebTV original, most of the richer metadata was already being scraped into
VideoMetadataand passed to the client — but onlysummarywas rendered. Everything else reached users solely through the JSON API. While wiring up the display, three real defects surfaced.Bugs fixed
related_documentswas always empty, site-wide.extractRelatedDocumentsmatched<a href="...">with the closing bracket right after the href. Real markup is<a href="..." target="_blank">, so the inner regex matched zero links on every asset. This was invisible because it degraded to an empty array, which is indistinguishable from "this asset has no related documents". Verified against live markup: old regex → 0 links, new → 2.Descriptions lost all block structure.
extractTextContentstrips every tag and collapses whitespace, fusing adjacent<li>items — the HLPF description read...Production Patterns SDG 9 and interlinkages.... Harmless while the text only fed JSON-LD; unacceptable now that it's displayed. Descriptions now useextractRichText, which preserves block boundaries as newlines.Scrape failures were swallowed.
catch { return createEmptyMetadata(); }made a network error, a 404, and a genuinely bare asset all look identical — which is why bug #1 survived. Failures now report to Sentry. Assets with no metadata block (media stakeouts, b-roll) carry nofield__label, so that marker distinguishes them from markup drift.What's displayed
Description (collapsed to three lines behind Show more / Show less, mirroring WebTV's own control) and Related Sites and Documents, which surfaces ODS document links. The toggle measures real overflow rather than guessing from string length. Sections render only when non-empty.
Topical subjects, corporate name, and the category breadcrumb were deliberately left out of the UI for now — their extractors already work and keep flowing to the JSON API.
API-visible behaviour changes
related_documentsgoes from always[]to populated.descriptiongains newlines at block boundaries (type unchanged,string | null).Not in scope
getVideoMetadatastill fetches/en/, so translated headings wrap English content. This is deliberate and now documented in the code: WebTV localizes the very DOM labels the parser keys on (Summary→Résumé,Subject Topical→Sujets), and not every asset has a translated page —/ar/asset/k1h/k1hoqorknt404s. Pointing the fetch at the request locale would silently return empty metadata for every non-English visitor. Fixing it properly needs a per-locale label map plus an/en/fallback. Follow-up.Persisting this metadata (rather than scraping it per request behind a 1-hour cache) is also deferred — it's the prerequisite for ever searching or faceting by topic.
Testing
parseVideoMetadata(html)split out as a pure function and tested against trimmed real-markup fixtures. The related-documents test fails onmain. Full suite: 228 passing; typecheck and lint clean.Also driven end-to-end against a local dev server on three assets chosen to cover the branches:
docs.un.orghrefs; toggle flips clamp 3 → none,aria-expandedfalse → true./fr/— renders "Sites web et documents connexes" / "Afficher plus", no raw i18n key leaked.🤖 Generated with Claude Code