fix(seo): remove blanket public/robots.txt shadowing the scoped app/robots.ts (unblocks Deck discoverability)#657
Open
Victor "David" Medina (Victor-David-Medina) wants to merge 1 commit into
Conversation
…obots.ts public/robots.txt served 'User-agent: * / Disallow: /' which, as a static file, overrides the Next.js app/robots.ts route in the App Router. That route is already the intended, carefully-scoped policy (allow /, /demo, /pricing, /for/, /scan, /benchmark, /templates; disallow /dashboard, /api, /auth, /council, /settings, ...). The stale blanket file self-throttled the public Deck marketing pages to zero crawl reach (measured Discoverability 50-52 in the 2026-07-01 design scorecard). Deleting the shadow lets the scoped app/robots.ts serve, unblocking /scan (the AEO front door) and the public marketing pages while keeping the authenticated app + api disallowed. Takes effect on the next Railway redeploy (verify /robots.txt at the edge). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
🛡️ Cascade Quality Score: 100/100
Threshold: 85/100 | Result: PASS ✅ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Deletes
public/robots.txt(a blanketUser-agent: * / Disallow: /) that was shadowing the intended, scopedapp/robots.tsroute.Why (root cause)
In the Next.js App Router, a static
public/robots.txtfile takes precedence over theapp/robots.tsroute. So the live edge (deck.relaylaunch.com/robots.txt) served a blanketDisallow: /— blocking all crawlers from the public Deck marketing pages — even thoughapp/robots.tsalready defines the correct policy:/,/demo,/pricing,/for/,/scan,/benchmark,/templates/dashboard,/api,/auth,/council,/settings,/rooms,/operations,/onboarding,/credits,/team,/memory,/monitoring,/pulse-opsThe 2026-07-01 design scorecard measured console Discoverability at 50-52 (Developing) because of this self-throttle. Deleting the shadow lets the scoped policy serve — unblocking
/scan(the AEO front-door surface) and the public marketing pages, while keeping the authenticated app + API disallowed.Not a strategy flip
This does not newly expose the app —
app/robots.tsalready disallows every authenticated/API path. It restores the policy the code already intended. (If a full pre-launch block were ever desired, that belongs inapp/robots.ts, not a stale static override.)Verify
curl https://deck.relaylaunch.com/robots.txtshould return the scopedapp/robots.tsoutput (allow list + disallow list + sitemap), notDisallow: /.