feat: send cookies with every request during a clone#58
Open
didimitrie wants to merge 1 commit into
Open
Conversation
Add a --cookie flag to `kage clone` that attaches cookies to all four
request paths a clone makes, so a site behind a login or a region/cookie
wall can be mirrored:
- Chrome page navigations (seeded into the browser before first load)
- asset downloads
- the robots.txt fetch
- the sitemap.xml fetch
The flag takes a single name=value pair, repeats for several cookies, or
accepts a whole Cookie header in one value ("a=1; b=2") pasted straight
from devtools. Browser cookies are scoped to the seed host and its
subdomains; the HTTP fetches send the same header. Off-domain assets are
left on the live web by default, so the cookies are not leaked to
third-party hosts.
Tests cover the header serialisation, the downloader/sitemap sending the
cookie, the Chrome cookie-param building, and the CLI flag parsing
(including malformed input). README, CLI reference, and CHANGELOG updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a
--cookieflag tokage cloneso a site behind a login or a region/cookie wall can be mirrored. The cookies ride on every request a clone makes, not just the page render:Cookieheader on the HTTP fetchrobots.txtfetchsitemap.xmlfetchThis keeps the authenticated session consistent across the whole run rather than only the rendered pages.
Usage
Notes
--cookieaccepts a singlename=value, repeats, or a full header in one value; malformed input is a clear usage error.Tests
Added coverage for:
Config.CookieHeaderserialisation (incl. empty-name skipping)CookieheaderfetchSitemapsending the cookieparseCookies, including malformed-input errors (theclipackage had no tests before)go build ./...,go vet ./...,gofmt, and the fullgo test ./...all pass.README, CLI reference (
docs/content/reference/cli.md), and CHANGELOG (Unreleased) updated.🤖 Generated with Claude Code
Closes #9