feat(mcp): download log content for self-hosted jobs in jobs_logs

[cchristous]

Summary

The jobs_logs MCP tool previously had split behavior:

• Hosted jobs: returned actual log content (up to 200 lines) with cursor-based pagination
• Self-hosted jobs: returned only a JWT token and URL, requiring the caller to manually curl the URL

This forced MCP clients (like Claude Code) to fall back to Bash curl to view self-hosted logs, defeating the purpose of having an MCP tool.

This PR makes fetchSelfHostedLogs download the log content via HTTP after generating the token, parse the JSON events response, and return a paginated preview — the same experience as hosted jobs. If the download fails for any reason (network error, non-200 status, JSON parse error), it falls back gracefully to the existing token/URL-only behavior with a logged warning.

Changes

mcp_server/pkg/tools/jobs/logs.go

• Added computePaginationWindow helper that computes slice indices, display metadata, and next cursor — used by both fetchHostedLogs and paginateSelfHostedLines to eliminate duplicated windowing logic
• computePaginationWindow only reports Truncated = true when the window genuinely doesn't cover all available lines (start > 0 || end < totalLines), rather than unconditionally when a cursor is provided
• Added logDownloader function type and threaded it through logsHandler → fetchSelfHostedLogs as a parameter, replacing the previous downloadLogsFn package-level variable (enables clean test injection without global mutation)
• Added downloadSelfHostedLogs function that GETs the logs URL, parses {"events": [{"output": "..."}]} JSON, and returns lines
• Added io.LimitReader with 10 MiB cap to prevent OOM from unexpectedly large responses — reads maxLogDownloadBytes+1 so that responses of exactly 10 MiB are accepted while anything larger is rejected with a descriptive errLogResponseTooLarge error; uses explicit int64 cast on both sides of the size comparison; the error message is derived from maxLogDownloadBytes to stay in sync if the constant changes
• downloadSelfHostedLogs accepts context.Context and uses http.NewRequestWithContext so the HTTP download respects parent cancellation/deadlines
• Uses a shared package-level selfHostedHTTPClient for connection pooling instead of creating a new http.Client per call; on non-200 responses the body is drained before returning the error so the TCP connection can be reused
• Added in-memory cache (logCache) keyed by job ID with 60-second TTL — avoids re-downloading the full log response on every pagination call, since self-hosted logs are fetched via a single HTTP GET with no server-side windowing; only finished jobs are cached — running jobs re-download on each call so new output is picked up immediately
• Cache stores token metadata (token, tokenType, tokenTtlSeconds, logsURL) so cached responses include the same fields as fresh responses; note that tokenTtlSeconds reflects the original TTL at issuance, not the remaining lifetime (the 60s cache TTL is well within the 300s token TTL so the token is always still valid)
• Both getCachedLogLines and cacheLogLines perform an expired-entry sweep on each call for consistent cleanup
• Cache is bounded to 10 entries (~100 MiB worst case) with LRU-by-expiry eviction to prevent unbounded memory growth
• Cache check is performed before gRPC GenerateToken call, so paginated requests that hit the cache skip the gRPC round-trip entirely (the token from the initial cache-miss call has a 300s TTL, well exceeding the 60s cache TTL)
• Empty download results (jobs still starting up) are intentionally not cached so they are re-checked on subsequent calls
• Added DownloadEmpty field to logsResult — when the log download succeeds but returns no events, formatSelfHostedLogsMarkdown now shows a distinct "no output yet, retry shortly" message instead of falling back to the token/curl instructions
• Updated fetchSelfHostedLogs to accept startingLine for pagination and jobFinished to control caching behavior
• Updated formatSelfHostedLogsMarkdown to show log preview when available, empty-download message when download succeeded but returned no lines, and token/URL fallback otherwise. When at cursor=0 the truncation message now says "This is the beginning of the log" instead of the misleading "No further logs are available"
• Removed dead if start <= 0 { start = 0 } guards from both formatHostedLogsMarkdown and formatSelfHostedLogsMarkdown — StartLine is always >= 0
• Updated logsFullDescription to reflect unified behavior
• Updated success tracking to mark success when preview lines OR token are present
• Stored tokenTypeToString result in a local variable to avoid redundant call

mcp_server/pkg/tools/jobs/register.go

• Updated Register to pass downloadSelfHostedLogs as the logDownloader argument to logsHandler

mcp_server/pkg/tools/jobs/jobs_test.go

• Extracted newSelfHostedTestEnv helper that creates the full test environment (httptest server, stubs, provider, handler, cache reset) and callLogsHandler helper for making requests — reduces ~40-60 lines of boilerplate per test
• newSelfHostedTestEnv uses t.Cleanup for automatic teardown and passes a test downloader directly to logsHandler — no global state mutation, safe for parallel use
• newSelfHostedTestEnv defaults job state to FINISHED; accepts selfHostedTestOpt to override (e.g. STARTED for running-job tests)
• Test downloader now asserts that the URL it receives matches the expected logs URL, verifying the correct URL is passed through the call chain
• Added shared constants (selfHostedTestJobID, selfHostedTestOrgID, selfHostedTestUser)
• Updated TestFetchSelfHostedLogs to use helpers and verify preview lines are returned
• Added TestSelfHostedLogsDownloadFailureFallback — HTTP 500 gracefully falls back to token/URL
• Added TestSelfHostedLogsEmptyDownloadFallback — empty events response shows distinct "no output yet" message and sets DownloadEmpty=true
• Added TestSelfHostedLogsPagination — 5 sub-tests mirroring hosted pagination test cases (including out-of-range cursor)
• Added TestDownloadSelfHostedLogs — 5 sub-tests for the download function (valid, empty, error, bad JSON, response too large)
• Added TestSelfHostedLogsCacheAvoidsDuplicateDownload — uses atomic.Int32 for the download counter to avoid a data race between the httptest handler goroutine and the test goroutine; verifies the HTTP server is only hit once across two handler calls for the same job, and asserts cached response includes Token/LogsURL metadata
• Added TestSelfHostedRunningJobSkipsCache — verifies that a STARTED job downloads logs on every call (no caching), confirming running jobs always get fresh output
• Added cursorBeyondEndReturnsEmptyPreview test case to both TestFetchHostedLogsPagination and TestSelfHostedLogsPagination — verifies a cursor past the end returns an empty preview with the correct backward cursor

Security considerations

• The logs URL is constructed from trusted internal sources (server BaseURL config, gRPC org service, UUID-validated jobID, gRPC-generated token) — not user-controllable
• HTTP response body is capped at 10 MiB via io.LimitReader to prevent OOM, with a clear error when the cap is hit
• HTTP download uses http.NewRequestWithContext to respect parent context cancellation
• Shared selfHostedHTTPClient has a 30-second timeout as a safety net; response bodies are drained on non-200 status to allow connection reuse
• JWT token (300s TTL) travels over HTTPS; only shown in markdown when log download fails (fallback path). When preview succeeds, the token is intentionally omitted from the markdown but remains in StructuredContent for programmatic MCP client access
• In-memory cache entries expire after 60 seconds; expired entries are evicted opportunistically on both read and write; cache is bounded to 10 entries with LRU-by-expiry eviction; only finished jobs are cached
• Token security reminder preserved in the fallback display path

Test plan

• All 25 tests pass (go test ./pkg/tools/jobs/... -v -count=1)
• go vet ./pkg/tools/jobs/... clean
• go build ./... clean
• Manual verification with a real self-hosted job via Claude Code MCP integration