[WIP] feat: implement OAuth client secret rotation

.gitignore

@@ -115,3 +115,6 @@ Thumbs.db
 
 # Claude Code
 .claude/
+
+# Superpowers documentation (development artifacts)
+docs/superpowers/

deploy/cloudrun/README.md

@@ -13,6 +13,7 @@ Deploy the Red Hat Lightspeed Agent for Google Cloud to Google Cloud Run for pro
   - [3. Set Up Cloud SQL Database](#3-set-up-cloud-sql-database)
   - [4. Redis Setup for Rate Limiting](#4-redis-setup-for-rate-limiting)
   - [5. Configure Secrets](#5-configure-secrets)
+  - [5a. Configure Secret Rotation Schedule](#5a-configure-secret-rotation-schedule)
   - [6. Copy MCP Image to GCR](#6-copy-mcp-image-to-gcr)
   - [7. Deploy](#7-deploy)
 - [Service Configuration](#service-configuration)
@@ -439,6 +440,108 @@ echo -n "postgresql+asyncpg://sessions:$SESSION_DB_PASSWORD@/agent_sessions?host
 # The CA certificate is stored separately (see Redis Setup step 3).
 ```
 
+### 5a. Configure Secret Rotation Schedule
+
+After secrets are populated, bootstrap the rotation schedule metadata and trigger plumbing:
+
+```bash
+./deploy/cloudrun/setup-secret-rotation.sh
+```
+
+This script configures:
+
+- **Secret Manager rotation metadata** (`next_rotation_time` + `rotation_period`) for:
+  - `redhat-sso-client-secret`
+  - `gma-client-secret`
+- **Secret Manager event notifications** by attaching a Pub/Sub topic to those secrets
+- **Pub/Sub subscription** to receive `SECRET_ROTATE` events
+
+> Note: this does not rotate secret values by itself. It configures schedule + `SECRET_ROTATE` notifications so a rotator worker can handle updates.
+
+**Configure Pub/Sub Push Endpoint (after deployment):**
+
+After deploying the marketplace handler to Cloud Run, configure the subscription to push events to the `/rotation` endpoint:
+
+```bash
+# Get the marketplace handler URL
+MARKETPLACE_HANDLER_URL=$(gcloud run services describe marketplace-handler \
+  --region=us-central1 \
+  --format='value(status.url)')
+
+# Get the Cloud Run service account (used for OIDC authentication)
+SERVICE_ACCOUNT=$(gcloud run services describe marketplace-handler \
+  --region=us-central1 \
+  --format='value(spec.template.spec.serviceAccountName)')
+
+# Configure push endpoint with OIDC authentication
+gcloud pubsub subscriptions modify secret-rotation-trigger-sub \
+  --push-endpoint="${MARKETPLACE_HANDLER_URL}/rotation" \
+  --push-auth-service-account="${SERVICE_ACCOUNT}" \
+  --project="${GOOGLE_CLOUD_PROJECT}"
+```
+
+This configures:
+- **Push endpoint**: `https://<marketplace-handler-url>/rotation`
+- **OIDC authentication**: Pub/Sub signs requests with service account JWT
+- **Audience**: Marketplace handler validates JWT audience matches service URL
+
+**Testing the rotation workflow:**
+
+⚠️ **Prerequisites:** The rotation endpoint is deployed and functional, but secret retrieval requires implementing API-based providers in `src/lightspeed_agent/rotation/providers.py`:
+
+1. **Implement `RedHatSSOSecretProvider._fetch_secret_from_api()`**
+   - Authenticate with Red Hat Identity Management API
+   - Request new OAuth client secret for SSO
+   - Return the generated secret as a string
+
+2. **Implement `GMASecretProvider._fetch_secret_from_api()`**
+   - Authenticate with Google Marketplace Admin API  
+   - Request client secret regeneration
+   - Return the generated secret as a string
+
+3. **Register providers in `create_default_registry()`**
+   ```python
+   # Uncomment these lines in providers.py:
+   registry.register("redhat-sso-client-secret", RedHatSSOSecretProvider())
+   registry.register("gma-client-secret", GMASecretProvider())
+   ```
+
+**Current state:** Rotation endpoint will return 500 error with "No provider registered for secret" until providers are implemented.
+
+Once providers are implemented and registered:
+
+```bash
+# Trigger immediate rotation for testing
+gcloud secrets update redhat-sso-client-secret \
+  --next-rotation-time="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
+  --project="${GOOGLE_CLOUD_PROJECT}"
+
+# Monitor rotation logs
+gcloud logging read "resource.type=cloud_run_revision \
+  AND resource.labels.service_name=marketplace-handler \
+  AND jsonPayload.message=~'event_type=secret_rotation_completed'" \
+  --limit=10 \
+  --format=json \
+  --project="${GOOGLE_CLOUD_PROJECT}"
+```
+
+**What happens during rotation:**
+
+1. **Secret Manager triggers event** when `next_rotation_time` is reached
+2. **Pub/Sub pushes notification** to `/rotation` endpoint with OIDC JWT token
+3. **Rotation endpoint validates** the Pub/Sub OIDC token (transport security)
+4. **Registry routes to provider** based on secret name in the event
+5. **Provider fetches new secret** from upstream API (Red Hat Identity or GMA)
+6. **Provider validates secret** (32+ bytes, 10+ unique characters)
+7. **Secret Manager stores new version** via `add_secret_version` API
+8. **Endpoint logs completion** with `event_type=secret_rotation_completed`
+
+**Error handling:**
+- Invalid OIDC token → 401 response (not retried)
+- Missing provider → 500 response (Pub/Sub retries)
+- API failure → 500 response (Pub/Sub retries)
+- Secret Manager failure → 500 response (Pub/Sub retries)
+
 ### 6. Copy MCP Image to GCR
 
 Cloud Run doesn't support Quay.io directly. Copy the MCP server image to GCR.

deploy/cloudrun/cleanup.sh

@@ -52,6 +52,8 @@ PUBSUB_INVOKER_SA="${PUBSUB_INVOKER_NAME}@${PROJECT_ID}.iam.gserviceaccount.com"
 # Pub/Sub configuration
 PUBSUB_TOPIC="${PUBSUB_TOPIC:-marketplace-entitlements}"
 PUBSUB_SUBSCRIPTION="${PUBSUB_SUBSCRIPTION:-${PUBSUB_TOPIC}-sub}"
+ROTATION_TOPIC="${ROTATION_TOPIC:-secret-rotation-trigger}"
+ROTATION_SUBSCRIPTION="${ROTATION_SUBSCRIPTION:-secret-rotation-trigger-sub}"
 
 # Parse arguments
 FORCE=false
@@ -82,9 +84,11 @@ echo ""
 echo "  - Cloud Run services: $SERVICE_NAME, $HANDLER_SERVICE_NAME"
 echo "  - Pub/Sub topic: $PUBSUB_TOPIC"
 echo "  - Pub/Sub subscription: $PUBSUB_SUBSCRIPTION"
+echo "  - Rotation Pub/Sub topic: $ROTATION_TOPIC"
+echo "  - Rotation Pub/Sub subscription: $ROTATION_SUBSCRIPTION"
 echo "  - Secrets: redhat-sso-client-id, redhat-sso-client-secret, database-url,"
 echo "             session-database-url, gma-client-id, gma-client-secret, dcr-encryption-key,"
-echo "             rate-limit-redis-url"
+echo "             rate-limit-redis-url, redis-ca-cert"
 echo "  - Service accounts: $SERVICE_ACCOUNT"
 echo "                      $PUBSUB_INVOKER_SA"
 echo ""
@@ -170,6 +174,7 @@ secrets=(
     "gma-client-secret"
     "dcr-encryption-key"
     "rate-limit-redis-url"
+    "redis-ca-cert"
 )
 
 for secret in "${secrets[@]}"; do
@@ -183,6 +188,29 @@ for secret in "${secrets[@]}"; do
     fi
 done
 
+# =============================================================================
+# Step 3b: Delete Secret Rotation Topic and Subscription
+# =============================================================================
+log_info "Deleting secret rotation Pub/Sub resources..."
+
+if gcloud pubsub subscriptions describe "$ROTATION_SUBSCRIPTION" --project="$PROJECT_ID" &>/dev/null; then
+    gcloud pubsub subscriptions delete "$ROTATION_SUBSCRIPTION" \
+        --project="$PROJECT_ID" \
+        --quiet
+    log_info "Rotation Pub/Sub subscription '$ROTATION_SUBSCRIPTION' deleted"
+else
+    log_info "Rotation Pub/Sub subscription '$ROTATION_SUBSCRIPTION' does not exist, skipping"
+fi
+
+if gcloud pubsub topics describe "$ROTATION_TOPIC" --project="$PROJECT_ID" &>/dev/null; then
+    gcloud pubsub topics delete "$ROTATION_TOPIC" \
+        --project="$PROJECT_ID" \
+        --quiet
+    log_info "Rotation Pub/Sub topic '$ROTATION_TOPIC' deleted"
+else
+    log_info "Rotation Pub/Sub topic '$ROTATION_TOPIC' does not exist, skipping"
+fi
+
 # =============================================================================
 # Step 4: Remove IAM Bindings and Delete Service Account
 # =============================================================================
@@ -266,6 +294,7 @@ echo ""
 echo "The following resources have been removed:"
 echo "  - Cloud Run services ($SERVICE_NAME, $HANDLER_SERVICE_NAME)"
 echo "  - Pub/Sub topic and subscription"
+echo "  - Secret rotation Pub/Sub subscription and trigger topic"
 echo "  - Secret Manager secrets"
 echo "  - Service accounts (runtime + Pub/Sub invoker) and IAM bindings"
 echo ""

deploy/cloudrun/setup-secret-rotation.sh

@@ -0,0 +1,126 @@
+#!/bin/bash
+# =============================================================================
+# Secret Rotation Bootstrap for Cloud Run Deployment
+# =============================================================================
+#
+# Configures Secret Manager-native rotation plumbing for production:
+# 1) Secret Manager rotation metadata (rotation period + next rotation time)
+# 2) Secret Manager event notifications to Pub/Sub topics on each secret
+# 3) Pub/Sub subscription to receive SECRET_ROTATE events for future rotator workflows
+#
+# This script does NOT rotate secret values by itself.
+# It only creates schedules and notification plumbing.
+#
+# Usage:
+#   ./deploy/cloudrun/setup-secret-rotation.sh
+#
+# Prerequisite:
+#   Run ./deploy/cloudrun/setup.sh first.
+#
+# Optional environment variables:
+#   GOOGLE_CLOUD_PROJECT            Required. GCP project id.
+#   GOOGLE_CLOUD_LOCATION           Optional. Scheduler region (default: us-central1).
+#   ROTATION_TOPIC                  Optional. Pub/Sub topic (default: secret-rotation-trigger).
+#   ROTATION_SUBSCRIPTION           Optional. Pub/Sub subscription name
+#                                   (default: secret-rotation-trigger-sub).
+#   ROTATION_NEXT_TIME              Optional. RFC3339 UTC timestamp for next rotation metadata.
+#
+# =============================================================================
+
+set -euo pipefail
+
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+RED='\033[0;31m'
+NC='\033[0m'
+
+log_info() { echo -e "${GREEN}[INFO]${NC} $1"; }
+log_warn() { echo -e "${YELLOW}[WARN]${NC} $1"; }
+log_error() { echo -e "${RED}[ERROR]${NC} $1"; }
+
+PROJECT_ID="${GOOGLE_CLOUD_PROJECT:-}"
+REGION="${GOOGLE_CLOUD_LOCATION:-us-central1}"
+ROTATION_TOPIC="${ROTATION_TOPIC:-secret-rotation-trigger}"
+ROTATION_SUBSCRIPTION="${ROTATION_SUBSCRIPTION:-secret-rotation-trigger-sub}"
+# Use first day of next month at midnight UTC by default.
+ROTATION_NEXT_TIME="${ROTATION_NEXT_TIME:-$(date -u -d "$(date -u +%Y-%m-01) +1 month" +%Y-%m-01T00:00:00Z)}"
+FULL_TOPIC_NAME="projects/${PROJECT_ID}/topics/${ROTATION_TOPIC}"
+
+if [[ -z "$PROJECT_ID" ]]; then
+    log_error "GOOGLE_CLOUD_PROJECT environment variable is required"
+    echo "  export GOOGLE_CLOUD_PROJECT=your-project-id"
+    exit 1
+fi
+
+# Secret rotation definitions:
+# secret_name|rotation_period_seconds
+# NOTE: currently set to 3600s (1 hour) for testing.
+ROTATION_DEFINITIONS=(
+    "redhat-sso-client-secret|3600"
+    "gma-client-secret|3600"
+)
+
+log_info "Configuring secret rotation bootstrap for project: $PROJECT_ID"
+log_info "Pub/Sub region (for subscription): $REGION"
+log_info "Rotation event topic: $FULL_TOPIC_NAME"
+log_info "Rotation event subscription: $ROTATION_SUBSCRIPTION"
+log_info "Initial next rotation timestamp: $ROTATION_NEXT_TIME"
+
+if ! gcloud pubsub topics describe "$ROTATION_TOPIC" --project="$PROJECT_ID" &>/dev/null; then
+    log_info "Creating Pub/Sub topic: $ROTATION_TOPIC"
+    gcloud pubsub topics create "$ROTATION_TOPIC" --project="$PROJECT_ID"
+else
+    log_info "Pub/Sub topic already exists: $ROTATION_TOPIC"
+fi
+
+# Create Secret Manager service identity (publisher principal for notifications).
+log_info "Ensuring Secret Manager service identity exists..."
+gcloud beta services identity create \
+  --service="secretmanager.googleapis.com" \
+  --project="$PROJECT_ID" --quiet >/dev/null 2>&1 || true
+
+project_number=$(gcloud projects describe "$PROJECT_ID" --format='value(projectNumber)')
+sm_service_account="service-${project_number}@gcp-sa-secretmanager.iam.gserviceaccount.com"
+
+log_info "Granting Pub/Sub publisher role to Secret Manager service account..."
+gcloud pubsub topics add-iam-policy-binding "$ROTATION_TOPIC" \
+  --member="serviceAccount:${sm_service_account}" \
+  --role="roles/pubsub.publisher" \
+  --project="$PROJECT_ID" --quiet >/dev/null
+
+if ! gcloud pubsub subscriptions describe "$ROTATION_SUBSCRIPTION" --project="$PROJECT_ID" &>/dev/null; then
+    log_info "Creating Pub/Sub subscription: $ROTATION_SUBSCRIPTION"
+    gcloud pubsub subscriptions create "$ROTATION_SUBSCRIPTION" \
+      --topic="$ROTATION_TOPIC" \
+      --project="$PROJECT_ID" --quiet
+else
+    log_info "Pub/Sub subscription already exists: $ROTATION_SUBSCRIPTION"
+fi
+
+for definition in "${ROTATION_DEFINITIONS[@]}"; do
+    IFS='|' read -r secret_name rotation_period <<< "$definition"
+
+    if ! gcloud secrets describe "$secret_name" --project="$PROJECT_ID" &>/dev/null; then
+        log_warn "Secret not found, skipping: $secret_name"
+        continue
+    fi
+
+    log_info "Configuring rotation + topic notification for $secret_name"
+    gcloud secrets update "$secret_name" \
+      --project="$PROJECT_ID" \
+      --add-topics="$FULL_TOPIC_NAME" \
+      --next-rotation-time="$ROTATION_NEXT_TIME" \
+      --rotation-period="${rotation_period}s" \
+      --quiet
+done
+
+echo ""
+log_info "Secret rotation bootstrap complete."
+echo ""
+echo "What is now configured:"
+echo "  - Secret Manager rotation metadata on 2 secrets"
+echo "  - Secret event notifications (including SECRET_ROTATE) to topic: $FULL_TOPIC_NAME"
+echo "  - Pub/Sub subscription to receive rotation events: $ROTATION_SUBSCRIPTION"
+echo ""
+echo "Next step:"
+echo "  - Deploy a rotator worker subscribed to '$ROTATION_SUBSCRIPTION' and handle eventType=SECRET_ROTATE."

deploy/cloudrun/setup.sh

@@ -358,14 +358,17 @@ echo "   echo -n 'rediss://REDIS_IP:6378/0' | gcloud secrets versions add rate-l
 echo "   # Download and store the Redis server CA certificate for TLS verification:"
 echo "   gcloud redis instances describe lightspeed-redis --region=\$REGION --project=$PROJECT_ID --format='value(serverCaCerts[0].cert)' | gcloud secrets versions add redis-ca-cert --data-file=- --project=$PROJECT_ID"
 echo ""
-echo "3. Copy the MCP server image to GCR (Cloud Run doesn't support Quay.io):"
+echo "3. Configure secret rotation schedules and Secret Manager Pub/Sub notifications:"
+echo "   ./deploy/cloudrun/setup-secret-rotation.sh"
+echo ""
+echo "4. Copy the MCP server image to GCR (Cloud Run doesn't support Quay.io):"
 echo "   docker pull quay.io/redhat-services-prod/insights-management-tenant/insights-mcp/red-hat-lightspeed-mcp:latest"
 echo "   docker tag quay.io/redhat-services-prod/insights-management-tenant/insights-mcp/red-hat-lightspeed-mcp:latest gcr.io/$PROJECT_ID/red-hat-lightspeed-mcp:latest"
 echo "   docker push gcr.io/$PROJECT_ID/red-hat-lightspeed-mcp:latest"
 echo ""
-echo "4. Build and deploy the agent (includes MCP sidecar):"
+echo "5. Build and deploy the agent (includes MCP sidecar):"
 echo "   ./deploy/cloudrun/deploy.sh --build --service all --allow-unauthenticated"
 echo ""
-echo "5. Get the service URL:"
+echo "6. Get the service URL:"
 echo "   gcloud run services describe $SERVICE_NAME --region=$REGION --project=$PROJECT_ID --format='value(status.url)'"
 echo ""

src/lightspeed_agent/marketplace/app.py

@@ -18,6 +18,7 @@
 
 from lightspeed_agent.config import get_settings
 from lightspeed_agent.marketplace.router import router as handler_router
+from lightspeed_agent.rotation import router as rotation_router
 from lightspeed_agent.probes import start_probe_server, stop_probe_server
 from lightspeed_agent.ratelimit import RateLimitMiddleware, get_redis_rate_limiter
 from lightspeed_agent.security import RequestBodyLimitMiddleware, SecurityHeadersMiddleware
@@ -123,6 +124,10 @@ def create_app() -> FastAPI:
     # This provides the /dcr endpoint that handles both Pub/Sub and DCR
     app.include_router(handler_router)
 
+    # Include the rotation router
+    # This provides the /rotation endpoint for Secret Manager Pub/Sub events
+    app.include_router(rotation_router)
+
     # Add rate limiting middleware for /dcr endpoint (IP-based, no auth on this service)
     app.add_middleware(RateLimitMiddleware, rate_limited_paths={"/dcr"})