Eai 1320 update domain

[pwistbac]

Domain Update Playbook - Complete Action Summary

Entry Point: update-domain.yaml

Pre-Tasks

1. Set kubeconfig environment (optional, if KUBECONFIG provided)
2. Validate required variables (NEW_DOMAIN, CERT_OPTION)
3. Validate certificate paths (if CERT_OPTION=provide)

---

Phase 1: Validation (validate.yaml)

1. Display validation header
2. Validate domain format - Ensures NEW_DOMAIN is a valid DNS name
3. Check cluster connectivity - Runs kubectl cluster-info
4. Get current domain from cluster-domain ConfigMap
5. Set current domain fact
6. Check if domain is changing - Fails if NEW_DOMAIN == CURRENT_DOMAIN
7. Get cert-manager status from ConfigMap
8. Verify ArgoCD installation - Checks for cluster-forge Application
9. Verify OpenBao installation - Checks StatefulSet readiness
10. Check for in-progress ArgoCD syncs - Warns if apps are syncing
11. Prompt to continue if syncs are in progress

---

Phase 2: Certificates (certificates.yaml)

Display DNS Requirements

1. Get LoadBalancer IP from envoy-gateway
2. Display DNS configuration instructions
   - Shows required A records: *.{NEW_DOMAIN} and k8s.{NEW_DOMAIN}
   - Shows kaytoo command for silogen.ai domains

Confirm Domain Update

1. Prompt for confirmation (unless DRY_RUN)
2. Abort if declined

Handle Certificates (based on CERT_OPTION)

If CERT_OPTION = "generate":

1. Generate self-signed certificate with SANs:
   - CN: {NEW_DOMAIN}
   - SANs: k8s.{NEW_DOMAIN}, kc.{NEW_DOMAIN}, *.{NEW_DOMAIN}
2. Create/update cluster-tls secret in envoy-gateway-system

If CERT_OPTION = "provide":

1. Validate certificate and key files exist
2. Extract CN and SANs from certificate
3. Warn if NEW_DOMAIN not in SANs
4. Prompt to continue
5. Create/update cluster-tls secret from provided files

If CERT_OPTION = "cert-manager":

1. Check if Certificate resource exists
2. Update Certificate spec with new dnsNames
3. Delete existing secret to trigger re-issuance
4. Wait for Certificate to become ready

---

Phase 3: Core Configuration Update (update_config.yaml)

Display Update Summary

1. Shows current domain, new domain, certificate option, and mode (DRY_RUN or live)

Update ConfigMap

1. Patch cluster-domain ConfigMap in default namespace
   - Sets DOMAIN: {NEW_DOMAIN}

Update OpenBao Secret

1. Execute kubectl exec into openbao-0 pod
2. Run bao kv put to update secret/cluster-domain with NEW_DOMAIN
3. Warn if update fails (continues on failure)

Update cluster-values Repository (GitOps)

1. Get Gitea admin token from gitea-admin-token secret
2. Fetch current values.yaml from cluster-values repo via Gitea API (using kubectl exec into gitea pod)
3. Create temp file and write current values
4. Update global.domain using yq
5. Read updated content and encode to base64
6. Commit to Gitea via API with message "Update domain to {NEW_DOMAIN}"
7. Clean up temp file

Update Keycloak Clients (update_keycloak.yaml)

1. Get AIRM admin credentials from airm-realm-credentials secret
2. Create ConfigMap with bash script containing:
   - update_client() - updates redirect URIs
   - update_client_post_logout_uris() - updates post-logout redirect URIs
   - update_client_root_url() - updates rootUrl
   - update_client_admin_url() - updates adminUrl
3. Create Kubernetes Job using amdenterpriseai/cluster-tool image
4. Job executes script which:
   - Gets Keycloak token using client_credentials grant
   - Updates each client by looking up UUID by clientId
   - Updates: AIRM UI Client, ArgoCD, Gitea, Minio clients
5. Wait for Job completion (60s timeout)
6. Display Job logs
7. Fail if Job failed

Update AIRM Cluster Records (update_airm_clusters.yaml)

1. Get AIRM admin credentials from airm-realm-credentials secret
2. Create ConfigMap with bash script
3. Create Kubernetes Job using amdenterpriseai/cluster-tool image
4. Job executes script which:
   - Gets access token from Keycloak
   - Fetches all clusters from /api/clusters
   - For each cluster:
   • Gets current cluster data
   • Updates kubeApiUrl to https://k8s.{NEW_DOMAIN}
   • Updates workbenchBaseUrl to https://aiwbui.{NEW_DOMAIN}
   • Sends PUT request to /api/clusters/{cluster_id}
5. Wait for Job completion (60s timeout)
6. Display Job logs
7. Fail if Job failed

Trigger ArgoCD Sync

1. Get current timestamp
2. Annotate cluster-forge Application with:
   - argocd.argoproj.io/refresh: hard
   - bloom.domain.update/timestamp: {timestamp}
3. Triggers hard refresh - ArgoCD detects parameter change and syncs

---

Phase 4: Verification (verify.yaml)

If not DRY_RUN:

1. Wait 2 seconds for changes to propagate
2. Verify ConfigMap update - Checks cluster-domain.data.DOMAIN == NEW_DOMAIN
3. Check Gateway configuration - Verifies spec.listeners[0].hostname
4. Display ArgoCD sync status with monitoring instructions

Display Post-Update Instructions

1. Shows next steps:
   - Update DNS records
   - Wait for ArgoCD sync (5-10 minutes)
   - Update external references (kubeconfig, integrations)
   - Test application access at new domain
2. Shows monitoring commands

---

Summary Statistics

Resources Updated:

• 1 ConfigMap (cluster-domain)
• 1 OpenBao secret (secret/cluster-domain)
• 1 Git repository (cluster-values)
• 4+ Keycloak clients (AIRM Admin, AIRM UI, ArgoCD, Gitea, Minio)
• N AIRM cluster records (all clusters in database)
• 1 TLS secret (cluster-tls)
• 1 ArgoCD Application annotation (triggers sync)

Jobs Created:

• 1 Keycloak update Job (auto-deleted after 5 min)
• 1 AIRM cluster update Job (auto-deleted after 5 min)

External APIs Used:

• Gitea API (cluster-values repo updates)
• Keycloak Admin API (client configuration)
• AIRM API (cluster records)