From cf16b4f2a545fdd7fdeefe5ee8ab1afe603de659 Mon Sep 17 00:00:00 2001 From: Manas Srivastava Date: Wed, 20 May 2026 23:00:43 +0530 Subject: [PATCH] =?UTF-8?q?chore(infra):=20retire=20self-hosted=20MinIO=20?= =?UTF-8?q?=E2=80=94=20DO=20Spaces=20is=20canonical=20(supersedes=20#4)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit DO Spaces (nyc3.digitaloceanspaces.com, bucket instant-shared) has been the active object-store backend in production since 2026-05-11. The self-hosted MinIO Deployment in instant-data is no longer in the request path for /storage/new — verified via live cluster: OBJECT_STORE_BACKEND=shared-key (alias → do-spaces) OBJECT_STORE_ENDPOINT=nyc3.digitaloceanspaces.com OBJECT_STORE_BUCKET=instant-shared This commit retires the local MinIO manifests and replaces every MINIO_* env injection on production k8s deployments with OBJECT_STORE_* sourced from instant-secrets / instant-infra-secrets. Supersedes PR #4 (2026-05-11), which had drifted from current master after the broad manifest reconciliation work landed. Manifests deleted: - k8s/data/minio.yaml (Deployment + PVC + ClusterIP + NodePort) - k8s/data/minio-bucket-init.yaml (one-shot Job creating instant-shared) - k8s/data/minio-secret.yaml (local-dev MinIO root creds) Env-var injection migrated MINIO_* → OBJECT_STORE_* in: - k8s/app.yaml (instant-api) — removed MINIO_ROOT_USER/PASSWORD optional refs; OBJECT_STORE_* keys already wired via prior storage-abstraction work. - k8s/worker/deployment.yaml (instant-worker) — storage_bytes scanner now reads OBJECT_STORE_* from instant-infra-secrets. - k8s/provisioner/deployment.yaml (instant-provisioner) — storage_bytes scanner block migrated. Also removes the dangling reference to a 'minio-secrets' Secret in the instant-infra namespace that was never defined in-tree (the in-repo Secret lived in instant-data). - k8s/configmap.yaml — removed MINIO_ENDPOINT / MINIO_BUCKET_NAME. - k8s/secrets.yaml (template) — removed MINIO_ROOT_USER/PASSWORD template keys; added OBJECT_STORE_* template keys for DO Spaces. Docs: - k8s/APPLY-CHECKLIST.md — new 'MinIO retirement' section with the post-merge operator cleanup commands (kubectl delete deploy/minio pvc/minio-data svc/{minio,minio-external} job/minio-bucket-init secret/minio-secrets in instant-data). Per CLAUDE.md rule 15 (infra has no auto-apply), the operator runs this manually after merge. Verification: - kubectl apply --dry-run=server -f k8s/{app,configmap,secrets, provisioner/deployment,worker/deployment}.yaml — all clean (no schema errors; pre-existing E2E_TEST_TOKEN hides-previous warning is unrelated to this change). - Live confirmation prod is on DO Spaces: instant-secrets has OBJECT_STORE_BACKEND=shared-key (alias to do-spaces), OBJECT_STORE_ENDPOINT=nyc3.digitaloceanspaces.com. Coverage block: Symptom: Self-hosted MinIO still deployed in instant-data while prod traffic is on DO Spaces Enumeration: rg -n -i 'minio' k8s/ --type yaml Sites found: 8 (3 manifest files + 5 env-injection blocks across app/configmap/worker/provisioner/secrets) Sites touched: 8 (3 deleted + 5 migrated to OBJECT_STORE_*) Coverage test: kubectl apply --dry-run=server clean across all mutated manifests (no resolveError on missing minio- secrets, no env-var redefinition warnings beyond the pre-existing E2E_TEST_TOKEN one) Live verified: Operator runs APPLY-CHECKLIST.md 'MinIO retirement' section commands post-merge; prod request path verified pre-PR via OBJECT_STORE_BACKEND env on live instant-secrets (DO Spaces, nyc3, instant-shared). Co-Authored-By: Claude Opus 4.7 (1M context) --- k8s/APPLY-CHECKLIST.md | 51 +++++++++++++ k8s/app.yaml | 21 ++---- k8s/configmap.yaml | 5 +- k8s/data/minio-bucket-init.yaml | 46 ------------ k8s/data/minio-secret.yaml | 16 ----- k8s/data/minio.yaml | 124 -------------------------------- k8s/provisioner/deployment.yaml | 56 ++++++++++++--- k8s/secrets.yaml | 17 +++-- k8s/worker/deployment.yaml | 60 ++++++++++++---- 9 files changed, 163 insertions(+), 233 deletions(-) delete mode 100644 k8s/data/minio-bucket-init.yaml delete mode 100644 k8s/data/minio-secret.yaml delete mode 100644 k8s/data/minio.yaml diff --git a/k8s/APPLY-CHECKLIST.md b/k8s/APPLY-CHECKLIST.md index 0b8c6b6..e7f3796 100644 --- a/k8s/APPLY-CHECKLIST.md +++ b/k8s/APPLY-CHECKLIST.md @@ -170,6 +170,57 @@ of silently regressing. --- +## MinIO retirement (2026-05-20) + +The self-hosted MinIO Deployment in `instant-data` was retired in +`chore/retire-self-hosted-minio-2026-05-20` (supersedes the stale PR #4 +from 2026-05-11). DO Spaces (`nyc3.digitaloceanspaces.com`, bucket +`instant-shared`) is the canonical production object-store backend, selected +by `OBJECT_STORE_BACKEND=do-spaces` in `instant-secrets` (and mirrored to +`instant-infra-secrets` for worker + provisioner storage_bytes scanners). + +After this PR merges, run the following on the prod cluster (and on any +local Rancher Desktop clusters that still have the legacy MinIO workload +deployed): + +```bash +# 1. Confirm context (do NOT run against the wrong cluster) +kubectl config current-context + +# 2. Inventory what's actually there before deleting anything +kubectl get deploy,pvc,svc,job,secret -n instant-data -l app=minio +kubectl get secret -n instant-data minio-secrets 2>/dev/null || echo "no minio-secrets" + +# 3. Remove the MinIO workload + storage + services + bootstrap Job + Secret +kubectl delete -n instant-data deploy/minio --ignore-not-found +kubectl delete -n instant-data pvc/minio-data --ignore-not-found +kubectl delete -n instant-data svc/minio svc/minio-external --ignore-not-found +kubectl delete -n instant-data job/minio-bucket-init --ignore-not-found +kubectl delete -n instant-data secret/minio-secrets --ignore-not-found + +# 4. Verify nothing left +kubectl get pods -n instant-data | grep -i minio # should print nothing + +# 5. (Optional) If the legacy `s3.instanode.dev` Ingress still points at +# the minio Service, delete or repoint it to DO Spaces. The Ingress +# object was untracked in this repo — confirm what's live: +kubectl get ingress -A | grep -i minio + +# 6. Sanity-check the storage hot path is unaffected +curl -sS https://api.instanode.dev/healthz | jq . +# /storage/new responses should reference *.digitaloceanspaces.com, +# not minio.instant-data.svc.cluster.local. +``` + +Rollback: revert the merge commit on master, re-apply the deleted +manifests from history (`git show ~1 -- k8s/data/minio*.yaml | +kubectl apply -f -`), and flip `OBJECT_STORE_BACKEND` back to `minio` in +`instant-secrets`. Storage data isn't lost in either direction — the PVC +was on local-path in Rancher Desktop only; DO Spaces holds the real +production object bytes. + +--- + ## Related files - `README.md` — secrets clobber warning (the same class of bug, but for diff --git a/k8s/app.yaml b/k8s/app.yaml index 7591c54..fcc5c2d 100644 --- a/k8s/app.yaml +++ b/k8s/app.yaml @@ -194,18 +194,9 @@ spec: name: instant-secrets key: R2_BUCKET_NAME optional: true - - name: MINIO_ROOT_USER - valueFrom: - secretKeyRef: - name: instant-secrets - key: MINIO_ROOT_USER - optional: true - - name: MINIO_ROOT_PASSWORD - valueFrom: - secretKeyRef: - name: instant-secrets - key: MINIO_ROOT_PASSWORD - optional: true + # MINIO_ROOT_USER/MINIO_ROOT_PASSWORD env refs retired 2026-05-20 + # (chore/retire-self-hosted-minio-2026-05-20). The api reads + # OBJECT_STORE_* below for the canonical DO Spaces backend. # ── Razorpay annual plan IDs (codified 2026-05-20) ──────────────── # Live since the annual-pricing toggle shipped. Required for any # *_yearly plan checkout to succeed. NOT optional in prod, but @@ -262,8 +253,10 @@ spec: optional: true # ── Object-store backend (storage abstraction, 2026-05-20) ──────── # OBJECT_STORE_BACKEND selects the impl: do-spaces (prod today), - # r2, s3, or minio. The rest are backend-specific. See - # common/storageprovider/factory.go. + # r2, or s3. The rest are backend-specific. See + # common/storageprovider/factory.go. The self-hosted MinIO backend + # was retired 2026-05-20 — DO Spaces (`nyc3.digitaloceanspaces.com`, + # bucket `instant-shared`) is the canonical production backend. - name: OBJECT_STORE_BACKEND valueFrom: secretKeyRef: diff --git a/k8s/configmap.yaml b/k8s/configmap.yaml index 7264f14..07b08ce 100644 --- a/k8s/configmap.yaml +++ b/k8s/configmap.yaml @@ -36,8 +36,9 @@ data: COMPUTE_PROVIDER: "k8s" STACK_EXPOSE_VIA: "nodeport" KUBE_NAMESPACE_APPS: "instant-apps" - MINIO_ENDPOINT: "minio.instant-data.svc.cluster.local:9000" - MINIO_BUCKET_NAME: "instant-shared" + # MINIO_ENDPOINT / MINIO_BUCKET_NAME retired 2026-05-20 (self-hosted MinIO + # retirement). Object storage now lives behind OBJECT_STORE_* in + # instant-secrets (DO Spaces, nyc3, bucket instant-shared). # Deploy / stack ingress wiring. DEPLOY_DOMAIN is the wildcard zone the # api builds per-deploy hostnames under (.deployment.instanode.dev); # without it the code falls back to the hardcoded default "instant.dev". diff --git a/k8s/data/minio-bucket-init.yaml b/k8s/data/minio-bucket-init.yaml deleted file mode 100644 index 8231b47..0000000 --- a/k8s/data/minio-bucket-init.yaml +++ /dev/null @@ -1,46 +0,0 @@ -# One-shot Job: creates the instant-shared bucket in MinIO after it starts. -# Idempotent — mc mb --ignore-existing is safe to re-run. -# Apply after minio.yaml: kubectl apply -f data/minio-bucket-init.yaml -apiVersion: batch/v1 -kind: Job -metadata: - name: minio-bucket-init - namespace: instant-data -spec: - template: - spec: - restartPolicy: OnFailure - initContainers: - - name: wait-for-minio - image: busybox:1.36 - command: - - sh - - -c - - | - until wget -q -O /dev/null http://minio.instant-data.svc.cluster.local:9000/minio/health/ready; do - echo "Waiting for MinIO..."; sleep 3 - done - echo "MinIO ready." - containers: - - name: create-bucket - image: minio/mc:RELEASE.2024-10-02T08-27-28Z - command: - - sh - - -c - - | - mc alias set myminio \ - http://minio.instant-data.svc.cluster.local:9000 \ - "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" && \ - mc mb --ignore-existing myminio/instant-shared && \ - echo "Bucket instant-shared is ready." - env: - - name: MINIO_ROOT_USER - valueFrom: - secretKeyRef: - name: minio-secrets - key: MINIO_ROOT_USER - - name: MINIO_ROOT_PASSWORD - valueFrom: - secretKeyRef: - name: minio-secrets - key: MINIO_ROOT_PASSWORD diff --git a/k8s/data/minio-secret.yaml b/k8s/data/minio-secret.yaml deleted file mode 100644 index c62f605..0000000 --- a/k8s/data/minio-secret.yaml +++ /dev/null @@ -1,16 +0,0 @@ -# MinIO root credentials — legacy local-dev path (in-cluster MinIO retired -# 2026-05-20). Operators MUST replace placeholders before applying. Local-dev -# convenience values were here historically; OSS publication requires -# placeholders so a fresh clone cannot accidentally come up with the same -# credentials as every other clone. -# Apply: kubectl apply -f data/minio-secret.yaml -# The API pod reads these same values from instant-secrets (instant namespace). -apiVersion: v1 -kind: Secret -metadata: - name: minio-secrets - namespace: instant-data -type: Opaque -stringData: - MINIO_ROOT_USER: "CHANGE_ME" - MINIO_ROOT_PASSWORD: "CHANGE_ME" diff --git a/k8s/data/minio.yaml b/k8s/data/minio.yaml deleted file mode 100644 index e10bd54..0000000 --- a/k8s/data/minio.yaml +++ /dev/null @@ -1,124 +0,0 @@ -# MinIO S3-compatible object storage — instant-data namespace -# Local dev backend for POST /storage/new. -# Apply: kubectl apply -f data/minio-secret.yaml && kubectl apply -f data/minio.yaml ---- -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: minio-data - namespace: instant-data -spec: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 10Gi ---- -apiVersion: apps/v1 -kind: Deployment -metadata: - name: minio - namespace: instant-data - labels: - app: minio -spec: - replicas: 1 - selector: - matchLabels: - app: minio - template: - metadata: - labels: - app: minio - spec: - containers: - - name: minio - image: minio/minio:RELEASE.2024-10-13T13-34-11Z - args: - - server - - /data - - --console-address - - :9001 - env: - - name: MINIO_ROOT_USER - valueFrom: - secretKeyRef: - name: minio-secrets - key: MINIO_ROOT_USER - - name: MINIO_ROOT_PASSWORD - valueFrom: - secretKeyRef: - name: minio-secrets - key: MINIO_ROOT_PASSWORD - ports: - - name: api - containerPort: 9000 - - name: console - containerPort: 9001 - resources: - requests: - memory: "128Mi" - cpu: "100m" - limits: - memory: "512Mi" - cpu: "500m" - readinessProbe: - httpGet: - path: /minio/health/ready - port: 9000 - initialDelaySeconds: 10 - periodSeconds: 10 - livenessProbe: - httpGet: - path: /minio/health/live - port: 9000 - initialDelaySeconds: 30 - periodSeconds: 30 - volumeMounts: - - name: data - mountPath: /data - volumes: - - name: data - persistentVolumeClaim: - claimName: minio-data ---- -# Cluster-internal service (used by the API pod) -apiVersion: v1 -kind: Service -metadata: - name: minio - namespace: instant-data - labels: - app: minio -spec: - selector: - app: minio - ports: - - name: api - port: 9000 - targetPort: 9000 - - name: console - port: 9001 - targetPort: 9001 ---- -# NodePort for external access from test machine / developer browser -apiVersion: v1 -kind: Service -metadata: - name: minio-external - namespace: instant-data - labels: - app: minio -spec: - type: NodePort - selector: - app: minio - ports: - - name: api - port: 9000 - targetPort: 9000 - nodePort: 30900 - - name: console - port: 9001 - targetPort: 9001 - nodePort: 30901 diff --git a/k8s/provisioner/deployment.yaml b/k8s/provisioner/deployment.yaml index 0b39001..e94b4d9 100644 --- a/k8s/provisioner/deployment.yaml +++ b/k8s/provisioner/deployment.yaml @@ -201,21 +201,55 @@ spec: value: "1" - name: K8S_MONGO_STORAGE_GI value: "1" - # ── MinIO storage bytes queries ─────────────────────────────────── - - name: MINIO_ENDPOINT - value: "minio.instant-data.svc.cluster.local:9000" - - name: MINIO_BUCKET_NAME - value: "instant-shared" - - name: MINIO_ROOT_USER + # ── Object-storage bytes queries ────────────────────────────────── + # Migrated 2026-05-20 from MINIO_* to OBJECT_STORE_* as part of + # chore/retire-self-hosted-minio-2026-05-20 (supersedes PR #4). + # Self-hosted MinIO retired; DO Spaces is the canonical backend. + # Also removes a previously dangling reference to a `minio-secrets` + # Secret in the instant-infra namespace that was never defined in + # the tree (the in-repo Secret lived in instant-data). + - name: OBJECT_STORE_BACKEND valueFrom: secretKeyRef: - name: minio-secrets - key: MINIO_ROOT_USER - - name: MINIO_ROOT_PASSWORD + name: instant-infra-secrets + key: OBJECT_STORE_BACKEND + optional: true + - name: OBJECT_STORE_ENDPOINT + valueFrom: + secretKeyRef: + name: instant-infra-secrets + key: OBJECT_STORE_ENDPOINT + optional: true + - name: OBJECT_STORE_REGION + valueFrom: + secretKeyRef: + name: instant-infra-secrets + key: OBJECT_STORE_REGION + optional: true + - name: OBJECT_STORE_ACCESS_KEY + valueFrom: + secretKeyRef: + name: instant-infra-secrets + key: OBJECT_STORE_ACCESS_KEY + optional: true + - name: OBJECT_STORE_SECRET_KEY + valueFrom: + secretKeyRef: + name: instant-infra-secrets + key: OBJECT_STORE_SECRET_KEY + optional: true + - name: OBJECT_STORE_BUCKET valueFrom: secretKeyRef: - name: minio-secrets - key: MINIO_ROOT_PASSWORD + name: instant-infra-secrets + key: OBJECT_STORE_BUCKET + optional: true + - name: OBJECT_STORE_SECURE + valueFrom: + secretKeyRef: + name: instant-infra-secrets + key: OBJECT_STORE_SECURE + optional: true # ── OTLP headers (NR license key) — CODIFIED INTENT ─────────────── # The live deployment has this as an inline `value:` literal (NR # license key embedded). Codified intent is to pull from diff --git a/k8s/secrets.yaml b/k8s/secrets.yaml index 78bfb85..37f0849 100644 --- a/k8s/secrets.yaml +++ b/k8s/secrets.yaml @@ -51,11 +51,18 @@ stringData: R2_API_TOKEN: "CHANGE_ME" R2_ENDPOINT: "CHANGE_ME" R2_BUCKET_NAME: "CHANGE_ME" - # MinIO credentials (same values as in infra/k8s/data/minio-secret.yaml). - # Used only by the legacy self-hosted MinIO Deployment, which was retired - # 2026-05-20 in favour of DigitalOcean Spaces. Operators MUST replace. - MINIO_ROOT_USER: "CHANGE_ME" - MINIO_ROOT_PASSWORD: "CHANGE_ME" + # Object-storage backend (canonical 2026-05-20 onward — self-hosted MinIO retired). + # OBJECT_STORE_BACKEND selects the impl: do-spaces (prod), r2, or s3. + # The remaining keys are backend-specific credentials. See + # common/storageprovider/factory.go. + OBJECT_STORE_BACKEND: "do-spaces" + OBJECT_STORE_ENDPOINT: "CHANGE_ME" + OBJECT_STORE_PUBLIC_URL: "CHANGE_ME" + OBJECT_STORE_REGION: "CHANGE_ME" + OBJECT_STORE_ACCESS_KEY: "CHANGE_ME" + OBJECT_STORE_SECRET_KEY: "CHANGE_ME" + OBJECT_STORE_BUCKET: "instant-shared" + OBJECT_STORE_SECURE: "true" # Comma-separated list of admin user emails (case-insensitive). Empty # / unset → admin endpoints reject every caller (closed-by-default). # Read by api/internal/middleware/admin.go on every request, no app diff --git a/k8s/worker/deployment.yaml b/k8s/worker/deployment.yaml index ff8049a..22a187e 100644 --- a/k8s/worker/deployment.yaml +++ b/k8s/worker/deployment.yaml @@ -165,28 +165,58 @@ spec: value: production - name: KUBE_NAMESPACE_APPS value: "instant-apps" - # MinIO storage_bytes scanner — added 2026-05-11 in worker - # PR #4. The scanner walks every storage resource's bucket - # once per scan interval, sums object sizes, and updates - # resources.storage_bytes in the platform Postgres. When - # MINIO_ENDPOINT is empty (e.g., in CI), the scanner runs - # as a fail-open no-op and emits a single warn-log line - # per storage resource per interval. - - name: MINIO_ENDPOINT - value: "minio.instant-data.svc.cluster.local:9000" - - name: MINIO_BUCKET_NAME - value: "instant-shared" - - name: MINIO_ROOT_USER + # Object-storage storage_bytes scanner. The scanner walks every + # storage resource's bucket once per scan interval, sums object + # sizes, and updates resources.storage_bytes in the platform + # Postgres. Backend is selected by OBJECT_STORE_BACKEND (today: + # do-spaces against nyc3.digitaloceanspaces.com, bucket + # instant-shared). When OBJECT_STORE_ENDPOINT is empty (e.g., in + # CI), the scanner runs as a fail-open no-op and emits a single + # warn-log line per storage resource per interval. + # + # Migrated 2026-05-20 from MINIO_* to OBJECT_STORE_* as part of + # chore/retire-self-hosted-minio-2026-05-20 (supersedes PR #4). + - name: OBJECT_STORE_BACKEND + valueFrom: + secretKeyRef: + name: instant-infra-secrets + key: OBJECT_STORE_BACKEND + optional: true + - name: OBJECT_STORE_ENDPOINT + valueFrom: + secretKeyRef: + name: instant-infra-secrets + key: OBJECT_STORE_ENDPOINT + optional: true + - name: OBJECT_STORE_REGION valueFrom: secretKeyRef: name: instant-infra-secrets - key: MINIO_ROOT_USER + key: OBJECT_STORE_REGION optional: true - - name: MINIO_ROOT_PASSWORD + - name: OBJECT_STORE_ACCESS_KEY valueFrom: secretKeyRef: name: instant-infra-secrets - key: MINIO_ROOT_PASSWORD + key: OBJECT_STORE_ACCESS_KEY + optional: true + - name: OBJECT_STORE_SECRET_KEY + valueFrom: + secretKeyRef: + name: instant-infra-secrets + key: OBJECT_STORE_SECRET_KEY + optional: true + - name: OBJECT_STORE_BUCKET + valueFrom: + secretKeyRef: + name: instant-infra-secrets + key: OBJECT_STORE_BUCKET + optional: true + - name: OBJECT_STORE_SECURE + valueFrom: + secretKeyRef: + name: instant-infra-secrets + key: OBJECT_STORE_SECURE optional: true # Brevo sender identity for the raw-HTML send path (added # 2026-05-15). The dashboard-template path was inheriting a