Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions k8s/APPLY-CHECKLIST.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,57 @@ of silently regressing.

---

## MinIO retirement (2026-05-20)

The self-hosted MinIO Deployment in `instant-data` was retired in
`chore/retire-self-hosted-minio-2026-05-20` (supersedes the stale PR #4
from 2026-05-11). DO Spaces (`nyc3.digitaloceanspaces.com`, bucket
`instant-shared`) is the canonical production object-store backend, selected
by `OBJECT_STORE_BACKEND=do-spaces` in `instant-secrets` (and mirrored to
`instant-infra-secrets` for worker + provisioner storage_bytes scanners).

After this PR merges, run the following on the prod cluster (and on any
local Rancher Desktop clusters that still have the legacy MinIO workload
deployed):

```bash
# 1. Confirm context (do NOT run against the wrong cluster)
kubectl config current-context

# 2. Inventory what's actually there before deleting anything
kubectl get deploy,pvc,svc,job,secret -n instant-data -l app=minio
kubectl get secret -n instant-data minio-secrets 2>/dev/null || echo "no minio-secrets"

# 3. Remove the MinIO workload + storage + services + bootstrap Job + Secret
kubectl delete -n instant-data deploy/minio --ignore-not-found
kubectl delete -n instant-data pvc/minio-data --ignore-not-found
kubectl delete -n instant-data svc/minio svc/minio-external --ignore-not-found
kubectl delete -n instant-data job/minio-bucket-init --ignore-not-found
kubectl delete -n instant-data secret/minio-secrets --ignore-not-found

# 4. Verify nothing left
kubectl get pods -n instant-data | grep -i minio # should print nothing

# 5. (Optional) If the legacy `s3.instanode.dev` Ingress still points at
# the minio Service, delete or repoint it to DO Spaces. The Ingress
# object was untracked in this repo — confirm what's live:
kubectl get ingress -A | grep -i minio

# 6. Sanity-check the storage hot path is unaffected
curl -sS https://api.instanode.dev/healthz | jq .
# /storage/new responses should reference *.digitaloceanspaces.com,
# not minio.instant-data.svc.cluster.local.
```

Rollback: revert the merge commit on master, re-apply the deleted
manifests from history (`git show <revert-sha>~1 -- k8s/data/minio*.yaml |
kubectl apply -f -`), and flip `OBJECT_STORE_BACKEND` back to `minio` in
`instant-secrets`. Storage data isn't lost in either direction — the PVC
was on local-path in Rancher Desktop only; DO Spaces holds the real
production object bytes.

---

## Related files

- `README.md` — secrets clobber warning (the same class of bug, but for
Expand Down
21 changes: 7 additions & 14 deletions k8s/app.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -194,18 +194,9 @@ spec:
name: instant-secrets
key: R2_BUCKET_NAME
optional: true
- name: MINIO_ROOT_USER
valueFrom:
secretKeyRef:
name: instant-secrets
key: MINIO_ROOT_USER
optional: true
- name: MINIO_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: instant-secrets
key: MINIO_ROOT_PASSWORD
optional: true
# MINIO_ROOT_USER/MINIO_ROOT_PASSWORD env refs retired 2026-05-20
# (chore/retire-self-hosted-minio-2026-05-20). The api reads
# OBJECT_STORE_* below for the canonical DO Spaces backend.
# ── Razorpay annual plan IDs (codified 2026-05-20) ────────────────
# Live since the annual-pricing toggle shipped. Required for any
# *_yearly plan checkout to succeed. NOT optional in prod, but
Expand Down Expand Up @@ -262,8 +253,10 @@ spec:
optional: true
# ── Object-store backend (storage abstraction, 2026-05-20) ────────
# OBJECT_STORE_BACKEND selects the impl: do-spaces (prod today),
# r2, s3, or minio. The rest are backend-specific. See
# common/storageprovider/factory.go.
# r2, or s3. The rest are backend-specific. See
# common/storageprovider/factory.go. The self-hosted MinIO backend
# was retired 2026-05-20 — DO Spaces (`nyc3.digitaloceanspaces.com`,
# bucket `instant-shared`) is the canonical production backend.
- name: OBJECT_STORE_BACKEND
valueFrom:
secretKeyRef:
Expand Down
5 changes: 3 additions & 2 deletions k8s/configmap.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,9 @@ data:
COMPUTE_PROVIDER: "k8s"
STACK_EXPOSE_VIA: "nodeport"
KUBE_NAMESPACE_APPS: "instant-apps"
MINIO_ENDPOINT: "minio.instant-data.svc.cluster.local:9000"
MINIO_BUCKET_NAME: "instant-shared"
# MINIO_ENDPOINT / MINIO_BUCKET_NAME retired 2026-05-20 (self-hosted MinIO
# retirement). Object storage now lives behind OBJECT_STORE_* in
# instant-secrets (DO Spaces, nyc3, bucket instant-shared).
# Deploy / stack ingress wiring. DEPLOY_DOMAIN is the wildcard zone the
# api builds per-deploy hostnames under (<id>.deployment.instanode.dev);
# without it the code falls back to the hardcoded default "instant.dev".
Expand Down
46 changes: 0 additions & 46 deletions k8s/data/minio-bucket-init.yaml

This file was deleted.

16 changes: 0 additions & 16 deletions k8s/data/minio-secret.yaml

This file was deleted.

124 changes: 0 additions & 124 deletions k8s/data/minio.yaml

This file was deleted.

56 changes: 45 additions & 11 deletions k8s/provisioner/deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -201,21 +201,55 @@ spec:
value: "1"
- name: K8S_MONGO_STORAGE_GI
value: "1"
# ── MinIO storage bytes queries ───────────────────────────────────
- name: MINIO_ENDPOINT
value: "minio.instant-data.svc.cluster.local:9000"
- name: MINIO_BUCKET_NAME
value: "instant-shared"
- name: MINIO_ROOT_USER
# ── Object-storage bytes queries ──────────────────────────────────
# Migrated 2026-05-20 from MINIO_* to OBJECT_STORE_* as part of
# chore/retire-self-hosted-minio-2026-05-20 (supersedes PR #4).
# Self-hosted MinIO retired; DO Spaces is the canonical backend.
# Also removes a previously dangling reference to a `minio-secrets`
# Secret in the instant-infra namespace that was never defined in
# the tree (the in-repo Secret lived in instant-data).
- name: OBJECT_STORE_BACKEND
valueFrom:
secretKeyRef:
name: minio-secrets
key: MINIO_ROOT_USER
- name: MINIO_ROOT_PASSWORD
name: instant-infra-secrets
key: OBJECT_STORE_BACKEND
optional: true
- name: OBJECT_STORE_ENDPOINT
valueFrom:
secretKeyRef:
name: instant-infra-secrets
key: OBJECT_STORE_ENDPOINT
optional: true
- name: OBJECT_STORE_REGION
valueFrom:
secretKeyRef:
name: instant-infra-secrets
key: OBJECT_STORE_REGION
optional: true
- name: OBJECT_STORE_ACCESS_KEY
valueFrom:
secretKeyRef:
name: instant-infra-secrets
key: OBJECT_STORE_ACCESS_KEY
optional: true
- name: OBJECT_STORE_SECRET_KEY
valueFrom:
secretKeyRef:
name: instant-infra-secrets
key: OBJECT_STORE_SECRET_KEY
optional: true
- name: OBJECT_STORE_BUCKET
valueFrom:
secretKeyRef:
name: minio-secrets
key: MINIO_ROOT_PASSWORD
name: instant-infra-secrets
key: OBJECT_STORE_BUCKET
optional: true
- name: OBJECT_STORE_SECURE
valueFrom:
secretKeyRef:
name: instant-infra-secrets
key: OBJECT_STORE_SECURE
optional: true
# ── OTLP headers (NR license key) — CODIFIED INTENT ───────────────
# The live deployment has this as an inline `value:` literal (NR
# license key embedded). Codified intent is to pull from
Expand Down
17 changes: 12 additions & 5 deletions k8s/secrets.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -51,11 +51,18 @@ stringData:
R2_API_TOKEN: "CHANGE_ME"
R2_ENDPOINT: "CHANGE_ME"
R2_BUCKET_NAME: "CHANGE_ME"
# MinIO credentials (same values as in infra/k8s/data/minio-secret.yaml).
# Used only by the legacy self-hosted MinIO Deployment, which was retired
# 2026-05-20 in favour of DigitalOcean Spaces. Operators MUST replace.
MINIO_ROOT_USER: "CHANGE_ME"
MINIO_ROOT_PASSWORD: "CHANGE_ME"
# Object-storage backend (canonical 2026-05-20 onward — self-hosted MinIO retired).
# OBJECT_STORE_BACKEND selects the impl: do-spaces (prod), r2, or s3.
# The remaining keys are backend-specific credentials. See
# common/storageprovider/factory.go.
OBJECT_STORE_BACKEND: "do-spaces"
OBJECT_STORE_ENDPOINT: "CHANGE_ME"
OBJECT_STORE_PUBLIC_URL: "CHANGE_ME"
OBJECT_STORE_REGION: "CHANGE_ME"
OBJECT_STORE_ACCESS_KEY: "CHANGE_ME"
OBJECT_STORE_SECRET_KEY: "CHANGE_ME"
OBJECT_STORE_BUCKET: "instant-shared"
OBJECT_STORE_SECURE: "true"
# Comma-separated list of admin user emails (case-insensitive). Empty
# / unset → admin endpoints reject every caller (closed-by-default).
# Read by api/internal/middleware/admin.go on every request, no app
Expand Down
Loading
Loading