diff --git a/README.md b/README.md index baaaee6..16d3fbf 100644 --- a/README.md +++ b/README.md @@ -155,32 +155,19 @@ vault write auth/kubernetes/role/external-secrets \ ### Verify Integration ```bash -# Create test secret vault kv put secret/test foo=bar +vault kv get secret/test # Check ESO synced it (ClusterSecretStore "vault" is pre-configured) kubectl get externalsecret -A -``` - -## Required Vault Secrets - -These secrets must exist in Vault before the corresponding apps can sync. - -### MinIO (`secret/minio`) - -```bash -# Generate 256-bit encryption key -ENCRYPTION_KEY=$(openssl rand -base64 32) -vault kv put secret/minio \ - access_key="minio-admin" \ - secret_key="$(openssl rand -hex 16)" \ - kms_secret_key="minio-encryption-key:${ENCRYPTION_KEY}" +# Clean up +vault kv delete secret/test ``` -**Important**: Back up `kms_secret_key` - losing it means losing access to encrypted data. +## Required Vault Secrets -### Other Secrets +These secrets must exist in Vault before the corresponding apps can sync. See [`docs/vault-secrets.md`](docs/vault-secrets.md) for ready-to-run provisioning commands. | Path | Keys | Used By | |------|------|---------| @@ -199,16 +186,6 @@ vault kv put secret/minio \ | `secret/docker-registry/ovh` | username, password | registry-secrets | | `secret/github-runner` | github_app_id, github_app_installation_id, github_app_private_key | arc-runners-public | -### Docker Registry (`secret/docker-registry/ovh`) - -```bash -vault kv put secret/docker-registry/ovh \ - username='' \ - password='' -``` - -To add or update a service password: `vault kv patch secret/postgresql -user-password=` - ## Platform Architecture (WIP) HDC splits workloads across namespaces by trust boundary and function: diff --git a/clusters/prod/apps/elasticsearch/Chart.yaml b/clusters/prod/apps/elasticsearch/Chart.yaml new file mode 100644 index 0000000..4a49905 --- /dev/null +++ b/clusters/prod/apps/elasticsearch/Chart.yaml @@ -0,0 +1,7 @@ +apiVersion: v2 +name: elasticsearch +version: 0.1.0 +dependencies: + - name: elasticsearch + version: "17.9.29" + repository: https://pilotdataplatform.github.io/helm-charts/ diff --git a/clusters/prod/apps/elasticsearch/application.yaml b/clusters/prod/apps/elasticsearch/application.yaml new file mode 100644 index 0000000..99bdffa --- /dev/null +++ b/clusters/prod/apps/elasticsearch/application.yaml @@ -0,0 +1,27 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: elasticsearch + namespace: argocd + annotations: + argocd.argoproj.io/sync-wave: "5" +spec: + project: default + source: + repoURL: https://github.com/PilotDataPlatform/pilot-hdc-platform-gitops.git + targetRevision: main + path: clusters/prod/apps/elasticsearch + helm: + valueFiles: + - ../../registry.yaml + - values.yaml + destination: + server: https://kubernetes.default.svc + namespace: utility + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true + - ServerSideApply=true diff --git a/clusters/prod/apps/elasticsearch/values.yaml b/clusters/prod/apps/elasticsearch/values.yaml new file mode 100644 index 0000000..1aca279 --- /dev/null +++ b/clusters/prod/apps/elasticsearch/values.yaml @@ -0,0 +1,51 @@ +elasticsearch: + global: + imageRegistry: n47w5524.c1.de1.container-registry.ovh.net + imagePullSecrets: + - docker-registry-secret + + image: + repository: hdc-services-external/bitnami/elasticsearch + tag: 7.17.3-debian-10-r29 + + # Single-node master (no split-brain risk with 1 node) + master: + replicas: 1 + heapSize: 1024m + resources: + requests: + cpu: 25m + memory: 2048Mi + persistence: + accessModes: + - ReadWriteOnce + + # Single data node + data: + replicas: 1 + heapSize: 1024m + resources: + requests: + cpu: 25m + memory: 2048Mi + persistence: + accessModes: + - ReadWriteOnce + size: 5Gi + + # Disabled node types (same as CSCS) + coordinating: + replicas: 0 + ingest: + replicas: 0 + curator: + enabled: false + metrics: + enabled: false + + # OVH nodes have vm.max_map_count=65530 (too low for ES, needs >=262144) + # Init container runs sysctl -w vm.max_map_count=262144 as privileged + sysctlImage: + enabled: true + repository: hdc-services-external/bitnami/bitnami-shell-archived + tag: 10-debian-10-r403 diff --git a/clusters/prod/apps/kafka/Chart.yaml b/clusters/prod/apps/kafka/Chart.yaml new file mode 100644 index 0000000..eb291bb --- /dev/null +++ b/clusters/prod/apps/kafka/Chart.yaml @@ -0,0 +1,7 @@ +apiVersion: v2 +name: kafka +version: 0.1.0 +dependencies: + - name: kafka + version: "20.0.3" + repository: https://pilotdataplatform.github.io/helm-charts/ diff --git a/clusters/prod/apps/kafka/application.yaml b/clusters/prod/apps/kafka/application.yaml new file mode 100644 index 0000000..2051797 --- /dev/null +++ b/clusters/prod/apps/kafka/application.yaml @@ -0,0 +1,27 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: kafka + namespace: argocd + annotations: + argocd.argoproj.io/sync-wave: "5" +spec: + project: default + source: + repoURL: https://github.com/PilotDataPlatform/pilot-hdc-platform-gitops.git + targetRevision: main + path: clusters/prod/apps/kafka + helm: + valueFiles: + - ../../registry.yaml + - values.yaml + destination: + server: https://kubernetes.default.svc + namespace: utility + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true + - ServerSideApply=true diff --git a/clusters/prod/apps/kafka/values.yaml b/clusters/prod/apps/kafka/values.yaml new file mode 100644 index 0000000..f6095ed --- /dev/null +++ b/clusters/prod/apps/kafka/values.yaml @@ -0,0 +1,112 @@ +kafka: + global: + imageRegistry: n47w5524.c1.de1.container-registry.ovh.net + imagePullSecrets: + - docker-registry-secret + + image: + repository: hdc-services-external/bitnami/kafka + + heapOpts: -Xms256M -Xmx256M + deleteTopicEnable: true + + replicaCount: 1 + defaultReplicationFactor: 1 + offsetsTopicReplicationFactor: 1 + transactionStateLogReplicationFactor: 1 + transactionStateLogMinIsr: 1 + + persistence: + enabled: true + size: 2Gi + + zookeeper: + image: + repository: hdc-services-external/bitnami/zookeeper + replicaCount: 1 + heapSize: 256 + persistence: + enabled: true + size: 1Gi + resources: + limits: + cpu: 500m + memory: 512Mi + + service: + type: ClusterIP + + extraDeploy: + - | + apiVersion: apps/v1 + kind: Deployment + metadata: + name: {{ include "kafka.name" . }}-connect + labels: {{- include "common.labels.standard" . | nindent 4 }} + app.kubernetes.io/component: connector + spec: + replicas: 1 + selector: + matchLabels: {{- include "common.labels.matchLabels" . | nindent 6 }} + app.kubernetes.io/component: connector + template: + metadata: + labels: {{- include "common.labels.standard" . | nindent 8 }} + app.kubernetes.io/component: connector + spec: + imagePullSecrets: + - name: docker-registry-secret + # initContainers: plugins-downloader removed — add back when CDC/ES connectors needed + containers: + - name: connect + image: n47w5524.c1.de1.container-registry.ovh.net/hdc-services-external/debezium/connect:1.1 + imagePullPolicy: IfNotPresent + ports: + - name: connector + containerPort: 8083 + volumeMounts: + - name: configuration + mountPath: /bitnami/kafka/config + - name: kafka-connect-plugins-dir + mountPath: /tmp + env: + - name: BOOTSTRAP_SERVERS + value: "kafka:9092" + - name: GROUP_ID + value: "sde_group" + - name: CONFIG_STORAGE_TOPIC + value: "sde_storage_topic" + - name: OFFSET_STORAGE_TOPIC + value: "sde_offset_topic" + - name: KAFKA_CONNECT_PLUGINS_DIR + value: "/kafka/connect,/tmp" + volumes: + - name: configuration + configMap: + name: {{ include "kafka.name" . }}-connect + - name: kafka-connect-plugins-dir + emptyDir: {} + - | + apiVersion: v1 + kind: ConfigMap + metadata: + name: {{ include "kafka.name" . }}-connect + labels: {{- include "common.labels.standard" . | nindent 4 }} + app.kubernetes.io/component: connector + data: + connect-standalone.properties: |- + bootstrap.servers = {{ include "kafka.name" . }}-0.{{ include "kafka.name" . }}-headless.{{ .Release.Namespace }}.svc.{{ .Values.clusterDomain }}:{{ .Values.service.port }} + - | + apiVersion: v1 + kind: Service + metadata: + name: {{ include "kafka.name" . }}-connect + labels: {{- include "common.labels.standard" . | nindent 4 }} + app.kubernetes.io/component: connector + spec: + ports: + - protocol: TCP + port: 8083 + targetPort: connector + selector: {{- include "common.labels.matchLabels" . | nindent 4 }} + app.kubernetes.io/component: connector diff --git a/clusters/prod/apps/keycloak-postgresql/Chart.yaml b/clusters/prod/apps/keycloak-postgresql/Chart.yaml new file mode 100644 index 0000000..681442e --- /dev/null +++ b/clusters/prod/apps/keycloak-postgresql/Chart.yaml @@ -0,0 +1,7 @@ +apiVersion: v2 +name: keycloak-postgresql +version: 0.1.0 +dependencies: + - name: postgresql + version: "15.5.17" + repository: https://pilotdataplatform.github.io/helm-charts/ diff --git a/clusters/prod/apps/keycloak-postgresql/application.yaml b/clusters/prod/apps/keycloak-postgresql/application.yaml new file mode 100644 index 0000000..515d787 --- /dev/null +++ b/clusters/prod/apps/keycloak-postgresql/application.yaml @@ -0,0 +1,27 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: keycloak-postgresql + namespace: argocd + annotations: + argocd.argoproj.io/sync-wave: "4" +spec: + project: default + source: + repoURL: https://github.com/PilotDataPlatform/pilot-hdc-platform-gitops.git + targetRevision: main + path: clusters/prod/apps/keycloak-postgresql + helm: + valueFiles: + - ../../registry.yaml + - values.yaml + destination: + server: https://kubernetes.default.svc + namespace: keycloak + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true + - ServerSideApply=true diff --git a/clusters/prod/apps/keycloak-postgresql/templates/external-secret.yaml b/clusters/prod/apps/keycloak-postgresql/templates/external-secret.yaml new file mode 100644 index 0000000..6fa19f7 --- /dev/null +++ b/clusters/prod/apps/keycloak-postgresql/templates/external-secret.yaml @@ -0,0 +1,21 @@ +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: keycloak-postgresql-credentials + namespace: keycloak +spec: + refreshInterval: 1h + secretStoreRef: + kind: ClusterSecretStore + name: vault + target: + name: keycloak-postgresql-credentials + data: + - secretKey: postgres-password + remoteRef: + key: secret/data/keycloak + property: postgres-password + - secretKey: password + remoteRef: + key: secret/data/keycloak + property: keycloak-user-password diff --git a/clusters/prod/apps/keycloak-postgresql/values.yaml b/clusters/prod/apps/keycloak-postgresql/values.yaml new file mode 100644 index 0000000..6bf2e66 --- /dev/null +++ b/clusters/prod/apps/keycloak-postgresql/values.yaml @@ -0,0 +1,30 @@ +postgresql: + fullnameOverride: keycloak-postgresql + + global: + imagePullSecrets: + - docker-registry-secret + postgresql: + auth: + database: bitnami_keycloak + username: bn_keycloak + existingSecret: keycloak-postgresql-credentials + secretKeys: + adminPasswordKey: postgres-password + userPasswordKey: password + + image: + repository: hdc-services-external/bitnami/postgresql + + primary: + persistence: + size: 5Gi + storageClass: "csi-cinder-high-speed" + + resources: + limits: + cpu: 500m + memory: 256Mi + requests: + cpu: 10m + memory: 64Mi diff --git a/clusters/prod/apps/message-bus-greenroom/Chart.yaml b/clusters/prod/apps/message-bus-greenroom/Chart.yaml new file mode 100644 index 0000000..526d5f5 --- /dev/null +++ b/clusters/prod/apps/message-bus-greenroom/Chart.yaml @@ -0,0 +1,7 @@ +apiVersion: v2 +name: message-bus-greenroom +version: 0.1.0 +dependencies: + - name: rabbitmq + version: "10.1.12" + repository: https://pilotdataplatform.github.io/helm-charts/ diff --git a/clusters/prod/apps/message-bus-greenroom/application.yaml b/clusters/prod/apps/message-bus-greenroom/application.yaml new file mode 100644 index 0000000..6319b4f --- /dev/null +++ b/clusters/prod/apps/message-bus-greenroom/application.yaml @@ -0,0 +1,27 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: message-bus-greenroom + namespace: argocd + annotations: + argocd.argoproj.io/sync-wave: "5" +spec: + project: default + source: + repoURL: https://github.com/PilotDataPlatform/pilot-hdc-platform-gitops.git + targetRevision: main + path: clusters/prod/apps/message-bus-greenroom + helm: + valueFiles: + - ../../registry.yaml + - values.yaml + destination: + server: https://kubernetes.default.svc + namespace: greenroom + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true + - ServerSideApply=true diff --git a/clusters/prod/apps/message-bus-greenroom/templates/external-secret.yaml b/clusters/prod/apps/message-bus-greenroom/templates/external-secret.yaml new file mode 100644 index 0000000..910c0b5 --- /dev/null +++ b/clusters/prod/apps/message-bus-greenroom/templates/external-secret.yaml @@ -0,0 +1,17 @@ +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: message-bus-greenroom-credentials + namespace: greenroom +spec: + refreshInterval: 1h + secretStoreRef: + kind: ClusterSecretStore + name: vault + target: + name: message-bus-greenroom-credentials + data: + - secretKey: rabbitmq-password + remoteRef: + key: secret/data/rabbitmq + property: password diff --git a/clusters/prod/apps/message-bus-greenroom/values.yaml b/clusters/prod/apps/message-bus-greenroom/values.yaml new file mode 100644 index 0000000..132435d --- /dev/null +++ b/clusters/prod/apps/message-bus-greenroom/values.yaml @@ -0,0 +1,32 @@ +rabbitmq: + fullnameOverride: message-bus-greenroom + + global: + imagePullSecrets: + - docker-registry-secret + + image: + repository: hdc-services-external/bitnami/rabbitmq + tag: 3.11.6-debian-11-r0 + + replicaCount: 1 + + auth: + username: greenroom + existingPasswordSecret: message-bus-greenroom-credentials + + persistence: + enabled: true + storageClass: "csi-cinder-high-speed" + size: 1Gi + + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: 500m + memory: 512Mi + + service: + type: ClusterIP diff --git a/clusters/prod/apps/minio/Chart.yaml b/clusters/prod/apps/minio/Chart.yaml new file mode 100644 index 0000000..7a50a59 --- /dev/null +++ b/clusters/prod/apps/minio/Chart.yaml @@ -0,0 +1,7 @@ +apiVersion: v2 +name: minio +version: 0.1.0 +dependencies: + - name: minio + version: "11.7.13" + repository: https://pilotdataplatform.github.io/helm-charts/ diff --git a/clusters/prod/apps/minio/application.yaml b/clusters/prod/apps/minio/application.yaml new file mode 100644 index 0000000..c70a26f --- /dev/null +++ b/clusters/prod/apps/minio/application.yaml @@ -0,0 +1,28 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: minio + namespace: argocd + annotations: + argocd.argoproj.io/sync-wave: "5" +spec: + project: default + source: + repoURL: https://github.com/PilotDataPlatform/pilot-hdc-platform-gitops.git + targetRevision: main + path: clusters/prod/apps/minio + helm: + valueFiles: + - ../../registry.yaml + - ../../versions.yaml + - values.yaml + destination: + server: https://kubernetes.default.svc + namespace: minio + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true + - ServerSideApply=true diff --git a/clusters/prod/apps/minio/templates/external-secret.yaml b/clusters/prod/apps/minio/templates/external-secret.yaml new file mode 100644 index 0000000..5c5dabd --- /dev/null +++ b/clusters/prod/apps/minio/templates/external-secret.yaml @@ -0,0 +1,27 @@ +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: minio-credentials + namespace: minio +spec: + refreshInterval: 1h + secretStoreRef: + kind: ClusterSecretStore + name: vault + target: + name: minio-credentials + data: + # Auth credentials + - secretKey: root-user + remoteRef: + key: secret/data/minio + property: access_key + - secretKey: root-password + remoteRef: + key: secret/data/minio + property: secret_key + # Encryption master key (format: key-name:base64-key) + - secretKey: kms-secret-key + remoteRef: + key: secret/data/minio + property: kms_secret_key diff --git a/clusters/prod/apps/minio/values.yaml b/clusters/prod/apps/minio/values.yaml new file mode 100644 index 0000000..a9184b6 --- /dev/null +++ b/clusters/prod/apps/minio/values.yaml @@ -0,0 +1,79 @@ +minio: + fullnameOverride: minio + + global: + imagePullSecrets: + - docker-registry-secret + + image: + repository: hdc-services-external/bitnami/minio + # tag from versions.yaml (2022.12.12-debian-11-r9) + + mode: standalone + + auth: + existingSecret: minio-credentials + # Chart expects keys: root-user, root-password + + persistence: + enabled: true + storageClass: "csi-cinder-high-speed" + accessMode: ReadWriteOnce + size: 5Gi + + resources: + requests: + memory: 256Mi + cpu: 100m + limits: + memory: 512Mi + cpu: 500m + + # Console ingress - not needed, internal access only + ingress: + enabled: false + + # S3 API ingress - public endpoint for S3_PUBLIC access + apiIngress: + enabled: true + ingressClassName: nginx + hostname: object.hdc.ebrains.eu + path: / + pathType: Prefix + servicePort: minio-api + annotations: + cert-manager.io/cluster-issuer: letsencrypt-prod + nginx.ingress.kubernetes.io/proxy-body-size: 20m + nginx.ingress.kubernetes.io/proxy-buffer-size: 512k + nginx.ingress.kubernetes.io/proxy-buffering: "on" + nginx.ingress.kubernetes.io/proxy-buffers-number: "8" + nginx.ingress.kubernetes.io/proxy-connect-timeout: 180s + nginx.ingress.kubernetes.io/proxy-max-temp-file-size: "0" + nginx.ingress.kubernetes.io/proxy-read-timeout: 180s + nginx.ingress.kubernetes.io/proxy-send-timeout: 180s + tls: true + + service: + type: ClusterIP + + # Bucket provisioning (runs as post-install/post-upgrade Job) + provisioning: + enabled: true + buckets: + - name: public-resources + - name: project-logos + # Anonymous read-only access via extraCommands because chart's native + # provisioning.policies only supports IAM policies (mc admin policy add) — + # no Principal field. Anonymous access needs "Principal":{"AWS":["*"]} + # which requires mc anonymous set-json instead. + # Note: chart sets mc alias as "provisioning" (not "local") + extraCommands: + - 'for BUCKET in public-resources project-logos; do printf ''{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["*"]},"Action":["s3:GetObject"],"Resource":["arn:aws:s3:::%s/*"]}]}'' "$BUCKET" > /tmp/anon-policy.json && mc anonymous set-json /tmp/anon-policy.json provisioning/$BUCKET; done' + + # Encryption via built-in master key (no KES needed) + extraEnvVars: + - name: MINIO_KMS_SECRET_KEY + valueFrom: + secretKeyRef: + name: minio-credentials + key: kms-secret-key diff --git a/clusters/prod/apps/postgresql/Chart.yaml b/clusters/prod/apps/postgresql/Chart.yaml new file mode 100644 index 0000000..445551f --- /dev/null +++ b/clusters/prod/apps/postgresql/Chart.yaml @@ -0,0 +1,7 @@ +apiVersion: v2 +name: postgresql +version: 0.1.0 +dependencies: + - name: postgresql + version: "15.5.17" + repository: https://pilotdataplatform.github.io/helm-charts/ diff --git a/clusters/prod/apps/postgresql/application.yaml b/clusters/prod/apps/postgresql/application.yaml new file mode 100644 index 0000000..e353f2f --- /dev/null +++ b/clusters/prod/apps/postgresql/application.yaml @@ -0,0 +1,28 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: postgresql + namespace: argocd + annotations: + argocd.argoproj.io/sync-wave: "4" +spec: + project: default + source: + repoURL: https://github.com/PilotDataPlatform/pilot-hdc-platform-gitops.git + targetRevision: main + path: clusters/prod/apps/postgresql + helm: + valueFiles: + - ../../registry.yaml + - ../../versions.yaml + - values.yaml + destination: + server: https://kubernetes.default.svc + namespace: utility + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true + - ServerSideApply=true diff --git a/clusters/prod/apps/postgresql/templates/_helpers.tpl b/clusters/prod/apps/postgresql/templates/_helpers.tpl new file mode 100644 index 0000000..e92f4c4 --- /dev/null +++ b/clusters/prod/apps/postgresql/templates/_helpers.tpl @@ -0,0 +1,6 @@ +{{/* +Compose full image path from global registry + repository + tag +*/}} +{{- define "postgresql.initImage" -}} +{{- .Values.postgresql.global.imageRegistry }}/{{- .Values.postgresql.image.repository }}:{{- .Values.postgresql.image.tag -}} +{{- end }} diff --git a/clusters/prod/apps/postgresql/templates/external-secret.yaml b/clusters/prod/apps/postgresql/templates/external-secret.yaml new file mode 100644 index 0000000..f8eccbb --- /dev/null +++ b/clusters/prod/apps/postgresql/templates/external-secret.yaml @@ -0,0 +1,49 @@ +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: postgresql-credentials + namespace: utility +spec: + refreshInterval: 1h + secretStoreRef: + kind: ClusterSecretStore + name: vault + target: + name: postgresql-credentials + data: + - secretKey: postgres-password + remoteRef: + key: secret/data/postgresql + property: postgres-password + - secretKey: metadata-user-password + remoteRef: + key: secret/data/postgresql + property: metadata-user-password + - secretKey: project-user-password + remoteRef: + key: secret/data/postgresql + property: project-user-password + - secretKey: auth-user-password + remoteRef: + key: secret/data/postgresql + property: auth-user-password + - secretKey: dataops-user-password + remoteRef: + key: secret/data/postgresql + property: dataops-user-password + - secretKey: notification-user-password + remoteRef: + key: secret/data/postgresql + property: notification-user-password + - secretKey: dataset-user-password + remoteRef: + key: secret/data/postgresql + property: dataset-user-password + - secretKey: approval-user-password + remoteRef: + key: secret/data/postgresql + property: approval-user-password + - secretKey: kg-integration-user-password + remoteRef: + key: secret/data/postgresql + property: kg-integration-user-password diff --git a/clusters/prod/apps/postgresql/templates/init-job.yaml b/clusters/prod/apps/postgresql/templates/init-job.yaml new file mode 100644 index 0000000..fcfc798 --- /dev/null +++ b/clusters/prod/apps/postgresql/templates/init-job.yaml @@ -0,0 +1,236 @@ +apiVersion: batch/v1 +kind: Job +metadata: + name: postgresql-init-users + namespace: utility + annotations: + argocd.argoproj.io/hook: PostSync + argocd.argoproj.io/hook-delete-policy: HookSucceeded +spec: + ttlSecondsAfterFinished: 300 + template: + spec: + restartPolicy: OnFailure + imagePullSecrets: + - name: docker-registry-secret + securityContext: + runAsNonRoot: true + runAsUser: 1001 + fsGroup: 1001 + seccompProfile: + type: RuntimeDefault + containers: + - name: init-users + image: {{ include "postgresql.initImage" . }} + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + resources: + requests: + cpu: 50m + memory: 64Mi + limits: + cpu: 200m + memory: 128Mi + env: + - name: PGHOST + value: postgres + - name: PGUSER + value: postgres + - name: PGPASSWORD + valueFrom: + secretKeyRef: + name: postgresql-credentials + key: postgres-password + - name: METADATA_USER_PASSWORD + valueFrom: + secretKeyRef: + name: postgresql-credentials + key: metadata-user-password + - name: PROJECT_USER_PASSWORD + valueFrom: + secretKeyRef: + name: postgresql-credentials + key: project-user-password + - name: AUTH_USER_PASSWORD + valueFrom: + secretKeyRef: + name: postgresql-credentials + key: auth-user-password + - name: DATAOPS_USER_PASSWORD + valueFrom: + secretKeyRef: + name: postgresql-credentials + key: dataops-user-password + - name: NOTIFICATION_USER_PASSWORD + valueFrom: + secretKeyRef: + name: postgresql-credentials + key: notification-user-password + - name: DATASET_USER_PASSWORD + valueFrom: + secretKeyRef: + name: postgresql-credentials + key: dataset-user-password + - name: APPROVAL_USER_PASSWORD + valueFrom: + secretKeyRef: + name: postgresql-credentials + key: approval-user-password + - name: KG_INTEGRATION_USER_PASSWORD + valueFrom: + secretKeyRef: + name: postgresql-credentials + key: kg-integration-user-password + command: + - /bin/bash + - -c + - | + set -e + + # Wait for postgres to be fully ready (not just accepting connections) + max_retries=60 # 60 × 10s = 10 min timeout + count=0 + until psql -h "$PGHOST" -U "$PGUSER" -c 'SELECT 1' >/dev/null 2>&1; do + count=$((count + 1)) + if [ $count -ge $max_retries ]; then + echo "ERROR: PostgreSQL not ready after $max_retries attempts" + exit 1 + fi + echo "Waiting for postgres... ($count/$max_retries)" + sleep 10 + done + + echo "Creating service users..." + + # Use psql variables with quote_literal() for safe password handling + # This prevents SQL injection if passwords contain quotes or special chars + + # metadata_user + psql -v ON_ERROR_STOP=1 -v pwd="$METADATA_USER_PASSWORD" <<'EOSQL' + SELECT 'CREATE USER metadata_user WITH PASSWORD ' || quote_literal(:'pwd') + WHERE NOT EXISTS (SELECT FROM pg_user WHERE usename = 'metadata_user') \gexec + SELECT 'ALTER USER metadata_user WITH PASSWORD ' || quote_literal(:'pwd') + WHERE EXISTS (SELECT FROM pg_user WHERE usename = 'metadata_user') \gexec + GRANT CONNECT ON DATABASE metadata TO metadata_user; + \c metadata + ALTER SCHEMA metadata OWNER TO metadata_user; + GRANT ALL PRIVILEGES ON SCHEMA metadata TO metadata_user; + GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA metadata TO metadata_user; + GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA metadata TO metadata_user; + ALTER DEFAULT PRIVILEGES IN SCHEMA metadata GRANT ALL ON TABLES TO metadata_user; + ALTER DEFAULT PRIVILEGES IN SCHEMA metadata GRANT ALL ON SEQUENCES TO metadata_user; + \c postgres + ALTER DATABASE metadata OWNER TO metadata_user; + EOSQL + + # project_user + psql -v ON_ERROR_STOP=1 -v pwd="$PROJECT_USER_PASSWORD" <<'EOSQL' + SELECT 'CREATE USER project_user WITH PASSWORD ' || quote_literal(:'pwd') + WHERE NOT EXISTS (SELECT FROM pg_user WHERE usename = 'project_user') \gexec + SELECT 'ALTER USER project_user WITH PASSWORD ' || quote_literal(:'pwd') + WHERE EXISTS (SELECT FROM pg_user WHERE usename = 'project_user') \gexec + GRANT CONNECT ON DATABASE project TO project_user; + GRANT ALL PRIVILEGES ON DATABASE project TO project_user; + ALTER DATABASE project OWNER TO project_user; + EOSQL + + # dataops_user + psql -v ON_ERROR_STOP=1 -v pwd="$DATAOPS_USER_PASSWORD" <<'EOSQL' + SELECT 'CREATE USER dataops_user WITH PASSWORD ' || quote_literal(:'pwd') + WHERE NOT EXISTS (SELECT FROM pg_user WHERE usename = 'dataops_user') \gexec + SELECT 'ALTER USER dataops_user WITH PASSWORD ' || quote_literal(:'pwd') + WHERE EXISTS (SELECT FROM pg_user WHERE usename = 'dataops_user') \gexec + GRANT CONNECT ON DATABASE dataops TO dataops_user; + GRANT ALL PRIVILEGES ON DATABASE dataops TO dataops_user; + ALTER DATABASE dataops OWNER TO dataops_user; + EOSQL + + # auth_user + psql -v ON_ERROR_STOP=1 -v pwd="$AUTH_USER_PASSWORD" <<'EOSQL' + SELECT 'CREATE USER auth_user WITH PASSWORD ' || quote_literal(:'pwd') + WHERE NOT EXISTS (SELECT FROM pg_user WHERE usename = 'auth_user') \gexec + SELECT 'ALTER USER auth_user WITH PASSWORD ' || quote_literal(:'pwd') + WHERE EXISTS (SELECT FROM pg_user WHERE usename = 'auth_user') \gexec + GRANT CONNECT ON DATABASE auth TO auth_user; + \c auth + GRANT ALL PRIVILEGES ON SCHEMA pilot_invitation TO auth_user; + GRANT ALL PRIVILEGES ON SCHEMA pilot_casbin TO auth_user; + GRANT ALL PRIVILEGES ON SCHEMA pilot_event TO auth_user; + GRANT ALL PRIVILEGES ON SCHEMA pilot_ldap TO auth_user; + GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA pilot_invitation TO auth_user; + GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA pilot_casbin TO auth_user; + GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA pilot_event TO auth_user; + GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA pilot_ldap TO auth_user; + GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA pilot_invitation TO auth_user; + GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA pilot_casbin TO auth_user; + GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA pilot_event TO auth_user; + GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA pilot_ldap TO auth_user; + ALTER DEFAULT PRIVILEGES IN SCHEMA pilot_invitation GRANT ALL ON TABLES TO auth_user; + ALTER DEFAULT PRIVILEGES IN SCHEMA pilot_casbin GRANT ALL ON TABLES TO auth_user; + ALTER DEFAULT PRIVILEGES IN SCHEMA pilot_event GRANT ALL ON TABLES TO auth_user; + ALTER DEFAULT PRIVILEGES IN SCHEMA pilot_ldap GRANT ALL ON TABLES TO auth_user; + ALTER DEFAULT PRIVILEGES IN SCHEMA pilot_invitation GRANT ALL ON SEQUENCES TO auth_user; + ALTER DEFAULT PRIVILEGES IN SCHEMA pilot_casbin GRANT ALL ON SEQUENCES TO auth_user; + ALTER DEFAULT PRIVILEGES IN SCHEMA pilot_event GRANT ALL ON SEQUENCES TO auth_user; + ALTER DEFAULT PRIVILEGES IN SCHEMA pilot_ldap GRANT ALL ON SEQUENCES TO auth_user; + \c postgres + ALTER DATABASE auth OWNER TO auth_user; + EOSQL + + # notification_user + psql -v ON_ERROR_STOP=1 -v pwd="$NOTIFICATION_USER_PASSWORD" <<'EOSQL' + SELECT 'CREATE USER notification_user WITH PASSWORD ' || quote_literal(:'pwd') + WHERE NOT EXISTS (SELECT FROM pg_user WHERE usename = 'notification_user') \gexec + SELECT 'ALTER USER notification_user WITH PASSWORD ' || quote_literal(:'pwd') + WHERE EXISTS (SELECT FROM pg_user WHERE usename = 'notification_user') \gexec + GRANT CONNECT ON DATABASE notifications TO notification_user; + GRANT ALL PRIVILEGES ON DATABASE notifications TO notification_user; + ALTER DATABASE notifications OWNER TO notification_user; + EOSQL + + # dataset_user + psql -v ON_ERROR_STOP=1 -v pwd="$DATASET_USER_PASSWORD" <<'EOSQL' + SELECT 'CREATE USER dataset_user WITH PASSWORD ' || quote_literal(:'pwd') + WHERE NOT EXISTS (SELECT FROM pg_user WHERE usename = 'dataset_user') \gexec + SELECT 'ALTER USER dataset_user WITH PASSWORD ' || quote_literal(:'pwd') + WHERE EXISTS (SELECT FROM pg_user WHERE usename = 'dataset_user') \gexec + GRANT CONNECT ON DATABASE dataset TO dataset_user; + GRANT ALL PRIVILEGES ON DATABASE dataset TO dataset_user; + ALTER DATABASE dataset OWNER TO dataset_user; + EOSQL + + # approval_user + psql -v ON_ERROR_STOP=1 -v pwd="$APPROVAL_USER_PASSWORD" <<'EOSQL' + SELECT 'CREATE USER approval_user WITH PASSWORD ' || quote_literal(:'pwd') + WHERE NOT EXISTS (SELECT FROM pg_user WHERE usename = 'approval_user') \gexec + SELECT 'ALTER USER approval_user WITH PASSWORD ' || quote_literal(:'pwd') + WHERE EXISTS (SELECT FROM pg_user WHERE usename = 'approval_user') \gexec + GRANT CONNECT ON DATABASE approval TO approval_user; + GRANT ALL PRIVILEGES ON DATABASE approval TO approval_user; + ALTER DATABASE approval OWNER TO approval_user; + ALTER DATABASE approval_user OWNER TO approval_user; + EOSQL + + # kg_integration_user (named schema like metadata) + psql -v ON_ERROR_STOP=1 -v pwd="$KG_INTEGRATION_USER_PASSWORD" <<'EOSQL' + SELECT 'CREATE USER kg_integration_user WITH PASSWORD ' || quote_literal(:'pwd') + WHERE NOT EXISTS (SELECT FROM pg_user WHERE usename = 'kg_integration_user') \gexec + SELECT 'ALTER USER kg_integration_user WITH PASSWORD ' || quote_literal(:'pwd') + WHERE EXISTS (SELECT FROM pg_user WHERE usename = 'kg_integration_user') \gexec + GRANT CONNECT ON DATABASE kg_integration TO kg_integration_user; + \c kg_integration + CREATE SCHEMA IF NOT EXISTS kg_integration; + ALTER SCHEMA kg_integration OWNER TO kg_integration_user; + GRANT ALL PRIVILEGES ON SCHEMA kg_integration TO kg_integration_user; + GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA kg_integration TO kg_integration_user; + GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA kg_integration TO kg_integration_user; + ALTER DEFAULT PRIVILEGES IN SCHEMA kg_integration GRANT ALL ON TABLES TO kg_integration_user; + ALTER DEFAULT PRIVILEGES IN SCHEMA kg_integration GRANT ALL ON SEQUENCES TO kg_integration_user; + \c postgres + ALTER DATABASE kg_integration OWNER TO kg_integration_user; + EOSQL + + echo "Service users created successfully!" diff --git a/clusters/prod/apps/postgresql/values.yaml b/clusters/prod/apps/postgresql/values.yaml new file mode 100644 index 0000000..4b9e651 --- /dev/null +++ b/clusters/prod/apps/postgresql/values.yaml @@ -0,0 +1,75 @@ +postgresql: + fullnameOverride: postgres + + global: + imagePullSecrets: + - docker-registry-secret + postgresql: + auth: + existingSecret: postgresql-credentials + secretKeys: + adminPasswordKey: postgres-password + + image: + repository: hdc-services-external/postgresql + + primary: + initdb: + scripts: + 00-create-databases.sh: | + #!/bin/bash + set -e + export PGPASSWORD=$POSTGRES_PASSWORD + + cat > /tmp/create_databases.sql <<'EOF' + -- Create databases (only those with services deployed) + SELECT 'CREATE DATABASE project' WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'project')\gexec + SELECT 'CREATE DATABASE metadata' WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'metadata')\gexec + SELECT 'CREATE DATABASE dataops' WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'dataops')\gexec + SELECT 'CREATE DATABASE auth' WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'auth')\gexec + SELECT 'CREATE DATABASE notifications' WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'notifications')\gexec + SELECT 'CREATE DATABASE approval' WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'approval')\gexec + SELECT 'CREATE DATABASE dataset' WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'dataset')\gexec + SELECT 'CREATE DATABASE approval_user' WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'approval_user')\gexec -- psql default DB for approval init container + SELECT 'CREATE DATABASE kg_integration' WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'kg_integration')\gexec + + -- kg_integration schema + \c kg_integration + CREATE SCHEMA IF NOT EXISTS kg_integration; + + -- metadata schemas + extensions + \c metadata + CREATE SCHEMA IF NOT EXISTS metadata; + CREATE EXTENSION IF NOT EXISTS ltree; + CREATE EXTENSION IF NOT EXISTS pg_cron; + SELECT cron.schedule('expire_REGISTERED_items', '0 */1 * * *', $$DELETE FROM metadata.items WHERE status='REGISTERED' AND last_updated_time < now() - interval '1 day'$$) WHERE NOT EXISTS (SELECT FROM cron.job WHERE jobname = 'expire_REGISTERED_items'); + SELECT cron.schedule('expire_job_history', '0 3 */1 * *', $$DELETE FROM cron.job_run_details WHERE end_time < now() - interval '30 days'$$) WHERE NOT EXISTS (SELECT FROM cron.job WHERE jobname = 'expire_job_history'); + + -- auth schemas + \c auth + CREATE SCHEMA IF NOT EXISTS pilot_invitation; + CREATE SCHEMA IF NOT EXISTS pilot_casbin; + CREATE SCHEMA IF NOT EXISTS pilot_event; + CREATE SCHEMA IF NOT EXISTS pilot_ldap; + EOF + + psql -U postgres -f /tmp/create_databases.sql + + extendedConfiguration: | + max_connections = 200 + shared_buffers = 128MB + cron.database_name = 'metadata' + + persistence: + size: 10Gi + storageClass: "csi-cinder-high-speed" + + resources: + limits: + cpu: 1 + memory: 512Mi + requests: + cpu: 20m + memory: 128Mi + + postgresqlSharedPreloadLibraries: "pgaudit,pg_cron" diff --git a/clusters/prod/apps/redis/Chart.yaml b/clusters/prod/apps/redis/Chart.yaml new file mode 100644 index 0000000..caff6b2 --- /dev/null +++ b/clusters/prod/apps/redis/Chart.yaml @@ -0,0 +1,7 @@ +apiVersion: v2 +name: redis +version: 0.1.0 +dependencies: + - name: redis + version: "16.11.2" + repository: https://pilotdataplatform.github.io/helm-charts/ diff --git a/clusters/prod/apps/redis/application.yaml b/clusters/prod/apps/redis/application.yaml new file mode 100644 index 0000000..e9f9cde --- /dev/null +++ b/clusters/prod/apps/redis/application.yaml @@ -0,0 +1,27 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: redis + namespace: argocd + annotations: + argocd.argoproj.io/sync-wave: "5" +spec: + project: default + source: + repoURL: https://github.com/PilotDataPlatform/pilot-hdc-platform-gitops.git + targetRevision: main + path: clusters/prod/apps/redis + helm: + valueFiles: + - ../../registry.yaml + - values.yaml + destination: + server: https://kubernetes.default.svc + namespace: redis + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true + - ServerSideApply=true diff --git a/clusters/prod/apps/redis/templates/external-secret.yaml b/clusters/prod/apps/redis/templates/external-secret.yaml new file mode 100644 index 0000000..955a85f --- /dev/null +++ b/clusters/prod/apps/redis/templates/external-secret.yaml @@ -0,0 +1,17 @@ +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: redis-credentials + namespace: redis +spec: + refreshInterval: 1h + secretStoreRef: + kind: ClusterSecretStore + name: vault + target: + name: redis-credentials + data: + - secretKey: password + remoteRef: + key: secret/data/redis + property: password diff --git a/clusters/prod/apps/redis/values.yaml b/clusters/prod/apps/redis/values.yaml new file mode 100644 index 0000000..866c63b --- /dev/null +++ b/clusters/prod/apps/redis/values.yaml @@ -0,0 +1,28 @@ +redis: + fullnameOverride: redis + + global: + imagePullSecrets: + - docker-registry-secret + + image: + repository: hdc-services-external/bitnami/redis + + architecture: standalone + + auth: + existingSecret: redis-credentials + existingSecretPasswordKey: password + + master: + persistence: + size: 1Gi + storageClass: "csi-cinder-high-speed" + + resources: + limits: + memory: 256Mi + cpu: 500m + requests: + memory: 128Mi + cpu: 100m diff --git a/docs/vault-secrets.md b/docs/vault-secrets.md new file mode 100644 index 0000000..ce5ba5d --- /dev/null +++ b/docs/vault-secrets.md @@ -0,0 +1,135 @@ +# Vault Secret Provisioning + +Ready-to-run commands for provisioning all required Vault secrets. Run these after Vault is bootstrapped (see main README). + +> **Per-environment**: each cluster has its own Vault instance. Run these commands against the target cluster's Vault (port-forward + `vault login` first). + +## PostgreSQL (`secret/postgresql`) + +```bash +vault kv put secret/postgresql \ + postgres-password=$(openssl rand -hex 24) \ + metadata-user-password=$(openssl rand -hex 24) \ + project-user-password=$(openssl rand -hex 24) \ + auth-user-password=$(openssl rand -hex 24) \ + dataops-user-password=$(openssl rand -hex 24) \ + notification-user-password=$(openssl rand -hex 24) \ + dataset-user-password=$(openssl rand -hex 24) \ + approval-user-password=$(openssl rand -hex 24) \ + kg-integration-user-password=$(openssl rand -hex 24) +``` + +To add or update a single service password: + +```bash +vault kv patch secret/postgresql -user-password=$(openssl rand -hex 24) +``` + +## Keycloak (`secret/keycloak`) + +```bash +vault kv put secret/keycloak \ + admin-password=$(openssl rand -hex 24) \ + postgres-password=$(openssl rand -hex 24) +``` + +## Redis (`secret/redis`) + +```bash +vault kv put secret/redis \ + password=$(openssl rand -hex 24) +``` + +**Use hex only** (`rand -hex`, not `rand -base64`). Some HDC services build `redis://` URIs without URL-encoding — base64 chars (`+`, `/`, `=`) break parsing. + +## MinIO (`secret/minio`) + +```bash +ENCRYPTION_KEY=$(openssl rand -base64 32) + +vault kv put secret/minio \ + access_key="minio-admin" \ + secret_key="$(openssl rand -hex 16)" \ + kms_secret_key="minio-encryption-key:${ENCRYPTION_KEY}" +``` + +**Important**: Back up `kms_secret_key` — losing it means losing access to encrypted data. + +## RabbitMQ (`secret/rabbitmq`) + +```bash +vault kv put secret/rabbitmq \ + username="greenroom" \ + password=$(openssl rand -hex 24) +``` + +## Docker Registry (`secret/docker-registry/ovh`) + +```bash +vault kv put secret/docker-registry/ovh \ + username='' \ + password='' +``` + +## Auth (`secret/auth`) + +```bash +vault kv put secret/auth \ + keycloak-client-secret='' +``` + +## Kong (`secret/kong`) + +```bash +vault kv put secret/kong \ + postgres-user="kong" \ + postgres-password=$(openssl rand -hex 24) +``` + +## Approval (`secret/approval`) + +```bash +vault kv put secret/approval \ + db-uri='postgresql://approval_user:@postgres.utility.svc:5432/approval' +``` + +Replace `` with the `approval-user-password` from `secret/postgresql`. + +## Download (`secret/download`) + +```bash +vault kv put secret/download \ + download-key=$(openssl rand -hex 32) +``` + +## KG Integration (`secret/kg-integration`) + +```bash +vault kv put secret/kg-integration \ + account-secret='' +``` + +## BFF CLI (`secret/bff-cli`) + +```bash +vault kv put secret/bff-cli \ + cli-secret='' \ + atlas-password='' \ + guacamole-jwt-public-key='' +``` + +## Guacamole (`secret/guacamole`) + +```bash +vault kv put secret/guacamole \ + pg-password=$(openssl rand -hex 24) +``` + +## GitHub Runner (`secret/github-runner`) — dev only + +```bash +vault kv put secret/github-runner \ + github_app_id='' \ + github_app_installation_id='' \ + github_app_private_key='' +```