diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 46d8a14e4a..bc1cc635b8 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -4,145 +4,18 @@ APL Core (App Platform for Linode) is a Kubernetes platform that integrates 30+ cloud-native applications (Istio, Argo CD, Keycloak, Tekton, Harbor, etc.) into a cohesive, multi-tenant PaaS. The codebase is a hybrid of TypeScript (CLI/operators), Helm charts, Helmfile manifests, and Go templates. -**Core Architecture:** User configuration (`env/` directory) → Helmfile bases → Helmfile releases → Helm charts → Kubernetes manifests - -## Critical Development Patterns - -### Values Flow (3-Stage Merge) - -Values are loaded in a strict 3-stage pipeline (see [ADR-2021-10-18](../adr/2021-10-18-defaults-and-derived.md)): - -1. **Defaults** (`helmfile.d/snippets/defaults.yaml`) - Static defaults, will eventually come from schema -2. **User Input** (`$ENV_DIR/env/**/*.yaml`) - User-provided configuration (NEVER write defaults/derived values here) -3. **Derived Values** (`helmfile.d/snippets/derived.gotmpl`) - Computed from defaults + user input - -**Critical Rule:** User input directory (`$ENV_DIR`) contains ONLY user-supplied values. Defaults and derived values must never be written back to `$ENV_DIR`. - -### Helmfile Release Patterns - -All Helmfile specs in `helmfile.d/` execute alphabetically. Use reusable anchors from `helmfile.d/snippets/templates.gotmpl`: - -- `*default` - Standard chart deployment. Values merged: `charts/{name}/values.yaml` → `values/{name}/{name}.gotmpl` → `.Values.apps.{name}._rawValues` -- `*raw` - Deploy additional K8s manifests (operators + CRs) from `values/{name}/{name}-raw.gotmpl` -- `*rawCR` - Deploy custom resources using the `raw-cr` chart -- `*jobs` - Deploy jobs to the `maintenance` namespace using `values/jobs/{name}.gotmpl` - -### Schema-Driven Validation - -All user-configurable parameters MUST be defined in `values-schema.yaml` (JSON Schema). Run `npm run validate-values` to validate. The schema serves as both validation and documentation. - -## CLI Commands & Workflow - -### Essential Commands - -```bash -# Bootstrap a new values repo (creates $ENV_DIR with defaults) -otomi bootstrap - -# Validate user configuration against schema -otomi validate-values - -# Validate rendered Kubernetes manifests -otomi validate-templates [-l name=myapp] - -# Render values for inspection -otomi values - -# Render chart values for a specific app -otomi x helmfile -l name=myapp write-values - -# Deploy all charts (or use -l name=myapp for selective deploy) -otomi apply [-l name=myapp] - -# Generate diff before applying -otomi diff [-l name=myapp] - -# Deploy to cluster (initial setup) -otomi install -``` - -### Development Setup - -```bash -# Install dependencies (helmfile, helm, kubectl, etc.) -npm run install-deps - -# Run CLI locally (bypass Docker) -export IN_DOCKER=false -export ENV_DIR=$PWD/tests/fixtures -export NODE_ENV=test - -# Compile TypeScript -npm run compile - -# Run tests -npm test -``` - -## Integrating a New Core App - -1. **Add Helm chart** to `charts/{myapp}/` (or vendor from upstream) -2. **Create values template** at `values/{myapp}/{myapp}.gotmpl` -3. **Define Helmfile release** in appropriate `helmfile.d/helmfile-*.yaml` file: - ```yaml - releases: - - name: myapp - installed: {{ .Values.apps.myapp.enabled }} - namespace: my-namespace - <<: *default # or *raw, *rawCR, *jobs - ``` -4. **Add schema** for user-configurable properties in `values-schema.yaml` under `.definitions.apps.properties.myapp` -5. **Configure defaults** in `helmfile.d/snippets/defaults.yaml` under `apps.myapp` -6. **Add namespace** (if needed) to `core.yaml` at `k8s.namespaces` -7. **Configure ingress** (if needed) in `core.yaml` at `adminApps` or `teamApps` - -## Docker-Based Execution - -The `binzx/otomi` script wraps all commands in Docker by default: - -- Uses `linode/apl-core:${otomi_version}` image -- Mounts `$ENV_DIR` as `/home/app/env/` -- Set `IN_DOCKER=false` to run locally (useful for cloud provider auth plugins) - -## Testing Strategy - -- Unit tests: `npm test` (Jest, located in `src/**/*.test.ts`) -- Integration tests: Use fixtures in `tests/fixtures/` with `NODE_ENV=test` -- Template validation: `otomi validate-templates` (validates all rendered manifests against K8s schemas) -- Policy tests: `npm run test:opa` (Rego policy testing) - -## Key Files & Directories - -| Path | Purpose | -| ---------------------- | -------------------------------------------- | -| `src/cmd/*.ts` | CLI command implementations | -| `helmfile.d/` | Helmfile specs (execute alphabetically) | -| `helmfile.d/snippets/` | Reusable templates, defaults, derived values | -| `charts/` | Helm charts (vendored and custom) | -| `values/` | Value templates for each chart | -| `values-schema.yaml` | JSON Schema for user configuration | -| `core.yaml` | Namespaces, ingress, team apps config | -| `binzx/otomi` | Bash wrapper for Docker-based execution | -| `adr/` | Architectural Decision Records | - -## Common Gotchas - -- **Helmfile labels:** Use `-l name=myapp` to select specific releases (not `-l app=myapp`) -- **Raw values override:** Use `apps.{name}._rawValues` to override chart values not in schema (use sparingly) -- **YAML anchors:** Search for `&anchorname` to find anchor definitions when you see `<<: *anchorname` -- **Keycloak integration:** Use `_derived.oidcBaseUrl`, `apps.keycloak.idp.clientID/clientSecret` for SSO -- **Untrusted CA:** Check `_derived.untrustedCA` to conditionally disable cert verification - -## Debugging Tips - -- Check deployment state: `otomi status` -- View traces on errors: Collected automatically in `otomi apply` failures -- Inspect Helmfile output: `otomi x helmfile -l name=myapp template` -- Local development: Use `$PWD/tests/fixtures` as `$ENV_DIR` -- Enable verbose logging: Add `-v` flag to any command - -## References - -- Full development guide: [docs/development.md](../docs/development.md) -- Architectural decisions: [adr/index.md](../adr/index.md) -- Public docs: https://techdocs.akamai.com/app-platform/docs/welcome +## Knowledge Base + +Use AGENTS.md files as your primary reference for understanding the codebase structure, conventions, and critical patterns. Each AGENTS.md file provides a comprehensive overview of its respective directory. + +| Path | Focus | +| ---------------------------------------------------------------- | --------------------------------------------------- | +| [`AGENTS.md`](AGENTS.md) | High level design | +| [`src/AGENTS.md`](src/AGENTS.md) | TypeScript source structure, conventions, dev setup | +| [`src/cmd/AGENTS.md`](src/cmd/AGENTS.md) | CLI command inventory, patterns | +| [`src/common/AGENTS.md`](src/common/AGENTS.md) | Shared utility modules, dependency graph | +| [`src/operator/AGENTS.md`](src/operator/AGENTS.md) | GitOps operator architecture, execution flow | +| [`helmfile.d/AGENTS.md`](helmfile.d/AGENTS.md) | Helmfile release phases, execution order | +| [`helmfile.d/snippets/AGENTS.md`](helmfile.d/snippets/AGENTS.md) | Critical templates, defaults, derived values | +| [`charts/AGENTS.md`](charts/AGENTS.md) | Custom vs vendored chart inventory | +| [`charts/team-ns/AGENTS.md`](charts/team-ns/AGENTS.md) | Team namespace chart (most complex) | diff --git a/.ignore b/.ignore new file mode 100644 index 0000000000..d19e7c237b --- /dev/null +++ b/.ignore @@ -0,0 +1,22 @@ +# we don't allow json files in the root except package.json + +.history +.tmp +_.bak +node_modules/ +/coverage/ +/dist/ +/env/ +_.DS*Store +.vscode/values-schema.yaml +*.env +/.secrets +chart/apl/values.schema.json +chart/apl/README.md +workflow/ +\_.new +.envrc +otomi.cpuprofile +/.idea/ +tmp +\*\*values-repo.yaml diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000000..4a1b2bed01 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,718 @@ +# APL Core — AI Agent Index + +> **App Platform (APL)** is a Kubernetes PaaS that integrates 30+ cloud-native applications into a cohesive, multi-tenant platform. +> This index provides the context AI agents need to perform software development tasks efficiently. + +**Stack:** TypeScript CLI + Kubernetes Operator + Helmfile/Helm + Go Templates +**Scale:** 3655 files, 357K lines, 54 charts, 76 TS source files + +## Subdirectory Knowledge Base + +| Path | Focus | +| ---------------------------------------------------------------- | --------------------------------------------------- | +| [`src/AGENTS.md`](src/AGENTS.md) | TypeScript source structure, conventions, dev setup | +| [`src/cmd/AGENTS.md`](src/cmd/AGENTS.md) | CLI command inventory, patterns | +| [`src/common/AGENTS.md`](src/common/AGENTS.md) | Shared utility modules, dependency graph | +| [`src/operator/AGENTS.md`](src/operator/AGENTS.md) | GitOps operator architecture, execution flow | +| [`helmfile.d/AGENTS.md`](helmfile.d/AGENTS.md) | Helmfile release phases, execution order | +| [`helmfile.d/snippets/AGENTS.md`](helmfile.d/snippets/AGENTS.md) | Critical templates, defaults, derived values | +| [`charts/AGENTS.md`](charts/AGENTS.md) | Custom vs vendored chart inventory | +| [`charts/team-ns/AGENTS.md`](charts/team-ns/AGENTS.md) | Team namespace chart (most complex) | + +--- + +## 1. Architecture Overview + +### Codebase Structure + +``` +/workspace +├── src/ # TypeScript CLI + Operator code +│ ├── cmd/ # CLI command implementations (apply, bootstrap, install, migrate, etc.) +│ ├── common/ # Shared utilities (git, k8s, values, crypto, helmfile wrapper) +│ ├── operator/ # APL Operator (watches CRDs, runs install/apply) +│ └── otomi.ts # CLI entrypoint +├── helmfile.d/ # Helmfile specs (executed alphabetically) +│ ├── helmfile-01..09.init # Core infrastructure (cert-manager, istio, keycloak, etc.) +│ ├── helmfile-03.databases # Platform databases (CloudNativePG) +│ ├── helmfile-10.monitoring # Monitoring stack (prometheus, alertmanager, grafana) +│ ├── helmfile-15.ingress-core# Core ingress + admin team namespace +│ ├── helmfile-20.ingress # External DNS +│ ├── helmfile-50.services # Optional services (trivy, kubeflow, kserve) +│ ├── helmfile-60.teams # Per-team releases (prometheus, grafana, tekton, team-ns) +│ ├── helmfile-70.shared # Shared services (harbor, oauth2-proxy, otomi-api, console) +│ ├── helmfile-90/91.artifacts# Raw K8s manifests (istio artifacts, otel artifacts) +│ └── snippets/ # Reusable templates, defaults, derived values +├── charts/ # Helm charts (custom + vendored) +│ ├── apl-*/ # Custom APL charts (operator, harbor-op, keycloak-op, gitea-op, network-policies) +│ ├── team-ns/ # Team namespace chart (RBAC, quotas, netpols, builds, ArgoCD, Kyverno) +│ ├── raw/ raw-cr/ # Charts for deploying raw K8s resources +│ ├── jobs/ # Chart for K8s Jobs/CronJobs +│ ├── skeleton/ # Template chart for new apps +│ └── / # Vendored upstream charts (ingress-nginx, keycloak, harbor, etc.) +├── values/ # Go template value files per chart +│ ├── /.gotmpl # Primary values template +│ └── /-raw.gotmpl # Raw K8s manifest templates +├── values-schema.yaml # JSON Schema for ALL user-configurable parameters +├── core.yaml # Namespace definitions + admin/team app ingress config +├── apps.yaml # App metadata (versions, descriptions, dependencies) +├── versions.yaml # Component version tags +├── values-changes.yaml # Values migration definitions (version-to-version) +└── tests/ # Test fixtures + integration tests +``` + +### Values Flow (3-Stage Merge) + +Every helmfile release loads values in this strict order: + +``` +1. DEFAULTS → helmfile.d/snippets/defaults.yaml (static defaults for all apps) +2. USER INPUT → $ENV_DIR/env/**/*.yaml (user-provided configuration) +3. DERIVED → helmfile.d/snippets/derived.gotmpl (computed: URLs, certs, Istio config) +``` + +**Critical Rule:** Never write defaults or derived values to `$ENV_DIR`. That directory holds ONLY user-supplied values. + +### Helmfile Release Anchors + +Defined in `helmfile.d/snippets/templates.gotmpl`: + +| Anchor | Chart Used | Values From | Purpose | +| ---------- | ----------------- | ----------------------------------------------------------- | ------------------------------- | +| `*default` | `charts/` | `values//.gotmpl` + `snippets/common.gotmpl` | Standard chart deployment | +| `*raw` | `charts/raw` | `values//-raw.gotmpl` | Deploy additional K8s manifests | +| `*rawCR` | `charts/raw-cr` | `values//-cr.gotmpl` + `snippets/common.gotmpl` | Deploy custom resources | +| `*jobs` | `charts/jobs` | `values/jobs/.gotmpl` | Jobs in `maintenance` namespace | +| `*otomiDb` | `charts/otomi-db` | `values//-otomi-db.gotmpl` | Platform database charts | + +### Helmfile Spec Pattern + +Every helmfile spec file follows this base loading pattern: + +```yaml +bases: + - snippets/defaults.yaml # Stage 1: defaults +--- +bases: + - snippets/env.gotmpl # Stage 2: user input from $ENV_DIR +--- +bases: + - snippets/derived.gotmpl # Stage 3: computed values +--- +{{ readFile "snippets/templates.gotmpl" }} # Load release anchors +{{- $v := .Values }} +{{- $a := $v.apps }} + +releases: + - name: myapp + installed: {{ $a | get "myapp.enabled" }} + namespace: my-namespace + <<: *default # Use anchor pattern +``` + +--- + +## 2. Key Concepts + +### APL operator modes + +Operation modes: + +- installer (src/installer.ts) +- operator (src/operator/apl-operator.ts) + +In the installer mode: + +- kubernetes maniefts are rendered and applied by helmfile + +In the operator mode: + +- App deployment is delegated to ArgoCD controller +- The operator only renders the values (using helmfile) and applies ArgoCD application manifests to the Kubernetes cluster +- The operator perform upgrades + +Operation modes are set based on the status stored in the kubernetes config map (see `APL_OPERATOR_STATUS_CM` variable) + +### Secrets + +All secrets are stored as SealedSecrets in git repository. Platform secres are deployed to the apl-secrets namespace. +The External Secrets Operator (ESO) is used to propagate platform secrets to the right namespace and expected format. + +### App Enablement + +Apps are toggled via `apps..enabled` in user config. Defaults are in `helmfile.d/snippets/defaults.yaml`. Some apps are always enabled (derived in `derived.gotmpl`): `argocd`, `cert-manager`, `ingress-nginx`, `istio`, `keycloak`, `sealed-secrets`. + +### Namespace Model + +Namespaces are defined in `core.yaml` under `k8s.namespaces`. Each entry can have: + +- `name` — K8s namespace name +- `app` — The app that owns this namespace (defaults to name) +- `disableIstioInjection` — Skip Istio sidecar +- `disablePolicyChecks` — Skip Kyverno checks +- `labels` — Extra labels + +Team namespaces follow the pattern `team-` and are managed by the `team-ns` chart. + +### Ingress Architecture + +Admin apps are defined in `core.yaml` under `adminApps`. Team apps under `teamApps`. Each entry configures: + +Ingress uses Gateway API (`HTTPRoute`) with Istio as the gateway implementation. +HTTPRoute binding to Gateway is set either in the `values//app.gotmpl` (if the corresponding charts/ delivers predefined routes) or oin the `values//-raw.gotmpl`. + +### Multi-Tenancy + +Teams are configured under `teamConfig.` in user values. Each team gets: + +- Dedicated namespace (`team-`) +- Network policies (default deny + platform allowlist) +- Resource quotas and limit ranges +- Optional managed monitoring (Grafana, Alertmanager, Prometheus) +- ArgoCD project + GitOps repo in Gitea +- Tekton dashboard for CI/CD +- Kyverno security policies +- Services, workloads, builds, secrets + +### Schema Validation + +All user-configurable parameters must be defined in `values-schema.yaml` (JSON Schema draft-07). The schema uses: + +- `$ref` for reusable definitions (resources, images, networking) +- `x-secret` annotation for sensitive values (triggers secret generation) +- `additionalProperties: false` to prevent typos +- Pattern validation for names, domains, resource quantities + +### Values Migration + +When schema changes between versions, `values-changes.yaml` defines migrations: + +- `deletions` — Remove deprecated keys +- `relocations` — Move keys to new paths +- `additions` — Add new keys with defaults +- `mutations` — Change existing values +- `customFunctions` — Complex migrations in TypeScript (`src/common/runtime-upgrades/`) +- `fileDeletions/fileAdditions` — File-level changes in `$ENV_DIR` + +Current schema version: **60** (see `versions.specVersion` in defaults.yaml) + +--- + +## 3. Agent Definitions + +### Agent: Network Policy Expert + +**Scope:** Create, modify, or troubleshoot Kubernetes NetworkPolicies for platform applications and teams. + +**Key Files:** +| File | Purpose | +|------|---------| +| `charts/apl-network-policies/templates/networkpolicies/*.yaml` | Platform-level network policies (per-app) | +| `charts/apl-network-policies/values.yaml` | Enable/disable flags (`netpols.: true`) | +| `values/apl-network-policies/apl-network-policies.gotmpl` | Values template passing config to chart | +| `charts/team-ns/templates/netpols/default-network-policies.yaml` | Team default policies (deny-all + platform allowlist) | +| `charts/team-ns/templates/netpols/custom-network-policies.yaml` | Team custom ingress policies (AllowAll/AllowOnly) | +| `charts/team-ns/templates/netpols/custom-istio-service-entries.yaml` | Team egress policies via Istio ServiceEntries | +| `values-schema.yaml` → `definitions.netpol` | Schema for user-defined network policies | +| `values-schema.yaml` → `definitions.appNetworkPolicyConfig` | Schema for per-app netpol toggle | +| `core.yaml` → `k8s.namespaces` | Namespace labels used in selectors | +| `helmfile.d/snippets/defaults.yaml` → `apps..networkPolicies.enabled` | Default netpol enablement per app | + +**Patterns:** + +- **Platform policies** (`apl-network-policies` chart): Each app gets a template file in `templates/networkpolicies/.yaml`, gated by `{{ if .Values.netpols. }}`. Policies use namespace selectors referencing labels from `core.yaml`. Common ingress sources: Istio gateway (via `.Values.ingressGatewaySelectors`), monitoring namespace, operators, internal namespace communication. +- **Team default policies** (`team-ns` chart): When `networkPolicy.ingressPrivate` is true, deploy deny-all + allow from platform services (istio-system, knative-serving, monitoring, tekton-pipelines, gitea). +- **Team custom policies**: Users define in `teamConfig..netpols[]` with schema `definitions.netpol`. Types: `ingress` (AllowAll from team namespaces, or AllowOnly from specific namespaces+labels) and `egress` (FQDN + port via Istio ServiceEntry). + +**How to add a network policy for a new platform app:** + +1. Create `charts/apl-network-policies/templates/networkpolicies/.yaml` +2. Gate with `{{- if .Values.netpols. }}` +3. Define podSelector, policyTypes, ingress rules using namespace/pod selectors +4. Add `netpols.` to the values template in `values/apl-network-policies/apl-network-policies.gotmpl` +5. Optionally add `apps..networkPolicies` schema entry in `values-schema.yaml` using `$ref: '#/definitions/appNetworkPolicyConfig'` +6. Set default in `helmfile.d/snippets/defaults.yaml` under `apps..networkPolicies.enabled` + +--- + +### Agent: New Application Integrator + +**Scope:** Add a new core or optional application to the APL platform. + +**Key Files:** +| File | Purpose | +|------|---------| +| `charts//` | Helm chart directory (custom or vendored) | +| `charts/skeleton/` | Template chart to copy from for custom charts | +| `values//.gotmpl` | Primary values template | +| `values//-raw.gotmpl` | Optional: raw K8s resources (RBAC, CRDs, etc.) | +| `helmfile.d/helmfile-*.yaml.gotmpl` | Helmfile release definition | +| `helmfile.d/snippets/defaults.yaml` | Default values for the app | +| `values-schema.yaml` → `properties.apps.` | User-configurable schema | +| `core.yaml` → `k8s.namespaces` | Namespace registration | +| `core.yaml` → `adminApps` or `teamApps` | Ingress configuration (if app has UI) | +| `apps.yaml` → `appsInfo.` | App metadata (version, description, links) | + +**Step-by-step procedure:** + +1. **Chart:** Place Helm chart in `charts//` (copy `charts/skeleton/` for custom, or vendor upstream) +2. **Values template:** Create `values//.gotmpl` — access platform values via `{{ $v := .Values }}`, `{{ $a := $v.apps }}` +3. **Helmfile release:** Add release in appropriate `helmfile.d/helmfile-*.yaml.gotmpl`: + ```yaml + - name: + installed: {{ $a | get ".enabled" }} + namespace: + <<: *default + ``` +4. **Defaults:** Add `apps.` section in `helmfile.d/snippets/defaults.yaml` with `enabled`, `resources`, `_rawValues: {}` +5. **Schema:** Add `apps.` under `properties.apps` in `values-schema.yaml` +6. **Namespace:** Add to `core.yaml` → `k8s.namespaces` (with appropriate flags) +7. **Ingress** (if app has web UI): Add to `core.yaml` → `adminApps` or `teamApps` +8. **Metadata:** Add to `apps.yaml` → `appsInfo.` (title, version, description, etc.) +9. **Raw artifacts** (optional): Create `values//-raw.gotmpl` and add `*raw` release + +**Placement rules for helmfile releases:** +| Phase | File | When to use | +|-------|------|-------------| +| 01-09 | `helmfile-0X.init` | Core infra that other apps depend on | +| 03 | `helmfile-03.databases` | Database releases using `*otomiDb` | +| 10 | `helmfile-10.monitoring` | Monitoring stack components | +| 15 | `helmfile-15.ingress-core` | Ingress-related core components | +| 20 | `helmfile-20.ingress` | DNS-related releases | +| 50 | `helmfile-50.services` | Optional/addon services | +| 60 | `helmfile-60.teams` | Per-team releases (iterated over teams) | +| 70 | `helmfile-70.shared` | Shared services (harbor, console, API) | +| 90-91 | `helmfile-90/91.artifacts` | Raw K8s manifest releases | + +--- + +### Agent: Ingress Expert + +**Scope:** Configure gateways, HTTPRoutes, OAuth2 authentication, and domain management. + +**Key Files:** +| File | Purpose | +|------|---------| +| `core.yaml` → `adminApps` | Deprecated | +| `core.yaml` → `teamApps` | Deprecated | +| `helmfile.d/snippets/routes.gotmpl` | HTTPRoute template (parentRefs, auth rules) | +| `helmfile.d/snippets/authpolicy-oauth2-ext.gotmpl` | OAuth2 external auth policy template | +| `helmfile.d/snippets/authpolicy-jwt.gotmpl` | JWT authentication policy template | +| `helmfile.d/snippets/serviceentry.gotmpl` | Istio ServiceEntry template for domain routing | +| `helmfile.d/snippets/derived.gotmpl` | Computed domain names, gateway names, TLS | +| `helmfile.d/snippets/domains.gotmpl` | Domain configuration helpers | +| `charts/team-ns/templates/routes.yaml` | Team service route rendering | +| `charts/team-ns/templates/ingress.yaml` | Deprecated | +| `charts/kubernetes-gateways/` | Gateway API gateway definitions | +| `charts/istio-gateway/` | Deprecated | +| `charts/ingress-nginx/` | Deprecated | +| `values/ingress-nginx/` | Deprecated| +| `values/istio-gateway/` | Deprecated | +| `values/kubernetes-gateways/` | Gateway API values | +| `values-schema.yaml` → `definitions.service` | Service/ingress schema | +| `values-schema.yaml` → `definitions.ingressClassParameters` | Ingress class config | +| `values-schema.yaml` → `properties.ingress` | Platform ingress class schema | + +**Concepts:** + +- Domains follow pattern `.` (admin) or `-.` (team) +- Derived values compute: `_derived.consoleDomain`, `_derived.giteaDomain`, `_derived.keycloakDomain`, etc. +- Gateway API `HTTPRoute` resources route to backend services +- OAuth2 proxy handles authentication (`auth: true` in core.yaml) +- Istio `ServiceEntry` resources make domains resolvable from within the mesh +- `ingress.platformClass` configures the main load balancer (IP, autoscaling, resources) +- Additional ingress classes via `ingress.classes[]` + +--- + +### Agent: Schema Manager + +**Scope:** Modify `values-schema.yaml` to add, change, or deprecate user-configurable parameters. + +**Key Files:** +| File | Purpose | +|------|---------| +| `values-schema.yaml` | THE schema file (JSON Schema draft-07 in YAML) | +| `helmfile.d/snippets/defaults.yaml` | Defaults that MUST match schema | +| `values-changes.yaml` | Migration definitions when schema changes | +| `src/common/runtime-upgrades/` | Custom migration functions | +| `src/cmd/validate-values.ts` | Schema validation command | +| `src/cmd/migrate.ts` | Migration execution | + +**Rules:** + +- Every user-configurable parameter MUST have a schema entry +- Use `$ref` to reference reusable definitions (`resources`, `image`, `idName`, etc.) +- Mark secrets with `x-secret: ''` (or `x-secret: '{{ randAlphaNum N }}'` for auto-generation) +- Use `additionalProperties: false` on app schemas to catch typos +- Always include `_rawValues: { $ref: '#/definitions/rawValues' }` for escape-hatch overrides +- When removing/renaming keys, add a migration in `values-changes.yaml` and bump `versions.specVersion` + +**Common schema patterns:** + +```yaml +# App with resources + enabled flag +apps: + myapp: + additionalProperties: false + properties: + _rawValues: + $ref: '#/definitions/rawValues' + enabled: + type: boolean + default: false + resources: + additionalProperties: false + properties: + main: + $ref: '#/definitions/resources' + sidecar: + $ref: '#/definitions/resources' +``` + +**Reusable definitions (most common):** +| Definition | Purpose | +|------------|---------| +| `resources` | CPU/memory requests+limits | +| `rawValues` | define collection of raw Kubernetes manifests | +| `image` / `imageSimple` | Container image config | +| `idName` | Lowercase DNS-safe name pattern | +| `domain` | Domain pattern | +| `autoscaling` / `autoscalingEnabled` | HPA config | +| `service` | Team service definition | +| `netpol` | Network policy definition | +| `workload` | ArgoCD workload definition | +| `build` | Tekton build definition | +| `secret` | External secret definition | +| `appNetworkPolicyConfig` | Per-app netpol toggle | + +--- + +### Agent: Team Configuration Manager + +**Scope:** Manage team namespaces, RBAC, resource quotas, security policies, and team-level services. + +**Key Files:** +| File | Purpose | +|------|---------| +| `charts/team-ns/` | THE team namespace chart | +| `charts/team-ns/values.yaml` | Team chart values structure | +| `charts/team-ns/templates/rbac.yaml` | Service accounts, roles, role bindings | +| `charts/team-ns/templates/limitrange.yaml` | Default resource limits for containers | +| `charts/team-ns/templates/quota.yaml` | Resource quotas (pods, load balancers) | +| `charts/team-ns/templates/netpols/` | Network policies (default + custom) | +| `charts/team-ns/templates/policies/` | Kyverno security policies | +| `charts/team-ns/templates/argocd/` | ArgoCD application + project templates | +| `charts/team-ns/templates/builds/` | Tekton build configurations (Docker, Buildpacks) | +| `charts/team-ns/templates/tekton-tasks/` | Tekton tasks (kaniko, git-clone, grype, buildpacks) | +| `charts/team-ns/templates/routes.yaml` | Service routing | +| `charts/team-ns/templates/ingress.yaml` | Ingress config | +| `charts/team-ns/templates/_helpers.tpl` | Helper templates | +| `values/team-ns/team-ns.gotmpl` | Values template for team-ns | +| `helmfile.d/helmfile-15.ingress-core.yaml.gotmpl` | Admin team namespace release | +| `helmfile.d/helmfile-60.teams.yaml.gotmpl` | Per-team releases (iterates over `teamConfig`) | +| `values-schema.yaml` → `definitions.team` | Full team configuration schema | +| `values-schema.yaml` → `definitions.teamSelfService` | Team self-service permissions | + +**Team iteration pattern (helmfile-60):** + +```gotmpl +{{- range $teamId, $team := omit $tc "admin" }} + - name: team-ns-{{ $teamId }} + installed: true + namespace: team-{{ $teamId }} + chart: ../charts/team-ns + values: + - ../values/team-ns/team-ns.gotmpl +{{- end }} +``` + +**Per-team releases deployed:** `tekton-dashboard-`, `prometheus-`, `grafana-dashboards-`, `team-ns-`, `team-secrets-`, `prometheus-msteams-` (if msteams alerts enabled). + +--- + +### Agent: Values Template Author + +**Scope:** Create or modify Go template value files in `values/`. + +**Key Files:** +| File | Purpose | +|------|---------| +| `values//.gotmpl` | Primary values template for each app | +| `values//-raw.gotmpl` | Raw K8s manifests template | +| `values/jobs/.gotmpl` | Job values templates | +| `helmfile.d/snippets/common.gotmpl` | Common values included in most releases | +| `helmfile.d/snippets/templates.gotmpl` | How values files are loaded per anchor type | + +**Template conventions:** + +```gotmpl +{{- $v := .Values }} # All merged values +{{- $a := $v.apps }} # All app configs +{{- $app := $a.myapp }} # Specific app config +{{- $tc := $v.teamConfig }} # Team configurations +{{- $d := $v._derived }} # Derived/computed values +{{- $provider := $v.cluster.provider }} # Cloud provider +{{- $domain := $v.cluster.domainSuffix }}# Cluster domain +``` + +**Accessing common derived values:** + +- `$v._derived.oidcBaseUrl` — Keycloak OIDC URL +- `$v._derived.untrustedCA` — Whether CA is untrusted +- `$v._derived.tlsSecretName` — TLS wildcard cert secret name +- `$v._derived.consoleDomain`, `giteaDomain`, `keycloakDomain`, etc. +- `$v._derived.ingressPublicGatewayName` — Istio ingress gateway name + +**Raw template pattern (`*-raw.gotmpl`):** + +```gotmpl +resources: + - apiVersion: v1 + kind: ConfigMap + metadata: + name: my-config + data: + key: value +``` + +--- + +### Agent: Helmfile Release Manager + +**Scope:** Add, modify, or reorder helmfile releases. + +**Key Files:** +| File | Purpose | +|------|---------| +| `helmfile.d/helmfile-*.yaml.gotmpl` | All release definitions (see placement table above) | +| `helmfile.d/snippets/templates.gotmpl` | Release anchor definitions | +| `helmfile.d/snippets/defaults.yaml` | Default values loaded by all specs | +| `helmfile.d/snippets/env.gotmpl` | User environment values | +| `helmfile.d/snippets/derived.gotmpl` | Computed values | + +**Adding a release:** + +```yaml +releases: + - name: my-app # Must match chart dir and values dir + installed: {{ $a | get "my-app.enabled" }} # Toggle via user config + namespace: my-namespace # Target K8s namespace + labels: # Optional labels for filtering + pkg: my-app + <<: *default # Use standard deployment anchor +``` + +**Release with raw artifacts:** + +```yaml + - name: my-app-artifacts + installed: {{ $a | get "my-app.enabled" }} + namespace: my-namespace + <<: *raw +``` + +**Release with labels for `otomi apply -l name=my-app`:** + +```yaml +labels: + tag: teams # Group label + team: { { $teamId } } # Team-specific label + pipeline: otomi-task-teams # Pipeline label +``` + +--- + +### Agent: TypeScript CLI Developer + +**Scope:** Work on CLI commands, operators, and TypeScript utility code. + +**Key Files:** +| File | Purpose | +|------|---------| +| `src/otomi.ts` | CLI entrypoint (yargs-based) | +| `src/cmd/index.ts` | Command registry | +| `src/cmd/*.ts` | Individual CLI commands | +| `src/common/hf.ts` | Helmfile wrapper (invokes helmfile with proper args) | +| `src/common/values.ts` | Values loading/merging logic | +| `src/common/k8s.ts` | Kubernetes API interactions | +| `src/common/bootstrap.ts` | Bootstrap logic (initial setup) | +| `src/common/utils.ts` | Shared utilities | +| `src/common/cli.ts` | CLI helper utilities | +| `src/common/constants.ts` | Constants and paths | +| `src/common/zx-enhance.ts` | Enhanced zx (Google's shell scripting lib) | +| `src/common/runtime-upgrade.ts` | Migration runner | +| `src/common/runtime-upgrades/` | Custom migration functions | +| `src/operator/` | APL Operator (watches CRDs, runs installs) | +| `tsconfig.json` / `tsconfig.build.json` | TypeScript config | +| `jest.config.ts` | Test config | +| `eslint.config.mjs` | Linting config | +| `babel.config.js` | Babel config | + +**Key CLI commands:** +| Command | Source | Purpose | +|---------|--------|---------| +| `otomi apply` | `src/cmd/apply.ts` | Deploy charts via helmfile | +| `otomi bootstrap` | `src/cmd/bootstrap.ts` | Initialize values repo | +| `otomi install` | `src/cmd/install.ts` | Full cluster installation | +| `otomi validate-values` | `src/cmd/validate-values.ts` | Validate user values against schema | +| `otomi validate-templates` | `src/cmd/validate-templates.ts` | Validate rendered K8s manifests | +| `otomi diff` | `src/cmd/diff.ts` | Show diff before applying | +| `otomi migrate` | `src/cmd/migrate.ts` | Run value migrations | +| `otomi values` | `src/cmd/values.ts` | Render merged values | +| `otomi x` | `src/cmd/x.ts` | Execute arbitrary helmfile commands | + +**Test patterns:** Tests are co-located as `*.test.ts` files. Use Jest with `npm test`. Fixtures in `tests/fixtures/`. + +**Dev setup:** + +```bash +export IN_DOCKER=false +export ENV_DIR=$PWD/tests/fixtures +export NODE_ENV=test +npm run compile +npm test +``` + +--- + +### Agent: Migration Author + +**Scope:** Define value migrations when schema changes between versions. + +**Key Files:** +| File | Purpose | +|------|---------| +| `values-changes.yaml` | Migration definitions (version → version) | +| `src/common/runtime-upgrade.ts` | Migration runner | +| `src/common/runtime-upgrades/` | Custom migration TypeScript functions | +| `src/cmd/migrate.ts` | Migration CLI command | +| `helmfile.d/snippets/defaults.yaml` → `versions.specVersion` | Current schema version | + +**Migration types:** + +```yaml +changes: + - version: 61 # Next version number + deletions: + - 'apps.old-app' # Remove deprecated keys + relocations: + - 'apps.foo.old': 'apps.foo.new' # Move keys + additions: + - apps.new-app.enabled: false # Add with default + mutations: + - apps.foo.bar: 'newValue' # Change values + customFunctions: + - myMigrationFunction # TypeScript function in runtime-upgrades/ + fileDeletions: + - env/teams/{team}/old-file.yaml + fileAdditions: + - env/teams/new-file.yaml +``` + +**After adding a migration:** Bump `versions.specVersion` in `helmfile.d/snippets/defaults.yaml`. + +--- + +### Agent: Monitoring Stack Expert + +**Scope:** Configure Prometheus, Grafana, Alertmanager, Loki, and OpenTelemetry. + +**Key Files:** +| File | Purpose | +|------|---------| +| `helmfile.d/helmfile-10.monitoring.yaml.gotmpl` | Monitoring stack releases | +| `helmfile.d/helmfile-60.teams.yaml.gotmpl` | Per-team Prometheus/Grafana/Alertmanager | +| `charts/kube-prometheus-stack/` | Prometheus operator chart | +| `charts/grafana-dashboards/` | Grafana dashboard definitions | +| `charts/loki/` | Loki log aggregation | +| `charts/otel-operator/` | OpenTelemetry operator | +| `charts/promtail/` | Promtail log shipper | +| `charts/prometheus-blackbox-exporter/` | Blackbox exporter | +| `charts/prometheus-msteams/` | MS Teams alerting bridge | +| `values/prometheus-operator/` | Prometheus operator values | +| `values/loki/` | Loki values | +| `values/otel-operator/` | OTel values | +| `values/grafana-dashboards/` | Dashboard values | +| `helmfile.d/snippets/alertmanager.gotmpl` | Alertmanager config template | +| `helmfile.d/snippets/alertmanager-teams.gotmpl` | Per-team alertmanager config | +| `helmfile.d/snippets/alertmanager/slack.gotmpl` | Slack integration | +| `helmfile.d/snippets/alertmanager/opsgenie.gotmpl` | Opsgenie integration | +| `helmfile.d/snippets/blackbox-targets.gotmpl` | Blackbox probe targets | +| `values-schema.yaml` → `definitions.alerts` | Alert configuration schema | +| `values-schema.yaml` → `properties.apps.prometheus` | Prometheus schema | +| `values-schema.yaml` → `properties.apps.loki` | Loki schema | + +**Team monitoring pattern:** + +- Each team can have managed monitoring (`teamConfig..settings.managedMonitoring.grafana/alertmanager`) +- Per-team Prometheus instance scrapes team namespace +- Per-team Grafana with OIDC auth + team-scoped datasources +- Per-team Alertmanager with configurable receivers (slack, msteams, opsgenie) +- Dashboards auto-provisioned from `grafana-dashboards` chart + +--- + +### Agent: Database Manager + +**Scope:** Manage CloudNativePG PostgreSQL databases for platform applications. + +**Key Files:** +| File | Purpose | +|------|---------| +| `helmfile.d/helmfile-03.databases.yaml.gotmpl` | Database releases | +| `charts/otomi-db/` | Database chart (wraps CloudNativePG) | +| `charts/cloudnative-pg/` | CloudNativePG operator chart | +| `charts/cloudnative-pg-plugin-barman-cloud/` | Backup plugin chart | +| `helmfile.d/snippets/defaults.yaml` → `databases` | Database defaults (keycloak, harbor, gitea) | +| `values-schema.yaml` → `properties.databases` (if exists) | Database schema | +| `values-schema.yaml` → `properties.platformBackups.database` | Backup configuration | + +**Platform databases:** keycloak, harbor, gitea — each has configurable replicas, storage size, resources, PostgreSQL parameters, and backup settings. + +--- + +## 4. Common Go Template Helpers + +Used across values templates: + +| Expression | Purpose | +| -------------------------------------- | ----------------------------------- | +| `$v.cluster.domainSuffix` | Cluster base domain | +| `$v.cluster.provider` | Cloud provider (`linode`, `custom`) | +| `$a \| get "app.key" defaultValue` | Safe key access with fallback | +| `$v \| dig "deep" "path" defaultValue` | Deep path access | +| `hasKey $dict "key"` | Check if key exists | +| `$v.otomi.isMultitenant` | Multi-tenancy flag | +| `$v._derived.*` | All computed values | +| `tpl (readFile "path") $v` | Render a sub-template with values | +| `toYaml \| nindent N` | YAML serialization with indentation | + +--- + +## 5. Testing & Validation + +| Command | Purpose | +| ------------------------------------------- | ------------------------------------- | +| `npm test` | Run Jest unit tests | +| `otomi validate-values` | Validate user config against schema | +| `otomi validate-templates [-l name=app]` | Validate rendered K8s manifests | +| `otomi diff [-l name=app]` | Preview changes before apply | +| `otomi x helmfile -l name=app template` | Render chart templates for inspection | +| `otomi x helmfile -l name=app write-values` | Render values for inspection | +| `npm run test:opa` | Run OPA/Rego policy tests | + +--- + +## 6. Conventions & Gotchas + +- **Helmfile labels:** Use `-l name=myapp` to select releases, not `-l app=myapp` +- **Raw values override:** `apps.._rawValues` overrides chart values not in schema (use sparingly) +- **YAML anchors:** Search for `&anchorname` when you see `<<: *anchorname` +- **Keycloak OIDC:** Use `_derived.oidcBaseUrl`, `apps.keycloak.idp.clientID/clientSecret` +- **Untrusted CA:** Check `_derived.untrustedCA` to conditionally disable cert verification +- **Chart naming:** Release name = chart directory name = values directory name (unless overridden in anchor) +- **App naming in schema vs defaults:** Some apps have different chartName in `apps.yaml` (e.g., `cnpg` → `cloudnative-pg`, `otel` → `otel-operator`, `trivy` → `trivy-operator`) +- **Team admin is special:** `team-admin` namespace is deployed in helmfile-15 (not helmfile-60) and has `networkPolicy: null`, `resourceQuota: null` +- **Alphabetical execution:** Helmfile specs run 01→91. Dependencies must be in earlier-numbered files. +- **.gitignore is minimal:** Only ignores `node_modules`, `package.json`, `bun.lock`, `.gitignore` itself. Most files ARE tracked. diff --git a/charts/AGENTS.md b/charts/AGENTS.md new file mode 100644 index 0000000000..57281c3611 --- /dev/null +++ b/charts/AGENTS.md @@ -0,0 +1,51 @@ +# charts/ — Helm Charts + +## OVERVIEW +Contains APL's custom infrastructure charts and 44 vendored upstream charts. +This directory provides the building blocks for the APL platform. + +## CUSTOM CHARTS +| Chart | Purpose | Complexity | +|-------|---------|------------| +| apl-operator/ | Core platform operator logic | Medium | +| apl-gitea-operator/ | Gitea lifecycle and configuration | Low | +| apl-harbor-operator/ | Harbor lifecycle and configuration | Low | +| apl-keycloak-operator/| Keycloak lifecycle and configuration | Low | +| apl-network-policies/ | Platform-wide default network policies | Medium | +| team-ns/ | Team namespace engine (RBAC, quotas, builds, ArgoCD) | HIGH | +| raw/ | Deploy arbitrary K8s manifests (ConfigMaps, etc.) | Low | +| raw-cr/ | Deploy arbitrary custom resources (CRs) | Low | +| jobs/ | Reusable templates for Kubernetes Jobs/CronJobs | Low | +| skeleton/ | Template chart for creating new custom apps | Starter | + +## VENDORED CHARTS +Upstream charts are mirrored here to ensure stability and local modification capability. +- **Core Platform:** argocd, cert-manager, istio-base, istio-gateway, istiod, keycloak, sealed-secrets, ingress-nginx +- **Monitoring:** kube-prometheus-stack, grafana-dashboards, loki, otel-operator, promtail, prometheus-blackbox-exporter, prometheus-msteams +- **CI/CD:** tekton-pipelines, tekton-triggers, tekton-dashboard +- **Storage/DB:** cloudnative-pg, cloudnative-pg-plugin-barman-cloud, harbor, gitea, valkey, rabbitmq +- **Security:** kyverno, trivy-operator, policy-reporter, oauth2-proxy, external-secrets +- **DNS/Ingress:** external-dns, cert-manager-webhook-linode, kubernetes-gateways +- **ML/Serverless:** knative-operator, kserve, kubeflow-pipelines +- **APL Services:** otomi-api, otomi-console, otomi-operator, otomi-db, linode-cfw +- **Utilities:** base, metrics-server, argocd-image-updater, wait-for + +## PATTERNS +- **Structure:** Every chart follows the standard Helm layout (Chart.yaml, values.yaml, templates/). +- **Helpers:** Critical naming and labeling logic resides in `templates/_helpers.tpl`. +- **Feature Toggles:** Templates use `{{- if }}` for logic (e.g., ingress, RBAC, persistence). +- **Static Defaults:** Chart `values.yaml` holds base defaults for the chart itself. +- **Dynamic Input:** Configuration is injected from `values/*.gotmpl` during Helmfile execution. +- **Escape Hatches:** Use `_rawValues` in user config to override chart values not in schema. +- **Skeleton:** Always use `charts/skeleton/` as the baseline for new custom charts. +- **Labels:** Consistently use `app.kubernetes.io/managed-by: otomi` and APL-specific labels. +- **Selectors:** Cross-app communication relies on stable labels defined in `_helpers.tpl`. + +## WHERE TO LOOK +- **Logic:** `templates/` — Examine for APL-specific logic, annotations, and labels. +- **Dependencies:** `Chart.yaml` — Check for upstream versions and sub-charts. +- **Default Values:** `values.yaml` — View the base chart configuration. +- **Multi-Tenancy:** `charts/team-ns/` — The primary engine for tenant isolation. +- **Manifest Injection:** `charts/raw/` or `charts/raw-cr/` for non-standard resources. +- **Job Templates:** `charts/jobs/` for platform maintenance or migration tasks. +- **Network Isolation:** `charts/apl-network-policies/` for cluster-wide restrictions. diff --git a/charts/team-ns/AGENTS.md b/charts/team-ns/AGENTS.md new file mode 100644 index 0000000000..e470ded6df --- /dev/null +++ b/charts/team-ns/AGENTS.md @@ -0,0 +1,43 @@ +# charts/team-ns/ — Team Namespace Chart + +## OVERVIEW +The core multi-tenancy engine of APL. Provisions isolated team environments with RBAC, quotas, +networking, security policies, and CI/CD (ArgoCD + Tekton) integration. + +## TEMPLATE STRUCTURE +- `templates/` + - `_helpers.tpl` — Label logic, domain math, Docker/volume config generation. + - `rbac.yaml` — Massive policy (SAs: team, kubectl, tekton; RoleBindings). + - `routes.yaml` — Gateway API `HTTPRoute` for team services. + - `ingress.yaml` — Legacy/Standard Ingress resources. (Deprecated) + - `limitrange.yaml` — Container resource defaults. + - `quota.yaml` — Team resource constraints (skipped for `team-admin`). + - `argocd/` — Team Apps/Projects for GitOps lifecycle. + - `builds/` — Tekton pipeline/build specs (Docker/Buildpacks). + - `netpols/` — Ingress/Egress isolation (Platform allowlist + Custom). + - `policies/` — Kyverno PSS (Baseline/Restricted) enforcement. + - `tekton-tasks/` — Reusable build steps (Kaniko, Grype, GitClone). + - `telemetry/` — Istio/OTel instrumentation. + +## KEY TEMPLATES +- `rbac.yaml`: Manages complex multi-identity RBAC for team users and automation. +- `netpols/default-network-policies.yaml`: Implements platform isolation (deny-all + core allow). +- `argocd/argocd-application-workload.yaml`: Bridges team config to ArgoCD deployments. + +## VALUE SOURCES +- Primary: `values/team-ns/team-ns.gotmpl` (Injected via Helmfile). +- Global: `helmfile.d/snippets/defaults.yaml`. +- Context: Iterated via `helmfile-60.teams.yaml.gotmpl` (except `team-admin`). + +## PATTERNS +- **Identity naming**: Resources suffix/prefix with `{{ $v.teamId }}`. +- **Namespace isolation**: Always targets `team-{{ $v.teamId }}`. +- **Labeling**: Every resource MUST have `otomi.io/team: {{ $v.teamId }}`. +- **Security**: Aggregates secrets for Gitea/Harbor/Internal-Registry. +- **Builds**: Standardized on Tekton with Kaniko for Docker or Buildpacks for source. + +## ANTI-PATTERNS +- **Hardcoded IDs**: Never use static team names; always use `teamId` from values. +- **Missing labels**: Resources without `otomi.io/team` will break platform logic. +- **Manual NS creation**: Let this chart handle namespace-scoped resource lifecycle. +- **Bypassing NetPols**: Adding global allows outside `custom-network-policies.yaml`. diff --git a/helmfile.d/AGENTS.md b/helmfile.d/AGENTS.md new file mode 100644 index 0000000000..31336110dc --- /dev/null +++ b/helmfile.d/AGENTS.md @@ -0,0 +1,46 @@ +# helmfile.d/ — Helmfile Release Definitions + +## OVERVIEW +Orchestrates 30+ cloud-native apps via ordered Helmfile specs using 3-stage value merge (defaults → user → derived). + +## EXECUTION ORDER +Specs execute alphabetically. Dependencies MUST be in earlier-numbered files. +| File | Phase | Key Releases | +|------|-------|--------------| +| `01-09.init` | Core Infra | Kyverno, Cert-Manager, Keycloak, External-Secrets | +| `03.databases`| DBs | CloudNativePG (Gitea, Keycloak, Harbor) using `*otomiDb` | +| `03.init` | Core Infra | Components requiring post-DB setup | +| `10.monitoring`| Observability | Prometheus, Grafana, Loki, OTel, Alertmanager | +| `15.ingress-core`| Ingress | Ingress-Nginx, Istio-Base, Admin Team Namespace | +| `20.ingress` | DNS | External-DNS | +| `50.services` | Optional | Knative, Kubeflow, Trivy, Kserve | +| `60.teams` | Per-Team | Tekton, Team-Prometheus/Grafana/Secrets (iterated) | +| `70.shared` | Platform | Harbor, OAuth2-Proxy, Otomi-API, Otomi-Console | +| `90-91.artifacts`| Manifests | Raw K8s artifacts for Istio and OTel | + +## RELEASE PATTERNS +Standardized loading via anchors in `snippets/templates.gotmpl`: +- `<<: *default`: Standard chart (uses `values//.gotmpl`) +- `<<: *raw`: Additional K8s manifests (uses `values//-raw.gotmpl`) +- `<<: *rawCR`: Custom Resources using `raw-cr` chart +- `<<: *jobs`: Maintenance jobs (uses `values/jobs/.gotmpl`) +- `<<: *otomiDb`: Database releases wrapping CloudNativePG + +## WHERE TO ADD NEW RELEASES +- **Core Infra:** `01-09.init` (ensure alphabetical ordering for dependencies) +- **DB-backed apps:** `03.databases` for the DB, then later for the app +- **Add-on services:** `50.services` +- **Team-scoped resources:** `60.teams` (requires iteration over `teamConfig`) + +## ANTI-PATTERNS +- **Ordering errors:** Placing a dependency in a higher-numbered file than its consumer +- **Hardcoding values:** Use `.Values` or `snippets/derived.gotmpl` instead +- **Direct $ENV_DIR access:** ALWAYS use `snippets/env.gotmpl` base +- **Skipping anchors:** Standard releases MUST use `*default`, `*raw`, etc. +- **Duplicate Prefixes:** `03.databases` vs `03.init` — remember alphabetical full-filename sorting +- **Manual Values Merge:** Use `snippets/common.gotmpl` within chart values for shared state +- **Recursive Bases:** Do NOT include bases that loop back to snippets + +## DIRECTORY STRUCTURE +- `snippets/`: Reusable Go templates, defaults, and derived values. +- `utils/`: Helper scripts for helmfile processing and manifest generation. diff --git a/helmfile.d/snippets/AGENTS.md b/helmfile.d/snippets/AGENTS.md new file mode 100644 index 0000000000..96baddec10 --- /dev/null +++ b/helmfile.d/snippets/AGENTS.md @@ -0,0 +1,50 @@ +# helmfile.d/snippets/ — Core Templates & Values + +## OVERVIEW +Core configuration layer defining the platform's 3-stage values merge and release templates. + +## FILE INVENTORY +| File | Purpose | +|------|---------| +| `defaults.yaml` | Stage 1: Static defaults for ALL apps (1203 lines). | +| `env.gotmpl` | Stage 2: Loads user values from `$ENV_DIR/env/**/*.yaml`. | +| `derived.gotmpl` | Stage 3: Computes `_derived.*` values (URLs, certs, gateways). | +| `templates.gotmpl` | Defines release anchors (`*default`, `*raw`, `*rawCR`, `*jobs`). | +| `common.gotmpl` | Shared values for releases (pull secrets, node selectors). | +| `routes.gotmpl` | HTTPRoute template for Gateway API routing and auth. | +| `domains.gotmpl` | Domain configuration and normalization helpers. | +| `authpolicy-jwt.gotmpl` | JWT authentication policy template. | +| `authpolicy-oauth2-ext.gotmpl` | OAuth2 external auth policy template. | +| `serviceentry.gotmpl` | Istio ServiceEntry for internal-to-external domain routing. | +| `alertmanager.gotmpl` | Platform-level Alertmanager configuration. | +| `alertmanager-teams.gotmpl` | Per-team Alertmanager configuration. | +| `alertmanager/opsgenie.gotmpl` | Opsgenie integration template for Alertmanager. | +| `alertmanager/slack.gotmpl` | Slack integration template for Alertmanager. | +| `blackbox-targets.gotmpl` | Targets for Prometheus blackbox prober. | +| `defaults.gotmpl` | Helper for loading default values in Go templates. | +| `grafana.gotmpl` | Grafana-specific configuration snippets. | +| `sops-env.gotmpl` | SOPS-encrypted environment variable loader. | +| `env.old.gotmpl` | Legacy environment loading (DEPRECATED). | +| `version-tags.gotmpl` | Component version mapping for platform services. | +| `provider-engine-map.gotmpl` | Maps cloud providers to specific engines. | +| `dockercfg.gotmpl` | Docker registry credential configuration. | +| `helmfile-utils.gotmpl` | Common Go template utilities for Helmfile specs. | + +## CRITICAL FILES +1. **`defaults.yaml`**: The source of truth for platform defaults. EVERY new app must register its default state, resources, and configuration here. It is the largest and most foundational file in the snippets directory, defining the initial state for the 3-stage merge. +2. **`derived.gotmpl`**: Computes complex values from raw user input and defaults. Crucial for understanding how OIDC URLs, TLS secrets, and gateway names are formed. It acts as the "logical" layer that transforms user intent into platform-specific configuration. +3. **`templates.gotmpl`**: Central registry of release patterns. Modifying an anchor here affects EVERY helmfile release that references it. It provides the standard `*default`, `*raw`, and `*jobs` building blocks used across all `helmfile-*.yaml` specs. + +## TEMPLATE HELPERS +- **Routing**: `routes.gotmpl` handles the mapping of hostnames to services with integrated Istio AuthPolicy support for Gateway API. +- **Domains**: `domains.gotmpl` provides functions to compute admin and team application domains consistently across the platform using cluster-level settings. +- **Monitoring**: `alertmanager/*.gotmpl` contains integration-specific templates for Slack and Opsgenie, while `blackbox-targets.gotmpl` manages probe targets for health monitoring. +- **Versions**: `version-tags.gotmpl` ensures consistent component versioning across all platform services and Helm charts. +- **Utility**: `helmfile-utils.gotmpl` contains generic helpers for string manipulation and template rendering. + +## ANTI-PATTERNS +- **Hardcoding**: Never hardcode values that belong in `defaults.yaml` or `derived.gotmpl`. +- **Direct Snippet Writes**: AI agents should never write temporary data to this directory. +- **Circular Logic**: Avoid cross-referencing between snippets that causes template recursion. +- **Stage Violation**: Don't put logic in `env.gotmpl` that belongs in `derived.gotmpl`. +- **Legacy Use**: Avoid modifying or using `env.old.gotmpl`; use the modern `env.gotmpl` instead. diff --git a/src/AGENTS.md b/src/AGENTS.md new file mode 100644 index 0000000000..37e08602b2 --- /dev/null +++ b/src/AGENTS.md @@ -0,0 +1,48 @@ +# src/ — TypeScript Source + +## OVERVIEW + +Core logic for APL CLI and GitOps Operator. Handles values merging, +Kubernetes orchestration, and Helmfile execution. + +## STRUCTURE + +- `cmd/`: Yargs command implementations (`{command, describe, builder, handler}`). +- `common/`: Core utilities (k8s, helmfile, values, git, crypto). +- `operator/`: GitOps reconciliation loop (watches Git repo, not CRDs). +- `otomi.ts`: Main entrypoint; registers commands from `cmd/index.ts`. +- `test-init.ts`: Jest global setup (mocks, console silencing). +- `stubs/`: Mock implementations for yargs and uuid used in tests. + +## WHERE TO LOOK + +- **Adding Commands**: New file in `src/cmd/`, register in `src/cmd/index.ts`. +- **K8s Interactions**: `src/common/k8s.ts` (1k+ lines, use caution). +- **Values/Migrations**: `src/common/values.ts` and `src/cmd/migrate.ts` (1.4k+ lines). +- **Git Operations**: `src/common/repo.ts` (700+ lines). +- **Helmfile Wrapper**: `src/common/hf.ts`. + +## CONVENTIONS + +- **Tests**: Co-locate `*.test.ts`. Use `ts-jest` and `test-init.ts`. +- **CLI**: Commands must export `CommandModule`. +- **Style**: No semicolons, single quotes, 120 char width. +- **TS**: Strict mode, ES2022, NodeNext resolution. +- **Lint**: No `++` (use `+= 1`), no `require`, no `param-reassign`. +- **Async**: Await all promises; `no-floating-promises` is enforced. + +## ANTI-PATTERNS + +- **God Files**: Do not grow `k8s.ts` or `migrate.ts`; extract logic to new modules. +- **CommonJS**: Never use `module.exports` or `require()`. +- **Mutation**: Avoid reassigning function parameters (`no-param-reassign`). +- **Mocks**: Avoid inline `jest.mock`; use `test-init.ts` or co-located mocks. + +## DEV SETUP + +```bash +export IN_DOCKER=false +export ENV_DIR=$PWD/tests/fixtures +export NODE_ENV=test +npm run compile && npm test +``` diff --git a/src/cmd/AGENTS.md b/src/cmd/AGENTS.md new file mode 100644 index 0000000000..91bba2b8d5 --- /dev/null +++ b/src/cmd/AGENTS.md @@ -0,0 +1,55 @@ +# src/cmd/ — CLI Commands + +OVERVIEW: Implementation of all `otomi` CLI commands using Yargs. + +| Command | File | Purpose | +| ------------------ | --------------------- | -------------------------------------------- | +| apply | apply.ts | Deploy charts via helmfile | +| apply-as-apps | apply-as-apps.ts | Deploy via ArgoCD (ArgoCD integration) | +| apply-teams | apply-teams.ts | Apply team-specific resources | +| bash | bash.ts | Interactive bash in container | +| bootstrap | bootstrap.ts | Initialize values repo | +| collect | collect.ts | Cluster diagnostics | +| commit | commit.ts | Validate + commit + push values | +| decrypt | decrypt.ts | Decrypt secrets files | +| destroy | destroy.ts | Destroy k8s resources | +| diff | diff.ts | Show diff before applying | +| encrypt | encrypt.ts | Encrypt secrets files | +| files | files.ts | Show values repo files | +| hf | hf.ts | Direct helmfile execution | +| install | install.ts | Full cluster installation | +| lint | lint.ts | Lint manifests via helmfile | +| migrate | migrate.ts | Migrate values between versions (1.4k lines) | +| playground | playground.ts | Dev playground | +| pull | pull.ts | Git pull + bootstrap | +| score-templates | score-templates.ts | Score template quality | +| server | server.ts | Server mode | +| status | status.ts | Show cluster status | +| sync | sync.ts | Sync k8s resources | +| template | template.ts | Export k8s resource templates | +| test | test.ts | Run cluster tests | +| traces | traces.ts | Collect failed resource traces | +| validate-cluster | validate-cluster.ts | Validate k8s version support | +| validate-templates | validate-templates.ts | Validate rendered manifests | +| validate-values | validate-values.ts | Validate user config against schema | +| values | values.ts | Render merged values | +| x | x.ts | Arbitrary helmfile execution | + +PATTERNS: + +- Export `CommandModule`: `{ command, describe, builder, handler }` +- Command registration: All files exported as array in `index.ts` +- Shared utilities: Call `../common/` for `hf.ts`, `values.ts`, `k8s.ts` +- Filtering: Use `-l/--label` for helmfile label selection + +WHERE TO LOOK: + +- `index.ts`: Command registry +- `migrate.ts`: Complex migration logic + functions +- `apply-as-apps.ts`: ArgoCD logic + +ANTI-PATTERNS: + +- Business logic in `handler`: Move to `../common/` +- Direct shell calls: Use `../common/zx-enhance.ts` +- Manual value merging: Use `../common/values.ts:getValues()` diff --git a/src/common/AGENTS.md b/src/common/AGENTS.md new file mode 100644 index 0000000000..b5cc3a5b8d --- /dev/null +++ b/src/common/AGENTS.md @@ -0,0 +1,37 @@ +# src/common/ — Shared Utilities + +## OVERVIEW + +Core logic for values orchestration, K8s/Helm wrappers, and platform state management used by CLI and Operator. + +## MODULE INVENTORY + +| Module | Purpose | +| -------------------- | ----------------------------------------------------------------- | +| `values.ts` | **PLATFORM ENGINE.** Merging, secrets, K8s detection, image tags. | +| `k8s.ts` | **GOD FILE.** K8s API client (Secrets, Apps, Helm, Exec, Patch). | +| `hf.ts` | Helmfile wrapper (`hf()`, `hfValues()`, `hfTemplate()`). | +| `repo.ts` | Values repository & team configuration file management. | +| `sealed-secrets.ts` | Encryption and manifest generation for Sealed Secrets. | +| `bootstrap.ts` | Initial environment/values repository setup logic. | +| `constants.ts` | File paths, environment variables, and platform constants. | +| `zx-enhance.ts` | Enhanced `zx` shell execution with robust error handling. | +| `runtime-upgrade.ts` | Migration runner for schema version upgrades. | +| `git-config.ts` | Git identity and authentication management. | +| `utils.ts` | Shared primitives (retry, sleep, object parsing). | + +## KEY MODULES & PATTERNS + +- **values.ts:** Central orchestrator for the 3-stage merge (Defaults -> User -> Derived). +- **k8s.ts:** Directly interacts with `@kubernetes/client-node`. Handles complex platform state like ArgoCD App reconciliation. +- **hf.ts:** Abstracts `helmfile` execution. Ensure `$ENV_DIR` is set before calling. +- **Dependency Flow:** `cmd/` & `operator/` -> `common/`. `common/` MUST NOT import from callers. +- **Constants:** Always use `constants.ts` for paths instead of hardcoded strings. + +## ANTI-PATTERNS + +- **Bootstrap Guard:** Never call `values.ts` functions within `bootstrap.ts` to avoid circular logic during init. +- **HF Naming:** `hf.ts` contains `withWorkloadValues`; treat as `withFiles` (pending rename). +- **Repo Workarounds:** `repo.ts` contains legacy "workloadValues" logic; avoid extending. +- **Async Safety:** Watch for `no-floating-promises` in entrypoints; ensure all `hf` calls are awaited. +- **Mutation:** Avoid `no-param-reassign` patterns found in `k8s.ts` when adding new methods. diff --git a/src/operator/AGENTS.md b/src/operator/AGENTS.md new file mode 100644 index 0000000000..f4baa454d6 --- /dev/null +++ b/src/operator/AGENTS.md @@ -0,0 +1,48 @@ +# src/operator/ — APL Operator + +## OVERVIEW + +GitOps-style operator managing platform installation and continuous reconciliation. +Drives CLI `apply` and `install` commands based on Git changes and periodic heartbeats. + +## EXECUTION FLOW + +1. **Bootstrap**: `main.ts` → `Installer.reconcileInstall()` (retries until success). +2. **Steady State**: `AplOperator.start()` launches parallel loops: + - **Poll (30s)**: `GitRepository` sync → Detect changes → `applyTeams()` or `apply()`. + - **Reconcile (5m)**: Forced `apply()` to ensure state consistency. +3. **Execution**: `AplOperations` wraps CLI commands (apply, install) using `runApplyIfNotBusy`. +4. **State**: `k8s.ts` updates ConfigMap heartbeats and apply status. +5. **Diagrams**: See `EXECUTION_FLOW.md` for detailed sequence diagrams. + +## KEY FILES + +| File | Role | +| ------------------- | --------------------------------------------------------------------- | +| `main.ts` | Entry point; switches from Install phase to GitOps phase. | +| `apl-operator.ts` | Core logic; manages Poll/Reconcile loops and concurrency locks. | +| `apl-operations.ts` | Integration layer; maps operator intent to CLI command handlers. | +| `installer.ts` | Finite state machine for initial platform deployment. | +| `git-repository.ts` | Git lifecycle; change detection (Teams-only vs. Global). | +| `k8s.ts` | Persistence; tracks operator health and apply results in K8s. | +| `validators.ts` | Bootstrapping; ensures environment variables and paths are valid. | +| `errors.ts` | Error hierarchy; specific types for Install vs. Operational failures. | +| `utils.ts` | Shared logic; logging decorators and async retry wrappers. | + +## PATTERNS + +- **Non-blocking loops**: Poll and Reconcile run independently; both use `isApplying` lock. +- **CLI Re-use**: Operator MUST call CLI logic via `AplOperations` to ensure consistency. +- **Granular Apply**: Differentiate between `applyTeams` (fast) and `apply` (full). +- **Heartbeat/Status**: Use `ConfigMaps` for observability instead of internal state. +- **Finite Retry**: `Installer` uses exponential backoff for the initial cluster setup. +- **Fail-fast Validation**: `validators.ts` checks environment before starting. + +## ANTI-PATTERNS + +- **CRD Watching**: Do NOT implement K8s controllers/watchers for configuration. +- **Stateful Logic**: Avoid keeping source-of-truth in memory; rely on Git/K8s. +- **Concurrent Applies**: Never bypass the `isApplying` lock in `apl-operator.ts`. +- **Direct Helmfile Calls**: Logic must pass through `AplOperations` command wrappers. +- **Silent Failures**: All errors must be wrapped in `OperatorError` or `InstallError`. +- **Mixing Phases**: Keep Installation logic strictly separate from GitOps reconciliation. diff --git a/values/AGENTS.md b/values/AGENTS.md new file mode 100644 index 0000000000..05d98f6499 --- /dev/null +++ b/values/AGENTS.md @@ -0,0 +1,210 @@ +# Values Directory — AI Agent Guide + +> Go template value files that configure Helm chart releases. Each subdirectory corresponds to a helmfile release. + +## Structure + +``` +values/ +├── / +│ ├── .gotmpl # Primary chart values (required) +│ ├── -raw.gotmpl # Raw K8s manifests via *raw anchor (optional) +│ ├── -otomi-db.gotmpl # Database chart values via *otomiDb anchor (optional) +│ ├── -cr.gotmpl # Custom resources via *rawCR anchor (optional) +│ └── -valkey.gotmpl # Valkey (Redis) sidecar values (optional) +├── jobs/ +│ └── scripts/ # Job script files +├── raw/ +│ └── istio-raw.gotmpl # Shared Istio raw resources +└── tests/ + └── connectivity-raw.gotmpl # Connectivity test manifests +``` + +**54 subdirectories** — one per app/component. + +## File Naming Conventions + +| Suffix | Helmfile Anchor | Chart Used | Purpose | +| ----------------------- | --------------- | ----------------- | ----------------------------- | +| `.gotmpl` | `*default` | `charts/` | Primary Helm values | +| `-raw.gotmpl` | `*raw` | `charts/raw` | Additional raw K8s manifests | +| `-cr.gotmpl` | `*rawCR` | `charts/raw-cr` | Custom resource manifests | +| `-otomi-db.gotmpl` | `*otomiDb` | `charts/otomi-db` | CloudNativePG database config | + +## Template Boilerplate + +Every `.gotmpl` file starts with standard variable bindings: + +```gotmpl +{{- $v := .Values }} # All merged values (defaults + user + derived) +{{- $a := $v.apps. }} # This app's config +{{- $k := $v.apps.keycloak }} # Keycloak config (if OIDC needed) +``` + +### Common Patterns + +```gotmpl +# Access derived values +$v._derived.untrustedCA # Whether CA is self-signed +$v._derived.oidcBaseUrl # Keycloak OIDC endpoint +$v._derived.tlsSecretName # Wildcard TLS secret name +$v._derived.consoleDomain # Console URL + +# Access cluster info +$v.cluster.domainSuffix # Base domain +$v.cluster.provider # Cloud provider + +# Safe key access with fallback +$a | get "some.key" "default" +$v | dig "deep" "path" "default" + +# Include shared templates +$httpRoute := tpl (readFile "../../helmfile.d/snippets/routes.gotmpl") $v | fromYaml + +# Conditional on app enabled +{{- if $v.apps.someApp.enabled }} + +# Image override for Linode LKE +{{- if $v.otomi.linodeLkeImageRepository }} +image: + repository: "{{ $v.otomi.linodeLkeImageRepository }}/registry/image" +{{- end }} + +# Node selector +{{- with $v.otomi | get "nodeSelector" nil }} +nodeSelector: + {{- range $key, $val := . }} + {{ $key }}: {{ $val }} + {{- end }} +{{- end }} +``` + +### Raw Template Pattern (`*-raw.gotmpl`) + +```gotmpl +resources: + - apiVersion: v1 + kind: ConfigMap + metadata: + name: my-config + data: + key: value +``` + +## App Inventory + +### Core Infrastructure + +| Directory | Files | Notes | +| --------------------- | --------------------------------------------------------- | -------------------------- | +| `cert-manager` | `.gotmpl`, `-raw.gotmpl` | TLS certificate management | +| `istio-base` | `.gotmpl` | Istio CRDs | +| `istiod` | `.gotmpl` | Istio control plane | +| `istio-gateway` | `egressgateway.yaml.gotmpl`, `ingressgateway.yaml.gotmpl` | Istio gateways | +| `istio-resources` | `-raw.gotmpl` | Istio policies, peer auth | +| `ingress-nginx` | `.gotmpl`, `-raw.gotmpl` | NGINX ingress controller | +| `kubernetes-gateways` | `.gotmpl`, `-raw.gotmpl` | Gateway API resources | +| `sealed-secrets` | `.gotmpl`, `-raw.gotmpl` | Sealed secrets controller | +| `external-secrets` | `.gotmpl`, `-raw.gotmpl` | External secrets operator | + +### Identity & Auth + +| Directory | Files | Notes | +| -------------- | -------------------------------------------- | --------------------------- | +| `keycloak` | `.gotmpl`, `-raw.gotmpl`, `-otomi-db.gotmpl` | Identity provider | +| `oauth2-proxy` | `.gotmpl`, `-raw.gotmpl` | OAuth2 authentication proxy | + +### GitOps & CI/CD + +| Directory | Files | Notes | +| ---------------------- | -------------------------------------------------------------- | --------------------- | +| `argocd` | `.gotmpl`, `-raw.gotmpl` | GitOps deployment | +| `argocd-image-updater` | `.gotmpl` | Auto image updates | +| `gitea` | `.gotmpl`, `-raw.gotmpl`, `-otomi-db.gotmpl`, `-valkey.gotmpl` | Git server | +| `tekton-pipelines` | `.gotmpl`, `-raw.gotmpl` | CI/CD pipelines | +| `tekton-triggers` | `.gotmpl` | Tekton event triggers | +| `tekton-dashboard` | `.gotmpl`, `-raw.gotmpl`, `-teams.gotmpl` | Tekton UI | + +### Platform Operators + +| Directory | Files | Notes | +| ----------------------- | ------------------------ | ------------------------- | +| `apl-operator` | `.gotmpl`, `-raw.gotmpl` | APL platform operator | +| `apl-gitea-operator` | `.gotmpl`, `-raw.gotmpl` | Gitea repo/org management | +| `apl-harbor-operator` | `.gotmpl`, `-raw.gotmpl` | Harbor project management | +| `apl-keycloak-operator` | `.gotmpl`, `-raw.gotmpl` | Keycloak realm management | +| `apl-network-policies` | `.gotmpl` | Platform network policies | + +### Monitoring & Observability + +| Directory | Files | Notes | +| ------------------------------ | ---------------------------------------------------------------------------------------------------- | ---------------------- | +| `prometheus-operator` | `.gotmpl`, `-raw.gotmpl`, `-team.gotmpl`, `pod-monitors.gotmpl`, `service-monitors.gotmpl`, `rules/` | Full monitoring stack | +| `grafana-dashboards` | `.gotmpl` | Dashboard provisioning | +| `loki` | `.gotmpl`, `-raw.gotmpl`, `auth-config.gotmpl` | Log aggregation | +| `promtail` | `.gotmpl` | Log shipping | +| `otel-operator` | `.gotmpl`, `-raw.gotmpl` | OpenTelemetry | +| `prometheus-blackbox-exporter` | `.gotmpl` | Endpoint probing | +| `prometheus-msteams` | `.gotmpl` | MS Teams alert bridge | + +### Databases + +| Directory | Files | Notes | +| ------------------------------------ | ------------------------ | ---------------------- | +| `cloudnative-pg` | `.gotmpl`, `-raw.gotmpl` | CloudNativePG operator | +| `cloudnative-pg-plugin-barman-cloud` | `.gotmpl` | Backup plugin | + +### Optional Services + +| Directory | Files | Notes | +| -------------------- | -------------------------------------------- | -------------------------- | +| `harbor` | `.gotmpl`, `-raw.gotmpl`, `-otomi-db.gotmpl` | Container registry | +| `trivy-operator` | `.gotmpl` | Vulnerability scanning | +| `kyverno` | `.gotmpl`, `-raw.gotmpl` | Policy engine | +| `policy-reporter` | `.gotmpl` | Policy violation reporting | +| `knative-operator` | `.gotmpl` | Knative operator | +| `knative-serving` | `-cr.gotmpl`, `-raw.gotmpl` | Serverless workloads | +| `kserve` | `.gotmpl` | ML model serving | +| `kubeflow-pipelines` | `.gotmpl`, `-raw.gotmpl` | ML pipelines | +| `rabbitmq` | `.gotmpl` | Message broker | + +### Platform UI & API + +| Directory | Files | Notes | +| ----------------- | ------------------------ | -------------------- | +| `otomi-console` | `.gotmpl` | Platform web console | +| `otomi-api` | `.gotmpl`, `-raw.gotmpl` | Platform API | +| `otomi-pipelines` | `.gotmpl` | Pipeline definitions | + +### Multi-Tenancy + +| Directory | Files | Notes | +| --------- | ---------------------------------------- | --------------------------------------------------------------- | +| `team-ns` | `.gotmpl` | Team namespace config (RBAC, quotas, netpols, builds, services) | +| `k8s` | `k8s-raw.gotmpl`, `k8s-raw-teams.gotmpl` | Raw K8s resources for platform and teams | + +### Other + +| Directory | Files | Notes | +| ----------------------------- | ------------------------ | --------------------- | +| `external-dns` | `.gotmpl`, `-raw.gotmpl` | DNS record management | +| `metrics-server` | `.gotmpl` | K8s metrics API | +| `linode-cfw` | `.gotmpl` | Linode Cloud Firewall | +| `cert-manager-webhook-linode` | `.gotmpl` | Linode DNS01 solver | +| `gitea-db-secret` | `-raw.gotmpl` | Gitea database secret | + +## How to Add Values for a New App + +1. Create `values//.gotmpl` with standard boilerplate +2. Optionally create `-raw.gotmpl` for additional K8s resources +3. Reference in helmfile release using appropriate anchor (`*default`, `*raw`, etc.) +4. Ensure defaults exist in `helmfile.d/snippets/defaults.yaml` +5. Add schema in `values-schema.yaml` + +## Key Rules + +- **Never hardcode secrets** — use `external-secrets` or `x-secret` schema annotation +- **Never write derived values** — they're computed in `helmfile.d/snippets/derived.gotmpl` +- **Match release name** — directory name must match helmfile release name +- **Use `_rawValues`** — for chart values not covered by schema (escape hatch) +- **Relative paths** — snippets are referenced as `../../helmfile.d/snippets/...`