Add CRD design, API compatibility, and Helm gating conventions

[kelos-bot]

/kind cleanup

What this PR does / why we need it:

Adds agent conventions derived from recent PR review feedback, applied consistently across CLAUDE.md/AGENTS.md and self-development/agentconfig.yaml.

New conventions (from this week's reviews):

1. Per-TaskSpawner configuration in CRD fields — Configuration that varies per TaskSpawner should be a CRD field, not a controller-level flag. Controller flags are for global settings only.
2. CRD API backward compatibility — Never rename or remove existing CRD JSON field tags, as this silently breaks existing manifests.
3. Gate optional CRDs behind Helm values — Helm templates referencing non-core CRDs (e.g., PodMonitoring) must be gated behind a values flag so installs don't fail on clusters without those CRDs.

Carried forward conventions (from prior reviews):

4. Consistent guidance across surfaces — Update all surfaces referencing the same topic together.
5. Provider-agnostic API design — Prefer generic abstractions over provider-specific CRD fields.
6. Idiomatic Helm values — Use nested dictionary structures, not flat string-based parameters.
7. Deploy-dev workflow sync — Update deploy-dev.yaml when adding controller flags or monitoring config.
8. Controller-driven migration — Prefer automated migration over manual user intervention for breaking changes.
9. Release note user actions — Release notes must describe required user actions for breaking changes.

Evidence from PR reviews:

• CRD field vs controller flag: PR Add ghproxy caching server for GitHub API GET traffic #838 — gjkim42 reviewed inline: "The new field should be added to the taskspawner CRD" (agent had added githubReadProxyBaseURL as a controller flag instead of a TaskSpawner CRD field)
• CRD API backward compatibility: PR Add ghproxy caching server for GitHub API GET traffic #838 — P1 review: "Renaming the CRD JSON field to githubReadBaseURL introduces a breaking API change without backward compatibility. Existing manifests using githubReadProxyBaseURL will silently stop configuring the read proxy."
• Gate optional CRDs: PR feat(spawner): add Prometheus metrics export #821 — P2 review: "PodMonitoring is a non-core CRD; rendering it unconditionally will cause Helm installs to fail on clusters that don't have monitoring.googleapis.com/v1 installed"
• Deploy-dev sync: PR feat(spawner): add Prometheus metrics export #821 — spawner PodMonitoring was pinned to kelos-system instead of workflow's KELOS_NAMESPACE
• Controller-driven migration: PR Replace app.kubernetes.io/* labels with kelos.dev/* labels #826 — gjkim42: "Let's auto-delete old spawner Deployments in the controller instead of requiring manual user action"
• Release note user actions: PR Replace app.kubernetes.io/* labels with kelos.dev/* labels #826 — gjkim42: "if a user needs an action, specify it in the release note"

Which issue(s) this PR is related to:

N/A

Special notes for your reviewer:

This PR replaces the previous content on this branch. All conventions are backed by specific PR review evidence listed above.

Does this PR introduce a user-facing change?

NONE

[cubic-dev-ai]

No issues found across 4 files