Skip to content

fix(pdb): require minAvailable < replicas so PodDisruptionBudgets stay satisfiable#110

Open
monrax wants to merge 1 commit into
mainfrom
fix/pdb-minavailable-invariant
Open

fix(pdb): require minAvailable < replicas so PodDisruptionBudgets stay satisfiable#110
monrax wants to merge 1 commit into
mainfrom
fix/pdb-minavailable-invariant

Conversation

@monrax

@monrax monrax commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

Summary

Both PodDisruptionBudgets could render as unsatisfiable on small clusters, silently producing disruptionsAllowed: 0 and blocking every voluntary eviction (kubectl drain, node cordon for maintenance, cluster-autoscaler scale-down). The templates rendered a PDB whenever the replica count met a fixed threshold (datanode >= 1, graylog >= 2) but never compared minAvailable to the actual replica count. This enforces the invariant minAvailable < replicas, matching the fail-fast idiom the MongoDB template already uses for arbiters.

Example of the broken case (DataNode, default minAvailable: 2):

datanode.replicas minAvailable disruptionsAllowed Result
1 2 0 drains hang forever
2 2 0 drains hang forever
3 2 1 works

Details

  • Replace the replica-threshold gate in both PDB templates with a fail when minAvailable >= replicas, mirroring custom/mongo-rs.yaml ("arbiters must be less than replicas"). The error names the misconfigured values.
  • Gate the DataNode PDB on datanode.enabled (it previously rendered even when the DataNode was disabled).
  • Align the DataNode minAvailable fallback (3 -> 2) with values.yaml, so nulling the value no longer silently raises the floor.
  • Add helm-unittest coverage for both PDBs (no-render defaults, enabled/satisfiable case, and the fail-fast misconfig cases).

Behavior change: a PDB enabled with minAvailable >= replicas now fails at render/install time instead of installing a silently-broken budget — the same surface a bad arbiter count already has.

Linked issues

No linked issue — surfaced while auditing the chart against the 2025-12 third-party evaluation (which flagged the PDB gating direction; this fixes the underlying minAvailable-vs-replicas relationship the gate never validated).

PR Checklist

Please check the items that apply to your change.

  • Tests added/updated
  • Documentation updated
  • This PR includes a new feature
  • This PR includes a bugfix
  • This PR includes a refactor

Testing Checklist

Static Validation

  • Linter check passes: helm lint ./charts/graylog
  • Helm renders local template sucessfully: helm template graylog ./charts/graylog --validate

Installation

  • Fresh installation completes successfully: helm install graylog ./charts/graylog
  • All pods reach Running state: kubectl rollout status statefulset/graylog
  • Helm tests pass: helm test graylog

Functional (if applicable)

  • Web UI accessible and login works
  • DataNodes visible in System > Cluster Configuration
  • Inputs can be created and receive data

Upgrade (if applicable)

  • Upgrade from previous release succeeds
  • Scaling up/down works correctly
  • Configuration changes apply correctly

Specific to this PR

  • helm unittest ./charts/graylog passes — 66 tests, incl. 10 new PDB cases
  • PDBs render at defaults with satisfiable minAvailable (graylog 1/2 replicas, datanode 2/3 replicas)
  • Enabling a PDB with minAvailable >= replicas fails with a clear message
  • DataNode PDB no longer renders when datanode.enabled=false

Notes for reviewers

  • Verify all applicable tests above pass
  • Validate that the linked issues are no longer reproducible, if applicable
  • Sync up with the author before merging
  • The commit history should be preserved - use rebase-merge or standard merge options when applicable

🤖 Generated with Claude Code

Both PodDisruptionBudgets rendered whenever replicas met a fixed
threshold (datanode >= 1, graylog >= 2) but never checked minAvailable
against the actual replica count. Enabling a PDB on a small cluster
(e.g. datanode.replicas=1 or 2 with the default minAvailable=2) produced
a budget with disruptionsAllowed=0, silently blocking every voluntary
eviction -- node drains for maintenance and cluster-autoscaler
scale-down would hang indefinitely.

Replace the replica-threshold gate with the same invariant the MongoDB
template already enforces for arbiters: fail fast when
minAvailable >= replicas, with a message pointing at the misconfigured
values. Also gate the datanode PDB on datanode.enabled (it previously
rendered even when the datanode was disabled) and align the datanode
minAvailable fallback (3 -> 2) with values.yaml so nulling the value no
longer silently raises the floor.

Adds helm-unittest coverage for both PDBs: no-render defaults, the
enabled/satisfiable case, and the fail-fast misconfig cases.

Co-Authored-By: Claude Opus 4.8 (1M context) <[EMAIL_ADDRESS_REDACTED]>
@monrax monrax requested a review from alix-graylog July 10, 2026 20:09
@monrax

monrax commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator Author

Live validation on EKS

Validated on EKS (2× m6i.2xlarge, MongoDB Community operator), installed with --set provider=aws.

  • Install with datanode.replicas=1 + datanode PDB enabled → fails fast: minAvailable (2) must be less than datanode.replicas (1)…
  • Install with graylog.replicas=2, minAvailable=2 + graylog PDB enabled → fails fast: must be less than graylog.replicas (2)
  • datanode.enabled=false + datanode PDB enabled → datanode PDB not rendered
  • Default install, both PDBs enabled → both PDBs created with correct minAvailable and tightened selectors (app.kubernetes.io/name + /instance)
  • Runtime status with pods ready → graylog-pdb and graylog-pdb-datanode both report ALLOWED DISRUPTIONS = 1
  • Evict one datanode → permitted (201 Success)
  • Evict a second datanode concurrently → blocked (429 TooManyRequests: … would violate the pod's disruption budget)
  • Evicted pod reschedules and ALLOWED DISRUPTIONS returns to 1

@monrax

monrax commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator Author

Foot-gun live-validated on the main branch

Installed main with datanode.replicas=1 + datanode.podDisruptionBudget.enabled=true:

  • Install succeeded silently with spec.minAvailable: 2 on a 1-replica DataNode
  • Runtime: currentHealthy=1, desiredHealthy=2, disruptionsAllowed=0
  • Evicting the DataNode (what kubectl drain issues) → 429 TooManyRequests — a node drain or autoscaler scale-down would hang indefinitely, with no warning at install

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant