Skip to content

fix(datanode): raise default data volume so search cache fits; document data tiering#112

Open
monrax wants to merge 2 commits into
mainfrom
fix/datanode-data-volume-default
Open

fix(datanode): raise default data volume so search cache fits; document data tiering#112
monrax wants to merge 2 commits into
mainfrom
fix/datanode-data-volume-default

Conversation

@monrax

@monrax monrax commented Jul 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

The default datanode.config.nodeSearchCacheSize (10gb) exceeded the default Data Node data volume (8Gi, ~7gb usable). Since node_search_cache_size reserves space on the Data Node data volume (disk, not memory), enabling an S3 (or path_repo) snapshot repository makes the Data Node fail preflight with not enough usable space for the node search cache and crash-loop. Plain installs are unaffected as the cache validation only runs when a repository is configured. This PR aligns the defaults and documents data tiering setup.

Details

  • values.yaml: datanode.persistence.data.size 8Gi -> 20Gi so the default 10gb cache fits alongside hot index data.
  • NOTES.txt: install/upgrade-time warning when data tiering is enabled and the cache does not fit the data volume (skipped when an existingClaim is used).
  • README.md: new Enable Data Tiering usage section (S3 client config, AWS https/path_style_access=false overrides, cache-vs-volume sizing note); ToC entries for Geolocation and Data Tiering; fixed a broken TLS anchor; repo doc links switched to absolute URLs so they resolve on ArtifactHub.
  • docs/data-tiering.md: new guide with AWS S3 and MinIO walkthroughs, linking to go2docs for the Enterprise warm-tier UI workflow.

PR Checklist

Please check the items that apply to your change.

  • Tests added/updated
  • Documentation updated
  • This PR includes a new feature
  • This PR includes a bugfix
  • This PR includes a refactor

Testing Checklist

Static Validation

  • Linter check passes: helm lint ./charts/graylog
  • Helm renders local template sucessfully: helm template graylog ./charts/graylog --validate

Installation

  • Fresh installation completes successfully: helm install graylog ./charts/graylog
  • All pods reach Running state: kubectl rollout status statefulset/graylog
  • Helm tests pass: helm test graylog

Functional (if applicable)

  • Web UI accessible and login works
  • DataNodes visible in System > Cluster Configuration
  • Inputs can be created and receive data

Upgrade (if applicable)

  • Upgrade from previous release succeeds
  • Scaling up/down works correctly
  • Configuration changes apply correctly

Specific to this PR

  • Reproduced the Data Node CrashLoopBackOff on the old 8Gi default + 10gb cache (not enough usable space... only 7gb available)
  • Confirmed the Data Node reaches 1/1 Running on the new 20Gi default
  • Confirmed the NOTES warning fires only when cache >= data volume (validated across scenarios via --dry-run=client)
  • Verified the full hot->warm lifecycle (repo verified, snapshot to bucket, searchable warm index) against real AWS S3
  • Verified the full hot->warm lifecycle against in-cluster MinIO
  • All testing done on EKS with a Graylog Enterprise license

Notes for reviewers

  • Verify all applicable tests above pass
  • Validate that the linked issues are no longer reproducible, if applicable
  • Sync up with the author before merging
  • The commit history should be preserved - use rebase-merge or standard merge options when applicable

🤖 Generated with Claude Code

monrax added 2 commits July 12, 2026 19:19
…ering enabled

The default datanode.config.nodeSearchCacheSize (10gb) exceeded the default
datanode data volume (8Gi, ~7gb usable). node_search_cache_size reserves disk on
the datanode data volume, so once an S3/path_repo snapshot repository is
configured the datanode fails preflight ("not enough usable space for the node
search cache") and CrashLoops. Plain installs are unaffected because the check
only runs when a repository is enabled.

- values.yaml: datanode.persistence.data.size 8Gi -> 20Gi so the default cache
  fits alongside hot index data.
- NOTES.txt: warn at install/upgrade time when tiering is enabled and the cache
  does not fit the data volume (skipped when an existingClaim is used).

Co-Authored-By: Claude Opus 4.8 (1M context) <[EMAIL_ADDRESS_REDACTED]>
- README: add an "Enable Data Tiering" usage section covering the S3 client
  config, the AWS https/path-style overrides, and the cache-vs-data-volume
  sizing note; add ToC entries for Geolocation and Data Tiering; fix a broken
  TLS anchor; use absolute URLs for repo doc links so they resolve on ArtifactHub.
- docs/data-tiering.md: new guide with AWS S3 and MinIO walkthroughs, handing
  off to go2docs for the Enterprise warm-tier UI workflow.

Co-Authored-By: Claude Opus 4.8 (1M context) <[EMAIL_ADDRESS_REDACTED]>
@monrax monrax requested a review from alix-graylog July 12, 2026 18:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant