Separate state retention settings from consensus data

[jbearer]

This allows state to be pruned more aggressively than consensus data. This may be desired because state takes up a lot of space, archival state is rarely accessed, and old state can always be re-derived from headers. Thus, pruning state aggressively may enable a node to keep more history of consensus data while using less space.

The logic for pruning is substantially reworked to separate (a) choosing which batch to prune based on retention settings and usage from (b) actually pruning a batch. This makes it much simpler to implement a complex policy such as having different retention policies for state vs consensus data.

[gemini-code-assist]

Code Review

This pull request introduces separate retention policies and pruning logic for Merklized state and consensus data, adding new configuration options and updating storage traits to support state-specific pruned height tracking. The SQL storage pruner has been refactored to handle these categories independently, with state pruning prioritized. Review feedback points out an inconsistency in timestamp usage during pruner initialization and suggests enhancing error messages by including the data category to improve debuggability.

[jbearer]

Unless I'm misunderstanding, this was a latent bug. We were considering the requested state snapshot to exist as long as height (the height of the most recently created snapshot) was not pruned. This would be a false positive whenever we were querying for an older snapshot that had been pruned.

Ultimately this ends up being ok, because if we load a path that is not actually present, we will end up computing a different commitment than the actual root of the tree, but it leads to an unintuitive edge case in the get_path verification logic and a confusing error message.

[claude]

Claude finished @jbearer's task in 3m 48s —— View job

---

Review of PR #4279: Separate state retention settings from consensus data

• Gather context (existing comments, diff)
• Review changed files in detail
• Post inline comments for specific issues
• Post summary review

---

Overall Assessment

This is a well-structured refactor that cleanly separates state pruning from consensus data pruning. The decomposition of the old monolithic prune() into PruneState / Pruner / PruneCategory / prune_batch() is a significant improvement in readability and maintainability. The priority ordering (always prune state before the corresponding consensus data) correctly respects the dependency between the two.

The new test test_separate_state_data_pruning validates the key scenario well — aggressive state pruning while preserving consensus data.

Issues Found

Bug: Archive mode doesn't clear state pruned height

In hotshot-query-service/src/data_source/storage/sql.rs:745, archive mode resets the pruned height to allow refetching of pruned data:

query("DELETE FROM pruned_height WHERE id = 1")
    .execute(conn.as_mut())
    .await?;

This only clears id = 1 (consensus PRUNED_HEIGHT_ID), but doesn't clear id = 2 (STATE_PRUNED_HEIGHT_ID). An archive node that previously had state pruned would still report that state as pruned and wouldn't reconstruct it.

This line isn't in the diff so I couldn't leave an inline comment — suggest changing to:

query("DELETE FROM pruned_height")
    .execute(conn.as_mut())
    .await?;

Fix this →

Minor: Stale comment in save_state_pruned_height

See inline comment — the comment says "id is set to 1" but the actual id is 2.

Suggestion: Richer tracing in prune_batch

See inline comment — including category and target height would improve production debugging.

Design Observations (non-blocking)

• Shared batch_size: State and consensus data share a single batch_size configuration. This is fine for now, but if state tables are significantly larger (as motivated in the PR description), it might be worth having an independent batch size for state in the future.

• Snapshot::Commit path and pruned state: In queries/state.rs, the Snapshot::Commit variant looks up a height from the header table (which may not be pruned) and then checks against state_pruned_height. This is correct — even if the header exists, the state nodes may have been pruned independently. The bug fix changing from height to created in the pruned check is also correct and well-explained in the self-review comment.

• Defaults match: The new state_minimum_retention and state_target_retention default to the same values as their consensus counterparts (1 day and 7 days), making this a backwards-compatible, opt-in feature.

---