feat: High availabilty via RAFT

[alpe]

Replaces #2836

[github-actions]

The latest Buf updates on your PR. Results from workflow CI / buf-check (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	⏩ skipped	✅ passed	✅ passed	Jan 9, 2026, 12:13 PM

[claude]

Claude finished @alpe's task in 2m 40s —— View job

---

Code Review: RAFT High Availability Implementation

I've completed a comprehensive review of this RAFT integration PR. This is a substantial architectural change introducing high availability through HashiCorp's RAFT consensus. Below are my findings organized by severity and category.

---

Critical Issues 🔴

1. Bootstrap Mode Restriction Blocks Non-Bootstrap Nodes

File: pkg/raft/node.go:108-115

The Start() method returns an error if Bootstrap is false, which prevents non-bootstrap nodes from joining an existing cluster:

if !n.config.Bootstrap {
    return fmt.Errorf("raft cluster requires bootstrap mode")
}

Impact: This makes it impossible to add new nodes to an existing cluster dynamically, severely limiting cluster scalability and operational flexibility.

Recommendation: Only execute bootstrap logic when Bootstrap is true. Allow non-bootstrap nodes to start and attempt to join existing cluster via AddPeer. Fix this →

---

2. Naked Returns Swallow Errors

File: node/full.go:113-119

Two critical error paths use naked return statements that will return (nil, nil) instead of propagating errors:

if leaderElection, err = newSingleRoleElector(leaderFactory); err != nil {
    return  // ❌ Should be: return nil, err
}

Impact: Initialization errors are silently ignored, leading to nil pointer dereferences and runtime panics.

Recommendation: Change both occurrences to return nil, err. Fix this →

---

3. Race Condition in Leader Election Sync Check

File: pkg/raft/election.go:96-109

The leadership transition logic has a TOCTOU (time-of-check-time-of-use) race:

raftSynced := d.node.waitForMsgsLanded(d.node.Config().SendTimeout) == nil
if d.node.leaderID() != d.node.NodeID() {  // ⚠️ Leadership could change between these lines
    continue
}
if !raftSynced || !runnable.IsSynced(d.node.GetState()) {
    // Pass on leadership
}

Impact: Could result in dual leadership or "split-brain" scenarios where two nodes believe they're leaders, potentially causing double-signing and chain forks.

Recommendation: Check leadership status before and after sync, with proper locking. Consider adding a monotonic leadership epoch/term counter. Fix this →

---

4. Missing Error Handling in RAFT State Persistence

File: block/internal/executing/executor.go:474-478

RAFT broadcast errors during block production could lead to state inconsistency:

if err := e.raftNode.Broadcast(e.ctx, raftState); err != nil {
    return fmt.Errorf("failed to propose block to raft: %w", err)
}
// ... but block is still committed locally even if RAFT broadcast failed

Impact: If RAFT broadcast fails, the local node commits the block but cluster doesn't receive it, causing divergence.

Recommendation: Either fail block production entirely on RAFT broadcast failure, or implement retry/rollback logic. Fix this →

---

High Priority Issues 🟡

5. Insufficient Context Cancellation Timeout

File: pkg/raft/election.go:68-70

Worker cancellation uses parent context which may be already cancelled:

workerCtx, cancel := context.WithCancel(ctx)

Issue: When switching roles, the parent context might be cancelled before workers gracefully shut down, leading to abrupt termination.

Recommendation: Use context.WithTimeout(context.Background(), 5*time.Second) for shutdown operations.

---

6. Potential Memory Leak in Apply Channel

File: pkg/raft/node.go:293-299

The FSM apply callback drops messages on full channel:

select {
case f.applyCh <- RaftApplyMsg{Index: log.Index, State: &state}:
default:
    f.logger.Warn().Msg("apply channel full, dropping message")
}

Issue: While non-blocking is good, dropping RAFT consensus messages silently could lead to followers permanently falling behind.

Recommendation: Add metrics for dropped messages and consider triggering a full sync if too many drops occur.

---

7. Missing Validation in State Transitions

File: pkg/raft/types.go:13-21

The assertValid function only checks height/timestamp ordering but doesn't validate:

• Chain continuity (hash chains)
• Non-negative timestamps
• Reasonable timestamp drift

Recommendation: Add comprehensive validation including hash chain verification, timestamp bounds, and state transition rules.

---

8. Inadequate Duplicate Server Detection

File: pkg/raft/node.go:377-391

deduplicateServers only deduplicates by ID, ignoring address conflicts:

key := server.ID  // ⚠️ Should also check for duplicate addresses

Issue: Two servers with different IDs but same address could cause connection issues.

Recommendation: Validate both ID and address uniqueness, return error on address conflicts.

---

Medium Priority Issues 🟢

9. Misleading Method Name

File: block/internal/cache/pending_headers.go:69-71

func (ph *PendingHeaders) GetLastSubmittedDataHeight() uint64

This method is on PendingHeaders but retrieves data height, which is confusing.

Recommendation: Rename to GetLastSubmittedHeaderHeight() for consistency. Fix this →

---

10. Debug Log at Wrong Level

File: block/internal/syncing/syncer.go:184

s.logger.Info().Msg("received raft block state")  // Should be Debug

Recommendation: Change to Debug() level to avoid log spam in production.

---

11. Non-Idiomatic Go Naming

File: pkg/rpc/server/http.go:143-146

NodeId string `json:"node_id"`  // Should be NodeID

Recommendation: Use NodeID per Go conventions for initialisms.

---

12. Redundant Type Conversion

File: block/internal/executing/executor.go:623-625

for i, tx := range batchData.Transactions {
    data.Txs[i] = tx  // types.Tx is already []byte
}

Recommendation: Replace loop with data.Txs = append(data.Txs, batchData.Transactions...) for efficiency.

---

Architecture & Design 📐

Strengths ✅

1. Clean separation of concerns - RAFT logic isolated in pkg/raft/, interfaces well-defined
2. Proper use of FSM pattern - State machine implementation follows RAFT best practices
3. Dynamic leader election - Elegant failover mechanism with DynamicLeaderElection
4. Non-blocking apply channel - Prevents RAFT cluster blocking on slow consumers
5. Comprehensive E2E tests - test/e2e/failover_e2e_test.go provides good coverage

Areas for Improvement ⚠️

1. Test coverage: Only 40% patch coverage (567 lines uncovered). Critical paths in pkg/raft/node.go (12% coverage) and block/internal/syncing/raft_retriever.go (0% coverage) need tests.

2. Configuration complexity: 8 new RAFT config options added without migration guide or validation rules.

3. Leadership transfer logic: The leadership transfer in election.go:104 needs retry logic and timeout handling.

4. Snapshot management: No tests for snapshot/restore functionality, which is critical for cluster recovery.

5. Metrics: Missing key metrics for:

   • Leader election count/duration
   • RAFT message latency
   • Follower lag
   • Dropped apply messages

---

Security Considerations 🔒

1. TLS Not Enforced: RAFT transport uses TCP without TLS. For production, mutual TLS should be mandatory.

2. No Authentication: RAFT AddPeer accepts any nodeID/address without authentication. Consider adding JWT or mTLS.

3. Snapshot Security: Snapshots stored on disk without encryption. Consider encrypting at rest.

4. Log Poisoning: No validation of log entries before applying to FSM beyond basic unmarshaling.

---

Performance & Resource Management ⚡

1. BoltDB Disk Usage: Two separate BoltDB files (raft-log.db and raft-stable.db) could grow large. Need log compaction strategy.

2. Sync Wait Timeout: waitForMsgsLanded uses SendTimeout which may be too short for large clusters under load.

3. Context Propagation: Several places create new background contexts instead of propagating parent context, losing trace information.

---

Testing Gaps 🧪

Required test additions:

1. Unit tests for:

   • pkg/raft/node.go bootstrap and non-bootstrap paths
   • pkg/raft/election.go all leadership transition scenarios
   • block/internal/syncing/raft_retriever.go apply loop edge cases

2. Integration tests for:

   • Adding node to existing cluster
   • Network partition scenarios
   • Leader crash during block production
   • Snapshot creation and restoration

3. Chaos tests for:

   • Random node failures
   • Network delays/packet loss
   • Disk I/O failures

---

Documentation 📝

Missing/Incomplete:

1. No ADR (Architecture Decision Record) for RAFT integration approach
2. Configuration parameters lack detailed descriptions of valid ranges
3. No runbook for operational procedures (adding nodes, leadership transfer, recovery)
4. Migration guide missing for existing deployments

---

Recommendations Summary

Before Merge:

1. ✅ Fix critical issues ABCI interface #1-4 (bootstrap, error handling, race conditions)
2. ✅ Add unit tests for core RAFT paths to reach >70% coverage
3. ✅ Add configuration validation with sensible defaults
4. ✅ Document RAFT configuration options in CLAUDE.md

Post-Merge (Can be follow-up PRs):

1. Add TLS support for RAFT transport
2. Implement comprehensive metrics and monitoring
3. Add chaos/resilience tests
4. Create operational runbook

---

Conclusion

This is a substantial and well-architected RAFT integration that brings true high availability to ev-node. The core design is sound with proper separation of concerns and elegant use of Go patterns. However, critical bugs in error handling and bootstrap logic must be fixed before merge to prevent production issues.

The 40% test coverage is concerning for such critical infrastructure. I strongly recommend adding tests for at least the core RAFT node lifecycle and leader election paths before merging.

Overall: Approve with required changes ✅ (after addressing critical issues #1-4)

---

[gemini-code-assist]

Summary of Changes

Hello @alpe, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the Rollkit network's resilience and fault tolerance by integrating the Raft consensus algorithm. The changes enable a robust failover mechanism, allowing nodes to seamlessly transition between leader and follower roles. This ensures continuous operation and consistent state across the network, even if individual nodes experience failures. The update touches various core components, from dependency management and block processing to RPC services and node lifecycle management, all aimed at achieving high availability.

Highlights

• Raft Consensus Integration: Introduced Raft consensus for high availability, enabling dynamic leader election and state replication. This involves new dependencies on github.com/hashicorp/raft and github.com/hashicorp/raft-boltdb.
• Node Architecture Refactoring: The node's core logic has been refactored to support a failover mechanism, allowing nodes to dynamically switch between aggregator (leader) and sync (follower) modes based on Raft's leadership status. This includes changes to how P2P clients are initialized and passed to node startup commands.
• Block Processing with Raft: The block executor now checks Raft leadership before producing new blocks and proposes the block state to the Raft cluster after execution, ensuring state consistency across the cluster.
• Cache and State Management Enhancements: Added methods to retrieve and persist the last submitted DA header and data heights from the cache. Logic was introduced to skip in-flight (temporary) headers and data during cache retrieval to prevent processing incomplete blocks.
• New RPC Endpoints and Configuration: Added new RPC endpoints for querying Raft node status (e.g., /raft/node) and introduced new configuration flags for Raft-specific settings like node_id, raft_addr, bootstrap, and peers.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces high availability via RAFT consensus, which is a significant architectural change. The implementation is extensive, touching many parts of the system from node startup to block processing and syncing. Overall, the changes are well-structured, introducing clear interfaces for RAFT integration and refactoring the node components to support dynamic role switching between leader and follower. I've identified a few critical issues related to error handling and startup logic that need to be addressed, along with some medium-severity issues for improving code clarity and maintainability. Great work on this complex feature.

I am having trouble creating individual review comments. Click here to see my feedback.

node/full.go (113-115)

The naked return here will not propagate the error from newSingleRoleElector. This will cause the function to return (nil, nil), masking the underlying error. The error should be returned to the caller.

		if leaderElection, err = newSingleRoleElector(leaderFactory); err != nil {
			return nil, err
		}

node/full.go (117-119)

Similar to the previous case, the naked return here will swallow the error from newSingleRoleElector. The error should be propagated up to the caller.

		if leaderElection, err = newSingleRoleElector(followerFactory); err != nil {
			return nil, err
		}

pkg/raft/node.go (111-113)

This check prevents a node from starting if Bootstrap is false, which is problematic for nodes joining an existing cluster. A new node attempting to join will fail to start. The bootstrap logic should only execute if n.config.Bootstrap is true, and the function should return nil otherwise, allowing non-bootstrap nodes to start and join a cluster.

block/internal/cache/pending_headers.go (69-71)

The method name GetLastSubmittedDataHeight is misleading as it's part of the PendingHeaders struct. For clarity and consistency, it should be renamed to GetLastSubmittedHeaderHeight.

This change will also require updating the call site in block/internal/cache/manager.go.

func (ph *PendingHeaders) GetLastSubmittedHeaderHeight() uint64 {
	return ph.base.getLastSubmittedHeight()
}

block/internal/executing/executor.go (570-572)

The explicit type conversion types.Tx(tx) is redundant since types.Tx is an alias for []byte, and tx is already of type []byte. The change to a direct assignment is good, but it seems this loop could be replaced with a single, more efficient append call.

	data.Txs = append(data.Txs, batchData.Transactions...)

block/internal/syncing/syncer.go (184)

This log message seems to be for debugging purposes, indicated by the +++ prefix. It should be logged at the Debug level instead of Info to avoid cluttering the logs in a production environment.

			s.logger.Debug().Uint64("header_height", state.LastSubmittedDAHeaderHeight).Uint64("data_height", state.LastSubmittedDADataHeight).Msg("received raft block state")

pkg/rpc/server/http.go (143-146)

To adhere to Go's naming conventions for initialisms, the struct field NodeId should be renamed to NodeID.

				NodeID   string `json:"node_id"`
			}{
				IsLeader: raftNode.IsLeader(),
				NodeID:   raftNode.NodeID(),

[codecov]

Codecov Report

❌ Patch coverage is 40.12672% with 567 lines in your changes missing coverage. Please review.
✅ Project coverage is 55.75%. Comparing base (453a8a4) to head (49ba50f).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/raft/node.go	12.12%	174 Missing ⚠️
pkg/raft/node_mock.go	45.40%	74 Missing and 21 partials ⚠️
block/internal/syncing/raft_retriever.go	0.00%	63 Missing ⚠️
node/full.go	32.30%	37 Missing and 7 partials ⚠️
block/internal/syncing/syncer.go	25.45%	39 Missing and 2 partials ⚠️
node/failover.go	74.45%	22 Missing and 13 partials ⚠️
block/internal/executing/executor.go	5.55%	30 Missing and 4 partials ⚠️
pkg/raft/election.go	80.23%	12 Missing and 5 partials ⚠️
pkg/rpc/server/http.go	6.66%	13 Missing and 1 partial ⚠️
block/components.go	27.27%	7 Missing and 1 partial ⚠️
... and 13 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2954      +/-   ##
==========================================
- Coverage   58.74%   55.75%   -3.00%     
==========================================
  Files          93      102       +9     
  Lines        8863     9814     +951     
==========================================
+ Hits         5207     5472     +265     
- Misses       3067     3714     +647     
- Partials      589      628      +39     

Flag	Coverage Δ
combined	55.75% <40.12%> (-3.00%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.