feat: leader election with kube api and send through k8s #271

Giwook-Han · 2025-11-18T04:43:21Z

Why is this change necessary?

Add sequencer failover process by using k8s.

Description

Add k8s lease logic (leader election logic) so that make multiple sequencer run for HA.
the lease has 15 secs of TTL and leader will renew it's lease every 3 secs.

Closes #262

Type of change

Checklist

I have tested my changes
I have updated documentation if needed
I have reviewed my code for best practices

Copilot

Pull Request Overview

This PR implements leader election for the Mojave sequencer using Kubernetes to enable high availability through automated failover. The solution uses the kube-leader-election library with a 15-second TTL lease that is renewed every 3 seconds by the current leader.

Key changes:

Added Kubernetes deployment manifests supporting 3-replica sequencer deployment with leader election
Implemented leader election logic in k8s_leader.rs that coordinates task management based on leadership status
Added comprehensive documentation for local Minikube setup and testing

Reviewed Changes

Copilot reviewed 15 out of 16 changed files in this pull request and generated 19 comments.

Show a summary per file

File	Description
k8s/setup.sh	Shell script to deploy sequencer resources to Kubernetes
k8s/ovh.setup.sh	OVH-specific deployment script with hardcoded kubeconfig path
k8s/service.sequencer.yaml	Service definition exposing HTTP and P2P ports
k8s/rbac.sequencer.yaml	RBAC configuration granting lease management permissions
k8s/pvc.yaml	Persistent volume configuration using hostPath (Minikube-specific)
k8s/ovh.pvc.yaml	OVH-specific PVC using Cinder storage
k8s/deploy.sequencer.yaml	Deployment with 3 replicas, environment variables, and volume mounts
docs/k8s.md	Comprehensive guide for Kubernetes deployment and failover testing
cmd/sequencer/src/main.rs	Refactored to detect K8s environment and delegate to leader election logic
cmd/sequencer/src/k8s_leader.rs	New module implementing Kubernetes leader election and task lifecycle management
cmd/sequencer/Cargo.toml	Added kube dependencies for K8s API interaction
Cargo.toml	Added workspace dependencies for k8s-openapi, kube, and kube-leader-election
crates/block-producer/src/block_producer.rs	Added error logging for block production failures
crates/batch-producer/Cargo.toml	Added mojave-utils dependency
justfile	Removed SKIP_BUILD environment variable check
Cargo.lock	Updated dependency versions and added K8s-related dependencies

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

k8s/deploy.sequencer.yaml

cmd/sequencer/src/k8s_leader.rs

docs/k8s.md

cmd/sequencer/src/k8s_leader.rs

k8s/setup.sh

k8s/deploy.sequencer.yaml

k8s/pvc.yaml

justfile

Signed-off-by: Sacha Froment <sfroment42@gmail.com>

* chore: add health check Signed-off-by: Sacha Froment <sfroment42@gmail.com> * refactor: change the way the main is started Signed-off-by: Sacha Froment <sfroment42@gmail.com> * chore: nitpick * mod: k8s yaml * chore: remove command in justfile * chore: fix just sequencer & node , fix kill process in start.sh * chore: rm * refactor: coordination k8s (#276) * refactor: coordination k8s Signed-off-by: Sacha Froment <sfroment42@gmail.com> * chore: save Signed-off-by: Sacha Froment <sfroment42@gmail.com> * chore: keep health alive Signed-off-by: Sacha Froment <sfroment42@gmail.com> * fix: start Signed-off-by: Sacha Froment <sfroment42@gmail.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore: remove useless Signed-off-by: Sacha Froment <sfroment42@gmail.com> * chore: review comment Signed-off-by: Sacha Froment <sfroment42@gmail.com> * chore: add back save on shutdown Signed-off-by: Sacha Froment <sfroment42@gmail.com> --------- Signed-off-by: Sacha Froment <sfroment42@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update justfile Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update cmd/sequencer/src/main.rs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update cmd/sequencer/src/cli.rs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update cmd/node/src/cli.rs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update cmd/sequencer/src/cli.rs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update justfile Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore: fix 1st call Signed-off-by: Sacha Froment <sfroment42@gmail.com> * Refactor: Use idiomatic assert! in CLI tests (#277) * Initial plan * Replace if-panic patterns with assert! in CLI tests Co-authored-by: sfroment <7238385+sfroment@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: sfroment <7238385+sfroment@users.noreply.github.com> * Fix missing assert in CLI test for command parsing (#278) * Initial plan * Fix matches! macro usage in sequencer CLI tests Added assert! around matches! calls in parse_stop_and_get_pub_key test to properly verify command types instead of just evaluating the boolean. Co-authored-by: sfroment <7238385+sfroment@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: sfroment <7238385+sfroment@users.noreply.github.com> --------- Signed-off-by: Sacha Froment <sfroment42@gmail.com> Co-authored-by: Giwook-Han <hkw00011@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: sfroment <7238385+sfroment@users.noreply.github.com>

Signed-off-by: Sacha Froment <sfroment42@gmail.com>

Copilot

Pull request overview

Copilot reviewed 32 out of 33 changed files in this pull request and generated 14 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

docs/k8s.md

crates/utils/src/health.rs

cmd/sequencer/src/cli.rs

docs/k8s.md

crates/coordination/src/k8s.rs

k8s/stateful.sequencer.yaml

k8s/secret.sequencer.yaml

justfile

docs/k8s.md

Signed-off-by: Sacha Froment <sfroment42@gmail.com>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot · 2025-12-03T09:21:37Z

@sfroment I've opened a new pull request, #281, to work on those changes. Once the pull request is ready, I'll request review from you.

Copilot · 2025-12-03T09:22:19Z

@sfroment I've opened a new pull request, #282, to work on those changes. Once the pull request is ready, I'll request review from you.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot · 2025-12-03T09:23:12Z

@sfroment I've opened a new pull request, #283, to work on those changes. Once the pull request is ready, I'll request review from you.

Copilot · 2025-12-03T09:24:03Z

@sfroment I've opened a new pull request, #284, to work on those changes. Once the pull request is ready, I'll request review from you.

* Initial plan * Fix StatefulSet name references in k8s documentation Co-authored-by: sfroment <7238385+sfroment@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: sfroment <7238385+sfroment@users.noreply.github.com>

…name (#282) * Initial plan * Fix documentation: update references from deploy.sequencer.yaml to stateful.sequencer.yaml Co-authored-by: sfroment <7238385+sfroment@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: sfroment <7238385+sfroment@users.noreply.github.com>

* Initial plan * Fix all references from deploy.sequencer.yaml to stateful.sequencer.yaml Co-authored-by: sfroment <7238385+sfroment@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: sfroment <7238385+sfroment@users.noreply.github.com>

* Initial plan * Call shutdown() on error paths to ensure consistent cleanup Co-authored-by: sfroment <7238385+sfroment@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: sfroment <7238385+sfroment@users.noreply.github.com>

Signed-off-by: Sacha Froment <sfroment42@gmail.com>

Giwook-Han requested a review from sfroment as a code owner November 18, 2025 04:43

Giwook-Han changed the title ~~feat: Leader election with kube api and send through k8s~~ feat: leader election with kube api and send through k8s Nov 18, 2025

sfroment requested a review from Copilot November 21, 2025 09:00

Copilot started reviewing on behalf of sfroment November 21, 2025 09:00 View session

Copilot finished reviewing on behalf of sfroment November 21, 2025 09:03

Copilot AI reviewed Nov 21, 2025

View reviewed changes

sfroment force-pushed the dev/giwook-Han/sequencer-failover branch from 561425b to 941f2cf Compare November 24, 2025 11:36

Giwook-Han and others added 18 commits November 25, 2025 18:40

init: add k8s yaml

7265941

feat: add k8s leader election logic

8d899bd

add: k8s yamls

ab72315

mod: add k8s leader shutdown

4c21205

add: k8s node yaml

156e9cb

fix: use pv&pvc to maintain status on k8s

47dd8c2

chore: yaml && add docs

1be0daf

feat: mod batch producer load batch number when it starts (#272)

bed857c

fix: cargo.toml

1c96a8e

chore: toml

9a6b3ae

mod: k8s + ovh

78df5f2

chore: lint

20df9c4

Signed-off-by: Sacha Froment <sfroment42@gmail.com>

chore: fix here we have 2 result imbricated

36eb566

Signed-off-by: Sacha Froment <sfroment42@gmail.com>

chore: change a bit

af05570

Signed-off-by: Sacha Froment <sfroment42@gmail.com>

feat: add node deploy

8435218

Signed-off-by: Sacha Froment <sfroment42@gmail.com>

chore: fix potential race

d77b6f8

Signed-off-by: Sacha Froment <sfroment42@gmail.com>

chore: update docs

a39ef6b

Signed-off-by: Sacha Froment <sfroment42@gmail.com>

chore: use anyhow error

b4c895f

Signed-off-by: Sacha Froment <sfroment42@gmail.com>

sfroment force-pushed the dev/giwook-Han/sequencer-failover branch from a398ffd to b4c895f Compare November 25, 2025 17:43

sfroment approved these changes Dec 3, 2025

View reviewed changes

sfroment and others added 3 commits December 3, 2025 09:53

chore: lint

2e182a2

Signed-off-by: Sacha Froment <sfroment42@gmail.com>

chore: lint

b9dcca4

Signed-off-by: Sacha Froment <sfroment42@gmail.com>

sfroment requested a review from Copilot December 3, 2025 09:13

Copilot started reviewing on behalf of sfroment December 3, 2025 09:14 View session

Copilot finished reviewing on behalf of sfroment December 3, 2025 09:16

Copilot AI reviewed Dec 3, 2025

View reviewed changes

sfroment and others added 2 commits December 3, 2025 10:20

chore: fix registry

ed8c4c4

Signed-off-by: Sacha Froment <sfroment42@gmail.com>

Update docs/k8s.md

525c108

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI mentioned this pull request Dec 3, 2025

Fix incorrect filename references in Kubernetes documentation #281

Merged

8 tasks

Copilot AI mentioned this pull request Dec 3, 2025

Fix documentation references to use correct StatefulSet manifest filename #282

Merged

8 tasks

Update docs/k8s.md

de5efe3

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI mentioned this pull request Dec 3, 2025

Fix StatefulSet name references in k8s documentation #283

Merged

4 tasks

Copilot AI mentioned this pull request Dec 3, 2025

Consolidate shutdown logic in K8s leader election error handling #284

Merged

5 tasks

Copilot AI and others added 5 commits December 3, 2025 10:32

fix: healt.port

70ba464

Signed-off-by: Sacha Froment <sfroment42@gmail.com>

sfroment merged commit 67bc8ce into main Dec 3, 2025
7 checks passed

feat: leader election with kube api and send through k8s #271

feat: leader election with kube api and send through k8s #271

Uh oh!

Conversation

Giwook-Han commented Nov 18, 2025

Why is this change necessary?

Description

Closes #262

Type of change

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI commented Dec 3, 2025

Uh oh!

Copilot AI commented Dec 3, 2025

Uh oh!

Copilot AI commented Dec 3, 2025

Uh oh!

Copilot AI commented Dec 3, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants