otaru

Over-Engineering at Its Finest.

Bare-metal k3s home lab and technical playground.

Current cluster shape:

• Dedicated 192.168.10.0/24 VLAN for cluster nodes and service virtual IPs
• k3s with embedded etcd on three Raspberry Pi 5 control-plane nodes
• Flannel wireguard-native for pod networking
• MetalLB + Envoy Gateway for service and ingress load balancing
• Istio ambient mesh for east-west traffic and Kiali for mesh observability

Architecture

Hardware

Node	Device	Role	Boot	Storage
raspberrypi-00	Raspberry Pi 5 8GB	Control plane	NVMe	Lexar NM620 256GB
raspberrypi-01	Raspberry Pi 5 8GB	Control plane	NVMe	Samsung 980 PRO 2TB (MZ-V8P2T0BW)
raspberrypi-02	Raspberry Pi 5 8GB	Control plane	NVMe	Crucial P3 Plus 4TB
raspberrypi-03	Raspberry Pi 4 Model B 8GB	Worker	SD	SanDisk Max Endurance 32 GB
ucg-ultra	UniFi Cloud Gateway Ultra	Router/Gateway	-	-
usw-ultra	UniFi Switch Ultra	PoE switch	-	-
rackmate-t1	GeeekPi DeskPi RackMate T1	Rack enclosure	-	-
rack-mount	GeeekPi 10" 2U Rack Mount	Pi rack mount	-	-

The cluster now runs embedded etcd on raspberrypi-00, raspberrypi-01, and raspberrypi-02. Longhorn uses the root filesystem on the NVMe-backed nodes directly. raspberrypi-03 remains a worker and is intentionally unschedulable for Longhorn storage.

Pending Node

Node	Device	Planned role	Boot	Storage
nuc-00	Intel NUC Mini PC Core i3-3217U 8GB	Future worker	mSATA/LVM	64 GB local storage

Network Layout

IP	Role
192.168.10.1	VLAN gateway
192.168.10.50	Kubernetes API VIP
192.168.10.51	Internal ingress VIP
192.168.10.60	raspberrypi-00
192.168.10.61	raspberrypi-01
192.168.10.62	raspberrypi-02
192.168.10.63	raspberrypi-03
192.168.10.64	nuc-00

Cluster Components

Category	Name	Description
Application	Blocky	Stateless DNS server
Application	changedetection.io	Self-hosted website change detection and alerting
Application	CyberChef	The Cyber Swiss Army Knife by GCHQ
Application	Excalidraw	Virtual whiteboard for sketching hand-drawn like diagrams
Application	Home Assistant	Home Automation
Application	JSON Crack	JSON, YAML, etc. visualizer and editor
Application	Kubernetes MCP Server	Model Context Protocol server for Kubernetes cluster operations
Application	TeslaMate	Self-hosted data logger for Tesla
Application	k3s-apiserver-loadbalancer	An operator to update the kubernetes service type to LoadBalancer
Application	冗PowerBot	Telegram bot tracks and counts individual message counts in groups
CI/CD	Argo CD	GitOps, drift detection, and reconciliation
CI/CD	Atlantis	OpenTofu Pull Request Automation, currently disabled in Argo CD
Connectivity	Envoy Gateway	Gateway API ingress controller with TLS termination and shared MetalLB virtual IP
Connectivity	Cloudflare Tunnel	Cloudflare Zero Trust Edge
Connectivity	Flannel	wireguard-native encrypted pod networking for k3s
Connectivity	Gateway API Kubernetes	Virtual IP and Layer 2 announcement for kubernetes service's External IP
Connectivity	Gateway API	Kubernetes standard CRDs for managing network traffic
Connectivity	MetalLB	Bare metal LoadBalancer implementation for service virtual IP allocation and L2 advertisement
Connectivity	httpbin	Generic health check service
Database	CloudNativePG Barman Cloud Plugin	PostgreSQL backup plugin for cloud storage providers
Database	CloudNativePG Clusters	Multi-cluster PostgreSQL management with B2 backup integration
Database	CloudNativePG	A Kubernetes operator that manages PostgreSQL clusters
Monitoring	Grafana	Grafana LGTM Stack. Visualisation dashboards
Monitoring	Heartbeats	Kubernetes operator for heartbeat monitoring
Monitoring	Kiali	Service mesh observability UI for Istio
Monitoring	Kubernetes Metrics Server	Scalable, efficient source of container resource metrics for Kubernetes built-in autoscaling pipelines
Monitoring	Monitoring Stack	Complete monitoring stack with Grafana, Prometheus, and Loki
Scheduling	Descheduler	Evicts pods for optimal cluster node utilisation
Scheduling	k8s-cleaner	Automated failed pod cleanup and periodic workload repaving
Scheduling	KEDA	Event Driven Autoscaler
Scheduling	Reloader	Watch changes in ConfigMap and Secret and do rolling upgrades
Security	1Password Connect	Proxy service for 1Password; acts as a secret provider
Security	External Secrets Operator	Extracts secrets from a secret provider
Security	amazon-eks-pod-identity-webhook	Amazon EKS Pod Identity Webhook for IRSA in bare metal Kubernetes clusters
Security	cert-manager	Manages TLS certificates via Let's Encrypt and ACME protocol
Security	Istio	Service mesh control plane and ambient dataplane (istiod, istio-cni, ztunnel)
Security	oidc-provider	Kubernetes OIDC provider and JWKS endpoint
Security	zizmor	Static analysis for GitHub Actions
Storage	Longhorn Config	Longhorn configuration and recurring jobs
Storage	Longhorn Volume Lib	Reusable volume templates for Longhorn storage
Storage	Longhorn	Distributed block storage system; backup and restore from/to remote destinations

IaaS, PaaS, and SaaS

Category	Name	Service	Description
CI/CD	GitHub	Actions	Run Terragrunt
CI/CD	Sourcery	AI Code Reviews	Instant feedback for Pull Requests
Connectivity	Cloudflare	Access	Edge Access Control
Connectivity	Cloudflare	DNS	Authoritative DNS Service
Connectivity	Cloudflare	Tunnel	Edge Connectivity
Connectivity	UniFi	Wifiman	VPN gateway to home network
Monitoring	Heartbeats	Heartbeats Operator	Kubernetes operator for heartbeat monitoring
Monitoring	WebGazer	Uptime Monitoring	Health Check
Security	1Password	Connect	Secrets Automation
Security	AWS	STS	OIDC/IRSA JWT token exchange
Security	Let's Encrypt	Let's Encrypt	Certificate Authority
Security	Snyk	Snyk	Detects vulnerabilities
Storage	AWS	DynamoDB	Database backend for 冗PowerBot
Storage	AWS	S3	OpenTofu Remote State
Storage	Backblaze	B2	Volume Backup

Bootstrap Cluster

1. Install Tooling

   brew install \
     ansible \
     direnv \
     go-jsonnet \
     helm \
     kubectl \
     opentofu \
     terragrunt

2. Add SSH Keys to known_hosts

   for ip in 192.168.10.{60..63}; do ssh-keygen -R "$ip" && ssh-keyscan "$ip" >> ~/.ssh/known_hosts; done

3. Set Up Service Credentials

   Follow the 1Password Connect Doc to create 1password-credentials.json and save the access token to the file token. Additionally, save your AWS Account ID to the file argocd-secret. Copy .envrc.sample to .envrc and fill in the required environment variables for Terragrunt and OpenTofu.

   ❯ tree $(pwd) -L 1
   /path/to/project/otaru
   ├── .envrc
   ├── .envrc.sample
   ├── 1password-credentials.json
   ├── 1password-credentials.json.sample
   ├── ...
   ├── argocd-secret
   ├── argocd-secret.sample
   ├── ...
   ├── token
   └── token.sample

4. Bootstrap Cluster

   make setup

Oopsy

Update host packages and reboot the entire cluster.

make maintenance

Upgrade k3s kubernetes version and restart workloads.

make upgrade

Unlock a LUKS node after boot.

make unlock raspberrypi-01

Wipe everything and start from scratch.

make nuke

Set up Raspberry Pi nodes and K3s cluster.

make setup