Talos Docker Compose Setup

This directory contains the Docker Compose configuration for bootstrapping a Talos Kubernetes cluster using PXE boot.

Container Overview

The setup consists of three Docker containers, each with a specific role:

1. matchbox: Serves as the PXE boot server and configuration generator

   • Downloads Talos OS kernel and initramfs
   • Generates Talos configurations for each node
   • Serves configurations over HTTP on port 8080
   • Automatically configures node-specific settings based on network-config.yaml

2. matchbox-tftp: Provides TFTP services for network booting

   • Implements PXE boot protocol over UDP port 69
   • Serves boot files to nodes during PXE boot process
   • Routes nodes to the appropriate configuration based on their MAC address

3. talos-bootstrap: Bootstraps and initializes the Talos Kubernetes cluster

   • Verifies connectivity to control plane nodes
   • Executes the cluster bootstrap process
   • Generates the kubeconfig file
   • Exports configuration files to host for easy access
   • Maps both ~/.kube and ~/.talos directories to provide access to configs

Environment Variables

The Docker Compose setup uses environment variables to configure the Talos cluster. These variables can be set in a .env file in this directory. A sample .env.example file is provided as a template.

Setting Up Environment Variables

1. Copy the example file to create your own .env file:

   cp .env.example .env

2. Edit the .env file to set your specific values:

   # Update with your actual values
   PROD_NAME=cluster.example.com
   PROD_ENDPOINT=https://api.cluster.example.com:6443
   # ... other variables

Available Environment Variables

Variable	Description	Example Value
GITHUB_REPO	GitHub repository for network config	https://github.com/yourusername/talos
FORCE_REGENERATE	Whether to regenerate existing configs	false
PROD_VIP	VIP for the Kubernetes API	192.168.1.100
PROD_NAME	Kubernetes cluster name	cluster.example.com
PROD_ENDPOINT	Kubernetes API endpoint	https://api.cluster.example.com:6443
PROD_DNS_DOMAIN	Kubernetes DNS domain	cluster.local
PROD_POD_SUBNET	Kubernetes pod subnet	10.244.0.0/16
PROD_SERVICE_SUBNET	Kubernetes service subnet	10.96.0.0/12
TALOS_VERSION	Talos version to use	https://pxe.factory.talos.dev/pxe/latest/metal-amd64
WIPE_DISK	Whether to wipe disks during install	true
MATCHBOX_HOST	Hostname for the matchbox server	matchbox.lan

DNS Configuration

Before starting the Docker Compose setup, ensure your DNS is properly configured:

1. Set up DNS entries for:
   • api.cluster.example.com → PROD_VIP
   • matchbox.lan → IP of the host running the matchbox container
   • Optionally, a wildcard entry *.cluster.example.com → PROD_VIP

Starting the Services

To start the services:

docker-compose up -d

All required containers are available on Docker Hub:

• quinneyd/talos-matchbox:v1.9.0
• quinneyd/talos-matchbox-tftp:v1.9.0
• quinneyd/talos-bootstrap:v1.9.0

The docker-compose file is pre-configured to use these images, so no local building is required.

When started, the services will:

1. Download the Talos kernel and initramfs
2. Generate Talos configurations for each node
3. Set up TFTP for PXE booting

Security Note

The .env file contains sensitive information and is excluded from git in the .gitignore file. Do not commit this file to your repository.

Network Configuration

The Talos nodes are configured with three separate network interfaces:

1. Primary Network (PRIMARY_NETWORK_CIDR)

   • Connected to bus path 0000:00:03.0
   • This is the only network with a gateway
   • Used for external connectivity and Kubernetes API access
   • VIP for control plane is configured on this network

2. Secondary Network (SECONDARY_NETWORK_CIDR)

   • Connected to bus path 0000:00:04.0
   • No gateway configured
   • Used for internal cluster communication

3. Tertiary Network (TERTIARY_NETWORK_CIDR)

   • Connected to bus path 0000:00:05.0
   • No gateway configured
   • Used for additional services or management

DHCP Configuration

For proper routing, the DHCP server should be configured to:

• Provide a gateway only on the primary network
• Not provide gateways on the secondary and tertiary networks

This ensures all external traffic is routed through the primary network interface.

The network configuration is automatically generated by the generate-configs.sh script, which creates the appropriate interface configuration for each node based on its bus path.

Network Interface Mapping

Bus Path	Physical Interface	Network Role	Purpose
0000:00:03.0	ens3	PRIMARY_NETWORK_CIDR	Primary network with VIP
0000:00:04.0	ens4	SECONDARY_NETWORK_CIDR	Secondary network
0000:00:05.0	ens5	TERTIARY_NETWORK_CIDR	Tertiary network

This mapping is tracked in the network-config.template.yaml file to avoid confusion.

Scripts Overview

The Docker containers use two main scripts to automate the Talos cluster lifecycle:

1. bootstrap-cluster.sh: Primary script to bootstrap the Talos cluster

   • Used by the talos-bootstrap container
   • Verifies node connectivity and health
   • Executes the bootstrap command and generates kubeconfig
   • Exports configuration files to host machine
   • Modified to ensure proper permissions on exported configs

2. generate-configs.sh: Generates Talos configurations for all nodes

   • Called by the matchbox container
   • Creates control plane and worker node configurations
   • Uses network-config.yaml as input
   • Outputs configurations to matchbox-data volume

Using Talos and Kubernetes Configuration Files

After a successful Talos cluster bootstrap, the necessary configuration files are automatically exported to your host machine:

1. Talos Configuration: Located at ~/.talos/config

   • Used by talosctl to interact with Talos nodes
   • Contains context information and authentication credentials
   • Use with: talosctl --nodes=<node-name> status

2. Kubernetes Configuration: Located at ~/.kube/config

   • Used by kubectl to interact with the Kubernetes cluster
   • Contains endpoints and authentication information
   • Use with: kubectl get nodes

Both configuration files will have proper permissions (644) to be used directly from your host. No need to connect to the container to interact with your cluster.

Example commands:

# Check Talos node status
talosctl --nodes=prodcp1 status

# Check Kubernetes nodes
kubectl get nodes

# View Kubernetes pods
kubectl get pods -A

If you need to regenerate the kubeconfig, you can run:

docker-compose run --rm talos-bootstrap talosctl --nodes=<node-name> kubeconfig