Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 25 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,12 +31,14 @@ The operator consists of two main components:
- Matches LoadBalancer IPs against defined IPRule policies (CIDR-based)
- Automatically generates IPRuleConfig resources for each Service
- Manages the Agent DaemonSet
- Supports IPv4 and IPv6 (Dual-Stack)

2. **Agent (DaemonSet)**:
- Runs on each node with hostNetwork access
- Applies/removes IP routing rules on the node
- Uses Linux netlink for direct kernel interaction
- Continuously reconciles the desired state
- Supports IPv4 and IPv6 rules

### What is Policy-Based Routing?

Expand All @@ -50,6 +52,7 @@ In a Kubernetes cluster with multiple network interfaces or load balancers, you
- **Provider-based Routing**: Route services from different tenants through different ISP uplinks
- **Traffic Segregation**: Physically separate production and test traffic
- **Geo-Routing**: Regionally distribute traffic based on LoadBalancer IP ranges
- **Dual-Stack Routing**: Handle IPv4 and IPv6 traffic with specific routing tables

#### How does it work?

Expand All @@ -58,15 +61,19 @@ The operator uses Linux **IP Rules** (see `ip rule`) to route traffic based on t
```bash
# Example: Traffic from Service 10.96.1.50 uses routing table 100
ip rule add from 10.96.1.50 lookup 100 priority 1000

# Example IPv6: Traffic from Service fd00::1 uses routing table 100
ip -6 rule add from fd00::1 lookup 100 priority 1000
```

Routing table 100 can then contain its own routes, e.g.:
```bash
# Table 100: Traffic via special gateway
ip route add default via 192.168.1.1 dev eth1 table 100
ip -6 route add default via fd00::1 dev eth1 table 100
```

**Result**: All packets originating from the Service IP `10.96.1.50` are routed through the gateway `192.168.1.1`, while other services use the default route.
**Result**: All packets originating from the Service IP `10.96.1.50` (or `fd00::1`) are routed through the gateway `192.168.1.1` (or `fd00::1`), while other services use the default route.

## 🏗️ Architecture

Expand Down Expand Up @@ -97,7 +104,7 @@ ip route add default via 192.168.1.1 dev eth1 table 100
│ │ │ │ IPRule │ │ IPRuleConfig │ │ Agent │ │ │ │
│ │ │ │ (User-def.) │ │ (Generated) │ │ (Deploy) │ │ │ │
│ │ │ │ │ │ │ │ │ │ │ │
│ │ │ │ cidr: ... │ │ serviceIP │ │ image: ... │ │ │ │
│ │ │ │ cidrs: ... │ │ serviceIP │ │ image: ... │ │ │ │
│ │ │ │ table: 100 │ │ table: 100 │ │ nodeSelect.│ │ │ │
│ │ │ │ priority │ │ state: ... │ │ │ │ │ │
│ │ │ └─────────────┘ └──────────────┘ └────────────┘ │ │ │
Expand Down Expand Up @@ -129,7 +136,7 @@ ip route add default via 192.168.1.1 dev eth1 table 100
│ │ │ │ │ │
│ │ │ ip rule show: │ │ │
│ │ │ 1000: from 10.96.1.50 lookup 100 │ │ │
│ │ │ 1000: from 10.96.1.51 lookup 200 │ │ │
│ │ │ 1000: from fd00::1 lookup 100 │ │ │
│ │ │ ... │ │ │
│ │ └─────────────────────────────────────────────────────┘ │ │
│ └────────────────────────────────────────────────────────────┘ │
Expand All @@ -147,7 +154,7 @@ ip route add default via 192.168.1.1 dev eth1 table 100
│ │
│ 1. Create IPRule │
├───────────────────────────────>│
│ (cidr: 192.168.1.0/24)
│ (cidrs: [192.168.1.0/24])
│ (table: 100, priority: 1000)│
│ │
│ 2. Create Service (LB) │
Expand Down Expand Up @@ -488,7 +495,8 @@ kind: IPRule
metadata:
name: datacenter-a-routing
spec:
cidr: "192.168.1.0/24"
cidrs:
- "192.168.1.0/24"
table: 100
priority: 1000
```
Expand All @@ -499,7 +507,7 @@ kubectl apply -f iprule-example.yaml

**Effect**: All services with LoadBalancer IPs in this range will automatically be configured with IP rules using routing table 100.

### Example 2: Multi-Datacenter Setup
### Example 2: Multi-Datacenter Setup (Dual-Stack)

```yaml
---
Expand All @@ -508,7 +516,9 @@ kind: IPRule
metadata:
name: datacenter-a
spec:
cidr: "10.0.0.0/16"
cidrs:
- "10.0.0.0/16"
- "fd00:10::/64"
table: 100
priority: 1000
---
Expand All @@ -517,7 +527,9 @@ kind: IPRule
metadata:
name: datacenter-b
spec:
cidr: "10.1.0.0/16"
cidrs:
- "10.1.0.0/16"
- "fd00:20::/64"
table: 200
priority: 1000
---
Expand All @@ -526,7 +538,8 @@ kind: IPRule
metadata:
name: datacenter-c-priority
spec:
cidr: "10.2.0.0/16"
cidrs:
- "10.2.0.0/16"
table: 300
priority: 2000
```
Expand All @@ -545,9 +558,11 @@ echo "200 datacenter_b" >> /etc/iproute2/rt_tables

# Configure routes in table 100
ip route add default via 192.168.1.1 dev eth1 table 100
ip -6 route add default via fd00::1 dev eth1 table 100

# Configure routes in table 200
ip route add default via 192.168.2.1 dev eth2 table 200
ip -6 route add default via fd00::2 dev eth2 table 200

# Ensure persistence with NetworkManager or systemd-networkd
```
Expand All @@ -573,6 +588,7 @@ kubectl logs -n ip-rule-operator-system deployment/ip-rule-operator-controller-m
# Check IP rules on a node
kubectl debug node/<node-name> -it --image=nicolaka/netshoot
ip rule show
ip -6 rule show
```

## 🔧 Development
Expand Down Expand Up @@ -718,4 +734,3 @@ For questions or issues:
---

**⭐ If you like this project, give it a star on GitHub!**

5 changes: 4 additions & 1 deletion api/v1alpha1/iprule_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,10 @@ type IPRuleSpec struct {
// Priority is the rule priority used. If 0, a default will be used by the agent
Priority int `json:"priority,omitempty"`
// SubnetTableMappings defines which routing table/priority to use for any LB IP within the given CIDR subnets
Cidr string `json:"cidr"`
// Deprecated: Use Cidrs instead
Cidr string `json:"cidr,omitempty"`
// Cidrs is a list of CIDR subnets (IPv4 and IPv6)
Cidrs []string `json:"cidrs,omitempty"`
}

// State constants for IPRuleConfig.Spec.State
Expand Down
82 changes: 82 additions & 0 deletions api/v1alpha1/iprule_webhook.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
/*
Copyright 2025 Marius Bertram.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package v1alpha1

import (
"context"
"fmt"
"net/netip"

"k8s.io/apimachinery/pkg/runtime"
ctrl "sigs.k8s.io/controller-runtime"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"sigs.k8s.io/controller-runtime/pkg/webhook"
"sigs.k8s.io/controller-runtime/pkg/webhook/admission"
)

// log is for logging in this package.
var iprulelog = logf.Log.WithName("iprule-resource")

// SetupWebhookWithManager will setup the manager to manage the webhooks
func (r *IPRule) SetupWebhookWithManager(mgr ctrl.Manager) error {
return ctrl.NewWebhookManagedBy(mgr).
For(r).
Complete()
}

// +kubebuilder:webhook:path=/validate-api-operator-brtrm-dev-v1alpha1-iprule,mutating=false,failurePolicy=fail,sideEffects=None,groups=api.operator.brtrm.dev,resources=iprules,verbs=create;update,versions=v1alpha1,name=viprule.kb.io,admissionReviewVersions=v1

var _ webhook.CustomValidator = &IPRule{}

// ValidateCreate implements webhook.CustomValidator so a webhook will be registered for the type
func (r *IPRule) ValidateCreate(ctx context.Context, obj runtime.Object) (admission.Warnings, error) {
iprulelog.Info("validate create", "name", r.Name)
return r.validateIPRule()
}

// ValidateUpdate implements webhook.CustomValidator so a webhook will be registered for the type
func (r *IPRule) ValidateUpdate(ctx context.Context, oldObj, newObj runtime.Object) (admission.Warnings, error) {
iprulelog.Info("validate update", "name", r.Name)
return r.validateIPRule()
}

// ValidateDelete implements webhook.CustomValidator so a webhook will be registered for the type
func (r *IPRule) ValidateDelete(ctx context.Context, obj runtime.Object) (admission.Warnings, error) {
iprulelog.Info("validate delete", "name", r.Name)
return nil, nil
}

func (r *IPRule) validateIPRule() (admission.Warnings, error) {
var warnings admission.Warnings

// Validate Cidrs
for _, cidr := range r.Spec.Cidrs {
if _, err := netip.ParsePrefix(cidr); err != nil {
return warnings, fmt.Errorf("invalid CIDR %q: %v", cidr, err)
}
}

// Validate legacy Cidr if present
if r.Spec.Cidr != "" {
if _, err := netip.ParsePrefix(r.Spec.Cidr); err != nil {
return warnings, fmt.Errorf("invalid legacy CIDR %q: %v", r.Spec.Cidr, err)
}
warnings = append(warnings, "field 'cidr' is deprecated, please use 'cidrs' instead")
}

return warnings, nil
}
9 changes: 7 additions & 2 deletions api/v1alpha1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions cmd/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,12 @@ func main() {
setupLog.Error(err, "unable to create controller", "controller", "Agent")
os.Exit(1)
}
if os.Getenv("ENABLE_WEBHOOKS") != "false" {
if err = (&apiv1alpha1.IPRule{}).SetupWebhookWithManager(mgr); err != nil {
setupLog.Error(err, "unable to create webhook", "webhook", "IPRule")
os.Exit(1)
}
}
// +kubebuilder:scaffold:builder

if metricsCertWatcher != nil {
Expand Down
59 changes: 59 additions & 0 deletions config/crd/bases/api.operator.brtrm.dev_agents.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,65 @@ spec:
description: NodeSelector restricts the target nodes on which the
agent pods will be scheduled.
type: object
resources:
description: Resources applied to the agent pods.
properties:
claims:
description: |-
Claims lists the names of resources, defined in spec.resourceClaims,
that are used by this container.

This is an alpha field and requires enabling the
DynamicResourceAllocation feature gate.

This field is immutable. It can only be set for containers.
items:
description: ResourceClaim references one entry in PodSpec.ResourceClaims.
properties:
name:
description: |-
Name must match the name of one entry in pod.spec.resourceClaims of
the Pod where this field is used. It makes that resource available
inside a container.
type: string
request:
description: |-
Request is the name chosen for a request in the referenced claim.
If empty, everything from the claim is made available, otherwise
only the result of this request.
type: string
required:
- name
type: object
type: array
x-kubernetes-list-map-keys:
- name
x-kubernetes-list-type: map
limits:
additionalProperties:
anyOf:
- type: integer
- type: string
pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
x-kubernetes-int-or-string: true
description: |-
Limits describes the maximum amount of compute resources allowed.
More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
type: object
requests:
additionalProperties:
anyOf:
- type: integer
- type: string
pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
x-kubernetes-int-or-string: true
description: |-
Requests describes the minimum amount of compute resources required.
If Requests is omitted for a container, it defaults to Limits if that is explicitly specified,
otherwise to an implementation-defined value. Requests cannot exceed Limits.
More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
type: object
type: object
tolerations:
description: Tolerations applied to the agent pods.
items:
Expand Down
11 changes: 8 additions & 3 deletions config/crd/bases/api.operator.brtrm.dev_iprules.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,15 @@ spec:
description: IpRuleSpec defines the desired state of IpRule.
properties:
cidr:
description: SubnetTableMappings defines which routing table/priority
to use for any LB IP within the given CIDR subnets
description: |-
SubnetTableMappings defines which routing table/priority to use for any LB IP within the given CIDR subnets
Deprecated: Use Cidrs instead
type: string
cidrs:
description: Cidrs is a list of CIDR subnets (IPv4 and IPv6)
items:
type: string
type: array
priority:
description: Priority is the rule priority used. If 0, a default will
be used by the agent
Expand All @@ -52,7 +58,6 @@ spec:
rules. If 0, a default will be used by the agent
type: integer
required:
- cidr
- table
type: object
status:
Expand Down
Loading
Loading