feat(perf): add PSI collectors for system and container pressure monitoring

[jra3]

Summary

Implements Pressure Stall Information (PSI) collectors for both system-level and per-container pressure monitoring. PSI quantifies resource contention (CPU, memory, I/O) to enable performance optimization and intelligent workload scheduling decisions.

Motivation

Resource pressure metrics are critical for:

• Detecting bottlenecks before they cause performance degradation
• Making informed scheduling decisions for workload placement
• Understanding true resource availability vs theoretical capacity
• Optimizing container resource limits based on actual contention

PSI provides kernel-level visibility into when tasks are stalled waiting for resources, distinguishing between "some" stalls (at least one task affected) and "full" stalls (all non-idle tasks blocked).

Implementation Details

System-Level PSI Collector

File: pkg/performance/collectors/psi.go (181 lines)

Collects pressure metrics from /proc/pressure/{cpu,memory,io}:

• Parses "some" and "full" stall metrics
• Captures avg10, avg60, avg300 percentages (short/medium/long-term trends)
• Records total cumulative stall time in microseconds
• Validates PSI availability at initialization (kernel 4.20+ required)

Container-Level PSI Collector

File: pkg/performance/collectors/cgroup_psi.go (140 lines)

Collects per-container pressure from cgroup PSI files:

• Uses containers.Discovery for automatic container detection
• Reads {cpu,memory,io}.pressure from each container's cgroup
• Supports both cgroup v2 (standard) and v1 (when available)
• Graceful degradation when PSI files are missing

OTEL Metric Transformation

File: internal/metrics/consumers/otel/transformer.go (+113 lines)

OpenTelemetry metric mapping:

• Gauges (Float64): avg10/avg60/avg300 percentages (current pressure levels)
• Counters (Int64): total stall time in microseconds (cumulative)
• Attributes: resource={cpu|memory|io}, stall_type={some|full}
• Container attributes: container.id, container.name, cgroup.path

Metric names:

• system.psi.pressure.avg10 (gauge, %)
• system.psi.pressure.avg60 (gauge, %)
• system.psi.pressure.avg300 (gauge, %)
• system.psi.pressure.total (counter, microseconds)

Type System

File: pkg/performance/types.go (+49 lines)

Added structured types:

• PSIStats: System-level pressure for all resources
• PSIResourceStats: Per-resource metrics (some/full × avg10/60/300/total)
• CgroupPSIStats: Per-container pressure with container metadata
• MetricTypePSI and MetricTypeCgroupPSI constants

Enabled in DefaultCollectionConfig for automatic collection.

Testing

System PSI Tests

File: pkg/performance/collectors/psi_test.go (468 lines)

Comprehensive coverage:

• Constructor validation (valid config, missing directories, kernel checks)
• PSI parsing with realistic kernel output
• Edge cases: zero pressure, high pressure (95%+), missing resources
• Error handling: malformed data, empty files, permission errors
• Real-world scenarios from production systems

Container PSI Tests

File: pkg/performance/collectors/cgroup_psi_test.go (236 lines)

Test scenarios:

• Single and multiple container collection
• Partial PSI availability (graceful degradation)
• Missing PSI files (cgroup v1 compatibility)
• Malformed data handling per resource

All tests use mock filesystems for isolation and reproducibility.

Graceful Degradation

The implementation handles various environments:

1. No PSI support: Constructor fails fast with clear error (kernel < 4.20)
2. Partial PSI: Collects available resources, skips missing (some kernels disable CPU PSI)
3. Container PSI unavailable: Skips containers without PSI files (cgroup v1)
4. Parse errors: Logs warning, continues with valid data

This ensures the agent works across diverse kernel configurations.

Kernel Compatibility

• Minimum kernel: 4.20 (PSI introduction)
• Cgroup v2: Full PSI support for all resources
• Cgroup v1: Limited/no PSI support (gracefully handled)
• CONFIG_PSI=n: Detected at initialization, clear error message

CI Improvements

File: .github/workflows/build_and_test.yml

Added linux-tools-generic package to ensure bpftool availability for vmlinux.h generation during eBPF builds. This fixes intermittent CI failures when building eBPF programs.

Related Issue

Closes #88

Files Changed

• Added: pkg/performance/collectors/psi.go (181 lines)
• Added: pkg/performance/collectors/psi_test.go (468 lines)
• Added: pkg/performance/collectors/cgroup_psi.go (140 lines)
• Added: pkg/performance/collectors/cgroup_psi_test.go (236 lines)
• Modified: pkg/performance/types.go (+49 lines)
• Modified: internal/metrics/event.go (+2 lines)
• Modified: internal/metrics/consumers/otel/transformer.go (+113 lines)
• Modified: .github/workflows/build_and_test.yml (+3 lines, -3 lines)

Total: +1,192 lines across 8 files

Testing Instructions

Unit Tests

make test-unit

All PSI collector tests use mock filesystems and run on any platform.

Integration Tests (Linux required)

# On Linux with kernel 4.20+
make test

# Check PSI availability
cat /proc/pressure/cpu
cat /proc/pressure/memory
cat /proc/pressure/io

Manual Testing

Deploy to a Kubernetes cluster and verify metrics:

kubectl logs -n antimetal-system <agent-pod> | grep -i psi

Expected output:

• System PSI metrics with resource={cpu,memory,io} attributes
• Container PSI metrics with container.id and cgroup.path attributes
• Graceful warnings for missing PSI files (expected on cgroup v1)

Performance Impact

• System PSI: 3 file reads per collection (~10KB total)
• Container PSI: 3 × N file reads (N = number of containers)
• Overhead: Negligible (<1ms for typical workloads)
• Collection frequency: Controlled by DefaultCollectionConfig

PSI files are kernel-maintained and reading them does not trigger expensive computations.

Future Enhancements

Potential follow-up work:

1. PSI threshold alerts for proactive intervention
2. Historical trend analysis for capacity planning
3. Correlation with cgroup throttling events
4. Integration with Kubernetes scheduler decisions

Related

• Closes Add PSI Collector #88
• Linear: https://linear.app/antimetal/issue/ENG-2380