process-supervisor

An escalating watchdog for long-lived native processes. Fires a domain-specific probe on a tick, tries a chain of soft recovery actions before killing anything, and stops trying if things are firing too often -- so a broken process doesn't get restarted a hundred times an hour while you're asleep.

The library is transport-agnostic. It knows nothing about sockets, HTTP, IPC, or how to kill a process. Domain code plugs in one HealthProbe, an ordered list of RecoveryActions, and a restart callback. The library handles streak filtering, escalation, and rate limiting.

Quick start

from supervisor import (
    Supervisor, RecoveryAction, RecoveryOutcome,
    HealthProbe, HealthStatus,
)
from supervisor.core import HealthCheck

class MyProbe(HealthProbe):
    def check(self) -> HealthCheck:
        if my_service_is_ok():
            return HealthCheck(HealthStatus.HEALTHY)
        return HealthCheck(HealthStatus.UNHEALTHY, diagnostic="ping failed")

def try_reconnect(ctx) -> RecoveryOutcome:
    if ctx.reconnect():
        return RecoveryOutcome.SUCCESS
    return RecoveryOutcome.RETRY

def try_reset_state(ctx) -> RecoveryOutcome:
    ctx.reset_state()
    return RecoveryOutcome.SUCCESS

def kill_and_restart(ctx) -> None:
    ctx.stop()
    ctx.start()

sv = Supervisor(
    probe=MyProbe(),
    actions=[
        RecoveryAction("reconnect", try_reconnect),
        RecoveryAction("reset-state", try_reset_state),
    ],
    on_exhausted=kill_and_restart,
    ctx=my_service_handle,
)
sv.run_forever()

See examples/flaky_service_demo.py for a self-contained working example: a small in-process HTTP server that fails every N-th request, wrapped in a Supervisor that clears its state on failure and kills the server thread only if that stops working.

What is interesting to read

docs/DESIGN.md walks through the decisions that make this different from systemd / supervisord / plain restart-on-crash:

• Coarse probe status, not booleans. UNKNOWN vs UNHEALTHY saves you during a network flake: a probe that times out due to DNS should not trigger a chain of restarts.
• The chain of soft actions. Restarts are expensive. Most transient faults recover from a much cheaper action (reset a value, reopen a connection). The chain lets you order recoveries by cost.
• Two breakers, not one. Recovery-rate and restart-rate are separate sliding windows. When recoveries burn too fast the chain short-circuits to a restart; when restarts burn too fast the supervisor stops trying and just logs.
• Episodes. Individual state samples mean little; what you want to log and count is the arc from "something went wrong" to a definite outcome. EpisodeTracker stitches them for you.

API surface

supervisor/
├── core.py        Supervisor, RecoveryAction, RecoveryOutcome, SupervisorConfig
├── probe.py       HealthProbe, HealthStatus, HealthCheck
└── episodes.py    EpisodeTracker, Episode, EpisodeOutcome

That's it. No decorators, no metaclasses, no async, no dependencies outside the Python standard library.

Testing

$ pip install pytest
$ python3 -m pytest tests/ -v

17 tests cover the escalation chain (SUCCESS / RETRY / ESCALATE), exception handling in actions, both breakers, the streak filter, UNKNOWN handling, and every episode outcome.

Requirements

• Python 3.10+
• No third-party runtime dependencies

License

MIT. See LICENSE.