Generic Alertmanager support with rich formatter

[ornariece]

Problem

Webhook detection currently checks for labels.job to identify Prometheus alerts. But job is just a scrape-config label that Prometheus happens to add — it's not part of the Alertmanager webhook format. Alerts from custom recording rules, Thanos, Mimir, or any other source routed through Alertmanager don't necessarily have it, so they silently fall through to the raw "Data received" fallback.

The formatter is also minimal — it shows status, a title, and three hardcoded labels. Annotations, timestamps, and actionable links (silence, source) are not surfaced.

There are also a few bugs: hasattr(dict, 'key') is used for annotation lookup, which is always False on dicts (keys are accessed via __getitem__, not attributes), so the description annotation is never read. Webhook detection only catches KeyError, missing IndexError/TypeError from malformed payloads. The slack-webhook branch uses if instead of elif, causing fall-through.

Solution

Generic detection: Check labels.alertname instead of labels.job. alertname is mandatory in every Alertmanager webhook per the data model. Since Prometheus alerts are sent through Alertmanager, they always have alertname too — nothing is lost. Grafana alerts are unaffected (they match first on grafana_folder).

Rich formatter: New alertmanager_to_markdown() produces one structured message per alert:

🔥 **InstanceDown** · critical

webserver.example.com of job node_exporter has been down for more than 5 minutes.

• **Instance:** webserver.example.com
• **Job:** node_exporter
• **Since:** 2024-03-30 08:00 UTC

Source · Silence

• Emoji status line with alert name + severity (doubles as push notification preview)
• Description from annotations (description preferred, summary fallback)
• All labels except alertname/severity as metadata
• Duration calculation for resolved alerts
• Pre-filled silence URL from externalURL
• Source link from generatorURL

Bug fixes (separate commits):

• hasattr → in for dict key lookup on annotations
• Catch IndexError and TypeError in addition to KeyError in webhook detection
• Fix if/elif fall-through in the slack-webhook dispatch branch
• e.with_traceback() (returns None) → str(e) in error messages
• Unicode bullet • instead of - list marker to avoid inconsistent <li> indentation on Element X iOS

Tests: Fixed the existing test (it called find_alert_type which doesn't exist — should be get_alert_type) and added coverage for the new formatter, helper functions, classification edge cases, and malformed payloads. 25 tests total.

Breaking changes

• prometheus-alert / prometheus-resolved type strings → alertmanager-alert / alertmanager-resolved
• prometheus_alert_to_markdown() removed (replaced by alertmanager_to_markdown())
• send_alert() takes a Request object instead of pre-read text (internal method)