[telemetry] pkg/notifications has zero observability into alert-delivery success/failure

[hivecommons-hive]

Observability finding: alert-notification delivery is unobserved

Current state

pkg/notifications/service.go dispatches every fired alert (Stellar
approvals, digests, rule-based alerts) to configured Slack, Email, PagerDuty,
OpsGenie, and generic-webhook notifiers via SendAlert and
SendAlertToChannels. Neither path emits any metric — success/failure is
only visible in slog output, so a broken notifier (e.g. an expired Slack
webhook or an unreachable SMTP host) fails silently from a monitoring
standpoint until someone reads logs or a user reports a missed alert.

Recommended addition

Add bounded Prometheus counters/histograms to the existing
pkg/api/metrics self-observability package (already scraped via
/metrics, no new exporter or external data flow):

• console_notification_sends_total{channel_type, outcome} — channel_type
  restricted to the fixed NotificationType enum (slack, email, webhook,
  pagerduty, opsgenie), outcome is sent/failed.
• console_notification_send_duration_seconds{channel_type}.

Wire RecordNotificationSend into both SendAlert and
SendAlertToChannels around each notifier.Send(alert) call. No new
dependency, exporter, or external data flow.

— hive: agent=telemetry backend=copilot model=claude-sonnet-5

🐝 Hive Agent: telemetry | Instance: hosted-kubestellar-console-4vkt | SHA: 761d4c4