Skip to content

observability.md

Latest commit

 

History

History
139 lines (122 loc) · 7.97 KB

observability.md

File metadata and controls

139 lines (122 loc) · 7.97 KB
Raw

Observability (Prometheus + mcp-prometheus)

An optional component, on by default (platform.observability in agentlab.yaml; skip it headlessly with agentlab configure --defaults --observability=false): a minimal Prometheus scraping the cluster plus mcp-prometheus, registered in muster as the prometheus family's member for the lab's cluster (x_prometheus_<tool> with management_cluster: <clusterName>-mcp-prometheus) — so Claude Code (or a kagent agent) can answer "how is the control plane's CPU?" or "how much memory does pod X use?" with real PromQL (x_prometheus_execute_query, …_execute_range_query, label/metadata discovery, and the rest of its 18 read-only tools).

Turned off on a lab that ran it, the next agentlab platform removes the stack: the mcp-prometheus HelmRelease, the kps release and the monitoring namespace (the prometheus-operator CRDs stay, as down keeps them). agentlab status reports the stack on its prometheus line, and names one left over while the switch is off.

The Prometheus is the Giant Swarm kube-prometheus-stack chart — the observability bundle's own pinned constituent (same version, gsoci images, curated relabelings). The bundle itself cannot run here: since v3 it renders Flux HelmReleases wired for the MC→WC model (a hardcoded remote kubeconfig Secret, dependsOn on HelmReleases this lab lacks, Alloy configs generated by observability-operator against Mimir) and it deliberately ships no local PromQL endpoint — collection goes Alloy → remote-write → Mimir on the management cluster. The lab installs the constituent directly with plain helm and re-enables the Prometheus server and node-exporter: prometheus-operator, kube-state-metrics, node-exporter and one emptyDir Prometheus (~5 pods, ~0.5 GB) scraping the apiserver, kubelet/cAdvisor (pod CPU/memory) and the node (control-plane CPU/memory). Values and the reasons for every toggle: internal/lab/templates/kube-prometheus-stack-values.yaml.tmpl; chart versions are Go-const pins in internal/lab/observability.go (like Flux's — no BOM covers them).

mcp-prometheus follows the mcp-kubernetes stance exactly: unauthenticated on the cluster network, muster is the single enforcement point. Its MCPServer CR comes from the chart's agent-platform-mcps.mcpServers values, the entry an installation writes for a management cluster: cluster: <clusterName>, group: prometheus, so the chart names it <clusterName>-mcp-prometheus, labels it tool-group: infrastructure and gives it the family block (muster.families.prometheus). What the lab exercises is the mcp-prometheus tool chain against a plain local Prometheus; the GS production shape (Alloy → Mimir, X-Scope-OrgID tenancy) is out of scope.

The platform's own monitors ride along: with observability on, the chart's muster ServiceMonitor + PrometheusRule, the kagent ServiceMonitor, the valkey PodMonitor and mcp-prometheus's own ServiceMonitor are all enabled, and the lab Prometheus scrapes them (its monitor/rule selectors are opened with *NilUsesHelmValues: false — upstream's default would only select monitors carrying the kps release label).

Backstage's own metrics views ride along too. The Clusters and Deployments pages query Mimir through gs-backend's MimirService, hardcoded to https://observability.<baseDomain>/prometheus/api/v1/query; the chart sets mimirEnabled: false because standalone installations have no such endpoint. With observability on, the lab provides exactly that endpoint — an HTTPRoute on the edge (observability.<domain>, /prometheus → the lab Prometheus served under that prefix, whose query API is what Mimir's is compatible with) — and overrides mimirEnabled: true in its app-config overlay. The gs frontend attributes samples without Mimir's cluster_id label to the installation itself, which is exactly right for a single-cluster lab, so the Deployments/Clusters metrics show real numbers. Note the endpoint is unauthenticated read-only PromQL on the (localhost-only) lab edge: a real MC fronts it with an auth gateway validating the Bearer token, plain Prometheus ignores it — wider than muster's OAuth, accepted for the lab.

Prometheus serves under /prometheus itself (routePrefix and externalUrl in its values), so the one public path carries both the query API and the web UI: agentlab open prometheus opens the UI there (graph, targets, alerts), with the same lab-CA trust offer as the portal. In the cluster the prefix applies too: mcp-prometheus dials http://prometheus-operated.monitoring.svc.cluster.local:9090/prometheus.

Alertmanager

An optional switch, off by default (platform.alertmanager, agentlab configure --alertmanager; needs platform.observability) for proving mcp-prometheus's alert tools (x_prometheus_get_alertmanager_alerts, mcp-prometheus 0.12 and newer). With it on, the kube-prometheus-stack runs its Alertmanager (the operator's alertmanager-operated Service on 9093, the chart's own config: every alert to the null receiver, nothing sent anywhere), the lab Prometheus fires one alert at all times, AgentlabWatchdog (vector(1), like the fleet's Watchdog), and mcp-prometheus gets ALERTMANAGER_URL pointing at that Service. agentlab platform-test's stage "Alertmanager alerts through muster" calls the tool as the signed-in person and expects AgentlabWatchdog; it is skipped with the switch off, or when the mcp-prometheus in place has no alert tool.

To prove an mcp-prometheus build or release candidate, swap its image in with platform.devImages.mcp-prometheus — a local build or a registry ref, side-loaded into the node like every Deployment target; the override and the pull policy ride the lab's mcp-prometheus HelmRelease's postRenderers (mcp-prometheus.yaml.tmpl), next to the dex-localhost sidecar:

platform:
  observability: true
  alertmanager: true
  devImages:
    mcp-prometheus: gsoci.azurecr.io/giantswarm/mcp-prometheus:0.12.0-rc.2

Then agentlab platform and agentlab platform-test.

Querying the lab Prometheus from the host

Two paths reach the lab Prometheus without a local port, and both name the lab cluster in the request itself:

  • the edge route — the endpoint Backstage's metrics views use: curl -sk "https://observability.<domain>/prometheus/api/v1/query?query=<PromQL>" (with :<gatewayPort> when it is not 443);
  • the API server's proxy — before the edge is up, or for the targets page: kubectl --context kind-<clusterName> get --raw '/api/v1/namespaces/monitoring/services/kps-kube-prometheus-stack-prometheus:9090/proxy/prometheus/api/v1/targets' (…/proxy/prometheus/api/v1/query?query=<PromQL> for a query).

A kubectl port-forward … <port>:9090 to a fixed local port is the path to avoid on a machine that talks to more than one cluster. When the port is taken — typically by a forward to another cluster's Prometheus that outlived the shell that started it — kubectl fails to bind, the failure is easy to lose (>/dev/null 2>&1 &), and every query to 127.0.0.1:<port> is answered by the other cluster's Prometheus: its targets, its pod names, its empty results. The lab's targets are pods in kind's 10.244.0.0/16, and its scrape pools are the monitors the lab renders (kps-…, agent-platform-connectivity-…, mcp-prometheus); a Prometheus that shows anything else is not the lab's.

agentlab platform-test grows a phase when the component is on: it lists the x_prometheus_* tools through muster, runs execute_query with up on the lab's cluster (management_cluster), asserts the platform itself is being scraped (muster, valkey, mcp-prometheus, and kagent when agents run all report up == 1), and then runs the Deployments page's exact workload query against the edge observability endpoint — proving Dex → muster → mcp-prometheus → Prometheus and the Backstage metrics path end to end. Logs: agentlab logs prometheus, agentlab logs mcp-prometheus.