Question: What are the best practices for testing exposed metrics compatibility?

[iypetrov]

What’s the recommended way to write compatibility tests for all metrics an app exposes? I want to detect breaking changes like renamed metrics, removed metrics, changed labels/types, etc. This should includes metrics from:

• Built-in collectors (go_*, process_*)
• Custom app metrics
• 3rd-party metrics (gRPC, OpenTelemetry, etc.)

P.S. I was thinking about using the Describe method from the Collector interface to discover all metrics a collector can expose. But I couldn’t find a way to ask a Registry for the list of registered collectors. Would it make sense to add something like this to support this kind of use case, or is there a recommended way to achieve it already?

[iypetrov]

@bwplotka wdyt?

[bwplotka]

Good question!

It depends on what lifecycle you want to do it in. I assume you're thinking about unit tests?

One way of doing this is through /metrics. See https://prometheus.io/docs/prometheus/latest/command-line/promtool/#promtool-check-metrics

P.S. I was thinking about using the Describe method from the Collector interface to discover all metrics a collector can expose. But I couldn’t find a way to ask a Registry for the list of registered collectors. Would it make sense to add something like this to support this kind of use case, or is there a recommended way to achieve it already?

There is https://pkg.go.dev/github.com/prometheus/client_golang@v1.23.2/prometheus#Registry.Describe but all fields are private. You could use Gather and it's similar to scrape - I think that's a reasonable approach.

Long term we thought about the direction of Otel schema:

• https://kccnceu2026.sched.com/event/2CVzR/schema-inference-and-automation-a-new-era-for-telemetry-management-nicolas-takashi-coralogix-arthur-silva-sens-grafana-labs
• Optional schematization of Prometheus metrics using Otel Schema prometheus#18349

When all metrics are defined in a known schema common Otel tools like weaver could be used. For intermitted state we could try to infer (see talk about) schema from metrics by scraping and then Otel tools would work too.

cc @ArthurSens @nicolastakashi

[MistaTwista]

i think that function proposal #2079 can be useful for your case too

[iypetrov]

My original goal was to make a validation for 3rd-party metrics on a unit-test level in order to make sure that I will not break existing dashboards/alerts. My problem with Gather, as of my understanding, is that if some code path doesn't happen the metrics might not be shown under /metrics endpoint so testing becomes much harder. I believe a good example for such use case is a migration from FluentBit v4 -> v5.

https://docs.fluentbit.io/manual/installation/whats-new-in-fluent-bit-v5.0#monitoring-changes
The fluentbit_hot_reloaded_times metric changed from a gauge to a counter, which makes it safe to use with PromQL functions such as rate() and increase().

For such use case weaver as I understand it, might be a solution, but isn't it a bit overkill?

[nicolastakashi]

Hey folks!
Sorry for the long delay to reply it.
Weaver is the current strategy available and I also learned about https://github.com/open-telemetry/semantic-conventions-conformance but I didn't went through it to understand how it works.

I'm also trying to propose something which could be more focused on the prometheus ecosystem and the client instrumentation: prometheus/proposals#86 feedback are very welcome