How to trace&debug Provider and Alert CRs?

[Unveraenderbar]

I'd like to make use of the GOTK notification API by implementing a generic webhook provider, so I deploy a HTTP endpoint (basic echo at the moment) and service and refer to it

$ kubextl -n flux-system exec test-pod -i -- curl -sX POST -H "Content-Type: application/json" -d '{"name": "Test"}' http://mygenericwebhook:8000; echo
{"status": "success", "message": {"name": "Test"}}
$ kubectl -n flux-system apply -f - <<EOF-
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: mygenericwebhook
  namespace: flux-system
spec:
  address: http://mygenericwebhook:8000
  secretRef:
    name: flux-system
  type: generic

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: mygenericwebhook
  namespace: flux-system
spec:
  eventSeverity: info
  eventSources:
  - kind: GitRepository
    name: '*'
  providerRef:
    name: mygenericwebhook
  summary: build and push OCI configuration

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: flux-system-repo
  namespace: flux-system
spec:
  interval: 1m0s
  ref:
    branch: main
  secretRef:
    name: flux-system
  timeout: 60s
  url: https://git.corp.int/scm/infra/flux-system
EOF

$ git commit --allow-empty -m 'test 2' && git push
[main 05f6eb6] test 2
Enumerating objects: 1, done.
Counting objects: 100% (1/1), done.
Writing objects: 100% (1/1), 185 bytes | 185.00 KiB/s, done.
Total 1 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote:
To https://git.corp.int/scm/infra/flux-system
   58f1c5a..05f6eb6  main -> main

$ flux reconcile source git flux-system-repo
► annotating GitRepository flux-system-repo in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision feature/flux-pilot@sha1:05f6eb616b61dbd7be6bb7a4b817c51f19a84962
$ kubextl -n flux-system logs -l app.kubernetes.io/component=notification-controller --tail 50000 | grep mygenericwebhook | jq .
{
  "level": "info",
  "ts": "2026-10-01T11:31:57.970Z",
  "msg": "metadata.finalizers: \"finalizers.fluxcd.io\": prefer a domain-qualified finalizer name including a path (/) to avoid accidental conflicts with other finalizer writers",
  "controller": "provider",
  "controllerGroup": "notification.toolkit.fluxcd.io",
  "controllerKind": "Provider",
  "Provider": {
    "name": "mygenericwebhook",
    "namespace": "flux-system"
  },
  "namespace": "flux-system",
  "name": "mygenericwebhook",
  "reconcileID": "16a3652e-2d19-4060-9817-cf7c4b3daf91"
}

As can be seen, the service is up and running, the notification CRs don't throw any errors – but no POST request gets routed to the webhook service ever.

How can I trace the flow of events in the notification subsystem to find out what's missing?

[matheuscscp]

If you don't see any log at all in notification-controller about the event, the reason may be that source-controller did not actually send the event you were expecting. We unfortunately don't have a table with the events sent by each Flux CRD, so at the moment you will need to look at the gitrepository_controller.go file in source-controller to find when/why/which events will be sent for GitRepository objects. You can also increase the log verbosity in the controllers to see if it gives you more info.

[Unveraenderbar]

I turned up log level in all GOTK controllers to debug, but could not find any additional log entries on the Alert or Provider described above.
And though I can see events related to my commits

$ kubectl get event | yq '.items[] | select(.involvedObject.name == "flux-system-repo" and .message == "stored*")'
apiVersion: v1
count: 1
eventTime: null
firstTimestamp: "2026-10-02T12:13:05Z"
involvedObject:
  apiVersion: source.toolkit.fluxcd.io/v1
  kind: GitRepository
  name: flux-system-repo
  namespace: flux-system
  resourceVersion: "765343898"
  uid: 1c4eed5c-b940-49a3-95d8-453d24752abc
kind: Event
lastTimestamp: "2026-10-02T12:13:05Z"
message: stored artifact for commit 'test 4'
metadata:
  annotations:
    source.toolkit.fluxcd.io/digest: sha256:46f6b0f6ac6f0825fa6fe72478cf29ed5741ea163671c230422353bd62a734e1
    source.toolkit.fluxcd.io/revision: main@sha1:4202c7137f4bd29a049d82a9afe812937ff4e1ce
  creationTimestamp: "2026-10-02T12:13:05Z"
  name: flux-system-repo.18dab55683e461d2
  namespace: flux-system
  resourceVersion: "765343910"
  uid: a66bf888-567b-4a7a-98a5-f010704963a0
reason: NewArtifact
reportingComponent: source-controller
reportingInstance: ""
source:
  component: source-controller
type: Normal
$ flux events --for gitrepository/flux-system-repo
LAST SEEN               TYPE    REASON                          OBJECT                           MESSAGE                     
11m                     Normal  NewArtifact                     GitRepository/flux-system-repo   stored artifact for commit 'test 4'

no POST request is sent to my receiver.

@matheuscscp is my understanding of the notification API correct in the sense that if I specify Providers/Alerts as given above, the notification controller should post events to my webhook service?

[matheuscscp]

I'm not sure about secretRef, is your HTTP server requiring auth? Also, is mygenericwebhook the Service object of your HTTP server? Is it in the flux-system namespace?

[Unveraenderbar]

The service is generic, and (together with the deployment) also in the flux system namespace.

$ kubectl get -o yaml svc mygenericwebhook
apiVersion: v1
kind: Service
metadata:
  name: mygenericwebhook
  namespace: flux-system
spec:
  clusterIP: 100.78.75.228
  clusterIPs:
  - 100.78.75.228
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - port: 8000
    protocol: TCP
    targetPort: 8000
  selector:
    app: mygenericwebhook
  sessionAffinity: None
  type: ClusterIP

Tried also with the FQDN i.e. mygenericwebhook.flux-system.svc.cluster.local, and registered all HelmChart, HelmRelease, and Kustomization CRs as event sources in the Alert CR, and removed the secret reference from the Provider CR... but still can't see any requests in the webhook deployment.

[matheuscscp]

I'm wondering if maybe the Flux NetworkPolicies are blocking the traffic from notification-controller to your HTTP server. Please try adding a NetworkPolicy to the flux-system namespace allowing this traffic flow. It could be that notification-controller is hanging on the TCP handshake due to the Flux NetworkPolicies, and never prints a log...

[Icaro0310]

The Alert/Provider resources themselves don't tell you much, the real signal is in notification-controller's logs. I tail them while forcing an event through:

kubectl -n flux-system logs deploy/notification-controller -f

You'll see lines per dispatch attempt, including the HTTP status your webhook returned. If nothing shows up at all, the Alert isn't matching, so double check eventSources, eventSeverity, and that the Provider is Ready (flux get providers / kubectl get alert -A).

Your curl test is fine for verifying the echo endpoint is reachable from the cluster, but note flux sends its own event JSON payload, not arbitrary bodies. If your endpoint only echoes, you'll just see the event object come back, which is exactly what you want for now.

—
More of my work lives here: github.com/Icaro0310 · Website: icaro0310.github.io

[tomm1990]

@Unveraenderbar Those NewArtifact lines are Kubernetes Events. notification-controller does not read them.

source-controller records that Event, then separately POSTs a copy to its --events-addr. The Provider webhook runs only if that POST hits the notification-controller event server. On a stock install the source-controller arg is:

--events-addr=http://notification-controller.flux-system.svc.cluster.local./

The same arg is patched onto kustomize-controller and helm-controller (manifests/bases/*/patch.yaml in flux2). If it is missing, flux events stays full and the echo server stays quiet.

Stop grepping mygenericwebhook. The accept/reject log is on the involved object, and it is written before the Provider is called (internal/server/event_handlers.go):

kubectl -n flux-system logs deploy/notification-controller --since=30m | grep -E 'dispatching event|discarding event'

Reconcile flux-system-repo once while that tail is running.

• dispatching event with flux-system-repo means an Alert matched and the webhook POST was started. The provider name is not on that line. A NetworkPolicy hang shows up about 15s later as failed to send notification (Provider timeout default). That is the case @matheuscscp described.
• discarding event, no alerts found for the involved object means the POST arrived and no Alert matched kind, name, and namespace.
• Neither line means the event server never got the POST. The policies in manifests/policies do not cause that: allow-egress allows all egress, and allow-webhooks only opens ingress port 9292 for Receivers.

If the next reconcile prints dispatching event and the echo server still sees nothing, the NetworkPolicy check is the right next step. If the line is missing, put --events-addr back on the emitting controller and restart it.

[Unveraenderbar]

Thanks a lot for the debugging help!!
@matheuscscp 's suspicionn was correct: after deleting the NetworkPolicy allow-webhooks, my dummy service is notified on new Flux events via JSON payloads.

As far as I understand k8s NetworkPolicies, it might be that the resource deployed on Flux bootstrapping is too "narrow" and would need a section allowing connections from pods (vs. only allowing them only from "namespace-outside")... maybe a hint in the notification API on checking network policies if no events become visible might be helpful.

[matheuscscp]

You're exactly right that you need a new NetworkPolicy to allow the right traffic flow, and that's much better than deleting the default one btw

[Unveraenderbar]

@matheuscscp yes, that would be nic – but unfortunately, as soon the default ingress policy (or any egress policy that tries to narrow down the connectivity to notification controller towards webhook pod) is set for the notification controller, no events are routed to the generic webhook. I'm not sure whether / to which degree network policies are additive, might be the default resource deployed during Flux bootstrapping would have to be patched in order to enable the event flow to the webhook pod.

[matheuscscp]

If you need a patch in the NetworkPolicy of the official Flux distribution you can do it with kustomize patches, regardless if you're using flux bootstrap or FluxInstance