Helm drift detection cannot recover from modified Service port

[jkroepke]

Summary

I'm trying to understand the expected behavior of HelmRelease drift detection when the server-side dry-run itself fails because of the drift.

I have a HelmRelease with drift detection enabled and server-side apply enabled for upgrades:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  labels:
    kustomize.toolkit.fluxcd.io/name: flux-operator
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: flux-operator
  namespace: flux-system
spec:
  chart:
    spec:
      chart: .
      interval: 12h
      reconcileStrategy: Revision
      sourceRef:
        kind: GitRepository
        name: flux-operator
      version: "*"

  driftDetection:
    mode: enabled

  install:
    crds: CreateReplace
    strategy:
      name: RetryOnFailure
      retryInterval: 5m

  interval: 5m
  releaseName: flux-operator
  serviceAccountName: kustomize-controller
  storageNamespace: flux-system
  suspend: false
  targetNamespace: flux-system
  timeout: 1m

  upgrade:
    crds: CreateReplace
    serverSideApply: enabled
    strategy:
      name: RetryOnFailure
      retryInterval: 5m

  valuesFrom:
    - kind: ConfigMap
      name: cluster-values
      valuesKey: values
    - kind: ConfigMap
      name: flux-operator-values
      valuesKey: values.yaml

Reproduction

To test drift detection, I manually edit a Service managed by the HelmRelease:

kubectl edit service flux-operator -n flux-system

For example, I change one Service port from:

port: 8080

to:

port: 8081

The edited Service itself remains valid.

I would expect drift detection to restore the Service to the state stored in the Helm release.

Instead, the HelmRelease gets stuck with:

Could not determine release state: unable to determine cluster state:
Service/flux-system/flux-operator dry-run failed (Invalid):
Service "flux-operator" is invalid:
[spec.ports[2].name: Duplicate value: "http",
 spec.ports[3].name: Duplicate value: "http-web"]

The duplicate port names do not exist in the manually edited Service. They seem to appear as part of the server-side dry-run used to determine the cluster state.

As long as the Service stays in this drifted state, Flux does not recover it automatically.

Controller logs

The helm-controller reports that the HelmChart is in sync, but reconciliation then fails while determining the cluster state:

2026-09-14T16:35:02+02:00
{"level":"info","ts":"2026-09-14T14:35:02.962Z","msg":"HelmChart/flux-system/flux-system-flux-operator with SourceRef 'GitRepository/flux-system/flux-operator' is in-sync","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"flux-operator","namespace":"flux-system"},"namespace":"flux-system","name":"flux-operator","reconcileID":"7f28270e-011d-4725-8b4f-3b5052dacc44"}

2026-09-14T16:35:03+02:00
{"level":"error","ts":"2026-09-14T14:35:03.457Z","msg":"Reconciler error","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"flux-operator","namespace":"flux-system"},"namespace":"flux-system","name":"flux-operator","reconcileID":"7f28270e-011d-4725-8b4f-3b5052dacc44","error":"cannot determine release state: unable to determine cluster state: Service/flux-system/flux-operator dry-run failed (Invalid): Service \"flux-operator\" is invalid: [spec.ports[2].name: Duplicate value: \"http\", spec.ports[3].name: Duplicate value: \"http-web\"]"}

Questions

Is this expected behavior?

1. Should a HelmRelease with driftDetection.mode: enabled be able to recover when the drift causes the SSA dry-run itself to become invalid?
2. Is this a known limitation related to the merge semantics of Service.spec.ports?
3. Is there an existing recovery strategy for cases where Flux cannot determine the cluster state because the dry-run fails?
4. Would it make sense to have an optional fallback strategy when normal SSA-based drift correction cannot recover the desired state?

I'm not necessarily suggesting that Flux should always perform an automatic delete/recreate or force replace, as this could be destructive for some resource types.

However, changing a single port on a Helm-managed Service can currently put the HelmRelease into a state where drift detection can detect that something is wrong, but cannot restore the desired state automatically.

[jkroepke]

Ref:

• When ServerSideDiff is enabled, modifying a service or container port number results in "duplicate" error. Diff cannot complete. argoproj/argo-cd#17717
• SSA with force conflicts should update Service kubernetes/kubernetes#130292

It seems a general purpose server-side apply problem?

[Slymaster]

This is the SSA limitation from kubernetes/kubernetes#130292, and helm-controller has a flag that resolves it.

Why the dry-run fails. For server-side apply, Service.spec.ports is a list map keyed by port + protocol (types.go). kubectl edit records its changes under the kubectl-edit field manager, so after your edit the port entries it touched belong to kubectl-edit, not to helm-controller. Drift detection does a dry-run apply of the release manifest (unstructured.go): the chart's 8080 entry is added back, the 8081 entry is kept because another manager owns it, and both carry the same name, hence Duplicate value. Since the error happens before any diff exists, helm-controller stops at "unable to determine cluster state" and never reaches correction (state.go). driftDetection.ignore doesn't help here, because ignored paths are only stripped after the dry-run (unstructured.go).

Recovery. Run helm-controller with --override-manager=kubectl-edit (available since helm-controller v1.5.0, repeatable; flags). Before diffing, fields owned by those managers are transferred to helm-controller (diff.go), so the dry-run drops the 8081 entry and drift correction restores the Service. With a FluxInstance:

spec:
  kustomize:
    patches:
      - target:
          kind: Deployment
          name: helm-controller
        patch: |
          - op: add
            path: /spec/template/spec/containers/0/args/-
            value: --override-manager=kubectl-edit

Add kubectl-patch and kubectl-client-side-apply the same way if those are used on the cluster too. With flux bootstrap, the same patch goes in flux-system/kustomization.yaml.

To unblock the current release right away, revert the port with kubectl edit: the entry key matches the chart's again, so the dry-run succeeds.

[jkroepke]

Thats a very use-full information! I think, this would be awesome that some SSA limitation can be mitigated that way. I will test it after my vacation.