TCP CR certificate is removed from disk while the bind still references it, making the next reload fatal

[daft-panda]

Environment

• Controller: v3.2.12 (helm chart 1.52.1), DaemonSet, useHostNetwork: true
• HAProxy: 3.2.21-dbe43be37
• Kubernetes: 1.36.2

Summary

A TCP CR bind using ssl_certificate is rendered as a direct crt <file> reference,
so unlike ingress frontend certificates it belongs to no crt-list. When the controller
decides that certificate is no longer in use, the runtime delete therefore always fails
— deleteRuntime asks HAProxy to remove it from a crt-list that does not exist, and the
del ssl cert fallback it never reaches is documented not to work for directly
referenced certificates either. refreshCerts unlinks the file anyway. The certificate
disappears while haproxy.cfg still names it, so the next reload aborts and the worker
exits. Observed on all three of our ingress pods.

Reproduction

apiVersion: ingress.v3.haproxy.org/v3
kind: TCP
metadata:
  name: app-grpc
  namespace: app
spec:
  - name: grpc
    frontend:
      name: app-grpc
      binds:
        v2053:
          address: "0.0.0.0"
          port: 2053
          ssl: true
          ssl_certificate: app-tls
          alpn: h2
    service:
      name: app-leader
      port: 8080

Sequence of events

applyBindOverride (pkg/handler/tcp-cr.go:251) rewrites the bind to the resolved path, so the config references the file directly:

bind 0.0.0.0:2053 name 0.0.0.0:2053 crt /etc/haproxy/certs/tcp/app_app-tls.pem ssl alpn h2

Ingress frontends instead reference the directory, which HAProxy exposes as a crt-list:

bind 0.0.0.0:443 name v4 crt /etc/haproxy/certs/frontend ssl alpn h2,http/1.1

On a running instance, only the frontend directory is a crt-list; the TCP certificate is loaded but belongs to none:

$ echo "show ssl crt-list" | socat stdio /var/run/haproxy-runtime-api.sock
/etc/haproxy/certs/frontend

$ echo "show ssl cert" | socat stdio /var/run/haproxy-runtime-api.sock
/etc/haproxy/certs/frontend/app_app-tls.pem
/etc/haproxy/certs/tcp/app_app-tls.pem

deleteRuntime (pkg/haproxy/certs/main.go:231) nonetheless passes the certificate directory as a crt-list name, which HAProxy correctly rejects:

INFO haproxy/certs/main.go:325 reload required : Runtime delete of cert file 'app_app-tls.pem'
failed : /var/run/haproxy-runtime-api.sock [3]  Can't delete the entry:
crt-list '/etc/haproxy/certs/tcp' does not exist!
[del ssl crt-list /etc/haproxy/certs/tcp /etc/haproxy/certs/tcp/app_app-tls.pem] not found

Because that error returns early, the CertEntryDelete fallback on line 245 is never reached — and per management.txt it could not have worked either: "del ssl cert … The deletion doesn't work with a certificate referenced directly with the crt directive in the configuration." So a reload is the only way to drop such a certificate, and the file must stay on disk until the config no longer names it.

Bug location

refreshCerts (pkg/haproxy/certs/main.go:322-332) schedules the removal regardless of the outcome:

if !crtOk || !crt.inUse {
    err := c.deleteRuntime(certDir, filename)
    if err != nil {
        instance.Reload("Runtime delete of cert file '%s' failed : %s", filename, certErrorForLog(err))
    } else {
        utils.GetLogger().Debugf("Runtime delete of cert ok [%s]", filename)
    }
    fs.AddDelayedFunc(filename, func() {                        // runs even though the delete failed
        logger.Error(os.Remove(path.Join(certDir, filename)))
    })
    delete(certs, certName)
}

The next reload then aborts:

[ALERT] (605) : config : parsing [/etc/haproxy/haproxy.cfg:127] : 'bind 0.0.0.0:2053' in section
'frontend' : unable to stat SSL certificate from file
'/etc/haproxy/certs/tcp/app_app-tls.pem' : No such file or directory.
[ALERT] (605) : config : Fatal errors found in configuration.
[WARNING] (69) : Failed to load worker (605) exited with code 1
ERROR haproxy/process/s6-overlay.go:63 failed to reload configuration

A reload succeeding at 05:08:08 and failing at 05:08:13 brackets the delayed removal.

Impact

While reloads are failing, server changes applied over the runtime API exist only in the running worker. The next reload that does succeed starts from the on-disk config and silently drops them. One pod came out of this with a backend permanently at

server SRV_1 127.0.0.1:1 disabled

while the other two carried the real endpoint, so a share of requests to that backend were answered 503. The controller did not correct it; the divergence persisted for hours until we deleted the pod. Possibly related to #768, which reports the same end state from a different trigger.

Suggested fix

Do not unlink the file when deleteRuntime returned an error. For certificates rendered as a direct crt <file> reference the runtime delete cannot succeed at all, so the file must survive until a reload has loaded a config that no longer references it.