External mode: controller exits with status 0 after fatal Kubernetes API initialization failure

[tomasptacnik-arch]

Summary

When running in --external mode, the HAProxy Kubernetes Ingress Controller
exits with status 0 after a fatal startup failure while retrieving the
Kubernetes version.

This causes a service manager using a failure-based restart policy, such as
systemd with Restart=on-failure, to treat the controller termination as
successful and leave it stopped.

The process returning status 0 is haproxy-ingress-controller, not the
HAProxy data-plane process.

Affected build

The installed binary reports:

root@host:~# /usr/local/bin/haproxy-ingress-controller --version
maxprocs: Leaving GOMAXPROCS=2: CPU quota undefined
HAProxy Ingress Controller dev f8ae6334
Build from: github.com/haproxytech/kubernetes-ingress
Git commit date: 2026-02-17T06:59:59Z
Ingress Management: default implementation

Please let us know if this behavior has already been corrected in a newer
release.

The configured HAProxy data-plane binary and host platform report:

HAProxy version 3.2.15-1ppa1~noble 2026/03/21
Status: long-term supported branch
Linux 6.8.0-124-generic x86_64

This HAProxy version is included for environment completeness. The process
that returned status 0 was the ingress controller identified above, not this
HAProxy binary.

Deployment mode

The controller runs outside the Kubernetes cluster using --external. It is
managed directly by systemd rather than by a Kubernetes Deployment or
DaemonSet.

The relevant, sanitized unit configuration is:

[Unit]
Description="HAProxy Kubernetes Ingress Controller"
Requires=network-online.target
After=network-online.target

[Service]
Type=simple
User=root
Group=root
ExecStart=/usr/local/bin/haproxy-ingress-controller \
  --program=/usr/sbin/haproxy \
  --config-dir=/opt/haproxy-ingress/config \
  --maps-dir=/opt/haproxy-ingress/config/maps \
  --runtime-dir=/tmp/haproxy-ingress \
  --external \
  --ingress.class=private-ingress \
  --configmap=haproxy-ingress/<redacted> \
  --default-backend-service=default/deny-all \
  --ipv4-bind-address=<redacted> \
  --disable-ipv6 \
  --http-bind-port=80 \
  --https-bind-port=443 \
  --stats-bind-port=8404
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGTERM
Restart=on-failure

[Install]
WantedBy=multi-user.target

Failure sequence

During a maintenance-triggered restart, the controller and the local DNS
resolver were started at nearly the same time:

06:45:56.949717  systemd reported haproxy-ingress.service started
06:45:56.951063  systemd began starting systemd-resolved.service
06:45:56.978992  the controller attempted its Kubernetes API lookup
06:45:57.039001  systemd recorded haproxy-ingress.service deactivated successfully
06:45:57.266661  systemd reported systemd-resolved.service started

The controller attempted DNS while the local resolver was still starting and
logged the following fatal error. Hostnames have been anonymized:

PANIC k8s/main.go:114 Unable to get Kubernetes version:
Get "https://kubernetes-api.example:6443/version":
dial tcp: lookup kubernetes-api.example on 127.0.0.53:53:
read: connection refused

Error: Get "https://kubernetes-api.example:6443/version":
dial tcp: lookup kubernetes-api.example on 127.0.0.53:53:
read: connection refused

The DNS failure occurred approximately 288 ms before systemd reported that
the resolver was ready. The transient dependency failure then cleared, but
the controller remained stopped.

Controller exit result

Systemd reported the ingress controller main process as follows:

Main PID: 341268 (code=exited, status=0/SUCCESS)

The effective service result was:

Restart=on-failure
Result=success
NRestarts=0
ExecMainCode=1
ExecMainStatus=0

ExecMainCode=1 is systemd's numeric representation of CLD_EXITED; it does
not mean application exit status 1. ExecMainStatus=0 is the actual
controller exit status.

The controller's PANIC log level is not visible to systemd. Because the
process returned status 0, systemd classified the termination as successful
and did not apply Restart=on-failure.

Actual behavior

1. The controller cannot resolve or contact the Kubernetes API during initial
   startup.
2. It logs a fatal PANIC error.
3. The controller exits with status 0.
4. Systemd records Result=success.
5. Restart=on-failure does not restart the controller.
6. The ingress service remains inactive after the transient dependency
   failure has cleared.

Expected behavior

In --external mode, if mandatory initialization fails and the controller
terminates, it should return a nonzero exit status so the host service manager
can apply its configured failure-restart policy.

Alternatively, the controller could retry the Kubernetes API connection
internally until startup succeeds or the process is explicitly stopped. It
should not return success after failing before it becomes operational.

Reproduction outline

1. Run haproxy-ingress-controller with --external under a process
   supervisor that records its exit status.
2. Configure access to the Kubernetes API through a hostname.
3. Make DNS resolution temporarily unavailable when the controller starts.
4. Start the controller.
5. Observe Unable to get Kubernetes version and the DNS error.
6. Check the controller's exit status.

Observed result: the controller exits with status 0.

Expected result: a nonzero exit status, allowing the process supervisor to
apply its failure-restart policy.

Operational impact

A sub-second DNS or Kubernetes API startup race becomes a persistent ingress
outage. In the observed event, the dependency recovered approximately 288 ms
after the failed lookup, but the controller remained stopped because its
termination was classified as successful.

Ordering the controller after the DNS resolver and using a broader systemd
restart policy are possible deployment mitigations. They do not replace the
expected process behavior: a controller that terminates because mandatory
startup failed should not report success.

Requested change

Could the fatal startup path return a nonzero exit status? If retrying the
initial Kubernetes API connection is the intended behavior, could the
controller remain running and retry instead of terminating successfully?