WS3 · cluster doctor follow-up checks — node fit, image pullability, in-cluster egress

[saadqbal]

Parent epic: tracebloc/client-runtime#116 (WS3). Follow-up to #88 / PR #89, which shipped the tracebloc cluster doctor MVP (6 checks) and deferred these three.

Checks to add

• Node fit — compare each node's allocatable CPU/memory/GPU against the resource requests the jobs-manager stamps on spawned training jobs (RESOURCE_REQUESTS, GPU_REQUESTS env). If no node can fit a job, surface ✖ "training jobs can't schedule" (the silent "Pending forever, no node" class). Read-only.
• Image pullability — when the chart uses a registry pull secret (tracebloc.useImagePullSecrets), verify that secret exists and is a well-formed kubernetes.io/dockerconfigjson in the namespace, so private-image pulls don't ImagePullBackOff. Read-only.
• In-cluster egress probe — the bigger one. Today's Backend egress check probes from the CLI host, which isn't the cluster's egress path (the cluster egresses via the egress-proxy). A real probe needs to run inside the cluster — either port-forward to the egress-proxy's own connectivity probe endpoint (preferred; reuses internal/submit/portforward.go, stays side-effect-light), or exec/spawn a probe pod (side-effecting; breaks doctor's read-only contract). Separate design + PR — do not rush into the read-only checks PR.

Sequencing

PR 1: node fit + image pullability (read-only, fit doctor's existing client-go pattern).
PR 2: in-cluster egress probe (after confirming the egress-proxy's probe contract in the client repo).

Target branch: develop.

[saadqbal]

Node-fit + image-pullability merged via #91 (→ develop). Remaining: the in-cluster egress probe (PR2) — investigation-first (confirm the egress-proxy's probe contract in the client repo before picking port-forward vs exec).

[saadqbal]

In-cluster egress probe shipped as a helm test in the client chart — egress-reachability-check (tracebloc/client#270) — not a doctor check. Scoping found the egress-proxy (Squid) has no queryable endpoint and the SB host is fetched post-auth (static nowhere), so a real in-cluster probe must run a pod; the codebase's pattern for that is a helm-test Job (mirrors egress-enforcement-check), which also keeps cluster doctor read-only. Backend reachability (the gating dependency) is covered; SB egress stays on the requests-proxy readiness check in doctor. All three #90 items now addressed — will close when tracebloc/client#270 merges.

[saadqbal]

All three follow-up checks are now merged:

• Node fit + image pullability → cli#91 (cluster doctor, 8 checks).
• In-cluster egress probe → shipped as the egress-reachability-check helm test in the client chart (feat: in-cluster backend-reachability helm test (WS3 / cli#90) client#270, merged to develop).

Closing. Remaining doctor/observability work (WS4 telemetry+alerts, WS5) is tracked in client-runtime#134.