Ingestor pinned to an amd64-only digest — arm64 data ingestion fails (ImagePullBackOff)

[LukasWodka]

Symptom

On arm64 hosts (Apple Silicon, AWS Graviton), data ingestion fails. The client install + connect succeed, but helm install tracebloc/ingestor … → the ingestor Job that jobs-manager spawns is stuck in ImagePullBackOff:

ghcr.io/tracebloc/ingestor@sha256:a0861ea9…
Failed to pull image: no match for platform in manifest: not found

Reproduced live on an Apple Silicon prod install (berlin-team, 2026-06-03). amd64/x86 hosts are unaffected.

Root cause

client/values.yaml pins images.ingestor.digest to the v0.3.1 baseline sha256:a0861ea9… (set in #160 / #161), which is an amd64-ONLY OCI index — no linux/arm64 entry. A digest-pinned pull on an arm64 node therefore has no platform match.

The floating :0.3 tag is now multi-arch (sha256:d361fa77…, amd64 + arm64 — data-ingestors started building arm64 after v0.3.1), but jobs-manager was still on the stale amd64 baseline (see follow-up 2).

Fix

Bump images.ingestor.digest to the multi-arch :0.3 index sha256:d361fa77…. Verified helm template renders the new digest into INGESTOR_IMAGE_DIGEST (no a0861ea9); helm lint clean. (PR linked below.)

Follow-ups (separate)

1. CI guard — assert the pinned ingestor digest is a multi-arch index before merge (would have caught Pin client chart's ingestor digest to v0.3.1 #160). Fits the installer-tests cross-arch coverage.
2. image-refresh — jobs-manager stayed on the amd64 baseline even though the image-refresh CronJob ran and :0.3 is multi-arch. Investigate whether image-refresh tracks the multi-arch float and/or whether auto-upgrade resets INGESTOR_IMAGE_DIGEST to the chart baseline.

Related

#160 (pinned the amd64-only v0.3.1 digest) · #158 (image-refresh) · backend#723 (sibling: mysql-client arm64).

Suggested fields: Work Type Bug · Squad Backend/Platform · Area Client (Helm) · Priority P1 (blocks arm64 ingestion entirely).

[LukasWodka]

Both follow-ups resolved (2026-06-03)

1. CI guard — DONE, in PR #187. Added a helm-ci job (ingestor-multiarch) that parses images.ingestor.digest and fails the build if it isn't a multi-arch index (linux/amd64 + linux/arm64) — would have caught the original amd64-only pin (#160). ghcr.io/tracebloc/ingestor is public, so it needs no secrets. Verified locally: passes on the new multi-arch d361fa77, fails on the old amd64-only a0861ea9. (#187 squashes the digest bump + the guard into one commit.)

2. image-refresh — NOT a bug. Investigation showed it works correctly; the failure was a timing artifact. Created-dates:

• amd64-only a0861ea9 (v0.3.1): built 2026-05-25.
• multi-arch d361fa77 (:0.3): amd64 built 2026-06-03 06:30 UTC, arm64 06:36 UTC — i.e., after this install.

So for the whole install window there was no arm64 ingestor image at all — the :0.3 float itself was the amd64-only a0861ea9, which image-refresh correctly tracked (no-op). Once the multi-arch build landed this morning, image-refresh picked it up on its next tick — confirmed in the live logs (latest=d361fa77 … digest changed (a0861ea9 -> d361fa77); set env). No image-refresh change needed.

Net: the durable fix is the multi-arch greenfield pin + the CI guard (both in #187); image-refresh handles ongoing rollouts correctly.

[LukasWodka]

Correction + final state. The digest fix actually landed on develop via #184 (v0.3.2 = the same multi-arch d361fa77 index), not this PR — so my note above overstated #187's scope.

#187 is now guard-only: it adds the ingestor-multiarch helm-ci job and no longer touches values.yaml (the digest is already correct on develop). The guard ran green in CI (Platforms: linux/amd64 linux/arm64 → OK) and fails on the old amd64-only a0861ea9.

So the split is: digest = fixed on develop (#184); regression guard = #187 (mergeable, all checks green, awaiting Asad's review).

[saadqbal]

This is now fixed by the v0.3.2 ingestor release + v1.4.4 chart, shipped earlier today:

• data-ingestors v0.3.2 is the first multi-arch ingestor image (linux/amd64 + linux/arm64), digest sha256:d361fa77…. The 0.3 float tag now points at it, so image-refresh has already rolled live clusters (including arm64) to the multi-arch digest on its ~15-min tick — no kubectl action needed.
• client v1.4.4 (chore(#184): pin greenfield ingestor baseline to v0.3.2 #185 → Sync develop → main for v1.4.4 chart release #188 → release) bumps the pinned greenfield baseline images.ingestor.digest from the amd64-only sha256:a0861ea9… (v0.3.1) to the multi-arch sha256:d361fa77…, so fresh installs on Apple Silicon / Graviton no longer ImagePullBackOff.

Heads-up: this overlapped with #187 — apologies for the parallel work, we were both on this within the same few minutes. Suggested close-out: confirm jobs-manager's INGESTOR_IMAGE_DIGEST env now resolves to d361fa77… on the affected berlin-team install (image-refresh should have set it), verify an ingestion run succeeds, then close. The ingestor-multiarch CI guard in #187 is the piece still worth landing so this can't regress.