Repository navigation
Description
Summary
The RustFS test fixture (hack/run-rustfs.sh) fails intermittently on the OpenShift CI
jobs (e2e-openshift, e2e-openshift-helm). The rustfs-bucket-init Job never completes
within its activeDeadlineSeconds: 300 and the step aborts before any VolSync e2e test
runs.
Since 2026-10-19 this is the single largest source of failures in those jobs — roughly
one run in three. It is not pool-specific and is unrelated to the PR under test.
Impact
Failure counts scraped from Prow (test-platform-results-public, job history since
2026-10-19), classified by failed step and log signature:
| Job | Runs | Failures | bucket-init failures |
|---|---|---|---|
pull-ci-backube-volsync-main-e2e-openshift |
49 | 10 | 6 |
pull-ci-backube-volsync-main-e2e-openshift-helm |
49 | 9 | 5 (+2 image-pull stalls) |
pull-ci-backube-volsync-main-e2e-openshift-fips |
50 | 15 | ~0 (cluster-pool / OLM failures instead) |
Affected clusters/pools include app-prow-small-aws-416-9-west2, -417-0-east2,
-420-west2, -421-west2, and app-prow-small-fips-aws-415-51-east2.
Observed behaviour
Every failure occurs in the first (non-TLS) ./hack/run-rustfs.sh invocation, right
after the chart installs successfully (so helm --wait reported the RustFS server Ready):
+ ./hack/run-rustfs.sh
===> Detected OpenShift; allowing SCC-assigned IDs and groups <===
Release "rustfs" does not exist. Installing it now.
NAME: rustfs
NAMESPACE: rustfs
STATUS: deployed
job.batch/rustfs-bucket-init-1792786234-62 created
error: timed out waiting for the condition on jobs/rustfs-bucket-init-1792786234-62
error: timed out waiting for the condition
...
Pods Statuses: 0 Active (0 Ready) / 0 Succeeded / 1 Failed
Events:
Normal SuccessfulCreate 5m21s job-controller Created pod: rustfs-bucket-init-...-svrw8
Normal SuccessfulDelete 21s job-controller Deleted pod: rustfs-bucket-init-...-svrw8
Warning DeadlineExceeded 20s job-controller Job was active longer than specified deadline
job.batch "rustfs-bucket-init-..." deleted
There are two distinct sub-modes:
bucket-initcontainer hangs (majority).kubectl logs job/...prints nothing,
which distinguishes this from a pod that never started (in theContainerCreating
case the same command prints
Error from server (BadRequest): container "bucket-init" in pod "..." is waiting to start: ContainerCreating).
The container therefore ran for the full 300 s and produced no output. Because every
rccall discards stderr (>/dev/null 2>&1) and the retry loop only sleeps/fails
between calls, a singlercinvocation must have blocked indefinitely — the
30-attempt retry budget never got a chance to fire — and the Job was SIGKILLed by
activeDeadlineSeconds.- Image pull stall. In some runs the pod is stuck in
ContainerCreatingfor the
entire 5 minutes (e.g.rustfs/rc:v0.1.36never finishes pulling on a freshly
provisioned cluster) and the deadline kills it.
In both cases the second (TLS) invocation on the same cluster always succeeds, which is
consistent with images already being cached at that point.
Suspected trigger
The first bucket-init hangs appear on 2026-10-20. On 2026-10-19,
6ce40288 (#2156, Renovate) bumped:
RUSTFS_SERVER_VERSION1.0.1 -> 1.0.3RUSTFS_CHART_VERSION1.0.0 -> 1.0.3
Runs between 2026-10-15 (when the RustFS fixture landed, #2152) and 2026-10-19 had no
bucket-init hangs. This correlation is a lead, not proven — the fixture currently
discards exactly the evidence needed to confirm it (see below).
Why the current failure mode is hard to debug
hack/run-rustfs.sh has three problems that turn a slow/hung dependency into an opaque
5-minute failure:
activeDeadlineSeconds: 300with no per-command timeout, so a hungrcburns the
whole budget instead of being retried (the retry loop never advances).rcoutput is discarded (>/dev/null 2>&1), and the failure path only prints
kubectl logs job/...+describe job. There is no pod event dump, no container
state/reason, no--previouslogs, and no RustFS server logs — so CI logs cannot
distinguish "DNS failure", "connection refused", "403 on credentials", or "server
accepted the connection and never replied".- The
kubectl waittimeout (300 s) equals the Job'sactiveDeadlineSeconds(300 s),
so the wait always loses the race, and the Job is deleted before any useful state is
captured.
Proposal
In hack/run-rustfs.sh (initialize_bucket):
- Bound every CLI call, e.g.
timeout 15 rc alias set ...,timeout 15 rc mb ...,
timeout 15 rc ls ..., keeping the existing attempt budget. A transient hang then
becomes a retry instead of a hard failure. - Stop discarding
rcstderr. Capture it to a file on the pod's existing/tmp
emptyDir and print it when the attempts are exhausted, so the real error is visible. - On failure, dump actionable diagnostics before deleting anything:
kubectl -n <ns> get pods -o wide,describe pod(including container state /
waiting reason),logs+logs --previous, namespace events, and the RustFS server
pod logs. - Make the
kubectl waittimeout longer thanactiveDeadlineSeconds(e.g. deadline
300 s / wait 360 s) so the wait observes the failure condition rather than racing
it, and delete the Job last. - Handle the image-pull sub-mode explicitly: either pre-pull
rustfs/rc:${RUSTFS_CLI_VERSION}before creating the Job, or detect a pod that has
never started and report the pull status (ErrImagePull/ImagePullBackOff/
pull latency) instead of a silent deadline expiry. - Consider raising
activeDeadlineSecondsonce bounded retries are in place.
Open questions
- Does
rc(rustfs/rustfs CLIv0.1.36) have an internal connect/read timeout? A client
that can block forever on an unreachable or unresponsive endpoint is worth reporting
upstream torustfs/rustfsregardless of this fixture. - Is the 1.0.1 -> 1.0.3 server/chart bump the actual trigger? The diagnostics above
should answer this on the first post-fix failure; a quick check would be to pin back to
1.0.1/1.0.0 and see whether the hangs stop.
Related observations (not this issue)
e2e-openshift-fipsmostly fails earlier, inocm-e2e-clusterpool-checkout
(ClusterNotReady - [Pending: True:NoClusters],Claim provision timed out) — cluster
pool capacity, not repo code./retestis the remedy.- Two
e2e-install-operatorfailures on 2026-10-22 hit
install plan is not available for the subscription volsync-v0-17-0-sub: timed out waiting for the conditionafter 15 m (OLM flake).
(AI generated)
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsDone
Activity
fix: bound and diagnose RustFS bucket-init hangs