Skip to content

e2e: RustFS bucket-init job hangs and times out on OpenShift CI #2182

Closed

Description

@JohnStrunk

Summary

The RustFS test fixture (hack/run-rustfs.sh) fails intermittently on the OpenShift CI
jobs (e2e-openshift, e2e-openshift-helm). The rustfs-bucket-init Job never completes
within its activeDeadlineSeconds: 300 and the step aborts before any VolSync e2e test
runs.

Since 2026-10-19 this is the single largest source of failures in those jobs — roughly
one run in three. It is not pool-specific and is unrelated to the PR under test.

Impact

Failure counts scraped from Prow (test-platform-results-public, job history since
2026-10-19), classified by failed step and log signature:

Job Runs Failures bucket-init failures
pull-ci-backube-volsync-main-e2e-openshift 49 10 6
pull-ci-backube-volsync-main-e2e-openshift-helm 49 9 5 (+2 image-pull stalls)
pull-ci-backube-volsync-main-e2e-openshift-fips 50 15 ~0 (cluster-pool / OLM failures instead)

Affected clusters/pools include app-prow-small-aws-416-9-west2, -417-0-east2,
-420-west2, -421-west2, and app-prow-small-fips-aws-415-51-east2.

Observed behaviour

Every failure occurs in the first (non-TLS) ./hack/run-rustfs.sh invocation, right
after the chart installs successfully (so helm --wait reported the RustFS server Ready):

+ ./hack/run-rustfs.sh
===> Detected OpenShift; allowing SCC-assigned IDs and groups <===
Release "rustfs" does not exist. Installing it now.
NAME: rustfs
NAMESPACE: rustfs
STATUS: deployed
job.batch/rustfs-bucket-init-1792786234-62 created
error: timed out waiting for the condition on jobs/rustfs-bucket-init-1792786234-62
error: timed out waiting for the condition
...
Pods Statuses:            0 Active (0 Ready) / 0 Succeeded / 1 Failed
Events:
  Normal   SuccessfulCreate  5m21s  job-controller  Created pod: rustfs-bucket-init-...-svrw8
  Normal   SuccessfulDelete  21s    job-controller  Deleted pod: rustfs-bucket-init-...-svrw8
  Warning  DeadlineExceeded  20s    job-controller  Job was active longer than specified deadline
job.batch "rustfs-bucket-init-..." deleted

There are two distinct sub-modes:

  1. bucket-init container hangs (majority). kubectl logs job/... prints nothing,
    which distinguishes this from a pod that never started (in the ContainerCreating
    case the same command prints
    Error from server (BadRequest): container "bucket-init" in pod "..." is waiting to start: ContainerCreating).
    The container therefore ran for the full 300 s and produced no output. Because every
    rc call discards stderr (>/dev/null 2>&1) and the retry loop only sleeps/fails
    between calls, a single rc invocation must have blocked indefinitely — the
    30-attempt retry budget never got a chance to fire — and the Job was SIGKILLed by
    activeDeadlineSeconds.
  2. Image pull stall. In some runs the pod is stuck in ContainerCreating for the
    entire 5 minutes (e.g. rustfs/rc:v0.1.36 never finishes pulling on a freshly
    provisioned cluster) and the deadline kills it.

In both cases the second (TLS) invocation on the same cluster always succeeds, which is
consistent with images already being cached at that point.

Suspected trigger

The first bucket-init hangs appear on 2026-10-20. On 2026-10-19,
6ce40288 (#2156, Renovate) bumped:

  • RUSTFS_SERVER_VERSION 1.0.1 -> 1.0.3
  • RUSTFS_CHART_VERSION 1.0.0 -> 1.0.3

Runs between 2026-10-15 (when the RustFS fixture landed, #2152) and 2026-10-19 had no
bucket-init hangs. This correlation is a lead, not proven — the fixture currently
discards exactly the evidence needed to confirm it (see below).

Why the current failure mode is hard to debug

hack/run-rustfs.sh has three problems that turn a slow/hung dependency into an opaque
5-minute failure:

  1. activeDeadlineSeconds: 300 with no per-command timeout, so a hung rc burns the
    whole budget instead of being retried (the retry loop never advances).
  2. rc output is discarded (>/dev/null 2>&1), and the failure path only prints
    kubectl logs job/... + describe job. There is no pod event dump, no container
    state/reason, no --previous logs, and no RustFS server logs — so CI logs cannot
    distinguish "DNS failure", "connection refused", "403 on credentials", or "server
    accepted the connection and never replied".
  3. The kubectl wait timeout (300 s) equals the Job's activeDeadlineSeconds (300 s),
    so the wait always loses the race, and the Job is deleted before any useful state is
    captured.

Proposal

In hack/run-rustfs.sh (initialize_bucket):

  • Bound every CLI call, e.g. timeout 15 rc alias set ..., timeout 15 rc mb ...,
    timeout 15 rc ls ..., keeping the existing attempt budget. A transient hang then
    becomes a retry instead of a hard failure.
  • Stop discarding rc stderr. Capture it to a file on the pod's existing /tmp
    emptyDir and print it when the attempts are exhausted, so the real error is visible.
  • On failure, dump actionable diagnostics before deleting anything:
    kubectl -n <ns> get pods -o wide, describe pod (including container state /
    waiting reason), logs + logs --previous, namespace events, and the RustFS server
    pod logs.
  • Make the kubectl wait timeout longer than activeDeadlineSeconds (e.g. deadline
    300 s / wait 360 s) so the wait observes the failure condition rather than racing
    it, and delete the Job last.
  • Handle the image-pull sub-mode explicitly: either pre-pull
    rustfs/rc:${RUSTFS_CLI_VERSION} before creating the Job, or detect a pod that has
    never started and report the pull status (ErrImagePull / ImagePullBackOff /
    pull latency) instead of a silent deadline expiry.
  • Consider raising activeDeadlineSeconds once bounded retries are in place.

Open questions

  • Does rc (rustfs/rustfs CLI v0.1.36) have an internal connect/read timeout? A client
    that can block forever on an unreachable or unresponsive endpoint is worth reporting
    upstream to rustfs/rustfs regardless of this fixture.
  • Is the 1.0.1 -> 1.0.3 server/chart bump the actual trigger? The diagnostics above
    should answer this on the first post-fix failure; a quick check would be to pin back to
    1.0.1/1.0.0 and see whether the hangs stop.

Related observations (not this issue)

  • e2e-openshift-fips mostly fails earlier, in ocm-e2e-clusterpool-checkout
    (ClusterNotReady - [Pending: True:NoClusters], Claim provision timed out) — cluster
    pool capacity, not repo code. /retest is the remedy.
  • Two e2e-install-operator failures on 2026-10-22 hit
    install plan is not available for the subscription volsync-v0-17-0-sub: timed out waiting for the condition after 15 m (OLM flake).

(AI generated)

Activity

  1. self-assigned this
    on Oct 23, 2026
  2. added a commit that references this issue on Oct 27, 2026
    b9dd00e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions