Skip to content

feat: push per-arch images in parallel before manifest list creation - #27

Open
yashvardhannanavati wants to merge 2 commits into
release-engineering:mainfrom
yashvardhannanavati:feat/parallel-push-per-arch-images
Open

yashvardhannanavati wants to merge 2 commits into
release-engineering:mainfrom
yashvardhannanavati:feat/parallel-push-per-arch-images

Conversation

@yashvardhannanavati

Copy link
Copy Markdown
Collaborator

Per-arch images were only built locally and never pushed to the registry before manifest list creation. This meant buildah manifest push --all had to push all architecture image layers for the first time (~4.5 min), then repeat for the second tag. Now per-arch images are pushed in parallel via ThreadPoolExecutor before manifest creation, so --all finds layers already present and only pushes the manifest list metadata.

@qodo-for-releng

Copy link
Copy Markdown

PR Summary by Qodo

Push per-arch images in parallel before multi-arch manifest creation

✨ Enhancement 🧪 Tests 🕐 40+ Minutes

Grey Divider

AI Description

• Push per-architecture images to the registry before creating the manifest list.
• Parallelize per-arch pushes to avoid duplicated layer uploads across tags.
• Add unit coverage for push behavior and parallel failure aggregation.
Diagram

graph TD
  A["MultiArchBuilder.build_all"] --> B["Build per-arch images"] --> C["Parallel push (ThreadPool)"] --> D["buildah push (per-arch)"] --> E["Create/push manifest list"] --> R[("Container registry")]
  D --> R
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Keep using only `buildah manifest push --all` (status quo)
  • ➕ Simpler control flow (no explicit per-arch pushes, no concurrency).
  • ➖ Slow when multiple tags are pushed because each tag re-uploads per-arch layers the first time.
  • ➖ Less explicit error reporting per-arch before manifest creation.
2. Use `skopeo copy --all` for multi-arch copy/push
  • ➕ Purpose-built for copying multi-arch images between transports/registries.
  • ➕ Potentially simpler retry semantics at the command level.
  • ➖ Introduces/leans on a different tool/behavior than the existing buildah-based flow.
  • ➖ Would still need a strategy to avoid duplicated uploads across tags depending on how tags are handled.
3. Sequential per-arch pushes (no ThreadPoolExecutor)
  • ➕ Easier to reason about; simpler logging order.
  • ➖ Doesn’t reduce wall-clock time as effectively; retains avoidable latency in CI/task execution.

Recommendation: The PR’s approach is the best fit: explicitly pushing per-arch images first makes the subsequent manifest push largely metadata-only, and parallelizing the pushes reduces wall-clock time. The added retry and aggregated error reporting improve robustness while keeping the overall buildah-centric workflow unchanged.

Files changed (2) +99 / -0

Enhancement (1) +36 / -0
multi-arch-builder.pyParallel per-arch registry pushes before manifest publication +36/-0

Parallel per-arch registry pushes before manifest publication

• Introduces a retried '_push_image()' helper that runs 'buildah push' to upload a single architecture image. Updates 'build_all()' to push all per-arch images to the registry concurrently and abort manifest creation if any push fails, reporting all failed images.

task/iib-image-builder-oci-ta/multi-arch-builder.py

Tests (1) +63 / -0
test_multi_arch_builder.pyUnit tests for '_push_image' and parallel push failure handling +63/-0

Unit tests for '_push_image' and parallel push failure handling

• Adds tests validating the exact 'buildah push' command and retry-surface error behavior for '_push_image'. Extends 'build_all()' tests to patch the new push step and verifies parallel push aggregates failures and prevents manifest creation on push errors.

tests/unit/test_multi_arch_builder.py

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 15, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 2:59 AM UTC · Completed 3:16 AM UTC
Commit: 8a575a9 · View workflow run →

@yashvardhannanavati
yashvardhannanavati force-pushed the feat/parallel-push-per-arch-images branch from 6270628 to d3f49fb Compare July 15, 2026 03:00
@qodo-for-releng

qodo-for-releng Bot commented Jul 15, 2026 •

Copy link
Copy Markdown

Code Review by Qodo


Action required

1. Manifest list pushes no longer retry on registry 5xx/403 🐞 Bug ⭐ New
Description
The new buildah regex in run_cmd makes buildah manifest push/add/create raise
ExternalServiceError on 403/5xx output, but _create_and_push_manifest_list only retries on
retry_if_exception_type(IIBError), and ExternalServiceError is a sibling of IIBError under
IIBBaseException, not a subclass. Before this change those transient failures raised IIBError
and were retried up to 3 times; now a single 500/503 from the registry during the manifest list push
fails the whole build on the first attempt.
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`_create_and_push_manifest_list` retries only on `IIBError`. Because of the broadened buildah regex in `run_cmd`, transient registry errors during `buildah manifest push` now raise `ExternalServiceError`, which is not a subclass of `IIBError`, so they are no longer retried.

## Fix Focus Areas
- task/iib-image-builder-oci-ta/multi-arch-builder.py[668-674]
- task/iib-image-builder-oci-ta/multi-arch-builder.py[135-147]

## Recommended Fix
Change the decorator to `retry=retry_if_exception_type((IIBError, ExternalServiceError))`. Another option is to retry only on `ExternalServiceError` and stop retrying the deterministic verification `IIBError`s. Add a unit test where `buildah manifest push` fails once with a `503 Service Unavailable` stderr and then succeeds.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

2. Push retry never triggers 🐞 Bug
Description
_push_image retries only ExternalServiceError, but run_cmd usually raises IIBError for buildah push
failures, so transient registry/network push errors will not be retried and can make builds flaky.
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`MultiArchBuilder._push_image()` is decorated to retry only `ExternalServiceError`, but `run_cmd()` raises `IIBError` for most `buildah push` failures. As a result, the new per-arch push step will often fail immediately without retrying, undermining the intended resiliency.

### Issue Context
`run_cmd()` only converts some buildah stderr patterns to `ExternalServiceError` (currently focused on build-container creation and a closed-pipe case). `buildah push` failures typically fall through to `IIBError`.

### Fix Focus Areas
- task/iib-image-builder-oci-ta/multi-arch-builder.py[75-131]
- task/iib-image-builder-oci-ta/multi-arch-builder.py[572-586]
- task/iib-image-builder-oci-ta/multi-arch-builder.py[748-766]

### Suggested fix
Either:
1) Broaden `_push_image` retry predicate to include `IIBError` (or a narrower subclass/marker for retryable push errors), **or**
2) Enhance `run_cmd()`'s buildah error classification to raise `ExternalServiceError` for retryable `buildah push` network/5xx conditions, keeping `_push_image` retry on `ExternalServiceError`.

Also consider surfacing the final exception details in the aggregated error to aid debugging (e.g., include `img -> str(exc)` in the raised message).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


View medium (1)
Remediation recommended
3. Ordinary build output can trigger needless retries 🐞 Bug ⭐ New
Description
The new pattern .*((?:403 Forbidden|50[0-9] \w+).*$) matches any stderr line containing a number
from 500 to 509 followed by a word (for example "500 packages" or "504 bytes"), and it applies to
every buildah command, including buildah bud. _build_image and _push_image retry on
ExternalServiceError, so a failed build whose output happens to contain such a number is classed
as a transient network error. That build is then re-run up to 3 times, each with a 3600s timeout,
and the error message points at a misleading cause.
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The broadened buildah network regex matches any line containing `50X <word>`, so permanent build failures can be classified as retryable `ExternalServiceError`.

## Fix Focus Areas
- task/iib-image-builder-oci-ta/multi-arch-builder.py[137-143]

## Recommended Fix
Match only real HTTP status text, e.g. `r".*((?:status code|received unexpected HTTP status|unexpected status)[^\n]*?(?:403 Forbidden|50[0-9] (?:Internal Server Error|Bad Gateway|Service Unavailable|Gateway Timeout)).*$)"`, or limit the new regex to `push`/`manifest` subcommands. Add a negative test where a stderr line such as `500 packages` does not produce an `ExternalServiceError`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


View low (2)
Informational
4. Parallel push failures lose their cause 🐞 Bug ⭐ New
Description
The parallel push loop in build_all logs each failure only as "%s: %s" with logger.error and
then raises a new IIBError that lists only the image names, with no from chaining and no
traceback. When a push fails, operators see only the stringified message for each image, and the
original ExternalServiceError, TimeoutExpired or OSError type is lost.
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Parallel push errors are logged without tracebacks, and the raised `IIBError` drops the original exceptions.

## Fix Focus Areas
- task/iib-image-builder-oci-ta/multi-arch-builder.py[895-904]

## Recommended Fix
Use `logger.error(..., exc_info=e)` for each failure. Include `str(e)` for each image in the aggregate message, and raise `IIBError(...) from next(iter(errors.values()))`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

5. Zero-worker threadpool crash 🐞 Bug
Description
build_all constructs ThreadPoolExecutor(max_workers=len(platform_images)), which raises ValueError
when platform_images is empty; this can happen if MultiArchBuilder is used with
BuildConfig.platforms=[], producing an unhandled exception type instead of a clear IIBError.
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`ThreadPoolExecutor(max_workers=len(platform_images))` raises `ValueError` when `platform_images` is empty. While the main `load_config_from_env()` path rejects empty `arches`, `BuildConfig` itself has no validation, so other call sites (or future refactors) can hit this and crash with an unhandled `ValueError`.

### Issue Context
`arches_from_metadata()` enforces at least one arch when using IIB metadata, but `BuildConfig.platforms` is a plain list field and can be empty when constructed programmatically.

### Fix Focus Areas
- task/iib-image-builder-oci-ta/multi-arch-builder.py[136-159]
- task/iib-image-builder-oci-ta/multi-arch-builder.py[288-314]
- task/iib-image-builder-oci-ta/multi-arch-builder.py[735-766]

### Suggested fix
Add an explicit guard before creating the executor, e.g.:
- if `not platform_images: raise IIBError("No platform images built; cannot push")`

This keeps failures consistent with the rest of the builder error handling and avoids leaking a low-level `ValueError`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


Comment on lines +572 to +585
@retry(
before_sleep=before_sleep_log(logger, logging.WARNING),
reraise=True,
retry=retry_if_exception_type(ExternalServiceError),
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=2),
)
def _push_image(self, image: str) -> None:
"""Push a single architecture image to the registry."""
logger.info("Pushing %s to registry", image)
run_cmd(
["buildah", "push", "--tls-verify=true", image, f"docker://{image}"],
exc_msg=f"Failed to push {image}",
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. Push retry never triggers 🐞 Bug ☼ Reliability

_push_image retries only ExternalServiceError, but run_cmd usually raises IIBError for buildah push
failures, so transient registry/network push errors will not be retried and can make builds flaky.
Agent Prompt
### Issue description
`MultiArchBuilder._push_image()` is decorated to retry only `ExternalServiceError`, but `run_cmd()` raises `IIBError` for most `buildah push` failures. As a result, the new per-arch push step will often fail immediately without retrying, undermining the intended resiliency.

### Issue Context
`run_cmd()` only converts some buildah stderr patterns to `ExternalServiceError` (currently focused on build-container creation and a closed-pipe case). `buildah push` failures typically fall through to `IIBError`.

### Fix Focus Areas
- task/iib-image-builder-oci-ta/multi-arch-builder.py[75-131]
- task/iib-image-builder-oci-ta/multi-arch-builder.py[572-586]
- task/iib-image-builder-oci-ta/multi-arch-builder.py[748-766]

### Suggested fix
Either:
1) Broaden `_push_image` retry predicate to include `IIBError` (or a narrower subclass/marker for retryable push errors), **or**
2) Enhance `run_cmd()`'s buildah error classification to raise `ExternalServiceError` for retryable `buildah push` network/5xx conditions, keeping `_push_image` retry on `ExternalServiceError`.

Also consider surfacing the final exception details in the aggregated error to aid debugging (e.g., include `img -> str(exc)` in the raised message).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Comment on lines +748 to +751
# Push per-arch images to registry in parallel so manifest push is fast
logger.info("Pushing per-arch images to registry in parallel")
with ThreadPoolExecutor(max_workers=len(platform_images)) as executor:
futures = {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Informational

2. Zero-worker threadpool crash 🐞 Bug ☼ Reliability

build_all constructs ThreadPoolExecutor(max_workers=len(platform_images)), which raises ValueError
when platform_images is empty; this can happen if MultiArchBuilder is used with
BuildConfig.platforms=[], producing an unhandled exception type instead of a clear IIBError.
Agent Prompt
### Issue description
`ThreadPoolExecutor(max_workers=len(platform_images))` raises `ValueError` when `platform_images` is empty. While the main `load_config_from_env()` path rejects empty `arches`, `BuildConfig` itself has no validation, so other call sites (or future refactors) can hit this and crash with an unhandled `ValueError`.

### Issue Context
`arches_from_metadata()` enforces at least one arch when using IIB metadata, but `BuildConfig.platforms` is a plain list field and can be empty when constructed programmatically.

### Fix Focus Areas
- task/iib-image-builder-oci-ta/multi-arch-builder.py[136-159]
- task/iib-image-builder-oci-ta/multi-arch-builder.py[288-314]
- task/iib-image-builder-oci-ta/multi-arch-builder.py[735-766]

### Suggested fix
Add an explicit guard before creating the executor, e.g.:
- if `not platform_images: raise IIBError("No platform images built; cannot push")`

This keeps failures consistent with the rest of the builder error handling and avoids leaking a low-level `ValueError`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 15, 2026 •

Copy link
Copy Markdown

Review

Findings

High

  • [logic-error] task/iib-image-builder-oci-ta/multi-arch-builder.py:671 — Retry regression in _create_and_push_manifest_list: the broadened network regex (line 141) now classifies transient HTTP errors (e.g. 503) as ExternalServiceError, but this method's retry decorator only catches IIBError. Before this PR, a 503 on buildah manifest push would fall through to raise IIBError(exc_msg) (retried). Now it raises ExternalServiceError (not retried), silently removing retry protection for transient registry errors during manifest operations. Both _push_image and _build_image correctly retry ExternalServiceError, making this the only method with the inconsistency.
    Remediation: Change the retry decorator to retry=retry_if_exception_type((IIBError, ExternalServiceError)).

Medium

  • [missing-test] tests/unit/test_multi_arch_builder.py — No test covers the interaction between the broadened network regex and the _create_and_push_manifest_list retry decorator. All existing tests call __wrapped__, bypassing the retry path entirely, so the regression above went undetected.
  • [doc-style] task/iib-image-builder-oci-ta/multi-arch-builder.py:680 — The docstring for _create_and_push_manifest_list was not updated: the new pushed_digests parameter and the changed return type (str instead of None) are undocumented.
  • [scope-bundling] task/iib-image-builder-oci-ta/multi-arch-builder.py — The PR bundles four independent concerns: (1) parallel push via ThreadPoolExecutor, (2) manifest tagging strategy redesign (push-primary + skopeo copy), (3) duplicate architecture detection, and (4) registry URL parsing fix (split → rsplit). Splitting these improves reviewability, revertibility, and bisectability.
  • [design-direction] task/iib-image-builder-oci-ta/multi-arch-builder.py:692 — The multi-tag manifest strategy was silently redesigned from an independent manifest-list-per-tag loop to push-primary + skopeo copy --preserve-digests. This behavioral change (tags now share a single manifest digest rather than being independently built) has registry semantics implications not documented in the PR body.
  • [incomplete-doc] task/iib-image-builder-oci-ta/README.md:10 — The build-multi-arch step description omits the new parallel per-arch push step between building and manifest list creation.
  • [incomplete-doc] README.md:26 — The task overview numbered list does not include the new parallel push step between steps 4 and 5.

Low

  • [edge-case] task/iib-image-builder-oci-ta/multi-arch-builder.py:655 — _verify_manifest_list uses self.config.platforms for arch_map lookup, but build_all() normalizes platforms only to a local variable. Whitespace-padded platforms would cause false verification failures (mitigated in practice by arches_from_metadata already stripping).
  • [edge-case] task/iib-image-builder-oci-ta/multi-arch-builder.py:141 — The new regex classifies HTTP 403 as transient ExternalServiceError and retries it, but 403 typically indicates permanent authorization failure.
  • [code-organization] task/iib-image-builder-oci-ta/multi-arch-builder.py:18 — Import ordering: from typing import Any placed before from dataclasses breaks the established alphabetical convention.
  • [doc-style] task/iib-image-builder-oci-ta/multi-arch-builder.py:600 — _push_image uses a single-line docstring; the codebase convention is multi-line Sphinx format with :param, :return:, :raises: entries.
  • [doc-style] task/iib-image-builder-oci-ta/multi-arch-builder.py:627 — _verify_manifest_list uses a single-line docstring for a four-parameter method; same Sphinx convention applies.
  • [tier-mismatch] task/iib-image-builder-oci-ta/multi-arch-builder.py:184 — Commits prefixed feat: include bug-fix-tier changes (rsplit fix, duplicate detection) that should use fix: prefix.
  • [stale-doc] README.md:143 — TestCreateAndPushManifestList coverage description references removed test test_creates_and_pushes_manifest_for_each_tag.
  • [missing-doc] README.md:143 — New test classes TestPushImage and TestVerifyManifestList not listed in the test-coverage table.
  • [incomplete-doc] README.md:147 — TestArchesFromMetadata row missing the new duplicate-architectures rejection test.
  • [incomplete-doc] task/iib-image-builder-oci-ta/README.md:40 — The arches field description omits the new duplicate-rejection validation constraint.
  • [stale-doc] CLAUDE.md:77 — Stale line-number references in CLAUDE.md and AGENTS.md for _build_image (:427→504), MultiArchBuilder (:353→423), generate_cache_locally (:295→365), and retry logic (:427/:524→504/~675).

Labels: PR introduces a performance enhancement to the Python build orchestrator with moderate risk score


Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR
Previous run

Review — Approve

PR: feat: push per-arch images in parallel before manifest list creation

Summary

This PR optimizes the multi-arch image build pipeline by pushing per-architecture images to the registry in parallel (via ThreadPoolExecutor) before manifest list creation, then using skopeo copy --all for additional tags instead of rebuilding the manifest. The changes are well-structured, correctly implemented, and thoroughly tested.

Changes reviewed

File What changed
multi-arch-builder.py New _push_image() with tenacity retry, new _verify_manifest_list(), parallel push in build_all(), skopeo copy for additional tags in _create_and_push_manifest_list(), new regex for transient buildah errors
test_multi_arch_builder.py New test classes for _push_image and _verify_manifest_list, parallel push failure aggregation test, updated manifest list tests

Analysis

Correctness ✅

  • Thread safety is sound — each _push_image call operates on a separate image; buildah push reads from local storage without write contention.
  • The tenacity retry decorator on _push_image correctly targets ExternalServiceError (transient network/registry errors), while non-retryable IIBError failures propagate immediately.
  • Error aggregation in the ThreadPoolExecutor block correctly collects all failures before raising a single descriptive IIBError.
  • The docker:// prefix on buildah manifest add is correct since per-arch images are now pushed to the registry before manifest creation — the manifest references the already-present remote layers.
  • skopeo copy --all for additional tags is a sound optimization that avoids redundant manifest construction and full re-push.
  • The new _verify_manifest_list() correctly compares found architectures against expected ones using self.config.arch_map.

Security ✅

  • TLS verification is maintained on all new buildah push commands (--tls-verify=true).
  • No new untrusted inputs or authentication changes.

Intent & scope ✅

  • Changes directly implement the stated optimization goal.
  • No unrelated behavioral changes beyond minor comment cleanup.

Test coverage ✅

  • New tests cover command construction, error propagation, manifest verification (happy path, missing arch, extra arch), and parallel failure aggregation.
  • Existing manifest list tests properly updated for the new skopeo copy flow and _verify_manifest_list call.

Findings

Low · _verify_manifest_list unused parameter
File: task/iib-image-builder-oci-ta/multi-arch-builder.py · Line ~596

The platform_images parameter is declared in the function signature but never referenced in the function body. The verification logic uses self.config.platforms (via arch_map) to determine expected architectures. Consider removing the parameter to avoid confusion, or document why it's reserved for future use.

Suggested remediation: Change signature to _verify_manifest_list(self, manifest_name: str) -> None and update the call site.

Low · Missing explicit --tls-verify=true on skopeo copy
File: task/iib-image-builder-oci-ta/multi-arch-builder.py · Line ~687

The skopeo copy --all command for additional tags omits --tls-verify=true. While skopeo defaults to TLS verification, every other registry-interacting command in this codebase (buildah push, buildah bud, _push_image) explicitly sets this flag. Adding it here would maintain consistency and defense-in-depth.

Suggested remediation: Add "--tls-verify=true" to the skopeo copy command arguments.

Previous run (2)

Review — comment

PR: #27 — feat: push per-arch images in parallel before manifest list creation
Status: Open (not draft) · Base: main · Head: aa0b1176

Summary

This PR parallelizes per-arch image builds and pushes using ThreadPoolExecutor with isolated buildah storage roots, adds a _verify_manifest_list pre-push check, and optimizes multi-tag publishing by pushing the manifest list once and using skopeo copy --all for additional tags. The production code logic is correct and the optimization strategy is sound. Findings are related to retry design and test coverage gaps.

Findings

1 · Nested retry causes unnecessary rebuilds on push failure

Severity Category File Line
medium correctness task/iib-image-builder-oci-ta/multi-arch-builder.py 593

_push_image retries up to 3× on ExternalServiceError (line 593–603). It is called inside _build_image, which also retries up to 3× on the same exception type (line 474–480). When a push persistently fails, _push_image exhausts its retries and re-raises ExternalServiceError → _build_image's retry catches it and re-runs the entire method, including a fresh buildah bud + architecture verification before retrying the push.

This means a pure push failure triggers up to 3 full rebuilds and 9 total push attempts. The rebuilds are unnecessary work.

Remediation: Either move _push_image out of _build_image (call it separately in build_all's parallel executor), or change _push_image to raise a non-retryable exception wrapper that _build_image's decorator won't catch.

2 · test_omits_binary_image_build_arg_when_unset no longer validates the build command

Severity Category File Line
medium test-integrity tests/unit/test_multi_arch_builder.py 475

This test was not updated to mock _push_image. After the PR's changes, _build_image calls run_cmd twice: once for buildah bud and once (via _push_image) for buildah push. The test checks mock_run_cmd.call_args[0][0] — the last call — which is now the push command, not the build command. The assertion "BINARY_IMAGE=" not in cmd passes trivially because the push command never contains BINARY_IMAGE=.

Remediation: Add @patch("multi_arch_builder.MultiArchBuilder._push_image") to this test (consistent with the other TestBuildImage tests that were updated) so that call_args captures the build command.

3 · No unit tests for isolated storage flag construction

Severity Category File Line
medium test-adequacy tests/unit/test_multi_arch_builder.py 493

The core new feature — buildah --root/--runroot flags, skopeo [overlay@...] container-storage prefix, and buildah push with storage flags — has no positive-path unit test. All parallel build tests either patch _build_image entirely or only exercise the error path. A regression in the storage flag construction (e.g., wrong index into storage_flags, missing overlay prefix) would go undetected.

Remediation: Add a test that calls _build_image.__wrapped__(builder, "amd64", "img:amd64", "/tmp/storage") with mocked run_cmd, _verify_image_architecture, and _push_image, then asserts that --root and --runroot flags appear in the buildah bud command. Similarly, add a test for _verify_image_architecture with non-empty storage_flags to verify the [overlay@...] prefix.

4 · Build storage created inside the Docker build context

Severity Category File Line
low correctness task/iib-image-builder-oci-ta/multi-arch-builder.py 800

build_base is set to os.path.join(self.config.context_path, ".build-storage"). This places the isolated buildah storage inside the directory that buildah bud uses as its build context. The directories are mostly empty when buildah bud runs (actual image data lands there during the build), and the current Dockerfiles use selective COPY rather than COPY ., so this is low-risk. However, if the Dockerfile pattern changes, this directory would be sent as part of the build context or even copied into the image.

Remediation: Consider placing build_base outside the context path — e.g., in a sibling directory or under /tmp.

5 · _verify_manifest_list accepts unused platform_images parameter

Severity Category File Line
low style task/iib-image-builder-oci-ta/multi-arch-builder.py 620

The method signature accepts platform_images: List[str] but the body uses self.config.platforms and self.config.arch_map exclusively. The parameter is never referenced.

Remediation: Remove the parameter or use it to derive expected architectures instead of self.config.platforms.

6 · Test misplacement and near-duplication

Severity Category File Line
low style tests/unit/test_multi_arch_builder.py 558

test_parallel_build_push_reports_all_failures lives in TestVerifyManifestList but exercises build_all(), not _verify_manifest_list. TestParallelBuildErrorCollection (line 605) contains a near-duplicate test for the same behavior using a slightly different mocking strategy.

Remediation: Move the build_all error-collection test out of TestVerifyManifestList and consolidate with TestParallelBuildErrorCollection.

Previous run (3)

Review

Verdict: comment · 1 medium, 1 low finding

Summary

This PR introduces parallel per-arch image builds and pushes via ThreadPoolExecutor, adds a pre-push step for per-arch images before manifest list creation, replaces the second manifest list build+push with a skopeo copy --all for additional tags, and adds a _verify_manifest_list guard. The approach is sound and well-tested — the core logic, parallel error collection, and manifest verification are all correct. Tests comprehensively cover the new methods and the updated flow.

Findings

1. _push_image retry decorator will not trigger on most push failures — medium

File: task/iib-image-builder-oci-ta/multi-arch-builder.py (lines 571–578)

_push_image uses retry_if_exception_type(ExternalServiceError), matching the pattern from _build_image. However, run_cmd only raises ExternalServiceError for buildah errors matching specific regex patterns designed for buildah bud failures ("error creating build container" with 403/50x status codes, or "read/write on closed pipe"). For buildah push failures, the error messages differ (e.g., "error pushing image", "error copying layers"), so neither regex matches and run_cmd falls through to raise IIBError(exc_msg) instead.

This means the tenacity retry is effectively a no-op for transient registry errors during push — exactly the scenario where retry is most valuable.

Remediation: Either:

  • Change the retry to retry_if_exception_type(IIBError) (consistent with _create_and_push_manifest_list's retry strategy), or
  • Extend the run_cmd network-error regexes to cover buildah push error patterns and raise ExternalServiceError for those

2. skopeo copy missing explicit --tls-verify=true — low

File: task/iib-image-builder-oci-ta/multi-arch-builder.py (lines 681–690)

The new skopeo copy --all command for copying manifests to additional tags omits --tls-verify=true. Skopeo defaults to TLS verification, so this is not a security issue. However, every other registry-facing command in the codebase (buildah push, buildah bud, buildah manifest push) explicitly sets --tls-verify=true. Adding it here would maintain consistency and defend against configuration-level overrides.

Remediation: Add "--tls-verify=true" to the skopeo copy command.

Notes

  • Thread safety: _build_image and _push_image are safe to call from ThreadPoolExecutor threads — they only read from self.config (immutable during build) and spawn independent subprocesses. Buildah's containers/storage layer uses file locking for concurrent access.
  • Test organization: test_parallel_push_reports_all_failures is placed under TestVerifyManifestList but tests push-error aggregation in build_all(). Minor organizational nit.
  • skopeo copy optimization: Using skopeo copy --all for the secondary tag instead of building+pushing a second manifest list is a good optimization — it performs a server-side copy rather than re-uploading all layers.
Previous run (4)

Review

Reason: stale-head

The review agent reviewed commit 62706281a66405401b5a4285deca54c8d1c07ef6 but the PR HEAD is now d3f49fb9335046ff9867a76f9a069f661101294c. This review was discarded to avoid approving unreviewed code.

@fullsend-ai-review

Copy link
Copy Markdown

/fs-review

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 15, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 4:06 AM UTC · Completed 4:11 AM UTC
Commit: 8a575a9 · View workflow run →

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review fullsend-ai-review Bot added the requires-manual-review Review requires human judgment label Jul 15, 2026
@yashvardhannanavati
yashvardhannanavati force-pushed the feat/parallel-push-per-arch-images branch from 8672be0 to aa0b117 Compare July 15, 2026 05:00
@fullsend-ai-review

fullsend-ai-review Bot commented Jul 15, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 5:01 AM UTC · Completed 5:09 AM UTC
Commit: 8a575a9 · View workflow run →

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review fullsend-ai-review Bot added requires-manual-review Review requires human judgment and removed requires-manual-review Review requires human judgment labels Jul 15, 2026
@yashvardhannanavati
yashvardhannanavati force-pushed the feat/parallel-push-per-arch-images branch from aa0b117 to 4478140 Compare July 15, 2026 06:10
@fullsend-ai-review

fullsend-ai-review Bot commented Jul 15, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 6:11 AM UTC · Completed 6:16 AM UTC
Commit: 8a575a9 · View workflow run →

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review fullsend-ai-review Bot added ready-for-merge All reviewers approved — ready to merge and removed requires-manual-review Review requires human judgment labels Jul 15, 2026
error_msg = str(e)
if "Manifest list not found locally." not in error_msg:
raise IIBError(f"Error removing local manifest list: {error_msg}")
logger.debug(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't the order incorrect? and should not be here error instead of debug?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or this is just in case when the message is different and exception will not be raised?

yashvardhannanavati and others added 2 commits November 9, 2026 13:38
…push

Speed up multi-arch builds by pushing per-arch images to the registry in
parallel via ThreadPoolExecutor before manifest list creation. This
pre-populates registry layers so `buildah manifest push --all` completes
faster (~4.5 min saved).

For the second manifest tag (commit SHA), use `skopeo copy --all` to
copy the manifest list server-side instead of rebuilding it from scratch.

Additional robustness improvements:
- Add `_verify_manifest_list` to confirm all expected architectures are
  present in the local manifest list before pushing
- Broaden HTTP error regex in `run_cmd` so push failures (403, 5xx) are
  classified as retryable `ExternalServiceError`
- Add 3600s timeout to `_push_image`
- Use `docker://` prefix in `buildah manifest add` for remote images
- Collect all parallel push errors before raising

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@yashvardhannanavati
yashvardhannanavati force-pushed the feat/parallel-push-per-arch-images branch from 4478140 to e6087f1 Compare November 9, 2026 20:36
@fullsend-ai-review

fullsend-ai-review Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 8:38 PM UTC · Completed 9:04 PM UTC

Commit: e6087f1 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $7.46

@fullsend-ai-review fullsend-ai-review Bot added risk/moderate PR risk: moderate and removed ready-for-merge All reviewers approved — ready to merge labels Nov 9, 2026
@fullsend-ai-review

Copy link
Copy Markdown

<!-- fullsend:risk-assessment -->
Risk Assessment: moderate (2/5)

Details

Moderate risk driven by a large blast radius from parallelism changes to the core build orchestration path, partially offset by good test coverage (50% test file ratio), no security-sensitive or CI workflow changes, and a known non-first-time author.

@fullsend-ai-review fullsend-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: The following review comments could not be posted on the diff (GitHub returned 422) and are included here instead:

  • task/iib-image-builder-oci-ta/multi-arch-builder.py (file-level): Line 671 ·
    [high] logic-error

Retry regression in _create_and_push_manifest_list: the broadened network regex (line 141) now classifies transient HTTP errors (e.g. 503) as ExternalServiceError, but this method's retry decorator only catches IIBError. Before this PR, a 503 on buildah manifest push would fall through to raise IIBError(exc_msg) (retried). Now it raises ExternalServiceError (not retried), silently removing retry protection for transient registry errors during manifest operations.

Suggested fix: Change the retry decorator to retry=retry_if_exception_type((IIBError, ExternalServiceError)) or retry=retry_if_exception_type(IIBBaseException) to cover the full exception hierarchy.

  • task/iib-image-builder-oci-ta/multi-arch-builder.py (file-level): Line 184 ·
    [low] tier-mismatch

Commits prefixed feat: include bug-fix-tier changes (rsplit fix, duplicate detection) that should use fix: prefix for accurate commit tier labeling.

Suggested fix: Separate bug fixes into fix:-prefixed commits.

@fullsend-ai-review fullsend-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the review comment for full details.

) -> None:
pushed_digests: dict[str, str],
) -> str:
"""

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[medium] doc-style

The docstring for _create_and_push_manifest_list was not updated: the new pushed_digests parameter and the changed return type (str instead of None) are undocumented.

Suggested fix: Add :param dict pushed_digests:, :return:, and :rtype: str to the docstring.

"Manifest list cannot be removed. No manifest list %s found", output_pull_spec
)
logger.info("Creating the manifest list %s locally", output_pull_spec)
primary_pull_spec = f"{image_name_repo}:{_tags[0]}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[medium] design-direction

The multi-tag manifest strategy was silently redesigned from an independent manifest-list-per-tag loop to push-primary plus skopeo copy --preserve-digests. Tags now share a single manifest digest rather than being independently built. This behavioral change has registry semantics implications not documented in the PR body.

Suggested fix: Document this strategy change explicitly in the PR body and explain why shared-digest aliasing is preferable to independent manifest creation.

for m in manifests
)
expected = sorted(
("linux", self.config.arch_map.get(platform, platform), pushed_digests[image])

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] edge-case

_verify_manifest_list uses self.config.platforms for arch_map lookup, but build_all() normalizes platforms only to a local variable. Whitespace-padded platforms would cause false verification failures, though mitigated in practice by arches_from_metadata already stripping.

Suggested fix: Normalize self.config.platforms in place at the start of build_all() so all downstream consumers use clean platform names.

# Check for transient network/registry errors on any buildah operation
network_regexes = [
r".*([e,E]rror:? creating build container).*(:?(403|50[0-9]|125)\s?.*$)",
r".*((?:403 Forbidden|50[0-9] \w+).*$)",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] edge-case

The new regex classifies HTTP 403 as transient ExternalServiceError and retries it, but 403 Forbidden typically indicates permanent authorization failure.

import subprocess
import sys
import tempfile
from concurrent.futures import ThreadPoolExecutor, as_completed

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] code-organization

Import ordering: from typing import Any placed before from dataclasses breaks the established alphabetical convention for from-import lines.

Suggested fix: Reorder the from-import block: concurrent.futures, dataclasses, pathlib, typing.

wait=wait_exponential(multiplier=2),
)
def _push_image(self, image: str) -> str:
"""Push an architecture image and return its remote manifest digest."""

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] doc-style

_push_image uses a single-line docstring; the codebase convention is multi-line Sphinx format with :param, :return:, :rtype:, and :raises: entries.

Suggested fix: Expand the docstring to multi-line Sphinx style with parameter, return, and exception documentation.

pushed_digests: dict[str, str],
remote_digest: str | None = None,
) -> None:
"""Verify a local or published list points to this run's pushed images."""

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] doc-style

_verify_manifest_list uses a single-line docstring for a four-parameter method; the codebase convention is multi-line Sphinx format.

Suggested fix: Expand to multi-line Sphinx style with :param:, :raises: entries.

@fullsend-ai-review
fullsend-ai-review Bot dismissed their stale review November 9, 2026 21:03

Superseded by updated review

@fullsend-ai-review fullsend-ai-review Bot added enhancement New feature or request python Pull requests that update python code labels Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request python Pull requests that update python code risk/moderate PR risk: moderate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants