Repository navigation
feat: push per-arch images in parallel before manifest list creation - #27
yashvardhannanavati wants to merge 2 commits into
Conversation
PR Summary by QodoPush per-arch images in parallel before multi-arch manifest creation
AI Description
Diagram
High-Level Assessment
Files changed (2)
|
|
🤖 Finished Review · ✅ Success · Started 2:59 AM UTC · Completed 3:16 AM UTC |
6270628 to
d3f49fb
Compare
| @retry( | ||
| before_sleep=before_sleep_log(logger, logging.WARNING), | ||
| reraise=True, | ||
| retry=retry_if_exception_type(ExternalServiceError), | ||
| stop=stop_after_attempt(3), | ||
| wait=wait_exponential(multiplier=2), | ||
| ) | ||
| def _push_image(self, image: str) -> None: | ||
| """Push a single architecture image to the registry.""" | ||
| logger.info("Pushing %s to registry", image) | ||
| run_cmd( | ||
| ["buildah", "push", "--tls-verify=true", image, f"docker://{image}"], | ||
| exc_msg=f"Failed to push {image}", | ||
| ) |
There was a problem hiding this comment.
1. Push retry never triggers 🐞 Bug ☼ Reliability
_push_image retries only ExternalServiceError, but run_cmd usually raises IIBError for buildah push failures, so transient registry/network push errors will not be retried and can make builds flaky.
Agent Prompt
### Issue description
`MultiArchBuilder._push_image()` is decorated to retry only `ExternalServiceError`, but `run_cmd()` raises `IIBError` for most `buildah push` failures. As a result, the new per-arch push step will often fail immediately without retrying, undermining the intended resiliency.
### Issue Context
`run_cmd()` only converts some buildah stderr patterns to `ExternalServiceError` (currently focused on build-container creation and a closed-pipe case). `buildah push` failures typically fall through to `IIBError`.
### Fix Focus Areas
- task/iib-image-builder-oci-ta/multi-arch-builder.py[75-131]
- task/iib-image-builder-oci-ta/multi-arch-builder.py[572-586]
- task/iib-image-builder-oci-ta/multi-arch-builder.py[748-766]
### Suggested fix
Either:
1) Broaden `_push_image` retry predicate to include `IIBError` (or a narrower subclass/marker for retryable push errors), **or**
2) Enhance `run_cmd()`'s buildah error classification to raise `ExternalServiceError` for retryable `buildah push` network/5xx conditions, keeping `_push_image` retry on `ExternalServiceError`.
Also consider surfacing the final exception details in the aggregated error to aid debugging (e.g., include `img -> str(exc)` in the raised message).
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| # Push per-arch images to registry in parallel so manifest push is fast | ||
| logger.info("Pushing per-arch images to registry in parallel") | ||
| with ThreadPoolExecutor(max_workers=len(platform_images)) as executor: | ||
| futures = { |
There was a problem hiding this comment.
2. Zero-worker threadpool crash 🐞 Bug ☼ Reliability
build_all constructs ThreadPoolExecutor(max_workers=len(platform_images)), which raises ValueError when platform_images is empty; this can happen if MultiArchBuilder is used with BuildConfig.platforms=[], producing an unhandled exception type instead of a clear IIBError.
Agent Prompt
### Issue description
`ThreadPoolExecutor(max_workers=len(platform_images))` raises `ValueError` when `platform_images` is empty. While the main `load_config_from_env()` path rejects empty `arches`, `BuildConfig` itself has no validation, so other call sites (or future refactors) can hit this and crash with an unhandled `ValueError`.
### Issue Context
`arches_from_metadata()` enforces at least one arch when using IIB metadata, but `BuildConfig.platforms` is a plain list field and can be empty when constructed programmatically.
### Fix Focus Areas
- task/iib-image-builder-oci-ta/multi-arch-builder.py[136-159]
- task/iib-image-builder-oci-ta/multi-arch-builder.py[288-314]
- task/iib-image-builder-oci-ta/multi-arch-builder.py[735-766]
### Suggested fix
Add an explicit guard before creating the executor, e.g.:
- if `not platform_images: raise IIBError("No platform images built; cannot push")`
This keeps failures consistent with the rest of the builder error handling and avoids leaking a low-level `ValueError`.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
ReviewFindingsHigh
Medium
Low
Labels: PR introduces a performance enhancement to the Python build orchestrator with moderate risk score Next steps:
Previous runReview — ApprovePR: feat: push per-arch images in parallel before manifest list creation SummaryThis PR optimizes the multi-arch image build pipeline by pushing per-architecture images to the registry in parallel (via Changes reviewed
AnalysisCorrectness ✅
Security ✅
Intent & scope ✅
Test coverage ✅
Findings
Previous run (2)Review — commentPR: #27 — feat: push per-arch images in parallel before manifest list creation SummaryThis PR parallelizes per-arch image builds and pushes using Findings1 · Nested retry causes unnecessary rebuilds on push failure
This means a pure push failure triggers up to 3 full rebuilds and 9 total push attempts. The rebuilds are unnecessary work. Remediation: Either move 2 ·
|
| Severity | Category | File | Line |
|---|---|---|---|
| medium | test-integrity | tests/unit/test_multi_arch_builder.py |
475 |
This test was not updated to mock _push_image. After the PR's changes, _build_image calls run_cmd twice: once for buildah bud and once (via _push_image) for buildah push. The test checks mock_run_cmd.call_args[0][0] — the last call — which is now the push command, not the build command. The assertion "BINARY_IMAGE=" not in cmd passes trivially because the push command never contains BINARY_IMAGE=.
Remediation: Add @patch("multi_arch_builder.MultiArchBuilder._push_image") to this test (consistent with the other TestBuildImage tests that were updated) so that call_args captures the build command.
3 · No unit tests for isolated storage flag construction
| Severity | Category | File | Line |
|---|---|---|---|
| medium | test-adequacy | tests/unit/test_multi_arch_builder.py |
493 |
The core new feature — buildah --root/--runroot flags, skopeo [overlay@...] container-storage prefix, and buildah push with storage flags — has no positive-path unit test. All parallel build tests either patch _build_image entirely or only exercise the error path. A regression in the storage flag construction (e.g., wrong index into storage_flags, missing overlay prefix) would go undetected.
Remediation: Add a test that calls _build_image.__wrapped__(builder, "amd64", "img:amd64", "/tmp/storage") with mocked run_cmd, _verify_image_architecture, and _push_image, then asserts that --root and --runroot flags appear in the buildah bud command. Similarly, add a test for _verify_image_architecture with non-empty storage_flags to verify the [overlay@...] prefix.
4 · Build storage created inside the Docker build context
| Severity | Category | File | Line |
|---|---|---|---|
| low | correctness | task/iib-image-builder-oci-ta/multi-arch-builder.py |
800 |
build_base is set to os.path.join(self.config.context_path, ".build-storage"). This places the isolated buildah storage inside the directory that buildah bud uses as its build context. The directories are mostly empty when buildah bud runs (actual image data lands there during the build), and the current Dockerfiles use selective COPY rather than COPY ., so this is low-risk. However, if the Dockerfile pattern changes, this directory would be sent as part of the build context or even copied into the image.
Remediation: Consider placing build_base outside the context path — e.g., in a sibling directory or under /tmp.
5 · _verify_manifest_list accepts unused platform_images parameter
| Severity | Category | File | Line |
|---|---|---|---|
| low | style | task/iib-image-builder-oci-ta/multi-arch-builder.py |
620 |
The method signature accepts platform_images: List[str] but the body uses self.config.platforms and self.config.arch_map exclusively. The parameter is never referenced.
Remediation: Remove the parameter or use it to derive expected architectures instead of self.config.platforms.
6 · Test misplacement and near-duplication
| Severity | Category | File | Line |
|---|---|---|---|
| low | style | tests/unit/test_multi_arch_builder.py |
558 |
test_parallel_build_push_reports_all_failures lives in TestVerifyManifestList but exercises build_all(), not _verify_manifest_list. TestParallelBuildErrorCollection (line 605) contains a near-duplicate test for the same behavior using a slightly different mocking strategy.
Remediation: Move the build_all error-collection test out of TestVerifyManifestList and consolidate with TestParallelBuildErrorCollection.
Previous run (3)
Review
Verdict: comment · 1 medium, 1 low finding
Summary
This PR introduces parallel per-arch image builds and pushes via ThreadPoolExecutor, adds a pre-push step for per-arch images before manifest list creation, replaces the second manifest list build+push with a skopeo copy --all for additional tags, and adds a _verify_manifest_list guard. The approach is sound and well-tested — the core logic, parallel error collection, and manifest verification are all correct. Tests comprehensively cover the new methods and the updated flow.
Findings
1. _push_image retry decorator will not trigger on most push failures — medium
File: task/iib-image-builder-oci-ta/multi-arch-builder.py (lines 571–578)
_push_image uses retry_if_exception_type(ExternalServiceError), matching the pattern from _build_image. However, run_cmd only raises ExternalServiceError for buildah errors matching specific regex patterns designed for buildah bud failures ("error creating build container" with 403/50x status codes, or "read/write on closed pipe"). For buildah push failures, the error messages differ (e.g., "error pushing image", "error copying layers"), so neither regex matches and run_cmd falls through to raise IIBError(exc_msg) instead.
This means the tenacity retry is effectively a no-op for transient registry errors during push — exactly the scenario where retry is most valuable.
Remediation: Either:
- Change the retry to
retry_if_exception_type(IIBError)(consistent with_create_and_push_manifest_list's retry strategy), or - Extend the
run_cmdnetwork-error regexes to coverbuildah pusherror patterns and raiseExternalServiceErrorfor those
2. skopeo copy missing explicit --tls-verify=true — low
File: task/iib-image-builder-oci-ta/multi-arch-builder.py (lines 681–690)
The new skopeo copy --all command for copying manifests to additional tags omits --tls-verify=true. Skopeo defaults to TLS verification, so this is not a security issue. However, every other registry-facing command in the codebase (buildah push, buildah bud, buildah manifest push) explicitly sets --tls-verify=true. Adding it here would maintain consistency and defend against configuration-level overrides.
Remediation: Add "--tls-verify=true" to the skopeo copy command.
Notes
- Thread safety:
_build_imageand_push_imageare safe to call fromThreadPoolExecutorthreads — they only read fromself.config(immutable during build) and spawn independent subprocesses. Buildah'scontainers/storagelayer uses file locking for concurrent access. - Test organization:
test_parallel_push_reports_all_failuresis placed underTestVerifyManifestListbut tests push-error aggregation inbuild_all(). Minor organizational nit. - skopeo copy optimization: Using
skopeo copy --allfor the secondary tag instead of building+pushing a second manifest list is a good optimization — it performs a server-side copy rather than re-uploading all layers.
Previous run (4)
Review
Reason: stale-head
The review agent reviewed commit 62706281a66405401b5a4285deca54c8d1c07ef6 but the PR HEAD is now d3f49fb9335046ff9867a76f9a069f661101294c. This review was discarded to avoid approving unreviewed code.
|
/fs-review |
|
🤖 Finished Review · ✅ Success · Started 4:06 AM UTC · Completed 4:11 AM UTC |
8672be0 to
aa0b117
Compare
|
🤖 Finished Review · ✅ Success · Started 5:01 AM UTC · Completed 5:09 AM UTC |
aa0b117 to
4478140
Compare
|
🤖 Finished Review · ✅ Success · Started 6:11 AM UTC · Completed 6:16 AM UTC |
| error_msg = str(e) | ||
| if "Manifest list not found locally." not in error_msg: | ||
| raise IIBError(f"Error removing local manifest list: {error_msg}") | ||
| logger.debug( |
There was a problem hiding this comment.
Isn't the order incorrect? and should not be here error instead of debug?
There was a problem hiding this comment.
Or this is just in case when the message is different and exception will not be raised?
…push Speed up multi-arch builds by pushing per-arch images to the registry in parallel via ThreadPoolExecutor before manifest list creation. This pre-populates registry layers so `buildah manifest push --all` completes faster (~4.5 min saved). For the second manifest tag (commit SHA), use `skopeo copy --all` to copy the manifest list server-side instead of rebuilding it from scratch. Additional robustness improvements: - Add `_verify_manifest_list` to confirm all expected architectures are present in the local manifest list before pushing - Broaden HTTP error regex in `run_cmd` so push failures (403, 5xx) are classified as retryable `ExternalServiceError` - Add 3600s timeout to `_push_image` - Use `docker://` prefix in `buildah manifest add` for remote images - Collect all parallel push errors before raising Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
4478140 to
e6087f1
Compare
|
🤖 Finished Review · ✅ Success · Started 8:38 PM UTC · Completed 9:04 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $7.46 |
|
<!-- fullsend:risk-assessment --> DetailsModerate risk driven by a large blast radius from parallelism changes to the core build orchestration path, partially offset by good test coverage (50% test file ratio), no security-sensitive or CI workflow changes, and a known non-first-time author. |
There was a problem hiding this comment.
Note: The following review comments could not be posted on the diff (GitHub returned 422) and are included here instead:
task/iib-image-builder-oci-ta/multi-arch-builder.py(file-level): Line 671 ·
[high] logic-error
Retry regression in _create_and_push_manifest_list: the broadened network regex (line 141) now classifies transient HTTP errors (e.g. 503) as ExternalServiceError, but this method's retry decorator only catches IIBError. Before this PR, a 503 on buildah manifest push would fall through to raise IIBError(exc_msg) (retried). Now it raises ExternalServiceError (not retried), silently removing retry protection for transient registry errors during manifest operations.
Suggested fix: Change the retry decorator to retry=retry_if_exception_type((IIBError, ExternalServiceError)) or retry=retry_if_exception_type(IIBBaseException) to cover the full exception hierarchy.
task/iib-image-builder-oci-ta/multi-arch-builder.py(file-level): Line 184 ·
[low] tier-mismatch
Commits prefixed feat: include bug-fix-tier changes (rsplit fix, duplicate detection) that should use fix: prefix for accurate commit tier labeling.
Suggested fix: Separate bug fixes into fix:-prefixed commits.
| ) -> None: | ||
| pushed_digests: dict[str, str], | ||
| ) -> str: | ||
| """ |
There was a problem hiding this comment.
[medium] doc-style
The docstring for _create_and_push_manifest_list was not updated: the new pushed_digests parameter and the changed return type (str instead of None) are undocumented.
Suggested fix: Add :param dict pushed_digests:, :return:, and :rtype: str to the docstring.
| "Manifest list cannot be removed. No manifest list %s found", output_pull_spec | ||
| ) | ||
| logger.info("Creating the manifest list %s locally", output_pull_spec) | ||
| primary_pull_spec = f"{image_name_repo}:{_tags[0]}" |
There was a problem hiding this comment.
[medium] design-direction
The multi-tag manifest strategy was silently redesigned from an independent manifest-list-per-tag loop to push-primary plus skopeo copy --preserve-digests. Tags now share a single manifest digest rather than being independently built. This behavioral change has registry semantics implications not documented in the PR body.
Suggested fix: Document this strategy change explicitly in the PR body and explain why shared-digest aliasing is preferable to independent manifest creation.
| for m in manifests | ||
| ) | ||
| expected = sorted( | ||
| ("linux", self.config.arch_map.get(platform, platform), pushed_digests[image]) |
There was a problem hiding this comment.
[low] edge-case
_verify_manifest_list uses self.config.platforms for arch_map lookup, but build_all() normalizes platforms only to a local variable. Whitespace-padded platforms would cause false verification failures, though mitigated in practice by arches_from_metadata already stripping.
Suggested fix: Normalize self.config.platforms in place at the start of build_all() so all downstream consumers use clean platform names.
| # Check for transient network/registry errors on any buildah operation | ||
| network_regexes = [ | ||
| r".*([e,E]rror:? creating build container).*(:?(403|50[0-9]|125)\s?.*$)", | ||
| r".*((?:403 Forbidden|50[0-9] \w+).*$)", |
There was a problem hiding this comment.
[low] edge-case
The new regex classifies HTTP 403 as transient ExternalServiceError and retries it, but 403 Forbidden typically indicates permanent authorization failure.
| import subprocess | ||
| import sys | ||
| import tempfile | ||
| from concurrent.futures import ThreadPoolExecutor, as_completed |
There was a problem hiding this comment.
[low] code-organization
Import ordering: from typing import Any placed before from dataclasses breaks the established alphabetical convention for from-import lines.
Suggested fix: Reorder the from-import block: concurrent.futures, dataclasses, pathlib, typing.
| wait=wait_exponential(multiplier=2), | ||
| ) | ||
| def _push_image(self, image: str) -> str: | ||
| """Push an architecture image and return its remote manifest digest.""" |
There was a problem hiding this comment.
[low] doc-style
_push_image uses a single-line docstring; the codebase convention is multi-line Sphinx format with :param, :return:, :rtype:, and :raises: entries.
Suggested fix: Expand the docstring to multi-line Sphinx style with parameter, return, and exception documentation.
| pushed_digests: dict[str, str], | ||
| remote_digest: str | None = None, | ||
| ) -> None: | ||
| """Verify a local or published list points to this run's pushed images.""" |
There was a problem hiding this comment.
[low] doc-style
_verify_manifest_list uses a single-line docstring for a four-parameter method; the codebase convention is multi-line Sphinx format.
Suggested fix: Expand to multi-line Sphinx style with :param:, :raises: entries.
Superseded by updated review
Per-arch images were only built locally and never pushed to the registry before manifest list creation. This meant
buildah manifest push --allhad to push all architecture image layers for the first time (~4.5 min), then repeat for the second tag. Now per-arch images are pushed in parallel via ThreadPoolExecutor before manifest creation, so--allfinds layers already present and only pushes the manifest list metadata.