Repository navigation
Fix index.db namespace collisions and support divergent-tag builds - #1378
Conversation
|
🤖 Finished Review · ✅ Success · Started 11:06 AM UTC · Completed 11:24 AM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $11.00 |
PR Summary by QodoPrevent index.db collisions and support divergent-tag builds
AI Description
Diagram
High-Level Assessment
Files changed (18)
|
Code Review by Qodo
1.
|
ReviewFindingsHigh
Medium
Low
Next steps:
Previous runReviewFindingsHigh
Medium
Low
Next steps:
Previous run (2)ReviewFindingsHigh
Medium
Low
Next steps:
Previous run (3)ReviewFindingsHigh
Medium
Low
Next steps:
Previous run (4)ReviewFindingsHigh
Medium
Low
Next steps:
Previous run (5)ReviewFindingsHigh
Medium
Low
Next steps:
Previous run (6)ReviewFindingsHigh
Medium
Low
Next steps:
Previous run (7)ReviewFindingsHigh
Medium
Low
Next steps:
Previous run (8)ReviewFindingsHigh
Medium
Low
Next steps:
|
|
🤖 Finished Review · ✅ Success · Started 1:26 AM UTC · Completed 1:45 AM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $9.84 |
|
Risk Assessment: moderate (2/5) DetailsTier 1 and Tier 2 signals are unchanged from the prior assessment (23 files, 3093 lines, large blast, protected-path count 1, test ratio 0.39, 2 authors) with only one minor fix commit added, keeping the composite at 2 under the 62/38 split with no linked issue. Previous runRisk Assessment: moderate (2/5) DetailsTier 1 signals are essentially unchanged from prior assessment (23 files, 3093 lines, large blast, protected-path count 1, test ratio 0.39) and Tier 2 shows the same multi-author pattern with only a minor additional fix commit, leaving the composite at 2 under the 62/38 split with no linked issue. Previous run (2)Risk Assessment: moderate (2/5) DetailsTier 1 signals are essentially unchanged from prior assessment (23 files vs 22, 2991 lines vs 2816, same large blast radius, protected-path count of 1, test coverage ratio 0.39), and Tier 2 shows only minor additional churn (2 commits in 30d including one fix) that does not warrant a score increase; composite remains 2 under the 62/38 split with no linked issue. Previous run (3)Risk Assessment: moderate (2/5) DetailsLarge change set (2816 lines, 22 files, large blast radius) with a substantial refactor of containerized_utils.py, but mitigated by solid test coverage (41%), no security or dependency signals, a known non-bot author, and very low git-history risk since most files are newly introduced with no churn or regression history on the master branch. Previous run (4)Risk Assessment: moderate (2/5) DetailsSignals are essentially unchanged from the prior assessment (21 files, ~2,517 lines, 0.38 test ratio, no security or CI changes, low 30d churn, stable file history), yielding the same composite of ~1.90, which rounds to 2 - moderate. Previous run (5)Risk Assessment: moderate (2/5) DetailsA large refactor across 21 files with 2,525 lines changed drives a high change-size signal, but adequate test coverage (38% test ratio), zero security or CI changes, low recent churn, and no regression history across primarily stable files produce a composite score of 1.95, rounding to moderate. Previous run (6)Risk Assessment: moderate (2/5) DetailsLarge change with 2049 lines across 21 files, but mitigated by stable git history (no recent fixes), reasonable test coverage (38%), and experienced author. Previous run (7)Risk Assessment: moderate (2/5) DetailsLarge PR (19 files, 1253 lines) with one protected path and moderate test coverage (42%), mostly new files with stable existing files showing low churn and long stability (avg 236 days since last change). |
Derive the index.db artifact and ImageStream tag from the index image's manifest digest (idb-<sha256>) instead of the pullspec string. Different content never collides; identical content addressed by different pullspecs (e.g. a released mirror) shares one artifact. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve three correctness issues raised by automated reviewers on the index.db naming / divergent-tag work: 1. remote_branch_exists no longer runs git ls-remote with strict=False, which swallowed network/auth failures as "branch absent" and could misroute a request onto the divergent build path. It now raises on command failure and accepts optional token args so the check works against private repos (auth injected into the URL, kept out of errors). 2. Divergent extraction now sources index.db/configs from the digest-resolved pullspec (from_index_resolved) instead of the mutable tag, so the extracted content matches the image the request already inspected during prebuild and cannot drift if the tag moves. 3. extract_catalog_and_db_from_image only falls back to the labeled db / empty db when the hidden-db path is genuinely absent. A new FileNotFoundInImageError (subclass of IIBError) distinguishes an absent path from a real registry/OCI/layer/tar failure, which now propagates instead of silently degrading to a wrong index.db. Also add clarifying comments: the empty index.db artifact tag is intentionally namespace-agnostic, test_get_artifact_combined_tag intentionally omits the pullspec hash, and drop a misleading :raises IIBError: from get_index_tag's docstring. Additionally, update stale test expectations uncovered by rebasing this work onto the digest-identity base (a4ed53b): test_utils.py carried duplicate copies of the oras_utils pullspec tests still asserting the old {image_name}-{tag} naming without mocking get_image_digest, and test_build_containerized_rm.py mocked containerized_utils._get_name_and_tag_from_pullspec, a symbol the digest-identity refactor removed from that module. Both are test-expectation fixes only; no production code changed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Also updates build_containerized_merge.py and build_containerized_regenerate_bundle.py, which call the same push_index_db_artifact/cleanup_on_failure interfaces changed in Task 5/6 but were not enumerated in the task-7 brief; mypy caught the now-invalid call signatures. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Update the containerized README's cache-naming section and an inline comment to reflect the content-digest identity scheme (replacing the earlier pullspec-hash description). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
bddca65 to
b73a69f
Compare
|
🤖 Finished Review · ✅ Success · Started 8:46 AM UTC · Completed 9:25 AM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $10.47 |
extract_files_from_image_non_privileged reconstructed the entire root
filesystem with tarfile.extractall to copy out one subpath. That fails on
real UBI/RHEL images: the 'data' filter rejects absolute symlinks
(/etc/alternatives, AbsoluteLinkError), 'tar' rejects escaping symlink
targets (/etc/crypto-policies, OutsideDestinationError), and
'fully_trusted' trips on read-only files overwritten across layers
(/etc/machine-id) -- plus tarfile ignores OCI whiteouts and hardlinks.
Delegate to 'oc image extract' instead, which applies OCI layer/whiteout
semantics correctly, runs unprivileged, and is already installed in the
worker base image and used elsewhere in the repo. The function signature
is unchanged, so callers are unaffected.
'oc' distinguishes files from directories by argument shape and exits 0
extracting nothing when a path is absent, so probe the directory form
('<dir>/*:<dst>') first, then the file form ('<file>:<dst>'), and treat
"nothing extracted" as FileNotFoundInImageError.
Tests rewritten to mock 'oc'/run_cmd instead of tarfile/skopeo internals.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
🤖 Finished Review · ❌ Failure (validation failed after 2 iteration(s)) · Started 9:20 AM UTC · Completed 10:02 AM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high |
|
🤖 Finished Review · ✅ Success · Started 4:19 AM UTC · Completed 4:40 AM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $10.83 |
'oc image extract' unpacks only file entries, so an empty index's /configs (no files) is indistinguishable from an absent path — both raise FileNotFoundInImageError. Rather than guess in the low-level extractor (the reverted dotfile approach), let extract_catalog_and_db_from_image decide: it holds the configs label as the signal that the image declares an FBC root, so it catches FileNotFoundInImageError and uses an empty catalog directory. The hidden index.db remains required and still fails with an onboarding error when absent, regardless of whether /configs is empty. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ab758de to
bde9a0c
Compare
|
🤖 Finished Review · ✅ Success · Started 8:14 AM UTC · Completed 8:36 AM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $11.76 |
Correctness:
- push_index_db_artifact hardcoded 'idb-<digest>' while the read path in
_get_content_addressed_artifact_tag went through
iib_index_db_artifact_tag_template. Identical under the default, silently
divergent under any site override -- every lookup would miss. Route the
write path through the same template.
- remote_branch_exists passed the bare branch name to 'git ls-remote', which
matches a pattern against the tail of a ref at slash boundaries: 'v4.17'
also matches 'refs/heads/team/v4.17'. A nested branch was reported as the
tag branch, misrouting a divergent-tag request onto the normal path where
the subsequent clone then fails. Pass the fully-qualified ref.
- extract_files_from_image_non_privileged stages with symlinks=True so a link
is never followed against the worker filesystem, but prepare_build_sources
then copies the configs tree with a plain copytree, which resolves it. An
image carrying '/configs/x -> /etc/passwd' would get host content committed
and pushed to GitLab. Reject symlinks escaping the extraction root in
extract_catalog_and_db_from_image, covering both copies, and fail closed
naming the offending path -- an FBC catalog is declarative JSON/YAML, so an
escaping link is not a legitimate shape for an index image to have.
- Chain the exceptions raised on missing index.db so the original traceback
survives for outer handlers.
Config:
- The tag template placeholders changed from {image_name}/{tag} to {digest}
when cache keys became content-addressed. A deployment carrying an override
from the old scheme failed deep inside a build with a bare KeyError;
validate at worker startup and name the setting instead. Only validated
when present, since absence means the class default applies. Documented in
the containerized README.
Cleanup:
- Move run_cmd to the module-level import (no circular-import reason; the
same module is already imported there) and repoint the tests onto
containerized_utils.run_cmd, matching every other mock in that file.
- Rename _get_artifact_combined_tag to _get_content_addressed_artifact_tag;
nothing is "combined" anymore, it is a pure content digest.
- Freeze BuildSources so the is_divergent guard, which decides whether an MR
may be merged, cannot be flipped after resolution.
- Note why prepare_git_repository_for_build is retained alongside
prepare_build_sources; add the missing :return: on remote_branch_exists.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
🤖 Finished Review · ✅ Success · Started 10:39 PM UTC · Completed 11:02 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $13.45 |
pull_index_db_artifact caught every IIBError out of get_oras_artifact and rewrote it as "No index.db found ... Onboard the image to build." A registry outage, an auth failure or a 5xx therefore told operators to re-onboard an image that was already onboarded. Add ArtifactNotFoundError(IIBError), following the FileNotFoundInImageError pattern: run_cmd classifies 'oras pull' stderr against the registry codes that mean the artifact is absent (MANIFEST_UNKNOWN, NAME_UNKNOWN, BLOB_UNKNOWN, "not found", 404) and raises it; everything else stays a plain IIBError. Deliberately does not match a bare 401/403, since an authenticated registry answers 404 for an absent tag. get_oras_artifact preserves the type through its cleanup wrapper, and pull_index_db_artifact keeps the onboarding message for a genuine miss while reporting a failed pull as what it is. The ImageStream fallback is unaffected, as ArtifactNotFoundError is still an IIBError. Also collapse the FileNotFoundInImageError docstring to a single line, moving its rationale to a comment above the class, to match every other exception in iib/exceptions.py. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
🤖 Finished Review · ✅ Success · Started 11:51 PM UTC · Completed 12:13 AM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $11.52 |
- Copy the divergent-path catalog with symlinks=True, matching the extraction copy. Escaping symlinks are already rejected, so the links that remain are internal and should be committed as links rather than dereferenced into duplicate regular files. - Move the rationale for FileNotFoundInImageError and ArtifactNotFoundError from block comments into their class docstrings, matching the rest of iib/exceptions.py. - Reject an iib_index_db_artifact_tag_template that cannot fit the "-<request_id>" suffix within the 128-character OCI tag limit, so a long custom prefix fails at worker startup instead of at push time. - Note at the three handler merge guards that "not sources.is_divergent" is defense-in-depth over the rejection in prepare_build_sources. - Clarify in AGENTS.md that only the git side rolls back on failure; content-addressed index.db artifacts are never overwritten. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
🤖 Finished Review · ✅ Success · Started 4:32 AM UTC · Completed 5:04 AM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $13.41 |
|
/fs-review |
|
🤖 Finished Review · ✅ Success · Started 11:36 AM UTC · Completed 12:00 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $10.40 |
There was a problem hiding this comment.
Note: The following review comments could not be posted on the diff (GitHub returned 422) and are included here instead:
docker/containerized/README.md(file-level): Line 58 · [medium] scope-alignment
The switch from '{image_name}-{tag}' to 'idb-{digest}' artifact keys is a hard, non-migrating cutover. The README documents orphaned cache entries and the existing cache-pruning cleanup, and a startup validator catches old-scheme config overrides. However, there is no migration tooling and operators must re-onboard affected images.
Suggested fix: Consider a one-time migration script or an operator runbook reference, though the README documentation and startup validator already mitigate the worst failure modes.
iib/workers/config.py(file-level): Line 126 · [low] API-contract
_REQUEST_ID_SUFFIX_RESERVE = 11 reserves space for a 10-digit request ID (32-bit range). If the database uses a 64-bit auto-incrementing primary key, request IDs could eventually exceed 10 digits.
iib/workers/tasks/git_utils.py(file-level): Line 156 · [low] API-shape-patterns
remote_branch_exists and validate_git_remote_branch both shell out to git ls-remote --heads but with different ref patterns. The new function uses refs/heads/{branch} for exact matching; the old function still uses bare branch name matching, which is susceptible to partial matches.
Suggested fix: Update validate_git_remote_branch to use refs/heads/{branch} for exact matching.
|
🤖 Finished Retro · ✅ Success · Started 2:54 PM UTC · Completed 3:11 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $5.99 |
Retro Analysis: PR #1378TimelinePR #1378 ("Fix index.db namespace collisions and support divergent-tag builds") was a large, human-authored PR (+2219/-874, 23 files, 21 commits) open for 11 days (Aug 30 - Sep 10). It went through 13 review agent runs at
Evidence for Existing IssuesReview agent timeout (2 failures): Both Sep 4 runs exhausted the 20-minute budget before writing output. The agent completed analysis and sub-agent orchestration but ran out of time at the output-write step. Evidence for agents#1157 (time-aware output writing), agents#1169 (output synthesis failure), fullsend#7042 (wrap-up signal on timeout). Note agents#1177 (closed Sep 5) addressed the same 20-min budget pattern. Redundant findings across 13 runs (~$135 total cost): The agent re-flagged dismissed findings repeatedly -- cache orphaning 10+ times, AGENTS.md protected-path 5+, secret exposure 3+ times after human rebuttal with evidence, symlinks 3+ times with contradictory guidance. Prior review provenance was False positives with wrong evidence: The agent claimed Domain knowledge gaps: lipoja caught 3 issues neither agent detected: (a) non-FBC image code paths are dead code since new IIB only supports FBC, (b) index.db extraction is broken because Konflux-built images don't include it, (c) OLM labels What the Review Agent Did WellDespite the noise, the agent produced genuinely useful findings: hardcoded tag format vs. configurable template (concrete correctness bug), TOCTOU on ConclusionAfter thorough deduplication against existing open issues, all improvement opportunities from this retro are already tracked. The highest-impact existing issues for this repo are iib#1366 (domain-specific AGENTS.md), agents#931 (enable delta review), agents#1092 (incremental review mode), and agents#1199 (load human review context). |
Summary
Fixes two architectural collisions in the containerized (git + ORAS + Konflux) worker architecture, where the
index.dbartifact identity and the git branch were derived from keys that are too weak.Problem 1 — namespace collision. The
index.dbORAS-artifact and ImageStream tags derived from repo-name + OCP version only, dropping registry/namespace. Soquay.io/redhat/fooandquay.io/redhat-pending/foocollapsed onto the same artifact reference and read/overwrote each other'sindex.db→ corruption.Problem 2 — divergent tags. The git branch was the OCP version read from image labels, so a divergent tag (
:test, or a timestampedv4.14-<ts>) resolved to the same branch as mainlinev4.14and collided in git.What changed
Part A — content-addressed artifact naming (clean cutover). The artifact/ImageStream tag is now keyed on the index image's manifest digest alone:
idb-{digest}, where{digest}is the full 64-char hex sha256 with nosha256:prefix (iib_index_db_artifact_tag_template, defaultidb-{digest}). Identical content addressed by different pullspecs (e.g. a released mirror) shares one artifact; different content never collides, so the namespace collision is structurally impossible rather than merely unlikely.Because the key is now content, artifacts are immutable: the write path keys on the output image digest, and there is no rollback-capture path to maintain. No dual-read — that would reintroduce the bug.
The template's placeholders changed with the scheme:
{image_name}and{tag}are no longer supplied,{digest}is. A deployment carrying an override from the old scheme is rejected at worker startup with aConfigErrornaming the setting, rather than failing part-way through a build.Part B — branch = image tag, with a normal/divergent discriminator. The git branch is now keyed on the image tag. For prod indexes
tag == ocp_version, so this is a no-op for the existing fleet.configs/+index.dbfrom the image withoc image extract(unprivileged; no tar handling in IIB). The hiddenindex.dbpath is required — an image without one was never onboarded to the containerized flow, and the request fails rather than silently building from a synthesised or labeled database. A declared-but-empty/configsis treated as an empty catalog, not a broken image. The build then reuses the base OCP branch's Konflux Component via a throw-away MR that is never merged, and rejects overwrite. The divergentindex.dbis never sourced from ORAS. Timestamped/point-in-time tags stay throw-away by design.Deployment
This is a breaking cutover for already-onboarded images. Read before rolling out.
The artifact key changes from
{image_name}-{tag}toidb-{digest}, so every artifact cached under the old scheme is orphaned. There is deliberately no read-through fallback: on the normal path a missing digest-keyed artifact fails the request withNo index.db found for the image <pullspec> (artifact <ref>). Onboard the image to build.Sourcing content from the image on a cache miss was implemented and then removed — it silently pushes a potentially staleindex.dbinto a content-addressed cache, which is worse than a loud failure.Consequence: the first request against each already-onboarded index image fails until that image is re-onboarded.
Before the upgrade:
iib_index_db_artifact_tag_templateis overridden in a sitesettings.py, update it to a{digest}-only template (or drop the override). Workers will refuse to start otherwise.After the upgrade:
3. Re-onboard each index image so its digest-keyed
index.dbartifact exists. Warming high-traffic indexes first limits user-visible failures.4. Orphaned old-scheme artifacts need no manual action; existing cache-pruning removes them.
A registry that cannot answer is now reported separately from a genuinely absent artifact, so an outage during the rollout will not tell operators to re-onboard an image that is already onboarded.
Commits
a4ed53bfix: key index.db artifact/ImageStream identity on image content digestfffa76efeat: addremote_branch_existsandget_index_taghelpers42ce2d9feat: unprivileged extraction of configs+index.db from index imageb0db932feat:prepare_build_sourcesorchestrates normal vs divergent build pathsde380a4/31d072a/abcb41afeat: route add / rm / fbc-operations through normal/divergent build sourcesd60faf9test: prove divergent never-merge guard withoverwrite=True66d231bdocs: document branch=tag convention and divergent-tag builds0da753d/4f4df4c/961cd7efeat: key index.db write on output image digest; drop rollback captureb3210b7fix: require hidden index.db, fail instead of falling back5124112fix: extract image files viaoc image extract, not tarfilebde9a0cfix: treat declared-but-empty/configsas an empty catalogScope
In scope: single-
from_indexrequest types —add,rm,fbc-operations. Out of scope (unchanged):merge-index-image,regenerate-bundle,create-empty-index(its empty-artifact ref is preserved byte-identical). Digest-referencedfrom_indexis deferred. The old (non-containerized) worker handlers are untouched.Type of change
Checklist
tox -e py312→ full suite passing, coverage gate met)tox -m static: black / flake8 / yamllint / mypy all OK)Unreleasedsection is empty by convention; entries are generated from PR titles at release time)🤖 Generated with Claude Code