Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
a4ed53b
fix: key index.db artifact/ImageStream identity on image content digest
yashvardhannanavati Sep 2, 2026
fffa76e
feat: add remote_branch_exists and get_index_tag helpers
yashvardhannanavati Aug 30, 2026
42ce2d9
feat: unprivileged extraction of configs+index.db from index image
yashvardhannanavati Aug 30, 2026
b0db932
feat: prepare_build_sources orchestrates normal vs divergent build paths
yashvardhannanavati Aug 30, 2026
de380a4
feat: route add requests through normal/divergent build sources
yashvardhannanavati Aug 30, 2026
31d072a
feat: route rm requests through normal/divergent build sources
yashvardhannanavati Aug 30, 2026
abcb41a
feat: route fbc-operations through normal/divergent build sources
yashvardhannanavati Aug 30, 2026
d60faf9
test: prove divergent never-merge guard with overwrite=True
yashvardhannanavati Aug 30, 2026
66d231b
docs: document branch=tag convention and divergent-tag builds
yashvardhannanavati Aug 30, 2026
8861631
fix: address code review findings on divergent-tag build path
yashvardhannanavati Sep 2, 2026
0da753d
feat: key index.db write on output image digest with warm-push
yashvardhannanavati Sep 2, 2026
4f4df4c
refactor: drop index.db rollback capture (content keys are immutable)
yashvardhannanavati Sep 2, 2026
961cd7e
feat: route handlers' index.db push through the output image digest
yashvardhannanavati Sep 2, 2026
ee6aa59
feat: bootstrap index.db from image on digest cache miss (read-through)
yashvardhannanavati Sep 2, 2026
b73a69f
docs: describe content-digest index.db cache naming
yashvardhannanavati Sep 2, 2026
b3210b7
fix: require hidden index.db, fail instead of falling back
yashvardhannanavati Sep 4, 2026
5124112
fix: extract image files via 'oc image extract', not tarfile
yashvardhannanavati Sep 4, 2026
bde9a0c
fix: treat declared-but-empty /configs as an empty catalog
yashvardhannanavati Sep 6, 2026
4a8357b
fix: address fullsend-ai-review findings on index.db naming
yashvardhannanavati Sep 9, 2026
55e7944
fix: distinguish a missing index.db artifact from a failed pull
yashvardhannanavati Sep 9, 2026
24296d6
fix: address latest fullsend-ai-review findings
yashvardhannanavati Sep 10, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,12 @@ Key modules:

## Pitfalls
- **Never use tools that require privileged access** - the production deployment does not have access to privileged containers.
- `overwrite_from_index_token` triggers an index.db push to Quay after build; if that push fails, the git commit must also be reverted.
- `overwrite_from_index_token` triggers an index.db push to Quay after build; if that push fails, the git commit must also be reverted. Only the git side rolls back — index.db artifacts are keyed by content digest and never overwritten, so there is nothing to undo on the artifact side.
- **Always revert the git commit on Konflux or push failure** — dangling commits break the next build. This includes when overwrite_from_index_token triggers an index.db push to Quay that fails. Mark the request failed immediately after reverting.
- **`index.db` lives in `/var/index_db`, not `/tmp`** — the task-start cleanup wipes `/tmp`; storing it there deletes the cache between requests.
- **`regenerate_bundle` always opens a PR** — source bundle overwrite is unsupported; don't add a direct-push path for this request type.
- **Tasks must be idempotent** — Celery retries; images may have already been pushed.
- **Never edit existing Alembic migrations** — generate a new revision instead.
- **API ↔ Worker task signatures must match** — renaming args silently breaks in-flight requests.
- **Avoid making changes to the old worker architecture** - Always ask before making changes to the following functions - `handle_add_request`, `handle_rm_request`, `handle_merge_request`, `handle_create_empty_index_request`, `handle_fbc_operation_request`, `handle_regenerate_bundle_request`
- **Divergent-tag requests never merge their MR** — they reuse the base OCP branch's Konflux Component and must stay throw-away; the guard is `overwrite_from_index and not sources.is_divergent`. Never source their index.db from ORAS.
Comment thread
lipoja marked this conversation as resolved.
Resolved
Comment thread
lipoja marked this conversation as resolved.
Resolved
Comment thread
lipoja marked this conversation as resolved.
Resolved
35 changes: 35 additions & 0 deletions docker/containerized/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -253,6 +253,41 @@ The worker configuration is in `docker/containerized/worker_config.py`. This fil
- Includes the containerized task modules
- Validates required configuration on startup

## Git Branch and Tag Semantics

The Git branch used for a request's catalog is keyed on the **image tag**, not a fixed OCP-version mapping. For the existing fleet this is a no-op, since prod index tags already equal the OCP version (e.g. `v4.19` → branch `v4.19`).

### Onboarding a Non-OCP Tag

To onboard a tag that isn't an OCP version (e.g. a custom or pre-release tag), a maintainer must:

1. Create a Git branch in the catalog repository named exactly after the tag.
2. Provision a Konflux Component for that branch, so pushes to it trigger a PipelineRun.

Once both exist, requests against that tag build and push normally, and `overwrite_from_index` is honored like any other branch.

### Divergent Tags (No Matching Branch)

If a request targets a tag with no corresponding Git branch — for example a timestamped, point-in-time tag cut from an existing index — IIB treats it as a **divergent tag**:

- Configs and `index.db` are extracted directly from the index image (unprivileged), not sourced from ORAS. Only the FBC configs and the hidden `index.db` are extracted; if the image carries no hidden `index.db`, the request fails ("no index.db found, onboard the image to build") — there is no labeled-db or empty-db fallback.
- The build reuses the base OCP branch's existing Konflux Component via a throw-away merge request.
- That MR is **never merged** — it is always closed after the pipeline completes (or on failure), regardless of outcome.
- `overwrite_from_index` is **rejected** for divergent-tag requests; there is no direct-push path.

This lets IIB build and validate a one-off tag without requiring per-tag branch/Component provisioning.

## Index DB Artifact and ImageStream Tag Naming

Cached `index.db` artifact tags (ORAS) and ImageStream tags are keyed on the index image's content (manifest) digest — `idb-<sha256>` — resolved via `skopeo inspect`, not on its pullspec. Because the key is the content itself, it is both namespace-safe and promotion-safe:

- **Namespace-safe:** two images that share a repository name in different registry namespaces (e.g. `quay.io/redhat/my-index:v4.17` vs `quay.io/redhat-pending/my-index:v4.17`) have different content and therefore different digests, so they never collide on the same cache tag.
- **Promotion-safe:** the same image content addressed by different pullspecs after a release or mirror (e.g. `quay.io/my-namespace/iib-pub:v4.17` → `registry.access.redhat.com/some-namespace/operator-index:v4.17`) preserves its manifest digest, so both pullspecs resolve to the *same* cache entry and share one `index.db`.

The tag is built from `iib_index_db_artifact_tag_template` (default `idb-{digest}`) on both the read and the write path. Its placeholders changed with this scheme: `{image_name}` and `{tag}` are no longer supplied, and `{digest}` is. A deployment carrying an override from the old scheme is rejected at worker startup with a `ConfigError` naming the setting, rather than failing part-way through a build.

Cache entries written under the previous pullspec-derived naming scheme are orphaned by this change — they are not migrated in place. On the normal path IIB never falls back to extracting `index.db` from the image: if the digest-keyed artifact is missing, the request fails with a "no index.db found for the image, onboard the image to build" error, and the image must be onboarded (which populates the artifact) before it can be built. Orphaned entries are cleaned up by the existing cache-pruning process rather than any code path in this workflow.
Comment thread
lipoja marked this conversation as resolved.
Resolved
Comment thread
yashvardhannanavati marked this conversation as resolved.
Resolved

## Differences from Traditional Workflow

| Aspect | Traditional Workflow | Containerized Workflow |
Expand Down
21 changes: 21 additions & 0 deletions iib/exceptions.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,27 @@ class IIBError(BaseException):
"""An error was encountered in IIB."""


class FileNotFoundInImageError(IIBError):
Comment thread
lipoja marked this conversation as resolved.
Resolved
Comment thread
yashvardhannanavati marked this conversation as resolved.
Resolved
Comment thread
yashvardhannanavati marked this conversation as resolved.
Resolved
"""
A requested path was not present in a container image.

Subclasses IIBError so existing "except IIBError" handlers still catch it, while
letting callers distinguish a genuinely absent path from a real extraction failure
(registry, OCI parsing, layer, or tar error).
"""


class ArtifactNotFoundError(IIBError):
"""
A requested OCI artifact was not present in the registry.

Subclasses IIBError for the same reason as FileNotFoundInImageError: callers that
only care that the pull failed keep working, while callers that must tell "this
image was never onboarded" apart from a registry outage or an auth failure can
catch this specifically.
"""


class ValidationError(BaseException):
"""Denote invalid input."""

Expand Down
59 changes: 58 additions & 1 deletion iib/workers/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ class Config(object):
iib_index_db_oras_auth_path: str = os.path.join(
os.path.expanduser('~'), '.docker', 'oras', 'config.json'
)
iib_index_db_artifact_tag_template: str = '{image_name}-{tag}'
iib_index_db_artifact_tag_template: str = 'idb-{digest}'
Comment thread
lipoja marked this conversation as resolved.
Resolved
Comment thread
lipoja marked this conversation as resolved.
Resolved
Comment thread
yashvardhannanavati marked this conversation as resolved.
Resolved
iib_index_db_artifact_template: str = '{registry}/index-db:{tag}'
# Whether to use OpenShift ImageStream cache for index.db artifacts
# Requires OpenShift cluster with ImageStream configured
Expand Down Expand Up @@ -383,6 +383,7 @@ def validate_celery_config(conf: app.utils.Settings, **kwargs) -> None:
_validate_multiple_opm_mapping(conf['iib_ocp_opm_mapping'])
_validate_iib_org_customizations(conf['iib_organization_customizations'])
_validate_konflux_config(conf)
_validate_index_db_artifact_tag_template(conf)

if conf.get('iib_aws_s3_bucket_name'):
if not isinstance(conf['iib_aws_s3_bucket_name'], str):
Expand Down Expand Up @@ -438,6 +439,62 @@ def validate_celery_config(conf: app.utils.Settings, **kwargs) -> None:
)


# Maximum length of an OCI reference tag, per the distribution spec grammar.
_OCI_MAX_TAG_LENGTH = 128
Comment thread
yashvardhannanavati marked this conversation as resolved.
Resolved
# Characters held back for the "-<request_id>" suffix the index.db push path appends:
# one separator plus a 10-digit request id, which covers the full 32-bit id range.
_REQUEST_ID_SUFFIX_RESERVE = 11
Comment thread
yashvardhannanavati marked this conversation as resolved.
Resolved


def _validate_index_db_artifact_tag_template(conf: app.utils.Settings) -> None:
"""
Ensure the index.db artifact tag template is keyed on the image content digest.

The template placeholders changed from ``{image_name}``/``{tag}`` to ``{digest}``
when index.db cache keys became content-addressed. A deployment carrying an
override from the old scheme would otherwise fail deep inside a build with a
bare ``KeyError``; catch it at worker startup and name the setting instead.

:param celery.app.utils.Settings conf: the Celery application configuration to validate
:raises iib.exceptions.ConfigError: if the template does not use ``{digest}``, or if it
cannot yield a tag within the OCI length limit
"""
template = conf.get('iib_index_db_artifact_tag_template')
if template is None:
# Not overridden, so the config class default applies. Only a site-level
# override can be wrong here, and an override is always present.
return

try:
template.format(digest='0' * 64)
Comment thread
yashvardhannanavati marked this conversation as resolved.
Resolved
except (KeyError, IndexError):
raise ConfigError(
'iib_index_db_artifact_tag_template must use only the "{digest}" placeholder. '
'index.db cache entries are keyed on the index image manifest digest; the '
f'"{{image_name}}"/"{{tag}}" placeholders are no longer supplied. Got: {template}'
)

if '{digest}' not in template:
raise ConfigError(
'iib_index_db_artifact_tag_template must contain the "{digest}" placeholder so '
f'cache entries stay keyed on image content. Got: {template}'
)

# An OCI reference tag is capped at 128 characters. The push path appends
# "-<request_id>" to the rendered tag for the per-request copy, so the template must
# leave room for that suffix too -- otherwise a long custom prefix passes startup and
# then fails at push time, after the image has already been built.
rendered_length = len(template.format(digest='0' * 64))
budget = _OCI_MAX_TAG_LENGTH - _REQUEST_ID_SUFFIX_RESERVE
if rendered_length > budget:
raise ConfigError(
'iib_index_db_artifact_tag_template renders a tag of '
f'{rendered_length} characters, which leaves no room for the "-<request_id>" '
f'suffix within the {_OCI_MAX_TAG_LENGTH}-character OCI tag limit. Keep the '
f'rendered tag at or under {budget} characters. Got: {template}'
)


def _validate_multiple_opm_mapping(iib_ocp_opm_mapping: Dict[str, str]) -> None:
"""
Validate iib_ocp_opm_mapping config variable.
Expand Down
55 changes: 32 additions & 23 deletions iib/workers/tasks/build_containerized_add.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
)
from iib.workers.tasks.celery import app
from iib.workers.tasks.containerized_utils import (
prepare_git_repository_for_build,
prepare_build_sources,
fetch_and_verify_index_db_artifact,
write_build_metadata,
git_commit_and_create_mr,
Expand Down Expand Up @@ -148,38 +148,41 @@ def handle_containerized_add_request(
distribution_scope = prebuild_info['distribution_scope']

index_to_gitlab_push_map = index_to_gitlab_push_map or {}
# Variables mr_details, last_commit_sha and original_index_db_digest
# needs to be assigned; otherwise cleanup_on_failure() fails when an exception is raised.
# Variables mr_details and last_commit_sha need to be assigned; otherwise
# cleanup_on_failure() fails when an exception is raised.
mr_details: Optional[Dict[str, str]] = None
last_commit_sha: Optional[str] = None
original_index_db_digest: Optional[str] = None

Opm.set_opm_version(from_index_resolved)

_update_index_image_build_state(request_id, prebuild_info)
present_bundles: List[BundleImage] = []
present_bundles_pull_spec: List[str] = []
with tempfile.TemporaryDirectory(prefix=f'iib-{request_id}-') as temp_dir:
branch = prebuild_info['ocp_version']

# Set up and clone Git repository
(
index_git_repo,
local_git_repo_path,
localized_git_catalog_path,
) = prepare_git_repository_for_build(
sources = prepare_build_sources(
request_id=request_id,
from_index=str(from_index),
from_index_resolved=from_index_resolved,
temp_dir=temp_dir,
branch=branch,
ocp_version=prebuild_info['ocp_version'],
index_to_gitlab_push_map=index_to_gitlab_push_map,
overwrite_from_index=overwrite_from_index,
)

# Pull index.db artifact (uses ImageStream cache if configured, otherwise pulls directly)
artifact_index_db_file = fetch_and_verify_index_db_artifact(
from_index=str(from_index),
temp_dir=temp_dir,
)
index_git_repo = sources.index_git_repo
local_git_repo_path = sources.local_git_repo_path
localized_git_catalog_path = sources.localized_git_catalog_path
branch = sources.target_branch

# Divergent path already has index.db extracted from the image; the normal
# path pulls it from ORAS. NEVER fall back to ORAS on the divergent path —
# that would read the base OCP branch's index.db.
if sources.index_db_path is not None:
artifact_index_db_file = sources.index_db_path
else:
artifact_index_db_file = fetch_and_verify_index_db_artifact(
from_index=str(from_index),
temp_dir=temp_dir,
)

msg = 'Checking if bundles are already present in index image'
log.info(msg)
Expand Down Expand Up @@ -330,20 +333,27 @@ def handle_containerized_add_request(

# Push updated index.db before merging the MR so that on failure both
# git and the index.db artifact remain consistent (MR stays open,
# cleanup_on_failure closes it and rolls back the artifact).
original_index_db_digest = push_index_db_artifact(
# cleanup_on_failure closes it; the content-addressed artifact needs no rollback).
push_index_db_artifact(
request_id=request_id,
from_index=str(from_index),
index_db_path=artifact_index_db_file,
operators=operators,
output_image=image_url,
overwrite_from_index=overwrite_from_index,
request_type='add',
)

# Merge or close the MR as the final step so that all side effects
# (replication, metadata, index.db push) have succeeded before git
# is advanced. This prevents git/index.db divergence on partial failure.
if overwrite_from_index:
#
# The `not sources.is_divergent` half is defense-in-depth: prepare_build_sources
# already rejects overwrite_from_index on the divergent path, so this cannot be
# reached with both set. It stays because a divergent build reuses the base OCP
# branch's Konflux Component -- merging its MR would publish one tag's content
# onto the shared branch.
if overwrite_from_index and not sources.is_divergent:
Comment thread
lipoja marked this conversation as resolved.
Resolved
merge_mr_after_build(mr_details, index_git_repo)
# Prevent cleanup_on_failure from trying to close an already-merged MR
mr_details = None
Expand All @@ -364,7 +374,6 @@ def handle_containerized_add_request(
request_id=request_id,
from_index=str(from_index),
index_repo_map=index_to_gitlab_push_map or {},
original_index_db_digest=original_index_db_digest,
reason=f"error: {e}",
)
raise IIBError(f"Failed to add bundles: {e}")
17 changes: 10 additions & 7 deletions iib/workers/tasks/build_containerized_create_empty_index.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,6 @@
opm_validate,
)
from iib.workers.tasks.oras_utils import (
_get_artifact_combined_tag,
_get_name_and_tag_from_pullspec,
get_oras_artifact,
)
Expand Down Expand Up @@ -176,7 +175,6 @@ def handle_containerized_create_empty_index_request(
index_git_repo: Optional[str] = None
last_commit_sha: Optional[str] = None
output_pull_spec: Optional[str] = None
original_index_db_digest: Optional[str] = None

with tempfile.TemporaryDirectory(prefix=f'iib-{request_id}-') as temp_dir:
branch = ocp_version
Expand All @@ -199,11 +197,16 @@ def handle_containerized_create_empty_index_request(
conf = get_worker_config()
Comment thread
lipoja marked this conversation as resolved.
Resolved
empty_tag = conf.get('iib_empty_index_db_tag', 'empty')

# Construct the pullspec for the empty index.db artifact
# Construct the pullspec for the empty index.db artifact.
# This tag intentionally does NOT use the content-digest key that
# _get_content_addressed_artifact_tag derives for real indexes: the empty index.db
# is a shared, content-free seed artifact keyed only by image name +
# "empty", so it is deliberately reusable across indexes rather than tied
# to any single image's manifest digest.
image_name, _ = _get_name_and_tag_from_pullspec(from_index)
Comment thread
lipoja marked this conversation as resolved.
Resolved
Comment thread
lipoja marked this conversation as resolved.
Resolved
empty_artifact_ref = conf['iib_index_db_artifact_template'].format(
registry=conf['iib_index_db_artifact_registry'],
tag=_get_artifact_combined_tag(image_name, empty_tag),
tag=f"{image_name}-{empty_tag}",
)

log.info('Fetching empty index.db from %s', empty_artifact_ref)
Expand Down Expand Up @@ -344,12 +347,13 @@ def handle_containerized_create_empty_index_request(

# Push the empty index.db with request ID tag
# Since overwrite_from_index is False, this will only push with request_id tag
# and will not overwrite the v4.x tag
original_index_db_digest = push_index_db_artifact(
# and will not overwrite the current artifact
push_index_db_artifact(
request_id=request_id,
from_index=from_index,
index_db_path=str(index_db_path),
operators=[], # Empty list since we're creating an empty index
output_image=image_url,
overwrite_from_index=False, # Always False for create_empty_index
request_type='create_empty_index',
)
Expand All @@ -372,7 +376,6 @@ def handle_containerized_create_empty_index_request(
request_id=request_id,
from_index=from_index,
index_repo_map=index_to_gitlab_push_map or {},
original_index_db_digest=original_index_db_digest,
reason=f"error: {e}",
)
raise IIBError(f"Failed to create empty index: {e}")
Expand Down
Loading
Loading