feat(cp): add --force to handle partially-populated destinations by renshao · Pull Request #2061 · oras-project/oras

renshao · 2026-05-25T06:33:56Z

Addresses #2072.

A pull-through cache (e.g. ACR Artifact Cache, AWS ECR pull-through cache,
GAR remote repositories) can report a manifest as "exists" while only
holding a subset of its referenced content. oras-go's copyGraph trusts
the destination's existence response and skips the sub-DAG, so the
final tag PUT is rejected by the registry with
'manifest blob unknown' because referenced platform manifests or blobs
are missing.

Add an opt-in --force flag on oras cp and oras push that wraps the
destination with a small GraphTarget reporting Exists()=false for any
manifest media type. This defeats the sub-DAG skip and forces the copy
to walk every referenced manifest, pushing any missing manifests or
blobs before the root tag is updated. Blob existence checks still
short-circuit, so already-present layers are not re-uploaded.

Changes:

cmd/oras/root/cp.go: register --force, document it in the examples,
and wrap dst with verifyingTarget when the flag is set.
cmd/oras/root/push.go: register --force and apply the same
verifyingTarget wrapper before tracking starts.
cmd/oras/root/cp_verify.go: verifyingTarget wrapper and
isManifestMediaType predicate, shared by cp and push.
cmd/oras/root/cp_verify_test.go: unit tests for the wrapper and
predicate.
test/e2e/suite/command/cp.go: e2e test that pre-populates a
destination with only the linux/amd64 platform and re-runs cp with
--force to push the remaining platform manifests of a multi-arch
image before tagging.

Default behavior is unchanged; --force is opt-in on both commands.

Please check the following list:

Does the affected code have corresponding tests, e.g. unit test, E2E test?
Does this change require a documentation update? - document the --force behaviour
Does this introduce breaking changes that would require an announcement or bumping the major version?
Do all new files have an appropriate license header?

TerryHowe · 2026-06-12T11:21:28Z

DCO issue https://github.com/oras-project/oras/pull/2061/checks?check_run_id=80464374792

Copilot

Pull request overview

This PR adds an opt-in --force mode to oras cp and oras push to better handle destinations (notably pull-through caches) that can report a manifest as “existing” while still missing referenced platform manifests/blobs, by forcing traversal of manifest nodes during graph copy.

Changes:

Add --force to oras cp and oras push, wrapping the destination with a VerifyingTarget that reports manifest descriptors as non-existent to defeat oras-go’s sub-DAG skip.
Introduce internal/contentutil helpers: IsManifestMediaType() and VerifyingTarget.Exists() override behavior, with unit tests.
Add an E2E test intended to validate copying into a partially populated destination using --force.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`cmd/oras/root/cp.go`	Adds `--force` flag and wraps the destination with `contentutil.VerifyingTarget` when enabled.
`cmd/oras/root/push.go`	Adds `--force` flag and applies the `contentutil.VerifyingTarget` wrapper before tracking/copy.
`internal/contentutil/verify.go`	Implements `IsManifestMediaType` and `VerifyingTarget` wrapper to force manifest traversal.
`internal/contentutil/verify_test.go`	Adds unit tests for manifest media-type detection and `Exists()` override behavior.
`test/e2e/suite/command/cp.go`	Adds an E2E scenario for `cp --force` against a “partially populated” destination.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

renshao · 2026-06-15T07:01:21Z

I've addressed all code review comments. Squashed commits and rebased from latest main. I ran all E2E tests in Ubuntu Linux, and they all pass.

I tested the --force flag in Windows and verify it works. See the screenshot below, for without vs with --force flag.

TerryHowe · 2026-06-15T14:06:40Z

+
+// IsManifestMediaType reports whether the given media type denotes a
+// manifest (i.e. a non-leaf node in a content-addressable graph).
+func IsManifestMediaType(mediaType string) bool {


I assume oras-go has nothing like this? This seems like a nice thing for oras-go to do.

Checked oras-go, there isn't something like this. I'm happy to create a new issue there and implement it. For this PR, let's keep it the current way. Once the oras-go has this feature we can de-dup later.

renshao · 2026-06-25T01:20:06Z

@TerryHowe
I've addressed your comments in a separate commit. Once you are happy I'll manually squash them into a single commit and rebase, so it is ready to merge. Thanks.

TerryHowe

I'd like @sajayantony to weigh in on the name of this option.

Other than that, generally looks good other than the fixture changes.

renshao · 2026-07-20T03:42:14Z

I'd like @sajayantony to weigh in on the name of this option.

Other than that, generally looks good other than the fixture changes.

@sajayantony I think we landed on --force as the name of this flag for oras cp and oras push. Could you confirm here so this PR is ready to merge, thanks!

Addresses oras-project#2072. A pull-through cache (e.g. ACR Artifact Cache, AWS ECR pull-through cache, GAR remote repositories) can report a manifest as "exists" while only holding a subset of its referenced content. oras-go's copyGraph trusts the destination's existence response and skips the sub-DAG, so the final tag PUT is rejected by the registry with 'manifest blob unknown' because referenced platform manifests or blobs are missing. Add an opt-in --force flag on `oras cp` and `oras push` that wraps the destination with a small GraphTarget reporting Exists()=false for any manifest media type. This defeats the sub-DAG skip and forces the copy to walk every referenced manifest, pushing any missing manifests or blobs before the root tag is updated. Blob existence checks still short-circuit, so already-present layers are not re-uploaded. The wrapper preserves the underlying scope hint so registry auth is unaffected. Changes: - cmd/oras/root/cp.go: register --force, document it in the examples, and wrap dst with the traversing target when the flag is set. - cmd/oras/root/push.go: register --force and apply the same wrapper before tracking starts. - internal/contentutil/traverse.go: traversing GraphTarget wrapper and manifest media-type predicate, shared by cp and push. - internal/contentutil/traverse_test.go: unit tests for the wrapper and predicate. - test/e2e/suite/command/cp.go: e2e test that uses an OCI image layout as the destination so a partial state can be staged (conformant registries reject indices whose children are missing). It fully copies a multi-arch image, deletes two platform manifest blobs on disk, then re-runs cp with --force and asserts the missing children are re-pushed before the tag is updated. Includes a negative control that runs a plain cp first and asserts the missing platform manifests remain absent, proving the bug exists and that --force fixes it. Default behavior is unchanged; --force is opt-in on both commands. Signed-off-by: Ren Shao <405516+renshao@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Ren Shao <shaoren@microsoft.com>

akashsinghal reviewed May 26, 2026

View reviewed changes

Comment thread internal/contentutil/traverse.go

akashsinghal reviewed May 26, 2026

View reviewed changes

Comment thread cmd/oras/root/cp_verify.go Outdated

akashsinghal reviewed May 26, 2026

View reviewed changes

Comment thread cmd/oras/root/cp.go

renshao force-pushed the verify-graph branch 2 times, most recently from bba852e to c17b8de Compare June 10, 2026 01:41

renshao changed the title ~~feat(cp): add --verify-graph to handle partially-populated destinations~~ feat(cp): add --force to handle partially-populated destinations Jun 10, 2026

renshao marked this pull request as ready for review June 10, 2026 03:08

renshao requested review from TerryHowe, Wwwsylvia, sabre1041, sajayantony and shizhMSFT as code owners June 10, 2026 03:08

TerryHowe requested a review from Copilot June 12, 2026 11:25

Copilot started reviewing on behalf of TerryHowe June 12, 2026 11:26 View session

Copilot AI reviewed Jun 12, 2026

View reviewed changes

Comment thread cmd/oras/root/push.go

Comment thread test/e2e/suite/command/cp.go Outdated

renshao force-pushed the verify-graph branch 2 times, most recently from b26caac to 206b90a Compare June 15, 2026 04:16

TerryHowe requested changes Jun 15, 2026

View reviewed changes

renshao force-pushed the verify-graph branch from 55343ab to bad27b6 Compare June 25, 2026 22:34

TerryHowe requested changes Jun 26, 2026

View reviewed changes

Comment thread test/e2e/testdata/zot/command/artifacts/index.json

renshao force-pushed the verify-graph branch from bad27b6 to 019e90d Compare July 20, 2026 03:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(cp): add --force to handle partially-populated destinations#2061

feat(cp): add --force to handle partially-populated destinations#2061
renshao wants to merge 1 commit into
oras-project:mainfrom
renshao:verify-graph

renshao commented May 25, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TerryHowe commented Jun 12, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

renshao commented Jun 15, 2026

Uh oh!

Uh oh!

TerryHowe Jun 15, 2026

Uh oh!

renshao Jun 25, 2026

Uh oh!

Uh oh!

renshao commented Jun 25, 2026

Uh oh!

TerryHowe left a comment

Uh oh!

Uh oh!

renshao commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

renshao commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TerryHowe commented Jun 12, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

renshao commented Jun 15, 2026

Uh oh!

Uh oh!

TerryHowe Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

renshao Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

renshao commented Jun 25, 2026

Uh oh!

TerryHowe left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

renshao commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

renshao commented May 25, 2026 •

edited

Loading