Repository navigation
Add a PadToSquare image transform - #1146
Open
Pierluigi Urru (stwykd) wants to merge 3 commits into
Open
Pierluigi Urru (stwykd) wants to merge 3 commits into
Pierluigi Urru (stwykd) wants to merge 3 commits into
Conversation
Pads an (H, W, C) image to a centred square of side max(H, W) with a configurable fill colour, the pad-to-square step some vision encoders are trained with before a fixed-size resize (LLaVA's expand2square). Square images pass through unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: af407617-e477-4690-b7f6-38e5f88f661b
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The tests omit portrait-image coverage for horizontal centering and odd right-side padding, and undisposed tensor handles leak wrappers.
2 open issues
What changed in this PR
Adds PadToSquare to the image-processing pipeline for vision encoders that require centered square padding before resizing.
Changes:
- Implements centered uint8 image padding with configurable scalar or per-channel fill.
- Registers
PadToSquarefor use in processor configurations. - Adds pixel-level tests for wide and already-square images using decode-only reference output.
| File | Description |
|---|---|
test/pp_api_test/test_processor.cc |
Tests padding dimensions, border colors, and preserved image pixels. |
test/data/processor/pad_to_square.json |
Configures padding with an explicit RGB fill. |
test/data/processor/decode_only.json |
Provides unpadded reference pixels for comparison. |
shared/api/image_transforms.hpp |
Implements padding and fill validation. |
shared/api/image_processor.cc |
Registers the new transform. |
Review effort: Balanced
Holds the result tensors in OrtxObjectPtr so the test no longer leaks them, and adds a tall image (horizontal offset), a single fill value for all channels and an out-of-range fill that fails at processor creation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: af407617-e477-4690-b7f6-38e5f88f661b
The tall test image has an odd width/height difference, so it gets a 102 column left border and a 103 column right one. Assert the border widths of every case, and for the tall image the left border, both image edges and the right border including its extra column. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: af407617-e477-4690-b7f6-38e5f88f661b
Contributor
|
/azp run |
|
Azure Pipelines: Successfully started running 2 pipeline(s). |
Author
|
Baiju Meswani (@baijumeswani) kunal-vaishnavi could you review this one too? it's the pad to square preprocessing the genai one needs. the ci-optional failure is pip failing to install build deps (collections.Callable), not this change |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
PadToSquareto the image processor transforms. It pads an(H, W, C)uint8 image to a square ofside
max(H, W), pasting it at row(S - H) / 2and column(S - W) / 2(rounded down, the same asLLaVA's
expand2square/ PILpaste). The border getsfill: one value per channel or a single value;the default is black. Square images pass through. Some fixed-resolution vision encoders are trained with
this step before a fixed-size
Resize, and no existing transform reproduces it.{ "operation": { "name": "pad_to_square", "type": "PadToSquare", "attrs": { "fill": [0, 0, 0] } } }Test:
ProcessorTest.TestPadToSquarechecks a wide PNG, a tall PNG with an odd width/heightdifference (the extra border column goes on the right) and a square JPEG against the decode-only
output (the border is the fill colour, the centre is the original pixels), a single
fillvalue for allchannels, and an out-of-range
fillthat fails at processor creation.PadToSquarefollowed byResizeLINEAR is bit-identical to PIL
paste+resize(BILINEAR).Companion to microsoft/onnxruntime-genai#2842 (a
llavamodel type in ONNX Runtime GenAI), which doesn't need thisPR to land.
cc Sayan Shaw (@sayanshaw24) kunal-vaishnavi