Skip to content

Modeling Algorithms - guard CreateSmoothed's fixed-stride shapes array against a section with more edges than section 1 - #1466

Open
gsdali wants to merge 2 commits into
Open-Cascade-SAS:masterfrom
gsdali:fix/thrusections-createsmoothed-section-edge-count
Open

gsdali wants to merge 2 commits into
Open-Cascade-SAS:masterfrom
gsdali:fix/thrusections-createsmoothed-section-edge-count

Conversation

@gsdali

@gsdali gsdali commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Description

BRepOffsetAPI_ThruSections::CreateSmoothed() derives nbEdges — the edge count it assumes
every section has — from section 1 alone (or section 2, if section 1 is a punctual/degenerate
vertex section). It then allocates shapes, an NCollection_Array1<TopoDS_Shape> sized exactly
nbSects * nbEdges, and fills it by walking each section's wire with a BRepTools_WireExplorer,
incrementing a running index with no bounds check.

Nothing enforces that every section actually has nbEdges edges. BRepFill_CompatibleWires (via
CheckCompatibility(true), the default) normally reconciles differing edge counts across sections
before CreateSmoothed() ever runs. With CheckCompatibility(false) ("no check"), that
reconciliation is skipped entirely, so a later section with more edges than section 1 makes the
fill loop walk past the end of shapes — an out-of-bounds write, observed as heap corruption and a
SIGSEGV/SIGBUS inside CreateSmoothed() once at least 3 sections are involved (2 sections always
take the CreateRuled() path instead, which does not share this allocation shape).

Confirmed via a minimal, from-scratch C++ reproducer with a custom signal handler
(backtrace_symbols_fd) for attribution: two matching circle sections build successfully; reusing
the same builder with a third, differently-shaped section (more edges than the first two) and
rebuilding SIGSEGVs against the stock library. It does not need a reused builder as such — what
actually varies is process/allocator state at the time of the overrun, so an otherwise-identical
single Build() call with all three sections added up front does not reliably crash even though
the same out-of-bounds write still occurs. Verified directly by instrumenting the fill loop: the
write index reaches one past shapes.Upper() before the corrupting assignment.

Fix

Before allocating shapes, walk every non-punctual section and count its edges; if any section's
count differs from nbEdges, set myStatus to BRepFill_ThruSectionErrorStatus_ProfilesInconsistent
and return, matching the early-return idiom this function already uses two lines below
(TS.IsNull() -> myStatus = Failed; return;). Punctual end sections (a degenerate vertex added
via AddVertex(), e.g. a cone's apex) are exempt, matching the existing w1Point/w2Point
handling throughout the rest of the function.

Testing

Two GTests added to the existing BRepOffsetAPI_ThruSections_Test.cxx:

  • MismatchedSectionEdgeCountFailsCleanlyWithoutCheck — the crash reproducer. Proved it fails
    first: linked against the unpatched archive, this test crashes (SIGBUS). With the fix, it passes
    (IsDone() == false, no crash).
  • MatchingSectionEdgeCountsStillSucceedWithoutCheck — regression guard: matching section edge
    counts under CheckCompatibility(false) still build, including a punctual end section.

All 5 tests in the file (the 3 pre-existing plus these 2) pass with the fix. Validated by
override-linking the patched translation unit ahead of the stock archive, both with and without
No_Exception/NDEBUG (matching a Release build configuration), confirming the fix eliminates the
out-of-bounds write itself rather than relying on Standard_OutOfRange's range check being
compiled in.

Formatted with this repo's own .clang-format (--dry-run --Werror clean on both changed files).

…y against a section with more edges than section 1

BRepOffsetAPI_ThruSections::CreateSmoothed() derives nbEdges from section 1
alone and allocates shapes sized nbSects * nbEdges, then fills it walking
each section's actual edges with no bounds check. With
CheckCompatibility(false), nothing reconciles differing section edge
counts first, so a later section with more edges than section 1 walks
past the end of shapes: an out-of-bounds write, observed as heap
corruption and a SIGSEGV/SIGBUS once at least 3 sections are involved (2
sections always take the CreateRuled() path instead).

Fix: count each non-punctual section's edges before allocating shapes; on
a mismatch, set myStatus to ProfilesInconsistent and return, matching the
early-return idiom this function already uses two lines below.

Adds two GTests to the existing BRepOffsetAPI_ThruSections_Test.cxx: the
crash reproducer (now fails cleanly instead of crashing) and a regression
guard for matching edge counts, including a punctual end section.
…xemption, DRY the predicate

The original fix's inequality test already rejected a section with MORE
edges than section 1 (the crashing direction) or FEWER (previously silent
success with misaligned per-section strides -- same contract violation,
just without a crash to signal it). Added a GTest proving the fewer-edges
direction: fails against pristine, unpatched source (IsDone() == true,
Shape().IsValid() == false), passes after the fix.

The punctual-section exemption now also verifies the section actually has
at least one edge before exempting it, rather than trusting
w1Point/w2Point's classification unconditionally -- those are vacuously
true for a wire with zero edges, which the fill loop's punctual branch
would otherwise walk with an uninitialized WireExplorer. Attempted to
reproduce a live crash for this specific case (including via a reused
builder) and could not; the check is kept as a correct, cheap hardening,
not as a proven fix for an observed crash.

isPunctualSection is now a single local lambda shared by this guard and
the pre-existing fill loop 15 lines below, which used to duplicate the
same two-clause boolean expression separately.
@gsdali
gsdali force-pushed the fix/thrusections-createsmoothed-section-edge-count branch from 2dbc7c4 to 94e1444 Compare September 2, 2026 00:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

1 participant