Skip to content

ci(interop): fix the lyrical ABI skew and make interop failures diagnosable - #305

Open
YuanYuYuan wants to merge 4 commits into
mainfrom
fix/interop-provenance
Open

ci(interop): fix the lyrical ABI skew and make interop failures diagnosable#305
YuanYuYuan wants to merge 4 commits into
mainfrom
fix/interop-provenance

Conversation

@YuanYuYuan

@YuanYuYuan YuanYuYuan commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

Three changes to the interop job. One turns the red lyrical leg green; two make the next failure diagnosable instead of a guess.

Fixes #303.

What this PR does

# Change Why
1 Install ros-<distro>-rclcpp explicitly The images pin ROS package versions but not the apt source. Naming rclcpp makes apt upgrade the copy baked into the image, so it matches the repo everything else comes from
2 Record the image, OS release, the six core ROS libraries, the four packages under test and libc6 before the tests run An interop run tests hiroz against a specific set of ROS binaries, and nothing recorded which
3 Retry the Nushell download, and stop piping it into tar A dropped connection failed a whole leg and reported it as a corrupt archive

Change 1 — the actual fix

ros:lyrical-ros-base ships rclcpp 32.0.0 (built 2026-06-06). The repository serves 32.0.2 (2026-07-30), the same build date as demo-nodes-cpp 0.37.9. Installing demo-nodes-cpp pulls the July binary but leaves June's librclcpp.so in place, because the dependency is already satisfied.

rclcpp::ExecutorOptions holds a pimpl and is stack-allocated in demo_nodes_cpp's send_request, so the layout mismatch makes ~ExecutorOptions() free a stack address. glibc reports double free or corruption (out) and the client aborts with exit 250.

The defect is not in hiroz, and not in zenoh: it reproduces on a stock image with the default RMW and no hiroz process. Full diagnosis, including the valgrind trace and the hypotheses ruled out, is in #303.

Change 3 — why it is not just a retry

The observed failure was:

curl: (56) Connection died, tried 5 times before giving up
gzip: stdin: unexpected end of file
tar: Error is not recoverable: exiting now      → exit 2

Piping curl into tar means the step's exit status comes from tar, so a network failure is indistinguishable from a bad tarball. The download now goes to a file first. --retry alone does not cover a connection reset mid-transfer; --retry-all-errors does.

Evidence

Check Result
Interop legs on 017fc88d humble, jazzy, kilted, lyrical — all SUCCESS
lyrical before this PR red since 2026-08-06, test_hiroz_add_two_ints_server_to_rcl_client aborting
Fix confirmed in a container, both directions without change 1: exit 250, double free or corruption. With it: Result of add_two_ints: 5, exit 0

Change 1 is not inert on the other distros, and the description previously implied it was. Naming rclcpp also upgrades it on humble: ros-humble-rclcpp moves from 16.0.19-1jammy.20260421 to 16.0.19-1jammy.20260724, so humble now runs a July librclcpp.so against April's rcl, rcutils, rmw and rosidl-runtime-c. jazzy and kilted are untouched (already the newest version).

That mixed set is safe today, and it was checked rather than assumed: diffing installed-versus-candidate headers inside the image for every package rclcpp depends on shows them byte-identical apart from a patch-version macro in rcpputils/version.h. Green on one run is evidence the suite passes, not evidence the inputs are unchanged — so the change is stated here rather than left to be discovered.

What this does not do

  • It does not fix the images. They will drift again whenever the repository moves ahead of them. The durable fix is for the images to pin their apt source, not only the package version — reported upstream as osrf/docker_images#897, and linked from lyrical interop: image ships rclcpp 32.0.0 while the repo ships 32.0.2, breaking ExecutorOptions ABI #303.
  • It fixes rclcpp only, which is sufficient today rather than in general. The image also bakes in rclcpp-action, -components, -lifecycle, tf2_ros, robot_state_publisher and rosbag2 at the older build, and ROS debs carry unversioned dependencies, so naming rclcpp cannot drag them along. After this change those image-built consumers disagree with the newer librclcpp.so — the skew is inverted rather than eliminated. It does not fire today, and that was tested rather than assumed: 32.0.2 removed a 16-byte member, so an old-layout caller over-allocates and the new library under-writes, which is harmless, whereas lyrical interop: image ships rclcpp 32.0.0 while the repo ships 32.0.2, breaking ExecutorOptions ABI #303 was the over-writing direction. Running component_container, buffer_server, robot_state_publisher and rosbag2 recorder under rmw_zenoh_cpp produced no corruption. It would matter if the interop suite grew a tf2 or rosbag2 test, or if the next upstream layout change went the other way.
  • Change 2 records versions but does not pin them. ROS apt repositories keep only the newest build of each package, so pinning is not usable here — a pin stops resolving the next time upstream rebuilds.

Breaking changes

None. CI only.

An interop run tests hiroz against a specific set of ROS packages, and
nothing recorded which. When the lyrical leg started failing (#303) the
first question -- what changed? -- could not be answered from the logs,
and re-running a job replaces its log, so the last-known-good run's
inputs were already gone.

Prints the image, the OS release, the four ROS packages under test and
libc6. That is what ruled out three hypotheses in #303: the image, the
package builds, and the C runtime.
The ros:<distro>-ros-base images pin ros-base to a fixed version, so the
rclcpp they bake in tracks that pin rather than the apt repository. Every
other ROS package this job installs comes from the repository. When the
image is behind, the two disagree on ABI.

On lyrical the image shipped rclcpp 32.0.0 (2026-06-06) while the repo
served 32.0.2 (2026-07-30, the same build date as demo-nodes-cpp 0.37.9).
ExecutorOptions holds a pimpl and is stack-allocated in demo_nodes_cpp's
send_request, so the layout mismatch made ~ExecutorOptions() free a stack
address; glibc reported it as double free and the client aborted.

Naming rclcpp explicitly makes apt upgrade it. Verified in a container:
without it the client aborts with exit 250, with it the client prints
Result of add_two_ints: 5 and exits 0.

Fixes #303.
A dropped connection failed a whole interop leg, and reported it as a
corrupt archive:

  curl: (56) Connection died, tried 5 times before giving up
  gzip: stdin: unexpected end of file
  tar: Error is not recoverable: exiting now   -> exit 2

Piping curl into tar means the step's exit code comes from tar, so a
network failure is indistinguishable from a bad tarball. Downloads to a
file first, then extracts.

--retry alone does not cover a connection reset mid-transfer;
--retry-all-errors does.
@YuanYuYuan YuanYuYuan changed the title ci(interop): record the ROS binaries each run tests against ci(interop): fix the lyrical ABI skew and make interop failures diagnosable Aug 13, 2026
…er test

The recording step listed rmw-zenoh-cpp, demo-nodes-cpp, example-interfaces
and action-tutorials-cpp. All four are installed from the repository on
every run, so they always read as current.

The skew that caused #303 lives in the core libraries baked into the
image -- rclcpp above all, the package this PR changes. A recurrence
would have printed four unchanged versions and reported that nothing
changed, which is the opposite of what the step is for.

Adds rclcpp, rcl, rmw, rcutils, rcpputils and rosidl-runtime-c.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

lyrical interop: image ships rclcpp 32.0.0 while the repo ships 32.0.2, breaking ExecutorOptions ABI

1 participant