ci(interop): fix the lyrical ABI skew and make interop failures diagnosable - #305
Open
YuanYuYuan wants to merge 4 commits into
Open
ci(interop): fix the lyrical ABI skew and make interop failures diagnosable#305YuanYuYuan wants to merge 4 commits into
YuanYuYuan wants to merge 4 commits into
Conversation
An interop run tests hiroz against a specific set of ROS packages, and nothing recorded which. When the lyrical leg started failing (#303) the first question -- what changed? -- could not be answered from the logs, and re-running a job replaces its log, so the last-known-good run's inputs were already gone. Prints the image, the OS release, the four ROS packages under test and libc6. That is what ruled out three hypotheses in #303: the image, the package builds, and the C runtime.
The ros:<distro>-ros-base images pin ros-base to a fixed version, so the rclcpp they bake in tracks that pin rather than the apt repository. Every other ROS package this job installs comes from the repository. When the image is behind, the two disagree on ABI. On lyrical the image shipped rclcpp 32.0.0 (2026-06-06) while the repo served 32.0.2 (2026-07-30, the same build date as demo-nodes-cpp 0.37.9). ExecutorOptions holds a pimpl and is stack-allocated in demo_nodes_cpp's send_request, so the layout mismatch made ~ExecutorOptions() free a stack address; glibc reported it as double free and the client aborted. Naming rclcpp explicitly makes apt upgrade it. Verified in a container: without it the client aborts with exit 250, with it the client prints Result of add_two_ints: 5 and exits 0. Fixes #303.
A dropped connection failed a whole interop leg, and reported it as a corrupt archive: curl: (56) Connection died, tried 5 times before giving up gzip: stdin: unexpected end of file tar: Error is not recoverable: exiting now -> exit 2 Piping curl into tar means the step's exit code comes from tar, so a network failure is indistinguishable from a bad tarball. Downloads to a file first, then extracts. --retry alone does not cover a connection reset mid-transfer; --retry-all-errors does.
…er test The recording step listed rmw-zenoh-cpp, demo-nodes-cpp, example-interfaces and action-tutorials-cpp. All four are installed from the repository on every run, so they always read as current. The skew that caused #303 lives in the core libraries baked into the image -- rclcpp above all, the package this PR changes. A recurrence would have printed four unchanged versions and reported that nothing changed, which is the opposite of what the step is for. Adds rclcpp, rcl, rmw, rcutils, rcpputils and rosidl-runtime-c.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three changes to the interop job. One turns the red
lyricalleg green; two make the next failure diagnosable instead of a guess.Fixes #303.
What this PR does
ros-<distro>-rclcppexplicitlyrclcppmakes apt upgrade the copy baked into the image, so it matches the repo everything else comes fromlibc6before the tests runtarChange 1 — the actual fix
ros:lyrical-ros-baseshipsrclcpp32.0.0 (built 2026-06-06). The repository serves 32.0.2 (2026-07-30), the same build date asdemo-nodes-cpp0.37.9. Installingdemo-nodes-cpppulls the July binary but leaves June'slibrclcpp.soin place, because the dependency is already satisfied.rclcpp::ExecutorOptionsholds a pimpl and is stack-allocated indemo_nodes_cpp'ssend_request, so the layout mismatch makes~ExecutorOptions()free a stack address. glibc reportsdouble free or corruption (out)and the client aborts with exit 250.The defect is not in hiroz, and not in zenoh: it reproduces on a stock image with the default RMW and no hiroz process. Full diagnosis, including the valgrind trace and the hypotheses ruled out, is in #303.
Change 3 — why it is not just a retry
The observed failure was:
Piping
curlintotarmeans the step's exit status comes fromtar, so a network failure is indistinguishable from a bad tarball. The download now goes to a file first.--retryalone does not cover a connection reset mid-transfer;--retry-all-errorsdoes.Evidence
017fc88dlyricalbefore this PRtest_hiroz_add_two_ints_server_to_rcl_clientabortingdouble free or corruption. With it:Result of add_two_ints: 5, exit 0Change 1 is not inert on the other distros, and the description previously implied it was. Naming
rclcppalso upgrades it on humble:ros-humble-rclcppmoves from16.0.19-1jammy.20260421to16.0.19-1jammy.20260724, so humble now runs a Julylibrclcpp.soagainst April'srcl,rcutils,rmwandrosidl-runtime-c. jazzy and kilted are untouched (already the newest version).That mixed set is safe today, and it was checked rather than assumed: diffing installed-versus-candidate headers inside the image for every package
rclcppdepends on shows them byte-identical apart from a patch-version macro inrcpputils/version.h. Green on one run is evidence the suite passes, not evidence the inputs are unchanged — so the change is stated here rather than left to be discovered.What this does not do
osrf/docker_images#897, and linked from lyrical interop: image ships rclcpp 32.0.0 while the repo ships 32.0.2, breaking ExecutorOptions ABI #303.rclcpponly, which is sufficient today rather than in general. The image also bakes inrclcpp-action,-components,-lifecycle,tf2_ros,robot_state_publisherandrosbag2at the older build, and ROS debs carry unversioned dependencies, so namingrclcppcannot drag them along. After this change those image-built consumers disagree with the newerlibrclcpp.so— the skew is inverted rather than eliminated. It does not fire today, and that was tested rather than assumed: 32.0.2 removed a 16-byte member, so an old-layout caller over-allocates and the new library under-writes, which is harmless, whereas lyrical interop: image ships rclcpp 32.0.0 while the repo ships 32.0.2, breaking ExecutorOptions ABI #303 was the over-writing direction. Runningcomponent_container,buffer_server,robot_state_publisherandrosbag2 recorderunderrmw_zenoh_cppproduced no corruption. It would matter if the interop suite grew a tf2 or rosbag2 test, or if the next upstream layout change went the other way.Breaking changes
None. CI only.