The Interop tests with ROS 2 lyrical job fails on every PR. Root cause found: it is not a hiroz defect, not a zenoh defect, and not an upstream code defect. It is an ABI skew between the ros:lyrical-ros-base image and the ROS apt repository, and hiroz CI can fix it in one line.
Reproduction — no hiroz, no zenoh
docker run --rm --network host ros:lyrical-ros-base bash -c '
apt-get update -qq && apt-get install -y -qq ros-lyrical-demo-nodes-cpp
source /opt/ros/lyrical/setup.bash
ros2 run demo_nodes_cpp add_two_ints_server &
sleep 5
ros2 run demo_nodes_cpp add_two_ints_client
'
# double free or corruption (out)
# [ros2run]: Aborted → exit 250
demo_nodes_cpp's own client crashes against demo_nodes_cpp's own server, on a stock image, with the default RMW. rmw_zenoh is not installed and hiroz is not running.
Root cause
| Package |
Version |
Built |
ros-lyrical-rclcpp installed from the image |
32.0.0 |
2026-06-06 |
ros-lyrical-rclcpp available in the repo |
32.0.2 |
2026-07-30 |
ros-lyrical-demo-nodes-cpp installed from the repo |
0.37.9 |
2026-07-30 |
The repo is self-consistent — rclcpp 32.0.2 and demo_nodes_cpp 0.37.9 were built the same day. The image is roughly two months behind it. apt-get install ros-lyrical-demo-nodes-cpp pulls the July binary but leaves the June librclcpp.so in place, because the dependency is satisfied loosely.
rclcpp::ExecutorOptions holds a pimpl, so ~ExecutorOptions() performs delete impl_. It is stack-allocated inside send_request. When the caller's layout does not match the loaded librclcpp.so, the destructor deletes whatever sits at that offset — here, a stack address. valgrind:
Invalid free() / delete / delete[] / realloc()
at operator delete(void*, unsigned long)
by rclcpp::ExecutorOptions::~ExecutorOptions() (librclcpp.so)
by send_request(...) → main
Address 0x1ffefffb60 is on thread 1's stack
glibc reports the same corruption as double free or corruption (out).
The fix, verified
Install rclcpp explicitly so apt upgrades it to the repo's version:
apt-get install -y ros-lyrical-demo-nodes-cpp ros-lyrical-rclcpp
ros-lyrical-rclcpp 32.0.2-1resolute.20260730.192517
[INFO] Result of add_two_ints: 5 → exit 0
The general rule: do not mix packages baked into a ROS image with newer packages from the repo. Either upgrade the ROS packages after apt-get update, or pin to the image's set.
What was ruled out along the way
| Hypothesis |
Verdict |
| A flake |
No — reproduces every time |
| Container image drift |
No — the tag was last updated 2026-08-04, before both the green and red runs |
| ROS package upgrade |
No — the repo carries a single build of each, predating both runs |
| Ubuntu/glibc update |
No — the failing job upgrades nothing from the Ubuntu mirror |
| hiroz's service-reply encoding |
No — the client prints Result of add_two_ints: 5 under valgrind; the data is correct, the crash is in teardown |
zenoh / rmw_zenoh_cpp |
No — reproduces with the default RMW and rmw_zenoh not installed |
Why it looked hiroz-shaped for so long: test_hiroz_add_two_ints_server_to_rcl_client was the only interop test asserting on a C++ child's exit status. Two of the three hiroz → C++ tests asserted nothing about the C++ process at all, so "8 of 9 pass" was an artifact of coverage. #306 fixes that; #304 made the client's own error message visible, which is what produced this diagnosis in a single run.
Actions
No hiroz source change is required, and the other three distro legs were never affected.
The
Interop tests with ROS 2 lyricaljob fails on every PR. Root cause found: it is not a hiroz defect, not a zenoh defect, and not an upstream code defect. It is an ABI skew between theros:lyrical-ros-baseimage and the ROS apt repository, and hiroz CI can fix it in one line.Reproduction — no hiroz, no zenoh
demo_nodes_cpp's own client crashes againstdemo_nodes_cpp's own server, on a stock image, with the default RMW.rmw_zenohis not installed and hiroz is not running.Root cause
ros-lyrical-rclcppinstalled from the imageros-lyrical-rclcppavailable in the reporos-lyrical-demo-nodes-cppinstalled from the repoThe repo is self-consistent —
rclcpp32.0.2 anddemo_nodes_cpp0.37.9 were built the same day. The image is roughly two months behind it.apt-get install ros-lyrical-demo-nodes-cpppulls the July binary but leaves the Junelibrclcpp.soin place, because the dependency is satisfied loosely.rclcpp::ExecutorOptionsholds a pimpl, so~ExecutorOptions()performsdelete impl_. It is stack-allocated insidesend_request. When the caller's layout does not match the loadedlibrclcpp.so, the destructor deletes whatever sits at that offset — here, a stack address. valgrind:glibc reports the same corruption as
double free or corruption (out).The fix, verified
Install
rclcppexplicitly so apt upgrades it to the repo's version:The general rule: do not mix packages baked into a ROS image with newer packages from the repo. Either upgrade the ROS packages after
apt-get update, or pin to the image's set.What was ruled out along the way
Result of add_two_ints: 5under valgrind; the data is correct, the crash is in teardownrmw_zenoh_cpprmw_zenohnot installedWhy it looked hiroz-shaped for so long:
test_hiroz_add_two_ints_server_to_rcl_clientwas the only interop test asserting on a C++ child's exit status. Two of the three hiroz → C++ tests asserted nothing about the C++ process at all, so "8 of 9 pass" was an artifact of coverage. #306 fixes that; #304 made the client's own error message visible, which is what produced this diagnosis in a single run.Actions
ros-${{ matrix.distro }}-rclcppto the interop job's apt install list. That is the whole fix.osrf/docker_images#897— the images pin the ROS package version but not the apt source. Includes the reproduction, the valgrind trace, and a verified fix (pin the source to a matching snapshot), offered as evidence rather than as a merge candidate: the real change belongs in the image template, and choosing the snapshot date is a policy decision for the maintainers. Not blocking us — ci(interop): fix the lyrical ABI skew and make interop failures diagnosable #305 fixes our side.No hiroz source change is required, and the other three distro legs were never affected.