Re-entrancy: the fix shape costs one drain thread per callback subscriber

[YuanYuYuan]

Part of #282.

Role: residual, introduced by the fix shape — the same category as #287. #250 does not have this defect by mistake; the defect follows from the only fix its API shape allows.

The cost

#250 moves the callout off the delivery thread onto a per-subscriber drain thread, spawned eagerly when the subscriber is built. A callback subscriber therefore costs a thread whether or not it ever receives a locally published sample.

Measured on Linux x86-64, release build, reading /proc/self/status before and after creating N subscribers:

N callback subscribers	threads before #250	threads after
1	6	7
10	6	16
40	6	46
100	6	106

Exactly baseline + N, on both Volatile and TransientLocal. A notifier-mode (queue) subscriber control stays flat at 6 even at N=100, which pins the growth to callback subscribers specifically.

Per subscriber, at N=100:

	before	after	delta
VmRSS	16.3 MB	17.7 MB	+12 KB
VmSize	364 MB	6,790 MB	+66.0 MB

Resident cost is negligible. The address-space figure is 66.0 MB per subscriber and perfectly linear across all four values of N. It decomposes as a 2 MB thread stack plus a 64 MB glibc per-thread malloc arena — capping MALLOC_ARENA_MAX floors the per-subscriber figure at exactly 2.03 MB:

MALLOC_ARENA_MAX	VmSize at N=100	per subscriber
unset	6,790 MB	66.0 MB
8	686 MB	3.3 MB
2	310 MB	2.03 MB
1	249 MB	2.03 MB

Threads return to baseline when the subscribers are dropped. There is no leak.

Scope — who actually pays

The ROS 2 path is unaffected. rmw-zenoh-rs builds subscriptions with build_with_notifier, so runs_user_code is false and no dispatcher is created. This cost falls on native Rust build_with_callback and hiroz-py callback subscribers only.

That also makes hiroz-via-rmw structurally identical to rmw_zenoh_cpp here, which creates no threads of its own at all — its zenoh callback enqueues and notifies, and the rclcpp executor runs the user callback. One executor serves every subscription, so upstream is O(1) in threads.

The difference is API shape, not implementation quality. A pull API (rmw_take + waitset, or a notifier) borrows the caller's thread. A push API — "give me a closure, I call it" — has no executor for the user to spin, so if the callout may not run on the delivery thread, the library must bring a thread. The cost is forced by the contract #250 has to honour.

The binding limit is address space, not memory

At realistic subscriber counts the resident cost is invisible. What binds first:

• Thread count against pids.max / ulimit -u. 100 subscribers is 106 threads; 1,000 is ~1,006.
• Address space against ulimit -v, a constrained container, or a 32-bit target — 66 MB per subscriber reaches limits long before RSS does.

Fix

Start the drain thread on first enqueue rather than at construction. On the plain path the queue is already conditional — a sample only enters it when a local publish is on the delivering thread, and everything else runs inline — so the thread is the only unconditional cost. A subscriber that never sees a session-local publish then costs nothing.

This is implemented on fix/lazy-dispatch-thread and verified in both directions:

	Result
With the fix, default parallel run	3/3 new tests, plus #250's existing suites 4/4 and 11/11
The same detector file against #250's unmodified eager tip	fails — creating 8 callback subscribers started 8 drain thread(s)

No PR is open yet, deliberately. The change depends on CallbackDispatcher, which exists only on #250's branch, so a PR would have to target that branch — the configuration that auto-closed #257 and #260 when their base was deleted on merge. It is held until #250 merges, then boundary-rebased onto main.

What this does not fix

• A TransientLocal callback subscriber still spawns on its first received sample, because always_shim enqueues unconditionally. Lazy spawn removes the cost for the ROS 2 default profile, not for durable subscribers under traffic.
• Many concurrently active callback subscribers still cost one thread each. Only a shared per-node dispatcher would make that O(1), and that trades away the per-topic isolation a dedicated thread provides — a single-threaded executor head-of-line blocks across subscriptions, which hiroz currently does not.
• MALLOC_ARENA_MAX is a deployment knob, not a library fix. It is process-global and its contention cost lands on the whole application, so hiroz should document it rather than call mallopt on a user's behalf.