[Zbig9000]

Summary

• Destroy every Vulkan device and the instance from an atexit handler registered after the instance and each device are created, so it runs before the driver's own exit handlers.
• Split the vk_device_struct destructor into an idempotent shutdown() that waits for the device to go idle, skips a device that was never created, and destroys the VMA allocator only when it holds no allocations.
• After the teardown, buffers, pinned host memory and backend contexts freed later by the application skip their Vulkan calls.
• New test-vulkan-teardown: initialise Vulkan0, allocate and free a buffer, exit; a second ctest entry exits with the buffer still allocated.

Error corrected

On the Jetson Orin (L4T 36.4, NVIDIA Tegra Vulkan driver 540.4.0, nvgpu), the speech CLIs finished their work, wrote their results, and then died with SIGSEGV at exit. The weekly bench therefore recorded 5 of the Jetson's 13 Vulkan cells as failed although their measurements were complete. Affected: Parakeet TDT, Parakeet Unified, Audio8 with the F16 codec decoder, Audio8 cloning with the F16 encoder.

The backend never destroyed its devices or its instance. Each device's own buffer type holds a shared_ptr to it, so ~vk_device_struct never ran, and vk_instance_t has no destructor. The NVIDIA libraries register their exit handlers when vkCreateInstance loads them. At exit, libEGL_nvidia's handler unmaps libnvidia-eglcore while a driver worker thread still runs code in it. Only vkDestroyDevice stops that thread. A crash-time signal handler showed the faulting pc in the unmapped libnvidia-eglcore range on that worker thread, right after the unmap.

A static destructor does not fix it. It is registered when libqvac-speech-ggml-vulkan loads, which is before the driver loads, so it runs after the driver's handlers. That version crashed in 20 of 20 runs. The fix registers the teardown with atexit after the driver is loaded.

Windows keeps the previous behaviour, because the exit handlers of a DLL run under the loader lock and Vulkan calls there can deadlock.

Validation

setup	runs	exit 0
Jetson Orin, parakeet --bench, unpatched speech@de1d0f5c	5	3
Jetson Orin, parakeet --bench, this branch	10	10
Jetson Orin, audio8-cli LM F16 + codec decoder F16, this branch	10	10

• Every Jetson run above completed its measurement. The Parakeet RTF is unchanged at 0.035.
• ctest -R test-vulkan passes 4/4 three times on the Jetson. It also passes on an RTX 5090 (proprietary driver) and on an AMD Raphael iGPU (RADV).
• test-backend-ops -o MUL_MAT and -o ADD pass on the RTX 5090.
• Pre-existing and unrelated: two q8_0 ADD cases (m=100,n=37,k=96, modes 1 and 2) give NaN on the AMD iGPU. The unpatched speech tip shows the same result.

Follow-up

• After the merge, the engine needs a ggml-speech registry cut at the merge commit on speech for the prebuilt addons. The weekly bench builds the speech head directly, so it picks this up on its next run.

🤖 Generated with Claude Code

[GustavoA1604]

[critical][pr-review-core] src/ggml-vulkan/ggml-vulkan.cpp — line 9327 (ggml_vk_shutdown_devices();)

An application cleanup handler registered before backend initialization now runs after device destruction. If it frees a retained Vulkan event, ggml_backend_vk_device_event_free() still calls destroySemaphore() and destroyEvent() on the destroyed device, which can crash during exit. Scheduler cleanup also reaches this path. Guard event destruction with device->destroyed, while still deleting the CPU objects, and add a regression test for this ordering.

[medium][coding-standards] tests/test-vulkan-teardown.cpp — line 25 (ggml_backend_free(backend);)

Both tests free the backend before teardown; the kept-buffer variant never frees its buffer. Neither exercises the new guards for resources freed after device destruction. Add a cleanup callback registered before backend initialization that frees retained device buffers, pinned host buffers, and the backend after teardown.

[Zbig9000]

[critical][pr-review-core] src/ggml-vulkan/ggml-vulkan.cpp — line 9327 (ggml_vk_shutdown_devices();)

An application cleanup handler registered before backend initialization now runs after device destruction. If it frees a retained Vulkan event, ggml_backend_vk_device_event_free() still calls destroySemaphore() and destroyEvent() on the destroyed device, which can crash during exit. Scheduler cleanup also reaches this path. Guard event destruction with device->destroyed, while still deleting the CPU objects, and add a regression test for this ordering.

[medium][coding-standards] tests/test-vulkan-teardown.cpp — line 25 (ggml_backend_free(backend);)

Both tests free the backend before teardown; the kept-buffer variant never frees its buffer. Neither exercises the new guards for resources freed after device destruction. Add a cleanup callback registered before backend initialization that frees retained device buffers, pinned host buffers, and the backend after teardown.

Both findings are addressed in 03171fa.

[critical][pr-review-core] src/ggml-vulkan/ggml-vulkan.cpp — line 9327 (ggml_vk_shutdown_devices();)
An application cleanup handler registered before backend initialization now runs after device destruction. If it frees a retained Vulkan event, ggml_backend_vk_device_event_free() still calls destroySemaphore() and destroyEvent() on the destroyed device, which can crash during exit. Scheduler cleanup also reaches this path. Guard event destruction with device->destroyed, while still deleting the CPU objects, and add a regression test for this ordering.

Fixed: ggml_backend_vk_device_event_free now calls ggml_vk_destroy_event_handles (semaphore, free and submitted events, the recorded event) only while device->destroyed is false, and deletes the CPU objects in every case. The same exit ordering also reached freed memory before any Vulkan call: the registry kept its device objects and device contexts in function-local statics that the runtime destroyed before such a handler ran. They now live for the whole process (ggml_vk_never_destroyed), as the CUDA backend’s device contexts already do. The regression test is the new ctest test-vulkan-teardown-free-after-teardown (see below); without the guard it aborts.

[medium][coding-standards] tests/test-vulkan-teardown.cpp — line 25 (ggml_backend_free(backend);)
Both tests free the backend before teardown; the kept-buffer variant never frees its buffer. Neither exercises the new guards for resources freed after device destruction. Add a cleanup callback registered before backend initialization that frees retained device buffers, pinned host buffers, and the backend after teardown.

Fixed: test-vulkan-teardown --free-after-teardown registers free_retained with std::atexit before ggml_backend_load_all, then initializes the backend, allocates a device buffer and a pinned host buffer, records an event, and returns; the handler frees the event, both buffers and the backend after the device teardown. It is wired as its own ctest in tests/CMakeLists.txt. The kept-buffer variant keeps leaking its buffer on purpose: it is the “buffer still alive at teardown” case the first commit fixed, and the new variant is the one that exercises the guards. The test passes on the RTX 5090 box and on the Jetson Orin, as reported by the session that ran it.

[github-actions]

Review Status

Current Status: ❌ PENDING
Approvals so far: none

Pending reviews: Needs 1 Management or Team Lead, and 1 more from Management, Team Lead, or Member.

[Zbig9000]

/review

[Alok-Ranjan23]

I reviewed the current head (7231ceeb) and found two teardown issues:

1. [P1] Destroy live Vulkan children before vkDestroyDevice. vk_device_struct::shutdown() calls device.destroy() while a retained buffer still owns a VMA allocation, and a retained backend can still own fences, command pools, semaphores, descriptor pools, and events. The new ggml_vk_cleanup() guard then skips their cleanup when the backend is freed after teardown. This violates VUID-vkDestroyDevice-device-05137 and makes the exit path dependent on driver behavior. Please release all live Vulkan children before destroying the device, while leaving their CPU wrappers safe to free later. See shutdown, late cleanup, and the Vulkan requirement.

2. [P2] Keep vk_instance alive for cleanup handlers registered before dynamic backend loading. With GGML_BACKEND_DL=ON, the test registers free_retained before ggml_backend_load_all() loads the Vulkan DSO. That DSO's nontrivial vk_instance static is constructed afterward, so its destructor runs before free_retained at exit. The latter still reaches vk_instance.devices through event and pinned-host-buffer frees. Leaking the registry device/context vectors does not preserve vk_instance itself, leaving those accesses outside its lifetime. Please retain the device directly in those resources or keep the lookup state alive through late cleanup. See test registration, vk_instance, event free, and host-buffer free.

I attempted a local dynamic Vulkan build, but CMake configuration stopped because SPIRV-HeadersConfig.cmake is unavailable, so these are source-review findings rather than runtime reproductions.

[Zbig9000]

I reviewed the current head (7231ceeb) and found two teardown issues:

1. **[P1] Destroy live Vulkan children before `vkDestroyDevice`.** `vk_device_struct::shutdown()` calls `device.destroy()` while a retained buffer still owns a VMA allocation, and a retained backend can still own fences, command pools, semaphores, descriptor pools, and events. The new `ggml_vk_cleanup()` guard then skips their cleanup when the backend is freed after teardown. This violates `VUID-vkDestroyDevice-device-05137` and makes the exit path dependent on driver behavior. Please release all live Vulkan children before destroying the device, while leaving their CPU wrappers safe to free later. See [shutdown](https://github.com/tetherto/qvac-ext-ggml/blob/7231ceeb60e427f0a64a11b47320dc9c70916be4/src/ggml-vulkan/ggml-vulkan.cpp#L1381), [late cleanup](https://github.com/tetherto/qvac-ext-ggml/blob/7231ceeb60e427f0a64a11b47320dc9c70916be4/src/ggml-vulkan/ggml-vulkan.cpp#L20143), and the [Vulkan requirement](https://docs.vulkan.org/refpages/latest/refpages/source/vkDestroyDevice.html).

2. **[P2] Keep `vk_instance` alive for cleanup handlers registered before dynamic backend loading.** With `GGML_BACKEND_DL=ON`, the test registers `free_retained` before `ggml_backend_load_all()` loads the Vulkan DSO. That DSO's nontrivial `vk_instance` static is constructed afterward, so its destructor runs before `free_retained` at exit. The latter still reaches `vk_instance.devices` through event and pinned-host-buffer frees. Leaking the registry device/context vectors does not preserve `vk_instance` itself, leaving those accesses outside its lifetime. Please retain the device directly in those resources or keep the lookup state alive through late cleanup. See [test registration](https://github.com/tetherto/qvac-ext-ggml/blob/7231ceeb60e427f0a64a11b47320dc9c70916be4/tests/test-vulkan-teardown.cpp#L86), [`vk_instance`](https://github.com/tetherto/qvac-ext-ggml/blob/7231ceeb60e427f0a64a11b47320dc9c70916be4/src/ggml-vulkan/ggml-vulkan.cpp#L3372), [event free](https://github.com/tetherto/qvac-ext-ggml/blob/7231ceeb60e427f0a64a11b47320dc9c70916be4/src/ggml-vulkan/ggml-vulkan.cpp#L23693), and [host-buffer free](https://github.com/tetherto/qvac-ext-ggml/blob/7231ceeb60e427f0a64a11b47320dc9c70916be4/src/ggml-vulkan/ggml-vulkan.cpp#L20425).

I attempted a local dynamic Vulkan build, but CMake configuration stopped because SPIRV-HeadersConfig.cmake is unavailable, so these are source-review findings rather than runtime reproductions.

Both fixed in 7cceb37.

P1, children destroyed after vkDestroyDevice. Each vk_device_struct now tracks its live backend contexts, events and VMA buffers. shutdown() cleans up the backends, destroys the event semaphores and events and releases the remaining buffers before it destroys the device; the CPU-side wrappers stay valid and a later free skips the Vulkan calls. ggml_vk_cleanup() also destroys the perf-logger query pool, which leaked before. The teardown ctests now run under VK_LAYER_KHRONOS_validation when the layer is installed and fail on any vkDestroy* validation error: without the release step the kept-buffer and free-after-teardown variants report 4 and 20 VUID-vkDestroyDevice-device-05137 errors, with it none.

P2, vk_instance lifetime with GGML_BACKEND_DL. vk_instance is never destroyed any more, so a cleanup handler registered before the Vulkan module loads can still free events and pinned host buffers through it.

Verified: ctest -R test-vulkan under validation on the RTX 5090 and the AMD iGPU (5/5, three runs each); the teardown test with the backend loaded as a module in all three modes (exit 0, no validation errors); on Jetson Orin 10/10 parakeet and audio8 runs with complete output and unchanged speed, and 5/5 ctests under its 1.3.204 layer. Your configure failure is reproducible: with dynamic backends and tests both on, test-vulkan-status links ggml-vulkan directly and CMake stops; that predates this PR, so I built the module-loaded teardown test by hand.

[Zbig9000]

/review