fix(waylib): fix Vulkan dmabuf RT and SHM upload via wlroots

[zccrs]

Summary

Fix Vulkan rendering without copying wlroots internals (supersedes the approach in #1132; core scanout path aligned with #1171).

Changes

• wlroots (thin waylib_ APIs)
  • waylib_vk_renderer_get_render_buffer_attribs() — reuse wlroots wlr_vk_render_buffer (same VkImage as tinywl/scene)
  • waylib_vk_renderer_record_render_buffer_acquire/release() — VK_QUEUE_FAMILY_FOREIGN_EXT ownership transfer (same as wlroots pass.c)
  • waylib_vk_renderer_flush_stage() — flush SHM stage CB before Qt samples
  • Export waylib_* in wlroots.syms
• waylib
  • Qt RHI render targets use the wlroots render_buffer image, not a parallel dmabuf import
  • Acquire before Qt pass / release after, then QRhiTexture::setNativeLayout(GENERAL)
  • Keep SHM partial updates + stage flush
  • ColorContentsMode and WRenderHelper::RenderTarget lifecycle
• effects / DTK override (Vulkan UI path)
  • Blur.qml: skip RenderBufferBlitter on Vulkan for now
  • Override DTK InWindowBlur → Treeland Blur
  • Unit test: tests/test_dtk_inwindowblur_override

Why not DCC modifier filtering

Filtering AMD DCC in output_pick_format was wrong: tinywl is fine with the same modifiers; stripping DCC black-screened some AMD machines. Dropped.

Why not #1132 style

Vendor wlroots → thin helpers, not a ~600-line copy of import logic.

Relation to #1171

Core scanout ownership model matches #1171 (get_render_buffer + FOREIGN acquire/release + QRhi layout sync). This PR stays smaller: no full presentation/blitter/trace rewrite yet.

摘要

复用 wlroots wlr_vk_render_buffer（与 tinywl 同路径），Qt 绘制前后做 FOREIGN_EXT acquire/release 并同步 QRhi layout；保留 SHM stage flush。不再平行 import dmabuf，也不再过滤 AMD DCC。核心与 #1171 对齐，范围更小。

Acknowledgements

Thanks to LFRon for identifying the Qt RHI/wlroots Vulkan integration issue and suggesting in #1132 that Treeland reuse wlroots APIs, which helped guide this implementation.

致谢

感谢 LFRon 发现 Qt RHI 与 wlroots Vulkan 集成问题，并在 #1132 中建议 Treeland 直接复用 wlroots 暴露的 API，为本次实现提供了重要参考。

Test plan

• cmake --build build --target waylib-wlroots waylibserver
• ctest -R test_dtk_inwindowblur_override (local)
• Nested Vulkan X11: SHM / EGL dmabuf / flower clients
• AMD Raven Vulkan without RADV_DEBUG=nodcc: no white/black/garbage
• Confirm SHM damage/partial update still works

[sourcery-ai]

Sorry @zccrs, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[sourcery-ai]

Reviewer's Guide

Fix Vulkan dmabuf render targets and SHM uploads in waylib by adding thin wlroots Vulkan wrappers, introducing a ColorContentsMode/RenderTarget lifecycle, and wiring color contents handling through outputs, layers, viewports, and buffer rendering paths.

Sequence diagram for Vulkan dmabuf render target lifecycle

sequenceDiagram
    actor Client
    participant OutputHelper
    participant WBufferRenderer
    participant WRenderHelper
    participant wlr_renderer
    participant QtRHI as QRhi

    Client->>OutputHelper: beginRender(pixelSize, format, flags, mode)
    OutputHelper->>WBufferRenderer: beginRender(pixelSize, dpr, format, flags, mode)
    WBufferRenderer->>WRenderHelper: acquireRenderTarget(renderControl, buffer, mode)
    WRenderHelper->>wlr_renderer: waylib_vk_renderer_import_dmabuf(renderer, dmabuf, vkImage)
    wlr_renderer-->>WRenderHelper: vkImage.image, vkImage.format
    WRenderHelper-->>WBufferRenderer: RenderTarget (vkImage, colorPreserved)
    WBufferRenderer->>QtRHI: renderNextFrame(renderer)
    WBufferRenderer->>WRenderHelper: prepareVulkanRenderTarget(commandBuffer, RenderTarget)
    WRenderHelper->>QtRHI: beginExternal()
    WRenderHelper->>QtRHI: vkCmdPipelineBarrier (GENERAL -> COLOR_ATTACHMENT_OPTIMAL)
    WRenderHelper->>QtRHI: endExternal()
    WBufferRenderer->>WRenderHelper: finishVulkanRenderTarget(commandBuffer, RenderTarget)
    WRenderHelper->>QtRHI: beginExternal()
    WRenderHelper->>QtRHI: vkCmdPipelineBarrier (COLOR_ATTACHMENT_OPTIMAL -> GENERAL)
    WRenderHelper->>QtRHI: endExternal()
    note over WRenderHelper,wlr_renderer: BufferData destructor calls
    WRenderHelper->>wlr_renderer: waylib_vk_imported_image_finish(renderer, vkImage)

Loading

File-Level Changes

Change	Details	Files
Introduce thin waylib Vulkan wrapper APIs on vendored wlroots to import dmabuf images as COLOR_ATTACHMENT targets and flush the Vulkan stage command buffer.	Add waylib_vk_imported_image struct and waylib_vk_renderer_import_dmabuf/waylib_vk_imported_image_finish/waylib_vk_renderer_flush_stage APIs to wlroots Vulkan headers and implementation. Implement dmabuf import helper in wlroots Vulkan texture code so waylib can get a VkImage/VkFormat with COLOR_ATTACHMENT usage. Export new waylib_* symbols from wlroots.syms for external use.	3rdparty/wlroots/include/wlr/render/vulkan.h 3rdparty/wlroots/render/vulkan/texture.c 3rdparty/wlroots/render/vulkan/renderer.c 3rdparty/wlroots/wlroots.syms
Refactor WRenderHelper and WBufferRenderer to manage a RenderTarget lifecycle, integrate ColorContentsMode, and correctly handle Vulkan dmabuf render targets and layout transitions around Qt RHI render passes.	Add WRenderHelper::RenderTarget opaque handle class backed by shared BufferData with weak_ptr so buffers can be tracked safely after destruction. Track per-buffer colorPreserved flag and use ColorContentsMode to derive QRhiTextureRenderTarget flags and software clear behavior, including resolveColorContentsMode and rhiRenderTargetFlags helpers. Change acquireRenderTarget/lastRenderTarget to return RenderTarget instead of raw QQuickRenderTarget plus buffer, and use shared_ptr for buffer list/last buffer tracking. Extend BufferData with Vulkan-specific state (renderer, waylib_vk_imported_image) and import dmabuf via waylib_vk_renderer_import_dmabuf when using wlr_renderer_is_vk(), instead of sampling-only wlr_texture images. Add Vulkan layout transition helpers in WRenderHelper (prepareVulkanRenderTarget/finishVulkanRenderTarget) to move imported images between GENERAL and COLOR_ATTACHMENT_OPTIMAL around Qt RHI command buffer execution. Update WBufferRenderer state to store WRenderHelper::RenderTarget, pass ColorContentsMode into beginRender, and use RenderTarget to select clear vs preserve behavior for software and RHI paths, including calling Vulkan prepare/finish helpers before/after rendering.	waylib/src/server/qtquick/wrenderhelper.h waylib/src/server/qtquick/wrenderhelper.cpp waylib/src/server/qtquick/private/wbufferrenderer_p.h waylib/src/server/qtquick/private/wbufferrenderer.cpp
Propagate ColorContentsMode through outputs, viewports, and layers, replacing the old PreserveColorContents flag and wiring it into buffer rendering and compositing decisions.	Introduce WGlobal::ColorContentsMode enum (DontCare/Clear/Preserve) and expose it as a QML enum. Add colorContentsMode property to WOutputLayer and WOutputViewport, with getters/setters and change signals, and remove PreserveColorContents from WOutputLayer::Flags. Change OutputHelper::beginRender/render/compositeLayers and WOutputRenderWindowPrivate::doRenderOutputs to take/use ColorContentsMode instead of a preserveColorContents bool, including controlling when a shadow renderer is used based on bufferRenderer()->isColorPreserved(). Wire colorContentsMode changes to scheduling frames (connect WOutputLayer::colorContentsModeChanged to WOutputHelper::scheduleFrame) and respect visualizeLayers override by forcing Preserve when visualizing. Adjust the QML-facing WOutputViewport property from preserveColorContents(bool) to colorContentsMode(ColorContentsMode).	waylib/src/server/kernel/wglobal.h waylib/src/server/qtquick/woutputlayer.h waylib/src/server/qtquick/woutputlayer.cpp waylib/src/server/qtquick/woutputviewport.h waylib/src/server/qtquick/woutputviewport.cpp waylib/src/server/qtquick/private/woutputviewport_p.h waylib/src/server/qtquick/woutputrenderwindow.cpp waylib/src/server/qtquick/woutputhelper.h waylib/src/server/qtquick/woutputhelper.cpp
Ensure SHM uploads with Vulkan renderer are flushed on wlroots’ stage command buffer before Qt samples textures, and clean up QRhiTexture ownership in the QSG texture provider.	Modify WSGTextureProviderPrivate to stop tracking a separate rhiTexture pointer and instead use qtTexture.rhiTexture(), resetting ownership and scheduling cleanup jobs correctly when buffers/textures change. Introduce flushVulkanStageIfNeeded() in WSGTextureProviderPrivate to call waylib_vk_renderer_flush_stage on the associated wlr_renderer when Vulkan is in use, and invoke it before updating/sampling the RHI texture and on buffer content updates. Remove mipmap filtering configuration from WSGTextureProvider to simplify texture setup.	waylib/src/server/qtquick/wsgtextureprovider.cpp
Miscellaneous plumbing and copyright updates related to the new rendering behavior.	Update copyrights on several waylib qtquick/kernel files to UnionTech Software Technology Co., Ltd. Include wrenderhelper.h/woutputhelper.h where needed due to new RenderTarget and ColorContentsMode usage. Tighten BufferData lifetime management and lastRenderTarget handling using shared_ptr/weak_ptr. Minor adjustments to RHI texture creation helpers (createRhiRenderTarget) to pass QRhiTextureRenderTarget::Flags explicitly and to support re-creation when ColorContentsMode changes.	waylib/src/server/qtquick/wrenderhelper.cpp waylib/src/server/qtquick/wrenderhelper.h waylib/src/server/qtquick/private/wbufferrenderer_p.h waylib/src/server/qtquick/private/wbufferrenderer.cpp waylib/src/server/qtquick/woutputrenderwindow.cpp waylib/src/server/qtquick/woutputviewport.cpp waylib/src/server/qtquick/private/woutputviewport_p.h waylib/src/server/qtquick/woutputlayer.cpp waylib/src/server/qtquick/woutputlayer.h waylib/src/server/kernel/wglobal.h waylib/src/server/qtquick/woutputhelper.h waylib/src/server/qtquick/woutputhelper.cpp waylib/src/server/qtquick/wsgtextureprovider.cpp

Possibly linked issues

• #: PR fixes Vulkan dmabuf/SHM rendering paths in waylib, directly addressing treeland’s broken Vulkan backend described.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Sorry @zccrs, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[deepin-bot]

TAG Bot

New tag: 0.8.17
DISTRIBUTION: unstable
Suggest: synchronizing this PR through rebase #1229

[Copilot]

Pull request overview

EN: This PR updates waylib’s QtQuick rendering pipeline and the vendored wlroots Vulkan backend to correctly support Vulkan dmabuf render targets (COLOR_ATTACHMENT) and ensure SHM uploads are visible to Qt by flushing wlroots’ Vulkan stage command buffer. It also introduces a tri-state color contents preservation mode that flows from outputs/layers into render-target creation.

**ZH：**本 PR 更新了 waylib 的 QtQuick 渲染链路以及内置 wlroots 的 Vulkan 后端：通过以 COLOR_ATTACHMENT 方式导入 dmabuf 修复 Vulkan render target，并在 Qt 采样前 flush wlroots 的 Vulkan stage 命令缓冲以保证 SHM 上传可见。同时引入三态的颜色内容保留策略，并贯穿 output/layer 到 render target 的创建与生命周期管理。

Changes / 变更点：

• Add thin wlroots Vulkan wrapper APIs (waylib_vk_renderer_import_dmabuf, waylib_vk_imported_image_finish, waylib_vk_renderer_flush_stage) and export waylib_* symbols. / 在 wlroots Vulkan 中新增 waylib_ 薄封装 API 并导出符号。
• Introduce WGlobal::ColorContentsMode and WRenderHelper::RenderTarget to manage render-target lifecycle + preserve/clear semantics. / 引入 ColorContentsMode 与 RenderTarget 句柄化生命周期管理。
• Wire the new mode and Vulkan barriers/flush through QtQuick buffer/output render path. / 将新模式与 Vulkan 屏障/flush 串到 QtQuick 渲染链路。

Reviewed changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
waylib/src/server/qtquick/wsgtextureprovider.cpp	Flush wlroots Vulkan stage CB before Qt samples; adjust QRhiTexture cleanup ownership.
waylib/src/server/qtquick/wrenderhelper.h	Add WRenderHelper::RenderTarget handle type and Vulkan prepare/finish hooks.
waylib/src/server/qtquick/wrenderhelper.cpp	Implement render-target tracking, COLOR_ATTACHMENT dmabuf import for Vulkan, and layout transitions around Qt render passes.
waylib/src/server/qtquick/woutputviewport.h	Replace boolean preserve flag with WGlobal::ColorContentsMode property.
waylib/src/server/qtquick/woutputviewport.cpp	Implement colorContentsMode getter/setter + signal.
waylib/src/server/qtquick/woutputrenderwindow.cpp	Plumb ColorContentsMode through output/layer rendering; adjust shadow renderer decisions.
waylib/src/server/qtquick/woutputlayer.h	Add colorContentsMode property; remove PreserveColorContents flag.
waylib/src/server/qtquick/woutputlayer.cpp	Implement colorContentsMode state + signal.
waylib/src/server/qtquick/woutputhelper.h	Switch render-target API to return WRenderHelper::RenderTarget.
waylib/src/server/qtquick/woutputhelper.cpp	Return RenderTarget directly; update last-target API.
waylib/src/server/qtquick/private/wqmlhelper.cpp	Add QML helper isVulkanBackend based on WRenderHelper::getGraphicsApi().
waylib/src/server/qtquick/private/wqmlhelper_p.h	Expose isVulkanBackend as a QML singleton property.
waylib/src/server/qtquick/private/woutputviewport_p.h	Store colorContentsMode in viewport private data.
waylib/src/server/qtquick/private/wbufferrenderer.cpp	Pass mode into beginRender; use render-target handle; add Vulkan prepare/finish calls.
waylib/src/server/qtquick/private/wbufferrenderer_p.h	Update renderer state to store RenderTarget + mode; add isColorPreserved().
waylib/src/server/kernel/woutput.cpp	Filter AMD DCC modifiers when Vulkan renderer active to avoid white scanout.
waylib/src/server/kernel/wglobal.h	Add WGlobal::ColorContentsMode enum for QML/C++ use.
tests/test_dtk_inwindowblur_override/main.cpp	New QtTest validating DTK InWindowBlur QML override behavior.
tests/test_dtk_inwindowblur_override/CMakeLists.txt	Add build/test wiring and DConfig test environment setup.
tests/CMakeLists.txt	Register the new test subdirectory.
src/core/qmlengine.h	Store URL interceptor lifetime member.
src/core/qmlengine.cpp	Install QQmlAbstractUrlInterceptor to redirect DTK InWindowBlur QML to Treeland override.
src/core/qml/overridable/InWindowBlur.qml	New override component backed by Treeland Blur.
src/core/qml/Effects/Blur.qml	Add Vulkan fallback path (skip RenderBufferBlitter backdrop capture on Vulkan).
src/CMakeLists.txt	Add QRC resource for the DTK override QML.
3rdparty/wlroots/wlroots.syms	Export waylib_* symbols from the vendored wlroots build.
3rdparty/wlroots/render/vulkan/texture.c	Implement dmabuf import wrapper + imported-image finish helper.
3rdparty/wlroots/render/vulkan/renderer.c	Implement stage-CB flush helper for Vulkan renderer.
3rdparty/wlroots/include/wlr/render/vulkan.h	Declare new waylib Vulkan wrapper APIs and imported-image struct.

Suppressed comments (1)

waylib/src/server/kernel/woutput.cpp:268

• EN: The wlr_renderer_is_vk(renderer) check needs ENABLE_VULKAN_RENDER guarding too; otherwise non-Vulkan builds won’t have the symbol and this block won’t compile. Please wrap the Vulkan-only modifier filtering with #ifdef ENABLE_VULKAN_RENDER.
  ZH: wlr_renderer_is_vk(renderer) 同样需要用 ENABLE_VULKAN_RENDER 保护，否则在未启用 Vulkan 的构建下该符号不可用，导致这里编译失败。建议将 Vulkan 专用的 modifier 过滤逻辑放进 #ifdef ENABLE_VULKAN_RENDER。

    const struct wlr_drm_format *effective_render_format = render_format;
    if (wlr_renderer_is_vk(renderer)) {
        for (size_t i = 0; i < render_format->len; i++) {
            uint64_t mod = render_format->modifiers[i];
            if (IS_AMD_FMT_MOD(mod)
                && ((mod >> AMD_FMT_MOD_DCC_SHIFT) & AMD_FMT_MOD_DCC_MASK)) {
                continue;
            }
            if (!wlr_drm_format_add(&dcc_filtered, mod)) {
                wlr_drm_format_finish(&dcc_filtered);
                return false;
            }
        }
        if (dcc_filtered.len > 0)
            effective_render_format = &dcc_filtered;
    }

[Copilot]

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated 2 comments.

[zzxyb]

晚点合入，在wlroots 0.20.2升级后再合

[deepin-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: zccrs, zzxyb

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• debian/deepin/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment