feat(mcp): drive the client UI, gated out of release builds - #9437
Draft
eordano wants to merge 1 commit into
Draft
feat(mcp): drive the client UI, gated out of release builds#9437eordano wants to merge 1 commit into
eordano wants to merge 1 commit into
Conversation
Contributor
|
New build in progress, come back later! |
eordano
added a commit
that referenced
this pull request
Jul 27, 2026
Rebases PR #9437's parity work onto dev and finishes it: the embedded MCP server can now drive the explorer-automation suite end to end, with no instrumented AltTester build. Rebase. #9339 was squash-merged into dev after this work branched, so the four tools that came over (get_network_log, list_ui_elements, get_ui_state, click_ui) are ported from the pre-review contract to dev's: the abstract McpTool base, DescribeInput(McpJsonSchema) instead of an InputSchema property, and the McpPointerEvent* naming. Nothing of dev's post-review version is reverted. Finding elements. The first pass could only enumerate uGUI Selectables, so about half of what the suite locates was invisible: its 118 By.PATH and 61 By.NAME locators are largely plain GameObjects, and presence/absence waits over them are the bulk of a UI suite. UiAutomation now walks the full active Transform tree under every root Canvas (plus the UI-Toolkit trees), iteratively, with a node cap and the nameFilter fast path. UiElementPath gained AltTester's By.PATH dialect — // descendant, / child, * wildcard, the Name[i] sibling indexer — matched against the whole candidate path so a query identifies the element and not an ancestor. An unmatched get_ui_state is an error result, which is how absence is expressed. Acting on them. click_ui and scroll hit-test through the live EventSystem at the element's screen position and dispatch at the real top hit, so a control covered by a modal returns clicked:false with the blocker instead of the false success a direct ExecuteEvents call produced; only an element with no raycastable graphic falls back to direct dispatch, and says so. set_ui_text covers uGUI InputField, TMP_InputField and UI-Toolkit TextField (the 9 SetText sites gate the whole auth path). press_key queues keyboard state onto the Input System device the client reads — the only seam reaching every DCLInput action, with the real action-map gating intact. Text reads now cover TextMeshPro, which is what the Explorer's labels actually are. Not in release builds. The MCP server itself keeps shipping everywhere — Creator Tools drives the installed client through it, so the transport and the scene-iteration tools cannot be compiled out. The surface added here to replace AltTester can be, and is: the six client-UI automation tools, the three reflection tools, the four Utils types behind them and their EditMode tests sit behind a new MCP_TEST_AUTOMATION define, on by default and removed for release by CloudBuild.PreExport. tools/list therefore differs by build, which the doc states. This matters now rather than in the abstract: sdk-commands only ever sets mcp into the decentraland:// link and DeepLinkAllowlist drops it, so the creator flow is broken today, and the natural fix is a loopback-gated allowlist entry for mcp. At that point a deep link can start the server on a shipped client and a command-line-only runtime flag stops being sufficient on its own. The compile gate is a property of the release binary that no allowlist edit can undo. That allowlist change is deliberately not made here. Reflection takes a second gate inside the builds that keep it: get/set_component_property and call_static_method are arbitrary in-process reads, writes and invocation, so --mcp-reflection (FeatureId.McpReflection) must also be passed, and they are absent from tools/list otherwise rather than refused when called. The suite passes both flags. docs/mcp-automation.md documents the tools, the path dialect, the AltTester coverage table and that reasoning. It also corrects a standing error: the doc claimed --mcp is accepted from a deep link, but the allowlist drops it, so the documented `npm run start -- --mcp` flow is silently a no-op — the fix belongs in sdk-commands, not in the allowlist. EditMode tests cover the pure parts (path building/matching, key-name parsing, the reflected property reads/writes, the network-log ring buffer). The hierarchy walk, hit-test and event dispatch are integration-only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
eordano
force-pushed
the
feat/mcp-cdpu-parity
branch
from
July 27, 2026 22:48
194b843 to
b5b09a1
Compare
Contributor
|
Lint in progress, come back later! |
eordano
changed the title
eordano
changed the title
Contributor
|
Tests: 24507 passed, 0 failed ✅ |
eordano
added a commit
that referenced
this pull request
Jul 28, 2026
Rebases PR #9437's parity work onto dev and finishes it: the embedded MCP server can now drive the client's own UI end to end, so a UI test suite needs no instrumented build. Rebase. #9339 was squash-merged into dev after this work branched, so the four tools that came over (get_network_log, list_ui_elements, get_ui_state, click_ui) are ported from the pre-review contract to dev's: the abstract McpTool base, DescribeInput(McpJsonSchema) instead of an InputSchema property, and the McpPointerEvent* naming. Nothing of dev's post-review version is reverted. Finding elements. The first pass could only enumerate uGUI Selectables, so about half of what a UI suite locates was invisible: real locators are largely plain GameObjects, and presence/absence waits over them are the bulk of such a suite. UiAutomation now walks the full active Transform tree under every root Canvas (plus the UI-Toolkit trees), iteratively, with a node cap and the nameFilter fast path. UiElementPath gained a path-expression dialect — // descendant, / child, * wildcard, the Name[i] sibling indexer — matched against the whole candidate path so a query identifies the element and not an ancestor. An unmatched get_ui_state is an error result, which is how absence is expressed. Acting on them. click_ui and scroll hit-test through the live EventSystem at the element's screen position and dispatch at the real top hit, so a control covered by a modal returns clicked:false with the blocker instead of the false success a direct ExecuteEvents call produced; only an element with no raycastable graphic falls back to direct dispatch, and says so. set_ui_text covers uGUI InputField, TMP_InputField and UI-Toolkit TextField, which is what the login email, OTP code, username and search fields need. press_key queues keyboard state onto the Input System device the client reads — the only seam reaching every DCLInput action, with the real action-map gating intact. Text reads now cover TextMeshPro, which is what the Explorer's labels actually are. Not in release builds. The MCP server itself keeps shipping everywhere — Creator Tools drives the installed client through it, so the transport and the scene-iteration tools cannot be compiled out. The surface added here can be, and is: the six client-UI automation tools, the three reflection tools, the four Utils types behind them and their EditMode tests sit behind a new MCP_TEST_AUTOMATION define, on by default and removed for release by CloudBuild.PreExport. tools/list therefore differs by build, which the doc states. This matters now rather than in the abstract: sdk-commands only ever sets mcp into the decentraland:// link and DeepLinkAllowlist drops it, so the creator flow is broken today, and the natural fix is a loopback-gated allowlist entry for mcp. At that point a deep link can start the server on a shipped client and a command-line-only runtime flag stops being sufficient on its own. The compile gate is a property of the release binary that no allowlist edit can undo. That allowlist change is deliberately not made here. Reflection takes a second gate inside the builds that keep it: get/set_component_property and call_static_method are arbitrary in-process reads, writes and invocation, so --mcp-reflection (FeatureId.McpReflection) must also be passed, and they are absent from tools/list otherwise rather than refused when called. McpServerGatingShould guards both gates: it asserts the tool count per build (17 shipped, +9 behind the define) so a tool cannot be added without deciding which side it belongs on, and drives the real FeaturesRegistry from synthetic app args to pin the flag matrix — including that --mcp-reflection is inert without --mcp. It sits outside the define, because asserting what the gate removes is only meaningful from outside it. docs/mcp-automation.md documents the tools, the path-expression syntax, what a UI suite can do with the surface, and the security reasoning. It also corrects a standing error: the doc claimed --mcp is accepted from a deep link, but the allowlist drops it, so the documented `npm run start -- --mcp` flow is silently a no-op — the fix belongs in sdk-commands, not in the allowlist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
eordano
force-pushed
the
feat/mcp-cdpu-parity
branch
from
July 28, 2026 00:03
b5b09a1 to
b2924a2
Compare
eordano
added a commit
that referenced
this pull request
Jul 28, 2026
Rebases PR #9437's parity work onto dev and finishes it: the embedded MCP server can now drive the client's own UI end to end, so a UI test suite needs no instrumented build. Rebase. #9339 was squash-merged into dev after this work branched, so the four tools that came over (get_network_log, list_ui_elements, get_ui_state, click_ui) are ported from the pre-review contract to dev's: the abstract McpTool base, DescribeInput(McpJsonSchema) instead of an InputSchema property, and the McpPointerEvent* naming. Nothing of dev's post-review version is reverted. Finding elements. The first pass could only enumerate uGUI Selectables, so about half of what a UI suite locates was invisible: real locators are largely plain GameObjects, and presence/absence waits over them are the bulk of such a suite. UiAutomation now walks the full active Transform tree under every root Canvas (plus the UI-Toolkit trees), iteratively, with a node cap and the nameFilter fast path. UiElementPath gained a path-expression dialect — // descendant, / child, * wildcard, the Name[i] sibling indexer — matched against the whole candidate path so a query identifies the element and not an ancestor. An unmatched get_ui_state is an error result, which is how absence is expressed. Acting on them. click_ui and scroll hit-test through the live EventSystem at the element's screen position and dispatch at the real top hit, so a control covered by a modal returns clicked:false with the blocker instead of the false success a direct ExecuteEvents call produced; only an element with no raycastable graphic falls back to direct dispatch, and says so. set_ui_text covers uGUI InputField, TMP_InputField and UI-Toolkit TextField, which is what the login email, OTP code, username and search fields need. press_key queues keyboard state onto the Input System device the client reads — the only seam reaching every DCLInput action, with the real action-map gating intact. Text reads now cover TextMeshPro, which is what the Explorer's labels actually are. Not in release builds. The MCP server itself keeps shipping everywhere — Creator Tools drives the installed client through it, so the transport and the scene-iteration tools cannot be compiled out. The surface added here can be, and is: the six client-UI automation tools, the three reflection tools, the four Utils types behind them and their EditMode tests sit behind a new MCP_TEST_AUTOMATION define, on by default and removed for release by CloudBuild.PreExport. tools/list therefore differs by build, which the doc states. This matters now rather than in the abstract: sdk-commands only ever sets mcp into the decentraland:// link and DeepLinkAllowlist drops it, so the creator flow is broken today, and the natural fix is a loopback-gated allowlist entry for mcp. At that point a deep link can start the server on a shipped client and a command-line-only runtime flag stops being sufficient on its own. The compile gate is a property of the release binary that no allowlist edit can undo. That allowlist change is deliberately not made here. Reflection takes a second gate inside the builds that keep it: get/set_component_property and call_static_method are arbitrary in-process reads, writes and invocation, so --mcp-reflection (FeatureId.McpReflection) must also be passed, and they are absent from tools/list otherwise rather than refused when called. McpServerGatingShould guards both gates: it asserts the tool count per build (17 shipped, +9 behind the define) so a tool cannot be added without deciding which side it belongs on, and drives the real FeaturesRegistry from synthetic app args to pin the flag matrix — including that --mcp-reflection is inert without --mcp. It sits outside the define, because asserting what the gate removes is only meaningful from outside it. docs/mcp-automation.md documents the tools, the path-expression syntax, what a UI suite can do with the surface, and the security reasoning. It also corrects a standing error: the doc claimed --mcp is accepted from a deep link, but the allowlist drops it, so the documented `npm run start -- --mcp` flow is silently a no-op — the fix belongs in sdk-commands, not in the allowlist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
eordano
force-pushed
the
feat/mcp-cdpu-parity
branch
from
July 28, 2026 01:11
b2924a2 to
300c108
Compare
Rebases PR #9437's parity work onto dev and finishes it: the embedded MCP server can now drive the client's own UI end to end, so a UI test suite needs no instrumented build. Rebase. #9339 was squash-merged into dev after this work branched, so the four tools that came over (get_network_log, list_ui_elements, get_ui_state, click_ui) are ported from the pre-review contract to dev's: the abstract McpTool base, DescribeInput(McpJsonSchema) instead of an InputSchema property, and the McpPointerEvent* naming. Nothing of dev's post-review version is reverted. Finding elements. The first pass could only enumerate uGUI Selectables, so about half of what a UI suite locates was invisible: real locators are largely plain GameObjects, and presence/absence waits over them are the bulk of such a suite. UiAutomation now walks the full active Transform tree under every root Canvas (plus the UI-Toolkit trees), iteratively, with a node cap and the nameFilter fast path. UiElementPath gained a path-expression dialect — // descendant, / child, * wildcard, the Name[i] sibling indexer — matched against the whole candidate path so a query identifies the element and not an ancestor. An unmatched get_ui_state is an error result, which is how absence is expressed. Acting on them. click_ui and scroll hit-test through the live EventSystem at the element's screen position and dispatch at the real top hit, so a control covered by a modal returns clicked:false with the blocker instead of the false success a direct ExecuteEvents call produced; only an element with no raycastable graphic falls back to direct dispatch, and says so. set_ui_text covers uGUI InputField, TMP_InputField and UI-Toolkit TextField, which is what the login email, OTP code, username and search fields need. press_key queues keyboard state onto the Input System device the client reads — the only seam reaching every DCLInput action, with the real action-map gating intact. Text reads now cover TextMeshPro, which is what the Explorer's labels actually are. Not in release builds. The MCP server itself keeps shipping everywhere — Creator Tools drives the installed client through it, so the transport and the scene-iteration tools cannot be compiled out. The surface added here can be, and is: the seven client-UI automation tools, the three reflection tools, the four Utils types behind them and their EditMode tests sit behind a new MCP_TEST_AUTOMATION define, on by default and removed for release by CloudBuild.PreExport. tools/list therefore differs by build, which the doc states. This matters now rather than in the abstract: sdk-commands only ever sets mcp into the decentraland:// link and DeepLinkAllowlist drops it, so the creator flow is broken today, and the natural fix is a loopback-gated allowlist entry for mcp. At that point a deep link can start the server on a shipped client and a command-line-only runtime flag stops being sufficient on its own. The compile gate is a property of the release binary that no allowlist edit can undo. That allowlist change is deliberately not made here. Reflection takes a second gate inside the builds that keep it: get/set_component_property and call_static_method are arbitrary in-process reads, writes and invocation, so --mcp-reflection (FeatureId.McpReflection) must also be passed, and they are absent from tools/list otherwise rather than refused when called. McpServerGatingShould guards both gates: it asserts the tool count per build (17 shipped, +10 behind the define) so a tool cannot be added without deciding which side it belongs on, and drives the real FeaturesRegistry from synthetic app args to pin the flag matrix — including that --mcp-reflection is inert without --mcp. It sits outside the define, because asserting what the gate removes is only meaningful from outside it. docs/mcp-automation.md documents the tools, the path-expression syntax, what a UI suite can do with the surface, and the security reasoning. It also corrects a standing error: the doc claimed --mcp is accepted from a deep link, but the allowlist drops it, so the documented `npm run start -- --mcp` flow is silently a no-op — the fix belongs in sdk-commands, not in the allowlist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
eordano
force-pushed
the
feat/mcp-cdpu-parity
branch
from
July 28, 2026 03:42
300c108 to
91c6618
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bring capabilities to debug the client, not just scenes, to the embedded MCP server.
Network (get_network_log):
Client UI automation (list_ui_elements / get_ui_state / click_ui):
Tests (folded into DCL.EditMode.Tests): McpNetworkLogBufferShould, UiElementPathShould. UI/engine walk + click are integration-only.
Docs: new sections in docs/mcp-automation.md (tool catalog + coverage + implementation map).