Skip to content

feat(screenshot): 新增原位截图翻译与自动重译 - #1292

Open
powerpan wants to merge 3 commits into
tisfeng:devfrom
powerpan:feat/in-place-screenshot-overlay
Open

powerpan wants to merge 3 commits into
tisfeng:devfrom
powerpan:feat/in-place-screenshot-overlay

Conversation

@powerpan

@powerpan powerpan commented Aug 30, 2026 •

Copy link
Copy Markdown

变更说明 / Summary

  • 新增“原位截图翻译”:框选固定区域后,以常驻浮窗按 OCR 布局覆写译文,并支持原文/译文切换、语言与服务选择、置顶、暂停、刷新、复制和重新框选。
  • 使用 ScreenCaptureKit 监控选区变化,并结合视觉去抖、语义差分、块级缓存和 generation guard,只对稳定变化的内容自动重新 OCR 与翻译。
  • 截图像素只在本地内存中处理,仅向用户选择的翻译服务发送 OCR 文本;本分支从最新 dev 独立适配,不包含其他未合并 PR 的实现。

关联 Issue / Linked Issues

无。 / None.

验证 / Verification

  • git diff --check
  • SwiftFormat:0/410 files require formatting
  • String Catalog JSON 与 47 个功能 key 的六语言覆盖校验
  • 两个 Info plist 与 Easydict.xcodeproj/project.pbxproj 的 plutil -lint
  • Debug build-for-testing
  • 15 个 focused suites,共 94 项测试通过、0 失败、0 跳过
  • Release build 并生成 Easydict.app
  • origin/dev..HEAD 两提交范围敏感信息扫描全部通过

截图 / Screenshots

请在 GitHub PR 页面补充截图。 / Please add screenshots on the GitHub PR page.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1603036398

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

// Call the original completion handler
captureCompletionHandler?(image)
captureCompletionHandler = nil
selectionCaptureCompletionHandler?(selectionResult)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Tear down overlays before invoking failure callbacks

When a selection is too small, this callback synchronously enters InPlaceTranslationWindowManager.presentSelectionFailureIfNeeded, which calls NSAlert.runModal() before execution can reach hideAllOverlayWindows(). The full-screen selector windows remain at .screenSaver level above the alert, so the UI can appear stuck with an inaccessible modal; tear down the overlays and event monitor before invoking the completion, or defer presentation until cleanup finishes.

Useful? React with 👍 / 👎.

/// optional plaintext debug log. The setting belongs to this fresh service instance only.
func configureForInPlaceRequest(_ service: QueryService) {
service.windowType = .fixed
service.allowsPlaintextRequestLogging = false

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Sanitize DeepSeek payload logs for in-place requests

When DeepSeek is selected and its stream produces a malformed SSE event, this flag has no effect because DeepSeekService.parseSSEEvent never checks it and logs the complete dataString at DeepSeekService.swift:228. That payload can contain generated translation text from the captured region, defeating this path's no-plaintext-log protection; make every eligible provider honor the flag or sanitize DeepSeek decode failures to structural metadata such as byte counts.

Useful? React with 👍 / 👎.

HStack(spacing: 6) {
liveToggle.labelsHidden()
narrowSourceMenu
Image(systemName: "arrow.right")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Use SFSafeSymbols for toolbar icons

This new SwiftUI toolbar hard-codes SF Symbol names through Image(systemName:) here and in the adjacent display and overflow buttons, bypassing the repository-required type-safe symbol API and its compile-time spelling and availability checks. Import SFSafeSymbols and use Image(systemSymbol:) with the corresponding symbol cases.

AGENTS.md reference: AGENTS.md:L23-L25

Useful? React with 👍 / 👎.

@powerpan
powerpan force-pushed the feat/in-place-screenshot-overlay branch from 1603036 to 24a2914 Compare August 31, 2026 15:44
Eric Pan added 3 commits August 31, 2026 23:45
现有截图翻译会把内容带入通用查询窗口,无法在固定区域持续保留原布局。阅读动态字幕、网页或不可复制内容时,用户需要一个常驻且明确控制隐私与费用的查看方式。

新增固定区域 ScreenCaptureKit 采集、本地布局 OCR、块级缓存和自动重译协调,并提供可缩放的原位翻译面板、语言与服务控制、暂停、刷新、复制和重新框选。请求边界会合并重复块、串行配额记账并拒绝旧 generation 回写。

此变更保留既有截图入口,在六种语言下提供完整设置和首次隐私说明。相关 130 项测试、Debug build-for-testing、Release 构建及独立复审均已通过,真实多屏、TCC、provider 和性能矩阵保留为发布前门禁。

----------------------------------------------------------------------

feat(screenshot): add in-place screenshot translation and live updates

Existing screenshot translation moves content into a general query window and cannot preserve layout over a fixed region. Reading changing subtitles, web pages, or non-copyable content needs a persistent view with explicit privacy and cost controls.

Add fixed-region ScreenCaptureKit capture, local layout OCR, block-level caching, and automatic retranslation together with a resizable in-place panel for language, provider, pause, refresh, copy, and reselection controls. The request boundary coalesces duplicate blocks, serializes quota accounting, and rejects stale generation updates.

This keeps existing screenshot entry points intact while adding complete settings and first-use disclosure in six locales. The 130 focused tests, Debug build-for-testing, Release build, and independent review pass, with real multi-display, TCC, provider, and performance matrices retained as release gates.
原位截图翻译提交已从最新 origin/dev 基线独立适配,需要以当前分支的实际验证取代原实现分支中的旧证据。

更新完成计划和历史,记录 94 项 focused tests、Debug build-for-testing、Release build 以及格式和工程校验结果。

这使 PR 的验证声明与当前上游基线一致,并明确配置备份 registry 仍由其他未合并变更负责。

----------------------------------------------------------------------

docs(screenshot): record upstream adaptation verification

The in-place screenshot translation change was adapted independently onto the latest origin/dev baseline, so its evidence must reflect the current branch instead of the original implementation branch.

Update the completed plan and history with the 94 focused tests, Debug build-for-testing, Release build, formatting, and project validation results.

This keeps the pull request claims aligned with the current upstream baseline and makes clear that configuration backup registry integration remains in separate unmerged work.
原位截圖翻譯錯誤沿用固定視窗結果卡片的展開狀態,導致已啟用的有道和內置 AI 等服務從選單消失。

讓服務資格只取決於固定視窗啟用狀態、翻譯能力和使用狀態,並補充服務來源提示、空狀態與對應回歸測試。

此修復讓折疊結果卡片不再改變原位翻譯服務選擇,同時保留服務停用、刪除和暫時錯誤的既有回退邊界。

----------------------------------------------------------------------

fix(screenshot): decouple in-place services from result cards

In-place screenshot translation reused the Fixed Window result-card expansion state, causing enabled providers such as Youdao and Built-in AI to disappear from its menus.

Base eligibility on Fixed Window enablement, translation capability, and usage status, and add provider-source guidance, an empty label, and regression coverage.

Collapsed result cards no longer change in-place provider selection, while existing fallback boundaries for disabled, removed, and temporarily failing services remain intact.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 24a2914573

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +250 to +253
await previousSession?.stop()
previousPanel?.contentView = nil
previousPanel?.orderOut(nil)
await session.start()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Start the session before exposing its controls

After reselection, the new panel is made interactive before this task finishes stopping the previous session. During that delay, changing live updates, languages, or the provider updates the ViewModel immediately, but the corresponding session command is silently rejected because InPlaceTranslationSession is still in .starting; session.start() then uses its original configuration, leaving the displayed controls out of sync with capture and translation behavior. Start the new session before presenting the panel, disable these controls until startup completes, or preserve mutations received while starting.

Useful? React with 👍 / 👎.

Comment on lines +193 to +194
if viewModel.lifecycle == .paused {
Text("in_place_screenshot_translation.status.paused")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Show capture failures before the paused label

When capture permission is revoked or the selected display disappears, handleCaptureFailure sets both the lifecycle to .paused and the availability to .permissionDenied or .displayDisconnected. This outer check therefore always renders only “Paused,” making the dedicated failure messages below unreachable in those production paths; display disconnection in particular has no other visible diagnosis. Prioritize capture availability over the generic paused state.

Useful? React with 👍 / 👎.

Comment on lines +212 to +213
case .debouncing:
Text("in_place_screenshot_translation.status.recovering_capture")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Don't report routine debouncing as capture recovery

Every ordinary visual change enters .debouncing while waiting for content to stabilize, so this mapping briefly displays “Recovering capture” during normal automatic retranslations even though capture has not failed. Use a distinct updating/debouncing message or retain the preceding status until recognition begins.

Useful? React with 👍 / 👎.

@powerpan
powerpan force-pushed the feat/in-place-screenshot-overlay branch from 24a2914 to 01d3737 Compare August 31, 2026 15:56

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 01d3737c37

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +163 to +169
if shouldResume {
lifecycle = .running
await startFrameSource()
guard lifecycle == .running, configuration.liveUpdatesEnabled else { return }
if let latestFrame {
scheduleProcessing(latestFrame, force: true)
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Resume invalidated work for paused sessions

When Auto Update is disabled, the initial OCR still runs while the lifecycle is .paused. If the user starts reselection and then cancels before that OCR finishes, suspendForReselection() cancels the pipeline and returns false, so this branch neither restarts capture nor reschedules the retained latestFrame; the restored panel remains indefinitely idle on its placeholder or partially processed snapshot. Reschedule invalidated processing independently of whether live capture should resume.

Useful? React with 👍 / 👎.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant