Skip to content

feat(diagnostics): measure first real app action - #364

Merged
NWYLZW merged 5 commits into
oneworks-ai:mainfrom
Nirvana-Jie:codex/issue-327-first-action-diagnostics
Aug 22, 2026
Merged

feat(diagnostics): measure first real app action#364
NWYLZW merged 5 commits into
oneworks-ai:mainfrom
Nirvana-Jie:codex/issue-327-first-action-diagnostics

Conversation

@Nirvana-Jie

@Nirvana-Jie Nirvana-Jie commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Related to [Performance][RFC] Desktop 启动分层:UI First、Core Ready 与后台能力增量就绪 #327.
  • Add a privacy-safe oneworks.app.first_action operation for the first real session action: submit, transport acceptance, first renderable assistant response, and terminal success/failure/termination.
  • Centralize existing-session sends, optimistic creation/retry, Interaction Panel annotations, and queued sends around one anonymous client-action-* correlation ID that the server reuses when the action actually runs.
  • Make correlation causal across history, app-level SSE, active-session WebSocket, reconnects, subscriber gaps, multiple panels, queue deletion, HTTP-ack loss, and app crash recovery.
  • Expose first-action P50/P95 distributions and a terminal-only success rate through the Relay diagnostics API.
  • Do not add a placeholder editor, queued draft, fake interactive state, or any other user-visible startup UI.

Why

Startup diagnostics previously ended at UI readiness, so they could not answer whether the user's first meaningful action was accepted and actually completed. This change adds that missing measurement boundary without putting diagnostics I/O on the application bootstrap critical path: the first-action operation and journal record are created lazily only after a real submit, and renderer reporting remains fire-and-forget.

Architecture and data truthfulness

  • The renderer locks the first submitted session and exact anonymous action ID. It never sends session IDs, action IDs, prompts, responses, paths, or error text to Desktop diagnostics.
  • A live transport must observe that exact user action before its later response or terminal state can advance the operation. A reconnect or a last-subscriber observation gap starts a new causal generation; reusing an open socket while observation remains continuous, or reconnecting an unrelated panel session, does not reset the tracked generation.
  • History can restore the same exact action, but cannot use an older completed snapshot or a later turn. The first later user message permanently supersedes the first-action observation.
  • Renderable-response detection shares the same predicate as the real message renderer, so hidden/standalone tool results and empty legacy content do not become fake response milestones.
  • Server status writes terminated before adapter shutdown and preserves failed/terminated if a later stop callback arrives.
  • Interrupted operations recovered on the next launch are exported as terminal facts, preventing crashes from disappearing from the success-rate denominator.
  • Relay success rate uses only terminal attempts; genuinely in-flight attempts remain visible as pending instead of being counted as failures.

This PR's measurable change

This PR establishes a measurement surface; it does not claim a startup-latency improvement. No same-artifact runtime A/B cohort was collected, so no latency percentage is reported.

Measurement capability Before After this PR
Privacy-safe first-action milestones None 6 closed milestones: submit, accepted, renderable response, success, failure, termination
Real submit entry classes correlated None 4: existing session, optimistic create/retry, annotation, queued send
First-action latency distributions None 4 P50/P95 series: app-start→submit and submit→accepted/response/success
Outcome accounting Unavailable Terminal success/failure/termination denominator with pending attempts reported separately
First-action journal writes before a real submit 0 0; the operation remains lazy and adds no bootstrap write
User-visible UI Existing real chat UI Unchanged

The next runtime cohort can now provide actual first-action latency and success-rate deltas. Provider-side first_token is deliberately not inferred from a complete renderer message and remains separate follow-up work.

Changelog

  • changelog/1.0.0-rc.4/readme.md

Screenshots

No visual behavior changes. This is the existing real chat surface whose message-renderability contract is reused for first.response.received; no placeholder or simulated composer was added.

Existing real One Works chat surface; visually unchanged by this PR

Validation

  • Targeted Vitest: 19 files / 207 tests passed.
  • pnpm typecheck.
  • pnpm exec eslint ..
  • pnpm exec dprint check.
  • git diff --check.
  • Client production Vite build: 8,488 modules transformed, passed with existing large-chunk warnings only.
  • Server runtime bundle, Desktop Electron build, and Relay server build passed during cross-module validation.
  • Independent local-correctness and global-architecture reviews passed with no P0–P3 findings.

Review Scope

  • Risk: medium
  • Applicable profiles: core review rules; Desktop/runtime readiness and diagnostics privacy boundaries
  • Evidence: causal state-machine tests, submit-path integration tests, crash-recovery/Relay aggregation tests, typecheck, lint, formatting, builds, and two independent reviews
  • Rule exceptions: none

Scope and follow-up

  • first.response.received means the renderer observed content that the real message UI can display. It is deliberately not provider-side first_token.
  • Relay workspace sessions can still be censored if the user leaves the active session route before completion. Missing samples must not be interpreted as success; an app-level Relay subscription remains follow-up work.
  • The Relay API exposes the summary in this PR; a Relay Admin category filter and visual scorecard remain separate follow-up work.
  • Configured external Channel/MCP, Windows/x64, large-history/plugin, and official signed-release cohorts remain open in [Performance][RFC] Desktop 启动分层:UI First、Core Ready 与后台能力增量就绪 #327.

Experience Review

  • 已判断是否需要沉淀经验
  • 如需要,已运行 $post-task-experience-review
  • reviewer PASS / NOT APPLICABLE 后才进入 merge

@vercel

vercel Bot commented Aug 13, 2026

Copy link
Copy Markdown

Someone is attempting to deploy a commit to the oneworks Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Experience Review

Before merge, make sure the PR body contains this completed checklist:

## Experience Review
- [x] 已判断是否需要沉淀经验
- [x] 如需要,已运行 `$post-task-experience-review`
- [x] reviewer `PASS` / `NOT APPLICABLE` 后才进入 merge

Checklist to confirm:

  • Decide whether this PR produced reusable project experience.
  • If experience should be captured, run $post-task-experience-review.
  • Merge only after the independent reviewer reports PASS or NOT APPLICABLE for the Experience Review Result.
  • For workflow, permission, or release-rule documentation changes, also add this checked section to the PR body after an independent read-only conflict review:
## Policy Conflict Review
- [x] Independent read-only reviewer checked workflow, permission, and release-rule conflicts and reported PASS

@Nirvana-Jie
Nirvana-Jie force-pushed the codex/issue-327-first-action-diagnostics branch from eff5226 to 86e9823 Compare August 13, 2026 11:51
@Nirvana-Jie
Nirvana-Jie marked this pull request as ready for review August 13, 2026 11:53
@Nirvana-Jie
Nirvana-Jie requested review from a team as code owners August 13, 2026 11:53

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 86e982387a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread apps/server/src/services/session/index.ts
@Nirvana-Jie
Nirvana-Jie force-pushed the codex/issue-327-first-action-diagnostics branch from d19a555 to 14c225c Compare August 14, 2026 06:33
@NWYLZW

NWYLZW commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

复审 latest head 14c225ca,当前结论:Needs changes

  1. [P1][REVIEW-001] 明确的提交失败不会终结首动作

    createDesktopFirstActionSubmitCoordinator.submit() 在 transport 前启动首动作,但 transport 抛错时既不标记 first.failed,也不标记 first.terminated。对于明确 HTTP 拒绝、创建失败后用户放弃等场景,action 不会进入服务端 history,后续状态观察也无法收口;该 attempt 会长期停留在 pending,并锁住后续首动作,导致 terminal-only success rate 被高估。

    建议区分明确拒绝、用户放弃与 ACK 不确定:明确拒绝记 first.failed,用户放弃记 first.terminated;网络断开等 ACK 不确定场景继续通过服务端因果事件协调,并提供有界 abandonment。请补对应回归测试。

    位置:apps/client/src/diagnostics/desktop-first-action-submit.ts:21-29

  2. [P2][REVIEW-001] Relay 日序列会拆分并重复统计跨日 operation

    diagnosticSeries() 先按 raw event 的 occurredAt 日期分组,再在每天内部聚合 operation。同一首动作如果午夜前提交、午夜后完成,或崩溃后隔天恢复,会在两天各算一次 attempt:第一天 pending、第二天 terminal,日成功率也不再对应提交 cohort。

    建议先按 operationId 聚合完整 operation,再按明确的 cohort 日期分桶,并补跨 UTC 日期与隔日 crash recovery 测试。

    位置:apps/relay-server/src/routes/diagnostics.ts:152-170

另外,当前 PR 与 main 存在冲突(CONFLICTING / DIRTY),修复后仍需更新分支并重新复审。项目内 CI 已通过;Vercel 失败信息是部署授权缺失。

@Nirvana-Jie
Nirvana-Jie force-pushed the codex/issue-327-first-action-diagnostics branch from 14c225c to 957f991 Compare August 20, 2026 16:58
@Nirvana-Jie

Nirvana-Jie commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

Addressed review comment issuecomment-5355678414 and rebased PR #364 onto main@6706bab3c20f3dbcb7cb4f52c3c1922b092d8e1c.

  • The first-action outcome is now explicit and privacy-safe: definite 4xx rejection records first.failed; user termination records first.terminated; 408/5xx and network/ACK uncertainty remain unresolved until causal observation. A main-owned 30-second abandonment timer settles genuinely unobserved attempts, while a retry plus causal observation clears the timer. No prompt, content, session identifier, or action identifier is transmitted.
  • Daily first-action series aggregate each operation first, then cohort by the real UTC first.submit timestamp. Terminal-only accounting and from filtering do not manufacture attempts or place an operation into a cohort without a real submit.

This rebase includes the current MR change set. The supplied validation evidence is 23 test files / 353 Vitest tests plus typecheck, full ESLint, dprint, client prepublish, relay build, and git diff --check passing. This comment intentionally reports only this MR's changes and validation; it does not include performance results from other MRs or claim an unmeasured performance ratio.

@NWYLZW
NWYLZW added this pull request to the merge queue Aug 22, 2026
Merged via the queue into oneworks-ai:main with commit 17bf3c8 Aug 22, 2026
26 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants