Skip to content

The guard for "no drafts in search" reads a field its own search never returns — and drafts are returned #2270

Description

@Ikalus1988

一句话

tests/test_mcp_server.py::test_no_drafts_in_search 声称检查"搜索结果里没有 draft 课程"。它读的是 r.get("status"),而 status 在默认档(compact)与 summary 档里根本不存在——只有 detail=full 才带。所以这个断言无论如何都是 0,永远无法失败。而 draft 课程确实出现在搜索结果里。

实测(2026-09-25,本机,真实 worker + 真实语料)

query: "search quota exhausted false zero results"
  detail=compact   status key: ABSENT   keys: id,title,problem,freshness,evidence_level,kind
  detail=summary   status key: ABSENT   keys: id,title,problem,freshness,evidence_level,domain,tags,fix,kind
  detail=full      status key: draft    keys: id,title,domain,tags,path,status,description,evidence_level,score,kind

语料里那篇课的状态:status = draft(命中结果就是它:search-quota-exhaustion-false-zero)

被断言的代码(tests/test_mcp_server.py):

resp = rpc("tools/call", {"name": "misakanet_search",
                          "arguments": {"query": "draft test lesson", "top": 10}})   # 默认 detail=compact
results = result.get("results", [])
draft_count = sum(1 for r in results if r.get("status") == "draft")
check("no drafts in results", draft_count == 0, ...)

r.get("status") 在 compact 档恒为 None → draft_count 恒为 0 → 这条 check 从来没有能力失败。它旁边的 check() 只是把布尔记进 PASS/FAIL 计数器,所以从 CI 的角度看它一直"通过"。

两个独立的问题,别混为一谈

(A) 一个不能失败的门禁。 这是本仓反复出现的那一类(#1822 的 --kv-only 提前 return、#2183 吞掉的退出码、#2525 之前的 identity aura 死读)。修法:断言真实属性——要么把结果里的 status 取出来(在 detail=full 下查),要么直接断言"结果里不存在 draft 课程 id"(不依赖投影暴露了什么字段)。

(B) draft 到底该不该出现在检索里——这是产品决定,不是 bug。 现状是:

  • 24 篇 status=draft 的课程在索引里,因此可被检索到(上面就是证据);
  • 它们只在 detail=full 才带 status,所以吃 compact 的 agent 无法知道自己拿到的是 draft;
  • 而这 24 篇的证据等级:23 篇 E0、1 篇 E1;其中 17 篇的 ## Verification 是 grep/echo/wc 占位符而不是真的修复校验(scripts/lesson_gate.py --existing 的警告,24 篇 0 阻塞项、46 条劝告)。

也就是说,本仓的信任语义是"标出证据等级而不是藏起内容"(compact 里就带 evidence_level),但唯一能说明"这是未定稿"的那个字段恰好不在默认档。这与今天刚修的 B25(evidence_level 只在 full 有、compact 没有)是同一个形状。

建议(两步,互不依赖)

  1. 无争议的一半:把 status(或仅 draft: true)在 compact 档按需暴露——只对 draft 课程加键,非 draft 的响应逐字节不变(与 summary_plain 的加法式约定一致,那套约定由 workers/d1-lesson-service.test.mjs 的 "keeps the exact legacy shape" 逐字节钉住)。这样 agent 至少能知道。

  2. 需要 owner 定的那一半:draft 是否应被检索。三种取法——
    (a) 保持可检索 + 标出 draft(= 上面第 1 步);
    (b) 完全排除 draft(那么 24 篇就是死重,要决定发布或删除);
    (c) 排除 E0 且 Verification 为占位符 的子集(17 篇 fanuc 系),其余保留。

    注意 (b) 会降低召回:search-quota-exhaustion-false-zero 是 C-07 的动机课程,里面写着"不要再执行旧版本的 rm misakanet/.quota.json",现在它是 draft——它正是最需要被搜到的那一类更正。

复跑方式

# 命中一片 draft 课程(默认档)
#   misakanet_search(query="search quota exhausted false zero results")
node --test workers/d1-lesson-service.test.mjs      # 逐字节的旧形状约定
python3 tests/test_mcp_server.py                    # 那条 check 所在的 smoke 脚本
python3 scripts/lesson_gate.py --existing --json $(python3 - <<'PY'
import json;print(" ".join(e["url"] for e in json.load(open("data/lessons.json")) if e.get("status")=="draft"))
PY
)                                                   # 24 篇 draft 的 0 阻塞 / 46 劝告

发现路径(留个记录,因为它是"分类积压"顺出来的):在做开放 issue 的机械五分类时,为了用生产检索判定"真盲区 vs 伪盲区",逐条驱动真实 worker 检索 issue 的技术签名,撞见 draft 课程被返回,进而发现那条 check 读了一个不存在的字段。



This repo is using Opire - what does it mean? 👇
💵 Everyone can add rewards for this issue commenting /reward 100 (replace 100 with the amount).
🕵️‍♂️ If someone starts working on this issue to earn the rewards, they can comment /try to let everyone know!
🙌 And when they open the PR, they can comment /claim #2270 either in the PR description or in a PR's comment.

🪙 Also, everyone can tip any user commenting /tip 20 @Ikalus1988 (replace 20 with the amount, and @Ikalus1988 with the user to tip).

📖 If you want to learn more, check out our documentation.

Activity

  1. zsxh1990 commented on Sep 28, 2026

    @zsxh1990
    Contributor

    !claim

  2. Ikalus1988 commented on Sep 28, 2026

    @Ikalus1988
    OwnerAuthor

    Live measurement, 2026-09-29 (this issue is closed; the additive half did not land)

    Closing this as completed is right for its first finding — the vacuous guard is gone (#2306, #2378).
    Its step 1 ("暴露 status … 只对 draft 课程加键,非 draft 的响应逐字节不变") is not in the code, and the
    gap is live on production today. Measured minutes ago, detail by detail, for the same draft lesson:

    misakanet_search(query="FANUC Robot Alarm Code Reference Table", top=1, detail=compact)
      keys: [evidence_level, freshness, id, kind, problem, title]        status: None
    misakanet_search(… detail=summary)
      keys: [domain, evidence_level, fix, freshness, id, kind, problem, tags, title]   status: None
    misakanet_search(… detail=full)
      keys: [created, description, domain, evidence_level, fix, id, kind, path, problem, score, status, tags, title, updated]
      status: "draft"
    misakanet_get_lesson(id="fanuc-alarm-code-reference")
      keys: [content, identity, path, trust_notice, voice]               status: None  (no evidence_level either)
    

    So the finding in the original report stands unchanged: a draft lesson is the top hit for its own title
    (fanuc-alarm-code-reference first), it is a status: draft file in lessons/contrib/, and both default
    detail levels — plus misakanet_get_lesson — are silent about it. misakanet_get_lesson is the second
    path, and it was not in the original report: its payload has no status at all, for any detail.

    The corpus is the other half of the measurement: production indexes 426 rows (GET /api/search-index
    → docCount 426, and canonical_lessons(lessons/) is exactly 426) while the repository's managed count is
    418 published lessons. Those 8 rows are the difference between "what the repo publishes" and "what
    search returns", which is also why the retrieval bench scores lower against production than against
    data/lessons.json — extra documents change IDF and displace expected lessons (measured in
    scripts/bench_production_recall.py).

    What this comment is not

    It is not a request to reopen the product question. Whether drafts should be searchable at all — the
    issue's own (a)/(b)/(c) — remains an owner decision, and the note in the original report is worth keeping in
    front of whoever decides it: search-quota-exhaustion-false-zero is a correction that is currently a
    draft, i.e. exactly the kind of lesson that needs to be findable.

    The additive half needs no such decision, so I have opened a PR for it: it appends status to a result
    only when the lesson is not published, so every published response stays byte-identical (the projection
    contract workers/d1-lesson-service.test.mjs pins), and it covers misakanet_get_lesson too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions