diff --git a/.gitignore b/.gitignore index 5890de5dcb..57f8d2f37c 100644 --- a/.gitignore +++ b/.gitignore @@ -310,3 +310,4 @@ docs-ui-refactor/ # Packaged Linsight skill bundles (scripts/pack_linsight_skill.sh output) /dist/ +.pnpm-store/ \ No newline at end of file diff --git a/docs/architecture/10-permission-rbac.md b/docs/architecture/10-permission-rbac.md index ffd9fab92e..3cf3d24001 100644 --- a/docs/architecture/10-permission-rbac.md +++ b/docs/architecture/10-permission-rbac.md @@ -1,7 +1,7 @@ # 用户、菜单与资源权限架构 > 现状版本：v3.0.0-beta1 / F048 -> 最后更新：2026-08-06 +> 最后更新：2026-08-13 > 规范来源：`features/v3.0.0-beta1/048-rebac-permission-model-grants/` BiSheng 的权限体系分为三个互不替代的层面： @@ -156,9 +156,12 @@ OpenFGA model。它是一个**不可变、可校验的全局权限语义版本** - 发布时通过一个 active release gate 原子切换新语义； - 既有 Grant 只引用稳定的 `model_key`，无需按资源/成员 fan-out 重写。 -代价是 Check/List 的关系图多一层 Catalog/model intersection。因此不能 -宣称零性能损失；Check、BatchCheck、ListObjects 必须由 BENCH-01 量化， -业务列表默认采用候选 cursor + BatchCheck。 +具体 action 的 Check/List 关系图仍包含 Catalog/model intersection，不能 +宣称零性能损失；Check、BatchCheck、ListObjects 必须由 BENCH-01 量化。 +`visible` 则是独立的单槽浅层执行索引，不展开具体 action 图。业务列表不设 +统一默认路径，而是按业务候选总量 `N_db`、用户可见数 `V`、可见率、业务过滤 +选择性、继承/CUSTOM 比例与端到端成本，在“完整可见 ID 优先”和“业务候选 +优先 + BatchCheck”之间逐入口选择。 ### 3.2 动作集合 @@ -177,8 +180,9 @@ OpenFGA model。它是一个**不可变、可校验的全局权限语义版本** | `publish` | workflow、assistant | | `unpublish` | workflow、assistant | -`visible` 是内部可见性 relation，不是可配置 action。调用方不能用 -`visible` 代替具体业务动作。 +`visible` 是内部可见性 relation，不是可配置 action。对资源存在任一合法 +授权来源即产生 shallow `visible` contribution；调用方不能用 `visible` +代替具体业务动作。 ### 3.3 level 与标准模型重算 @@ -203,6 +207,13 @@ OpenFGA model。它是一个**不可变、可校验的全局权限语义版本** level。选择项失效或未分配 level 时保留选择来源，但不产生运行时权限； active 自定义 model 若最终没有有效动作会阻止发布。 +`active` 只表示模型能否作为新增授权或 MOVE 的目标，不是既有授权的 +运行时开关：模型停用后禁止新增/转入，但既有 Grant 的 `visible`、具体 +action 和 `manage_permission` 均保持，且仍允许精确 REMOVE/MOVE OUT。 +模型删除不要求先停用，但必须先清理所有 active/pending/failed Grant、 +assignee、source projection 和 live tuple 残留；引用与残留均为零后，新 +Catalog 才移除 `model_key`，历史 release 仍作为 RETIRED 快照保留。 + --- ## 4. Grant、assignee、来源与 owner @@ -251,6 +262,24 @@ tenant + resource_type + resource_id + model_key 关系。能否授予同级 model 由 `allow_same_level` 和 `manage_permission` 共同约束。 +### 4.4 单槽 visible 来源投影 + +每个有效 owner/Grant assignee 来源都编译为一条可重建 contribution，写入 +`permission_visible_source_projection`。投影保留 source owner、model、 +locator、version、operation/migration item 和 checksum，用于引用计数、 +迁移追溯、对账与清理，但不参与读取时的 ALLOW 判定。 + +OpenFGA 资源上只有一个直接 `visible` relation，不存在 `visible_a`、 +`visible_b`、slot 或全局 switch。同一 resource/relation/subject 即使来自 +direct、部门、用户组、protected owner 或多个 model，也只保留一条聚合 +live tuple；撤销单一来源只退休对应 contribution，最后一个来源退休时才 +删除聚合 tuple。system/public/shared 和继承语义同样投影为资源本级 shallow +visible，但不伪造为普通 Grant source。 + +Grant mutation 在同一 durable projection operation 中冻结 action tuple 与 +visible contribution 的 after-state，并在一个 OpenFGA Write 中原子提交； +编译后的 writes+deletes 超过 90 时整体拒绝，不拆批报告部分成功。 + --- ## 5. 资源 mode 与继承 @@ -370,18 +399,37 @@ business resolve → VerifiedPermissionTarget → consistency marker → OpenFGA Check ``` -业务列表： +业务列表按入口采用两种路径之一： ```text -业务数据库 keyset cursor 取 ≤100 个候选 +可见 ID 优先：完整消费 StreamedListObjects(visible) +→ 正常 EOF、去重、tenant fence 和容量检查后一次性交付 ID 集 +→ 业务数据库按 ID 过滤、状态、排序并加载详情 + +候选优先：业务数据库 keyset cursor 取有界候选 → 每个候选由业务 adapter 验证 -→ OpenFGA BatchCheck -→ 保留允许项并继续 cursor +→ OpenFGA BatchCheck(visible) +→ 保留允许项，不足一页则从最后扫描候选继续 ``` -`ListObjects` 只允许已通过 BENCH-01 的特定入口使用。它不负责加载 -业务数据、tenant/status 过滤或通用分页；结果超过批准上限时必须报错， -不能把 OpenFGA 默认 1,000 条上限静默当作全集。 +完整可见枚举只允许通过 `PermissionService.list_visible_objects()` 使用， +底层必须完整消费 StreamedListObjects。deadline、取消、服务错误、异常对象 +类型、非正常结束或超过调用方容量均返回 25014，不交付部分前缀，也不从 SQL +source projection 补 ALLOW。该路径不执行 super admin / tenant admin 扩权。 + +当前入口选择： + +- `knowledge/space/joined`：可见 ID 优先，容量 5,000；按 500 ID 分块查询 + 数据库，排除 canonical 本人创建并稳定排序，不读 `space_channel_member`、 + 成员角色，也不重复 visible 检查； +- `knowledge/space/department`：先从部门绑定与部门表得到小候选集，再做一次 + 有界 `batch_check_visible`，最后查询空间详情； +- 知识空间文件/文件夹列表：业务候选优先，使用稳定 `(sort_key,id)` cursor + 跨批填页；即使多数子资源继承父空间权限，也必须对最终子资源做 BatchCheck。 + +具体 action 的 `ListObjects` 仍只允许 BENCH-01 批准的特定入口使用。它不 +负责加载业务数据、tenant/status 过滤或通用分页；结果超过批准上限时必须 +报错，不能把 OpenFGA 默认限制静默当作全集。 --- @@ -483,6 +531,7 @@ REMOVE，必须携带 resource、Catalog 和 assignee 乐观版本。 - `resource_permission_mode` - `permission_projection_operation` - `permission_projection_tuple` +- `permission_visible_source_projection` ### 9.3 正式数据迁移 @@ -513,9 +562,11 @@ src/backend/scripts/migrate_f048_permission_data.py Celery/Linsight 暂停任务消费； 2. D1：API 正常启动链执行 Alembic upgrade，确认单 head； 3. D2：运维进入 backend 容器执行脚本；脚本确认 ready F048 heartbeat=0， - 两次稳定源扫描并冻结 run/item/checksum； -4. D3：在同一 Store 发布一个 F048 model，分批写 SQL/FGA； -5. D4：higher-consistency 验证后，仅删除记录到 run 的 legacy tuple；两份 legacy + 两次稳定源扫描并冻结唯一 run/item/checksum； +4. D3：直接从旧系统 Config、tuple、owner 和 canonical 业务事实编译最终 + Grant/assignee/source contribution，在同一 Store 发布一个最终 F048 model，分批写 SQL/FGA； +5. D4：验证 contribution、aggregate/live tuple、source/target checksum、无来源 tuple=0、 + streamed visible 集合与 canonical oracle 一致后，仅删除记录到 run 的 legacy tuple；两份 legacy Config 原始行保留为只读排障证据，但不再参与任何运行时读写或授权判断； 6. D5：迁移成功后重启全部 backend 进程，自动发现新 model 并绑定 SQL CURRENT Catalog； 7. D6：smoke、全实例 heartbeat 和语义校验通过后，迁移门禁自动解除。 @@ -523,8 +574,9 @@ src/backend/scripts/migrate_f048_permission_data.py 脚本 DB scan batch=500，FGA write batch≤90，每批写 checkpoint。崩溃后 必须携带同一个 `run-id` 从 frozen items 前向恢复。 -没有 preview、dry-run、rollback、cleanup、Store switch 或 model A/B -并行窗口。迁移失败时保持应用迁移门禁和 F048 runtime 不就绪，修复同一 run 的前向路径。 +没有 preview、dry-run、rollback、cleanup、Store switch、中间 F048 model、 +model A/B 或第二次迁移。迁移失败时保持应用迁移门禁和 F048 runtime 不就绪， +只允许携带同一 `run-id`、固定 Store/source model 的前向恢复。 --- @@ -570,12 +622,19 @@ OpenFGA 使用 JSON log，并暴露 `:2112/metrics`。RPC histogram 在部署中 - `permission_decision` - `permission_projection` +- `permission_visibility_projection` +- `permission_visible_list` - `permission_catalog_publish` - `permission_roster_explain` - `permission_migration` 日志不得记录用户名、部门名、资源名、token 或 legacy Config 原文。 +`permission_visibility_projection` 记录 source count、unique tuple、stale/orphan、 +checksum 与耗时；`permission_visible_list` 记录 strategy、candidate/visible/scanned、 +scan amplification、stream_completed、capacity 及 DB/FGA/total 耗时。无来源 tuple、 +删除残留、stream incomplete、joined 容量达到 80% 或候选扫描放大超过阈值必须告警。 + --- ## 12. 失败策略与不变量 @@ -611,11 +670,13 @@ OpenFGA 使用 JSON log，并暴露 `:2112/metrics`。RPC histogram 在部署中 | Grant policy | `bisheng/permission/domain/services/grant_service.py` | | mode policy | `bisheng/permission/domain/services/mode_service.py` | | durable projection | `bisheng/permission/domain/services/projection_service.py` | +| shallow visible 编译与对账 | `bisheng/permission/domain/services/visibility_projection_service.py` | | 业务 target registry | `bisheng/permission/application/resource_authorization.py` | | runtime facade | `bisheng/permission/application/runtime.py` | | API composition | `bisheng/api/services/f048_permission_runtime.py` | | Catalog/Grant/decision endpoints | `bisheng/permission/api/endpoints/` | | Alembic DDL | `bisheng/core/database/alembic/versions/f048_permission_model_grants.py` | +| visible source DDL | `bisheng/core/database/alembic/versions/v3_0_0_f048_visible_source_projection.py` | | 数据迁移 CLI | `scripts/migrate_f048_permission_data.py` | | BENCH-01 | `scripts/benchmark_f048_permission_paths.py` | diff --git a/features/v3.0.0-beta1/048-rebac-permission-model-grants/bench-01-flat-visible-report-20260813.md b/features/v3.0.0-beta1/048-rebac-permission-model-grants/bench-01-flat-visible-report-20260813.md new file mode 100644 index 0000000000..b596b11099 --- /dev/null +++ b/features/v3.0.0-beta1/048-rebac-permission-model-grants/bench-01-flat-visible-report-20260813.md @@ -0,0 +1,371 @@ +# BENCH-01 初版：知识空间 `visible` 展平关系性能验证 + +## 0. v1.15.1 单槽直接 `visible` 发布门禁（2026-08-13） + +### 0.1 结论 + +T162 发布门禁通过。在 `192.168.106.116` 的测试后端容器内，使用官方 +OpenFGA v1.15.1 预编译产物（SHA-256 +`5f06f0d263a450fcc081e5e7b6bec4649e8da5f6c9d84c9718e5ba213158eec9`）启动仅监听 +`127.0.0.1` 的内存 datastore 实例，不修改 v1.14.2 业务容器，不写业务 Store。 + +30 轮正式样本与 2 轮预热均使用 `HIGHER_CONSISTENCY`，串行并发度 1。 +完整消费 StreamedListObjects 至正常 EOF 后才记录成功；所有结果数和集合 +checksum 与 canonical oracle 一致，错误率为 0，720/720 个请求都能关联服务端 +dispatch/datastore read。 + +### 0.2 固定环境与校验和 + +| 项目 | 值 | +|---|---| +| OpenFGA | v1.15.1，commit `1db35fb8b33d7512666e49e6b75cbd2cca8c4694` | +| 运行方式 | 测试容器内独立进程，memory datastore，HTTP `127.0.0.1:18080` | +| Store | `01KZXGMMNNVM0CR1SK7YV0JF29` | +| Authorization Model | `01KZXGMY99KRNR71W8DCS3SA05` | +| model checksum | `cdcb8c337e756305966082e0d9cd5540e794692e614a048de9f8efbbf168350f` | +| required relations checksum | `54a9eeff8904de4b8b237a019aa7080ac0d53183ae7e35ee98d7237a7da877d4` | +| contract checksum | `80a80489bf4f3ae01f5b1f3519cd275b10a92f16dd58df5fa1378de48c608912` | +| dataset/source/visible checksum | `d083c56f...ab22` / `9e2e7699...0df0` / `f185c204...994e` | +| OpenFGA tuple 数 | 41,666 | +| 业务分布 | 代表性合成数据，非生产脱敏样本 | + +目标 DSL 的列表执行关系只有一个可直接写入的单槽： + +```text +resource#visible: + [user, + user:*, + department#member, + department#subtree_member, + user_group#member, + user_group#admin, + tenant#member] +``` + +`ordinary/protected` 仅保留在 SQL source projection 的来源审计字段，不再是 OpenFGA +`visible` 查询图的中间 relation。system/public/shared/parent 继承由各 Owner adapter +计算有效来源后同样投影为直接 `visible` tuple；具体 action 仍保持 +Grant/Model/Catalog ReBAC 图。模型中没有 A/B relation 或 switch。 + +### 0.3 完整可见枚举结果 + +| `N_db` | `V` | 来源 | P50 | P95 | P99 | 门槛 | 结果 | +|---:|---:|---|---:|---:|---:|---:|---| +| 10,000 | 10 | direct | 2.936 ms | 4.094 ms | 4.293 ms | 50 ms | PASS | +| 10,000 | 100 | department userset | 4.969 ms | 6.630 ms | 8.041 ms | 100 ms | PASS | +| 10,000 | 1,000 | group userset | 20.006 ms | 25.167 ms | 27.962 ms | 300 ms | PASS | +| 10,000 | 5,000 | system wildcard | 84.307 ms | 106.659 ms | 109.454 ms | 1,000 ms | PASS | +| 100,000 | 10 | direct | 4.058 ms | 5.618 ms | 7.110 ms | 50 ms | PASS | +| 100,000 | 100 | department userset | 4.977 ms | 7.203 ms | 22.753 ms | 100 ms | PASS | +| 100,000 | 1,000 | group userset | 20.589 ms | 26.592 ms | 46.046 ms | 300 ms | PASS | +| 100,000 | 5,000 | direct+department+group 重叠 | 84.357 ms | 119.068 ms | 129.705 ms | 1,000 ms | PASS | + +Check 合并 240 个样本，P50/P95/P99 为 `3.455/6.095/8.540 ms`。逻辑 +BatchCheck 20/50/100 的 P95 为 `11.736/20.738/39.422 ms`；v1.15.1 单次 +BatchCheck 最多 50 项，因此 100 候选使用业务侧 `2×50` 并合并结果，没有 +发送非法的 100-check 请求。 + +### 0.4 业务路径对比 + +| 入口 | 选定路径 | `N_db` | `V` | `p=V/N_db` | 候选通过率 | 扫描行 | 放大 | P50/P95/P99 | +|---|---|---:|---:|---:|---:|---:|---:|---:| +| joined | visible ID-first | 100,000 | 100 | 0.001 | 1.00 | 100 | 5.0 | 6.049 / 8.660 / 9.463 ms | +| department | candidate-first | 200 | 100 | 0.5 | 0.50 | 50 | 2.5 | 16.677 / 24.900 / 30.473 ms | +| file | candidate-first | 100,000 | 5,000 | 0.05 | 0.95 | 100 | 2.0 | 36.326 / 49.156 / 63.290 ms | + +这里的 `N_db`、排序、候选通过率和扫描行由 checksum-pinned 合成业务数据 +驱动，不冒充生产 MySQL/DM8 延迟。该结果确认本次入口选择：joined 在资源总量大、 +可见范围小时使用 ID-first；department 候选小、file 在父级继承下候选通过率高， +使用 candidate-first + 分块 BatchCheck。MySQL/DM8 的分块 `IN`、排序与 cursor 一致性仍由 +后续业务入口集成测试覆盖。 + +### 0.5 首轮失败及修正 + +首轮 v1.15.1 运行暴露 builder 虽然已把 Grant-derived tuple 放到资源上，但 +`visible` 仍定义为 `permission_enabled AND (ordinary_custom/protected/system/inherit)`。强一致实时 +样本中，`V=100/1,000/5,000` 约为 `148 ms/1.33 s/6.8 s`，5,000 结果产生 +约 10,000 dispatch，明确不达标。根因是查询图仍然按每个候选资源计算中间 +intersection，而不是部门/用户组 userset本身。 + +修正为可直接写入的单一 `visible` relation 后，同样数据、一致性和服务版本下 +才得到上述通过结果。这个失败过程保留为门禁价值证据：不能只看 tuple 是否 +在 resource 上，必须验证最终业务 relation 的完整 DSL 和服务端执行计数。 + +## 1. 结论 + +在 `192.168.106.116` 的现有 OpenFGA 服务中新建隔离 Store，使用相同数据分别测试： + +- 当前深层关系：`visible = permission_enabled AND (Grant/Model/Catalog 等可见分支)`； +- 展平关系：基准中命名为 `flat_visible`，直接投影 + `[user, department#member, user_group#member] -> resource`。 + +测试结果支持展平方案：在 10,000 个资源、用户可见 1,000 个资源、 +`HIGHER_CONSISTENCY` 条件下，`flat_visible` P95 为 **19.281 ms** 且返回完整； +当前深层 `visible` 在约 3 秒 deadline 后仅返回 **659/1,000**，HTTP 状态仍为 200。 + +`flat_visible` 只是 A/B 测试中的对照名称。若采用该方案，建议直接把业务 `visible` +改为展平后的有效可见关系，不长期维护两个相同业务语义的 relation。 + +## 2. 测试边界 + +| 项目 | 值 | +|---|---| +| 测试日期 | 2026-08-13 | +| OpenFGA 环境 | `192.168.106.116` 当前服务 | +| OpenFGA 版本 | v1.14.2 | +| 数据存储 | 当前 OpenFGA MySQL datastore | +| 隔离 Store | `01KZX2TTNEEMZWN3R8DE3C08JM` | +| Authorization Model | `01KZX2TTPP0VH72QPYCPF2P0YQ` | +| 业务 Store | 全程未写入 | +| 资源规模 | 1,000、10,000 | +| 可见范围 | 直接用户 10、部门 100、用户组 1,000 | +| Tuple 数 | 66,446 | +| ListObjects deadline | 约 3 秒（当前服务配置） | +| ListObjects max results | 1,000（当前服务默认值） | + +每组关系使用相同资源和相同授权主体；返回结果校验数量和集合 checksum。 +强一致每组运行 3 次，默认一致性每组运行 5 次，串行执行，未进行并发压测。 + +## 3. 模型对照 + +### 3.1 当前深层 `visible` + +```text +resource#visible + = permission_enabled + AND ( + protected_visible FROM grant + OR (ordinary_visible FROM grant AND custom_mode) + OR system_visible + ) + +grant#ordinary_visible + = ordinary_assignee AND active FROM model + +model#active + = active FROM release + +release#active + = enabled_marker AND active FROM catalog +``` + +ListObjects 需要反向展开 resource、Grant、assignee、Model、Release、Catalog 及 +intersection/union 分支。 + +### 3.2 展平关系 + +```text +resource#flat_visible: + [user, + department#member, + department#subtree_member, + user_group#member, + user_group#admin] +``` + +基准数据同时保留深层 Grant 事实和等价的展平 tuple，确保两条查询路径期望集合相同。 + +## 4. 核心结果 + +### 4.1 10,000 资源 + +#### HIGHER_CONSISTENCY + +| 可见来源 | 期望结果 | 深层 `visible` P95 | 深层结果 | 展平 P95 | 展平结果 | +|---|---:|---:|---:|---:|---:| +| 直接用户 | 10 | 3,007.933 ms | 10/10 | 2.900 ms | 10/10 | +| 部门 | 100 | 3,011.952 ms | 100/100 | 6.054 ms | 100/100 | +| 用户组 | 1,000 | 3,022.130 ms | 首个样本 659/1,000，集合不稳定 | 19.281 ms | 1,000/1,000 | + +深层关系在 10 和 100 结果场景虽然最终返回完整，但耗时已经达到 deadline； +1,000 结果场景出现 HTTP 200 下的部分结果，不能作为安全的资源范围使用。 + +#### 默认一致性 + +| 可见来源 | 深层 `visible` P50/P95 | 展平 P50/P95 | 展平集合 | +|---|---:|---:|---:| +| 直接用户 10 | 3,007.003 / 3,008.364 ms | 2.441 / 11.487 ms | 完整、稳定 | +| 部门 100 | 3,005.682 / 3,006.727 ms | 5.078 / 5.551 ms | 完整、稳定 | +| 用户组 1,000 | 871.046 / 1,565.028 ms | 13.685 / 15.963 ms | 完整、稳定 | + +默认一致性的深层关系受 iterator cache 状态影响明显；展平关系不依赖热缓存也保持毫秒级。 + +### 4.2 1,000 资源 + +#### HIGHER_CONSISTENCY + +| 可见来源 | 深层 `visible` P95 | 展平 P95 | 展平结果 | +|---|---:|---:|---:| +| 直接用户 10 | 3,010.499 ms | 3.332 ms | 完整、稳定 | +| 部门 100 | 3,018.562 ms | 5.380 ms | 完整、稳定 | +| 用户组 1,000 | 3,010.112 ms，结果不完整 | 16.375 ms | 完整、稳定 | + +#### 默认一致性 + +| 可见来源 | 深层 `visible` P50/P95 | 展平 P50/P95 | +|---|---:|---:| +| 直接用户 10 | 31.092 / 151.122 ms | 3.007 / 3.144 ms | +| 部门 100 | 31.175 / 265.583 ms | 4.676 / 4.748 ms | +| 用户组 1,000 | 33.855 / 612.956 ms | 12.345 / 15.495 ms | + +## 5. OpenFGA 执行成本 + +以下为 10,000 资源强一致场景的代表请求： + +| 可见范围 | 关系 | query time | dispatch | datastore query | datastore item | +|---:|---|---:|---:|---:|---:| +| 10 | 深层 `visible` | 3,003 ms | 4,573 | 14,203 | 8,086 | +| 10 | 展平 | 1 ms | 10 | 4 | 10 | +| 100 | 深层 `visible` | 3,008 ms | 4,249 | 12,860 | 7,624 | +| 100 | 展平 | 3 ms | 102 | 7 | 101 | +| 1,000 | 深层 `visible` | 3,003 ms | 4,375 | 12,110 | 8,821 | +| 1,000 | 展平 | 15 ms | 1,001 | 5 | 1,001 | + +展平查询的成本基本跟用户实际可见结果数相关，而不是跟平台资源总数及 +Grant/Model/Catalog 图的反向展开规模相关。 + +## 6. 方案评估 + +### 6.1 性能判断 + +展平关系在本轮 10,000 资源测试中满足“平台资源大、用户可见范围小”的目标： + +- 可见 10：强一致 P95 2.900 ms； +- 可见 100：强一致 P95 6.054 ms； +- 可见 1,000：强一致 P95 19.281 ms； +- 三种主体来源均返回完整且 checksum 稳定； +- 没有依赖默认一致性的热缓存才能达标。 + +因此，从读取性能和 ListObjects 完整性看，展平 `visible` 方案合理。 + +### 6.2 上线前必须解决的投影语义 + +展平关系是执行索引，Grant/Model/Catalog 和 SQL Grant 仍是事实来源。实现必须覆盖： + +1. Grant 新增、撤销和模型替换； +2. Catalog 发布导致模型动作/等级变化，以及模型 active 只改变可分配状态、既有投影不变； +3. `CUSTOM/INHERIT` 模式切换； +4. protected owner 与 ordinary assignee； +5. direct、department、department subtree、user group member/admin； +6. public、tenant shared、system visible； +7. 同一主体通过多个来源获得同一资源时，撤销一个来源不得误删最终 `visible`； +8. 资源删除、租户隔离和投影失败闭合。 + +部门和用户组应继续使用 userset tuple，不展开为每用户每资源 tuple，避免组织变更造成 +用户级 fan-out。 + +### 6.3 写放大 + +本轮 11,000 个资源共写入 66,446 条 tuple。展平方案相对深层事实增加一条 +“有效主体 -> 资源”的执行 tuple；真实增量与有效授权主体/资源组合数量相关。 + +上线前还应补充 mutation benchmark： + +- 单Grant新增/撤销； +- 同主体多来源去重与最后来源撤销； +- 1,000/10,000 资源下模型停用不扫描 Grant/不改 visible 的验证，以及 Catalog 发布； +- 部门/用户组成员变更； +- projection ledger重试与失败恢复。 + +## 7. 建议 + +1. 直接展平现有业务 `visible`，不要长期保留 `visible` 和 `list_visible` 两套相同语义； +2. `visible` 只表达“用户是否能看到资源”，具体 edit/download/manage 等动作继续使用 + 当前 Grant/Model/Catalog 关系； +3. 新增受限 `list_visible_objects(resource_type, actor, max_results)` 入口，只允许经过 + BENCH-01 的资源类型； +4. 当前 OpenFGA `listObjects-max-results=1000`，入口必须显式检测/约束用户可见结果上限， + 不能把 1,000 当作天然全集； +5. 完成投影一致性和写路径性能验证后，再把知识空间 `joined` 改为 + `ListObjects(visible) -> DB按ID查询 -> 排除本人创建`。 + +## 8. A/B 槽与组织 userset 补充对照测试（弃选证据） + +### 8.1 目的与数据 + +为验证目标设计中的 A/B 槽是否削弱展平枚举性能，2026-08-13 在同一隔离 Store 追加： + +- Authorization Model：`01KZX7WGH8773H1EV083Y5Q7T6`； +- OpenFGA：v1.14.2，build `c8a0e7b553ec322edfaa88948d57ba73bcad8883`，容器镜像 ID + `sha256:65c4db264a403bfd62387fed4fe48f1b7cdda694bdac5f493a8f3c5a6460b4ba`； +- 资源类型：`bench_ab_space_10000_v1`，10,000 个资源； +- 新增 tuple：13,365；A/B 两槽数据相同，每个资源都有 + `visibility_switch -> permission_visibility_switch:bench-ab-current-v1`； +- 可见分布：直接用户 10、部门 userset 100、用户组 userset 1,000、三种来源重叠后的 + 混合集合 1,000； +- A 槽、B 槽分别激活测试；每组普通 ListObjects 强一致 10 次、默认一致性 20 次； +- 切换 A→B 仅原子删除 `slot_a_active`、写入 `slot_b_active`； +- 另用 StreamedListObjects 强一致完整消费 10 次，记录首条与完整结果耗时。 + +测试模型中： + +```text +visible = + (ordinary_visible_a AND slot_a_active FROM visibility_switch) + OR + (ordinary_visible_b AND slot_b_active FROM visibility_switch) +``` + +部门和用户组仍为 `department#member`、`user_group#member` userset，没有展开成员。 + +### 8.2 普通 ListObjects 结果 + +以下为 10,000 资源、HIGHER_CONSISTENCY P95；所有 240 个采样均成功，结果数量精确、 +checksum 稳定： + +| 当前槽 | 可见来源 | 结果数 | 裸展平 `flat_visible` | 当前底层槽 relation | 业务 `visible` | +|---|---|---:|---:|---:|---:| +| A | 直接用户 | 10 | 5.830 ms | 2.938 ms | 51.133 ms | +| A | 部门 userset | 100 | 9.672 ms | 6.759 ms | 50.686 ms | +| A | 用户组 userset | 1,000 | 16.289 ms | 17.040 ms | 63.233 ms | +| A | 直接+部门+用户组重叠 | 1,000 | 16.242 ms | 35.443 ms | 48.964 ms | +| B | 直接用户 | 10 | 3.634 ms | 2.894 ms | 52.810 ms | +| B | 部门 userset | 100 | 5.292 ms | 6.642 ms | 53.643 ms | +| B | 用户组 userset | 1,000 | 14.632 ms | 20.458 ms | 87.976 ms | +| B | 直接+部门+用户组重叠 | 1,000 | 12.758 ms | 13.177 ms | 80.975 ms | + +A/B 两个槽返回集合完全相同，切换不改变授权语义。部门和用户组 userset 的底层槽查询仍 +与直接展平关系同一数量级；主要新增耗时来自 `visible` 的 switch/intersection，而不是 +userset 本身。 + +1,000 资源对照模型 `01KZX8617BJHZCM7PKHGH7EHGB` 中，B 槽业务 `visible` 强一致 P95： + +| 可见来源 | 结果数 | 1,000 资源 | 10,000 资源 | +|---|---:|---:|---:| +| 直接用户 | 10 | 20.794 ms | 52.810 ms | +| 部门 userset | 100 | 25.211 ms | 53.643 ms | +| 用户组 userset | 1,000 | 37.753 ms | 87.976 ms | +| 混合重叠 | 1,000 | 25.371 ms | 80.975 ms | + +这说明当前“每个资源一个 switch link + intersection”的表达仍受资源总量影响；虽然远低于 +深层 Grant/Model/Catalog 约 3 秒的结果，但没有完全达到裸展平查询只随实际可见结果增长的 +理想特征。 + +### 8.3 StreamedListObjects 结果 + +10,000 资源、B 槽激活、HIGHER_CONSISTENCY，所有 120 个采样均完整且 checksum 稳定： + +| 可见来源 | 结果数 | 裸槽完整消费 P95 | `visible` 首条 P95 | `visible` 完整消费 P95 | +|---|---:|---:|---:|---:| +| 直接用户 | 10 | 22.924 ms | 70.421 ms | 71.376 ms | +| 部门 userset | 100 | 15.537 ms | 65.760 ms | 69.083 ms | +| 用户组 userset | 1,000 | 76.960 ms | 68.421 ms | 96.934 ms | +| 混合重叠 | 1,000 | 56.592 ms | 91.241 ms | 124.356 ms | + +StreamedListObjects 解决普通 ListObjects 结果上限与静默截断风险，但不会消除模型图本身的 +解析成本。当前合成数据下仍低于 125ms P95，不能据此直接外推并发或生产长尾。 + +### 8.4 结论与设计影响 + +1. 部门、用户组授权可以保留 userset，不必展开为用户级 tuple；本轮未发现 userset 破坏 + 底层槽 relation 的列表性能，其成本仍主要随用户实际可见资源数增长。 +2. A/B 双写明确增加存储与写放大；语义上只有当前激活槽贡献结果，A→B 切换前后集合一致， + 但不能据此假定 OpenFGA 物理执行时完全不遍历未激活分支。 +3. 但是当前基于资源 `visibility_switch` + intersection 的运行时选槽给 10,000 资源带来 + 约 40～75ms P95 附加开销，并表现出资源总量相关性。 +4. 最新产品语义已经明确：模型停用只禁止新增或变更授权，已有 Grant 不受影响；模型删除前 + 必须先撤销或替换全部绑定。因此不再存在“停用时瞬时撤销上万绑定”的全局原子切换需求。 +5. 结合 2 倍写放大和约 40～75ms P95 的 switch/intersection 附加开销，目标方案选择单槽 + 浅层 `visible`，A/B 仅保留为弃选方案的实测证据，不进入生产 DSL、迁移和运行时。 +6. 上线门禁仍需在 pinned v1.15.1、目标完整单槽 DSL、生产脱敏分布和并发条件下复测普通 + Check、BatchCheck、StreamedListObjects、Grant 来源撤销/替换、模型停用投影不变和删除引用门禁。 diff --git a/features/v3.0.0-beta1/048-rebac-permission-model-grants/design.md b/features/v3.0.0-beta1/048-rebac-permission-model-grants/design.md index 2f64fe7718..b6ddd836c5 100644 --- a/features/v3.0.0-beta1/048-rebac-permission-model-grants/design.md +++ b/features/v3.0.0-beta1/048-rebac-permission-model-grants/design.md @@ -6,11 +6,11 @@ > - 本文定义选定架构、OpenFGA 新模型、数据/API 契约、发布一致性和迁移 runbook； > - 后续 [tasks.md](./tasks.md) 只拆实现与验证任务，不复制本文论证。 -**状态**：✅ 功能与迁移脚本开发完成；Design ★ 与 Tasks ★ 均已确认；本地 E2E 经用户确认不执行 +**状态**：🔲 F048 尚未上线；本次“单槽浅层 visible、模型停用/删除语义及数据驱动列表路径”Design 24 项复审 LGTM，待用户 Design ★ 确认 **关联**：[spec.md](./spec.md) · [产品说明](./product-authorization-model-and-migration-guide.md) · [release contract](../release-contract.md) **版本**：v3.0.0-beta1 **功能分支**：`feat/v3.0.0-beta1/048-rebac-permission-model-grants` -**最后更新**：2026-07-31 +**最后更新**：2026-08-13 --- @@ -25,6 +25,11 @@ MySQL/DM8 中可审计、可版本化的 Action/Model/Grant 控制面，以及 `INHERIT`/`CUSTOM` 与受保护 owner。Platform 和 Client 使用同一后端契约解释模型、来源、等级与模式。 +本次增量只展平固定 `visible` 执行关系，并为每个 canonical 授权来源维护单槽浅层投影； +模型停用只禁止新增或变更授权，已有 Grant 继续生效，删除前必须先撤销或替换全部绑定。 +列表按真实候选量、可见率、继承比例和端到端成本，在“完整可见 ID 优先”与 +“业务候选优先 + BatchCheck”之间逐入口选择。具体 action 的最终 ReBAC 语义保持不变。 + ### 1.2 非目标 - 不引入 DENY/黑名单；有效授权仍是白名单并集。 @@ -42,6 +47,9 @@ MySQL/DM8 中可审计、可版本化的 Action/Model/Grant 控制面，以及 - 不在 Alembic revision、应用启动钩子、API 或 Celery 中执行旧权限数据读取、转换、回填、 清理或 OpenFGA tuple 迁移；数据迁移只由 `src/backend/scripts/` 下的专用运维脚本执行。 - 不在本期引入新的 UI、状态管理或 OpenFGA SDK 依赖。 +- 不把具体 action 全部展平，不把管理员管理范围并入个人 `visible`，不把 SQL/source projection + 用作 OpenFGA 故障时的 ALLOW fallback。 +- 不规定所有业务列表永久使用同一种权限筛选顺序，也不把 116 合成 BENCH-01 当作生产结论。 --- @@ -49,7 +57,7 @@ MySQL/DM8 中可审计、可版本化的 Action/Model/Grant 控制面，以及 - 全局架构铁律直接遵循 [Constitution C1–C7](../../../docs/constitution.md)，本节不另行 复制或重新定义。 -- 遵循版本契约 INV-9～INV-26，尤其是： +- 遵循版本契约 INV-9～INV-27，尤其是： - 具体动作只由 OpenFGA 最终裁决； - Config 大 JSON 不再是运行时真相； - 生产 Check/List/Write 必须显式固定 Authorization Model ID； @@ -76,7 +84,8 @@ MySQL/DM8 中可审计、可版本化的 Action/Model/Grant 控制面，以及 旧 model 只记录 predecessor 审计链，不能构造 compatibility client，不能在 SQL CURRENT Catalog 不匹配时仅凭“它是最新模型”跨过这一步。 - OpenFGA 单次 Write 的 writes+deletes 上限为 100，且同一请求整体原子成功或失败。 - 所有可见状态切换必须把最终 commit 控制在该上限内。 + 每个 Grant/assignee 来源变更必须把自身 action 与 visible delta 的最终 commit 控制在该上限内； + 不把多个独立绑定的清理伪装成一次跨批原子删除。 - OpenFGA 默认读可能命中缓存；安全变更后的读使用 `HIGHER_CONSISTENCY`， 并在服务端缓存窗口内保持资源/全局“近期变更”标记。窗口取部署中 query/iterator/ controller cache TTL 的最大值+5s；当前 compose 最大值为 30s，因此默认 35s， @@ -86,6 +95,15 @@ MySQL/DM8 中可审计、可版本化的 Action/Model/Grant 控制面，以及 - 当前代表性 DM8/OpenFGA 环境已有单次 Check 约 8–15ms 的证据 （F036 `loadtest-report.md`）；生产脱敏分布尚未附入仓库，属于切换门禁 `BENCH-01`，不能把该环境数据冒充生产结论。 +- 2026-08-13 在 116 的隔离性能 Store、OpenFGA v1.14.2 上完成了首轮合成 BENCH-01： + 10,000 个资源、单用户可见 10/100/1,000 时，直接展平 relation 的 P95 分别约为 + 2.9/6.1/19.3ms 且集合完整；当前深层 `visible` 在相同量级约 3s，1,000 目标只返回 + 659。该结果只证明“展平可见关系”值得进入目标模型与业务链路复测，不证明 116 版本可替代 + pinned v1.15.1，也不单独决定任何业务接口的列表路径。 +- 同一 Store 的补充 A/B 槽实验表明：10,000 资源下，裸单槽直接 relation 的 P95 约 + 2.9～19.3ms，而经资源级 switch/intersection 的 A/B 查询约 53～88ms，且双写增加约 2 倍 + Grant-derived visible tuple。既然停用不撤销已有授权、删除又要求先清零绑定，A/B 不再提供 + 必需的业务原子性，反而把永久读成本和写放大施加到所有请求；因此目标采用单槽浅层投影。 ### 2.1 本 Design 采用并等待 ★ 确认的产品选择 @@ -98,6 +116,9 @@ MySQL/DM8 中可审计、可版本化的 Action/Model/Grant 控制面，以及 | OQ-05 | viewer=false、editor=false、manager=false、owner=true | 无 `manage_permission` 的模型该值不产生能力；owner 可授同级 | | OQ-06 | 文件预览不设置 PermissionAction | 预览不调用动作鉴权；只有原件/打包下载检查 `download`；知识库 RAG 仍检查库 `use` | | OQ-07 | 选择 A：退役既有 F018 owner 交接 | F048 启服时移除该 API，本期不实现 protected owner transfer；其他 ordinary owner 仍可并存 | +| OQ-08 | 按代表性实际业务数据选择列表路径 | 可见 ID 优先与业务候选优先均合法；比较候选总量、可见率、继承/CUSTOM 比例、业务选择性、页大小、扫描放大、完整性和端到端成本 | +| OQ-09 | 个人可见列表不因管理员身份扩权 | `visible` Check/BatchCheck/枚举不执行 super_admin/tenant_admin 全量短路；具体管理动作仍按 Constitution C4 的身份 gate | +| OQ-10 | 停用保留已有授权；删除前清零绑定 | inactive 只禁止新增/变更到该模型，不改变既有 Grant；删除必须先撤销或替换全部绑定，引用、来源投影与残留 tuple 清零后才允许完成 | | 迁移发布 | 由部署平台或入口层停止业务流量后，在 backend 容器内对现有 Store 直接迁移，不支持预演和回滚 | API、Celery、Linsight 只注册 lazy Permission Runtime Context，首次权限调用才校验 Store/model/Catalog 并初始化；部门投影作为依赖 Permission Runtime 的独立 lazy Context。旧 model 下首次权限调用 fail closed，但应用不把迁移状态耦合到 `/health` 或全局 HTTP/WS/Worker 门禁；迁移脚本输出和退出码是运维状态的权威来源。发布一个新不可变 model ID，源校验、tuple 转换、旧 tuple 退役和目标校验在同一正式 run 完成；迁移后重启并只运行新 model | | 数据升级职责 | Alembic 仅执行 DDL；`src/backend/scripts/` 专用脚本执行数据迁移 | 先完成 schema upgrade，再由 `migrate_f048_permission_data.py --apply` 创建/续跑正式 data migration run；服务启动不自动迁数据 | @@ -148,19 +169,12 @@ MySQL/DM8 中可审计、可版本化的 Action/Model/Grant 控制面，以及 从结构上阻止跨模型拼接等级、`manage_permission` 和同级策略。代价是每次动作 Check 比旧四档直连多经过 Grant→ModelRelease→Catalog，且包含 intersection；这是有意识 用读路径深度换全局原子发布和无 fan-out 更新，不得宣称“零性能损失”。 -- **查询策略**： - - 单资源 `Check/BatchCheck` 是默认路径，BENCH-01 分别测 direct、department、 - group、继承和多 Grant 的 dispatch/read 数与 P50/P95/P99； - - 高频业务列表由业务 Service 按租户、状态、关键字和 cursor 有界加载候选， - 再把本页/小批 ID 交给 `PermissionService.batch_check_actions()`；权限模块不查询业务表； - - `ListObjects` 只用于调用方确实需要完整授权对象集合、并且该资源类型的规模/模型路径 - 已通过 BENCH-01 的场景。它不得成为通用分页实现，也不得把默认 1,000 结果当作全集； - - OpenFGA 官方明确指出 `and` / `but not` 比 `or` 昂贵，ListObjects 性能随模型复杂度、 - tuple 数和结果规模显著变化，因此最终是否保留某一 ListObjects 入口由实测决定。 -- **何时重新考虑**：Pinned OpenFGA 的真实 ListObjects/Check 证明该额外深度无法满足 - §7 性能门槛时，先比较“业务候选 + BatchCheck”与扁平化 execution projection； - 若两者都不达标，再评估按 action 物化的读模型或官方原生 role primitive，不能退回 - SQL/Config 第二 PDP。 +- **查询策略**：具体业务 action 保持三层交集并使用单资源 Check 或有界 BatchCheck；固定 + `visible` 按决策 13 物化为派生执行投影，避免完整可见枚举反向遍历全部 Grant/model/catalog。 + 业务列表使用哪种顺序不再由本决策统一规定，按决策 14 的业务数据含义选择。 +- **何时重新考虑**：Pinned OpenFGA 的真实 action Check/BatchCheck 不能满足 §7 门槛时， + 再评估按高频 action 增加受控执行投影；不能退回 SQL/Config 第二 PDP，也不能因为 + `visible` 已展平就默认展平全部动作。 ### 决策 3：普通与受保护 assignee 在同一个 Grant 中分 relation @@ -244,7 +258,7 @@ MySQL/DM8 中可审计、可版本化的 Action/Model/Grant 控制面，以及 ProjectionService 写入 Redis recent-change marker，在 server cache TTL+5s 内 Check/List 使用 `HIGHER_CONSISTENCY`；marker 读取失败也用 higher consistency。 - **选定**：C。 -- **原因**：失效点集中，不把缓存钩子散到业务域；撤权/停用后没有旧 ALLOW，新增授权后 +- **原因**：失效点集中，不把缓存钩子散到业务域；撤权、绑定替换或模型动作收紧后没有旧 ALLOW，新增授权后 也没有旧 DENY 阻挡 read-after-write。 - **何时重新考虑**：新模型在真实负载下不达标；新增缓存也必须以 catalog release、 Authorization Model ID 和资源 projection version 为 key，并证明所有删除可感知。 @@ -381,6 +395,84 @@ reader 同时检查三个 marker scope：Catalog publish 用 global；tenant/dep - **何时重新考虑**：以后若平台提供经过审计的独立一次性 Job runner，可以由它封装同一 脚本入口；Alembic DDL-only 和“数据迁移不得在应用启动时自动执行”的边界不重新开放。 +### 决策 13：`visible` 使用单槽浅层执行投影，具体 action 保持规范 ReBAC 图 + +- **备选**： + - A. 保持 `resource → grant → model release → catalog` 的深层 `visible`，直接使用 + ListObjects。写入最少，但 BENCH-01 已证明大资源、小可见范围时反向枚举不可接受，且会 + 受服务端 deadline/结果上限影响。 + - B. 把每个有效来源直接投影到资源的单个浅层 `visible` relation，并用 source projection + 做跨来源引用计数；来源撤销/替换按单个绑定 operation 原子维护 action 与 visible delta。 + - C. 使用 A/B 双槽与全局 switch，允许未来一次批量政策变更先 staging 再原子切换，但所有 + 查询多一层 intersection，Grant-derived visible tuple 和常规写入约翻倍。 +- **选定**：B。 +- **原因**：任何定义有效、未删除模型下的既有 Grant 都意味着可见，`visible` 不需要在线反向 + 展开模型动作；inactive 只控制后续能否选择该模型，不参与现有 Grant 的授权图。删除模型前 + 已要求把每个 Grant/assignee 撤销或替换并完成对账，所以最终删除只是引用为零后的控制面 + 墓碑操作，不存在“删除时批量撤掉上万绑定”的原子切换需求。部门和用户组仍以 userset 投影， + 不按成员展开。SQL 的 Model/Grant/Assignee 和各 Owner 业务域事实仍是控制面真相，source + projection 与 OpenFGA tuple 只是不可独立编辑的派生执行索引，不能在 Check/List 失败后回退 + SQL 产生 ALLOW。 +- **写放大边界**：动作、等级、名称、同级策略和模型 active 变化都不重写 visible；新增、撤销 + 或替换一个 Grant/assignee 时，只维护该来源及同 key 聚合 tuple。部门/用户组主体保持 + `department#member/subtree_member`、`user_group#member/admin`，不与成员数相乘。 +- **一致性边界**：每个绑定撤销/替换都是独立、幂等、可审计的业务 operation，并在一个 + OpenFGA Write 中提交其 action 与 visible delta。批量清理可以逐绑定续跑，但模型删除持续被 + 引用计数和残留 checksum 阻断，不能在清理中途报告删除成功。 +- **何时重新考虑**：未来若产品明确要求“停用一个模型必须瞬时撤销其全部既有授权”，或存在 + 其他必须跨越 100-tuple 边界原子切换的全局可见政策，再单独评估版本化索引/A-B；不能在没有 + 该语义的情况下承担永久 switch 查询成本，也不能用数据库 ALLOW 替代 OpenFGA。 + +#### 顶层枚举资源与继承型子资源的边界 + +- `knowledge_space`、`knowledge_library`、workflow/assistant/tool/channel/dashboard 等需要完整 + 可见 ID 枚举的顶层资源，`visible` 由本地直接单槽与浅层 `system_visible` 组成；Grant 来源 + 直接投影到本级，public/shared 仍由显式系统 relation 计算，不进入 Grant source projection。 +- `folder`、`knowledge_file` 的列表采用业务候选优先，因此不把父资源授权展开到每个后代。 + 它们的 `visible` 是“本地直接单槽 + `inherit_mode` 下的 parent visible + 不受 mode gate 的 + system visible”。CUSTOM 子资源的本地 Grant 仍产生直接单槽，继承来源不产生子资源 source 行。 +- 该边界保证知识空间 `ListObjects` 不反向遍历 Grant/model，同时保留文件候选 BatchCheck 的 + 父级继承语义；禁止为了统一形态向海量 folder/file 扇出 visible tuple。 + +### 决策 14：列表先枚举可见 ID 还是先取业务候选，由数据分布与端到端成本决定 + +- **备选**： + - A. 所有列表都先 ListObjects。资源总量大、用户可见少时有利，但可见集合大或业务过滤 + 选择性强时会多枚举、多回表，并且不天然提供业务排序/分页。 + - B. 所有列表都先查业务候选再 BatchCheck。高可见率且业务候选可被索引强过滤时有利， + 但资源上万、用户只见少量时，为填一页会出现很大的候选扫描放大。 + - C. 按入口记录代表性实际分布，以同一业务过滤/排序/分页语义比较两条完整链路。 +- **选定**：C；不在运行时逐请求猜测路径，本次由每个入口的设计登记选择，数据分布显著变化后 + 重新压测和评审。 +- **统一测量口径**：至少记录业务候选总量 `N_db`、可见数 `V`、可见率 `p=V/N_db`、继承与 + `CUSTOM` 比例、过滤选择性、请求条数 `k`、候选优先实际扫描量与放大倍数、权限调用 + P50/P95/P99、业务 DB 回表/排序成本及完整性。比较： + + ```text + T_id_first = T_list_visible_complete + T_db_filter_sort_by_visible_ids + T_candidate_first = Σ(T_db_cursor_batch + T_batch_visible), until k items or exhausted + scan_amplification = scanned_candidates / max(returned_items, 1) + ``` + +- **入口初始选择**： + - `/knowledge/space/mine`：仍由知识空间 Owner 域按 canonical “本人创建”事实查询，不把它 + 改造成权限枚举入口；本次只删除根目录文件数和部门展示元数据装饰。 + - `/knowledge/space/joined`：目标语义是“完整可见集合中他人创建的知识空间”，预计采用可见 + ID 优先；权限集合完成后由知识空间 Repository 限定 tenant/ACTIVE/type、排除本人创建并按 + `update_time` 排序。`space_channel_member`、成员角色和管理员身份都不是候选必要条件； + “本人创建”复用 mine 的 canonical creator predicate，只作排除条件，不作权限 ALLOW。 + - `/knowledge/space/department`：先由组织/知识空间 Owner 域查询当前用户所属部门及有效空间 + 绑定得到小候选 ID 集，再 `batch_check_visible`，最后按通过的 ID 查询空间详情；不重复逐项 + visible、不返回 `manage_permission`、根目录文件数或部门展示元数据。若实际候选量不再小， + 按本决策重新比较 ID-first，而不是把当前顺序变成永久规则。 + - 知识空间文件/文件夹列表：预计采用业务候选优先；Repository 先按 parent、状态、关键字、 + 排序和稳定 cursor 取批次，再 BatchCheck 最终 `visible`。不足一页继续扫描，响应 cursor + 指向最后扫描的业务候选而非最后返回的可见项。 + - 上述“预计”必须由 §7 的代表性业务数据门禁确认；若结果相反，可调整路径但不得改变业务 + 过滤、排序、分页连续性和最终权限语义。 +- **何时重新考虑**：`N_db/V/p`、业务过滤选择性、继承/CUSTOM 比例或扫描放大跨越已登记容量 + 区间，或另一条路径连续三次同口径基准的 P95 总体成本低 20% 以上时，重新评审入口登记。 + --- ## 4. 系统现状与目标结构 @@ -411,6 +503,26 @@ reader 同时检查三个 marker scope：Catalog publish 用 global；tenant/dep | Platform `PermissionListTab.tsx` / Client 同名文件 | 593 / 664 行，按主体聚合旧档位；Client 已超 600 行 | 先拆 RosterTable/SourceBadge/ModeHeader，再接新契约 | | 两端 `PermissionGrantTab.tsx` | 提交 relation+model_id | 改稳定 model_key、assignee row、来源/版本契约 | +#### 4.1.1 2026-08-13 仓库实现状态、部署现状与本次差量 + +F048 代码虽已完成原版规范化 Catalog/Grant/Assignee、durable projection 和深层 Authorization +Model，但**尚未上线**；正式迁移仍只存在一次，源端仍是 §4.1 的旧 Config/四档 relation/ +creator/成员等事实，目标端必须直接生成本文最终单槽浅层 visible model，不存在“旧 F048 → successor +F048”的第二次生产迁移。本次尚未合入原 F048 实现的差量如下： + +| 当前代码事实 | 当前后果 | 本次目标 | +|---|---|---| +| `authorization_model_f048.py` 的 `visible` 仍从 resource 反向经过 Grant、ModelRelease/Catalog intersection，并叠加 mode/parent/system | 单资源 Check 可用，但大资源/小可见范围的 ListObjects 反向枚举很慢 | 决策13的单槽浅层 visible；具体 action 图不变 | +| `PermissionActionService.check_visible/batch_check_visible` 调 `_identity_shortcut` | super_admin/tenant_admin 对个人 visible 自动 ALLOW | visible 专用路径只保留 tenant fence，不做管理员扩权短路 | +| `list_action_objects` 明确拒绝 `visible`，管理员返回 `None`，底层调用普通非流式 ListObjects | 没有能证明完整性的 `list_visible_objects` 合同 | 完整消费 StreamedListObjects，正常结束/容量检查后才交付 | +| `KnowledgeSpaceService.get_my_followed_spaces` 先读 active non-creator membership；普通用户再由 `_scan_space_action_ids("visible")` 每 100 条扫描全部空间，最后 `_format_accessible_spaces` 重查 visible；管理员跳过扫描 | joined 同时依赖 membership、全量候选 BatchCheck、重复 visible 和管理员特例，资源上万时成本与语义均不符合 AC-172 | 先完整枚举 visible ID，再由业务库排除 canonical creator；不读 membership/role、不重复 Check | +| 原迁移脚本只生成深层 `visible` 所需的 Grant/model/resource 关系，没有 source projection 与浅层 tuple | 若只改目标 DSL/读入口，正式旧系统迁移后完整枚举没有数据，或 Check/枚举语义分叉 | 在同一个 `migrate_f048_permission_data.py` 正式 run 中随 Grant/assignee 映射同步生成单槽可见投影并纳入 D4 校验 | +| 116 BENCH-01 原型只在隔离 Store 测裸 direct relation，未进入 builder/runtime/业务库 | 只能证明引擎方向，不能作为已实现功能或生产性能结论 | §7 第二阶段验证目标 DSL 与 joined/department/file 端到端链路 | + +因此本文 §4.2 以后描述的是**待 Design ★ 后实现的目标状态**；原 F048 已实现部分继续作为 +基线、source projection 新表、25014、原迁移脚本的可见投影补充和列表入口都尚未编码。任何环境首次 +上线 F048 时都从旧系统直接迁到该目标状态，不发布中间的深层-visible F048 model。 + ### 4.2 启服后新执行链 权限模块的边界是**业务无关的授权计算与权限事实投影**。它不得 import 或调用 @@ -437,7 +549,7 @@ Repository 读取并验证 tenant/current owner/parent/type/status，再输出 只消费 DTO、旧权限事实与 Config，不把业务查询放入 permission domain。该 coordinator 不注册为应用运行时 Service。 -单资源业务动作链为： +单资源**具体业务动作**链为： ```text 业务 Endpoint @@ -458,6 +570,12 @@ Repository 读取并验证 tenant/current owner/parent/type/status，再输出 增加资源查询职责。任何 HTTP 请求中的 `tenant_id`、`resource_version` 都不能直接构造 target；必须由注册的业务 Service 产生。 +固定 `visible` 是个人内容范围，不是管理员管理范围。其单资源 Check/BatchCheck 和完整枚举 +均执行 tenant fence、Catalog/model pin、一致性选择与 OpenFGA 查询，但**不执行** +`super_admin/tenant_admin` 全量 ALLOW 短路；管理员若没有 Grant、组织、共享、public 或其他 +明确资源来源，个人列表中就不可见。平台/租户管理入口需要查看全量资源时，继续由独立的 +系统身份业务流程加载，并对后续具体管理动作按上面的 C4 顺序校验，不能借用 `visible` 伪装来源。 + 常规业务 Repository 仍只查当前 tenant。跨 tenant 的 F017 共享由 `ResourceShareService` / 各资源业务 Service 持有唯一例外： @@ -469,16 +587,28 @@ target；必须由注册的业务 Service 产生。 4. 它再构造 VerifiedPermissionTarget，并按需要调用具体 action Check； 5. PermissionService 不读取共享资源行、不解析资源状态，也不创建跨 tenant 普通 Grant。 -业务列表默认只有一种高频合法形态： +业务列表有两种同等合法的编排，入口必须按决策 14 登记一种： + +```mermaid +flowchart LR + A["可见 ID 优先"] --> B["OpenFGA 完整枚举 visible ID"] + B --> C["业务库按 ID + tenant/status/filter 查询"] + C --> D["业务排序与分页"] + E["业务候选优先"] --> F["业务库按 filter/sort/cursor 取有界批次"] + F --> G["OpenFGA BatchCheck visible"] + G --> H{"已填满一页或候选耗尽?"} + H -- "否" --> F + H -- "是" --> I["返回结果和最后扫描候选 cursor"] +``` -1. 业务 Repository 用 DM8-safe cursor、租户、状态、关键字等条件有界扫描候选； -2. 业务 Service 把一批 VerifiedPermissionTarget 交给 - `PermissionService.batch_check_actions()`； -3. 按具体 action 过滤并继续有界凑页，不 fetch-all。 +可见 ID 优先只允许调用 `PermissionService.list_visible_objects()`，它必须完整消费 +StreamedListObjects 并在正常终止后才返回 ID 集；deadline、取消、服务端错误或应用容量上限 +都返回明确错误，不能把已收到的前缀包装为成功。业务 Service 仍需按 tenant、状态、类型、 +关键字和其他业务条件回表；FGA 对象 ID 不能替代业务存在性或排序。 -只有资源规模、结果上限、超时和 nested intersection 已通过 BENCH-01 的入口，才可由业务 -Service 使用受限 `ListObjects(user, can_, type)` 或 streamed variant 获取授权 ID， -随后仍由业务 Service 加载和校验业务数据。不得用 `visible` 候选证明 +业务候选优先必须使用 DM8-safe 稳定 cursor 与有界批次，将业务侧已验证 target 交给 +`batch_check_visible()`；若本批不足一页则继续扫描，不能 fetch-all，也不能把“本批没有更多 +可见项”误报为业务候选已耗尽。两条路径都只证明 `visible`，不得证明 `edit/download/use/...`，不得在 FGA 结果后读取 SQL 模型补 ALLOW。 ### 4.3 初始动作 Catalog @@ -530,11 +660,16 @@ CatalogService 的生成规则固定为；每次 action 的 level/active/scope - 每个自定义模型保留管理员显式选择；过滤 inactive/unassigned/越 scope 动作后， 等级重新取剩余动作最高 level，不按等级补动作； - action scope 在每个具体资源 relation 上取交集；模型在目标资源上没有任何适用 action - 时，该 Grant 仍可通过 active model 产生 `visible`，但不产生任何可配置业务 action； -- action 变为 inactive/unassigned 后，引用它的 active 自定义模型必须在 draft 中修复或 - 停用；不能把空的 effective model 发布为 active。 + 时，该 Grant 仍产生 `visible`，但不产生任何可配置业务 action； +- `permission_model.active` 只在新增/变更 Grant 的 command 校验中表示“可分配”。它不进入 + OpenFGA 运行时 action/visible 图；停用前的既有 Grant 继续引用当前 Catalog 中该模型定义。 + 对 inactive 模型修改动作仍会影响既有 Grant，必须经过与 active 模型相同的 impact、确认和 + Catalog 原子发布； +- action 变为 inactive/unassigned 后，若自定义模型有效动作变空，有引用时可作为 inactive + 的仅可见模型保留以便逐步撤销/替换；无引用时应直接删除或修复。空模型不得设为 active， + 也不得用于新增授权。 -对任一 active 模型 `m`： +对任一当前 Catalog 中定义有效且未删除的模型 `m`： ```text effective_actions(m, resource_type) @@ -558,7 +693,7 @@ Catalog projection 据此写 `can_grant_level_1..4` marker。resource 的 grant- ### 4.4 OpenFGA 新关系骨架 以下 DSL 是规范骨架；实际 Python JSON builder 必须为每个具体资源类型展开合法 relation， -并由 OpenFGA model tests 锁定。为简洁只展示 `edit` 和 grant-level： +并由 OpenFGA model tests 锁定。为简洁只展示 `edit`、grant-level 和浅层可见关系： ```fga model @@ -576,14 +711,13 @@ type permission_model_release define enabled_marker: [user:*] define edit_marker: [user:*] define grant_level_1_marker: [user:*] - define active: enabled_marker and active from catalog - define can_edit: active and edit_marker - define can_grant_level_1: active and grant_level_1_marker + define published: enabled_marker and active from catalog + define can_edit: published and edit_marker + define can_grant_level_1: published and grant_level_1_marker type permission_model relations define release: [permission_model_release] - define active: active from release define can_edit: can_edit from release define can_grant_level_1: can_grant_level_1 from release @@ -600,8 +734,6 @@ type permission_grant define ordinary_assignee: [user, department#member, department#subtree_member, user_group#member, user_group#admin] define protected_assignee: [user] - define ordinary_visible: ordinary_assignee and active from model - define protected_visible: protected_assignee and active from model define ordinary_can_edit: ordinary_assignee and can_edit from model define protected_can_edit: protected_assignee and can_edit from model define ordinary_can_grant_level_1: @@ -610,6 +742,11 @@ type permission_grant protected_assignee and can_grant_level_1 from model ``` +`permission_grant` 不再作为 `visible` 的在线反向枚举路径；它继续是具体 action、可授等级和 +来源解释的规范 ReBAC 图。VisibilityProjectionCompiler 为每个定义有效、未删除模型下的有效来源生成 +资源本级展平 relation，保留 `user`、`department#member/subtree_member`、 +`user_group#member/admin` 等 projected subject，不把 userset 展开成用户列表。 + resource helper 展开为；实际 DSL 为每个 action 建立中间 relation，以满足 schema 1.1 对 union/intersection 的结构要求： @@ -619,10 +756,14 @@ permission_enabled= [user:*] # 创建最终启用、删除首先撤 custom_mode = [user:*] inherit_mode = [user:*] # 仅 folder/file parent = [canonical parent] # 始终 mirror 业务直接父级 +ordinary_visible = [user, department#member, department#subtree_member, + user_group#member, user_group#admin] +protected_visible = [user] +explicit_system_visibility = [user, user:*, tenant#member] ordinary_or_protected_visible = - protected_visible FROM grant - OR (ordinary_visible FROM grant AND custom_mode) + protected_visible + OR (ordinary_visible AND custom_mode) OR (visible FROM parent AND inherit_mode) system_visible = @@ -663,6 +804,11 @@ can_grant_level_ = permission_enabled AND ( - dashboard 使用与其他本期顶级资源相同的 Grant/action 骨架；只有 llm_server、llm_model 保留 allowlist 内旧 relation，首批业务动作 API 不接受这两类。 +系统/public/shared 资源关系本身已经是浅层直接来源，不进入 Grant-derived source projection； +它们继续经 `explicit_system_visibility` 和 parent 传播参与同一个 `visible` 结果。 +父级继承也不对子孙物化，避免父空间一次授权按目录/文件数量 fan-out。顶级 knowledge_space +没有 parent，完整枚举的主路径因此是资源本级展平 Grant 来源与本级 system 来源。 + `permission_enabled` 是资源生命周期 fence：新资源先离线 staging mode、protected owner、 parent 和 system tuples，最后写一个 marker 才可见；删除先原子撤销 marker，再分批清理。 因此资源的 Grant 数超过单次 Write 上限时也不会留下部分可访问状态。 @@ -683,7 +829,34 @@ commit: WRITE user:* active permission_catalog_release: ``` -最后两条在同一个 OpenFGA Write 中提交，因此没有新旧 Catalog release 并行 ALLOW 的窗口。 +Catalog commit 始终为上述 2 tuple。动作、等级、模型定义和可分配状态变化不修改浅层 visible； +其中模型定义变化通过 Catalog 切换原子影响既有 Grant 的具体 action，模型 active 变化只更新 +SQL 控制面的新授权校验状态。只有 Grant/assignee 来源新增、撤销、替换或资源 mode/lifecycle +变化时，才通过对应 projection operation 修改 visible tuple。 + +```mermaid +sequenceDiagram + participant API as "Grant/Model API" + participant SQL as "SQL Control Plane/Ledger" + participant FGA as "OpenFGA" + opt "可选：先停用，阻止新增绑定" + API->>SQL: "停用模型：CAS active=false" + SQL-->>API: "已有 Grant/visible 不变" + end + loop "先逐项撤销或替换全部绑定" + API->>SQL: "提交独立、幂等的 binding operation" + SQL->>FGA: "同一 Write 提交 action + 单槽 visible delta" + FGA-->>SQL: "higher-consistency 校验后 finalize" + end + API->>SQL: "删除模型：检查绑定引用与残留 checksum" + alt "仍有引用或残留" + SQL-->>API: "25004：继续清理或 reconcile" + else "引用与残留均为零" + SQL->>FGA: "发布不再包含该模型的新 Catalog" + SQL->>SQL: "当前 Catalog 删除模型；历史快照保留审计" + SQL-->>API: "删除成功" + end +``` ### 4.5 SQL 数据结构 @@ -704,10 +877,10 @@ tuple fingerprint 固定为 | 表 | 关键字段 | 约束 / 用途 | |---|---|---| -| `permission_catalog_release` | `id`, `release_key`, `version`, `status`, `write_fenced`, `predecessor_id`, `required_authorization_model_release_id`, `draft_owner_id`, `idempotency_key`, `projection_checkpoint`, `expires_at`, `published_at`, `checksum`, `commit_checksum` | `release_key`、`version`、`idempotency_key` 唯一；DRAFT/PROJECTING/COMMITTED/CURRENT/RETIRED/FAILED_CLOSED | +| `permission_catalog_release` | `id`, `release_key`, `version`, `status`, `write_fenced`, `predecessor_id`, `required_authorization_model_release_id`, `draft_owner_id`, `idempotency_key`, `projection_checkpoint`, `expires_at`, `published_at`, `checksum`, `commit_checksum` | `release_key`、`version`、`idempotency_key` 唯一；DRAFT/PROJECTING/COMMITTED/CURRENT/RETIRED/FAILED_CLOSED；Catalog 不记录 visible 槽位，source projection 单独对账 | | `permission_action` | `id`, `catalog_release_id`, `code`, `name`, `level nullable`, `active`, `sort_order` | unique(release, code)；null level=未分配 | | `permission_action_resource_scope` | `id`, `action_id`, `resource_type` | unique(action, resource_type) | -| `permission_model` | `id`, `catalog_release_id`, `model_key`, `normalized_name`, `name`, `kind`, `config_scope`, `derived_level nullable`, `active`, `allow_same_level`, `legacy_source_key` | unique(release, model_key/name)；`config_scope=PLATFORM`；标准字段由 Service 固定 | +| `permission_model` | `id`, `catalog_release_id`, `model_key`, `normalized_name`, `name`, `kind`, `config_scope`, `derived_level nullable`, `active`, `allow_same_level`, `legacy_source_key` | unique(release, model_key/name)；`config_scope=PLATFORM`；`active` 只表示可用于新增/变更授权，不影响既有 Grant；标准字段由 Service 固定 | | `permission_model_action` | `id`, `model_id`, `action_id` | unique(model, action) | | `permission_catalog_projection_tuple` | `id`, `catalog_release_id`, `phase`, `sequence`, `action`, `fga_user`, `relation`, `fga_object`, `tuple_fingerprint`, `status` | unique(release, phase, tuple_fingerprint)；全局 Catalog staging 的可恢复逐项日志 | @@ -721,9 +894,10 @@ Authorization Model 已注册 action 的 level/active；代码引入新 action | 表 | 关键字段 | 约束 / 用途 | |---|---|---| -| `permission_grant` | `id`, `tenant_id`, `resource_type`, `resource_id`, `model_key`, `state`, `version`, `projection_state` | unique(tenant, resource_type, resource_id, model_key)；停用后复用同一逻辑 Grant | +| `permission_grant` | `id`, `tenant_id`, `resource_type`, `resource_id`, `model_key`, `state`, `version`, `projection_state` | unique(tenant, resource_type, resource_id, model_key)；模型停用不改变已有 Grant state；删除模型前引用必须为零 | | `permission_grant_assignee` | `id`, `tenant_id`, `grant_id`, `subject_type`, `subject_id`, `userset_relation`, `include_children`, `source_type`, `source_ref`, `source_locator`, `source_fingerprint`, `projected_subject`, `protected`, `state`, `version` | unique(tenant, grant, source_fingerprint)；来源独立，客户端不能提交 source/protected | | `resource_permission_mode` | `id`, `tenant_id`, `resource_type`, `resource_id`, `mode`, `parent_type`, `parent_id`, `version`, `projection_state`, `operation_id` | unique(tenant, resource_type, resource_id)；parent 是 canonical mirror | +| `permission_visible_source_projection` | `id`, `tenant_id`, `resource_type`, `resource_id`, `visibility_class`, `projected_subject`, `source_kind`, `source_owner_key`, `source_locator`, `source_fingerprint`, `contribution_fingerprint`, `model_key nullable`, `source_version`, `tuple_fingerprint`, `state`, `operation_id nullable`, `migration_item_id nullable` | unique(tenant, resource_type, resource_id, visibility_class, projected_subject, contribution_fingerprint)；由 canonical 来源派生的单槽当前执行索引，只供编译、引用计数、迁移追溯、对账与清理，不参与 ALLOW | `source_locator` 是服务端规范化的非空自然键，例如： @@ -742,9 +916,19 @@ snapshot:folder:300:assignee:891 物理上由 `permission_grant_assignee.protected=true` 表达，不再建一张会与 assignee 漂移的重复表。 -相同 OpenFGA subject 可能由多个 source row 投影；删除 tuple 前必须按 -`(grant_id, protected, projected_subject)` 做引用计数，只有最后一个 active source -消失才删除。 +可见 projection 不能只复用 `source_fingerprint`：同一个 `direct:user:42` 可以同时存在于 +viewer 和 editor 两个 Grant。`source_owner_key` 固定为 canonical owner（例如 +`grant_assignee:`、`share:`、`public_policy:`）， +`contribution_fingerprint=sha256(source_kind + "\0" + source_owner_key + "\0" + +source_fingerprint + "\0" + coalesce(model_key, ""))`。这样模型/Grant 贡献可分别退役，而聚合 +tuple 仍只在最后一个 contribution 消失时删除。 + +相同 OpenFGA subject 可能由多个 Grant、模型或 source row 投影到同一资源 tuple。具体 action +仍按 `(grant_id, protected, projected_subject)` 引用计数；展平可见 tuple 则必须按 +`(tenant, resource_type, resource_id, visibility_class, projected_subject)` +聚合 `permission_visible_source_projection`。撤销来源时先把 after-state 与 operation 同事务冻结， +只有最后一个 active contribution 消失才删除对应 OpenFGA tuple。该表禁止业务 API +直接编辑，也禁止 PermissionService 用它补 ALLOW；其内容可随 canonical 来源重建。 #### 4.5.3 投影、Authorization Model 与迁移表 @@ -769,6 +953,9 @@ MigrationRun/Item 只允许专用平台运维 Repository 访问；该 Repository - Grant 列表：`(tenant_id, resource_type, resource_id, state, id)`； - assignee 解释：`(tenant_id, grant_id, state, id)`、`(tenant_id, subject_type, subject_id, state)`； +- 可见来源聚合：`(tenant_id, resource_type, resource_id, visibility_class, + projected_subject, state)`、`(model_key, state, tenant_id, id)`；索引只服务投影编译/对账， + 不作为线上权限查询； - mode：唯一键外加 `(tenant_id, projection_state, update_time)`； - projection retry：`(status, update_time, id)`； - migration resume：`(run_id, status, id)`、`(run_id, severity, difference_type)`； @@ -803,11 +990,11 @@ DRAFT(SQL 完整快照) → 重算 impact；与用户确认的 checksum 不同则释放 fence 并返回 EXPIRED → PROJECTING（release 本身是 durable operation head） → 分批 staging model releases（不 active） - → OpenFGA model tests + impact checksum + → OpenFGA model tests + action impact checksum → 预置 global recent-change marker；失败则不 commit → 原子 active old→new（COMMITTED） - → SQL CURRENT + old RETIRED（FINALIZED） - → 保留 marker 至 OpenFGA cache TTL+5s，释放 publish fence + → SQL CURRENT + old RETIRED + → FINALIZED；保留 marker 至 OpenFGA cache TTL+5s，释放 publish fence ``` 每个 tenant 权限写在 prepare 事务中短暂锁 current release，只有 @@ -815,30 +1002,40 @@ DRAFT(SQL 完整快照) 新 operation 不存在检查后插入的竞态。commit 前失败会清除旧 current 的 fence； commit 后直到 finalize 都保持 fence，finalize 时新 current 才解除。 -若在 commit 前失败，旧 Catalog 完整有效；若 commit 后进程崩溃，reconciler 读 active tuple -确认 checksum 后 finalize。COMMITTED 已是完整的新执行策略，runtime 决策不依赖 SQL -status 补充 ALLOW；但所有权限写持续被 fence 拒绝，直到 SQL finalize。未知状态标 +若在 commit 前失败，旧 Catalog 完整有效；若 commit 后进程崩溃，reconciler 读 Catalog +active 与 checksum 后继续 finalize。COMMITTED 已是完整的新 action 定义；浅层 visible 不随 +Catalog active 或模型可分配状态改变。runtime 决策不依赖 SQL status 补充 ALLOW；但所有 +权限写持续被 fence 拒绝，直到 SQL finalize。未知状态标 `FAILED_CLOSED` 并冻结新 Catalog/Grant/mode/resource 写，不反向猜测。 -commit timeout 后以 higher consistency 读取 old/new active：仅 new active 视为 committed， -仅 old active 才可重试原 commit；两者同时或都不 active 属于不变量破坏，保持全局 fence -并人工修复。 +commit timeout 后以 higher consistency 读取 old/new active：保持 before 才可重试，达到 after +才确认 committed；其他状态保持全局 fence 并人工修复，绝不能任选一侧作为真相。 + +模型 active 变化不处理任何既有 Grant/Assignee 或 visible tuple。`DELETE_MODEL` 不要求先 +inactive，但要求有效、待处理和失败恢复中的 Grant/assignee 引用全部为零；最终删除前比较 +canonical 空引用、单槽 source projection 与 live FGA tuple checksum。任一残留先进入 +reconcile，未清零不得成功删除；最终删除通过新 Catalog 不再包含该 `model_key` 生效，旧 +Catalog/ModelRelease 作为 RETIRED 审计快照保留，不物理改写历史。最终删除不承担批量撤权 +或投影清理。 #### Grant mutation 1. 接收业务 Service 已生成的 VerifiedPermissionTarget；校验其 tenant/context version、 - 当前 Catalog、target model active 与 `expected_resource_version`。direct user 是否为 + 当前 Catalog 与 `expected_resource_version`。ADD/MOVE 的 target model 必须 active；REMOVE + 允许精确撤销 inactive model 上的既有 assignee，以便引用清零和最终删除。direct user 是否为 active member、department/group 是否同 tenant 由 Tenant/Department/UserGroup 业务 Service 先验证并生成 canonical subject；权限模块只接受该服务端结果和 userset allowlist， 不回查业务表； 2. 对普通用户以 OpenFGA `can_manage_permission` + `can_grant_level_n` 重校验； -3. SQL 事务写 PENDING rows + operation/tuple delta； -4. 计算规范化 tuple delta；预置 resource recent-change marker，预置失败则不写 FGA； +3. SQL 事务写 PENDING rows + operation/tuple delta，并写入单槽 after-state 的 + `permission_visible_source_projection`；inactive model 的既有 binding 仍产生 contribution； +4. 按跨 Grant/source 聚合引用计数计算规范化 action + 单槽 visible tuple delta；预置 + resource recent-change marker，预置失败则不写 FGA； 5. 一次 OpenFGA Write 提交 add/move/revoke； 6. higher-consistency Check 验证安全后果； 7. SQL finalize、写审计，marker 保留到 OpenFGA cache TTL+5s 后自然过期，返回新 version。 -HTTP 每次最多 50 个 change item；编译后的 `writes + deletes` 必须 ≤90，为 Grant link、 -model link 和服务端安全 tuple 预留余量。最终以编译结果为准，超限返回 25013，不能静默 +HTTP 每次最多 50 个 change item；单槽 visible tuple 也计入编译后的 `writes + deletes`， +总量必须 ≤90，为 Grant link、model link 和服务端安全 tuple 预留余量。最终以编译结果为准，超限返回 25013，不能静默 分批返回部分成功。客户端超时以同一 idempotency key 重试，不得换 key 猜测状态。 FGA Write 超时属于 `COMMIT_UNKNOWN`：reconciler 用 higher consistency 对比 operation @@ -863,7 +1060,8 @@ dry-run 必须先核对 ledger、CURRENT Catalog 的 Store/model pin 与 scope f protected 属性复制下来；public/shared 等 system 来源继续由 system relation 传播， 不进入快照。若 subject+model 与本资源 protected assignee 重合，只保留本资源 protected 来源；staging 先以 higher consistency 识别往返切换保留的同 key tuple，只补写 - 尚未达到 after state 的 tuple，完成后一个 Write 删除 `inherit_mode`、写 `custom_mode`。 + 尚未达到 after state 的 action tuple 和单槽展平 visible tuple，完成后一个 Write 删除 + `inherit_mode`、写 `custom_mode`。 - `CUSTOM→INHERIT`：确认合法直接 parent；一个 Write 删除 `custom_mode`、写 `inherit_mode`；本级 ordinary assignee 立即不生效，随后逐 source 标 RETIRED 并清理。 同一 Grant 若仍有 protected source 必须保持 ACTIVE/link；只有最后一个 active source @@ -886,7 +1084,7 @@ dry-run 必须先核对 ledger、CURRENT Catalog 的 Store/model pin 与 scope f - COPY(INHERIT) 在新 parent 下创建 INHERIT；COPY(CUSTOM) 复制 ordinary source rows， protected owner 按新资源规则创建。 - delete 先撤销 `permission_enabled` 并禁止新 mutation，再分批清 - resource→grant/mode/system tuples，由 Owner Feature 完成资源删除；失败保留可恢复 + resource→grant/mode/system/单槽可见 tuples 与 source projection，由 Owner Feature 完成资源删除；失败保留可恢复 operation，资源保持 fail closed，不留下可授权的悬空 Grant。 - department create/move/archive/member 变更由 DepartmentService 校验业务规则后调 projection ledger； 手工维护、SSO/F015 全量同步和组织 reconcile 都必须在**同一个 SQL 事务**内写业务状态并 @@ -903,9 +1101,10 @@ dry-run 必须先核对 ledger、CURRENT Catalog 的 Store/model pin 与 scope f | `permission/domain/repositories/` | Catalog/Grant/mode/projection 持久化，以及供专用数据脚本调用的 run/item Repository | 不判权限；不主动扫描业务表或自动启动数据迁移 | | `permission/domain/services/catalog_service.py` | draft、影响分析、标准模型生成、Catalog publish 编排 | 不直接 ORM/FGA HTTP | | `.../projection_service.py` | operation、tuple phase、幂等提交、reconcile、recent marker | 不决定谁可授权 | +| `.../visibility_projection_service.py` | 从 canonical Grant assignee 编译单槽浅层 visible contribution、跨来源引用计数、checksum/reconcile；迁移器与运行时复用同一纯编译器 | 不展开部门/用户组成员；system/public/shared 仍由各 Owner projection 处理；不接受业务编辑；不参与 Check/List fallback | | `.../grant_service.py` | 来源、模型、版本、protected 和同级规则编排 | 不读取 Config | | `.../mode_service.py` | 已验证 parent/mode context 的 impact/snapshot/projection | 不查询或移动业务资源，不维护第二 parent | -| `.../permission_service.py` | 唯一权限 facade：check/batch-check/受限 list、Grant 与 projection | 不查询业务资源，不判断状态/父级，不 fallback ALLOW | +| `.../permission_service.py` | 唯一权限 facade：具体 action check/batch-check、visible check/batch-check/完整流式枚举、Grant 与 projection | 不查询业务资源，不判断状态/父级，不 fallback ALLOW；个人 visible 不做管理员扩权短路 | | `.../permission_explain_service.py` | 已经 FGA 判定后的 opaque subject/source/model ID、scope 与 protected 说明 | 不产生 ALLOW，不查询用户/部门/用户组/资源名称 | | `permission/application/resource_permission_coordinator.py` | 接收业务侧 VerifiedPermissionTarget，编排通用权限 HTTP 契约；通过显式 identity display port 批量补充展示名 | 不直接调用业务 Repository；target 不能来自客户端字段；展示名不参与授权 | | 各资源 `*Service` / `ResourceAuthorizationPort` | 查询资源、校验 tenant/status/parent/业务规则，生成 verified target，调用权限 facade | 不解析 Catalog/Grant，不二次 ALLOW | @@ -918,7 +1117,7 @@ dry-run 必须先核对 ledger、CURRENT Catalog 的 Store/model pin 与 scope f | `department_change_handler.py` | 新模型的 parent+child 对称 tuple operation | 不展开部门用户/资源 Grant | | `telemetry_search/.../DashboardService` | dashboard 业务加载、preset/status/组件规则、verified target 与 lifecycle projection | 不再调旧 AccessType/DASHBOARD relation | | `core/database/alembic/versions/.py` | 新表、列、索引、约束等 MySQL/DM8 schema DDL | 不读取/转换/回填/清理业务数据，不访问 Config/OpenFGA，不 import 数据迁移脚本 | -| `scripts/migrate_f048_permission_data.py` | 唯一数据迁移入口；初始化 app context，执行 source validation→publish-new-model→控制面/tuple 转换→retire legacy→checkpoint；`migrate` 必须显式 `--apply` | 不创建/修改 schema，不注册 API/Celery/startup，不直接使用业务 ORM，不提供 inventory/dry-run/rollback | +| `scripts/migrate_f048_permission_data.py` | 唯一数据迁移入口；初始化 app context，执行 old source validation→publish 最终单槽浅层 visible model→控制面/Grant/可见 source projection/tuple 转换→retire legacy→checkpoint；`migrate` 必须显式 `--apply` | 不创建/修改 schema，不注册 API/Celery/startup，不直接使用业务 ORM，不提供 inventory/dry-run/rollback，不发布中间深层-visible F048 model | | `scripts/README.md` | 记录运维停流/暂停任务、schema head、容器内正式命令、verify、重启、退出码、恢复流量和失败续跑方式 | 不保存凭据或生产数据 | | permission notification adapter | 从 assignee/source/model 生成通知 | 不再解析 relation/binding | | Platform permission components | Catalog 配置、资源授权 | store 不直接 HTTP | @@ -960,9 +1159,14 @@ domain 对业务 model/repository 的 import 必须由架构测试禁止。正 | 21 | action code 对应不可变 Authorization Model relation | 只在 SQL 新增 action 会得到永远 DENY 或非法查询 | `CatalogService.validate_required_model_release` | | 22 | F017 完整行需要窄范围跨 tenant 读取 | 普通 ORM auto-filter 会误挡共享；裸 bypass 会泄露 | `SharedResourceRepository.get_authorized_by_ids` | | 23 | 资源清理可能超过 100 tuple | 直接分批删除会保留部分 ALLOW | `PermissionService.delete_resource_boundary` | -| 24 | 新模型的 ListObjects 路径含 Grant/model/catalog intersection | 把它用于所有列表会放大延迟、触发 1,000 上限或拖慢 Check | business cursor + `batch_check_actions`，BENCH-01 单独测 ListObjects | +| 24 | 具体 action 的 ListObjects 仍含 Grant/model/catalog intersection；固定 visible 则改为单槽浅层投影 | 因 visible 已优化就把所有 action 列表都改成完整枚举，仍会放大延迟或不完整 | 具体 action 保持 Check/BatchCheck；`list_visible_objects` 独立 BENCH/allowlist | | 25 | Alembic revision 在 API 启动前执行且必须 DDL-only；F048 数据转换是独立 scripts 作业 | 把回填/Config 扫描/FGA 写入塞进 `upgrade()` 会阻塞或破坏启动，且无法按 run/checkpoint 续跑；放进 lifespan/Celery 又可能多实例并发 | `core/database/alembic/versions/.py` + `scripts/migrate_f048_permission_data.py` + `scripts/README.md` | | 26 | SSO/F015 部门同步原先先提交业务行，再 best-effort 更新 FGA；归档恢复到同一 parent 时字段 diff 还是空 | 进程崩溃会留下无 ledger 的永久 SQL/FGA 分叉；恢复行继续缺 parent/child mirror | `DepartmentTopologyProjectionService.aupsert_synced_department/aarchive_synced_department` + durable scope tests | +| 27 | OpenFGA 普通 ListObjects 可受 deadline 和最大结果数限制；HTTP 200 不等价于业务全集 | joined 等可见 ID 优先列表会静默漏空间 | `FGAClient.stream_list_objects` + `PermissionService.list_visible_objects` 正常终止/容量断言 | +| 28 | 展平 tuple 不携带 Grant/model/source ID，同一 subject-resource 可由多个来源贡献 | 撤销一个来源时误删其他仍有效来源的可见性，或删除模型后留下无来源 tuple | `permission_visible_source_projection` + 跨来源引用计数/reconcile | +| 29 | 模型 active 被误当成运行时撤权开关会制造资源规模 fan-out | 停用一个广泛使用的模型时产生不必要的大批 tuple 清理和复杂原子切换 | active 只在 Grant command 中控制可分配性；已有 Grant 不变，删除前逐绑定清零并由引用门禁阻断最终删除 | +| 30 | 管理员管理范围与个人内容可见范围不是同一业务语义 | super_admin 身份把“我加入的”扩成平台全量 | visible 专用 facade 禁止 identity shortcut；具体 action 保持 C4 顺序 | +| 31 | 列表路径成本取决于候选规模、可见率、业务过滤和继承比例 | 把 joined 的 ID-first 或文件的 candidate-first 机械推广到所有列表 | §3 决策14入口登记 + BENCH-01 业务链路阶段 | --- @@ -1117,10 +1321,12 @@ idempotency_key + confirmed=true`，不接受客户端重新提交影响数字 Catalog change 只允许 `ASSIGN_ACTION_LEVEL`、`SET_ACTION_ACTIVE`、`CREATE_MODEL`、 `UPDATE_MODEL`、`SET_MODEL_ACTIVE`、`DELETE_MODEL`、`SET_ALLOW_SAME_LEVEL`；不存在 -`CREATE_ACTION_CODE`。`DELETE_MODEL` 仅允许 inactive custom 且跨 tenant Grant 引用为零。 -标准模型动作由 level 重新生成，自定义模型只保留显式动作。任何 active 自定义模型因 -动作停用/未分级而变成空集，或任一 Grant 上模型与资源没有可用 action，都是 publish -blocker。模型不含 `manage_permission` 时提交 `allow_same_level=true` 直接拒绝。 +`CREATE_ACTION_CODE`。`DELETE_MODEL` 只允许 custom 且跨 tenant 的有效、待处理和失败恢复中 +Grant/assignee 引用均为零，并要求 source projection/live tuple 残留为零；不强制先 inactive。 +标准模型动作由 level 重新生成，自定义模型只保留显式动作。active 自定义模型因动作停用/ +未分级而变成空集时必须在同一 draft 中修复或设为 inactive；若已有引用，inactive 空模型仅 +保留可见性以便逐项替换/撤销。模型在某资源上没有适用 action 不阻断该资源的仅可见 Grant。 +模型不含 `manage_permission` 时提交 `allow_same_level=true` 直接拒绝。 预设仅是 API 返回的初始化动作集合，不存“预设→模型”运行时关联。 #### Frontend interaction @@ -1129,6 +1335,9 @@ blocker。模型不含 `manage_permission` 时提交 `allow_same_level=true` 直 保存先展示动作/标准模型/自定义模型与 Grant 影响，确认后才 publish； - 标准模型名称、等级、动作、删除控件只读；仅模型实际包含 `manage_permission` 时显示 allow_same_level；自定义模型编辑器实时用服务端同一纯函数预览派生等级； +- inactive 模型不出现在新增/变更授权的目标模型选择器中，但引用它的既有成员行仍展示原模型、 + 可见性和具体动作，并允许有权用户把该行 MOVE 到 active 模型或 REMOVE；若该既有模型包含 + `manage_permission`，其授权能力也继续有效，不能由前端擅自隐藏； - Platform 与 Client 的资源弹窗共用 response schema，但保持各自 Zustand/Recoil 和 request wrapper；INHERIT 行只读，CUSTOM 行依据 `editable`，protected 始终锁定； - 没有 `can_manage_permission` 的用户不请求 roster，只请求 my-permissions；前端不靠 @@ -1142,7 +1351,10 @@ blocker。模型不含 `manage_permission` 时提交 `allow_same_level=true` 直 |---|---|---| | `PermissionService.check_action(target, action, actor)` | 已加载资源的业务 Service | 唯一具体动作入口；target 必须是服务端 verified DTO | | `PermissionService.batch_check_actions(targets, action, actor)` | 业务列表/按钮装饰 | 只检查调用方给出的有界 target；不加载候选 | -| `PermissionService.list_action_objects(type, action, actor)` | 仅 BENCH-01 批准的业务 Service | 返回授权 ID；不加载业务数据、不保证通用分页 | +| `PermissionService.check_visible(target, actor)` | 详情/单资源可见判断 | 与可见枚举同一 relation/pin/consistency；不执行管理员身份扩权短路 | +| `PermissionService.batch_check_visible(targets, actor)` | 候选优先业务列表 | 有界 BatchCheck；不加载候选；不因管理员身份扩权 | +| `PermissionService.list_visible_objects(type, actor, max_results)` | 经入口登记的可见 ID 优先列表 | 完整消费 StreamedListObjects，正常终止后一次性交付去重 ID；超时、取消、非正常结束或超过容量明确失败；不加载业务数据 | +| `PermissionService.list_action_objects(type, action, actor)` | 仅以后经专项 BENCH 批准的非 visible 场景 | 返回具体 action 授权 ID；本次业务列表不新增调用方 | | `PermissionService.authorize(target, projection)` | 各 Owner Feature 创建流程 | 保留 C4 统一入口；只投影 mode/protected/parent/system + enable | | `PermissionService.project_parent_change(target, old_parent, new_parent)` | folder/file Owner Service | 业务侧先完成移动合法性与 parent context 校验 | | `PermissionService.project_copy(target, source_permission_snapshot)` | copy Owner Service | 只按 verified snapshot 投影 ordinary/protected | @@ -1151,6 +1363,28 @@ blocker。模型不含 `manage_permission` 时提交 `allow_same_level=true` 直 | `PermissionDataMigrationRepository` | `migrate_f048_permission_data.py` only | 持久化 run/item/checkpoint；不注册到 HTTP、worker 或 startup | | `FGAClient.for_model(target_model_id)` | data migration script / release bootstrap only | 始终使用现有 Store；只为新 tuple Write/目标 Check 构造新 model client，source model ID 只进审计，不构造 legacy/dual client | +`list_visible_objects` 不是公开 HTTP，也不返回异步迭代器给业务层：它在权限边界内去重并只在 +stream 正常结束后交付不可变 ID 集，防止调用方提前消费不完整前缀。每个业务入口必须显式传 +经过 BENCH 审核的 `max_results`；`joined` 首发容量定为 5,000 个可见知识空间，超过即返回 +25014 并告警，不退回全租户候选扫描。业务 Repository 的 `IN` 参数分块上限为 500，最终按 +同一 `order_by` 做稳定归并；BENCH-01 必须同时验证 MySQL/DM8 结果一致。若实际用户持续接近 +该容量，应先设计 permission-aware cursor/索引连接，不得简单提高到无界内存集合。 + +知识空间 `joined` 的完整内部调用链固定为： + +```text +KnowledgeSpaceService.get_joined_spaces(actor, order_by) + → PermissionService.list_visible_objects("knowledge_space", actor, max_results=5000) + → KnowledgeSpaceRepository.list_by_visible_ids( + ids, tenant=current, status=ACTIVE, type=SPACE, + exclude_creator=canonical_creator(actor), order_by=validated_order) + → 返回空间基础列表 +``` + +该链不读 `space_channel_member`、不计算成员角色、不重复 `check_visible`，也不装饰 +`manage_permission`、根目录文件数或部门元数据。主动订阅只有在其 canonical 来源已经投影出 +有效 visible 时才出现；SQL 订阅行本身不能绕过 OpenFGA 补 ALLOW。 + 旧 `FineGrainedPermissionService.has_any_permission_async(permission_ids=...)`、 `PermissionService.check(relation=...)` 对首批类型在新模式不可调用；架构测试维护 allowlist， 只允许 llm_server/llm_model legacy internal adapter。dashboard 必须通过新 action facade。 @@ -1164,7 +1398,7 @@ release contract 分配模块 `250`： | 25001 | InvalidCatalogAction | 未知、未分级、不适用 action；旧 payload | | 25002 | PermissionVersionConflict | Catalog/resource/assignee optimistic version 冲突 | | 25003 | ImmutableStandardModel | 修改标准模型受保护字段 | -| 25004 | PermissionModelStateConflict | inactive/被引用模型删除/空模型 | +| 25004 | PermissionModelStateConflict | 使用 inactive 模型新增/变更授权、被引用或有残留关系的模型删除、active 空模型 | | 25005 | GrantLevelForbidden | 无 manage、越级、同级策略拒绝 | | 25006 | ProtectedAssignmentMutation | 普通成员修改 protected/self assignment | | 25007 | InvalidPermissionMode | 无 parent、非法模式、操作冲突 | @@ -1174,6 +1408,7 @@ release contract 分配模块 `250`： | 25011 | AuthorizationModelMismatch | store/model/checksum/实例 pin 不一致 | | 25012 | PermissionImpactExpired | draft 过期/current release 已变化 | | 25013 | PermissionMutationTooLarge | tuple 原子上限无法满足 | +| 25014 | PermissionEnumerationIncomplete | StreamedListObjects 未正常结束、结果超过入口容量或无法证明集合完整 | 通用 deny 继续用 19000，OpenFGA unavailable 用 19002，非法 resource type/id 用 19003； 不得创建相同含义的新码。 @@ -1187,9 +1422,10 @@ release contract 分配模块 `250`： | tenant / department / user_group FGA tuples | 新模型保留关系 | 组织同步遗漏会影响所有 Grant | | F017 `shared_with` 与 `ResourceShareService` | 系统只读关系 | 不能误转 tenant 内普通 Grant | | F027/F036/F040 cursor/有界扫描 | 列表编排 | 不能退回 fetch-all + Python filter | +| KnowledgeSpace/Department Owner Repository | canonical creator、tenant/status/type/filter/sort、部门与空间绑定、500-ID 分块稳定归并 | creator 语义或排序键变化会破坏 mine/joined 去重和分页；不得把 membership role 当 creator/visible | | AuditLogDao structured insert | 审计 | metadata 需限长且不存姓名/密钥 | | Redis | recent-change marker、迁移锁 | 故障时 higher consistency / DB lock，不降级 ALLOW | -| OpenFGA v1.15.1 | Check/List/Write | 版本/digest 是发布契约的一部分 | +| OpenFGA v1.15.1 | Check/BatchCheck/StreamedListObjects/Write | 版本/digest 是发布契约的一部分；只有正常结束的 stream 可作为完整集合；升级 deadline/resolve 配置必须重跑 BENCH | | Alembic 单 head + MySQL/DM8 DDL | `uv run alembic upgrade head` | revision 只能建/改 schema；若混入数据逻辑或产生多 head，部署必须失败 | | backend scripts 运行环境 | cwd=`src/backend/`、live `config`、完整 app context | 配置或工作目录不一致会迁移到错误 DB/Store；脚本不得由服务启动路径隐式调用 | @@ -1200,7 +1436,7 @@ release contract 分配模块 `250`： | C1 DDD | PASS | 业务 Service→业务 Repository→verified target→Permission facade；Permission domain 禁止 import 业务 model/repository；数据迁移仅由 scripts 编排业务 source port；旧 2,211 行 endpoint 拆分 | | C2 双 DB | PASS | Alembic revision 只做 MySQL/DM8 DDL；数据迁移脚本经 Repository/cursor 执行，无 JSON runtime query/partial index/方言 upsert | | C3 多租户 | PASS | Grant/mode/assignee/operation 带真实 tenant；自动 filter 强制 import；仅数据迁移脚本、平台超管 Catalog 全局影响聚合与 F017 精确共享 adapter 可窄范围 bypass | -| C4 权限 | PASS（含下述措辞修订） | 业务侧先验证资源 target；PermissionService 内 super_admin→target tenant gate→tenant_admin→具体 action OpenFGA；管理/可授等级也在 FGA | +| C4 权限 | PASS（含下述措辞修订） | 具体 action 仍是业务侧验证 target→super_admin→tenant fence→tenant_admin→OpenFGA；个人 `visible` facade 刻意不做管理员扩权短路，并由同一 OpenFGA relation 执行 Check/BatchCheck/完整枚举；系统管理列表走独立身份业务入口 | | C5 错误码 | PASS | release contract 分配 250；通用 deny/unavailable 复用 190 | | C6 安全 | PASS | 无 secret/PII tuple；日志只记录 opaque ID/hash；故障 fail closed | | C7 前端 | PASS | Platform/Client 使用各自 wrapped request + react-query；store 不 HTTP；无新库 | @@ -1229,17 +1465,20 @@ Constitution 头部要求 law change 经过 PR review；用户 Design ★ 确认 1. **纯模型测试**：真实 OpenFGA v1.15.1 容器运行 model tests，覆盖： - standard/custom 动作； - - active Catalog/model； + - active Catalog 与模型可分配状态不进入既有 Grant runtime； - direct+department+group 并集与独立撤销； - `department#member` 与递归 `department#subtree_member`、树移动与 resolve-depth blocker； - same-source grant level，禁止跨模型拼接； - ordinary/protected + CUSTOM/INHERIT + 始终存在的 canonical parent； - permission_enabled create/delete fence； - public/shared 只读沿 parent 传播且不受普通 mode gate 影响； - - dashboard visible/edit/delete/manage_permission 与 preset system visibility。 + - dashboard visible/edit/delete/manage_permission 与 preset system visibility； + - 单槽浅层 visible 与深层 canonical oracle 对 direct、部门、 + 用户组、多 Grant、多来源、protected、system、CUSTOM/INHERIT 的集合完全一致。 2. **Catalog publisher crash matrix**：在 SQL prepare、FGA stage、FGA commit、SQL finalize 每个边界注入故障，证明旧/新只会完整生效且 retry 幂等；覆盖 impact checksum 变化、 - publish fence、recent marker 预置失败，以及 action 缺失于 pinned model 时拒绝。 + publish fence、recent marker 预置失败、2-tuple Catalog commit，以及 action 缺失于 pinned + model 时拒绝；模型 active 改变不得产生 visible tuple delta。 3. **Grant/mode concurrency**：双管理员旧 version、重复 idempotency、ADD+REMOVE 同 tuple、 protected collision、50/51 change item 与 90/91/100 compiled tuple 边界。 4. **C3/双库**：MySQL+DM8 Alembic DDL、单 head、unique、cursor、FOR UPDATE、tenant @@ -1247,7 +1486,12 @@ Constitution 头部要求 law change 经过 PR review；用户 Design ★ 确认 5. **数据迁移脚本**：确认 Alembic revision 无 DML/业务 import；覆盖损坏 JSON、孤儿/跨 tenant、edited system model、空 custom、非单调 manage、 child expansion collapse、CREATOR/`user_id` 差异保留、多 ordinary owner、F018 历史交接、 - Store ID 保持不变、dual model 禁用、旧 tuple 退役、checkpoint 断点续跑与前向修复。 + Store ID 保持不变、dual model 禁用、旧 tuple 退役、checkpoint 断点续跑与前向修复；每条 + 每个合法 target assignee 同步生成单条 contribution，同 key 多来源只写一个聚合 tuple， + inactive 模型的既有合法 binding 仍生成、orphan binding 不生成，system 来源不进入该表， + D4 Check/BatchCheck/streamed list + 与 canonical oracle checksum 一致；有效 manual subscription、同 tuple 去重、department_admin + 派生 membership、已撤权历史 active row 与证据冲突 blocker 分别覆盖。 6. **业务回归**：knowledge/library/folder/file/workflow/assistant/tool/channel/dashboard 的全部旧 permission_id/AccessType call sites 改为 action；dashboard 删除从 edit 改 delete； 文件 preview 不做 action、download 必须做 download；列表以 reference oracle 对比。 @@ -1260,26 +1504,39 @@ Constitution 头部要求 law change 经过 PR review；用户 Design ★ 确认 10. **权限边界架构测试**：permission domain 不得 import 业务 ORM、DAO、Repository 或 `telemetry_search/knowledge/workflow/tool/channel` Service；伪造 HTTP tenant/status/version 不能生成 VerifiedPermissionTarget；各业务 Service 的 tenant/status/parent 校验有独立测试。 +11. **来源收敛**：同一 subject-resource 由两个 Grant、两个模型、direct+department 或 + subscription+group 同时贡献；逐个撤销只在最后来源消失时删单槽 tuple。模型 inactive 后 + 既有结果不变且不能新增绑定；最终删除在引用/残留非零时被阻断；COMMIT_UNKNOWN/reconcile + 后 source projection、live tuple 与 canonical oracle 一致，任何无来源 tuple 都成为 blocker。 +12. **列表合同**：StreamedListObjects 正常结束、deadline、取消、服务错误、5,000/5,001 容量； + joined 排除本人创建且不读 membership/role，管理员不扩权；department 候选只 BatchCheck 一次； + file 候选不足一页时跨批凑页并以最后扫描候选生成 cursor。MySQL/DM8 对分块 ID 稳定归并一致。 ### 7.2 量化门槛 | 维度 | 门槛 | |---|---| | Check | 同一脱敏数据与并发下新读路径 P95 ≤ `max(旧路径 P95×1.20, 旧路径 P95+5ms)`；错误率 <0.1%；记录 dispatch 与 datastore read 数 | -| BatchCheck | 业务侧 20/50/100 个有界候选分别测 P50/P95/P99；新读路径 P95 ≤ 同规模旧路径×1.25 | -| ListObjects | 与 BatchCheck 分开报告 direct/department/group/inherit、多 Grant、结果 10/100/1000；不得静默截断，未达标入口改业务 cursor+BatchCheck | -| 高频业务列表 | F040 当前版本为 oracle；可见集合/cursor fingerprint 一致，P95 不退化 >20%，不 fetch-all | -| Catalog 更新 | FGA staging O(模型数×动作数)，commit 恰好 2 tuple；与 resource/assignee 数无关 | -| Grant mutation | ≤50 change item 且 compiled delta≤90；一个 commit Write≤90；不得重写同 Grant 其他 assignee | +| BatchCheck | 业务侧 20/50/100 个有界候选分别测 P50/P95/P99；记录目标 visible DSL 的 dispatch/datastore read；新读路径 P95 ≤ 同规模旧路径×1.25 | +| 完整 visible 枚举 | pinned v1.15.1、目标单槽浅层 DSL，资源 10k/100k、可见 10/100/1000/5000，覆盖 direct/department/group/system 与多来源；stream 必须正常结束且集合 checksum=canonical oracle，P95 分别 ≤50/100/300/1000ms，任何成功响应不得截断 | +| 列表路径对比 | 对 joined/department/file 分别记录 `N_db/V/p`、继承/CUSTOM、过滤选择性、k、扫描放大、DB/权限/总体 P50/P95/P99；选定路径 P95 不高于另一条 1.20 倍且不能牺牲完整性；若差异在 20% 内，选择分页更稳定、维护更简单者 | +| 高频业务列表 | F040 当前版本为业务语义 oracle；可见集合/排序/cursor fingerprint 一致，P95 不退化 >20%，不 fetch-all；joined 5,000 容量和 DB 500-ID 分块在 MySQL/DM8 通过 | +| Catalog 更新 | action/model definition staging O(模型数×动作数)，commit=2 tuple；模型 active 变化只更新 Catalog/SQL 可分配状态，不扫描 Grant、不写 visible tuple；发布 finalize 前保持 write fence | +| Grant mutation | ≤50 change item 且 action+单槽 visible compiled delta≤90；一个 commit Write≤90；不得重写同 Grant 其他 assignee或同资源其他来源 | | Mode switch | staging 每批≤90 tuple；commit=2 mode tuple；任何时刻无 local+parent 普通权限并集 | | 部门集合 | 生产脱敏 P99 树深在 pinned `resolve-node-limit`（`OPENFGA_RESOLVE_NODE_LIMIT`）安全范围内；成员/移动不重写资源 Grant | -| data migration script | DB scan batch=500，FGA write batch≤90，每批 checkpoint；RSS≤512MiB；无全量 JSON 常驻；Alembic upgrade 不扫描业务数据 | -| 安全变更一致性 | revoke/inactive/策略收紧成功后立即 higher-consistency 验证；随后 cache window 不出现旧 ALLOW | - -`BENCH-01`：在实现/发布验证环境用生产脱敏分布覆盖 tenant/resource（含 dashboard）、 -model/Grant/assignee、部门树深度、folder/file 权限链深度及组合最坏路径，记录 -P50/P95/P99 与既有旧读路径基线对比并附 checksum。它是新读路径的性能验收，不是生产迁移 dry-run； -未通过时不得进入正式升级窗口，也不能以 109 环境数据豁免，需回到 Design 调整。 +| data migration script | DB scan batch=500，action+单槽 visible FGA write batch≤90，每批 checkpoint；RSS≤512MiB；无全量 JSON 常驻；source/聚合 tuple checksum 一致后才允许退役旧 tuple；Alembic upgrade 不扫描业务数据 | +| 安全变更一致性 | revoke/replace/模型动作或策略收紧成功后立即 higher-consistency 验证；随后 cache window 不出现旧 ALLOW；模型 inactive 不属于撤权 | + +`BENCH-01` 分两阶段：首轮报告 +[bench-01-flat-visible-report-20260813.md](./bench-01-flat-visible-report-20260813.md) 是 116 +v1.14.2 隔离 Store 的合成引擎能力证据；第二阶段必须在 pinned v1.15.1 使用本文单槽浅层/ +mode/system/inherit 的目标 DSL，并用生产脱敏或可证明代表性的分布覆盖 tenant/resource（含 +knowledge_space 与 file）、model/Grant/assignee、部门树深度及组合最坏路径。第二阶段同时跑 +joined/department/file 的两条完整业务链路，记录 source checksum、visible checksum、 +P50/P95/P99、DB SQL/扫描行数、FGA dispatch/read 和错误/超时。它是本次实现和路径启用的 +发布 blocker，不是生产迁移 dry-run；未通过不得开发/启用 `list_visible_objects`，需回到 +Design 调整。 ### 7.3 可观测 @@ -1290,6 +1547,8 @@ P50/P95/P99 与既有旧读路径基线对比并附 checksum。它是新读路 | `permission_decision` | tenant, resource_type, action, outcome/reason, auth_model_id, catalog_release, consistency, elapsed_ms | | `permission_projection` | operation_type/status, tuple_count, retry_count, scope_type, elapsed_ms | | `permission_catalog_publish` | old/new release, model_count, action_count, impacted counts, stage/commit elapsed | +| `permission_visibility_projection` | catalog, source_count, unique_tuple_count, project/reconcile/checksum elapsed, stale/orphan counts | +| `permission_visible_list` | tenant, resource_type, strategy, candidate_count, visible_count, scanned_count, scan_amplification, stream_completed, capacity, db/fga/total elapsed | | `permission_roster_explain` | resource_type, model_key, source_type, grant_count, assignee_count, page_size, elapsed_ms | | `permission_migration` | run/phase/status, scanned/created/skipped/blocked/diff counts, checkpoint | @@ -1301,10 +1560,14 @@ P50/P95/P99 与既有旧读路径基线对比并附 checksum。它是新读路 - migration blocker>0、checkpoint 停滞、semantic diff 未批准； - D5 后任一实例仍引用旧 model、启用 dual model，或 forward-fix operation `FAILED_CLOSED`； - Catalog 同时出现 0 或 >1 CURRENT。 +- source projection 与 live visible checksum 不一致、模型删除前引用或残留不为零、无来源 tuple>0； +- visible stream 未正常结束、joined 达到 80% 容量、candidate-first 扫描放大 P95>10，或入口 + 实际 `N_db/V/p` 连续 7 天超出 BENCH 登记区间。 审计复用 `AuditLogDao`，事件名固定为 `permission.catalog.{draft,publish}`、`permission.model.{create,update,state}`、 `permission.grant.mutate`、`permission.mode.switch`、`permission.resource.{move,copy,delete}`, +`permission.visibility.{project,reconcile}`、 `permission.migration.{start,checkpoint,retire_legacy,verify,activate,forward_fix}`。 每条记录包含 operator/tenant/target、idempotency/operation ID、before/after version+checksum、影响计数、result、error code、时间；成员姓名、Config 原文和 token @@ -1347,6 +1610,19 @@ direct member、department member 和无权限用户；文档/日志不保存固 6. 正式迁移期间保持 API/Celery/Linsight 全部停止；启服时采集所有 instance heartbeat， Store ID 必须仍是迁移前的现有值，model ID/Catalog release 必须全部是新值，且 `dual_model_mode=false`；不以 `/api/v1/env` 版本字段代替。 +7. 用普通用户、平台超管和租户管理员分别访问 + `GET /api/v1/knowledge/space/joined?order_by=update_time`：三者都只返回自身来源可见且非本人 + 创建的空间；关闭一个 direct 来源但保留 department/group 来源时空间仍在，删除最后来源后 + 消失；SQL trace 不得查询 `space_channel_member`，OpenFGA 不得逐空间 BatchCheck。 +8. 访问 `GET /api/v1/knowledge/space/department?order_by=update_time`：先得到部门绑定候选， + 单次有界 BatchCheck 后查详情；响应无 manage_permission、文件数和部门元数据。构造文件列表 + 首批可见项不足一页，确认继续扫描且 next cursor 不重复/漏项。 +9. 在隔离测试环境让 StreamedListObjects 超时及生成 5,001 个可见空间，接口必须分别返回明确 + 25014 且不返回部分列表；恢复 5,000 以内后 checksum 与 canonical oracle 一致。 +10. 停用一个已被多个资源使用的自定义模型：原成员的 visible/具体 action/manage_permission + Check 前后保持一致，新增或 MOVE 到该模型返回 25004，且 projection ledger 没有 visible + delta；直接删除继续返回 25004。逐项撤销或替换全部绑定并完成 source/live checksum 对账后， + 通过 `DELETE_MODEL` draft 发布不再包含该模型的新 Catalog，历史 RETIRED 快照仍可审计。 ### 7.5 Spec 追踪矩阵 @@ -1368,7 +1644,12 @@ direct member、department member 和无权限用户；文档/日志不保存固 | AC-127～136 | §4.3 生成公式、§6.1 Catalog validation、§8.1 legacy model mapping | | AC-137～147 | §4.5 规范化表、§4.6 ledger、§8 migration item/Config cleanup | | AC-153～158 | §2.1、§3 决策1/9/12、§4.2～4.3、§4.5.4、§6.2、§8、dashboard/preview/download/F018/边界与 Alembic/scripts 职责测试 | +| AC-159～171 | §3 决策13、§4.4～4.6 单槽浅层 visible/source projection/引用计数、§6.2 完整流式枚举、来源收敛与 crash/reconcile 测试 | +| AC-172 | §3 决策14、§6.2 joined 调用链；可见 ID 优先、canonical creator 排除、无 membership/role/admin 扩权依赖 | +| AC-173～176 | §3 决策14、§4.2 双路径编排、§7.1～7.3 列表合同/业务分布 BENCH/重评告警 | +| AC-177 | §4.1.1、§8.1～8.4；旧系统单次正式 run 直接生成最终单槽浅层 model、Grant/Assignee source projection 与 visible tuple，D4 三种 visible 查询同源校验 | | NFR-01～14 | §4.2/4.5/4.6、§6.5、§7.1～7.4 与 BENCH-01 | +| NFR-15～18 | §3 决策13～14、§6.2、§7.1～7.3 的端到端成本、来源收敛容量、扫描放大和代表性数据记录 | --- @@ -1388,6 +1669,7 @@ direct member、department member 和无权限用户；文档/日志不保存固 | 损坏/重复/冲突 Config | migration blocker item | 保留 config key、row version 与字段 locator；不采用默认值/空数组 | | include_children root binding + child tuples | 一个 root assignee + `department#subtree_member` | child tuple 只核对；新模型新增对称 child mirror | | current business owner/creator field + owner tuples | 一个 protected source + 0..N ordinary owner sources | 按 §3 决策9；多 owner 本身不阻断 | +| active knowledge-space non-creator membership | direct Grant assignee，`source_locator=space_membership:` | 仅由 Knowledge 迁移 adapter 证明是仍有效的 manual subscribe/invite/approval 业务来源时迁移；与同一直接 tuple/binding 指向相同模型时合并为同一授权 contribution 但保留来源追溯；`department_admin` 等派生 membership 改从 canonical 部门 Grant 迁移；无法区分历史残留与有效订阅时阻断/人工确认，不得仅凭任意 active row 复活授权 | | legacy `failed_tuple` pending/dead | evidence-based reconciliation item | 依次核对 Store 最终状态、资源是否仍存在、旧模型明确拒绝、资源新事实重建结果；tenant/department member 由对应业务域 adapter 给出 canonical state 并进入目标 tuple 写/删计划；只有无法取得上述证据的记录才阻断 | | tuple 指向已删除的 canonical resource | stale-resource audit item | 不创建虚构资源或 Grant；目标 tuple 校验通过后，与其他 legacy tuple 一起删除 | | folder/file 本级 ordinary | CUSTOM + 有效来源快照 | canonical parent tuple 仍保留，但普通权限不读取它 | @@ -1395,6 +1677,21 @@ direct member、department member 和无权限用户；文档/日志不保存固 | shared_with / public / system identity | 新 system relations | 不进普通 Grant | | dashboard owner/editor/viewer tuple 与 DASHBOARD(_WRITE) | dashboard Grant/action | owner/manager/editor/viewer 按通用模型迁；读为 fixed visible，write→edit，删除单独转 delete | | llm_server/llm_model old tuple | 新模型中的 legacy allowlist | 不进入 Catalog action | +| 每条映射后合法模型的 Grant assignee | 一条 `permission_visible_source_projection` contribution | ordinary/protected 分 relation；`source_owner_key=grant_assignee:`；显式 inactive 模型的既有合法 binding 仍迁移并继续有效 | +| 同一 resource/relation/projected_subject 的多个有效 contribution | 一个聚合 OpenFGA visible tuple | 迁移 item 保留全部来源，tuple 只写一次；任一来源撤销不能删除其他来源的 tuple | +| system/public/shared 可见来源 | `explicit_system_visibility` 及既有 system relation | 不进入 Grant source projection；按 Owner adapter 映射并与最终 `visible` oracle 一并校验 | + +可见迁移不是从旧 `view_*`/viewer tuple 旁路再造一套授权：迁移器必须先按现有规则形成唯一 +Model/Grant/Assignee canonical target，再由同一 target 编译器产生单槽 contribution。旧直接 tuple +和 Config binding 同时指向一个授权时，仍只形成一份 assignee/contribution；无直接 tuple 的 +orphan binding 继续只审计，不得因为需要可见索引而复活。folder/file 的继承可见性继续由 +parent+mode 计算，不对子孙物化；部门/用户组保持 userset，不展开成员。 + +主动订阅的迁移边界同样遵守“业务来源与执行 tuple 双证据优先”：有效 manual membership 是 +Knowledge Owner 域的来源事实，但旧系统中已知可能存在未随撤权清理的历史 active row；因此 +adapter 必须结合 relation、旧直接 tuple/binding、审批/订阅状态与来源字段形成确定结论。确认为 +历史残留的 row 只进入审计/清理，不生成 Grant；证据冲突进入 blocker，不能为了 joined 列表 +完整而批量把所有 active membership 重新授权。 owner adapter 的最低证据： @@ -1431,9 +1728,9 @@ protected owner transfer 不再是开放项：F048 启服构建必须退役 F018 |---|---|---| | D0 运维停流 | 无业务流量和任务消费 | 由部署/入口层停止 HTTP/WS 业务流量并暂停 Celery/Linsight 消费；记录现有 Store ID、旧 model ID、Config/业务表 watermark 与 environment fingerprint；应用 `/health` 只表示进程存活 | | D1 Schema Migration | 进程存活、F048 不就绪 | 正常启动链只执行 `alembic upgrade head` 的 MySQL/DM8 DDL；验证单 head 与新表/索引/约束存在；revision 不创建 run、不读取或写入业务数据、不访问 OpenFGA | -| D2 Data Migration Script 初始化与控制面转换 | backend 容器可进入、运维停流持续 | 运维进入 backend 容器，从 `src/backend/` 以 live `config` 执行 `scripts/migrate_f048_permission_data.py migrate ... --apply`；脚本验证 D1/schema fingerprint，创建唯一正式 run，在**同一 Store**发布新 model 并跑 model tests，再导入 Action/Catalog/标准及自定义 Model、binding→Grant/assignee、mode 和人工项 | -| D3 迁移并退役旧数据 | 运维停流持续 | 原地复用仍合法的 tenant/department/user_group/system/shared/parent tuple；按 item/checkpoint 写新 Catalog/Grant/mode tuple，逐批核对后删除已迁移资源的旧四档/废弃 relation tuple；旧 Config 大 JSON 原始行只读保留供排障，不再参与运行时 | -| D4 数据脚本 verify | 运维停流持续、尚未重启 | 执行同一脚本的 `verify --run-id`；blocker=0；来源/目标计数和 checksum 一致；source item 必须在应用层按 `(source_kind, source_locator)` 排序；已迁移类型 legacy tuple 计数为零；新 model 高风险语义通过；run 转 `READY_TO_START` | +| D2 Data Migration Script 初始化与控制面转换 | backend 容器可进入、运维停流持续 | 运维进入 backend 容器，从 `src/backend/` 以 live `config` 执行 `scripts/migrate_f048_permission_data.py migrate ... --apply`；脚本验证 D1/schema fingerprint，创建唯一正式 run，在**同一 Store**直接发布包含单槽浅层 visible 的最终 F048 model 并跑 model tests，再导入 Action/Catalog/标准及自定义 Model、binding→Grant/assignee、mode 和人工项；不发布中间 F048 model | +| D3 迁移并退役旧数据 | 运维停流持续 | 原地复用仍合法的 tenant/department/user_group/system/shared/parent tuple；按同一 item/checkpoint 写新 Catalog/Grant/mode/action tuple，由 target Grant assignee 编译单槽 source projection 和聚合 visible tuple；system/public/shared 按 Owner adapter 写直接 system tuple；source/聚合 tuple checksum 一致后逐批删除已迁移资源的旧四档/废弃 relation tuple；旧 Config 大 JSON 原始行只读保留供排障，不再参与运行时 | +| D4 数据脚本 verify | 运维停流持续、尚未重启 | 执行同一脚本的 `verify --run-id`；blocker=0；来源/目标计数和 checksum 一致；source item 必须在应用层按 `(source_kind, source_locator)` 排序；单槽 contribution 与聚合 tuple checksum 一致、无来源 visible tuple=0；单资源 visible 与完整枚举均等于 canonical oracle；已迁移类型 legacy tuple 计数为零；新 model 高风险语义通过；run 转 `READY_TO_START` | | D5 重启并自动发现 | 重启→新 model | 数据迁移成功后重启 API、Celery、Linsight 等服务；全部实例在首次权限调用时延迟初始化 Permission Runtime，发现唯一同名 Store/latest model，并验证它与 SQL CURRENT Catalog 的 ACTIVE release 一致；smoke 通过后由运维恢复流量和任务消费 | | D6 前向运行 | 新 model | 正常读写只走新 model + projection ledger；问题只做前向修复；旧 model ID 仅作为 OpenFGA 不可删除的历史版本存在 | @@ -1446,6 +1743,7 @@ CREATED → MODEL_PUBLISHED → MIGRATING_CONTROL_PLANE → MIGRATING_TUPLES + → MIGRATING_VISIBILITY → RETIRING_LEGACY → VERIFYING → READY_TO_START @@ -1460,6 +1758,27 @@ CREATED `authorization_model_release` 置为 `ACTIVE` 并退役来源 release；D5 是应用重启与启服阶段， 不再修改 data migration run，也不引入第三个 migration CLI 子命令。 +#### 已完成旧版 F048 迁移环境的前向对账 + +旧版 F048 migration run 已到 `READY_TO_START/COMPLETED` 时不得重置、篡改或创建第二个正式 +migration run。使用 `scripts/reconcile_f048_visible_projection.py` 执行独立的前向对账： + +1. 默认 dry-run，以 SQL `PermissionGrant/PermissionGrantAssignee` 为 Grant 可见来源真相，重算 + source projection 与聚合直接 tuple；不扫描 OpenFGA tuple，也不从 system/public/shared + 关系反推授权。 +2. `--apply` 必须确认 Store ID 和操作人，并要求 runtime heartbeat 和在途 projection operation + 均为零；旧 model 切换还必须显式指定 `--allow-model-upgrade`。生产、开发、测试使用同一逻辑， + 不按环境名放宽门禁。 +3. 若 CURRENT Catalog 仍引用旧 F048 model，在同一 Store 发布最终 immutable model；按 + `projected_subject + visible + resource` 去重后，用 `on_duplicate=ignore` 幂等确保所有 Grant + 直接 tuple，higher-consistency BatchCheck 全部通过后才激活 SQL source projection，再发布一个 + 无业务动作/模型变化的新 Catalog release 绑定新 Authorization Model release，最后退休旧 release。 +4. 部门和用户组保持 userset subject，不展开成员用户；脚本不扫描或删除多余 direct visible tuple， + 因为它可能由 system owner 等非 Grant canonical 来源贡献。存在 stale Grant source projection 时 + apply 整体阻断，来源分类完成前不得把“无 Grant source”直接解释为应撤销。 +5. 任一步失败保持停流，重复执行同一命令按 model checksum、Catalog idempotency key、source + fingerprint 和 tuple key 前向续跑；成功后重启服务，由稳定 Store name 自动发现新 model。 + ### 8.3 正式迁移命令与启服门禁 Schema 命令只有 `alembic upgrade head`，revision 只做 DDL。F048 数据迁移入口固定为 @@ -1470,7 +1789,8 @@ Celery import/调用。`migrate` 必须显式带 `--apply`，否则 argparse 返 正式 `migrate --apply` 校验 schema fingerprint，按稳定 Store name 自动发现唯一现有 Store/latest source model 并将二者写入 durable run，在该 Store 发布新 model，然后在同一个真实 run 内完成 source -validation、规范化 SQL、新 tuple 写入、旧 tuple/Config 退役。resume 必须复用 run 中的 +validation、规范化 SQL、Grant/Assignee 与单槽可见 source projection、新 tuple 写入、旧 +tuple/Config 退役。resume 必须复用 run 中的 Store 与 source/target model ID，不接受调用方替换。报告由真实 item 聚合为 CSV/NDJSON + checksum，artifact 只用于审计，不成为运行时真相。 @@ -1488,13 +1808,18 @@ CSV/NDJSON + checksum，artifact 只用于审计，不成为运行时真相。 每个 batch 固定 source key range，把以下内容同时写入 migration item： - normalized source checksum 与目标 SQL row checksum； -- 同一 Store 中 target tuple 与待删除 legacy tuple 的 fingerprint 集及实际核对结果； +- 同一 Store 中 action/mode/system/单槽 visible target tuple 与待删除 legacy tuple 的 + fingerprint 集及实际核对结果； +- 每个 target Grant assignee 对应的 contribution fingerprint、visibility class、聚合 tuple + fingerprint，以及同 key 其他 contribution 的引用计数； - created/reused/legacy-deleted/skipped/deduplicated/blocked 数； - checkpoint 的最后 source locator、累计 checksum 与 lease version。 - model/tuple/mode mapper 的 INFO/BLOCKER 差异明细；`mapping-blocked` 必须先持久化具体 source key、 message、severity 与 difference type，并同步 run.blocker_count，不能只在 CLI 输出聚合错误码。 -删除旧 tuple 前，migration item 必须先持久化完整 source snapshot 和目标 checksum。 +删除旧 tuple 前，migration item 必须先持久化完整 source snapshot、Grant/Assignee target、单槽 +source projection 与全部目标 tuple checksum；source 与聚合 tuple 未达到相同完整 target checksum 时不得进入 +`RETIRING_LEGACY`。 进程在 SQL commit、目标 tuple Write 或 legacy delete 后崩溃时，resume 读取 SQL 与同一 Store 的目标/遗留 fingerprint：目标已存在且 legacy 已删除则 finalize；目标存在但 legacy 仍在则只续删；两者都未提交才重放；其他混合集标 `FAILED_CLOSED`，前向修正后重新 verify。 @@ -1522,6 +1847,11 @@ D5 必须同时满足： 并允许 0..N 个 ordinary owner；knowledge_space/channel 的 CREATOR/`user_id` 差异已 preservation-first 映射或人工批准；每个 system-owned exception 命中显式 allowlist； - SQL/OpenFGA count、source/target checksum、Catalog/model checksum 全部一致； +- `permission_visible_source_projection` 对每个合法 Grant assignee 可见来源恰有一条 + contribution；显式 inactive 模型的既有 binding 继续迁移，orphan binding 为零；聚合 tuple + checksum 与来源一致，无来源 visible tuple=0； +- 按 tenant/resource type 的 canonical oracle 对单资源 `visible`、BatchCheck 与完整 streamed + visible 枚举做集合 checksum，比对结果完全一致；管理员身份不参与该 oracle 扩权； - 已迁移资源类型的旧四档/废弃 relation tuple=0；两份旧 Config 大 JSON 原始行只读保留， 且旧 Config 运行时读取/写入/授权引用=0； 仍合法的 system/shared/parent/组织 tuple fingerprint 与 D0 source snapshot 一致； @@ -1559,18 +1889,23 @@ D3 已完成全部旧运行数据退役，D6 没有延后的 cleanup 窗口。 `FineGrainedPermissionService`、legacy shadow write、creator fallback 和 F018 route 对已迁移资源均不可达； 4. 仅 llm_server/llm_model 显式 legacy allowlist 保留并有独立测试；dashboard 旧 adapter 删除； -5. 审计 Store ID 未变化、旧/新 model ID、legacy delete checksum 和最终 target checksum。 +5. 审计 Store ID 未变化、旧/新 model ID、legacy delete checksum、单槽 visibility checksum 和 + 最终 target checksum。 --- ## 9. 已知短板与后续改进 -- 生产脱敏分布尚未落入仓库；`BENCH-01` 是进入正式升级窗口前的发布 blocker，不影响 - Design 代码实现，但不能在 E2E/发布评审中标绿。它验证新读路径性能，不是迁移预演。 -- Catalog 完整快照与 model release staging 是 O(模型数×动作数)，刻意换取原子性； - 当前无生产模型规模证据支持更复杂的增量图。 +- 现有 BENCH-01 只是 116、OpenFGA v1.14.2、裸展平 relation 的 10k 合成结果；补充 A/B + 实验用于比较永久 switch 成本，但仍没有覆盖完整 mode/system/inherit、业务 DB 与代表性分布。 + 第二阶段在 pinned + v1.15.1 通过前，Design 可以评审，但不得实现或启用生产 `list_visible_objects` 路径。 +- Catalog 完整快照与 model release staging 是 O(模型数×动作数)；模型 active 变化不扫描 + 资源 Grant、不修改 visible。模型定义变化仍通过 Catalog 2-tuple commit 原子影响既有 Grant + 的具体 action；write fence 窗口必须由 impact、模型规模和真实 BENCH 控制。 - Catalog publish 从最终 impact 重算到 SQL finalize 会全局拒绝 Grant/mode/resource 权限写； - 读取不中断。BENCH-01 必须量出该窗口并给 Platform 明确维护提示，不能静默排队后过期执行。 + visible 读取不随可分配状态变化。BENCH-01 必须量出该窗口并给 Platform 明确维护提示，不能 + 静默排队后过期执行。 - dashboard 已纳入本期，现状缺少创建 owner projection 且删除复用 can_edit；两处都必须 在切换前改完，不能保留 legacy allowlist。 - 文件预览明确不设置 PermissionAction；原件/打包下载使用 `download`，知识库 RAG 使用库 @@ -1582,11 +1917,14 @@ D3 已完成全部旧运行数据退役，D6 没有延后的 cleanup 窗口。 - 权限数据升级是 `src/backend/scripts/` 中的人工运维步骤，不随 Alembic 或服务启动自动 执行；这增加了发布 runbook 操作，但避免不可控 DML/FGA 写阻塞数据库结构升级或多实例 并发迁移。只有未来具备独立审计 Job runner 时才封装该脚本，不能迁回 Alembic/lifespan。 -- OpenFGA 新模型深度比旧模型增加，`department#subtree_member` 还受部门树深度影响； - pinned `resolve-node-limit` 与脱敏 P99 必须共同入门禁。只有真实 BENCH-01 失败才允许 - 重新设计，不能先用 SQL PDP“优化”绕过。 -- Catalog/Model/Grant 的 intersection 对 Check 是常数级额外图遍历，对 ListObjects - 可能被反向枚举放大；因此 ListObjects 不是默认分页路径，是否启用按入口由 BENCH-01 决定。 +- 具体 action 的新模型深度比旧模型增加，`department#subtree_member` 还受部门树深度影响； + pinned `resolve-node-limit` 与脱敏 P99 必须共同入门禁。`visible` 展平只解决可见枚举，不证明 + 具体 action 也应物化；不能用 SQL PDP“优化”绕过。 +- 单槽浅层 visible 增加 source projection 表和每个唯一资源/subject 的聚合 tuple；父级继承、 + department/group 成员不展开，system 来源不进该表。需要持续观察 Store tuple、SQL 行数、 + projection reconcile 与 orphan 差异；超过容量预算时重新设计版本化索引，而不是取消审计。 +- `joined` 首发完整集合容量为 5,000；超过会明确失败而不是漏项。这在语义上安全，但可用性 + 有上限；达到 80% 容量即触发 permission-aware cursor/索引连接的后续设计。 --- @@ -1594,6 +1932,12 @@ D3 已完成全部旧运行数据退役，D6 没有延后的 cleanup 窗口。 | 日期 | 改动 | 触发原因 | |---|---|---| +| 2026-08-13 | 对单槽浅层 visible、inactive 既有授权保持、删除零引用门禁、旧系统单次迁移、列表路径、契约/依赖/测试/可观测执行 24 项 Design 接手测试与 Constitution Check；复审 LGTM，停在 Design ★ | `/sdd-review ... design` | +| 2026-08-13 | 将模型 `active` 收窄为“是否可用于新增/变更授权”：停用不影响已有 Grant；删除必须先撤销或替换全部绑定，并在引用/source projection/live tuple 清零后完成。可见执行关系改为单槽浅层 `visible`，移除 A/B 槽、switch、双写和 Catalog 4-tuple 切换；保留来源引用计数、ledger、reconcile 与旧系统单次迁移 | 用户确认界面语义“关闭后不能再用它授权，已有授权不受影响；删除必须先清理绑定关系” | +| 2026-08-13 | 按 F048 未上线事实重新完成 24 项 Design 接手测试：旧系统单次迁移、A/B 可见投影、订阅/历史 membership 证据边界、D4 完整性门禁和 Release Contract 均一致；复审 LGTM，停在 Design ★ | `/sdd-review ... design` | +| 2026-08-13 | 用户澄清 F048 尚未上线：删除“旧 F048 → successor model”二次迁移和独立 migration run/checkpoint，把 A/B 可见 source projection、switch 与完整性校验合并到原 `migrate_f048_permission_data.py` 从旧系统执行的唯一正式 run | 用户迁移拓扑纠正 | +| 2026-08-13 | 首轮 24 项 Design 接手测试修正现状快照、跨模型同来源 fingerprint 冲突与完整枚举错误码缺口；迁移拓扑后续按上一行重新修订并复审 | `/sdd-review ... design` | +| 2026-08-13 | 仅展平固定 `visible`，引入 A/B 可见槽、全局原子 switch、跨来源 projection 引用计数与模型删除收敛；个人 visible 取消管理员扩权短路；列表按实际数据在完整可见 ID 优先和业务候选优先间选择，并登记 mine/joined/department/file 初始路径、完整流式枚举与 BENCH-01 第二阶段 | 用户要求大资源/小可见范围高性能枚举，同时明确文件列表等高可见率场景应由业务数据含义决定路径 | | 2026-08-06 | 迁移流量控制归属运维部署/入口和队列操作；删除应用全局 HTTP/WS/Worker 迁移门禁假设，`/health` 仅表示进程存活；Permission Runtime 与部门投影改为首次权限调用延迟初始化 | 用户明确迁移期间无人使用平台由运维保证，业务代码不承载一次性运维流程 | | 2026-08-05 | 增加 `reconcile_f048_projection_operations.py` dry-run/apply 运维入口，以 durable ledger 和 live Store/model/scope fence 恢复指定 operation | 116 环境需要安全恢复 11/15/18，并为后续同类 in-flight publish blocker 提供可重复执行的前向修复入口 | | 2026-08-05 | projection 恢复 PREPARED 时以 higher consistency 过滤已满足的 STAGE tuple，只补写缺失项后再 commit | 116 环境 `CUSTOM→INHERIT→CUSTOM` 往返时保留 Grant link，重复 WRITE 被 OpenFGA 拒绝并留下 3 条 PREPARED operation | diff --git a/features/v3.0.0-beta1/048-rebac-permission-model-grants/e2e-checklist.md b/features/v3.0.0-beta1/048-rebac-permission-model-grants/e2e-checklist.md index 78daf9e4a3..92873dba70 100644 --- a/features/v3.0.0-beta1/048-rebac-permission-model-grants/e2e-checklist.md +++ b/features/v3.0.0-beta1/048-rebac-permission-model-grants/e2e-checklist.md @@ -7,6 +7,33 @@ > 数据前缀：`e2e-f048-permission-` > 安全要求：禁止连接生产；禁止删除此前缀之外的任何数据。 +## 0. 2026-08-13 可见性增量必验矩阵 + +以下场景是本次单槽 `visible` 增量的发布门禁，不能被后文旧 F048 通用清单替代： + +- [ ] 创建五个由他人创建的知识空间，分别通过 direct user、department userset、 + user-group userset、主动订阅迁移来源和 system/shared Owner 来源授权同一普通用户； + `GET /knowledge/space/joined` 返回五个空间且不依赖 `space_channel_member` 行或成员角色。 +- [ ] 同一用户创建的空间即使存在 `visible` 也不出现在 joined；平台超管和租户管理员使用 + 相同账号来源集合时结果与普通用户一致，不因管理员身份扩展到全部空间。 +- [ ] 为同一知识空间制造 direct + department + group 多来源；逐个撤销时，前两个撤销不删除 + 聚合 `visible`，最后一个来源撤销后 Check、BatchCheck 和完整枚举才同时变为不可见。 +- [ ] 停用仍被引用的自定义模型：已有成员的 visible、具体 action 和 + `manage_permission` 不变；ADD/MOVE 目标不再包含该模型，Platform/Client 既有行仍展示并可 + MOVE/REMOVE。删除不要求先停用，但引用或 projection/live residual 非零时返回 25004。 +- [ ] 部门空间列表先查询部门绑定得到有限候选空间 ID，只执行一次分块 BatchCheck；不先枚举 + 全平台知识空间，也不重复逐项 visible 校验，响应不返回 `manage_permission` 或部门元数据。 +- [ ] 文件列表按业务排序跨至少三批取候选并 BatchCheck；首批可见项不足时继续扫描，填满页面 + 后 cursor 指向最后消费候选，下一页无重复/遗漏；父空间可见不能替代子项最终 visible。 +- [ ] 为一个用户准备 5,001 个知识空间 visible 结果，joined 返回明确容量错误，不返回 200 + 截断集合；4,000 个结果触发 80% capacity 告警指标。 +- [ ] 在 StreamedListObjects 输出部分结果后断开连接，joined 整体失败且 + `stream_completed=false`；停止 OpenFGA 后 Check/BatchCheck/list 均不从 SQL、旧 binding、 + creator 或管理员身份回退产生 ALLOW。 +- [ ] 校验 `permission_visibility_projection` 与 `permission_visible_list` 指标包含 source/tuple/ + checksum/stale/orphan、strategy/candidate/visible/scanned/amplification/capacity 和各阶段耗时， + 且不包含姓名、资源名、Config 原文或 token。 + ## 1. 环境与账号 - [ ] API、Platform、Client、Worker、Linsight Worker 使用同一构建版本。 @@ -65,7 +92,10 @@ - [ ] 查看者/编辑者/管理者/所有者的 key、名称、等级、动作集合不可编辑或删除。 - [ ] 标准模型仅在包含 `manage_permission` 时显示“允许授予同级”开关。 - [ ] 自定义模型的等级由最高动作自动得出；空动作、未分级动作或越界动作无法保存。 -- [ ] 停用仍被 Grant 引用的模型后，其 Grant fail closed；不能直接不可恢复删除。 +- [ ] 停用仍被 Grant 引用的模型后，不能再用该模型新增或变更授权，但已有 Grant 的可见性、 + 具体动作和 `manage_permission` 保持不变；引用未清零时删除返回 `25004`。 +- [ ] 逐项撤销或替换全部 Grant/assignee 后，source projection 与 live visible tuple 对账为零， + 此时允许删除模型；停用不是删除前置条件。 - [ ] 模型发布前后既有 Grant ID 与 assignee 数不因动作变更发生 fan-out 重写。 ## 4. 两端成员与模式界面 diff --git a/features/v3.0.0-beta1/048-rebac-permission-model-grants/e2e-test-report.md b/features/v3.0.0-beta1/048-rebac-permission-model-grants/e2e-test-report.md index 60cbe13f50..7e2e14ba77 100644 --- a/features/v3.0.0-beta1/048-rebac-permission-model-grants/e2e-test-report.md +++ b/features/v3.0.0-beta1/048-rebac-permission-model-grants/e2e-test-report.md @@ -1,251 +1,112 @@ -# F048 E2E 覆盖报告：ReBAC Permission Model Grants +# F048 可见性增量 E2E 覆盖报告 -> 执行日期：2026-07-30 -> 分支：`feat/v3.0.0-beta1/048-rebac-permission-model-grants` -> 整体状态：**NOT RUN BY SCOPE — 实现验证通过，本地真实环境 E2E 经用户确认不执行** +> 执行日期：2026-08-13 +> 分支：`feat/3.0.0-beta1` +> 整体状态：**PARTIAL — 实现与真实 OpenFGA 集成通过，专用业务 E2E 和真实 MySQL/DM8 未执行** ## 1. 结论 -F048 的功能实现、契约测试、迁移脚本专项、两端权限 UI 专项测试、生产构建、 -架构守卫和新增代码质量检查已通过。用户于 2026-07-30 明确确认本地不执行 -真实环境 E2E，因此专用 API、OpenFGA v1.15.1、MySQL/DM8、D0～D6 启服链路和 -生产分布 BENCH-01 在本次开发完成口径中记为 `NOT RUN BY SCOPE`。 - -这不是实现失败，也不是生产发布批准。本报告保留未执行范围和后续清单，防止把 -“功能与迁移脚本开发完成”误解为“目标部署环境已验证”。 - -## 2. 本次环境 - -| 项目 | 结果 | +本次单槽浅层 `visible`、来源投影、joined ID-first、department/file candidate-first、模型 +inactive 生命周期和单 run 迁移均已实现并完成定向回归。116 隔离内存 Store 上的真实 +OpenFGA v1.15.1 集成测试 5/5 通过，并发现、修复了 v1.15.1 +StreamedListObjects 的 `result` 包络兼容问题。 + +当前不能给出生产发布 PASS：没有配置专用 F048 业务部署账号，也没有 disposable MySQL 与 +DM8 DSN，因此真实 API E2E 为 6 skip，数据库集成的 2 个外部用例为 skip。正式发布仍须完成 +[手工与运维验证清单](./e2e-checklist.md)，尤其是五类 joined 来源、5,001 容量、故障无 SQL +fallback、MySQL/DM8 和 D0～D6 迁移链路。 + +## 2. 自动化结果 + +| 验证层 | 结果 | 判定 | +|---|---:|---| +| T144～T188 各波次定向测试 | 全部通过 | PASS | +| Wave 13 后端回归 | 51 passed, 2 skipped | PASS；skip 为 MySQL/DM8 环境门控 | +| 116 OpenFGA v1.15.1 集成 | 5 passed | PASS | +| OpenFGA 客户端单测 | 16 passed | PASS | +| F048 扩展全集 | 562 passed, 7 skipped, 11 failed | PARTIAL；见 §5 | +| 专用真实 API E2E | 6 skipped | NOT RUN；未设置 `F048_E2E=1` 和专用账号 | +| Platform 可见性增量 | 3 files / 24 tests passed | PASS | +| Client GrantTab 增量 | 9 tests passed | PASS | +| Platform / Client ESLint | 两端全量通过 | PASS | +| Platform / Client strict typecheck | 361 / 1214 strict files passed | PASS | +| `pnpm check-i18n` | no new i18n drift | PASS | +| Ruff（本次后端范围） | all checks passed | PASS | +| Architecture Guard / `git diff --check` | passed | PASS | + +## 3. 真实 OpenFGA v1.15.1 + +环境为 `192.168.106.116` 上只监听 `127.0.0.1:18080` 的隔离内存实例，通过 SSH 本地转发 +执行测试；每个测试创建并删除自己的 `f048-integration` Store，不写业务 Store。 + +| 项目 | 证据 | |---|---| -| 工作区 | `/Users/zhangguoqing/works/bisheng` | -| Python | 仓库 `src/backend/.venv` / Python 3.11 | -| Platform / Client | 仓库现有 Node 依赖 | -| Backend | 未启动；`http://localhost:7860/health` 连接失败，HTTP `000` | -| Docker | 本次范围决策后未用于 F048 验证；未启动 F048 容器或持久化测试数据 | -| OpenFGA | 无可用的 disposable v1.15.1 实例和 digest 元数据 | -| MySQL / DM8 | 无两套 disposable F048 schema DSN | -| BENCH fixture | 仅 synthetic CI harness，`production_derived=false` | -| 已知集成主机 | `192.168.106.116` 网络可达，但当前会话 SSH 认证失败；未执行任何远程写操作，也无法确认其是否为专用 F048 环境 | - -## 3. 自动化结果 - -| 验证层 | 命令摘要 | 结果 | 判定 | -|---|---|---:|---| -| F048 后端专项 | `pytest test/permission/test_f048_*.py ...` | 394 passed, 7 skipped | 实现 PASS；真实环境项未执行 | -| 本次变更文件回归 | 107 个已修改/新增 backend 测试文件 | 913 passed, 13 skipped | PASS；skip 均为显式外部环境门控 | -| 部门/SSO/组织同步整目录 | `pytest test/department test/sso test/org_sync` | 446 passed | PASS | -| BENCH-01 合同 | `pytest test_f048_performance_contract.py` | 12 passed | harness PASS | -| OpenFGA/DB gate | OpenFGA + database integration tests | 2 passed, 7 skipped | NOT RUN BY SCOPE | -| 真实 API E2E | `pytest test_e2e_f048_permission_model_grants.py` | 6 skipped | NOT RUN BY SCOPE | -| Platform 权限专项 | 13 个 F048/权限测试文件 | 57 passed | PASS | -| Client 权限专项 | 7 个 F048/权限测试文件 | 23 passed | PASS | -| Platform build | `npm run build` | success | PASS | -| Client build | `npm run build` | success | PASS | -| Python compile | 全部已修改/新增 backend Python | success | PASS | -| Architecture Guard | `scripts/arch-guard.sh` | success | PASS | -| Patch format | staged + unstaged `git diff --check` | success | PASS | -| 新增 Python Ruff/format | 127 个新增文件 | all checks passed / all formatted | PASS | -| 修改存量关键 Ruff | `ruff --select F,E9` | 本次引入项为 0；剩余 13 条为改动前基线 | PASS_WITH_BASELINE | -| 完整 permission 回归 | `pytest test/permission -q` | 540 passed, 7 skipped, 3 failed | 有既有基线失败 | - -### 3.1 F048 后端专项 - -```bash -cd src/backend -uv run pytest \ - test/permission/test_f048_*.py \ - test/channel/test_f048_channel_permissions.py \ - test/knowledge/test_f048_preview_download_permissions.py \ - test/tool/test_f048_tool_permissions.py -q -``` - -结果：`394 passed, 7 skipped, 212 warnings in 3.41s`。 +| 版本 | v1.15.1，commit `1db35fb8b33d7512666e49e6b75cbd2cca8c4694` | +| 二进制 SHA-256 | `e54deb580c4de50dbd61d689ced8a6b8ed0ed26024add74def1b0a392c512a9c` | +| 测试 | `F048_OPENFGA_INTEGRATION=1 ... pytest test_f048_openfga_integration.py -q` | +| 结果 | `5 passed` | -7 个 skip 均是显式外部环境门控： +覆盖单槽模型结构（无 A/B/switch）、direct/department/group/system、具体 action 不展平、 +Check/BatchCheck/StreamedListObjects 集合一致、higher consistency、Store-scoped legacy tuple +删除、Write 原子性及 resolve depth。 -- OpenFGA v1.15.1 disposable integration：5； -- MySQL/DM8 disposable schema integration：2。 - -覆盖 Catalog、action level 全模型重算、标准/自定义模型、Grant/source、 -多 owner、INHERIT/CUSTOM、projection ledger、business verified target、 -dashboard、preview/download、API/Worker/Linsight pin、D0～D6 coordinator、 -legacy retirement、迁移脚本和 BENCH 合同。 - -### 3.2 真实 OpenFGA 与数据库 - -```bash -cd src/backend -uv run pytest \ - test/permission/test_f048_openfga_integration.py \ - test/permission/test_f048_database_integration.py -q -rs -``` +首次运行暴露真实 v1.15.1 NDJSON 行为为： -结果：`2 passed, 7 skipped`。 - -- 2 个通过项证明 F048 Alembic revision 的静态 DDL-only 合同； -- 5 个 OpenFGA 测试要求 `F048_OPENFGA_INTEGRATION=1`、批准的 - `openfga/openfga:v1.15.1@sha256:` 和 runtime metadata； -- 2 个数据库测试要求 `F048_DATABASE_INTEGRATION=1`、专用 MySQL/DM8 DSN、 - schema 名含 `f048_test` 以及显式 destructive-test ack。 - -未执行项包括真实 Check/BatchCheck/ListObjects、同 Store 新 model、显式 model -tuple 写删、higher consistency、atomic Write、resolve node limit，以及 MySQL/DM8 -upgrade、lease/checkpoint/resume/checksum 对齐。 - -### 3.3 BENCH-01 - -```bash -cd src/backend -uv run pytest test/permission/test_f048_performance_contract.py -q -``` - -结果：`12 passed`。当前仓库 fixture： - -| 字段 | 值 | -|---|---| -| contract checksum | `12654227efbcdb9ebc2effdd1da5c668723cdcd96c4333eb621f0229b190dc14` | -| dataset checksum | `ad302adc12b1807080f85c238b36c7e93a75ef1c39165e6250a3138710f6e54b` | -| authorization model checksum | `edf2c67ebabf5cd24bd9d520181e50ee12c9cf667061488c6886db2253042ee4` | -| dataset source | `synthetic-ci-harness-only` | -| production derived | `false` | - -合同和脚本会验证 Check、BatchCheck 20/50/100、ListObjects -direct/department/group/inherit/multi-grant、10/100/1000 结果、完整结果 checksum -以及业务 cursor + BatchCheck，并输出 P50/P95/P99、错误率、dispatch 和 datastore -query count。 - -synthetic fixture 即使使用 `--allow-synthetic` 也只能验证 harness，报告必须保持 -`release_ready=false`。正式通过必须使用批准的脱敏生产分布 fixture。 - -### 3.4 两端前端 - -Platform： - -```bash -cd src/frontend/platform -npm test -- --run \ - src/test/f048ActionLevelBoard.test.tsx \ - src/test/f048DashboardPermissions.test.tsx \ - src/test/f048ModelEditor.test.tsx \ - src/test/f048PermissionApi.test.ts \ - src/test/f048PermissionDialog.test.tsx \ - src/test/f048PermissionGrantTab.test.tsx \ - src/test/f048PermissionI18n.test.ts \ - src/test/f048PermissionRoster.test.tsx \ - src/test/f048RolesAndPermissions.test.tsx \ - src/test/knowledgeSpaceGrantSubjects.test.tsx \ - src/test/permissionHookCache.test.tsx \ - src/test/subjectSearchUser.test.tsx \ - src/test/systemPageTabsVisibility.test.tsx -npm run build +```json +{"result":{"object":"workflow:direct"}} ``` -结果：`13 test files / 57 tests passed`，生产构建成功。除 12 个 F048/权限文件外， -同时覆盖 `systemPageTabsVisibility.test.tsx` 的系统管理页签边界。仅有存量 -`jsx` DOM attribute、Browserslist 和 bundle size 警告。 - -Client： - -```bash -cd src/frontend/client -npm run test:ci -- --runTestsByPath \ - src/api/permission.test.ts \ - src/components/ChannelMemberDialog.test.tsx \ - src/components/permission/PermissionDialog.test.tsx \ - src/components/permission/PermissionGrantTab.test.tsx \ - src/components/permission/PermissionListTab.test.tsx \ - src/components/permission/permissionI18n.test.ts \ - src/pages/knowledge/FilePreview/TopBar.test.tsx -npm run build -``` - -结果：`7 suites / 23 tests passed`，生产构建成功。仅有存量 Browserslist、 -bundle size、PWA glob 和 runtime font 警告。 - -### 3.5 专用真实 API E2E - -新增： -`src/backend/test/e2e/test_e2e_f048_permission_model_grants.py`。 - -它只在 `F048_E2E=1` 时运行，双重清理 -`e2e-f048-permission-*` 工作流，覆盖： - -- 四个标准模型和 Catalog 超管边界； -- 新资源 `CUSTOM`、受保护 creator owner 及禁止删除； -- viewer Grant 幂等新增、roster 唯一、具体 `visible` ALLOW 和 `delete` DENY； -- 非 `manage_permission` 用户不能读取 roster； -- 撤销后的随后读取立即 DENY； -- 跨租户业务目标不可解析； -- 无 canonical parent 的顶级 workflow 不能切换 `INHERIT`； -- 未登记 action 返回 `25001`。 - -本机执行： - -```bash -cd src/backend -uv run pytest test/e2e/test_e2e_f048_permission_model_grants.py -q -``` - -结果：`6 skipped`，原因是未设置 `F048_E2E=1`，且本机 backend/Docker -确实不可用。正式命令和账号变量见 -[E2E 手工与运维验证清单](./e2e-checklist.md)。 - -## 4. 完整 permission 回归的既有失败 - -```bash -cd src/backend -uv run pytest test/permission -q -``` - -结果：`540 passed, 7 skipped, 3 failed`。3 个失败全部位于 -`test_f027_role_scope_nullsafe_unique.py`： - -1. `test_upgrade_adds_scope_key_and_swaps_to_nullsafe_unique_constraint` -2. `test_upgrade_dedupes_existing_null_scope_collisions_before_creating_constraint` -3. `test_downgrade_restores_previous_constraint_and_drops_scope_key` - -前两个失败是 F027 测试桩构造的 `role` table 缺少 `department_id`；第三个测试桩 -与当前 F027 downgrade 的约束探测不一致。它们不经过 F048 revision、Catalog、 -Grant、OpenFGA model 或新迁移脚本，本次没有扩大范围修复。F048 专项为全绿。 +客户端此前只接受顶层 `object`，会把正常流误判为不完整。修复后同时兼容 v1.15.1 包络和兼容 +代理的未包络形式，单测 16/16、真实集成 5/5 通过。 -## 5. AC 覆盖状态 +## 4. 可见性增量覆盖 -| AC 类别 | 自动化状态 | 仍需真实环境 | +| 合同 | 自动化证据 | 状态 | |---|---|---| -| Catalog / action / model（AC-01～18、148～149、156） | Domain/API/UI 专项通过 | 真实发布确认、并发管理员、审计 | -| Grant / 来源 / owner（AC-19～27、36～44、157） | Domain/API/UI 专项通过 | 真实用户/部门/组成员变化链路 | -| Check / List / fail closed（AC-28～35、69～70） | adapter、合同和故障单测通过 | pinned OpenFGA、真实故障注入 | -| INHERIT / CUSTOM（AC-45～57、150～152） | mode/domain/adapter 专项通过 | 两端交互、真实移动/复制 | -| roster / explain（AC-58～65） | 两端组件及 API 专项通过 | Platform/Client 浏览器清单 | -| dashboard（AC-153） | 后端与 Platform 专项通过 | 真实 dashboard 全动作与分享链接 | -| preview/download（AC-154） | 后端与 Client 专项通过 | 原件/打包文件真实 HTTP 下载 | -| business boundary（AC-155） | architecture/static/adapter 测试通过 | SQL/log 运行时观察 | -| 数据迁移（AC-71～147、158） | mapper/coordinator/CLI/DDL 合同通过 | MySQL、DM8、D0～D6 维护窗口 | -| 性能（AC-28～35、69） | BENCH harness 合同通过 | production-derived fixture 正式跑数 | - -## 6. 未执行的生产发布验证项 - -若后续需要确认某个目标环境可发布，仍应补齐以下四类证据： - -1. **真实 OpenFGA**：批准的 v1.15.1 tag@digest、runtime metadata、同一 Store、 - 一个新 runtime model、5 个集成测试全部通过； -2. **真实数据库和迁移**：MySQL 与 DM8 disposable integration 全绿，并在维护窗口 - 完成 D0～D6、全实例 heartbeat/pin 和旧 tuple/Config 清理证明； -3. **真实 API/UI**：专用 E2E 6/6 通过，并完成 Platform、Client、dashboard、 - preview/download、故障注入、跨租户、Worker/Linsight 清单； -4. **正式 BENCH-01**：批准的脱敏生产分布 fixture，`production_derived=true`， - 所有阈值通过且 `release_ready=true`。 - -本次按用户确认不继续建立本地 E2E 环境。已知主机 -`root@192.168.106.116` 在 2026-07-30 的只读 SSH 探测中返回认证失败，因此本轮既没有 -取得容器/版本证据，也没有对该主机、数据库或 OpenFGA 做任何修改。 - -本次范围结论： - -- T140 以“记录范围决策和未执行证据报告”收口； -- 可以确认 F048 功能、DDL 与正式数据迁移/校验脚本开发完成； -- 不把该结论表述为目标环境已完成发布验证； -- 不执行生产数据迁移； -- 不声明 OpenFGA 性能无损； -- 不通过 model A/B、切 Store、预演或应用级回滚规避门禁。 +| Grant 即可见、模型 inactive 不撤权 | visibility compiler / grant policy / migration tests | PASS | +| 多来源最后撤销才删除聚合 tuple | visibility projection/reconcile tests | PASS | +| 单资源、BatchCheck、完整 stream 同语义 | permission service + 真实 OpenFGA | PASS | +| joined 为完整 visible ID-first、排除本人、管理员无扩权 | `test_space_joined_visible_ids.py` | PASS | +| joined 5,000 上限与无静默截断 | permission service / joined tests | PASS | +| department 候选后一次分块 BatchCheck | department permission tests | PASS | +| file 跨批扫描、稳定 cursor、最终子项 visible | file pagination tests | PASS | +| OpenFGA 异常无 SQL fallback | permission service / business authorization tests | PASS | +| 来源/列表指标与告警、不记录 PII | visibility observability tests | PASS | +| inactive 行两端保留、target 过滤、精确 MOVE/REMOVE | Platform 5 tests / Client 9 tests | PASS | +| 删除零引用/零 residual、停用非前置 | ModelEditor / catalog / D4 verifier tests | PASS | +| 旧系统单 run 迁移、无 A/B 或二次迁移 | migration CLI/coordinator/verifier tests | PASS | + +## 5. 未关闭的环境项与基线项 + +### 5.1 环境门控 + +- 真实 MySQL/DM8：新来源投影表、唯一键、索引、cursor/checkpoint 和 contribution 用例已编写， + 但未提供 `F048_MYSQL_TEST_DSN` / `F048_DM8_TEST_DSN`，2 个 live 用例跳过。 +- 真实业务 API E2E：`test_e2e_f048_permission_model_grants.py` 使用 + `e2e-f048-permission-` 前缀双重清理；当前未提供专用部署和三类账号，6 个用例跳过。 +- 页面手工验证、5,001 条真实容量、OpenFGA 故障注入、D0～D6 迁移及生产脱敏 BENCH + 尚未执行，不能据此批准发布。 + +### 5.2 F048 扩展全集的 11 个失败 + +`pytest test/permission/test_f048_*.py ...` 得到 `562 passed, 7 skipped, 11 failed`。复跑首个失败 +及相关单文件后，失败边界为： + +- decision/grant/dashboard 的旧 HTTP/Service 测试没有注入新的 async `permission_actor` 身份解析 + 依赖，单文件同样会尝试初始化未注册的 `permission_runtime`；这是测试 harness 未随共享身份 + 入口更新，不是此次 visible 集合断言失败； +- 既有 domain-boundary 测试仍识别到 `grant_subject_service.py` 中 5 个历史业务 ORM import； +- 两个 Linsight 用例的预期错误文案仍为旧 `F048 OpenFGA runtime`，当前实现返回 + `permission runtime is not ready`； +- 一个 migration runtime 用例只在扩展全集顺序下缺少 tenant Context，单独复跑通过，属于测试 + Context 隔离问题。 + +这些失败未指向本次单槽 visible 语义结果不一致，但仍是可复现的历史测试/架构债务，不能将 +“扩展全集”标成 PASS。若发布门禁要求整个 F048 历史全集全绿，应先修复共享测试 harness、 +domain boundary 与 Context 隔离后复跑。 + +## 6. 发布判定 + +本次开发代码可进入后续专用集成环境验证；当前报告判定为 **PARTIAL**，不是发布批准。 +完成清单中的真实 MySQL/DM8、业务 API/UI、故障注入、5,001 容量和 D0～D6 后，才可将状态 +更新为 PASS。 diff --git a/features/v3.0.0-beta1/048-rebac-permission-model-grants/product-authorization-model-and-migration-guide.md b/features/v3.0.0-beta1/048-rebac-permission-model-grants/product-authorization-model-and-migration-guide.md index 815f3070de..cc91ca75f7 100644 --- a/features/v3.0.0-beta1/048-rebac-permission-model-grants/product-authorization-model-and-migration-guide.md +++ b/features/v3.0.0-beta1/048-rebac-permission-model-grants/product-authorization-model-and-migration-guide.md @@ -306,7 +306,8 @@ Grant、资源关联和 assignee 都不需要重写。 同理： -- 停用模型：移除或关闭模型的 `active` 投影； +- 停用模型：只关闭该模型后续被选中用于新增或变更授权的能力，已有 Grant 的 OpenFGA + action 与 visible 投影保持不变； - 修改同级授权策略：只调整该模型可授予的目标等级能力； - 删除某动作：只更新模型自身的动作投影。 @@ -507,7 +508,7 @@ OpenFGA 是执行面的最终裁决者，MySQL 是控制面的业务真相。 |---|---| | 动作名称、等级、适用资源 | 当前用户是否能执行具体动作 | | 标准/自定义模型定义 | 沿模型、Grant、主体和 parent 计算 | -| 模型 active 与同级策略配置 | `can_edit`、`can_download` 等最终结果 | +| 模型 active（仅控制后续可分配性）与同级策略配置 | `can_edit`、`can_download` 等既有 Grant 的最终结果 | | Grant、assignee 和来源明细 | 用户、部门、用户组的关系集合计算 | | 权限模式和界面展示 | 资源继承关系的权限传播 | | 审计、发布状态、迁移记录 | Check、ListObjects、ListUsers | @@ -851,7 +852,7 @@ tuple 本身属于 Store，不属于某个 model ID：迁移器按 Store - `edit`、`download`、`delete`、`manage_permission`； - dashboard visible/edit/delete/manage_permission； - 文件预览不做 action Check，原件/打包下载检查 `download`； -- 模型 active、动作等级和同级授权策略； +- 模型 active 只影响新增/变更授权且不改变既有 Grant、动作等级和同级授权策略； - SQL 与现有 Store 的来源/目标计数、checksum 和 blocker； - 首批已迁移资源类型的旧四档/旧模型关系为零；旧 Config 原始行保留数仅用于审计， 其运行时引用必须为零； @@ -1042,12 +1043,15 @@ ListObjects。正式 relation `visible`（即这里讨论的 `can_view` 语义 ### 模型 active 有什么作用？ -它是模型的总开关： +它是模型的“可分配”开关，不是运行时撤权总开关： - inactive 模型不能新增授权； -- 既有 Grant 保留用于审计； -- 既有 Grant 不再产生可见性、具体动作或授权他人的能力； -- 不需要逐个删除所有 Grant。 +- 既有 Grant、可见性、具体动作和授权他人的能力保持不变； +- 如果要删除模型，必须先逐项撤销或替换全部 Grant/assignee 绑定； +- 只有引用、来源投影和残留 OpenFGA tuple 都清零后，最终删除才会成功。 + +因此模型停用不需要 A/B 可见槽。单槽浅层 `visible` 只在 Grant/assignee 来源新增、撤销、 +替换或资源生命周期变化时更新。 ### 标准模型修改“允许同级”会发布新 Authorization Model 吗？ diff --git a/features/v3.0.0-beta1/048-rebac-permission-model-grants/spec.md b/features/v3.0.0-beta1/048-rebac-permission-model-grants/spec.md index 1ba51d3101..fccb8f1591 100644 --- a/features/v3.0.0-beta1/048-rebac-permission-model-grants/spec.md +++ b/features/v3.0.0-beta1/048-rebac-permission-model-grants/spec.md @@ -7,7 +7,7 @@ > OpenFGA DSL、tuple 形状、API、Service、脚本参数、批次和部署命令属于后续 > [design.md](./design.md) 与 [tasks.md](./tasks.md)，不在本文档重复。 -**状态**：✅ Spec、Design ★、Tasks ★ 已确认；功能与迁移脚本开发完成；本地 E2E 经用户确认不执行 +**状态**：原 F048 Spec、Design ★、Tasks ★ 已确认；本次“高效可见资源枚举”Spec 迭代已由用户于 2026-08-13 指示进入 Design **关联 PRD**：[3.0-beta1 ReBAC 逻辑优化](https://dataelem.feishu.cn/wiki/TYmHw4nPzitTQnkUjW8c5NFKnQg) @@ -42,7 +42,7 @@ - 将按资源和目标角色重复的细粒度权限收敛为统一动作目录。 - 为动作配置唯一的 1～4 级，并由等级生成四个累计标准模型。 -- 支持自定义模型、模型生效状态，以及模型级 `manage_permission` 同级授权策略。 +- 支持自定义模型、模型可分配状态，以及模型级 `manage_permission` 同级授权策略。 - 以 PermissionModel 与 PermissionGrant 表达“模型能力”和“资源授权主体集合”。 - 让进入资源 ReBAC 的请求以 OpenFGA 对具体动作的结果作为唯一权限结论。 - 支持用户、部门和既有合法主体类型的多来源授权并集与独立撤销。 @@ -59,6 +59,14 @@ - 提供逐项映射、幂等断点续跑、迁移后校验、启服门禁、前向修复和旧路径退役。 - 数据库结构变更只通过 DDL-only Alembic revision 完成；旧权限数据、Config 与 OpenFGA tuple 的读取、转换、回填、清理和校验由 `src/backend/scripts/` 下的专用数据迁移脚本完成。 +- 根据各业务列表的实际数据规模、可见比例、继承比例和分页特征选择权限筛选路径；允许先取得 + 完整有效可见资源 ID 集合后查询业务数据，也允许先取得业务候选再批量筛选并按需续取， + 但选型必须有代表性业务数据和性能测量支撑。 +- 保证 Grant/主体来源撤销及自定义模型删除前的绑定清理完成后，有效可见结果及时、完整 + 收敛，且不影响同一主体从其他仍有效来源获得的可见性；模型停用只禁止新增授权，已有 + 授权继续生效。 +- 首个业务验收范围包括知识空间“我加入的”：用户可见且非本人创建的知识空间均应纳入， + 不限于直接授权或成员表来源。 ### 本次明确排除 @@ -78,6 +86,8 @@ 仅作为 OpenFGA 不可修改/删除的历史版本存在。 - 不在 Alembic revision、应用启动钩子、API 或 Celery 中执行权限数据迁移；服务不会在 启动时自动回填旧数据。 +- 不在本 Spec 固化可见关系的 OpenFGA DSL、tuple 形状、写入批次、分页协议或缓存实现。 +- 不让资源可见性替代编辑、下载、搜索、RAG、删除或权限管理等具体动作鉴权。 --- @@ -90,8 +100,11 @@ | 标准模型 | 查看者、编辑者、管理者、所有者四个系统模型，分别累计包含 1、1～2、1～3、1～4 级动作 | | 自定义模型 | 管理员选择有效动作形成的模型；等级由最高动作等级自动得出 | | PermissionGrant | 某一资源与某一 PermissionModel 的授权集合；其 assignee 可以是用户、部门或既有合法主体 | -| 模型 `active` | 模型是否可被授权以及其既有 Grant 是否能产生权限的总开关；不是业务动作，也不参与等级计算 | +| 模型 `active` | 模型是否可被用于新增或变更授权的可分配开关；不是运行时撤权开关，不参与等级计算，停用不影响已有 Grant 的可见性和具体动作 | | 资源可见 | 主体至少通过一个有效 Grant 与资源建立关系后，可以看到资源列表项和基础元数据 | +| 有效可见集合 | 指定主体、租户和资源类型下，由全部当前有效授权来源与明确系统级可见来源共同形成的完整资源 ID 集合；它只证明资源可见，不证明任何具体动作 | +| 控制面真相 | 权限模型、Grant、Grant 主体和权限模式以权限控制面的 canonical 持久事实为准；组织成员、系统身份和业务资源状态以各自 Owner 业务域的 canonical 事实为准 | +| 权限执行面 | OpenFGA 对已发布控制面事实进行 Check、List 和 Write 解释，并为进入资源 ReBAC 的请求给出最终权限结论；它不是数据库故障时可绕过的缓存 | | 具体动作 | 编辑、下载、删除、分享、使用、权限管理等业务行为；必须按该动作本身判定 | | `INHERIT` | 普通授权沿直接权限父级计算，当前资源不能维护普通本级成员 | | `CUSTOM` | 当前资源形成权限边界，只使用本级普通 Grant；业务结构父级仍保留 | @@ -113,7 +126,8 @@ 3. 撤销用户 `001` 的直接授权后，只删除模型 `900` 的来源； 4. 部门来源仍在，因此 `download` 继续允许，`edit` 在没有其他来源时拒绝； 5. 如果模型 `900` 成功增加 `share`，所有引用模型 `900` 的 Grant 同时获得该动作，不逐资源、逐主体重建授权； -6. 如果模型 `900` 被停用，其全部 Grant 保留用于审计，但不再产生可见性、`edit` 或 `share`。 +6. 如果模型 `900` 被停用，不能再用它新增或变更授权，但已有 Grant 继续产生可见性、 + `edit` 和 `share`；如需删除模型，必须先撤销或替换这些 Grant/assignee 绑定。 --- @@ -145,6 +159,16 @@ 的升级方式，在迁移完成后只额外重启一次服务；外部流量在整个过程保持关闭，失败时保持维护 并前向修复，在新逻辑稳定后确认旧路径已退役。 +### US-07 资源列表使用者 + +作为资源列表使用者，我希望每个列表根据实际业务数据规模和可见分布选择总体成本更低的权限 +筛选路径，并在数据分布变化后能够重新评估，而不是由资源层级或单一经验永久决定。 + +### US-08 权限模型维护者 + +作为权限模型维护者，我希望停用自定义模型时只禁止后续使用它授权、已有授权不受影响； +需要删除模型时，系统先要求我撤销或替换全部绑定，并在绑定清零后完成残留关系对账。 + --- ## 3. 验收标准 @@ -175,7 +199,7 @@ - **AC-05** — THE SYSTEM SHALL 不再把 `view_space`、`view_folder`、`view_file`、`view_channel`、`view_app`、`view_kb`、`view_tool` 作为可配置动作展示。 - **AC-06** — WHEN 动作等级成功变化, THE SYSTEM SHALL 按新等级重新得出四个标准模型的累计动作结果，并将影响范围提供给管理员确认和审计。 -### 3.2 标准模型、自定义模型与生效状态 +### 3.2 标准模型、自定义模型与可分配状态 - **AC-07** — THE SYSTEM SHALL 固定提供查看者、编辑者、管理者、所有者四个标准模型，其等级分别为 1、2、3、4。 - **AC-08** — THE SYSTEM SHALL 使查看者包含全部 1 级有效动作、编辑者包含 1～2 级、管理者包含 1～3 级、所有者包含 1～4 级。 @@ -185,9 +209,9 @@ - **AC-12** — WHEN 自定义模型没有动作、包含未分级动作或包含对其声明范围不合法的动作, THE SYSTEM SHALL 将空动作模型显示为“未定级”并拒绝保存不合法结果。 - **AC-13** — WHEN 自定义模型成功增加或移除动作, THE SYSTEM SHALL 让所有引用该模型的有效 Grant 使用同一新动作结果，不要求重新给每个资源或主体授权。 - **AC-14** — WHEN 模型变更将影响既有 Grant, THE SYSTEM SHALL 在生效前展示受影响的资源数、Grant 数和主体来源数，并要求明确确认。 -- **AC-15** — WHILE 模型为 inactive, THE SYSTEM SHALL 禁止新增授权，并使其既有 Grant 不产生资源可见性、具体动作或授权他人的能力。 +- **AC-15** — WHILE 模型为 inactive, THE SYSTEM SHALL 禁止使用该模型新增 Grant assignee、把既有 assignee 变更到该模型或复制出新的该模型绑定；其停用前已存在且仍有效的 Grant/assignee SHALL 继续产生与停用前一致的资源可见性、具体动作和权限管理能力。 - **AC-16** — IF 模型更新未完整生效, THEN THE SYSTEM SHALL 继续使用更新前的完整有效状态，不得呈现部分动作已生效的混合结果。 -- **AC-17** — WHEN 自定义模型仍被 Grant 引用, THE SYSTEM SHALL 禁止不可恢复的直接删除；管理员可以停用模型，并在引用清零或完成替换后执行最终清理。 +- **AC-17** — WHEN 自定义模型仍被任何有效或待处理 Grant/assignee 引用, THE SYSTEM SHALL 禁止删除；管理员必须先撤销或替换全部绑定，待引用计数、来源投影和残留 OpenFGA tuple 对账均为零后，才能执行最终删除。停用不是删除前的强制步骤，也不得自动撤销已有绑定。 - **AC-18** — WHEN 管理员选择“协作编辑”“权限管理”“高级管理”或其他预设创建自定义模型, THE SYSTEM SHALL 只用预设初始化动作选择；最终动作和等级以保存内容为准，预设后续变化不得自动改写已保存模型。 ### 3.3 PermissionGrant 与多来源并集 @@ -200,11 +224,11 @@ - **AC-24** — WHEN 用户退出某部门但仍有直接授权, THE SYSTEM SHALL 仅失去该部门来源的动作，直接授权继续有效。 - **AC-25** — WHEN 完全相同的资源、模型、主体和来源被重复提交, THE SYSTEM SHALL 得到幂等结果，不创建重复的有效授权。 - **AC-26** — WHEN 同一主体对同一资源同时拥有不同模型, THE SYSTEM SHALL 分别保留各模型来源，不以“最高等级模型”覆盖较低等级模型的独有动作或授权策略。 -- **AC-27** — WHEN Grant 关联的模型 inactive、缺失或不合法, THE SYSTEM SHALL 使该 Grant fail closed，并将异常暴露给审计和修复流程。 +- **AC-27** — WHEN Grant 关联的模型缺失、已删除或定义不合法, THE SYSTEM SHALL 使该 Grant fail closed，并将异常暴露给审计和修复流程；模型仅为 inactive 不属于异常，既有 Grant 继续有效。 ### 3.4 资源可见性与具体动作最终鉴权 -- **AC-28** — WHEN 主体通过至少一个 active 模型和有效 Grant 与资源建立关系, THE SYSTEM SHALL 允许其看见该资源的列表项和基础元数据。 +- **AC-28** — WHEN 主体通过至少一个定义有效且未删除的模型和有效 Grant 与资源建立关系, THE SYSTEM SHALL 允许其看见该资源的列表项和基础元数据；模型是否仍可用于新增授权不影响既有 Grant。 - **AC-29** — WHEN 主体没有任何有效 Grant 且没有其他明确的系统级可见关系, THE SYSTEM SHALL 不返回该资源。 - **AC-30** — WHEN 请求通过 C4 明确的系统级身份策略后仍需进入资源 ReBAC，且业务操作需要某一具体动作, THE SYSTEM SHALL 以 OpenFGA 对该动作的结果作为最终权限结论。 - **AC-31** — WHEN 列表候选已经按该列表所要求的具体动作完成权限筛选, THE SYSTEM SHALL 直接使用该结果，不再调用其他数据源做第二次权限裁决。 @@ -213,6 +237,27 @@ - **AC-34** — IF 一个需要进入资源 ReBAC 的请求遇到 OpenFGA 不可用、超时或结果不可判定, THEN THE SYSTEM SHALL 返回明确失败，不得改用旧四档关系、Config binding、creator 或数据库细粒度权限产生 ALLOW；既有系统级身份策略必须在请求进入 ReBAC 前明确确定，不能因故障临时启用。 - **AC-35** — WHERE 多租户开启, THE SYSTEM SHALL 只对业务层已限定在当前租户范围内的资源执行权限判定，且不得解析跨租户 Grant。 +#### 3.4.1 高效可见资源枚举与来源收敛 + +- **AC-159** — WHEN 主体通过任一有效 Grant（包括由部门、用户组等组织主体关系解析出的有效 Grant）或明确的系统级资源可见来源获得资源授权, THE SYSTEM SHALL 将该资源纳入其有效可见集合；组织成员关系本身不得脱离资源授权来源单独产生可见性，资源可见也不得依赖模型是否包含某个可配置业务动作。 +- **AC-160** — WHEN 代表性业务数据证明“先取得完整可见资源 ID，再查询业务数据”的总体查询与权限成本更低, THE SYSTEM SHALL 支持该列表采用可见资源 ID 优先路径，并保证返回集合完整、租户隔离正确且后续业务条件仍被应用。 +- **AC-161** — THE SYSTEM SHALL 让同一当前生效权限版本、同一一致性要求和同一资源范围下的单资源 `visible` 结果与可见资源枚举遵循相同语义；任一资源不得长期出现单资源结果为 ALLOW 但枚举缺失，或单资源结果为 DENY 但枚举返回。 +- **AC-162** — WHEN 采用完整可见资源 ID 枚举的列表，其结果超过一次响应能够完整承载的数量, THE SYSTEM SHALL 提供可继续取得全部结果的明确合同，或返回明确的容量错误；不得以成功响应静默截断或返回不完整集合。 +- **AC-163** — WHEN 平台超管或租户管理员访问面向个人的资源可见列表, THE SYSTEM SHALL 按与普通用户相同的有效资源来源计算可见集合，不得仅因管理员身份自动获得平台或租户全部内容资源；系统管理入口的管理范围继续由独立业务身份策略决定，且不得伪装成个人资源可见来源。 +- **AC-164** — WHEN 自定义模型被停用, THE SYSTEM SHALL 只阻止后续使用该模型新增或变更绑定，不得删除该模型既有 Grant/assignee、来源投影或可见关系，也不得改变这些既有授权产生的具体动作结果。 +- **AC-165** — WHEN 自定义模型仍被任一 Grant 引用, THE SYSTEM SHALL 继续按 AC-17 阻止最终删除；WHEN 引用已清零且执行最终删除, THE SYSTEM SHALL 在报告删除成功前确认该模型不再贡献任何有效可见结果，并清理所有能够追溯到该模型的残留可见关系。 +- **AC-166** — WHEN Grant、Grant assignee、部门/用户组成员关系或其他可见来源被撤销, THE SYSTEM SHALL 只撤销该来源对有效可见集合的贡献；不得删除同一主体从其他仍有效来源获得的可见性。 +- **AC-167** — IF 来源撤销、绑定替换或删除前清理的权限发布未完整生效, THEN THE SYSTEM SHALL 不得报告该项绑定变更或模型删除已成功，不得用模型删除掩盖残留来源，并必须保留可审计、可重试和可对账的失败状态。模型停用只变更可分配状态，不触发已有授权清理。 +- **AC-168** — THE SYSTEM SHALL 使每条有效可见结果都能追溯到当前有效的模型/Grant/主体来源，或可识别的组织、系统、共享等 Owner 业务域事实；无法追溯的残留可见关系不得继续产生 ALLOW，并必须进入修复与审计范围。 +- **AC-169** — THE SYSTEM SHALL 以权限控制面的 canonical 持久事实作为 PermissionModel、PermissionGrant、Grant assignee 和权限模式的唯一控制面真相，以各 Owner 业务域的 canonical 记录作为组织成员、系统身份、共享关系和资源状态的控制面真相；不得把同一事实复制成另一套可独立编辑的业务真相。 +- **AC-170** — THE SYSTEM SHALL 以 OpenFGA 作为上述事实发布后的权限执行面；SQL 或 Owner 业务域记录存在但尚未在 OpenFGA 生效时，不得报告权限已生效，也不得在 OpenFGA 拒绝、超时或不可用时直接用数据库记录产生 ALLOW。 +- **AC-171** — WHEN 权限执行面出现缺失、重复或无来源的可见结果, THE SYSTEM SHALL 能从控制面 canonical 事实进行对账与收敛，并保证修复只影响差异来源，不改变其他合法来源。 +- **AC-172** — WHEN 查询知识空间“我加入的”, THE SYSTEM SHALL 返回用户有效可见集合中由他人创建的知识空间；直接授权、部门、用户组、主动订阅及其他合法来源只要最终产生有效可见性都应纳入，不得以某一种成员登记或成员角色作为必要条件，且本人创建的知识空间继续排除。 +- **AC-173** — WHEN 代表性业务数据证明知识空间文件或文件夹列表中大部分候选继承父级权限、候选可见率较高，且“先取业务候选再批量筛选”的总体成本更低, THE SYSTEM SHALL 先按既有业务范围、排序和稳定游标取得一批候选，再批量筛选候选的可见性；若可见项不足请求条数，则继续取得后续候选并重复筛选，直到达到请求条数或业务候选耗尽。 +- **AC-174** — WHEN 候选列表中的文件或文件夹通过父级继承或本级自定义权限获得可见性, THE SYSTEM SHALL 按其最终有效权限决定是否返回；候选来源、父级继承概率或已对父知识空间的权限判定不得替代子资源最终可见性筛选。 +- **AC-175** — WHEN 为一个业务列表选择可见资源 ID 优先或业务候选优先路径, THE SYSTEM SHALL 基于具有代表性的实际业务数据评估资源候选总量、单用户可见数量及比例、继承与 `CUSTOM` 比例、业务过滤选择性、请求页大小、填满一页所需候选扫描量、权限调用成本和结果完整性；不得仅按资源类型、层级或理论复杂度决定。 +- **AC-176** — WHEN 已选列表路径所依据的数据分布发生显著变化，或另一条路径经同口径测量后总体成本更低, THE SYSTEM SHALL 允许重新评审和调整路径；调整前后必须保持相同的业务过滤、排序、分页连续性和最终权限语义。 + ### 3.5 `manage_permission` 与可授予等级 每个包含 `manage_permission` 的模型独立选择以下策略之一： @@ -397,6 +442,11 @@ 不得保留旧 binding 写入入口或 dual/legacy model client。 - **AC-101** — IF 启服后安全验证失败, THEN THE SYSTEM SHALL 重新进入维护、停止权限写入并 在新 model 与新控制面上前向修复；不得恢复旧 model、回放旧 Config、逐请求 fallback 或静默放行。 +- **AC-177** — WHEN F048 从尚未上线前的旧权限系统执行正式迁移, THE SYSTEM SHALL 在同一个 + PermissionMigrationRun 中从旧 Config、旧四档关系和 Owner 业务事实生成最终 PermissionModel、 + Grant、Assignee 及其完整可见执行投影；不得先上线或迁入一个缺少展平可见关系的中间 F048 + model，也不得为本次修改另建“旧 F048 → 新 F048”的第二套生产迁移。启服前必须证明单资源 + visible、批量 visible 与完整可见枚举均和同一 canonical 目标来源集合一致。 ### 4.7 启服后监控、前向修复与启服前旧路径退役 @@ -581,9 +631,9 @@ MySQL/DM8 关系表是权限配置、模型定义和绑定关系的控制面真 | EC-02 | 用户同时通过直接、部门、用户组获得相同模型 | 权限只计算一次语义结果，但来源分别保留；撤销任一来源不影响其余来源 | | EC-03 | 用户的多个模型有相同等级但动作不同 | 动作取并集；不得只保留一个“最高等级”模型 | | EC-04 | 一个模型允许同级、另一个同级模型禁止同级 | 分别计算后取可授予目标并集，不使用全局开关 | -| EC-05 | 模型被停用但仍有大量 Grant | 立即停止产生权限，Grant 留作审计和恢复，不触发资源×主体重写 | +| EC-05 | 模型被停用但仍有大量 Grant | 只禁止新增或变更到该模型；已有 Grant 与权限结果保持不变，不触发资源×主体重写 | | EC-06 | 动作从低等级调整到高等级 | 标准模型累计结果与引用该动作的自定义模型派生等级按新版本一致生效 | -| EC-07 | 动作被停用导致自定义模型无有效动作 | 该模型不得继续 active；系统 fail closed 并提示修复 | +| EC-07 | 动作被停用导致自定义模型无有效动作 | 该模型不得继续 active 或用于新增授权；已有 Grant 仅保留 visible、不产生已移除动作，并提示修复或逐项替换/撤销 | | EC-08 | `CUSTOM` 资源在业务树中移动 | 结构父级改变，本级普通 Grant 保留，权限不自动继承新父级 | | EC-09 | `INHERIT` 资源的父级被删除 | 删除方必须重挂、级联处理或拒绝；不能留下悬空权限来源 | | EC-10 | OpenFGA 返回未知或超时 | 请求明确失败，任何旧路径不得补充 ALLOW | @@ -597,6 +647,16 @@ MySQL/DM8 关系表是权限配置、模型定义和绑定关系的控制面真 | EC-18 | 被编辑的旧系统模型与新标准模型动作不一致 | 创建历史自定义模型快照并迁移其原绑定，不污染新标准模型 | | EC-19 | `include_children=true` 曾把一个根部门展开成多个子部门 tuple | 仅保留根部门 assignee 与范围，展开 tuple 只参与核对，不形成独立授权 | | EC-20 | Config JSON 损坏、binding 指向缺失模型或同一 tuple 匹配冲突 binding | 记录原始定位并阻断相关迁移，不使用默认模型、空动作或最高等级猜测 | +| EC-21 | 被停用模型关联大量 Grant，且部分用户还有其他来源 | 该模型贡献和其他来源均保持不变；若后续逐项撤销该模型绑定，只在最后一个有效来源消失时删除聚合可见关系 | +| EC-22 | 自定义模型引用清零后仍存在可追溯到该模型的残留可见关系 | 最终删除不得报告成功；完成残留清理和对账后才可成功 | +| EC-23 | 同一空间同时由直接授权、部门、用户组和订阅产生可见性 | 任一来源撤销不影响其余来源；最后一个有效来源撤销后空间才从有效可见集合消失 | +| EC-24 | 用户可见资源数量超过单次响应上限 | 明确继续获取全部结果或返回容量错误，不得用 HTTP 成功响应静默截断 | +| EC-25 | 超管没有任何普通用户可见来源 | 个人资源可见列表不因超管身份返回平台全部资源；管理入口仍按独立身份策略工作 | +| EC-26 | SQL/Owner 业务域已有授权事实，但 OpenFGA 发布失败 | 权限不得报告已生效；保留失败状态并重试、对账，不从数据库直接补充 ALLOW | +| EC-27 | 文件列表首批业务候选经权限筛选后不足一页 | 继续从稳定业务游标取得后续候选并批量筛选，直到填满一页或候选耗尽；不得提前返回伪造的“已结束”状态 | +| EC-28 | 文件列表大多数子资源继承父知识空间，仅少量子资源为 `CUSTOM` | 若实测支持候选优先路径，则仍对候选执行统一的最终可见性筛选；不得因高继承比例默认全部候选可见 | +| EC-29 | 同一列表在不同规模环境中的可见比例明显不同 | 分别按同口径数据评估总体成本；不能把一个小数据环境的结论未经验证直接固化到大数据环境 | +| EC-30 | 可见 ID 优先与候选优先的实测成本接近 | 优先保证完整性、稳定分页和实现可维护性，并在 Design 中记录选择依据与重新评估条件 | --- @@ -604,23 +664,23 @@ MySQL/DM8 关系表是权限配置、模型定义和绑定关系的控制面真 ### 6.1 安全 -- **NFR-01** — 未知动作、未分级动作、inactive 模型、异常 Grant 和权限服务故障必须默认拒绝。 +- **NFR-01** — 未知动作、未分级动作、缺失或已删除模型、异常 Grant 和权限服务故障必须默认拒绝；inactive 只禁止新增或变更授权，不得让既有合法 Grant fail closed。 - **NFR-02** — 任何客户端提交的模型等级、可授予范围、来源标签和可编辑状态均不可信，服务端必须基于当前有效事实决定。 - **NFR-03** — 权限成员名单、组织信息和模型影响范围属于受保护数据，只向明确有权角色开放。 ### 6.2 一致性 - **NFR-04** — 模型发布、Grant 变更、权限模式切换和迁移批次必须幂等、可重试、可恢复，且不能对用户暴露半完成权限。 -- **NFR-05** — 安全相关撤权、模型停用和同级策略收紧在成功返回后不得继续被旧缓存放行。 +- **NFR-05** — 安全相关撤权、绑定替换、模型动作收紧和同级策略收紧在成功返回后不得继续被旧缓存放行；模型停用不改变既有授权结果。 - **NFR-06** — 所有资源授权结果必须能追溯到同一租户内的有效模型、Grant 和主体来源。 ### 6.3 性能与规模 -- **NFR-07** — 修改一个已被大量资源和主体引用的 PermissionModel 时，授权数据写入量不得与“资源数 × 用户/部门数”成比例。 +- **NFR-07** — 修改 PermissionModel 的动作、等级、名称或同级策略时，不得重写资源授权；模型可分配状态改变时不得重建其派生可见执行投影。部门/用户组授权始终保留 userset，不得按成员展开为“资源数 × 用户数”。 - **NFR-08** — 新增或撤销一个 Grant assignee 时，不得重写同一 Grant 中其他 assignee 或该模型在其他资源上的授权。 - **NFR-09** — 部门和用户组授权不得按成员展开；组织成员变化不得触发资源授权全量重写。 - **NFR-10** — 资源列表继续满足既有 cursor、批量候选和有界扫描契约；不得恢复为租户全量加载后逐项权限过滤再分页。 -- **NFR-11** — 性能、容量和迁移时限的量化门槛在 Design 阶段基于生产脱敏分布确定，并作为发布门禁；本 Spec 不预设未经测量的数值。 +- **NFR-11** — 两类列表路径的量化性能与容量门槛必须基于 BENCH-01 和代表性业务数据在 Design 阶段确定，并作为发布门禁；任何成功响应都必须满足对应列表的完整性与分页合同，不得以性能优化为由静默漏项。 ### 6.4 兼容与可观测 @@ -628,11 +688,19 @@ MySQL/DM8 关系表是权限配置、模型定义和绑定关系的控制面真 - **NFR-13** — 必须能按租户、资源类型、动作、模型、Grant 和来源观察权限判定、批量判定、候选枚举的数量、耗时、错误、拒绝原因及版本。 - **NFR-14** — 必须能观察模型影响范围、模型停用、Grant 变更、模式切换、迁移进度、差异、人工项、启服和前向修复状态。 +### 6.5 可见枚举专项 + +- **NFR-15** — 列表路径选型必须比较端到端总体成本，而不能只比较单次权限调用：可见资源 ID 优先路径计入可见枚举和业务详情查询成本，业务候选优先路径计入候选查询、为填满页面产生的扫描放大和批量权限判断成本。 +- **NFR-16** — 模型删除前的绑定清理和来源撤销必须有容量验收，证明大规模受影响关系下可按绑定幂等续跑、对账，且删除在引用与残留关系未清零时持续被阻断；模型停用不进入该清理流程。 +- **NFR-17** — WHEN 列表选择业务候选优先路径, THE SYSTEM SHALL 使用有界批量权限判断和稳定业务游标；可见项不足一页时允许继续扫描后续候选，并观测为填满页面产生的候选扫描放大。 +- **NFR-18** — 列表选型基准必须记录测试数据是否来自生产代表性分布、样本规模和关键比例；纯合成数据可以验证权限引擎能力，但不能单独证明某个业务接口应永久采用哪条路径。 + --- ## 7. 已确认产品决策 -下列 OQ-01～OQ-07 已由用户在 2026-07-29 明确确认并回写；本节不存在剩余开放项： +下列 OQ-01～OQ-07 已由用户在 2026-07-29 明确确认，OQ-08～OQ-10 已由用户在 +2026-08-13 明确确认并回写；本节不存在剩余开放项： | ID | 决策项 | 2026-07-29 确认结果 | |----|-----------|---------------------| @@ -643,6 +711,9 @@ MySQL/DM8 关系表是权限配置、模型定义和绑定关系的控制面真 | OQ-05 | 四个标准模型的初始“是否允许授予同级”分别是什么？ | viewer=false、editor=false、manager=false、owner=true；无 `manage_permission` 时该值不产生能力 | | OQ-06 | 文件预览与下载如何鉴权？ | 预览不设置 PermissionAction；只有原件/打包下载检查 `download` | | OQ-07 | 既有 F018 owner 交接在 F048 后是否继续开放？ | 选择 A：F048 启服时退役 F018，本期不实现 owner transfer；创建者 protected owner 保持不可转让，其他 owner 仍可并存 | +| OQ-08 | 不同资源列表如何选择权限筛选路径？ | 以代表性实际业务数据比较两条路径的端到端总体成本后决定，不按资源层级设置永久禁令。当前预期是知识空间“我加入的”更适合可见 ID 优先、知识空间文件列表更适合业务候选优先，但两项都必须由对应数据分布与 BENCH 结果确认，并在分布变化后允许重新评审 | +| OQ-09 | 超管是否因系统身份在个人可见列表中自动看见全部资源？ | 否；个人内容可见范围与普通用户采用相同来源语义，管理入口的系统身份判断保持独立 | +| OQ-10 | 自定义模型停用或最终删除时如何处理已有授权和可见结果？ | 停用只禁止新增或变更到该模型，已有授权不受影响；删除必须先撤销或替换全部 Grant/assignee 绑定，引用、来源投影与残留 tuple 清零后才允许最终删除。因此不需要为模型停用引入 A/B 可见槽 | 以下结论已由当前方案确认： @@ -654,7 +725,7 @@ MySQL/DM8 关系表是权限配置、模型定义和绑定关系的控制面真 - 权限继承复用既有 `parent`，不新增 `permission_parent`。 - 复制资源时保留源资源的权限模式；`CUSTOM` 副本复制普通白名单，受保护授权按新资源规则生成。 - 对进入资源 ReBAC 的请求，OpenFGA 的具体动作结果是最终权限结论，不再由 Config 自定义模型做第二次裁决。 -- 模型 `active` 是可授权和可生效的总开关，不是业务动作。 +- 模型 `active` 只是可用于新增/变更授权的开关，不是运行时撤权开关；已有 Grant 不受停用影响。 - 现有模型定义、模型动作和资源绑定迁入规范化关系表；Config 大 JSON 在切换后不再参与运行时。 - Authorization Model 沿用现有 Store；运维停止业务流量和任务消费后，从 backend 容器发布 新 model、原地迁移并退役旧 tuple，校验后重启服务且只启用新 model，再由运维恢复访问。 @@ -668,6 +739,15 @@ MySQL/DM8 关系表是权限配置、模型定义和绑定关系的控制面真 - user-owned 资源必须建立受保护 owner；经 adapter 代码 allowlist 与 canonical business predicate 双重确认的 platform system-owned 资源不得伪造用户 owner，只由既有 Constitution C4 system identity 管理。 +- 权限模型、Grant、Grant 主体和权限模式的控制面真相位于规范化 MySQL/DM8 表；部门、 + 用户组、系统身份、共享与资源状态继续以各自 Owner 业务域为真相；OpenFGA 是发布后的 + 唯一权限执行面，不是可以用数据库结果绕过的缓存。 +- 每个有效授权来源都应产生资源可见性；个人可见列表不因超管/租户管理员身份自动扩大， + 具体管理范围由独立业务入口判断。 +- “先枚举可见 ID”和“先取业务候选再批量筛选”都是合法路径；选择依据是对应接口的实际 + 候选规模、可见比例、继承比例、分页扫描放大和两条路径端到端实测成本。当前对知识空间 + “我加入的”和文件/文件夹列表的方向已由 Design 登记为待 BENCH-01 第二阶段确认的预期， + 不是永久禁令。 --- @@ -686,6 +766,14 @@ MySQL/DM8 关系表是权限配置、模型定义和绑定关系的控制面真 所有阻断项为零，其他差异均有批准记录；脚本输出和退出码已归档。 - MySQL、DM8、Platform、Client、频道入口、后台任务及既有高频列表性能回归通过。 - 同 Store 新 model 启服、前向故障处置和旧运行数据退役均完成验证。 +- BENCH-01 覆盖大资源量、多档可见范围和候选优先分页场景，分别给出两条路径在代表性业务 + 数据分布下的端到端成本、完整性和适用边界；报告和原始结果随发布证据归档。 +- 自动化验收覆盖停用后已有授权保持且新增授权被拒绝、引用未清零时删除阻断、引用清零后最终删除、多个可见来源并存、 + 最后来源撤销、发布失败和残留可见关系对账，证明 Check 与可见资源枚举一致。 +- 知识空间“我加入的”回归覆盖直接授权、部门、用户组、主动订阅、本人创建排除和超管场景， + 且不依赖某一种成员登记或成员角色得出候选空间。 +- 知识空间文件/文件夹列表回归覆盖父级继承、子资源 `CUSTOM`、首批过滤不足一页、候选耗尽、 + 稳定排序与游标连续性，证明不会漏项、重复、越权或先枚举全部可见子资源。 --- @@ -706,6 +794,7 @@ MySQL/DM8 关系表是权限配置、模型定义和绑定关系的控制面真 - [v3.0.0-beta1 Release Contract](../release-contract.md) - [产品可读：Authorization Model 接管与数据迁移说明](./product-authorization-model-and-migration-guide.md) +- [BENCH-01：大资源量、小可见范围的可见枚举性能初版报告](./bench-01-flat-visible-report-20260813.md) - [旧版 F004 ReBAC Core](../../v2.5.0/004-rebac-core/spec.md) - [旧版 F006 权限数据迁移](../../v2.5.0/006-permission-migration/spec.md) - [旧版 F007 Resource Permission UI](../../v2.5.0/007-resource-permission-ui/spec.md) diff --git a/features/v3.0.0-beta1/048-rebac-permission-model-grants/tasks.md b/features/v3.0.0-beta1/048-rebac-permission-model-grants/tasks.md index 935d153d65..23d8fe05c2 100644 --- a/features/v3.0.0-beta1/048-rebac-permission-model-grants/tasks.md +++ b/features/v3.0.0-beta1/048-rebac-permission-model-grants/tasks.md @@ -11,10 +11,10 @@ | 步骤 | 状态 | 备注 | |---|---|---| -| spec.md | ✅ 已评审 | 用户已确认 | -| design.md | ✅ 已评审 | 24 项复审 LGTM；用户于 2026-07-29 明确确认 Design ★ | -| tasks.md | ✅ 已确认 | 21 项 `/sdd-review ... tasks` 评审 LGTM；用户于 2026-07-30 明确确认 Tasks ★ | -| 实现 | ✅ 已完成 | 140 / 140 收口；功能与迁移脚本完成，本地 E2E 经用户于 2026-07-30 明确确认不执行 | +| spec.md | ✅ 已评审 | 原 Spec 已确认；新增可见性与停用/删除语义已经回写 | +| design.md | ✅ 已评审 | 24 项复审 LGTM；用户于 2026-08-13 明确确认 Design ★ | +| tasks.md | ✅ 已拆解 | T144～T194 已按单槽 visible、迁移、列表入口和旧迁移环境对账拆解 | +| 实现 | ⏳ 环境验证中 | T144～T188、T191～T194 已实现并完成定向回归；T189 真实 MySQL/DM8、业务 API/UI、故障注入与迁移窗口验证待执行，T190 待其闭环 | --- @@ -36,6 +36,11 @@ `task_prerun` 恢复 `current_tenant_id` ContextVar 的链路；不得用默认 tenant 猜测调用范围。 - 本计划不使用“测试降级”；需要 OpenFGA、MySQL、DM8 的测试标为中央集成环境执行，但仍保留 自动化断言，不把本 Feature 范围内验证延迟到未来。 +- T001～T143 记录的是本次可见性增量前的已完成实现基线，其中关于 inactive fail closed、 + 深层 visible 等描述不代表本次目标语义；T144～T194 是经本次 Design ★ 确认后的唯一增量 + 实施计划，不能把历史任务的完成标记当作单槽 visible/停用保留授权已经实现的证据。 +- 本增量不新增 Worker/Celery 调用入口；既有 Worker 若经共享 Permission facade 使用新模型， + 继续由 `before_task_publish` header 传 `tenant_id`、`task_prerun` 恢复 ContextVar，禁止默认租户。 ### 跨 Feature 与共享文件影响 @@ -44,7 +49,7 @@ | F004/F006/F007/F008 | F048 直接替代旧资源四档、Config 模型与成员 UI；T126～T134 必须证明旧路径对已迁移资源不可达 | | F017 shared resource | T063/T064 只保留精确 ID 跨 tenant 只读业务加载，不把 shared relation 改成普通 Grant | | F018 owner transfer | 按 OQ-07 与 INV-25，T049/T050 删除交接 API；其他 owner 仍作为 ordinary source 并存 | -| F027/F036/F040 列表性能 | T035/T131/T132 保留业务 cursor+BatchCheck 默认路径，ListObjects 只有 BENCH-01 通过的入口可启用 | +| F027/F036/F040 列表性能 | 原任务的“统一候选优先”结论已被本次 Design 取代；T160～T172 按代表性数据分别验证可见 ID 优先与业务候选优先，不恢复 fetch-all | | `core/openfga/client.py` / `discovery.py` / `manager.py` | T017～T020 是共享运行时变更；llm_server/llm_model 只保留显式 legacy allowlist，其他资源禁止 dual；latest 只用于启动发现并必须匹配 SQL CURRENT Catalog | | Constitution C4 | T027/T028 将 F048 原子路径切到 durable projection ledger；旧单 tuple 路径的 `failed_tuple` 不被误删 | @@ -1165,6 +1170,467 @@ --- +## Wave 9 — 单槽 visible 基础设施与权限核心 + +### 基础设施 + +- [x] **T144：补齐可见枚举错误码与领域 schema** + - **文件**：`src/backend/bisheng/common/errcode/permission.py`, + `src/backend/bisheng/permission/domain/schemas/f048.py` + - **逻辑**：登记 `25014 PermissionEnumerationIncomplete`；定义内部完整枚举请求/结果、 + `max_results`、正常结束状态和 source projection DTO。不得把部分结果包装成成功响应， + 不新增公开 ListObjects HTTP API。 + - **依赖**：T001, T002 + +- [x] **T145：建立单槽 visible source projection 表与 DDL** + - **文件**：`src/backend/bisheng/permission/domain/models/projection.py`, + `src/backend/bisheng/core/database/alembic/versions/v3_0_0_f048_visible_source_projection.py` + - **逻辑**：新增 `permission_visible_source_projection`，字段和唯一键严格按 Design §4.5.2， + 不含 `visibility_slot`；索引支持 resource/subject 聚合、model 引用清理和 migration item + 对账。revision 只做 MySQL/DM8 可移植 DDL；正式数据迁移开始后不提供应用级 downgrade， + revision downgrade 只允许在尚未创建 data migration run 时按 child→table 删除新结构。 + - **依赖**：T003, T005, T008 + +### 后端 Domain（Test-First） + +- [x] **T146：source projection Repository 合同测试** + - **文件**：`src/backend/test/permission/test_f048_repositories.py` + - **测试**：验证单槽 contribution 唯一性、同 subject 多来源引用计数、model/source cursor、 + operation/migration item 关联、tenant 自动过滤、残留 checksum 和幂等 retire；禁止 + `visibility_slot` 字段与跨租户聚合。 + - **覆盖 AC**：AC-165, AC-166, AC-168, AC-169, AC-171 + - **依赖**：T145 + +- [x] **T147：实现 source projection Repository** + - **文件**：`src/backend/bisheng/permission/domain/repositories/projection_repository.py` + - **逻辑**：提供 contribution upsert/retire、resource-subject active count、model 引用/残留 + cursor、operation checksum 和 migration batch API；只读写 permission 控制面表，不产生 ALLOW。 + - **验收**：T146 全部通过 + - **依赖**：T146 + +- [x] **T148：单槽 Authorization Model 语义测试** + - **文件**：`src/backend/test/permission/test_f048_authorization_model.py` + - **测试**：用真实 model JSON 证明 resource `visible` 只经过单槽 ordinary/protected/system/ + parent+mode，不存在 A/B/switch；inactive 模型的既有 Grant 继续 visible/具体 action/ + `manage_permission`，但模型缺失或已删除 fail closed；管理员身份不扩大个人 visible。 + - **覆盖 AC**：AC-15, AC-27, AC-28, AC-159, AC-161, AC-163, AC-164 + - **依赖**：T144 + +- [x] **T149：实现单槽 Authorization Model builder** + - **文件**：`src/backend/bisheng/core/openfga/authorization_model_f048.py` + - **逻辑**：删除 model active 对既有 Grant 的运行时交集和深层 visible 反向枚举；保留 Catalog + published/action 图，新增 ordinary/protected 浅层 visible relation，system/parent/mode 语义 + 不变；不得引入 `permission_visibility_switch` 或 `visible_a/visible_b`。 + - **验收**：T148 全部通过且 model checksum 更新 + - **依赖**：T148 + +- [x] **T150：模型停用与删除零引用门禁测试** + - **文件**：`src/backend/test/permission/test_f048_model_policy.py`, + `src/backend/test/permission/test_f048_catalog_service.py` + - **测试**：覆盖 inactive 禁止 ADD/MOVE target、既有 Grant 和 manage 能力保持、停用不产生 + visible delta、删除不要求先停用、任一 active/pending/failed Grant 或 source/live 残留均 + 阻断删除；零引用时新 Catalog 移除 model_key，RETIRED 历史快照保留。 + - **覆盖 AC**：AC-15, AC-17, AC-27, AC-164, AC-165, AC-167 + - **依赖**：T147, T149 + +- [x] **T151：实现模型可分配状态与删除协议** + - **文件**：`src/backend/bisheng/permission/domain/services/model_policy.py`, + `src/backend/bisheng/permission/domain/services/catalog_service.py` + - **逻辑**：把 `active` 收窄为 Grant command 的 target 可分配校验；impact 仍覆盖 inactive + 模型定义变化对既有 Grant 的 action 影响。`DELETE_MODEL` 在跨 tenant 引用和残留 checksum + 为零后，通过新 Catalog 不再包含 model_key 生效，不物理改写历史 release。 + - **验收**：T150 全部通过 + - **依赖**：T150 + +- [x] **T152：VisibilityProjectionCompiler 来源聚合测试** + - **文件**：`src/backend/test/permission/test_f048_visibility_projection.py` + - **测试**：覆盖 direct/department/subtree/group/protected、多模型、多来源、visibility-only + 模型和 inactive 既有 binding；同 resource/relation/subject 只写一个 live tuple，撤销一个来源 + 保留其他来源，最后来源才删除；system 来源不写入 Grant source projection。 + - **覆盖 AC**：AC-159, AC-164, AC-165, AC-166, AC-168, AC-169, AC-171 + - **依赖**：T147, T149 + +- [x] **T153：实现 VisibilityProjectionCompiler** + - **文件**：`src/backend/bisheng/permission/domain/services/visibility_projection_service.py`, + `src/backend/bisheng/permission/domain/services/projection_plan.py` + - **逻辑**：从 canonical Grant assignee 编译单槽 contribution、引用计数与 aggregate tuple delta； + contribution fingerprint 包含 source owner/model，聚合 key 不含 model/slot。编译器为纯授权投影， + 不查询业务表、不展开部门/用户组成员、不参与读取 fallback。 + - **验收**：T152 全部通过 + - **依赖**：T152 + +- [x] **T154：Grant mutation 与单槽投影原子性测试** + - **文件**：`src/backend/test/permission/test_f048_grant_sources.py`, + `src/backend/test/permission/test_f048_projection_service.py` + - **测试**：覆盖 ADD/MOVE/REMOVE 的 action+visible 同 operation 提交、inactive target 拒绝、 + inactive source 允许精确撤销、跨来源最后引用、50/51 change 和 90/91 tuple 边界、marker + 预置失败与 COMMIT_UNKNOWN；不得双写 A/B。 + - **覆盖 AC**：AC-15, AC-164, AC-166, AC-167, AC-170 + - **依赖**：T151, T153 + +- [x] **T155：接入 Grant mutation 单槽投影** + - **文件**：`src/backend/bisheng/permission/domain/services/grant_service.py`, + `src/backend/bisheng/permission/domain/services/projection_service.py` + - **逻辑**：在同一 SQL prepare 冻结 assignee 与 contribution after-state，预置 recent marker， + 一个 OpenFGA Write 提交 action 和单槽 visible delta，higher-consistency 校验后 finalize； + 编译后超过 90 整体拒绝，不跨批报告部分成功。 + - **验收**：T154 全部通过 + - **依赖**：T154 + +- [x] **T156：完整 visible 枚举 facade 测试** + - **文件**：`src/backend/test/permission/test_f048_fga_client.py`, + `src/backend/test/permission/test_f048_permission_service.py` + - **测试**：覆盖 StreamedListObjects 正常结束、去重、deadline/取消/服务错误、容量 + 5,000/5,001、tenant fence、recent marker consistency 和管理员无扩权；单资源 Check、 + BatchCheck 与完整枚举集合 checksum 相同，SQL/source projection 不补 ALLOW。 + - **覆盖 AC**：AC-160, AC-161, AC-162, AC-163, AC-168, AC-169, AC-170, AC-171 + - **依赖**：T149, T155 + +- [x] **T157：实现 StreamedListObjects 与 list_visible_objects** + - **文件**：`src/backend/bisheng/core/openfga/client.py`, + `src/backend/bisheng/permission/domain/services/permission_action_service.py` + - **逻辑**：client 完整消费 stream 并显式 model pin/consistency；permission facade 只在正常 + 结束且未超过调用方 `max_results` 时一次性交付不可变去重 ID 集。visible 路径只做 tenant + fence，不执行 super_admin/tenant_admin shortcut，不返回部分前缀。 + - **验收**：T156 全部通过 + - **依赖**：T156 + +- [x] **T158：单槽残留对账与恢复测试** + - **文件**：`src/backend/test/permission/test_f048_visibility_reconcile.py` + - **测试**：构造缺失、重复、无来源和 source/live checksum 混合集；证明只修复差异来源， + 不删除其他贡献；删除模型在 reconcile 完成前持续返回 25004，FAILED_CLOSED 不被脚本猜测放行。 + - **覆盖 AC**：AC-165, AC-167, AC-168, AC-170, AC-171 + - **依赖**：T153, T155 + +- [x] **T159：实现 visible source reconcile** + - **文件**：`src/backend/bisheng/permission/domain/services/visibility_projection_service.py`, + `src/backend/scripts/reconcile_f048_projection_operations.py` + - **逻辑**：从 canonical contribution 重算 aggregate checksum，默认 dry-run；`--apply` 只通过 + 领域 reconcile operation 补差异。Store/model/scope fence、ledger 或来源不完整时保持 + FAILED_CLOSED，脚本不得直接 UPDATE operation 状态或绕过 PermissionService 写 tuple。 + - **验收**：T158 全部通过 + - **依赖**：T158 + +--- + +## Wave 10 — BENCH 门禁与知识空间列表接入 + +### 性能门禁（先于业务入口） + +- [x] **T160：扩展 BENCH-01 性能合同测试** + - **文件**：`src/backend/test/permission/test_f048_performance_contract.py` + - **测试**：固定 10k/100k 资源、visible 10/100/1,000/5,000、direct/department/group/system/ + 多来源数据 checksum；比较单槽 ListObjects、20/50/100 BatchCheck 和业务 candidate scan， + 断言 stream 完整、无 A/B relation、结果不静默截断，并记录 `N_db/V/p` 与扫描放大。 + - **覆盖 AC**：AC-160, AC-161, AC-162, AC-163, AC-168, AC-175, AC-176 + - **依赖**：T157 + +- [x] **T161：实现 BENCH-01 v1.15.1 数据集与脚本** + - **文件**：`src/backend/scripts/benchmark_f048_permission_paths.py`, + `src/backend/test/permission/fixtures/f048_bench_contract.synthetic.json` + - **逻辑**：输出 model/dataset/source/visible checksum、P50/P95/P99、dispatch/datastore reads、 + DB rows 和 scan amplification；支持单槽 Check/BatchCheck/StreamedListObjects 及 joined/ + department/file 两条完整链路，不连接生产、不充当迁移 dry-run。 + - **验收**：T160 全部通过 + - **依赖**：T160 + +- [x] **T162：执行 pinned v1.15.1 BENCH-01 发布门禁** + - **文件**：`features/v3.0.0-beta1/048-rebac-permission-model-grants/bench-01-flat-visible-report-20260813.md` + - **验证**：在镜像 digest 固定的 v1.15.1 环境运行 T161；记录完整 DSL、并发与代表性分布。 + 完整枚举 P95 按 Design §7.2 的 50/100/300/1,000ms 门槛，集合 checksum 必须等于 canonical + oracle；joined 只有门禁通过才允许接入 ID-first。报告保留 v1.14.2/A-B 数据为历史对照， + 不把合成结果冒充生产分布。 + - **覆盖 AC**：AC-160, AC-161, AC-162, AC-163, AC-168, AC-175, AC-176 + - **依赖**：T149, T157, T161 + +### 已完成的轻量列表基线 + +- [x] **T163：mine 轻量列表回归测试** + - **文件**：`src/backend/test/knowledge/test_space_listing_pin_source.py` + - **测试**：证明 mine 仍按 DB order+用户 pin 分组，且响应不再包含/计算根目录 `file_num` + 或部门装饰字段；已于提交 `e14d64f73` 通过。 + - **覆盖 AC**：AC-31, AC-175, AC-176 + - **依赖**：T140 + +- [x] **T164：实现 mine 轻量列表响应** + - **文件**：`src/backend/bisheng/knowledge/domain/schemas/knowledge_space_schema.py`, + `src/backend/bisheng/knowledge/domain/services/knowledge_space_service.py` + - **逻辑**：`KnowledgeSpaceListItemResp` 只保留基础空间与 pin/follow/subscription/role； + `_format_member_spaces` 不调用根文件计数和部门元数据装饰。已提交 `e14d64f73`。 + - **验收**：T163 全部通过 + - **依赖**：T163 + +- [x] **T165：department 候选优先与轻量响应测试** + - **文件**：`src/backend/test/department/test_department_knowledge_space_service.py` + - **测试**：从部门绑定空间候选执行一次 visible BatchCheck，仅把可见 ID 交给轻量 DB formatter； + 不读取 `space_channel_member`、不重复 visible、不补 manage/file count/部门元数据。已于提交 + `e14d64f73` 通过。 + - **覆盖 AC**：AC-161, AC-175, AC-176 + - **依赖**：T140 + +- [x] **T166：实现 department 后端轻量列表** + - **文件**：`src/backend/bisheng/knowledge/domain/services/department_knowledge_space_service.py` + - **逻辑**：binding candidates→单次 bounded visible BatchCheck→`_format_basic_spaces`；只返回 + 业务基础字段与 pin 状态。已提交 `e14d64f73`。 + - **验收**：T165 全部通过 + - **依赖**：T165 + +### 前端 Client（Test-First） + +- [x] **T167：Client department spaceKind 映射测试** + - **文件**：`src/frontend/client/src/api/knowledge.test.ts` + - **测试**：department API 不再返回部门元数据时，Client 根据调用入口稳定映射 + `spaceKind="department"`；mine/joined 仍为 normal，不依赖后端 `space_kind` 默认值。 + - **覆盖 AC**：AC-161, AC-175, AC-176 + - **依赖**：T166 + +- [x] **T168：复核 Client department spaceKind 映射实现** + - **文件**：`src/frontend/client/src/api/knowledge.ts` + - **逻辑**：`getDepartmentSpacesApi` 在 `mapSpace` 后补稳定 `spaceKind="department"`；代码已在 + `e14d64f73` 预置，本任务以 T167、单文件 ESLint 和 Client strict typecheck 全通过为完成条件。 + - **验收**：T167 全部通过 + - **依赖**：T167 + +### 后端业务列表（Test-First） + +- [x] **T169：joined 可见 ID 优先合同测试** + - **文件**：`src/backend/test/knowledge/test_space_joined_visible_ids.py` + - **测试**：direct/department/group/manual subscription/其他合法来源全部由完整 visible ID + 集进入候选；排除 canonical 本人创建、应用 tenant/status/type/order 过滤；不读 membership/ + role、不重复 Check、不返回 manage/file count/部门元数据。普通用户、超管、租管采用相同 + 个人 visible 来源；stream 失败或 5,001 明确返回 25014。 + - **覆盖 AC**：AC-159, AC-160, AC-161, AC-162, AC-163, AC-168, AC-169, AC-170, AC-171, AC-172 + - **依赖**：T162 + +- [x] **T170：实现 joined 可见 ID 优先链路** + - **文件**：`src/backend/bisheng/knowledge/domain/services/knowledge_space_service.py` + - **逻辑**：`list_visible_objects("knowledge_space", max_results=5000)`→DB 500-ID 分块详情查询→ + canonical creator 排除与稳定 `order_by` 归并；删除 `_scan_space_action_ids("visible")`、 + membership/role candidate 和 `_format_accessible_spaces` 的重复 visible。主动订阅只有已投影 + 为 canonical Grant/source 时生效。 + - **验收**：T169 全部通过 + - **依赖**：T169 + +- [x] **T171：文件/文件夹候选优先稳定游标测试** + - **文件**：`src/backend/test/knowledge/test_file_visible_candidate_pagination.py` + - **测试**：高继承/高可见率、少量 CUSTOM deny、首批不足一页、跨多批填页、候选耗尽、 + 排序同值和 cursor 重试；每批只做 bounded BatchCheck，父空间 ALLOW 不能替代子资源最终 + visible，页间不重复/漏项并记录 scan amplification。 + - **覆盖 AC**：AC-161, AC-173, AC-174, AC-175, AC-176 + - **依赖**：T162 + +- [x] **T172：实现文件/文件夹候选优先续取** + - **文件**：`src/backend/bisheng/knowledge/domain/services/knowledge_space_service.py`, + `src/backend/bisheng/knowledge/domain/models/knowledge_file.py` + - **逻辑**：业务 Repository 以稳定 `(sort_key,id)` cursor 有界取候选，Service 对每批构造 + verified target 并 `batch_check_visible`；不足 page_size 继续扫描，next cursor 基于最后扫描 + 候选而非最后可见项。不得先枚举全部可见子资源或把继承概率当 ALLOW。 + - **验收**：T171 全部通过 + - **依赖**：T171 + +--- + +## Wave 11 — 旧系统单次迁移与 D4 门禁 + +### 后端 scripts / Migration（Test-First） + +- [x] **T173：旧来源到单槽 contribution 迁移测试** + - **文件**：`src/backend/test/permission/test_f048_tuple_mapper.py`, + `src/backend/test/permission/test_f048_migration_coordinator.py` + - **测试**：旧四档 tuple+唯一 binding→canonical Grant/assignee→一条 contribution；显式 + inactive 模型的既有合法 binding 继续迁移，orphan binding 不复活，direct+membership + 同授权去重但保留追溯；system/public/shared 不写 Grant projection；不得生成 switch/A-B。 + - **覆盖 AC**：AC-159, AC-164, AC-166, AC-168, AC-169, AC-177 + - **依赖**：T153, T162 + +- [x] **T174：实现迁移 target 单槽编译** + - **文件**：`src/backend/bisheng/permission/migration/f048_tuple_mapper.py`, + `src/backend/bisheng/permission/migration/f048_coordinator.py` + - **逻辑**：在原唯一 PermissionMigrationRun 中复用 VisibilityProjectionCompiler，从旧 Config/ + tuple/Owner facts 直接生成最终 Grant/assignee/source/aggregate tuple；每批≤90、持久化 source + 与 target checksum 后才退休 legacy，不发布中间深层-visible model。 + - **验收**：T173 全部通过 + - **依赖**：T173 + +- [x] **T175：D4 单槽完整性与删除门禁测试** + - **文件**：`src/backend/test/permission/test_f048_migration_verifier.py`, + `src/backend/test/permission/test_f048_migration_runtime.py` + - **测试**：验证每个合法 assignee 一条 contribution、source/aggregate checksum、无来源 tuple=0、 + inactive binding 保持、orphan=0、单资源/BatchCheck/streamed list oracle 一致；任一 residual + 或不完整 stream 阻断 READY_TO_START/模型删除，resume 只前向修复。 + - **覆盖 AC**：AC-161, AC-165, AC-167, AC-168, AC-170, AC-171, AC-177 + - **依赖**：T174 + +- [x] **T176：实现 D4 单槽 verifier** + - **文件**：`src/backend/bisheng/permission/migration/f048_verifier.py`, + `src/backend/bisheng/permission/migration/f048_runtime_verification.py` + - **逻辑**：移除 A/B/switch gate，新增单槽 contribution/aggregate/live checksum、模型引用残留、 + streamed visible 完整性和 canonical oracle；source item 继续按 `(source_kind,source_locator)` + 在应用层排序，preserved tuple 继续排除计划退休项。 + - **验收**：T175 全部通过 + - **依赖**：T175 + +- [x] **T177：数据迁移 CLI 单 run 合同测试** + - **文件**：`src/backend/test/permission/test_f048_migration_cli.py` + - **测试**：`migrate --apply` 和 `verify --run-id` 只复用原 run/checkpoint；拒绝 second migration、 + 中间 F048 model、A/B 参数、Store 替换和 startup/Celery 调用；resume 固定 durable Store/model。 + - **覆盖 AC**：AC-167, AC-170, AC-177 + - **依赖**：T176 + +- [x] **T178：更新正式迁移入口与 runbook** + - **文件**：`src/backend/scripts/migrate_f048_permission_data.py`, + `src/backend/scripts/README.md` + - **逻辑**：CLI 调用 T174/T176 的单槽 migration/verifier；runbook 写明 D0 停流、DDL-only + Alembic、同 Store 单 model、单 run checkpoint、失败保持维护与前向修复，删除 A/B/switch + 命令和二次迁移说明。不得保存凭据或把 `--apply` 描述成 dry-run。 + - **验收**：T177 全部通过 + - **依赖**：T177 + +--- + +## Wave 12 — Platform / Client 模型生命周期交互 + +### 前端 Platform（Vitest Test-First） + +- [x] **T179：Platform 模型停用/删除交互测试** + - **文件**：`src/frontend/platform/src/test/f048ModelEditor.test.tsx` + - **测试**：停用提示为“不能再用它授权，已有授权不受影响”；有引用或 residual 的 delete + 显示 25004 指引且不暗示停用即撤权；零引用后才能确认 DELETE_MODEL draft，历史影响数可见。 + - **覆盖 AC**：AC-15, AC-17, AC-164, AC-165, AC-167 + - **依赖**：T151 + +- [x] **T180：实现 Platform ModelEditor 生命周期语义** + - **文件**：`src/frontend/platform/src/pages/SystemPage/components/permission/ModelEditor.tsx` + - **逻辑**：active switch 只控制可分配性；删除对话框展示引用/残留 blocker 和先撤销或替换 + 绑定的操作指引。复用现有 i18n key 和 request wrapper，不引入新状态库或硬编码中文。 + - **验收**：T179、Platform 单文件 lint/typecheck 通过 + - **依赖**：T179 + +- [x] **T181：Platform GrantTab inactive 行测试** + - **文件**：`src/frontend/platform/src/test/f048PermissionGrantTab.test.tsx` + - **测试**：inactive 模型不出现在 ADD/MOVE target，但既有行仍展示原模型和权限，可 MOVE + 到 active 模型或 REMOVE；包含 manage_permission 的 inactive 来源仍允许管理，不由 UI 隐藏。 + - **覆盖 AC**：AC-15, AC-164, AC-166 + - **依赖**：T151 + +- [x] **T182：实现 Platform GrantTab inactive 展示** + - **文件**：`src/frontend/platform/src/components/bs-comp/permission/PermissionGrantTab.tsx` + - **逻辑**：target options 过滤 inactive，现有 assignee row 不过滤；保留 editable/protected/ + source 服务端字段，MOVE/REMOVE 精确使用 assignee version，不在前端重算权限。 + - **验收**：T181、Platform 单文件 lint/typecheck 通过 + - **依赖**：T181 + +### 前端 Client（Jest Test-First） + +- [x] **T183：Client GrantTab inactive 行测试** + - **文件**：`src/frontend/client/src/components/permission/PermissionGrantTab.test.tsx` + - **测试**：与 Platform 同合同：新增/变更目标不含 inactive，既有 inactive 行及其来源/动作 + 保持展示并可精确 MOVE/REMOVE，protected 行仍锁定，403 不在组件分支处理。 + - **覆盖 AC**：AC-15, AC-164, AC-166 + - **依赖**：T151 + +- [x] **T184：实现 Client GrantTab inactive 展示** + - **文件**：`src/frontend/client/src/components/permission/PermissionGrantTab.tsx` + - **逻辑**：使用 Client request wrapper/react-query v4 与本地状态；按服务端 target/row 字段 + 分别过滤和展示，不新增 Recoil，不硬编码中文，不增加 403 业务分支。 + - **验收**：T183、Client 单文件 lint/strict typecheck 通过 + - **依赖**：T183 + +--- + +## Wave 13 — 集成、可观测、E2E 与文档收口 + +### 后端集成与可观测（Test-First） + +- [x] **T185：真实 OpenFGA v1.15.1 单槽集成测试** + - **文件**：`src/backend/test/permission/test_f048_openfga_integration.py` + - **测试**：在固定 digest 验证单槽 model tests、direct/department/group/system、多来源、 + inactive 既有授权保持、具体 action 不展平、Check/BatchCheck/StreamedListObjects 同集合、 + higher consistency、atomic Write 与 max resolve depth；模型中不得存在 A/B/switch relation。 + - **覆盖 AC**：AC-15, AC-28, AC-159, AC-161, AC-163, AC-164, AC-166, AC-170, AC-171 + - **依赖**：T157, T159, T162 + +- [x] **T186：MySQL/DM8 单槽 schema 与迁移集成测试** + - **文件**：`src/backend/test/permission/test_f048_database_integration.py` + - **测试**：在 disposable MySQL/DM8 验证新表/索引/唯一键、tenant filter、cursor/checkpoint、 + single contribution、resume 与 D4 checksum；Alembic 只做 DDL且单 head，正式脚本不由启动调用。 + - **覆盖 AC**：AC-165, AC-167, AC-168, AC-169, AC-171, AC-177 + - **依赖**：T145, T178 + +- [x] **T187：visible 投影与列表可观测测试** + - **文件**：`src/backend/test/permission/test_f048_visibility_observability.py` + - **测试**：断言 projection source/unique tuple/reconcile/checksum/stale/orphan 指标，以及列表 + strategy/candidate/visible/scanned/amplification/stream_completed/capacity/DB-FGA-total 耗时；日志 + 不含姓名、资源名、Config 原文或 token，达到容量/放大/无来源阈值告警。 + - **覆盖 AC**：AC-167, AC-168, AC-171, AC-175, AC-176 + - **依赖**：T159, T170, T172 + +- [x] **T188：实现 visible 投影与列表可观测** + - **文件**：`src/backend/bisheng/permission/domain/services/visibility_projection_service.py`, + `src/backend/bisheng/permission/domain/services/permission_action_service.py` + - **逻辑**：按 Design §7.3 写结构化 metric-log 与审计 ID/checksum；无来源 tuple、删除残留、 + stream incomplete、joined 容量 80% 和 candidate scan amplification 超阈值告警，不记录 PII。 + - **验收**：T187 全部通过 + - **依赖**：T187 + +### E2E 与文档 + +- [ ] **T189：执行 F048 可见性增量 E2E** + - **文件**：`features/v3.0.0-beta1/048-rebac-permission-model-grants/e2e-checklist.md`, + `features/v3.0.0-beta1/048-rebac-permission-model-grants/e2e-test-report.md` + - **验证**：使用 `/e2e-test` 生成/执行 API E2E 与页面手工清单；覆盖模型停用保持/新增拒绝、 + 删除零引用门禁、多来源最后撤销、joined 五类来源+本人创建排除+管理员无扩权、department + 单次 BatchCheck、file 跨批填页、5,001 容量错误、OpenFGA 故障无 SQL fallback，以及 + Platform/Client inactive 行一致。运行 backend focused suite、arch-guard、frontend lint/ + typecheck/check-i18n；无法执行的真实环境项必须明确记录，不能写成通过。 + - **覆盖 AC**：AC-15, AC-17, AC-27, AC-28, AC-159, AC-160, AC-161, AC-162, AC-163, AC-164, AC-165, AC-166, AC-167, AC-168, AC-169, AC-170, AC-171, AC-172, AC-173, AC-174, AC-175, AC-176, AC-177 + - **依赖**：T168, T170, T172, T178, T180, T182, T184, T185, T186, T188 + +- [ ] **T190：回写权限架构现状与增量偏差** + - **文件**：`docs/architecture/10-permission-rbac.md`, + `features/v3.0.0-beta1/048-rebac-permission-model-grants/tasks.md` + - **逻辑**：实现完成后把单槽 shallow visible、source projection、inactive 可分配语义、删除 + 零引用协议、完整 streamed 枚举、数据驱动列表路径和旧系统单 run 迁移写为当前事实；记录 + 实际偏差与验证证据，不保留 A/B/深层 visible 为运行时说明，不修改 Constitution C1～C7。 + - **验收**：T189 报告无未关闭 blocker，`git diff --check` 与文档本地链接检查通过 + - **依赖**：T189 + +--- + +## Wave 14 — 旧版 F048 迁移环境前向对账 + +- [x] **T191：修正顶层枚举与子资源继承 visible 边界测试** + - **文件**：`src/backend/test/permission/test_f048_authorization_model.py` + - **测试**：顶层资源保持直接单槽+浅层 system 分支；folder/file 保留本地直接、parent+mode + 继承和 system 传播；不存在 A/B/switch，也不展开部门/用户组成员或父级来源。 + - **依赖**：T148, T160, T171 + +- [x] **T192：实现 F048 v2 visible 模型边界** + - **文件**：`src/backend/bisheng/core/openfga/authorization_model_f048.py` + - **逻辑**：Authorization Model 版本升为 `f048-v2`；只为需要完整枚举的顶层资源展平 Grant + visible，folder/file 继续通过 canonical parent 继承，system/public/shared 不受 mode gate。 + - **验收**：T191 与 model checksum 稳定性测试通过 + - **依赖**：T191 + +- [x] **T193：生产级 visible 对账命令测试** + - **文件**：`src/backend/test/permission/test_f048_visible_reconcile_cli.py` + - **测试**：覆盖默认 dry-run、apply Store 二次确认、仅相同 projected subject 的聚合去重、部门/ + 用户组 userset 不展开，以及 duplicate-ignore 幂等补写与 higher-consistency 批量验证。 + - **依赖**：T159, T192 + +- [x] **T194：实现旧迁移环境前向对账与 immutable model/Catalog 切换** + - **文件**：`src/backend/scripts/reconcile_f048_visible_projection.py`, + `src/backend/bisheng/permission/application/catalog_api.py`, `src/backend/scripts/README.md` + - **逻辑**：以 SQL Grant/assignee 重建 source projection；默认 dry-run，apply 要求停流、Store + /operator 确认、无在途 operation，旧 model 切换需显式确认；同 Store 发布/复用最终 model， + 以 duplicate-ignore 确保全部 Grant visible tuple 并 higher-consistency 验证，成功后再激活来源表， + 最后用 no-op Catalog release 前向切换且不修改历史 migration run；不扫描/删除既有 tuple， + stale source 阻断自动写入。 + - **验收**：T193、Catalog runtime 回归、ruff、diff-check 通过 + - **依赖**：T192, T193 + +--- + ## 实际偏差记录 > 只记录一句话指针；设计原因和反直觉事实回写 [design.md](./design.md)。 @@ -1197,3 +1663,7 @@ - 116 的 D4 verify 进一步发现 MySQL collation 顺序与 Python 冻结顺序不同，以及 preserved 核对仍包含计划删除的 stale/canonical-false tuple；T143 只修正取证算法，不修改已经完成的 target tuple 写入和 legacy tuple 退休结果。 +- T185 在 116 的真实 OpenFGA v1.15.1 集成中发现 StreamedListObjects NDJSON 使用 + `{"result":{"object":"..."}}` 包络；client 增加该官方运行时形态并保留未包络代理兼容， + 单槽集合语义不变。T189 当前为 PARTIAL：真实 MySQL/DM8、业务 API/UI、故障注入、5,001 + 容量和 D0～D6 尚无环境证据，因此 T189/T190 不标完成。 diff --git a/features/v3.0.0-beta1/048-rebac-permission-model-grants/test-focus-checklist.md b/features/v3.0.0-beta1/048-rebac-permission-model-grants/test-focus-checklist.md index fc3a8d1a44..22c1443fce 100644 --- a/features/v3.0.0-beta1/048-rebac-permission-model-grants/test-focus-checklist.md +++ b/features/v3.0.0-beta1/048-rebac-permission-model-grants/test-focus-checklist.md @@ -51,9 +51,10 @@ - [ ] 修改一个已被多个资源使用的模型，确认页准确展示受影响资源、授权组和主体数量；取消确认时 线上权限不发生变化。 - [ ] 确认模型变更后，所有引用该模型的资源同时获得或失去对应动作，不需要逐个资源重新授权。 -- [ ] 停用模型后不能再选择该模型新增成员，已有成员立即失去该模型带来的资源可见性和具体动作； - 重新启用后按原授权恢复。 -- [ ] 被资源使用的模型不能直接删除；必须先停用或解除全部引用。 +- [ ] 停用模型后不能再选择该模型新增成员或把成员变更到该模型；已有成员的资源可见性、具体 + 动作和权限管理能力保持不变，重新启用不会重复成员或改写授权。 +- [ ] 被资源使用的模型不能直接删除；必须先撤销或替换全部 Grant/assignee 引用并完成残留 + source projection/OpenFGA tuple 对账。停用不是删除前置条件。 ### 场景 03：资源所有者添加成员 @@ -90,7 +91,8 @@ 删除、分享、使用、发布或权限管理分别被拒绝。 - [ ] 列表按钮的显示状态、详情页按钮状态和实际接口执行结果一致；隐藏按钮不能代替后端鉴权。 - [ ] 用户没有任何有效成员关系时，资源不出现在列表、搜索和选择器中，直接访问详情也不能进入。 -- [ ] 停用模型、撤销成员或切换继承来源后，列表、详情和实际操作在同一次变更后保持一致。 +- [ ] 停用模型后列表、详情和实际操作保持原有授权结果，仅新增/变更入口拒绝该模型；撤销成员 + 或切换继承来源后，列表、详情和实际操作在同一次变更后保持一致。 ### 场景 07：文件夹和文件默认继承上级权限 @@ -180,7 +182,8 @@ - [ ] 九类资源的列表、搜索、最近使用和选择器只返回当前用户可见的资源。 - [ ] 混合准备直接、部门、用户组和继承权限数据，翻页后不漏数据、不重复，列表总数与实际可见资源一致。 -- [ ] 授权、撤销、模型停用或模式切换后，重新查询列表立即反映最新结果。 +- [ ] 授权、撤销或模式切换后，重新查询列表立即反映最新结果；模型停用前后已有授权列表结果 + 保持一致，仅新增/变更授权入口不再展示该模型。 - [ ] 具有大量可见资源的用户打开列表、翻页和搜索时响应时间可接受，不出现持续加载或超时。 ### 场景 17：模型变更对存量资源的影响 diff --git a/features/v3.0.0-beta1/release-contract.md b/features/v3.0.0-beta1/release-contract.md index c031b4e8f9..bdbd4f06bc 100644 --- a/features/v3.0.0-beta1/release-contract.md +++ b/features/v3.0.0-beta1/release-contract.md @@ -26,7 +26,7 @@ | PermissionActionResourceScope | F048-rebac-permission-model-grants | 动作与适用资源类型的规范化关联，不使用 JSON 数组作为运行时真相 | | PermissionCatalogRelease | F048-rebac-permission-model-grants | 动作、模型和模型动作的完整版本快照；由单一 active Catalog tuple 原子切换执行状态 | | PermissionCatalogProjectionTuple | F048-rebac-permission-model-grants | PLATFORM 全局 Catalog release 的分批 staging、commit checksum 与逐 tuple 恢复状态；不伪造 tenant_id | -| PermissionModel | F048-rebac-permission-model-grants | 四个标准模型、自定义模型、派生等级、动作集合、生效状态和模型级同级授权策略 | +| PermissionModel | F048-rebac-permission-model-grants | 四个标准模型、自定义模型、派生等级、动作集合、可分配状态和模型级同级授权策略 | | PermissionModelAction | F048-rebac-permission-model-grants | 权限模型与细粒度动作的规范化多对多关联 | | PermissionGrant | F048-rebac-permission-model-grants | 某一资源与某一权限模型之间的授权集合及其用户、部门、用户组等主体来源 | | PermissionGrantAssignee | F048-rebac-permission-model-grants | Grant 中的直接用户、部门、用户组等主体、范围、来源和受保护属性 | @@ -34,6 +34,7 @@ | ProtectedPermissionAssignment | F048-rebac-permission-model-grants | 系统创建且不能由普通成员管理接口删除或降级的资源授权 | | PermissionProjectionOperation / PermissionProjectionTuple | F048-rebac-permission-model-grants | tenant 级 Grant/mode/resource 到 OpenFGA 的幂等发布意图、分阶段 tuple、commit、补偿和失败关闭状态 | | AuthorizationModelRelease / PermissionMigrationRun / PermissionMigrationItem | F048-rebac-permission-model-grants | 现有 OpenFGA Store 中的新 Authorization Model 版本、唯一生产固定版本，以及由 `src/backend/scripts/` 专用数据迁移脚本写入的逐项映射、checkpoint、旧 tuple 退役、校验、启服和人工处置结果；不是 Alembic revision 状态 | +| PermissionVisibleSourceProjection | F048-rebac-permission-model-grants | 原 F048 正式迁移和后续运行时从 canonical Grant assignee 生成的展平可见派生索引；随同一 PermissionMigrationRun/Item 追溯；system/public/shared 继续由各 Owner 事实与 system tuple 追溯；均不可独立编辑或参与数据库 ALLOW | **规则**： - 非 Owner Feature 的 AC 中不得出现其他对象的"创建/修改/删除"行为，只能"读取"或"调用" Owner 的 Service @@ -67,6 +68,7 @@ | INV-24 | 标准模型按等级累计动作；自定义模型只产生其显式选择的动作，派生等级只用于分类和可授予边界，不能自动补齐该等级及以下的其他动作。本不变量替代 v2.5.0 INV-7 在 F048 资源权限范围内的旧四档金字塔语义 | PermissionAction, PermissionModel | F048 | | INV-25 | user-owned 资源创建仍必须通过 `PermissionService.authorize()` 为创建者建立受保护 owner Grant 并遵守失败补偿；一个资源可以同时有多个 owner，其他 owner 作为独立普通来源存在。F048 启服后不再要求继续写旧资源 `owner` tuple。只有经资源 adapter 代码 allowlist 与 canonical business predicate 双重确认的 platform system-owned 资源可以不伪造用户 owner，并继续只由 C4 system identity 管理。OQ-07 已选择 A：F048 启服时退役既有 F018 owner 交接 API，本期不实现 protected owner transfer；创建者 protected owner 不可通过普通成员接口删除或转让 | ProtectedPermissionAssignment, PermissionGrant | F048 | | INV-26 | F048 的 Alembic revision 只允许 MySQL/DM8 schema DDL，不得读取、转换、回填、去重、清理或 seed 旧权限数据，也不得访问 OpenFGA。所有旧 Config、业务事实和 tuple 数据迁移必须由运维人员在已启动但 F048 未就绪的 backend 容器内，通过 `src/backend/scripts/` 下的专用脚本于 schema upgrade 成功后显式执行；不得由 API、Celery 或应用启动钩子自动触发 | PermissionMigrationRun, PermissionMigrationItem | F048 | +| INV-27 | 权限模型、Grant、Grant 主体和权限模式以规范化 MySQL/DM8 关系表为控制面真相；组织成员、系统身份和资源状态以各自 Owner 业务域的 canonical 事实为真相；OpenFGA 是这些事实发布后的唯一权限执行面。每条有效资源可见结果必须可追溯到至少一个当前有效来源；模型停用只禁止新增或变更授权，已有授权保持有效；模型删除前必须撤销或替换全部绑定，并在引用、来源投影和残留 tuple 清零后才允许删除。来源撤销只清除该来源贡献，并保证 Check 与可见资源枚举一致，不得删除其他仍有效来源的可见性 | PermissionModel, PermissionGrant, PermissionGrantAssignee, AuthorizationModelRelease | F048 | （INV-1~7 为 v2.6.0 存量不变量，继续有效，见 `features/v2.6.0/release-contract.md`。） @@ -112,7 +114,7 @@ | 模块编码 (MMM) | 模块 | Owner Feature | |----------------|------|---------------| | —（不新增） | 既有功能体验优化与引用溯源 | F043 复用工作流/报告既有错误响应；F044 验证失败是业务结果（状态=异常）而非错误响应，不占码；F045/F046 纯前端；F047 复用 citation 子系统与 F029 权限过滤的既有错误响应 | -| 250 | ReBAC 权限 Catalog、Grant、投影与迁移 | F048；25001～25013，具体语义见 F048 Design §6.3 | +| 250 | ReBAC 权限 Catalog、Grant、投影、迁移与完整枚举 | F048；25001～25014，具体语义见 F048 Design §6.3 | --- @@ -129,3 +131,7 @@ | 2026-07-29 | 纠正 INV-18/23 迁移拓扑：沿用现有 Store、只运行新 model；同 Store 原地转换并在启服前退役旧 tuple/Config，不创建或维护第二 Store/model runtime | F048 | | 2026-07-29 | 新增 INV-26 并修订 INV-18：Alembic revision 仅负责 MySQL/DM8 schema DDL；F048 旧权限数据和 OpenFGA tuple 迁移由 `src/backend/scripts/` 专用脚本执行，禁止 migration/lifespan/API/Celery 混入数据迁移 | F048 | | 2026-07-31 | 简化 F048 升级顺序：沿用既有“更新镜像并启动→容器内执行数据脚本”流程；旧 model 下进程只进入不就绪运维态并由应用门禁自动拒绝 HTTP/WS，脚本通过后重启一次即自动恢复访问，不再要求先停止容器、人工切换入口或设置停服变量 | F048 | +| 2026-08-13 | 基于 BENCH-01 追加数据驱动的列表权限策略与来源收敛合同：可见 ID 优先和业务候选优先均为合法路径，按代表性实际业务数据中的候选规模、可见/继承比例、分页扫描放大和端到端成本决定并允许重新评审；个人可见列表不因超管身份自动扩大；明确 SQL/Owner 业务域控制面真相与 OpenFGA 唯一执行面边界，并新增 INV-27 | F048 | +| 2026-08-13 | 为 StreamedListObjects 未完整终止或超过业务容量上限分配 25014，禁止把枚举前缀作为成功全集返回 | F048 | +| 2026-08-13 | 纠正 F048 可见投影迁移拓扑：F048 尚未上线，不新增旧 F048 到新 F048 model 的二次迁移；原 PermissionMigrationRun/Item 从旧 Config/四档关系和 Owner 事实直接生成最终单槽浅层 visible model、Grant/Assignee、PermissionVisibleSourceProjection 与 tuple | F048 | +| 2026-08-13 | 明确模型停用/删除语义：停用只禁止新增或变更授权，已有 Grant 保持有效；删除必须先清零或替换全部绑定并完成残留投影对账。因停用不再触发批量撤权，F048 可见执行投影采用单槽浅层 `visible`，不引入 A/B 槽与运行时 switch | F048 | diff --git a/src/backend/bisheng/common/errcode/permission.py b/src/backend/bisheng/common/errcode/permission.py index 9296702cb6..6be959e65d 100644 --- a/src/backend/bisheng/common/errcode/permission.py +++ b/src/backend/bisheng/common/errcode/permission.py @@ -112,3 +112,8 @@ class PermissionImpactExpiredError(BaseErrorCode): class PermissionMutationTooLargeError(BaseErrorCode): Code: int = 25013 Msg: str = "Permission mutation exceeds the atomic tuple limit" + + +class PermissionEnumerationIncompleteError(BaseErrorCode): + Code: int = 25014 + Msg: str = "Permission object enumeration did not complete" diff --git a/src/backend/bisheng/core/database/alembic/versions/v3_0_0_f048_visible_source_projection.py b/src/backend/bisheng/core/database/alembic/versions/v3_0_0_f048_visible_source_projection.py new file mode 100644 index 0000000000..8fd2a1e395 --- /dev/null +++ b/src/backend/bisheng/core/database/alembic/versions/v3_0_0_f048_visible_source_projection.py @@ -0,0 +1,130 @@ +"""F048: add the single-slot visible source projection index. + +The table is rebuildable control-plane state. This revision is DDL-only; the +formal F048 migration script owns all contribution backfill and verification. + +Revision ID: f048_visible_source_projection +Revises: linsight_pending_files +Create Date: 2026-08-13 +""" + +from collections.abc import Sequence + +import sqlalchemy as sa +from alembic import op + +from bisheng.core.database.alembic_helpers.online import table_exists + +revision: str = "f048_visible_source_projection" +down_revision: str | Sequence[str] | None = "linsight_pending_files" +branch_labels: str | Sequence[str] | None = None +depends_on: str | Sequence[str] | None = None + +_TABLE = "permission_visible_source_projection" + + +def upgrade() -> None: + if table_exists(_TABLE): + return + + op.create_table( + _TABLE, + sa.Column("id", sa.BigInteger(), primary_key=True, autoincrement=True), + sa.Column("tenant_id", sa.BigInteger(), nullable=False), + sa.Column("resource_type", sa.String(64), nullable=False), + sa.Column("resource_id", sa.String(64), nullable=False), + sa.Column("visibility_class", sa.String(64), nullable=False), + sa.Column("projected_subject", sa.String(256), nullable=False), + sa.Column("source_kind", sa.String(64), nullable=False), + sa.Column("source_owner_key", sa.String(256), nullable=False), + sa.Column("source_locator", sa.String(256), nullable=False), + sa.Column("source_fingerprint", sa.CHAR(64), nullable=False), + sa.Column("contribution_fingerprint", sa.CHAR(64), nullable=False), + sa.Column("model_key", sa.String(64), nullable=True), + sa.Column("source_version", sa.BigInteger(), nullable=False), + sa.Column("tuple_fingerprint", sa.CHAR(64), nullable=False), + sa.Column( + "state", + sa.String(64), + nullable=False, + server_default=sa.text("'PENDING'"), + ), + sa.Column("operation_id", sa.BigInteger(), nullable=True), + sa.Column("migration_item_id", sa.BigInteger(), nullable=True), + sa.Column( + "create_time", + sa.DateTime(), + nullable=False, + server_default=sa.text("CURRENT_TIMESTAMP"), + ), + sa.Column( + "update_time", + sa.DateTime(), + nullable=False, + server_default=sa.text("CURRENT_TIMESTAMP"), + ), + sa.ForeignKeyConstraint( + ["operation_id"], + ["permission_projection_operation.id"], + ondelete="RESTRICT", + ), + sa.ForeignKeyConstraint( + ["migration_item_id"], + ["permission_migration_item.id"], + ondelete="RESTRICT", + ), + sa.UniqueConstraint( + "tenant_id", + "resource_type", + "resource_id", + "visibility_class", + "projected_subject", + "contribution_fingerprint", + name="uq_perm_visible_source_contribution", + ), + ) + op.create_index("ix_permission_visible_source_projection_tenant_id", _TABLE, ["tenant_id"]) + op.create_index( + "ix_perm_visible_resource_subject", + _TABLE, + [ + "tenant_id", + "resource_type", + "resource_id", + "visibility_class", + "projected_subject", + "state", + ], + ) + op.create_index( + "ix_perm_visible_model_state", + _TABLE, + ["model_key", "state", "tenant_id", "id"], + ) + op.create_index( + "ix_perm_visible_source_owner", + _TABLE, + ["tenant_id", "source_kind", "source_owner_key", "state", "id"], + ) + op.create_index( + "ix_perm_visible_operation", + _TABLE, + ["tenant_id", "operation_id", "state", "id"], + ) + op.create_index( + "ix_perm_visible_migration_item", + _TABLE, + ["migration_item_id", "state", "id"], + ) + + +def downgrade() -> None: + """Drop the table before any formal F048 data-migration run is created. + + After a formal run starts, the release runbook forbids application-level + downgrade and requires forward repair. The DDL revision deliberately does + not inspect business rows to enforce that operational precondition. + """ + + if table_exists(_TABLE): + op.drop_table(_TABLE) diff --git a/src/backend/bisheng/core/database/alembic/versions/v3_0_0_linsight_pending_files.py b/src/backend/bisheng/core/database/alembic/versions/v3_0_0_linsight_pending_files.py new file mode 100644 index 0000000000..dcc8af1a35 --- /dev/null +++ b/src/backend/bisheng/core/database/alembic/versions/v3_0_0_linsight_pending_files.py @@ -0,0 +1,48 @@ +"""Add linsight_session_version.pending_files (deferred attachment ingestion). + +Attachment ingestion moved out of the submit request and into the Linsight +worker: submit now only records the RAW submitted file refs here, and the worker +materializes them into ``files`` before the run starts. ``files`` therefore keeps +its contract (only fully-ingested entries), so every downstream reader +(_init_file_directory / prepare_file_list / the workspace drawer) is untouched. + +Nullable for backward compatibility: rows written before this column existed read +as NULL, i.e. nothing pending. ``JsonType`` keeps MySQL/DM8 compatible. + +Revision ID: linsight_pending_files +Revises: f048_migration_item_message_longtext +Create Date: 2026-08-13 +""" + +from __future__ import annotations + +from collections.abc import Sequence +from typing import Union + +import sqlalchemy as sa +from alembic import op + +from bisheng.core.database.dialect_helpers import JsonType, column_exists + +revision: str = "linsight_pending_files" +down_revision: Union[str, Sequence[str], None] = "f048_migration_item_message_longtext" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + +_TABLE = "linsight_session_version" +_COLUMN = "pending_files" + + +def upgrade() -> None: + conn = op.get_bind() + if not column_exists(conn, _TABLE, _COLUMN): + op.add_column( + _TABLE, + sa.Column(_COLUMN, JsonType, nullable=True, comment="Submitted file refs awaiting worker-side ingestion"), + ) + + +def downgrade() -> None: + conn = op.get_bind() + if column_exists(conn, _TABLE, _COLUMN): + op.drop_column(_TABLE, _COLUMN) diff --git a/src/backend/bisheng/core/openfga/authorization_model_f048.py b/src/backend/bisheng/core/openfga/authorization_model_f048.py index 9bc7952c16..bef22a0aef 100644 --- a/src/backend/bisheng/core/openfga/authorization_model_f048.py +++ b/src/backend/bisheng/core/openfga/authorization_model_f048.py @@ -12,7 +12,7 @@ from hashlib import sha256 from typing import Any -MODEL_VERSION = "f048-v1" +MODEL_VERSION = "f048-v2" DEFAULT_ACTION_CODES: tuple[str, ...] = ( "manage_permission", @@ -95,6 +95,13 @@ "use": frozenset({"knowledge_library", "workflow", "assistant", "tool"}), } +# These resource types are enumerated directly by ListObjects. Folder and +# file visibility stays compositional because their list endpoints fetch a +# business candidate page first and then BatchCheck inherited permissions. +FLAT_VISIBLE_RESOURCE_TYPES: frozenset[str] = frozenset(set(MIGRATED_RESOURCE_TYPES) - set(PARENT_TYPES)) | frozenset( + OWNER_PROJECTION_RESOURCE_TYPES +) + def _this() -> dict: return {"this": {}} @@ -137,6 +144,16 @@ def _subject_types() -> list[dict]: ] +def _visible_subject_types() -> list[dict]: + """Subjects that canonical owner adapters may flatten into visibility.""" + + return [ + *_subject_types(), + {"type": "tenant", "relation": "member"}, + _wildcard_user_type(), + ] + + def _base_type_definitions() -> list[dict]: return [ {"type": "user", "relations": {}, "metadata": None}, @@ -209,9 +226,10 @@ def _model_release_type(action_codes: tuple[str, ...]) -> dict: relations: dict[str, dict] = { "catalog": _this(), "enabled_marker": _this(), + "published": _from("catalog", "active"), "active": _intersection( _computed("enabled_marker"), - _from("catalog", "active"), + _computed("published"), ), } metadata: dict[str, dict] = { @@ -222,7 +240,7 @@ def _model_release_type(action_codes: tuple[str, ...]) -> dict: marker = f"{action}_marker" relations[marker] = _this() relations[f"can_{action}"] = _intersection( - _computed("active"), + _computed("published"), _computed(marker), ) metadata[marker] = {"directly_related_user_types": [_wildcard_user_type()]} @@ -230,7 +248,7 @@ def _model_release_type(action_codes: tuple[str, ...]) -> dict: marker = f"grant_level_{level}_marker" relations[marker] = _this() relations[f"can_grant_level_{level}"] = _intersection( - _computed("active"), + _computed("published"), _computed(marker), ) metadata[marker] = {"directly_related_user_types": [_wildcard_user_type()]} @@ -266,14 +284,6 @@ def _permission_grant_type(action_codes: tuple[str, ...]) -> dict: "model": _this(), "ordinary_assignee": _this(), "protected_assignee": _this(), - "ordinary_visible": _intersection( - _computed("ordinary_assignee"), - _from("model", "active"), - ), - "protected_visible": _intersection( - _computed("protected_assignee"), - _from("model", "active"), - ), } metadata = { "model": {"directly_related_user_types": [{"type": "permission_model"}]}, @@ -309,30 +319,28 @@ def _system_relation( *, type_name: str, parent_types: tuple[str, ...], - action: str | None, + action: str, ) -> dict: - children: list[dict] = [] - if action is None: + children: list[dict] = [_computed(f"system_{action}_marker")] + if type_name in SYSTEM_SHARED_ACTION_TYPES[action]: children.extend( ( - _computed("system_visible_marker"), _computed("public_reader"), _from("shared_with", "member"), ) ) - if parent_types: - children.append(_from("parent", "system_visible")) - else: - children.append(_computed(f"system_{action}_marker")) - if type_name in SYSTEM_SHARED_ACTION_TYPES[action]: - children.extend( - ( - _computed("public_reader"), - _from("shared_with", "member"), - ) - ) - if parent_types: - children.append(_from("parent", f"system_can_{action}")) + if parent_types: + children.append(_from("parent", f"system_can_{action}")) + return _union(*children) + + +def _system_visible_relation(*, parent_types: tuple[str, ...]) -> dict: + children: list[dict] = [ + _computed("public_reader"), + _from("shared_with", "member"), + ] + if parent_types: + children.append(_from("parent", "system_visible")) return _union(*children) @@ -344,20 +352,8 @@ def _resource_type(type_name: str, action_codes: tuple[str, ...]) -> dict: "custom_mode": _this(), "shared_with": _this(), "public_reader": _this(), - "system_visible_marker": _this(), "system_download_marker": _this(), "system_use_marker": _this(), - "ordinary_visible_from_grant": _from("grant", "ordinary_visible"), - "protected_visible_from_grant": _from("grant", "protected_visible"), - "ordinary_custom_visible": _intersection( - _computed("ordinary_visible_from_grant"), - _computed("custom_mode"), - ), - "system_visible": _system_relation( - type_name=type_name, - parent_types=parent_types, - action=None, - ), } metadata: dict[str, dict] = { "grant": {"directly_related_user_types": [{"type": "permission_grant"}]}, @@ -365,31 +361,43 @@ def _resource_type(type_name: str, action_codes: tuple[str, ...]) -> dict: "custom_mode": {"directly_related_user_types": [_wildcard_user_type()]}, "shared_with": {"directly_related_user_types": [{"type": "tenant"}]}, "public_reader": {"directly_related_user_types": [_wildcard_user_type()]}, - "system_visible_marker": {"directly_related_user_types": [_wildcard_user_type()]}, "system_download_marker": {"directly_related_user_types": [_wildcard_user_type()]}, "system_use_marker": {"directly_related_user_types": [_wildcard_user_type()]}, } - - visible_children = [ - _computed("protected_visible_from_grant"), - _computed("ordinary_custom_visible"), - _computed("system_visible"), - ] if parent_types: relations["parent"] = _this() relations["inherit_mode"] = _this() - relations["inherited_visible"] = _intersection( - _from("parent", "visible"), - _computed("inherit_mode"), - ) - visible_children.append(_computed("inherited_visible")) metadata["parent"] = {"directly_related_user_types": [{"type": parent_type} for parent_type in parent_types]} metadata["inherit_mode"] = {"directly_related_user_types": [_wildcard_user_type()]} - relations["visible"] = _intersection( - _computed("permission_enabled"), - _union(*visible_children), + relations["system_visible"] = _system_visible_relation( + parent_types=parent_types, ) + if type_name in FLAT_VISIBLE_RESOURCE_TYPES: + # Keep the top-level hot path as one direct relation so reverse + # enumeration is proportional to the user's visible set. Public and + # tenant-shared predicates stay as a second shallow system branch. + relations["visible"] = _union( + _this(), + _computed("system_visible"), + ) + else: + # Child resources are checked from business candidates. Their local + # custom grants remain direct, while ordinary inheritance and system + # visibility stay in the graph instead of being fanned out to every + # descendant. + relations["inherited_visible"] = _intersection( + _from("parent", "visible"), + _computed("inherit_mode"), + ) + relations["visible"] = _union( + _this(), + _computed("inherited_visible"), + _computed("system_visible"), + ) + metadata["visible"] = { + "directly_related_user_types": _visible_subject_types(), + } for action in action_codes: relations[f"ordinary_can_{action}_from_grant"] = _from( diff --git a/src/backend/bisheng/core/openfga/client.py b/src/backend/bisheng/core/openfga/client.py index f52dd2c937..191705a7ce 100644 --- a/src/backend/bisheng/core/openfga/client.py +++ b/src/backend/bisheng/core/openfga/client.py @@ -6,7 +6,9 @@ from __future__ import annotations +import json import logging +from collections.abc import AsyncIterator from typing import Any import httpx @@ -165,19 +167,56 @@ async def list_objects( data = await self._post(f"/stores/{self._store_id}/list-objects", body) return data.get("objects", []) + async def stream_list_objects( + self, + user: str, + relation: str, + type: str, + consistency: str | None = None, + ) -> tuple[str, ...]: + """Consume StreamedListObjects completely before returning any object.""" + + body = { + "user": user, + "relation": relation, + "type": type, + "authorization_model_id": self._model_id, + } + if consistency: + body["consistency"] = consistency + objects: list[str] = [] + async for item in self._streamed_post( + f"/stores/{self._store_id}/streamed-list-objects", + body, + ): + # OpenFGA v1.15.1 wraps each NDJSON item in ``result``. Accept the + # unwrapped shape as well for compatible proxies and test doubles. + result = item.get("result", item) + object_key = result.get("object") if isinstance(result, dict) else None + if not isinstance(object_key, str) or not object_key: + raise FGAClientError("OpenFGA StreamedListObjects returned an invalid item") + objects.append(object_key) + return tuple(objects) + # ── Tuple CRUD ─────────────────────────────────────────────── async def write_tuples( self, writes: list[dict] | None = None, deletes: list[dict] | None = None, + *, + ignore_duplicate_writes: bool = False, ) -> None: """Batch write and/or delete tuples. Each tuple: {"user": "user:7", "relation": "owner", "object": "workflow:abc"} Raises FGAWriteError when the atomic request is invalid or fails. """ - body = self._build_write_body(writes, deletes) + body = self._build_write_body( + writes, + deletes, + ignore_duplicate_writes=ignore_duplicate_writes, + ) if body is None: return @@ -193,9 +232,15 @@ def write_tuples_sync( self, writes: list[dict] | None = None, deletes: list[dict] | None = None, + *, + ignore_duplicate_writes: bool = False, ) -> None: """Synchronous tuple write for Celery tasks without an asyncio loop.""" - body = self._build_write_body(writes, deletes) + body = self._build_write_body( + writes, + deletes, + ignore_duplicate_writes=ignore_duplicate_writes, + ) if body is None: return @@ -211,6 +256,8 @@ def _build_write_body( self, writes: list[dict] | None = None, deletes: list[dict] | None = None, + *, + ignore_duplicate_writes: bool = False, ) -> dict | None: """Assemble the OpenFGA write request body, or None when nothing to do.""" operation_count = len(writes or ()) + len(deletes or ()) @@ -218,7 +265,12 @@ def _build_write_body( raise FGAWriteError(f"OpenFGA Write exceeds {OPENFGA_WRITE_TUPLE_LIMIT} tuple operations") body: dict[str, Any] = {} if writes: - body["writes"] = {"tuple_keys": [self._tuple_key(t) for t in writes]} + write_payload: dict[str, Any] = { + "tuple_keys": [self._tuple_key(t) for t in writes], + } + if ignore_duplicate_writes: + write_payload["on_duplicate"] = "ignore" + body["writes"] = write_payload if deletes: body["deletes"] = {"tuple_keys": [self._tuple_key(t) for t in deletes]} return body if body else None @@ -412,6 +464,40 @@ async def _post(self, path: str, body: dict) -> dict: raise FGAClientError(f"OpenFGA {resp.status_code}: {detail}") return resp.json() + async def _streamed_post( + self, + path: str, + body: dict, + ) -> AsyncIterator[dict[str, Any]]: + """Yield validated NDJSON objects and fail if the stream is incomplete.""" + + try: + async with self._http.stream( + "POST", + path, + json=body, + headers={"Accept": "application/x-ndjson"}, + ) as resp: + if resp.status_code >= 400: + await resp.aread() + raise FGAClientError(f"OpenFGA {resp.status_code}: {resp.text[:500]}") + async for line in resp.aiter_lines(): + if not line.strip(): + continue + try: + item = json.loads(line) + except json.JSONDecodeError as exc: + raise FGAClientError("OpenFGA streamed an invalid NDJSON item") from exc + if not isinstance(item, dict): + raise FGAClientError("OpenFGA streamed a non-object NDJSON item") + yield item + except FGAClientError: + raise + except (httpx.ConnectError, httpx.TimeoutException) as exc: + raise FGAConnectionError(f"OpenFGA stream incomplete: {exc}") from exc + except httpx.HTTPError as exc: + raise FGAClientError(f"OpenFGA stream HTTP error: {exc}") from exc + def _post_sync(self, path: str, body: dict) -> dict: """POST JSON synchronously and return parsed response.""" try: diff --git a/src/backend/bisheng/knowledge/domain/models/knowledge.py b/src/backend/bisheng/knowledge/domain/models/knowledge.py index 1bc8fe5452..a9ace44dfa 100644 --- a/src/backend/bisheng/knowledge/domain/models/knowledge.py +++ b/src/backend/bisheng/knowledge/domain/models/knowledge.py @@ -706,6 +706,31 @@ async def async_get_spaces_by_ids(cls, space_ids: list[int], order_by: str = "up result = await session.exec(statement) return result.all() + @classmethod + async def async_get_joined_spaces_by_visible_ids( + cls, + space_ids: list[int], + *, + tenant_id: int, + exclude_creator_id: int, + order_by: str = "update_time", + ) -> list[Knowledge]: + """Load one bounded joined-space ID chunk under canonical DB filters.""" + + if not space_ids: + return [] + statement = select(Knowledge).where( + Knowledge.id.in_(space_ids), + Knowledge.tenant_id == tenant_id, + Knowledge.type == KnowledgeTypeEnum.SPACE.value, + Knowledge.state == KnowledgeState.PUBLISHED.value, + Knowledge.user_id != exclude_creator_id, + ) + statement = cls._apply_space_order(statement, order_by) + async with get_async_db_session() as session: + result = await session.exec(statement) + return result.all() + @classmethod def get_public_spaces(cls, order_by: str = "update_time") -> list[Knowledge]: """Get all PUBLIC and APPROVAL Knowledge Spaces (Knowledge Square)""" @@ -884,8 +909,8 @@ async def async_count_public_spaces(cls, keyword: str | None = None) -> int: @staticmethod def _apply_space_order(statement, order_by: str): if order_by == "create_time": - return statement.order_by(Knowledge.create_time.desc()) + return statement.order_by(Knowledge.create_time.desc(), Knowledge.id.desc()) elif order_by == "name": - return statement.order_by(Knowledge.name.asc()) + return statement.order_by(Knowledge.name.asc(), Knowledge.id.asc()) else: - return statement.order_by(Knowledge.update_time.desc()) + return statement.order_by(Knowledge.update_time.desc(), Knowledge.id.desc()) diff --git a/src/backend/bisheng/knowledge/domain/models/knowledge_space_file.py b/src/backend/bisheng/knowledge/domain/models/knowledge_space_file.py index 004d4c6fdc..720936f965 100644 --- a/src/backend/bisheng/knowledge/domain/models/knowledge_space_file.py +++ b/src/backend/bisheng/knowledge/domain/models/knowledge_space_file.py @@ -14,7 +14,7 @@ ) # F027 AD-14: file extension priority for "file_type" sort order. -# Same 15-WHEN ranking used by `SpaceFileDao.order_field_text`'s SQL CASE. +# Same 16-WHEN ranking used by `SpaceFileDao.order_field_text`'s SQL CASE. # Files not matching any of these (folders, unknown extensions) get rank 999. _EXT_PRIORITIES: list[tuple] = [ ("pdf", 1), @@ -231,7 +231,7 @@ async def async_list_children( statement = statement.where(build_keyset_where(sort_cols, tuple(cursor), descending=descending)) if page_size: statement = statement.limit(page_size) - statement = statement.order_by(text(cls.order_field_text(order_field, order_sort))) + statement = statement.order_by(text(f"{cls.order_field_text(order_field, order_sort)}, id desc")) else: if page and page_size: statement = statement.offset((page - 1) * page_size).limit(page_size) diff --git a/src/backend/bisheng/knowledge/domain/schemas/knowledge_space_schema.py b/src/backend/bisheng/knowledge/domain/schemas/knowledge_space_schema.py index 9c7fefad25..4b52a99178 100644 --- a/src/backend/bisheng/knowledge/domain/schemas/knowledge_space_schema.py +++ b/src/backend/bisheng/knowledge/domain/schemas/knowledge_space_schema.py @@ -15,6 +15,19 @@ class SpaceSubscriptionStatusEnum(str, Enum): NOT_SUBSCRIBED = "not_subscribed" +class KnowledgeSpaceListItemResp(KnowledgeBase): + """Lightweight knowledge-space list item without detail-only counters or metadata.""" + + id: int = Field(..., description="Knowledge Space ID") + is_pinned: bool = Field(default=False, description="Knowledge Space pinned by current user or not") + is_followed: bool = Field(default=False, description="Knowledge Space followed by current user or not") + subscription_status: SpaceSubscriptionStatusEnum = Field( + default=SpaceSubscriptionStatusEnum.NOT_SUBSCRIBED, + description="Current user subscription status", + ) + user_role: UserRoleEnum | None = Field(default=None, description="Knowledge Space user role") + + class KnowledgeSpaceCreateReq(BaseModel): name: str = Field(..., max_length=200, description="Knowledge Space Name") description: str | None = Field(None, description="Knowledge Space Description") diff --git a/src/backend/bisheng/knowledge/domain/services/department_knowledge_space_service.py b/src/backend/bisheng/knowledge/domain/services/department_knowledge_space_service.py index 2e1619ef63..d086d1f1b2 100644 --- a/src/backend/bisheng/knowledge/domain/services/department_knowledge_space_service.py +++ b/src/backend/bisheng/knowledge/domain/services/department_knowledge_space_service.py @@ -20,7 +20,7 @@ from bisheng.knowledge.domain.models.department_knowledge_space import ( DepartmentKnowledgeSpaceDao, ) -from bisheng.knowledge.domain.models.knowledge import AuthTypeEnum, KnowledgeDao +from bisheng.knowledge.domain.models.knowledge import AuthTypeEnum, KnowledgeDao, KnowledgeRead from bisheng.knowledge.domain.schemas.knowledge_space_schema import ( DepartmentKnowledgeSpaceBatchCreateReq, DepartmentKnowledgeSpaceVisibilityReq, @@ -468,8 +468,7 @@ async def get_user_department_spaces( request: Request, login_user: UserPayload, order_by: str = "update_time", - ) -> list[KnowledgeSpaceInfoResp]: - members = await SpaceChannelMemberDao.async_get_user_space_members(login_user.user_id) + ) -> list[KnowledgeRead]: all_bindings = await DepartmentKnowledgeSpaceDao.aget_all() candidate_ids = [int(binding.space_id) for binding in all_bindings] action_map = await batch_check_business_actions( @@ -478,18 +477,12 @@ async def get_user_department_spaces( resource_ids=candidate_ids, actions=("visible",), ) - space_ids = {space_id for space_id in candidate_ids if "visible" in action_map.get(str(space_id), frozenset())} + space_ids = [space_id for space_id in candidate_ids if "visible" in action_map.get(str(space_id), frozenset())] if not space_ids: return [] - filtered_members = [member for member in members if int(member.business_id) in space_ids] svc = KnowledgeSpaceService(request=request, login_user=login_user) - return await svc._format_accessible_spaces( - list(space_ids), - order_by, - memberships=filtered_members, - required_action="visible", - ) + return await svc._format_basic_spaces(space_ids, order_by) @classmethod async def get_all_department_spaces( diff --git a/src/backend/bisheng/knowledge/domain/services/knowledge_space_service.py b/src/backend/bisheng/knowledge/domain/services/knowledge_space_service.py index 37ae31371f..fb5f579b59 100644 --- a/src/backend/bisheng/knowledge/domain/services/knowledge_space_service.py +++ b/src/backend/bisheng/knowledge/domain/services/knowledge_space_service.py @@ -6,6 +6,7 @@ from dataclasses import replace from datetime import datetime from pathlib import Path +from time import perf_counter from typing import TYPE_CHECKING from urllib.parse import urlparse @@ -41,6 +42,7 @@ SpaceTenantMismatchError, ) from bisheng.common.errcode.llm import WorkbenchEmbeddingError +from bisheng.common.errcode.permission import PermissionEnumerationIncompleteError from bisheng.common.models.space_channel_member import ( BusinessTypeEnum, MembershipStatusEnum, @@ -48,6 +50,7 @@ SpaceChannelMemberDao, UserRoleEnum, ) +from bisheng.common.services.metric_log import emit_metric from bisheng.core.context.tenant import get_current_tenant_id from bisheng.core.database import get_async_db_session from bisheng.core.storage.minio.minio_manager import get_minio_storage_sync @@ -86,6 +89,7 @@ FolderUploadItem, KnowledgeSpaceFileResponse, KnowledgeSpaceInfoResp, + KnowledgeSpaceListItemResp, SpaceSubscriptionStatusEnum, ) from bisheng.knowledge.domain.services.knowledge_audit_telemetry_service import ( @@ -112,6 +116,7 @@ ) from bisheng.permission.application.business_authorization import ( batch_check_business_actions, + batch_check_business_visible, check_business_action, ) from bisheng.permission.application.identity import resolve_permission_actor @@ -178,6 +183,8 @@ _logger = logging.getLogger(__name__) _CHILD_PERMISSION_SCAN_BATCH_SIZE = 100 +_JOINED_VISIBLE_MAX_RESULTS = 5_000 +_JOINED_DB_ID_BATCH_SIZE = 500 # F040: keyword-search batch-scan window. A batch shrunk by ReBAC visibility # filtering is refilled from the next OFFSET window, so this only bounds per-round # DB fetch + visibility evaluation, not the page size. @@ -1435,7 +1442,52 @@ async def update_knowledge_space( # ──────────────────────────── Listings ──────────────────────────────────── - async def _format_member_spaces(self, members: list[SpaceChannelMember], order_by: str) -> list[KnowledgeRead]: + async def _format_basic_spaces(self, space_ids: list[int], order_by: str) -> list[KnowledgeRead]: + """Format an already-authorized space list with only per-user pin state.""" + if not space_ids: + return [] + + spaces = await KnowledgeDao.async_get_spaces_by_ids(space_ids, order_by) + return await self._format_basic_space_rows(spaces) + + async def _format_basic_space_rows(self, spaces: list[Knowledge]) -> list[KnowledgeRead]: + """Format already-loaded rows without permission, count, or metadata reads.""" + + if not spaces: + return [] + pinned_ids = await KnowledgeSpaceUserPinDao.list_pinned_space_ids(self.login_user.user_id) + results = [ + KnowledgeRead( + **space.model_dump(), + is_pinned=space.id in pinned_ids, + ) + for space in spaces + ] + return [item for item in results if item.is_pinned] + [item for item in results if not item.is_pinned] + + @staticmethod + def _sort_joined_space_rows(spaces: list[Knowledge], order_by: str) -> list[Knowledge]: + """Restore one deterministic business order after chunked ID queries.""" + + if order_by == "name": + return sorted(spaces, key=lambda row: (row.name.casefold(), int(row.id or 0))) + if order_by == "create_time": + return sorted( + spaces, + key=lambda row: (row.create_time or datetime.min, int(row.id or 0)), + reverse=True, + ) + return sorted( + spaces, + key=lambda row: (row.update_time or datetime.min, int(row.id or 0)), + reverse=True, + ) + + async def _format_member_spaces( + self, + members: list[SpaceChannelMember], + order_by: str, + ) -> list[KnowledgeSpaceListItemResp]: if not members: return [] @@ -1452,7 +1504,7 @@ async def _format_member_spaces(self, members: list[SpaceChannelMember], order_b if one.id in pinned_ids: pinned_spaces.append( - KnowledgeSpaceInfoResp( + KnowledgeSpaceListItemResp( **one.model_dump(), is_pinned=True, user_role=member_conf.user_role, @@ -1462,7 +1514,7 @@ async def _format_member_spaces(self, members: list[SpaceChannelMember], order_b ) else: normal_spaces.append( - KnowledgeSpaceInfoResp( + KnowledgeSpaceListItemResp( **one.model_dump(), is_pinned=False, user_role=member_conf.user_role, @@ -1470,11 +1522,9 @@ async def _format_member_spaces(self, members: list[SpaceChannelMember], order_b is_followed=True, ) ) - ordered = pinned_spaces + normal_spaces - await self._populate_root_file_counts(ordered) - return await self._decorate_department_metadata(ordered) + return pinned_spaces + normal_spaces - async def get_my_created_spaces(self, order_by: str = "update_time") -> list[KnowledgeRead]: + async def get_my_created_spaces(self, order_by: str = "update_time") -> list[KnowledgeSpaceListItemResp]: members = await SpaceChannelMemberDao.async_get_user_created_members(self.login_user.user_id) if members: department_space_ids = set( @@ -1526,23 +1576,60 @@ async def get_my_managed_spaces(self, order_by: str = "name") -> list[KnowledgeR ) async def get_my_followed_spaces(self, order_by: str = "update_time") -> list[KnowledgeRead]: - """ - Return the spaces the current user follows (non-creator). - Pinned spaces always appear first; within each pinned/non-pinned group - the caller-specified order_by is applied. - """ - # Fetch members ordered by is_pinned DESC so we know which are pinned - members = await SpaceChannelMemberDao.async_get_user_followed_members(self.login_user.user_id) - space_ids = {int(member.business_id) for member in members} - if not self.login_user.is_admin(): - space_ids.update(await self._scan_space_action_ids("visible")) - return await self._format_accessible_spaces( - list(space_ids), - order_by, - memberships=members, - exclude_created=True, - required_action="visible", + """Return complete personally visible spaces created by another user.""" + + started = perf_counter() + runtime = await get_f048_runtime() + actor = await self._permission_actor() + fga_started = perf_counter() + visible = await runtime.list_visible_objects( + actor, + resource_type="knowledge_space", + max_results=_JOINED_VISIBLE_MAX_RESULTS, ) + fga_elapsed_ms = (perf_counter() - fga_started) * 1000 + try: + visible_ids = [int(resource_id) for resource_id in visible.object_ids] + except (TypeError, ValueError) as exc: + raise PermissionEnumerationIncompleteError( + msg="Knowledge-space visible enumeration returned an invalid resource ID", + exception=exc, + ) from exc + + spaces: list[Knowledge] = [] + db_started = perf_counter() + for offset in range(0, len(visible_ids), _JOINED_DB_ID_BATCH_SIZE): + spaces.extend( + await KnowledgeDao.async_get_joined_spaces_by_visible_ids( + visible_ids[offset : offset + _JOINED_DB_ID_BATCH_SIZE], + tenant_id=actor.current_tenant_id, + exclude_creator_id=actor.user_id, + order_by=order_by, + ) + ) + result = await self._format_basic_space_rows(self._sort_joined_space_rows(spaces, order_by)) + emit_metric( + "permission_visible_list", + tenant=actor.current_tenant_id, + resource_type="knowledge_space", + strategy="visible_ids_first_joined", + candidate_count=len(visible_ids), + visible_count=len(visible_ids), + scanned_count=len(visible_ids), + scan_amplification=1 if visible_ids else 0, + stream_completed=True, + capacity=_JOINED_VISIBLE_MAX_RESULTS, + db_elapsed_ms=(perf_counter() - db_started) * 1000, + fga_elapsed_ms=fga_elapsed_ms, + total_elapsed_ms=(perf_counter() - started) * 1000, + returned_count=len(result), + alert=( + "capacity_80_percent" + if len(visible_ids) >= _JOINED_VISIBLE_MAX_RESULTS * 0.8 + else None + ), + ) + return result async def alist_mine_and_joined_cursor( self, @@ -2030,14 +2117,14 @@ async def _filter_visible_child_items( for resource_type, resource_ids in by_type.items(): if not resource_ids: continue - action_map = await self._batch_actions( - resource_type, - resource_ids, - ("visible",), + visible_map = await batch_check_business_visible( + self.login_user, + resource_type=resource_type, + resource_ids=resource_ids, ) for resource_id in resource_ids: permissions[(resource_type, str(resource_id))] = ( - {"visible"} if "visible" in action_map.get(str(resource_id), frozenset()) else set() + {"visible"} if visible_map.get(str(resource_id), False) else set() ) return [ item @@ -2062,19 +2149,30 @@ async def _scan_visible_child_items( page_size: int, cursor: list | None = None, exclude_file_ids: list[int] | None = None, - ) -> tuple[list[KnowledgeFile], bool]: + ) -> tuple[list[KnowledgeFile], bool, list | None]: """F027 cursor-paginated scan: keep fetching batches via keyset, fold through ReBAC filtering, stop once we've accumulated ``page_size + 1`` visible items (the +1 probes ``has_more``) or the DB is exhausted. - Returns ``(visible_page_items, has_more)`` — the visible items are - already truncated to ``page_size`` if ``has_more`` is True. + Returns visible items, ``has_more``, and the last consumed candidate + key. The response cursor must resume after that candidate, not after the + last visible row, so denied rows are not rescanned on the next page. """ from bisheng.knowledge.domain.models.knowledge_space_file import _compute_ext_rank_python visible_page_items: list[KnowledgeFile] = [] permission_context = await self._build_child_permission_context(space_id) batch_cursor: list | None = list(cursor) if cursor else None + resume_cursor: list | None = None + scanned_candidates = 0 + + def candidate_cursor(item: KnowledgeFile) -> list: + return [ + item.file_type, + _compute_ext_rank_python(item.file_name), + item.update_time, + item.id, + ] while True: batch_items = await SpaceFileDao.async_list_children( @@ -2098,28 +2196,56 @@ async def _scan_visible_child_items( space_id=space_id, context=permission_context, ) - for item in visible_batch: - visible_page_items.append(item) - if len(visible_page_items) > page_size: - # Got the +1 probe — done scanning. - return visible_page_items[:page_size], True + visible_ids = {item.id for item in visible_batch} + for item in batch_items: + if item.id in visible_ids: + if len(visible_page_items) == page_size: + amplification = scanned_candidates / max(len(visible_page_items), 1) + emit_metric( + "permission_visible_list", + resource_type="space_child", + strategy="candidate_first_batch_check", + candidate_count=scanned_candidates, + visible_count=len(visible_page_items), + scanned_candidates=scanned_candidates, + returned_items=len(visible_page_items), + scanned_count=scanned_candidates, + scan_amplification=amplification, + stream_completed=True, + has_more=True, + alert="scan_amplification" if amplification > 10 else None, + ) + return visible_page_items, True, resume_cursor + visible_page_items.append(item) + scanned_candidates += 1 + resume_cursor = candidate_cursor(item) # Advance batch_cursor to the LAST DB row of this batch (not last # visible) so the next batch picks up strictly after; if we used # the last visible, items filtered out between them would be # re-emitted on the next batch. last_db = batch_items[-1] - batch_cursor = [ - last_db.file_type, - _compute_ext_rank_python(last_db.file_name), - last_db.update_time, - last_db.id, - ] + batch_cursor = candidate_cursor(last_db) if len(batch_items) < _CHILD_PERMISSION_SCAN_BATCH_SIZE: break - return visible_page_items, False + amplification = scanned_candidates / max(len(visible_page_items), 1) + emit_metric( + "permission_visible_list", + resource_type="space_child", + strategy="candidate_first_batch_check", + candidate_count=scanned_candidates, + visible_count=len(visible_page_items), + scanned_candidates=scanned_candidates, + returned_items=len(visible_page_items), + scanned_count=scanned_candidates, + scan_amplification=amplification, + stream_completed=True, + has_more=False, + alert="scan_amplification" if amplification > 10 else None, + ) + return visible_page_items, False, None async def _scan_visible_search_items( self, @@ -2202,7 +2328,6 @@ async def list_space_children( from bisheng.common.cursor import CursorDecodeError, decode_cursor, encode_cursor from bisheng.common.errcode.knowledge_space import KnowledgeSpaceInvalidCursorError from bisheng.common.schemas.api import PageInfiniteCursorData - from bisheng.knowledge.domain.models.knowledge_space_file import _compute_ext_rank_python await self._require_read_permission(space_id) if parent_id: @@ -2229,7 +2354,7 @@ async def list_space_children( if self.version_repo is not None: exclude_file_ids = await self.version_repo.find_non_primary_file_ids() or None - visible_page_items, has_more = await self._scan_visible_child_items( + visible_page_items, has_more, scan_cursor = await self._scan_visible_child_items( space_id=space_id, parent_id=parent_id, file_ids=file_ids, @@ -2248,17 +2373,8 @@ async def list_space_children( data = await self._handle_file_folder_extra_info(visible_page_items) next_cursor: str | None = None - if has_more and visible_page_items: - last = visible_page_items[-1] - next_cursor = encode_cursor( - ( - last.file_type, - _compute_ext_rank_python(last.file_name), - last.update_time, - last.id, - ), - context=context, - ) + if has_more and scan_cursor: + next_cursor = encode_cursor(scan_cursor, context=context) return PageInfiniteCursorData( data=data, diff --git a/src/backend/bisheng/linsight/api/endpoints/linsight.py b/src/backend/bisheng/linsight/api/endpoints/linsight.py index 428682d279..6e241d5c5d 100644 --- a/src/backend/bisheng/linsight/api/endpoints/linsight.py +++ b/src/backend/bisheng/linsight/api/endpoints/linsight.py @@ -202,7 +202,7 @@ async def event_generator(): response_data = { "message_session": message_session_model.model_dump(), - "linsight_session_version": linsight_session_version_model.model_dump(), + "linsight_session_version": linsight_session_version_model.public_dump(), } except BaseErrorCode as e: # Typed business errors (folder-upload limits, …) keep their own code so @@ -577,7 +577,7 @@ async def get_linsight_session_version_list( # version's linked task message_id + liked so the standalone linsight page can # rate through the shared /liked endpoint and re-highlight on reload (same as # the in-conversation task turn). - version_dumps = [model.model_dump() for model in linsight_session_version_models] + version_dumps = [model.public_dump() for model in linsight_session_version_models] # ChatMessage is tenant-aware too — without the same widening, a cross-tenant # share recipient silently loses the like/dislike state. with bypass_tenant_filter_if(share_link is not None): diff --git a/src/backend/bisheng/linsight/domain/models/linsight_session_version.py b/src/backend/bisheng/linsight/domain/models/linsight_session_version.py index fe5d5a322c..231ad27c5f 100644 --- a/src/backend/bisheng/linsight/domain/models/linsight_session_version.py +++ b/src/backend/bisheng/linsight/domain/models/linsight_session_version.py @@ -78,6 +78,17 @@ class LinsightSessionVersionBase(SQLModelSerializable): files: list[dict] | None = Field( None, description="Uploaded files list:", sa_column=Column(JsonType, nullable=True) ) + # Raw submitted file refs the worker still has to ingest. Attachment parsing + # runs the full ETL (minutes on a multi-PDF batch) and used to happen inside + # the submit request, which nginx cut at 300s; it now happens in the worker + # right before the run. ``files`` keeps meaning "ingested and usable", so + # nothing downstream had to learn about this staging column. Worker-only: + # API responses go through ``public_dump``, which drops it. + pending_files: list[dict] | None = Field( + None, + description="Submitted file refs awaiting worker-side ingestion", + sa_column=Column(JsonType, nullable=True), + ) # F035: per-task execution model id chosen at submit time (nullable; falls # back to the tenant linsight_default_model_id when empty). model: str | None = Field(None, description="Per-task execution model id", sa_type=Text, nullable=True) @@ -143,6 +154,19 @@ class LinsightSessionVersion(LinsightSessionVersionBase, table=True): __tablename__ = "linsight_session_version" + def public_dump(self) -> dict: + """``model_dump`` for anything that leaves the server. + + Drops ``pending_files``: each entry carries the presigned temp-bucket + link the browser uploaded with (7-day validity), and the version list is + reachable through a share link — echoing them would hand a share + recipient direct URLs to the submitter's originals. The worker reads the + column off the row / Redis snapshot, never off an API response. + """ + data = self.model_dump() + data.pop("pending_files", None) + return data + class LinsightSessionVersionDao: """ diff --git a/src/backend/bisheng/linsight/domain/services/agent_factory.py b/src/backend/bisheng/linsight/domain/services/agent_factory.py index 804571545d..a9ed2751dc 100644 --- a/src/backend/bisheng/linsight/domain/services/agent_factory.py +++ b/src/backend/bisheng/linsight/domain/services/agent_factory.py @@ -271,7 +271,7 @@ def _build_linsight_system_prompt( exec_line = ( " - 需要资料时用 search_knowledge_base 检索知识库/知识空间；" "读写文件用 write_file / read_file / edit_file / ls。\n" - " - 用户上传的音视频（mp3/mp4 等）已在 submit 阶段 ASR 转写为 uploads/*.md；" + " - 用户上传的音视频（mp3/mp4 等）已由 ASR 转写为 uploads/*.md；" " 中 path 为转写文本、name 为原始文件名。" "read_file(path) 即可获取内容，勿推断为「扩展名标注有误」或「实际是文本文件」。" ) @@ -293,7 +293,7 @@ def _build_linsight_system_prompt( " - 读写文件用 write_file / read_file / edit_file / ls；若用户上传了文件，" "用 ls / read_file 在工作区中查阅。本次任务没有可检索的知识库/知识空间，" "请基于已有资料与自身知识完成，不要调用任何知识库检索工具。\n" - " - 用户上传的音视频（mp3/mp4 等）已在 submit 阶段 ASR 转写为 uploads/*.md；" + " - 用户上传的音视频（mp3/mp4 等）已由 ASR 转写为 uploads/*.md；" " 中 path 为转写文本、name 为原始文件名。" "read_file(path) 即可获取内容，勿推断为「扩展名标注有误」或「实际是文本文件」。" ) @@ -758,7 +758,7 @@ async def create_linsight_agent( from deepagents import create_deep_agent svid = svid or session_model.id - model = await _resolve_model(session_model, model_id) + model, supports_vision = await _resolve_model(session_model, model_id) if backend is None: backend = _default_backend(svid, file_dir) @@ -792,7 +792,7 @@ async def create_linsight_agent( middlewares: list = [ build_resilience_middleware(linsight_conf, is_subagent=False, budget_sink=turn_budget_sink), build_tool_loop_breaker_middleware(linsight_conf, is_subagent=False), - *build_binary_guards(has_code_interpreter), + *build_binary_guards(has_code_interpreter, supports_vision=supports_vision), ] # F035 Track D — skills (RE-ENABLED 2026-06-24, Fork X). The run's allowed skill @@ -866,7 +866,7 @@ async def create_linsight_agent( # reading an original would fail the whole task. It sees the same code # interpreter as the main graph (not in _SUBAGENT_TOOL_DENY), so the flag # carries over; revisit if it is ever added to that deny list. - *build_binary_guards(has_code_interpreter), + *build_binary_guards(has_code_interpreter, supports_vision=supports_vision), # Same tail language directive on the subagent's own stack (last -> after # its TodoList/Filesystem framework prompts), so the researcher also # reasons in the user's language. @@ -899,12 +899,21 @@ async def create_linsight_agent( ) -async def _resolve_model(session_model, model_id: str | None) -> BaseChatModel: +async def _resolve_model(session_model, model_id: str | None) -> tuple[BaseChatModel, bool]: """Resolve the per-task model via LLMService (design §2.2 / §2.2.1). Priority: per-task ``model_id`` -> tenant Linsight default model. Tenant resolution + share fallback live inside ``get_bisheng_linsight_llm`` (INV-T18: tenant_id is threaded explicitly in the Worker subprocess). + + Returns ``(model, supports_vision)``. ``supports_vision`` is the admin-declared + ``WSModel.visual`` flag — the 视觉 column in 系统模型设置 → 工作台模型, which + daily chat already gates image attachments on. Linsight picks its model from + that SAME list (``models[]``, the id ``linsight_default_model_id`` selects + from), so the lookup is always resolvable; a model id that is somehow absent + reads as False, matching the field's own default. Returned from here rather + than fetched separately because ``workbench_conf`` is already in hand — + ``get_workbench_llm`` re-validates model refs and is not free to call twice. """ workbench_conf = await LLMService.get_workbench_llm(tenant_id=session_model.tenant_id) linsight_conf = settings.get_linsight_conf() @@ -920,11 +929,16 @@ async def _resolve_model(session_model, model_id: str | None) -> BaseChatModel: "linsight_default_model_id is not configured" ) - return await LLMService.get_bisheng_linsight_llm( + supports_vision = any( + str(entry.id) == str(resolved_id) and bool(entry.visual) for entry in (workbench_conf.models or []) + ) + + model = await LLMService.get_bisheng_linsight_llm( invoke_user_id=session_model.user_id, model_id=resolved_id, temperature=linsight_conf.default_temperature, ) + return model, supports_vision def _default_backend(svid: str, file_dir: str | None): diff --git a/src/backend/bisheng/linsight/domain/services/binary_content_guard.py b/src/backend/bisheng/linsight/domain/services/binary_content_guard.py index 47d0b2a90e..67394721b6 100644 --- a/src/backend/bisheng/linsight/domain/services/binary_content_guard.py +++ b/src/backend/bisheng/linsight/domain/services/binary_content_guard.py @@ -31,8 +31,22 @@ ``video`` in particular MUST die here: it raises inside langchain-core before any HTTP call, so ``llm_error_classifier`` never sees a status code to bucket. -``image`` blocks are left alone on purpose — they are the one multimodal shape -mainstream providers do accept, and dropping them would break vision models. +``image`` blocks get their own two-part treatment, because two independent things +can make an image unsendable: + +- **Can this model see at all?** ``supports_vision`` is the admin-declared + ``WSModel.visual`` flag behind the 视觉 checkbox in 系统模型设置 → 工作台模型 + (daily chat already gates attachments on it, ``chat_service._process_agent_files``). + When it is off, an image is refused at the TOOL layer with an actionable hint — + the model never receives a payload its endpoint would reject. +- **Where is the model willing to see it?** deepagents' ``read_file`` returns an + image as a block on the TOOL message (``deepagents/middleware/filesystem.py``), + a shape some endpoints refuse outright: Kimi K3 answers ``image_url parts are + supported only in user messages`` with a 400 that kills the session. Every + provider accepts an image in the USER role, so ``_sanitize_messages`` MOVES it + there rather than maintaining a per-vendor table that would always lag the next + model. Dropping it instead was rejected — it would regress the scanned-page + workflow this passthrough exists for. """ from __future__ import annotations @@ -41,7 +55,7 @@ from typing import Any from langchain.agents.middleware.types import AgentMiddleware, ModelRequest, ModelResponse -from langchain_core.messages import ToolMessage +from langchain_core.messages import HumanMessage, ToolMessage from langgraph.prebuilt.tool_node import ToolCallRequest from langgraph.types import Command from loguru import logger @@ -58,6 +72,11 @@ # us. `image` is intentionally absent (see module docstring). _BLOCKED_BLOCK_TYPES = frozenset({"file", "audio", "video", "input_audio"}) +# The two shapes an image arrives in: deepagents' standard block (`image` + +# `base64`) and the OpenAI-native `image_url`. Both convert to the same outgoing +# `image_url` part, so both need relocating. +_IMAGE_BLOCK_TYPES = frozenset({"image", "image_url"}) + # A `decode("utf-8", errors="replace")` of binary bytes is dominated by U+FFFD. # Real text files carry a few at most (a stray mis-encoded byte), so a small # ratio separates the two cleanly. NUL bytes never appear in legitimate text. @@ -99,6 +118,34 @@ def _binary_read_hint(file_path: str, has_code_interpreter: bool) -> str: ) +def _no_vision_hint(file_path: str, has_code_interpreter: bool) -> str: + """Replacement for an image read when the run's model has no vision. + + Deliberately names the cause: a model that silently receives nothing and is + left to guess will confabulate a description of the page. Telling it plainly + that it cannot see, and that saying so is the correct outcome, is the honest + behaviour — the deliverable then carries "not visually verified" instead of an + invented reading. + """ + if has_code_interpreter: + route_zh = "- 若内容重要：用 bisheng_code_interpreter 从图片的来源文件中提取文字/数据（例如用 fitz 抽 PDF 那一页的文本）。\n" + route_en = "If the content matters, extract it from the source file with bisheng_code_interpreter instead." + else: + route_zh = "- 本次没有可用的代码执行工具，无法从原件提取内容。\n" + route_en = "No code execution tool is available this run to extract it from the source." + return ( + f"[System] `{file_path}` 是图片，而本次运行所用的模型未被标记为具备视觉能力" + f"（系统模型设置 → 工作台模型的「视觉」列），因此无法把图片交给你查看。\n" + f"{route_zh}" + f"- 不要再对该路径重复调用 read_file，结果不会变。\n" + f"- 不要凭猜测描述图片内容；如果确实无法获得，请在交付物中如实说明该处未经视觉核对。\n" + f"[System] `{file_path}` is an image and the model configured for this run is not marked " + f"vision-capable, so it cannot be shown to you. {route_en} Do not call read_file on this " + f"path again, and never guess at the image's contents — state plainly in the deliverable " + f"that it could not be checked visually." + ) + + def _looks_like_mojibake(text: str) -> bool: """True when ``text`` is binary bytes that got lossily decoded as UTF-8.""" if not text or len(text) < _MOJIBAKE_MIN_CHARS: @@ -148,6 +195,13 @@ def _is_valid_base64(value: Any) -> bool: return set(head) <= _B64_CHARS +def _has_image_block(content: Any) -> bool: + """True when ``content`` carries at least one image block, in either shape.""" + if not isinstance(content, list): + return False + return any(isinstance(b, dict) and b.get("type") in _IMAGE_BLOCK_TYPES for b in content) + + def _broken_image_payload(block: dict) -> bool: """True when an ``image`` block carries a payload that is not valid base64. @@ -178,11 +232,15 @@ class BinaryReadGuardMiddleware(AgentMiddleware): has_code_interpreter: whether the sandboxed code interpreter is bound for this run. Keeps the hint's suggested route in lockstep with the tools actually available (same contract as the uploaded-files pointer block). + supports_vision: whether the run's model is declared vision-capable + (``WSModel.visual``). Defaults to False — fail closed, the same way an + unset checkbox reads in daily chat. """ - def __init__(self, has_code_interpreter: bool = False) -> None: + def __init__(self, has_code_interpreter: bool = False, supports_vision: bool = False) -> None: super().__init__() self._has_code_interpreter = has_code_interpreter + self._supports_vision = supports_vision async def awrap_tool_call( self, @@ -221,6 +279,13 @@ async def awrap_tool_call( logger.info("[linsight-binary-guard] read_file returned an invalid image payload for {}", file_path) return _replace_tool_content(result, _binary_read_hint(file_path, self._has_code_interpreter)) + # A model with no declared vision capability cannot use an image block: the + # endpoint 400s on it, and a text-only model that somehow accepted it would + # see nothing. Refuse here, where the hint can still name the reason. + if not self._supports_vision and _has_image_block(content): + logger.info("[linsight-binary-guard] image read on a non-vision model for {}", file_path) + return _replace_tool_content(result, _no_vision_hint(file_path, self._has_code_interpreter)) + return result @@ -240,39 +305,116 @@ class ModelContentGuardMiddleware(AgentMiddleware): has_code_interpreter: whether the sandboxed code interpreter is bound this run. Gates the replacement text's suggested route, same lockstep rule as ``BinaryReadGuardMiddleware`` and the uploaded-files pointer block. + supports_vision: whether the run's model is declared vision-capable. The + tool guard already refuses image reads when it is off, so this is the + backstop for images that entered some other way — a replayed + checkpoint from a turn that ran on a vision model, most of all. """ - def __init__(self, has_code_interpreter: bool = False) -> None: + def __init__(self, has_code_interpreter: bool = False, supports_vision: bool = False) -> None: super().__init__() self._has_code_interpreter = has_code_interpreter + self._supports_vision = supports_vision async def awrap_model_call( self, request: ModelRequest, handler: Callable[[ModelRequest], Awaitable[ModelResponse]], ) -> ModelResponse: - sanitized, stripped = _sanitize_messages(request.messages, self._has_code_interpreter) + sanitized, stripped, relocated = _sanitize_messages( + request.messages, self._has_code_interpreter, supports_vision=self._supports_vision + ) if stripped: logger.warning("[linsight-content-guard] stripped {} block(s) before model call", sorted(stripped)) + if relocated: + logger.info("[linsight-content-guard] relocated {} tool-message image(s) into a user turn", relocated) + if stripped or relocated: request = request.override(messages=sanitized) return await handler(request) -def _sanitize_messages(messages: list, has_code_interpreter: bool = False) -> tuple[list, set[str]]: - """Return ``(messages, stripped_types)`` with blocked blocks turned into text.""" +def _image_source_path(message: Any) -> str: + """Best-effort path of the file a tool-message image came from. + + deepagents stamps ``read_file_path`` on the multimodal branch; anything else + (a future tool, a replayed checkpoint written by an older build) has no marker + and gets an empty label rather than a fabricated one. + """ + kwargs = getattr(message, "additional_kwargs", None) + if isinstance(kwargs, dict): + return str(kwargs.get("read_file_path") or "") + return "" + + +def _relocated_image_message(entries: list[tuple[str, dict]]) -> HumanMessage: + """Carry tool-returned images into a synthesized USER turn. + + The header exists to stop the model reading this as a fresh instruction from + the human: it arrives in the user role for protocol reasons only. Each image + is preceded by its own label so several relocated reads stay distinguishable. + """ + content: list[dict] = [ + { + "type": "text", + "text": ( + "[System] Image(s) returned by a tool call are relayed here because this model " + "endpoint only accepts images in user messages. This is NOT a new instruction " + "from the user — continue the task in progress." + ), + } + ] + for path, block in entries: + label = f"Image from `{path}`:" if path else "Image from the preceding tool call:" + content.append({"type": "text", "text": label}) + content.append(block) + return HumanMessage(content=content) + + +def _sanitize_messages( + messages: list, has_code_interpreter: bool = False, *, supports_vision: bool = False +) -> tuple[list, set[str], int]: + """Return ``(messages, stripped_types, relocated_count)``. + + Three transforms, all request-only — the persisted history and the checkpoint + keep the original shape, so this is reversible and never corrupts state: + + 1. blocked block types (``file``/``audio``/``video``) become text; + 2. with ``supports_vision`` off, images become text wherever they sit; + 3. otherwise images inside a ``ToolMessage`` are MOVED into a synthesized user + turn (``pending``/``flush`` below carry the ordering rule). + """ route = ( " Use bisheng_code_interpreter to inspect the original file instead." if has_code_interpreter else " No code execution tool is available this run; answer from the parsed text view." ) stripped: set[str] = set() - out = [] + relocated = 0 + out: list = [] + # Images pulled out of the tool run currently being walked, in encounter order. + # An OpenAI-compatible endpoint requires every tool message answering one + # assistant `tool_calls` batch to be CONTIGUOUS and to precede any other role, + # so the carrier turn cannot go straight after the image-bearing tool message: + # it is flushed when the tool run ends (or at the end of the list, which is the + # common case — the model reads an image and the request goes out right there). + pending: list[tuple[str, dict]] = [] + + def flush() -> None: + if pending: + out.append(_relocated_image_message(list(pending))) + pending.clear() + for message in messages: + is_tool = isinstance(message, ToolMessage) + if not is_tool: + flush() + content = getattr(message, "content", None) if not isinstance(content, list): out.append(message) continue + source_path = _image_source_path(message) if is_tool else "" new_content = [] changed = False for block in content: @@ -287,9 +429,9 @@ def _sanitize_messages(messages: list, has_code_interpreter: bool = False) -> tu } ) continue - # `image` stays (it is the one shape endpoints accept) — unless its - # payload is not real base64, in which case forwarding it either 400s - # the request or feeds the model noise it will confabulate over. + # Checked BEFORE the relocation branch: forwarding a corrupt payload + # either 400s the request or feeds the model noise it will confabulate + # over, and moving it to a user turn would not make it any more valid. if isinstance(block, dict) and _broken_image_payload(block): stripped.add("image:invalid-base64") changed = True @@ -301,6 +443,31 @@ def _sanitize_messages(messages: list, has_code_interpreter: bool = False) -> tu } ) continue + if isinstance(block, dict) and block.get("type") in _IMAGE_BLOCK_TYPES and not supports_vision: + stripped.add("image:no-vision") + changed = True + new_content.append( + { + "type": "text", + "text": "[System] An image was removed here: the model configured for this " + "run is not marked vision-capable, so it cannot be shown one. Do not guess " + f"at its contents.{route}", + } + ) + continue + if is_tool and isinstance(block, dict) and block.get("type") in _IMAGE_BLOCK_TYPES: + pending.append((source_path, block)) + relocated += 1 + changed = True + where = f"`{source_path}`" if source_path else "this path" + new_content.append( + { + "type": "text", + "text": f"[System] The image read from {where} is attached to the user message " + f"that follows this tool batch; read it from there.", + } + ) + continue new_content.append(block) if not changed: @@ -312,18 +479,23 @@ def _sanitize_messages(messages: list, has_code_interpreter: bool = False) -> tu new_content = [{"type": "text", "text": "[System] Unsupported attachment removed."}] out.append(message.model_copy(update={"content": new_content})) - return out, stripped + flush() + return out, stripped, relocated -def build_binary_guards(has_code_interpreter: bool) -> list[AgentMiddleware]: +def build_binary_guards(has_code_interpreter: bool, *, supports_vision: bool) -> list[AgentMiddleware]: """The pair of guards every graph that owns ``read_file`` must carry. Middleware is per-subgraph in langgraph: a subagent's model and tool calls are NOT wrapped by the parent graph's stack, so each one needs its own instances. Returning them from a single factory means a new subagent (e.g. the planned data-analyst) gets both by construction instead of by remembering to. + + ``supports_vision`` is keyword-only and has NO default on purpose: both guards + fail closed, so a new call site that forgot it would silently switch image + reads off rather than raise. Make the caller state it. """ return [ - BinaryReadGuardMiddleware(has_code_interpreter=has_code_interpreter), - ModelContentGuardMiddleware(has_code_interpreter=has_code_interpreter), + BinaryReadGuardMiddleware(has_code_interpreter=has_code_interpreter, supports_vision=supports_vision), + ModelContentGuardMiddleware(has_code_interpreter=has_code_interpreter, supports_vision=supports_vision), ] diff --git a/src/backend/bisheng/linsight/domain/services/resilience_middleware.py b/src/backend/bisheng/linsight/domain/services/resilience_middleware.py index 4ad8909d73..52a42dd22c 100644 --- a/src/backend/bisheng/linsight/domain/services/resilience_middleware.py +++ b/src/backend/bisheng/linsight/domain/services/resilience_middleware.py @@ -76,6 +76,53 @@ _TRUNCATION_FINISH_REASONS = frozenset({"length", "max_tokens", "max_output_tokens"}) +# --------------------------------------------------------------------------- +# Incomplete-stream guard +# +# A provider can close the SSE stream mid-turn WITHOUT raising anything: httpx +# sees a clean EOF, the openai SDK simply stops yielding chunks, and langchain +# hands back whatever it had accumulated. Nothing in the retry path above ever +# runs, because nothing failed. +# +# Measured on 114 (2026-08-13, grok4.6 via tokenrouter, 147k-token context): the +# model emitted one line of narration ("正在撰写完整参数清单与 D2 数据表。") and the +# stream closed before the ``write_file`` call it was about to make. The result +# was an AIMessage with NO tool calls — which deepagents reads as "the agent is +# done" — so a run with 3 of its 5 steps still pending completed "successfully" +# with that narration as its final answer and as its only deliverable. +# +# Fingerprint of a stream that never finished (all three must hold): +# - no finish_reason / stop_reason — a completed OpenAI-compatible or Anthropic +# stream always carries one; +# - no input token usage — the usage chunk is the last thing a completed stream +# sends, and ``input_tokens`` is never legitimately 0 (a provider that omits +# usage entirely still reports finish_reason, so this stays a confirmation +# signal, never the sole trigger); +# - no tool call at all — the only shape that silently routes the graph to END. +# A stream cut mid-tool-call is deliberately EXCLUDED: the graph keeps running +# (the tool errors out and the L2 truncation nudge / L3 loop breaker take +# over), so it is neither silent nor worth re-sending a six-figure-token +# request over. +# --------------------------------------------------------------------------- + + +class IncompleteStreamError(ConnectionError): + """The provider closed the response stream before the model finished its turn. + + Subclasses ``ConnectionError`` deliberately — that IS what happened at the + transport layer, and it makes ``classify_behavior`` bucket this as RETRYABLE and + ``label_error`` render the network-timeout card, without teaching the shared + classifier about a Linsight-only exception type. + """ + + +# Retries for an incomplete stream. Deliberately small and SEPARATE from both the +# exception-retry and truncation budgets: every attempt re-sends the entire request +# (147k input tokens in the measured case), so this spends a bounded amount of money +# to avoid silently truncating the run. +_INCOMPLETE_STREAM_RETRY_LIMIT = 2 + + # --------------------------------------------------------------------------- # Turn budget + soft landing # @@ -213,6 +260,30 @@ def _is_truncated_tool_call(response: object) -> bool: return bool(getattr(ai, "tool_calls", None)) or bool(getattr(ai, "invalid_tool_calls", None)) +def _is_incomplete_stream_response(response: object) -> bool: + """True when the provider closed the stream before the model finished its turn. + + Vendor-agnostic, and deliberately a three-way conjunction — see the + ``IncompleteStreamError`` block above for why each conjunct is needed and why a + stream cut mid-tool-call is excluded. + """ + ai = _response_ai_message(response) + if ai is None: + return False + meta = getattr(ai, "response_metadata", None) or {} + # No provider metadata at all → not a provider stream (a synthetic/degraded + # message, or a non-streaming shim). langchain fills model_name from the very + # first chunk, so a real stream — finished or cut off — always carries something. + if not meta: + return False + if meta.get("finish_reason") or meta.get("stop_reason"): + return False + if getattr(ai, "tool_calls", None) or getattr(ai, "invalid_tool_calls", None): + return False + usage = getattr(ai, "usage_metadata", None) or {} + return not usage.get("input_tokens") + + def _with_truncation_nudge(request: ModelRequest) -> ModelRequest: """A new request with the corrective nudge appended (ephemeral — retry only).""" return request.override(messages=[*request.messages, HumanMessage(content=_TRUNCATION_NUDGE)]) @@ -562,6 +633,7 @@ async def awrap_model_call( current, budget_key = self._apply_turn_budget(request) exc_attempts = 0 trunc_attempts = 0 + incomplete_attempts = 0 while True: try: response = await handler(current) @@ -594,6 +666,32 @@ async def awrap_model_call( ) current = _with_truncation_nudge(current) continue + # Incomplete-stream guard: a stream closed mid-turn yields a tool-call-less + # AIMessage that the graph reads as a clean finish. Re-send the call; only + # when it keeps coming back incomplete do we fail (main graph) / degrade + # (subagent) — the run is never allowed to silently pass off a cut-off + # narration as its answer. + if _is_incomplete_stream_response(response): + if incomplete_attempts < _INCOMPLETE_STREAM_RETRY_LIMIT: + delay = self._delay(incomplete_attempts) + logger.warning( + "[linsight-resilience] incomplete stream response " + "(no finish_reason, no usage, no tool call) " + "(attempt {}/{}); sleeping {:.1f}s", + incomplete_attempts + 1, + _INCOMPLETE_STREAM_RETRY_LIMIT, + delay, + ) + incomplete_attempts += 1 + if delay > 0: + await asyncio.sleep(delay) + continue + return self._degrade_or_raise( + IncompleteStreamError( + f"provider closed the response stream before the turn finished " + f"({_INCOMPLETE_STREAM_RETRY_LIMIT + 1} attempts)" + ) + ) # Two-phase turn accounting: the soft-landing STAGE had to be picked before # the call (it shapes the request), but whether this turn did any real work # is only knowable from the response. Refund here — the loop's SINGLE @@ -613,6 +711,7 @@ def wrap_model_call( current, budget_key = self._apply_turn_budget(request) exc_attempts = 0 trunc_attempts = 0 + incomplete_attempts = 0 while True: try: response = handler(current) @@ -632,6 +731,20 @@ def wrap_model_call( trunc_attempts += 1 current = _with_truncation_nudge(current) continue + # Incomplete-stream guard — see the async twin above. + if _is_incomplete_stream_response(response): + if incomplete_attempts < _INCOMPLETE_STREAM_RETRY_LIMIT: + delay = self._delay(incomplete_attempts) + incomplete_attempts += 1 + if delay > 0: + time.sleep(delay) + continue + return self._degrade_or_raise( + IncompleteStreamError( + f"provider closed the response stream before the turn finished " + f"({_INCOMPLETE_STREAM_RETRY_LIMIT + 1} attempts)" + ) + ) # Refund a pure state-maintenance turn — see the async twin above for why # this sits at the loop's single success exit. if _is_state_only_turn(response): diff --git a/src/backend/bisheng/linsight/domain/services/stream_event_mapper.py b/src/backend/bisheng/linsight/domain/services/stream_event_mapper.py index 21fb0d4314..e4546d8ff4 100644 --- a/src/backend/bisheng/linsight/domain/services/stream_event_mapper.py +++ b/src/backend/bisheng/linsight/domain/services/stream_event_mapper.py @@ -111,7 +111,12 @@ class _TodoProjection: class _OpenCall: """Buffered tool-call *start* frame, keyed by call_id (design §3.4).""" + # Emitted/persisted id — globally unique for this svid, see ``StreamContext.tool_seq``. call_id: str + # The provider's own tool_call id. Every in-memory bookkeeping dict + # (``open_calls`` / ``orphan_ends`` / ``tool_arg_buffers``) stays keyed by THIS, + # because that is what comes back on ``ToolMessage.tool_call_id``. + raw_call_id: str task_id: str name: str params: Any @@ -169,6 +174,18 @@ class StreamContext: # welded reasoning from different runs into one block, spliced new text into # the middle of old text, and left history ordered against its own timestamps. run_token: str = field(default_factory=lambda: uuid4().hex[:8]) + # monotonic counter giving each TOOL call a distinct, stable call_id. + # + # A provider only has to make ``tool_calls[].id`` unique WITHIN one response — + # the OpenAI contract is just "the id a ToolMessage refers back to", never a + # cross-request guarantee. tokenrouter/kimi-k3 mints ``:`` + # and restarts the index at 0 every turn, so ``bisheng_code_interpreter:0`` + # came back 156 times in one run. ``add_execution_task_step`` upserts by + # call_id across the WHOLE history, so all 156 collapsed onto one row and the + # step flow showed "运行代码 1 次". Mixed with ``run_token`` for the same reason + # thinking ids are (see above): a mapper is per-RUN, so the counter alone would + # collide again after an ask_user resume. + tool_seq: int = 0 # terminal收口 dedup: first terminal wins, later ones dropped (§3.7) terminated: bool = False @@ -496,14 +513,14 @@ def _accumulate_tool_args(self, message: Any) -> None: continue self.ctx.tool_arg_buffers[cid] = self.ctx.tool_arg_buffers.get(cid, "") + (tcc.get("args") or "") - def _parse_arg_buffer(self, call_id: str) -> dict[str, Any] | None: + def _parse_arg_buffer(self, raw_call_id: str) -> dict[str, Any] | None: """Parse a call's reassembled arg buffer once it is a complete JSON object. Returns ``None`` while the buffer is still partial (mid-stream JSON does not parse) — strict ``json.loads`` succeeds only when the object closes, so this fires exactly once per call, when its args finish streaming. """ - raw = self.ctx.tool_arg_buffers.get(call_id) + raw = self.ctx.tool_arg_buffers.get(raw_call_id) if not raw: return None try: @@ -522,7 +539,9 @@ def _enrich_delegation_from_buffer(self, open_call: _OpenCall) -> bool: """ if open_call.step_type != "subagent" or open_call.delegate_goal: return False - args = self._parse_arg_buffer(open_call.call_id) + # RAW id: ``tool_arg_buffers`` is filled by _accumulate_tool_args from the + # provider's own chunk ids, so the minted call_id would never hit. + args = self._parse_arg_buffer(open_call.raw_call_id) if not args: return False goal = args.get("description") or args.get("instruction") or "" @@ -565,9 +584,16 @@ def _handle_tool_starts(self, tool_calls: list[dict[str, Any]], ns: str | None) # segment boundary (段流重构 2026-06). task_id = self.ctx.svid for tc in tool_calls: - call_id = tc.get("id") or tc.get("call_id") - if not call_id: + raw_call_id = tc.get("id") or tc.get("call_id") + if not raw_call_id: continue + # Mint the id we emit and persist. The provider's id is only unique + # within one response (see StreamContext.tool_seq), and the persistence + # layer upserts by call_id across the whole history — so a colliding id + # silently overwrites an earlier, unrelated call. Keeping the raw id as + # the prefix leaves the row greppable ("which tool, which turn"). + self.ctx.tool_seq += 1 + call_id = f"{raw_call_id}#{self.ctx.run_token}:{self.ctx.tool_seq}" name = tc.get("name", "") args = tc.get("args") step_type = self._infer_step_type(name, ns) @@ -599,6 +625,7 @@ def _handle_tool_starts(self, tool_calls: list[dict[str, Any]], ns: str | None) open_call = _OpenCall( call_id=call_id, + raw_call_id=raw_call_id, task_id=task_id, name=name, params=args, @@ -607,7 +634,21 @@ def _handle_tool_starts(self, tool_calls: list[dict[str, Any]], ns: str | None) namespace=ns, delegate_goal=delegate_goal, ) - self.ctx.open_calls[call_id] = open_call + # An earlier call with the SAME provider id is still open — it never got + # an end frame (HITL park, recursion cut, budget refusal) and is now + # being displaced. Its half-written arg buffer would otherwise keep + # accumulating under this call's deltas: usually that just fails + # json.loads and the delegation goal degrades to empty, but a leftover + # that happens to CLOSE against this call's first fragment parses fine + # and hands us the previous call's goal. Drop it here — this is the one + # moment we can tell "stale" from "mid-stream", because a normally + # finished call was popped by its end frame. Deltas that follow refill + # the buffer from scratch. + if raw_call_id in self.ctx.open_calls: + self.ctx.tool_arg_buffers.pop(raw_call_id, None) + # Keyed by the RAW id: the end frame arrives as ToolMessage.tool_call_id, + # which carries the provider's value, not ours. + self.ctx.open_calls[raw_call_id] = open_call if ns: extra_info["namespace"] = ns @@ -627,26 +668,32 @@ def _handle_tool_starts(self, tool_calls: list[dict[str, Any]], ns: str | None) ) # An orphan end may already be buffered for this call_id (§3.7). - buffered = self.ctx.orphan_ends.pop(call_id, None) + buffered = self.ctx.orphan_ends.pop(raw_call_id, None) if buffered is not None: events.append(self._build_end_step(open_call, buffered)) return events - def _handle_tool_end(self, message: Any, call_id: str, ns: str | None) -> list[BaseEvent]: + def _handle_tool_end(self, message: Any, raw_call_id: str, ns: str | None) -> list[BaseEvent]: + """Close the open call ``ToolMessage.tool_call_id`` points at. + + ``raw_call_id`` is the PROVIDER's id — every dict touched here is keyed by + it. The emitted frame's id comes from ``open_call.call_id`` (the minted, + globally-unique one), so start and end still merge into a single history row. + """ output = self._message_text(message) payload = {"output": output} - open_call = self.ctx.open_calls.pop(call_id, None) + open_call = self.ctx.open_calls.pop(raw_call_id, None) if open_call is None: # End arrived before start — buffer it (§3.7 reordering). - self.ctx.orphan_ends[call_id] = payload + self.ctx.orphan_ends[raw_call_id] = payload return [] # Last-chance goal/params fill: if _reconcile_open_delegations never ran # for this call (e.g. the args finished streaming in the same chunk that # also carried the tool result), recover them from the buffer so the end # frame still carries the delegation goal. The buffer is then disposable. self._enrich_delegation_from_buffer(open_call) - self.ctx.tool_arg_buffers.pop(call_id, None) + self.ctx.tool_arg_buffers.pop(raw_call_id, None) return [self._build_end_step(open_call, payload)] def _build_end_step(self, open_call: _OpenCall, payload: dict[str, Any]) -> ExecStep: diff --git a/src/backend/bisheng/linsight/domain/services/workbench_impl.py b/src/backend/bisheng/linsight/domain/services/workbench_impl.py index d5178768f3..bf062c8cf3 100644 --- a/src/backend/bisheng/linsight/domain/services/workbench_impl.py +++ b/src/backend/bisheng/linsight/domain/services/workbench_impl.py @@ -4,6 +4,7 @@ import os import time import uuid +from collections.abc import Awaitable, Callable from dataclasses import dataclass from io import BytesIO from typing import Any @@ -229,6 +230,7 @@ async def submit_user_question( submit_obj: LinsightQuestionSubmitSchema, login_user: UserPayload, display_files: list[dict] | None = None, + defer_ingest: bool = True, ) -> tuple[MessageSession, LinsightSessionVersion]: """ Submit user issue and create session @@ -241,6 +243,12 @@ async def submit_user_question( ChatMessage so the uploaded attachments render after a refresh, mirroring the daily-chat question envelope. ``None`` for the legacy /linsight entry (it renders attachments its own way). + defer_ingest: Park the raw file refs in ``pending_files`` and let the + worker ingest them (``ingest_pending_files``) instead of parsing + them here. Ingestion runs the full ETL — 12 PDFs measured at 19 + minutes — so inline parsing put the whole batch inside one HTTP + request, which nginx cut at 300s. Set False only where the caller + genuinely needs the files materialized before it returns. Returns: tuple: (Message Session Model, Inspiration Conversation Version Model) @@ -249,6 +257,10 @@ async def submit_user_question( LinsightError: When creating a session fails """ try: + # Metadata-only preconditions, always in-request: they carry the typed + # codes the frontend branches on, which a deferred ingest could only + # report minutes later as a generic task failure. + cls.validate_submitted_files(submit_obj.files) # Continue an existing session when session_id is supplied, else # start a fresh one. Continuing reuses the MessageSession and only # appends a new version, so follow-up rounds stay in one 会话 (F035). @@ -271,7 +283,12 @@ async def submit_user_question( # svid up-front and use it both for ingestion and the version row id. svid = uuid.uuid4().hex # Process files (if present) — after chat_id is finalized - processed_files = await cls._process_submitted_files(submit_obj.files, svid, login_user.user_id) + if defer_ingest: + processed_files = None + pending_files = [f.model_dump() for f in (submit_obj.files or [])] or None + else: + processed_files = await cls._process_submitted_files(submit_obj.files, svid, login_user.user_id) + pending_files = None if not continuing: # F035 Track J (unified conversation model): a task turn is not a @@ -315,6 +332,7 @@ async def submit_user_question( organization_knowledge_ids=submit_obj.organization_knowledge_ids, knowledge_space_ids=submit_obj.knowledge_space_ids, files=processed_files, + pending_files=pending_files, model=submit_obj.model, skills=submit_obj.skills, ) @@ -323,18 +341,23 @@ async def submit_user_question( # F035 Track J: land the user question in the unified conversation # stream so the round reads as one Q→A pair regardless of task mode. # (The bot answer turn is written at completion in task_exec.) - # Annotate the persisted attachments with each file's parse result so - # the attachment chip can show a "parse failed" state after a refresh. await linsight_execute_utils.persist_task_user_turn( chat_id=chat_id, user_id=login_user.user_id, question=submit_obj.question, - # Attachments that came in through the shared upload endpoint are - # still sitting in the temp bucket; the ones ingested by linsight - # already have an object_name and are skipped. + # Deferring the ingest means EVERY attachment is still sitting in + # the temp bucket here, so all of them get promoted — the chat + # copy can no longer piggyback on the workspace original the + # ingest used to have written by now. That is one extra + # server-side copy_object per file, against a message whose + # attachments would otherwise vanish with the temp bucket's 3-day + # rule if the worker never ran. The parse result (video cover, + # failed state) is stamped on later by the worker, via the + # session_version_id pointer below. files=await promote_chat_attachments( - cls._annotate_display_files(display_files, processed_files), login_user.user_id + cls.annotate_display_files(display_files, processed_files), login_user.user_id ), + session_version_id=svid, ) return message_session, linsight_session_version @@ -349,7 +372,7 @@ async def submit_user_question( raise cls.LinsightError(f"Failed to submit user question: {e!s}") @staticmethod - def _annotate_display_files(display_files: list[dict] | None, processed_files: list | None) -> list[dict] | None: + def annotate_display_files(display_files: list[dict] | None, processed_files: list | None) -> list[dict] | None: """Stamp each persisted attachment with its parse result (by file_id). ``display_files`` are the daily-shape dicts the frontend renders; the @@ -357,6 +380,11 @@ def _annotate_display_files(display_files: list[dict] | None, processed_files: l ``parsing_status`` per file. Merging them lets the attachment chip show a "parse failed" state on reload instead of a normal-looking attachment the model can't actually use. + + Called twice per deferred turn: once here with ``processed_files=None`` + (a passthrough — submit no longer knows anything about the files) and + again from the worker once the ingest produced them + (``linsight_execute_utils.annotate_task_user_turn_files``). """ if not display_files: return display_files @@ -379,7 +407,10 @@ def _annotate_display_files(display_files: list[dict] | None, processed_files: l # Ingestion already persisted the original image bytes for the # workspace preview; naming it here lets the conversation resolve # a fresh link for it too, the same way the other chat modes do. - if p.get("original_file_path"): + # Never overwrite a name that is already set: with the ingest + # deferred, submit promotes the attachment out of the temp bucket + # itself, and THAT key is the one conversation deletion sweeps. + if p.get("original_file_path") and not item.get("object_name"): item["object_name"] = p["original_file_path"] annotated.append(item) return annotated @@ -389,9 +420,69 @@ async def _get_redis(cls): """Return the redis client (indirection so tests can patch it).""" return await get_redis_client() + @classmethod + def validate_submitted_files(cls, files: list[SubmitFileSchema] | None) -> None: + """Cheap, pure-metadata preconditions — must stay in the request. + + These are the only source of the typed codes the frontend branches on + (11021/11022/11023) and of the "file still parsing" rejection. They touch + no MinIO, no Redis, no bytes; deferring them to the worker would turn a + precise, immediate error into a generic task failure minutes later. + """ + if not files: + return + for file in files: + if file.parsing_status != "completed": + raise cls.LinsightError(f"file {file.file_name} status is error: {file.parsing_status}") + # Folder upload: reject an over-sized batch before any byte is copied. + cls._validate_folder_upload(files) + + @classmethod + async def ingest_pending_files( + cls, + session_model: LinsightSessionVersion, + *, + on_progress: Callable[[int, int, str], Awaitable[None]] | None = None, + should_abort: Callable[[], bool] | None = None, + ) -> bool: + """Materialize the attachments a deferred submit left on the row. + + Returns True when it ingested something. Rebinds ``files`` and clears + ``pending_files`` on the passed model — the CALLER persists (the worker + has to refresh the Redis snapshot in the same breath, so it owns the + write). + + The rebind is deliberate: ``JsonType`` is a plain JSON column with no + ``MutableList`` wrapper anywhere in this repo, so mutating the list in + place emits no UPDATE at all and the ingest would silently vanish. + """ + pending = session_model.pending_files + if not pending: + return False + + files = [SubmitFileSchema(**item) for item in pending] + processed = await cls._process_submitted_files( + files, + session_model.id, + session_model.user_id, + on_progress=on_progress, + should_abort=should_abort, + ) + + existing = session_model.files or [] + session_model.files = existing + (processed or []) + session_model.pending_files = None + return True + @classmethod async def _process_submitted_files( - cls, files: list[SubmitFileSchema] | None, chat_id: str, user_id: int = 0 + cls, + files: list[SubmitFileSchema] | None, + chat_id: str, + user_id: int = 0, + *, + on_progress: Callable[[int, int, str], Awaitable[None]] | None = None, + should_abort: Callable[[], bool] | None = None, ) -> list | None: """Process submitted files (F035: offload-first ingestion). @@ -415,6 +506,14 @@ async def _process_submitted_files( Args: files: List of submitted file references. chat_id: Session id; scopes the workspace prefix. + on_progress: Awaited after each file with ``(done, total, file_name)``. + Only the worker passes it, to keep the timeline alive while a + deferred ingest runs; ``None`` keeps the original behaviour. + should_abort: Polled BETWEEN files; truthy stops the loop and returns + what was ingested so far. It bounds a stop to the file currently + being parsed, not to the whole batch — an ETL already in flight + (600s ceiling) still runs to completion, because nothing here can + interrupt it. Returns: List of processed file metadata dicts (one per submitted file). @@ -422,20 +521,26 @@ async def _process_submitted_files( if not files: return None - for file in files: - if file.parsing_status != "completed": - raise cls.LinsightError(f"file {file.file_name} status is error: {file.parsing_status}") - - # Folder upload: reject an over-sized batch before any byte is copied. - cls._validate_folder_upload(files) + cls.validate_submitted_files(files) # Daily-bucket files (unified-resource) are parsed on-the-fly; only the # linsight-pipeline files need a Redis temp_info lookup. linsight_files = [f for f in files if not f.file_url] redis_keys = [f"{cls.FILE_INFO_REDIS_KEY_PREFIX}{f.file_id}" for f in linsight_files] redis_client = await cls._get_redis() - temp_list = await redis_client.amget(redis_keys) if redis_keys else [] - temp_by_id = {f.file_id: t for f, t in zip(linsight_files, temp_list)} + temp_values = await redis_client.amget(redis_keys) if redis_keys else [] + # ``amget`` DROPS misses (``[loads(v) for v in values if v is not None]``), + # so a single expired temp key makes the returned list shorter than the + # keys — zipping it positionally would then staple B's markdown_file_path + # onto A. Only pair by position when the lengths still prove alignment; + # otherwise re-read key by key so each file gets its own value (or None). + if len(temp_values) == len(linsight_files): + temp_by_id = {f.file_id: t for f, t in zip(linsight_files, temp_values)} + else: + temp_by_id = { + f.file_id: await redis_client.aget(f"{cls.FILE_INFO_REDIS_KEY_PREFIX}{f.file_id}") + for f in linsight_files + } minio_client = await get_minio_storage() @@ -452,14 +557,37 @@ async def _process_submitted_files( len(media_files), ",".join(f.file_name for f in media_files), ) + total = len(files) for submit_file in files: - if submit_file.file_url: - entry = await cls._ingest_daily_file(submit_file, chat_id, minio_client, user_id, used_names) - else: - entry = await cls._ingest_one_file( - submit_file, temp_by_id.get(submit_file.file_id), chat_id, minio_client, used_names - ) + if should_abort is not None and should_abort(): + logger.info(f"ingest aborted after {len(processed_files)}/{total} files chat_id={chat_id}") + break + try: + if submit_file.file_url: + entry = await cls._ingest_daily_file(submit_file, chat_id, minio_client, user_id, used_names) + else: + entry = await cls._ingest_one_file( + submit_file, temp_by_id.get(submit_file.file_id), chat_id, minio_client, used_names + ) + except Exception as e: + # Degrade this file, never the batch. The daily branch already + # degrades internally, but the linsight branch makes bare + # copy_object calls, so one MinIO hiccup abandoned every + # remaining attachment. Since the ingest moved into the worker + # that loss is also silent — the run would simply proceed with + # fewer files than the user attached. + logger.exception(f"attachment ingest failed name={submit_file.file_name!r} chat_id={chat_id}: {e}") + entry = { + "file_id": submit_file.file_id, + "original_filename": submit_file.file_name, + "relative_path": submit_file.relative_path, + "parsing_status": "failed", + "valid": False, + "error_message": str(e), + } processed_files.append(entry) + if on_progress is not None: + await on_progress(len(processed_files), total, submit_file.file_name) return processed_files @@ -1681,6 +1809,16 @@ def sort_tasks_by_chain(tasks: list[Any]) -> list[Any]: """ Sort task list by task chain previous_task_idYes Noneis the first task,next_task_idYes Noneis the last task. + + Every row is emitted AT MOST ONCE. The rows come from ``_save_task_info``, + which only ever inserts new ids and never rewrites an existing row's + previous/next pointers — so when the model reshapes its plan, the freshly + inserted rows form a second chain that runs straight into the first one, + and BOTH have ``previous_task_id is None``. Walking each head and blindly + extending replayed the shared tail once per head: session 8a570723 on 114 + held 11 rows and this returned 20, which the panel counted as 18 todos. + ``visited`` also makes the walk safe against a cycle, which would + otherwise spin here forever. """ if not tasks: return [] @@ -1692,23 +1830,21 @@ def sort_tasks_by_chain(tasks: list[Any]) -> list[Any]: start_tasks = [task for task in tasks if task.previous_task_id is None] sorted_tasks = [] + visited: set[str] = set() for start_task in start_tasks: # Build task chains from each start node current_task = start_task - chain = [] - while current_task is not None: - chain.append(current_task) + while current_task is not None and current_task.id not in visited: + visited.add(current_task.id) + sorted_tasks.append(current_task) # Setujunext_task_idFind next task next_task_id = current_task.next_task_id current_task = task_dict.get(next_task_id) if next_task_id else None - sorted_tasks.extend(chain) - # Dealing with possible orphaned tasks (neitherpreviousNothing, either!nextpointing to them) - processed_ids = {task.id for task in sorted_tasks} - orphan_tasks = [task for task in tasks if task.id not in processed_ids] + orphan_tasks = [task for task in tasks if task.id not in visited] sorted_tasks.extend(orphan_tasks) return sorted_tasks diff --git a/src/backend/bisheng/linsight/domain/task_exec.py b/src/backend/bisheng/linsight/domain/task_exec.py index 524c3b5411..9b7793dab4 100644 --- a/src/backend/bisheng/linsight/domain/task_exec.py +++ b/src/backend/bisheng/linsight/domain/task_exec.py @@ -126,6 +126,14 @@ async def ensure_linsight_permission_runtime(manager=None) -> dict: # Headroom for the once-per-run before_agent nodes and the closing turn. _RECURSION_LIMIT_MARGIN = 20 +# Machine name on the attachment-ingest progress row. The client matches on it (and +# on extra_info.ingest_progress) to render localized copy, so it is a CONTRACT: this +# string is mirrored in client/src/components/Linsight/Execution/execTypes.ts and +# changing one side alone leaves the row rendering its raw token to the user. +_INGEST_STEP_NAME = "ingest_uploads" +# Phases that close the row (status="end"); anything else keeps it spinning. +_INGEST_TERMINAL_PHASES = frozenset({"done", "failed", "aborted"}) + def _resolve_recursion_limit(linsight_conf) -> int: """LangGraph ``recursion_limit`` for one task run — always above the turn budget. @@ -225,14 +233,35 @@ async def _managed_execution(self): # Start Termination Monitoring await self._start_termination_monitor(session_model) - # Initialization file directory - self.file_dir = await self._init_file_directory(session_model) + # Claim the session BEFORE the (now possibly multi-minute) attachment + # ingest: a duplicate queue item for the same svid — submit enqueues + # AND the browser's start-execute may still land — is rejected by + # _is_session_in_progress, which keys only on IN_PROGRESS. It also + # brings the row into the worker-startup crash sweep's scan, which + # force-FAILs an IN_PROGRESS row whose owner node is dead; a worker + # killed mid-ingest is thus reported as failed instead of sitting at + # NOT_STARTED forever with nobody to pick it up (the attachments + # survive on pending_files, and /workbench/continue re-ingests them). + await self._update_session_status(session_model, SessionVersionStatusEnum.IN_PROGRESS) # F035 problem 2: ensure the session-level pseudo task row exists so # planning/wrap-up/direct-answer steps (mapper routes them to - # task_id = svid) are persisted and survive a refresh. + # task_id = svid) are persisted and survive a refresh. It also carries + # the ingest progress steps pushed just below. await self._ensure_session_pseudo_task(session_model) + await self._ingest_pending_attachments(session_model) + # The ingest can hold the run for minutes, and _execute_workflow opens + # with an UNCONDITIONAL IN_PROGRESS write. A stop that landed inside + # that window would be overwritten by it, and nothing writes a + # terminal status afterwards — the session then shows as running + # forever, unstoppable. Re-read the authoritative status here rather + # than trusting the monitor's polling interval to have caught up. + await self._check_termination_now() + + # Initialization file directory + self.file_dir = await self._init_file_directory(session_model) + yield session_model finally: @@ -389,10 +418,18 @@ async def _managed_resume(self): session_model = await self._get_session_model(self.session_version_id) try: await self._start_termination_monitor(session_model) - self.file_dir = await self._init_file_directory(session_model) # F035 problem 2: resume/continue can also produce session-level - # (task_id = svid) steps; ensure the pseudo task row is present. + # (task_id = svid) steps; ensure the pseudo task row is present. It + # now has to precede the ingest, which hangs its progress steps off it. await self._ensure_session_pseudo_task(session_model) + # Normally a no-op: the fresh run already drained pending_files. It is + # NOT dead code, because this is the only second chance the column + # ever gets — a worker killed mid-ingest leaves the row FAILED with + # pending_files intact, and FAILED is one of the two statuses + # /workbench/continue accepts. Without this the follow-up turn would + # run with no attachments at all and the refs would sit there forever. + await self._ingest_pending_attachments(session_model) + self.file_dir = await self._init_file_directory(session_model) yield session_model finally: await self._cleanup_resources() @@ -687,7 +724,11 @@ async def _get_llm(self, session_model: LinsightSessionVersion) -> BaseChatModel Worker subprocess). """ try: - return await _resolve_model(session_model, getattr(session_model, "model", None)) + # ``_resolve_model`` also reports the model's declared vision capability, + # which only the agent's guards consume — this helper LLM never carries + # attachments, so the flag is dropped here. + model, _supports_vision = await _resolve_model(session_model, getattr(session_model, "model", None)) + return model except Exception as e: # Keep the generic user-facing message, but chain + log the real # cause so the original stack trace surfaces (project error-handling @@ -697,6 +738,124 @@ async def _get_llm(self, session_model: LinsightSessionVersion) -> BaseChatModel "The task has been terminated, please contact the administrator to check the status of the Ideas task execution model" ) from e + async def _push_ingest_step( + self, + svid: str, + call_id: str, + *, + phase: str, + done: int, + total: int, + file_name: str = "", + output: str = "", + ) -> None: + """Publish one attachment-ingest progress frame on the session pseudo task. + + Carries only DATA — the row's wording lives in the client's locale files. + A backend-formatted label would be a Chinese string on a Japanese user's + timeline, and it lands in persisted history, so it would stay wrong after + a language switch. ``ExecStep.extra_info`` is the documented channel for + exactly this ("额外信息，包含文件上传等其他信息") and the client already + threads it through to ``MergedStep.extraInfo``. + + Stable ``call_id`` on purpose: ``add_execution_task_step`` upserts by it, + so the twelve progress frames of a twelve-file batch collapse into ONE + history row that keeps re-rendering, instead of twelve dead rows. + """ + try: + step = ExecStep( + task_id=svid, + call_id=call_id, + call_reason="", + name=_INGEST_STEP_NAME, + step_type="tool", + status="end" if phase in _INGEST_TERMINAL_PHASES else "start", + output=output, + extra_info={"ingest_progress": {"phase": phase, "done": done, "total": total, "file_name": file_name}}, + ) + await self._state_manager.add_execution_task_step(svid, step=step) + await self._state_manager.push_message( + MessageData(event_type=MessageEventType.TASK_EXECUTE_STEP, data=step.model_dump()) + ) + except Exception as e: + # Progress rendering is not worth failing an ingest over. + logger.warning(f"Failed to push ingest progress step for {svid}: {e}") + + async def _ingest_pending_attachments(self, session_model: LinsightSessionVersion) -> None: + """Materialize deferred attachments before anything reads them. + + Deadline: this MUST complete before ``_init_file_directory`` (the local + prefetch), before ``_generate_tools`` (the code interpreter snapshots + ``os.walk(file_dir)``) and before ``prepare_file_list`` (the 可用文件 + pointer block) — and before WorkspaceBackend serves ``read_file``, since + ingestion is what physically PUTs ``workspace/{svid}/uploads/*``. + + A per-file failure never gets here: ``_process_submitted_files`` degrades + it to ``valid=False`` and carries on, so a ten-file task still runs when + one attachment is unreadable. Anything that DOES escape is systemic — + storage unreachable, a malformed ref — and is raised, not swallowed. + Swallowing it would leave ``files`` empty and let the agent answer a + question about "the attached report" with no attachment at all: a + confident answer about a document it never saw, with one muted timeline + row as the only trace. ``pending_files`` is left intact so + /workbench/continue can retry the same batch. + """ + pending = session_model.pending_files + if not pending: + return + + svid = session_model.id + total = len(pending) + call_id = f"ingest_uploads_{svid}" + + async def on_progress(done: int, count: int, file_name: str) -> None: + await self._push_ingest_step(svid, call_id, phase="running", done=done, total=count, file_name=file_name) + + # The progress frames render as ordinary tool rows (step_type="tool"), which + # the step logger deliberately does not spell out; log the boundaries here + # so "where did the 19 minutes go" stays greppable from the worker log. + logger.info(f"Ingesting {total} deferred attachment(s): session_version_id={svid}") + try: + # First frame before any byte is touched: the worker has already taken + # the task off the queue, so without this the timeline sits empty for + # however long the first file takes to parse. + await self._push_ingest_step(svid, call_id, phase="running", done=0, total=total) + + ingested = await LinsightWorkbenchImpl.ingest_pending_files( + session_model, + on_progress=on_progress, + should_abort=lambda: self._is_terminated, + ) + if not ingested: + return + + # Writes the row and THEN the Redis snapshot (set_session_version_info + # is DB-first). The order is load-bearing: the IN_PROGRESS flip above + # already cached a snapshot carrying files=None, and + # get_session_version_info reads Redis first — skipping this refresh + # would shadow the freshly ingested files for the whole run. + await self._state_manager.set_session_version_info(session_model) + except Exception as e: + logger.exception(f"Pending attachment ingest failed: session_version_id={svid}: {e}") + # The raw exception text rides in `output` (a diagnostic detail the row + # shows verbatim), never in the label. + await self._push_ingest_step(svid, call_id, phase="failed", done=0, total=total, output=str(e)[:500]) + raise TaskExecutionError(f"Failed to ingest uploaded attachments: {e}") from e + + # The chips on the question row were persisted before anything was known + # about these files; stamp the parse result onto them now that it exists. + # Best-effort by design — a history nicety must not fail a run whose + # attachments are already in place. + try: + await linsight_execute_utils.annotate_task_user_turn_files(session_model) + except Exception as e: + logger.warning(f"Failed to annotate question attachments for {svid}: {e}") + + done = len(session_model.files or []) + phase = "aborted" if self._is_terminated else "done" + logger.info(f"Deferred attachment ingest {phase} {done}/{total}: session_version_id={svid}") + await self._push_ingest_step(svid, call_id, phase=phase, done=done, total=total) + @create_cache_folder_async async def _init_file_directory(self, session_model: LinsightSessionVersion) -> str: """Initialization file directory""" @@ -1565,6 +1724,18 @@ def _check_termination(self): logger.info("Termination signal detected, ready to terminate agent task") raise UserTerminationError("Task terminated by user") + async def _check_termination_now(self): + """``_check_termination`` against the authoritative status, not the flag. + + The monitor only samples every ``USER_TERMINATION_CHECK_INTERVAL``, so + right after a step that can run for minutes the in-process flag may still + say "running" for a session the user already stopped. Callers that are + about to write status unconditionally need the fresh answer. + """ + if await self._check_user_termination(): + self._is_terminated = True + self._check_termination() + async def _start_termination_monitor(self, session_model: LinsightSessionVersion): """Start Termination Monitoring""" diff --git a/src/backend/bisheng/linsight/domain/utils.py b/src/backend/bisheng/linsight/domain/utils.py index baf1b5884e..3720459c8e 100644 --- a/src/backend/bisheng/linsight/domain/utils.py +++ b/src/backend/bisheng/linsight/domain/utils.py @@ -606,11 +606,12 @@ async def enqueue_session_for_execution(session_model: LinsightSessionVersion) - handoff event, and the browser then called start-execute. Anything that cut the stream before that second call — a refresh, a closed tab, a proxy timeout — left the session parked at NOT_STARTED with nothing to pick it up. That is - not hypothetical: a task with 12 attachments spent minutes parsing them + not hypothetical: a task with 12 attachments spent 19 minutes parsing them INSIDE the submit request, the user gave up waiting, and the session sat in the table untouched (the conversation lost its task row too, so even the - task-mode badge disappeared). Submitting now enqueues server-side, so the - task runs whether or not the client is still listening. + task-mode badge disappeared). Ingestion has since moved into the worker, so + the submit request is short again — but server-side enqueue stays, because + "the task runs" must not depend on the client still listening. Enqueueing twice is harmless: the executor re-reads the session and bails via ``_is_session_in_progress`` when it is already running, so the client's @@ -699,11 +700,23 @@ async def get_task_feedback_by_version(session_id: str) -> dict[str, dict]: return result -async def persist_task_user_turn(chat_id: str, user_id: int, question: str, files: list | None = None) -> ChatMessage: +async def persist_task_user_turn( + chat_id: str, + user_id: int, + question: str, + files: list | None = None, + session_version_id: str | None = None, +) -> ChatMessage: """F035 Track J (TJ-3): persist the task user turn into the unified conversation. Mirrors the daily-chat question envelope (workstation/chat_service) so the user turn renders identically whether or not task mode was on for the round. + + ``session_version_id`` is stamped into ``extra`` the same way the bot task + turn carries it. That pointer is what lets the worker find this exact row + again: with the attachment ingest deferred, everything the parse learns about + the files (the video poster frame above all) only exists minutes after this + row is written. """ return await ChatMessageDao.ainsert_one( ChatMessage( @@ -716,12 +729,64 @@ async def persist_task_user_turn(chat_id: str, user_id: int, question: str, file category="question", message=json.dumps({"query": question or "", "files": files or []}, ensure_ascii=False), files=json.dumps(files) if files else None, - extra="{}", + extra=json.dumps({"linsight_session_version_id": session_version_id}) if session_version_id else "{}", source=0, ) ) +async def annotate_task_user_turn_files(session_model: LinsightSessionVersion) -> bool: + """Back-fill the question row's attachment chips with the ingest result. + + Submit writes the question turn before a single byte has been touched, so the + per-file parse result it used to carry (``valid`` / ``parsing_status`` / + ``error_message``, and the video ``cover_filepath`` the chip actually renders) + has to be written back once the worker has it. Without this a task-mode video + loses its thumbnail on every reload — the live session only has it because + the client stamps its own copy in memory. + + ``object_name`` is deliberately left alone: submit already promoted each + attachment out of the temp bucket, and that key is the one conversation + deletion sweeps. Returns True when a row was actually rewritten. + """ + from bisheng.linsight.domain.services.workbench_impl import LinsightWorkbenchImpl + + processed = session_model.files or [] + if not processed or not session_model.session_id: + return False + + rows = await ChatMessageDao.aget_messages_by_chat_id( + chat_id=session_model.session_id, category_list=["question"], limit=1000 + ) + for row in reversed(rows): + if row.is_bot: + continue + try: + row_svid = json.loads(row.extra or "{}").get("linsight_session_version_id") + except (json.JSONDecodeError, TypeError): + continue + if row_svid != session_model.id: + continue + + try: + payload = json.loads(row.message or "{}") + except (json.JSONDecodeError, TypeError): + return False + files = payload.get("files") if isinstance(payload, dict) else None + if not files: + return False + + annotated = LinsightWorkbenchImpl.annotate_display_files(files, processed) + if annotated == files: + return False + payload["files"] = annotated + row.message = json.dumps(payload, ensure_ascii=False) + row.files = json.dumps(annotated) + await ChatMessageDao.aupdate_message_model(row) + return True + return False + + def _extract_user_query(message: str | None) -> str: """Unwrap the daily question envelope ``{"query","files"}``; fall back to raw text.""" if not message: diff --git a/src/backend/bisheng/permission/application/business_authorization.py b/src/backend/bisheng/permission/application/business_authorization.py index fab8e5281b..cb246fa43f 100644 --- a/src/backend/bisheng/permission/application/business_authorization.py +++ b/src/backend/bisheng/permission/application/business_authorization.py @@ -135,3 +135,61 @@ async def batch_check_business_actions( if is_allowed: result[resource_id].add(action) return {resource_id: frozenset(action_codes) for resource_id, action_codes in result.items()} + + +async def batch_check_business_visible( + login_user: LoginPermissionIdentity, + *, + resource_type: str, + resource_ids: Iterable[str | int], +) -> dict[str, bool]: + """Resolve bounded business candidates and check final visible without admin expansion.""" + + normalized_ids = tuple(dict.fromkeys(str(value) for value in resource_ids)) + if not normalized_ids: + return {} + + actor = await resolve_permission_actor(login_user) + registry = await get_f048_resource_registry() + resolved = await asyncio.gather( + *( + registry.resolve( + resource_type=resource_type, + resource_id=resource_id, + actor=actor, + action="visible", + ) + for resource_id in normalized_ids + ), + return_exceptions=True, + ) + result = dict.fromkeys(normalized_ids, False) + targets: list[VerifiedPermissionTarget] = [] + target_ids: list[str] = [] + for resource_id, resolution in zip(normalized_ids, resolved, strict=True): + if isinstance( + resolution, + ( + InvalidCatalogActionError, + PermissionInvalidResourceError, + PermissionPublishNotReadyError, + ), + ): + continue + if isinstance(resolution, BaseException): + raise resolution + targets.append(resolution) + target_ids.append(resource_id) + + runtime = await get_f048_runtime() + for offset in range(0, len(targets), _MAX_BATCH_CHECKS): + batch_targets = tuple(targets[offset : offset + _MAX_BATCH_CHECKS]) + batch_ids = target_ids[offset : offset + _MAX_BATCH_CHECKS] + allowed = await runtime.batch_check_actions( + actor, + batch_targets, + "visible", + ) + for resource_id, is_allowed in zip(batch_ids, allowed, strict=True): + result[resource_id] = bool(is_allowed) + return result diff --git a/src/backend/bisheng/permission/application/catalog_api.py b/src/backend/bisheng/permission/application/catalog_api.py index d0e4d5bfe1..c60f68b801 100644 --- a/src/backend/bisheng/permission/application/catalog_api.py +++ b/src/backend/bisheng/permission/application/catalog_api.py @@ -44,6 +44,7 @@ PermissionModel, PermissionModelAction, PermissionProjectionOperation, + PermissionVisibleSourceProjection, ) from bisheng.permission.domain.schemas import ( CatalogChangeRequest, @@ -68,11 +69,11 @@ ) from bisheng.permission.domain.services.model_policy import ( CustomModelSelection, + ModelReferenceSummary, PermissionModelImpact, PermissionModelRelease, derive_permission_models, effective_model_action_codes, - ensure_model_deletable, ) SessionFactory = Callable[[], AbstractAsyncContextManager[AsyncSession]] @@ -325,6 +326,73 @@ async def authorization_release( raise AuthorizationModelMismatchError(msg="Catalog authorization model release is missing") return row + async def bind_draft_authorization_release( + self, + *, + draft_id: int, + authorization_release_id: int, + ) -> None: + """Bind an unpublished complete draft to an immutable model release. + + Normal Catalog edits inherit the current authorization-model release. + A forward model upgrade uses this narrow maintenance hook after the + no-op draft has been fully materialized and before publication. + """ + + with bypass_tenant_filter(): + async with self._session_factory() as session: + async with session.begin(): + draft = ( + ( + await session.execute( + select(PermissionCatalogRelease) + .where(PermissionCatalogRelease.id == draft_id) + .with_for_update() + ) + ) + .scalars() + .first() + ) + model_release = await session.get( + AuthorizationModelRelease, + authorization_release_id, + ) + if draft is None: + raise PermissionPublishNotReadyError(msg="Authorization model Catalog draft is missing") + if draft.required_authorization_model_release_id == authorization_release_id and draft.status in { + "DRAFT", + "PROJECTING", + "COMMITTED", + "CURRENT", + }: + return + if draft.status != "DRAFT": + raise PermissionPublishNotReadyError( + msg="Authorization model can only be bound to a DRAFT Catalog" + ) + predecessor = ( + await session.get(PermissionCatalogRelease, draft.predecessor_id) + if draft is not None and draft.predecessor_id is not None + else None + ) + predecessor_model = ( + await session.get( + AuthorizationModelRelease, + predecessor.required_authorization_model_release_id, + ) + if predecessor is not None + else None + ) + if ( + model_release is None + or predecessor_model is None + or model_release.store_id != predecessor_model.store_id + ): + raise AuthorizationModelMismatchError( + msg="Authorization model release Store differs from the Catalog predecessor" + ) + draft.required_authorization_model_release_id = authorization_release_id + async def grant_references( self, ) -> dict[str, tuple[str, ...]]: @@ -349,6 +417,73 @@ async def grant_references( ) return {key: tuple(sorted(values)) for key, values in references.items()} + async def model_reference_summaries( + self, + model_keys: tuple[str, ...], + ) -> dict[str, ModelReferenceSummary]: + """Build the fail-closed cross-tenant deletion audit for model keys.""" + + summaries = {model_key: ModelReferenceSummary() for model_key in model_keys} + if not summaries: + return summaries + with bypass_tenant_filter(): + async with self._session_factory() as session: + grants = list( + ( + await session.execute( + select(PermissionGrant).where( + col(PermissionGrant.model_key).in_(model_keys), + PermissionGrant.state != "INACTIVE", + ) + ) + ).scalars() + ) + sources = list( + ( + await session.execute( + select(PermissionVisibleSourceProjection).where( + col(PermissionVisibleSourceProjection.model_key).in_(model_keys), + PermissionVisibleSourceProjection.state != "RETIRED", + ) + ) + ).scalars() + ) + + grants_by_model: dict[str, list[PermissionGrant]] = {} + for grant in grants: + grants_by_model.setdefault(grant.model_key, []).append(grant) + sources_by_model: dict[str, list[PermissionVisibleSourceProjection]] = {} + for source in sources: + if source.model_key is not None: + sources_by_model.setdefault(source.model_key, []).append(source) + + for model_key in model_keys: + model_grants = grants_by_model.get(model_key, ()) + model_sources = sources_by_model.get(model_key, ()) + residual_rows = sorted( + "\0".join( + ( + source.contribution_fingerprint, + source.tuple_fingerprint, + source.state, + ) + ) + for source in model_sources + ) + summaries[model_key] = ModelReferenceSummary( + active_grant_count=sum(grant.state == "ACTIVE" for grant in model_grants), + pending_grant_count=sum(grant.state == "PENDING" for grant in model_grants), + failed_grant_count=sum(grant.state not in {"ACTIVE", "PENDING"} for grant in model_grants), + active_source_count=sum(source.state == "ACTIVE" for source in model_sources), + pending_source_count=sum(source.state == "PENDING" for source in model_sources), + failed_source_count=sum(source.state not in {"ACTIVE", "PENDING"} for source in model_sources), + live_tuple_count=len( + {source.tuple_fingerprint for source in model_sources if source.state == "ACTIVE"} + ), + residual_checksum=(sha256("\n".join(residual_rows).encode()).hexdigest() if residual_rows else None), + ) + return summaries + async def prepare_publish( self, *, @@ -1352,6 +1487,14 @@ async def create_draft( request.changes, before, ) + custom_keys = {model.model_key for model in customs} + deleted_model_keys = tuple( + sorted( + model.model_key + for model in before.model_release.models + if model.kind == "CUSTOM" and model.model_key not in custom_keys + ) + ) try: draft = await self._service.build_draft( CatalogDraftBuildInput( @@ -1365,6 +1508,7 @@ async def create_draft( custom_models=customs, standard_allow_same_level=standard_policy, grant_references=(await self._state.grant_references()), + model_reference_summaries=(await self._state.model_reference_summaries(deleted_model_keys)), draft_owner_id=operator_id, idempotency_key=request.idempotency_key, expires_at=_utc_now_naive() + timedelta(minutes=10), @@ -1572,29 +1716,6 @@ async def _apply_changes( ) elif kind == CatalogChangeType.DELETE_MODEL: model = self._custom_model(change.model_key, custom_by_key) - derived = next(item for item in before.model_release.models if item.model_key == model.model_key) - references = await self._state.grant_references() - reference_count = len(references.get(model.model_key, ())) - try: - # Judge "is it disabled" on the state this batch publishes, not - # on the base release. Reading the base meant a batch that - # deactivates and then deletes was refused for being active, - # forcing two separate publications to remove one model. - ensure_model_deletable( - replace(derived, active=model.active), - reference_count=reference_count, - ) - except ValueError as exc: - # 25004 covers several model-state conflicts, so its generic - # copy cannot name the blocker. Say which one it is, and how - # much is in the way, or the caller only learns "state does - # not allow this". - raise PermissionModelStateConflictError( - exception=exc, - msg=str(exc), - reason="referenced_by_grants" if reference_count else "model_is_active", - reference_count=reference_count, - ) from exc del custom_by_key[model.model_key] elif kind == CatalogChangeType.SET_ALLOW_SAME_LEVEL: if change.allow_same_level is None or not change.model_key: diff --git a/src/backend/bisheng/permission/application/control_state.py b/src/backend/bisheng/permission/application/control_state.py index 6f110511c5..504f8448d2 100644 --- a/src/backend/bisheng/permission/application/control_state.py +++ b/src/backend/bisheng/permission/application/control_state.py @@ -27,9 +27,13 @@ PermissionGrantAssignee, PermissionModel, PermissionModelAction, + PermissionVisibleSourceProjection, ResourcePermissionMode, ) -from bisheng.permission.domain.schemas import VerifiedPermissionTarget +from bisheng.permission.domain.schemas import ( + VerifiedPermissionTarget, + VisibleSourceProjectionDTO, +) from bisheng.permission.domain.services.grant_service import ( GrantMutationContext, ) @@ -51,6 +55,9 @@ PermissionSourceExplanation, ) from bisheng.permission.domain.services.projection_plan import ProjectionOutcome +from bisheng.permission.domain.services.visibility_projection_service import ( + VisibilityProjectionCompilation, +) @dataclass(frozen=True, slots=True) @@ -279,6 +286,32 @@ async def load_grants( ) return tuple(result) + async def load_visible_sources( + self, + *, + target: VerifiedPermissionTarget, + ) -> tuple[VisibleSourceProjectionDTO, ...]: + """Load the complete active contribution set for one resource scope.""" + + async with get_async_db_session() as session: + rows = list( + ( + await session.execute( + select(PermissionVisibleSourceProjection) + .where( + PermissionVisibleSourceProjection.tenant_id == target.tenant_id, + PermissionVisibleSourceProjection.resource_type == target.resource_type, + PermissionVisibleSourceProjection.resource_id == target.resource_id, + PermissionVisibleSourceProjection.state == "ACTIVE", + ) + .order_by(PermissionVisibleSourceProjection.id) + ) + ) + .scalars() + .all() + ) + return tuple(self._visible_source_snapshot(row) for row in rows) + async def inherited_grants( self, *, @@ -622,6 +655,7 @@ async def prepare_grants( self, context: GrantMutationContext, grants: tuple[GrantSnapshot, ...], + visibility: VisibilityProjectionCompilation | None, *, idempotency_key: str, operation_id: int, @@ -670,14 +704,21 @@ async def prepare_grants( ) .values(state="PENDING_DELETE") ) + if visibility is not None: + await self._prepare_visible_sources( + session, + tenant_id=context.target.tenant_id, + visibility=visibility, + operation_id=operation_id, + ) async def finalize_grants( self, context: GrantMutationContext, grants: tuple[GrantSnapshot, ...], + visibility: VisibilityProjectionCompilation | None, outcome: ProjectionOutcome, ) -> None: - del outcome async with get_async_db_session() as session: async with session.begin(): for grant in grants: @@ -717,6 +758,13 @@ async def finalize_grants( version=PermissionGrantAssignee.version + 1, ) ) + if visibility is not None: + await self._finalize_visible_sources( + session, + tenant_id=context.target.tenant_id, + visibility=visibility, + operation_id=outcome.operation_id, + ) async def allocate_source_ids(self, count: int) -> tuple[int, ...]: if count < 0: @@ -783,6 +831,7 @@ async def prepare_mode( await self.prepare_grants( grant_context, grants, + None, idempotency_key=idempotency_key, operation_id=operation_id, ) @@ -806,7 +855,7 @@ async def finalize_mode( models=tuple(grant.model for grant in grants), grants=grants, ) - await self.finalize_grants(grant_context, grants, outcome) + await self.finalize_grants(grant_context, grants, None, outcome) async with get_async_db_session() as session: async with session.begin(): await session.execute( @@ -952,6 +1001,29 @@ def _source_snapshot( version=row.version, ) + @staticmethod + def _visible_source_snapshot( + row: PermissionVisibleSourceProjection, + ) -> VisibleSourceProjectionDTO: + return VisibleSourceProjectionDTO( + tenant_id=int(row.tenant_id or 0), + resource_type=row.resource_type, + resource_id=row.resource_id, + visibility_class=row.visibility_class, + projected_subject=row.projected_subject, + source_kind=row.source_kind, + source_owner_key=row.source_owner_key, + source_locator=row.source_locator, + source_fingerprint=row.source_fingerprint, + contribution_fingerprint=row.contribution_fingerprint, + model_key=row.model_key, + source_version=row.source_version, + tuple_fingerprint=row.tuple_fingerprint, + state=row.state, + operation_id=row.operation_id, + migration_item_id=row.migration_item_id, + ) + @staticmethod def _owner_projection_grants( context: OwnerProjectionContext, @@ -1067,6 +1139,113 @@ async def _upsert_grant( row.projection_state = projection_state return row + @staticmethod + async def _prepare_visible_sources( + session, + *, + tenant_id: int, + visibility: VisibilityProjectionCompilation, + operation_id: int, + ) -> None: + desired = (*visibility.active_sources, *visibility.retired_sources) + for source in desired: + row = ( + ( + await session.execute( + select(PermissionVisibleSourceProjection) + .where( + PermissionVisibleSourceProjection.tenant_id == tenant_id, + PermissionVisibleSourceProjection.resource_type == source.resource_type, + PermissionVisibleSourceProjection.resource_id == source.resource_id, + PermissionVisibleSourceProjection.visibility_class == source.visibility_class, + PermissionVisibleSourceProjection.projected_subject == source.projected_subject, + PermissionVisibleSourceProjection.contribution_fingerprint + == source.contribution_fingerprint, + ) + .with_for_update() + ) + ) + .scalars() + .first() + ) + if row is None: + row = PermissionVisibleSourceProjection( + tenant_id=tenant_id, + resource_type=source.resource_type, + resource_id=source.resource_id, + visibility_class=source.visibility_class, + projected_subject=source.projected_subject, + source_kind=source.source_kind, + source_owner_key=source.source_owner_key, + source_locator=source.source_locator, + source_fingerprint=source.source_fingerprint, + contribution_fingerprint=source.contribution_fingerprint, + model_key=source.model_key, + source_version=source.source_version, + tuple_fingerprint=source.tuple_fingerprint, + state="PENDING", + operation_id=operation_id, + migration_item_id=source.migration_item_id, + ) + session.add(row) + continue + + immutable_fields = ( + "resource_type", + "resource_id", + "visibility_class", + "projected_subject", + "source_kind", + "source_owner_key", + "source_locator", + "source_fingerprint", + "contribution_fingerprint", + "model_key", + "tuple_fingerprint", + ) + if any(getattr(row, field) != getattr(source, field) for field in immutable_fields): + raise PermissionVersionConflictError( + msg="Visible source contribution fingerprint collision", + ) + if source.source_version < row.source_version: + raise PermissionVersionConflictError(msg="Visible source version is stale") + row.source_version = source.source_version + row.state = "PENDING" + row.operation_id = operation_id + session.add(row) + + await session.flush() + + @staticmethod + async def _finalize_visible_sources( + session, + *, + tenant_id: int, + visibility: VisibilityProjectionCompilation, + operation_id: int, + ) -> None: + for sources, state in ( + (visibility.active_sources, "ACTIVE"), + (visibility.retired_sources, "RETIRED"), + ): + fingerprints = tuple(source.contribution_fingerprint for source in sources) + if not fingerprints: + continue + result = await session.execute( + update(PermissionVisibleSourceProjection) + .where( + PermissionVisibleSourceProjection.tenant_id == tenant_id, + PermissionVisibleSourceProjection.operation_id == operation_id, + col(PermissionVisibleSourceProjection.contribution_fingerprint).in_(fingerprints), + col(PermissionVisibleSourceProjection.state).in_(("PENDING", state)), + ) + .values(state=state) + ) + if result.rowcount != len(fingerprints): + raise PermissionVersionConflictError( + msg="Visible source after-state changed before projection finalize", + ) + @staticmethod async def _upsert_assignee( session, @@ -1207,6 +1386,7 @@ async def prepare( self, context, grants, + visibility, *, idempotency_key, operation_id, @@ -1214,12 +1394,13 @@ async def prepare( await self._state.prepare_grants( context, grants, + visibility, idempotency_key=idempotency_key, operation_id=operation_id, ) - async def finalize(self, context, grants, outcome) -> None: - await self._state.finalize_grants(context, grants, outcome) + async def finalize(self, context, grants, visibility, outcome) -> None: + await self._state.finalize_grants(context, grants, visibility, outcome) class SqlModeState: diff --git a/src/backend/bisheng/permission/application/runtime.py b/src/backend/bisheng/permission/application/runtime.py index 899a71c67a..b9f7454f5a 100644 --- a/src/backend/bisheng/permission/application/runtime.py +++ b/src/backend/bisheng/permission/application/runtime.py @@ -36,7 +36,10 @@ stable_grant_key, ) from bisheng.permission.domain.models import ProjectionOperationStatus -from bisheng.permission.domain.schemas import VerifiedPermissionTarget +from bisheng.permission.domain.schemas import ( + VerifiedPermissionTarget, + VisibleObjectEnumerationResult, +) from bisheng.permission.domain.services.grant_service import ( CanonicalGrantChange, GrantCapability, @@ -190,6 +193,21 @@ async def list_action_objects( max_results=max_results, ) + async def list_visible_objects( + self, + actor: PermissionActor, + *, + resource_type: str, + max_results: int, + ) -> VisibleObjectEnumerationResult: + """Expose complete visible enumeration through the sole online facade.""" + + return await self._decision.list_visible_objects( + actor, + resource_type=resource_type, + max_results=max_results, + ) + async def get_permission_version( self, *, @@ -577,6 +595,7 @@ async def build_grant_context( target=target, models=models, ) + visible_sources = await self._state.load_visible_sources(target=target) system_authorized = self._system_authorized(actor, target) capabilities = () if system_authorized else await self._grant_capabilities(actor, target) return GrantMutationContext( @@ -590,6 +609,7 @@ async def build_grant_context( capabilities=capabilities, models=models, grants=grants, + existing_visible_sources=visible_sources, ) async def grantable_models( diff --git a/src/backend/bisheng/permission/application/sql_runtime.py b/src/backend/bisheng/permission/application/sql_runtime.py index dd51323568..1c9b6aa2bb 100644 --- a/src/backend/bisheng/permission/application/sql_runtime.py +++ b/src/backend/bisheng/permission/application/sql_runtime.py @@ -27,6 +27,7 @@ PermissionCatalogRelease, PermissionGrant, PermissionGrantAssignee, + PermissionVisibleSourceProjection, ResourcePermissionMode, ) from bisheng.permission.domain.repositories.projection_repository import ( @@ -602,6 +603,48 @@ async def finalize( ) ) + visible_sources = list( + ( + await session.execute( + select(PermissionVisibleSourceProjection) + .where( + PermissionVisibleSourceProjection.tenant_id == plan.tenant_id, + PermissionVisibleSourceProjection.operation_id == operation_id, + PermissionVisibleSourceProjection.state == "PENDING", + ) + .with_for_update() + ) + ) + .scalars() + .all() + ) + for source in visible_sources: + prefix = "grant_assignee:" + if source.source_kind != "GRANT_ASSIGNEE" or not source.source_owner_key.startswith(prefix): + raise PermissionPublishNotReadyError( + msg="Visible source recovery cannot infer its canonical owner state", + ) + try: + assignee_id = int(source.source_owner_key.removeprefix(prefix)) + except ValueError as exc: + raise PermissionPublishNotReadyError( + msg="Visible source recovery owner identity is invalid", + ) from exc + assignee_state = ( + await session.execute( + select(PermissionGrantAssignee.state).where( + PermissionGrantAssignee.tenant_id == plan.tenant_id, + PermissionGrantAssignee.id == assignee_id, + ) + ) + ).scalar_one_or_none() + if assignee_state not in {"ACTIVE", "INACTIVE"}: + raise PermissionPublishNotReadyError( + msg="Visible source recovery owner state is incomplete", + ) + source.state = "ACTIVE" if assignee_state == "ACTIVE" else "RETIRED" + session.add(source) + target_mode = self._target_mode(plan) mode_values: dict[str, object] = { "version": plan.target_version, diff --git a/src/backend/bisheng/permission/domain/models/__init__.py b/src/backend/bisheng/permission/domain/models/__init__.py index 2e95f4aeb4..b9d18706c3 100644 --- a/src/backend/bisheng/permission/domain/models/__init__.py +++ b/src/backend/bisheng/permission/domain/models/__init__.py @@ -27,8 +27,10 @@ from .projection import ( PermissionProjectionOperation, PermissionProjectionTuple, + PermissionVisibleSourceProjection, ProjectionOperationStatus, ProjectionTupleStatus, + VisibleSourceProjectionState, ) __all__ = [ @@ -50,8 +52,10 @@ "PermissionModelAction", "PermissionProjectionOperation", "PermissionProjectionTuple", + "PermissionVisibleSourceProjection", "ProjectionOperationStatus", "ProjectionState", "ProjectionTupleStatus", "ResourcePermissionMode", + "VisibleSourceProjectionState", ] diff --git a/src/backend/bisheng/permission/domain/models/projection.py b/src/backend/bisheng/permission/domain/models/projection.py index 1e564f1fd2..c92a7336a7 100644 --- a/src/backend/bisheng/permission/domain/models/projection.py +++ b/src/backend/bisheng/permission/domain/models/projection.py @@ -40,6 +40,13 @@ class ProjectionTupleStatus(StrEnum): FAILED_CLOSED = "FAILED_CLOSED" +class VisibleSourceProjectionState(StrEnum): + PENDING = "PENDING" + ACTIVE = "ACTIVE" + RETIRED = "RETIRED" + FAILED_CLOSED = "FAILED_CLOSED" + + class PermissionProjectionOperation(SQLModelSerializable, table=True): __tablename__ = "permission_projection_operation" __table_args__ = ( @@ -178,3 +185,117 @@ class PermissionProjectionTuple(SQLModelSerializable, table=True): server_default=UPDATE_TIME_SERVER_DEFAULT, ), ) + + +class PermissionVisibleSourceProjection(SQLModelSerializable, table=True): + """Rebuildable source index for one flattened visible relation.""" + + __tablename__ = "permission_visible_source_projection" + __table_args__ = ( + UniqueConstraint( + "tenant_id", + "resource_type", + "resource_id", + "visibility_class", + "projected_subject", + "contribution_fingerprint", + name="uq_perm_visible_source_contribution", + ), + Index( + "ix_perm_visible_resource_subject", + "tenant_id", + "resource_type", + "resource_id", + "visibility_class", + "projected_subject", + "state", + ), + Index( + "ix_perm_visible_model_state", + "model_key", + "state", + "tenant_id", + "id", + ), + Index( + "ix_perm_visible_source_owner", + "tenant_id", + "source_kind", + "source_owner_key", + "state", + "id", + ), + Index( + "ix_perm_visible_operation", + "tenant_id", + "operation_id", + "state", + "id", + ), + Index( + "ix_perm_visible_migration_item", + "migration_item_id", + "state", + "id", + ), + ) + + id: int | None = Field( + default=None, + sa_column=Column(BigInteger, primary_key=True, autoincrement=True), + ) + tenant_id: int | None = Field( + default=None, + sa_column=Column(BigInteger, nullable=False, index=True), + ) + resource_type: str = Field(sa_column=Column(String(64), nullable=False)) + resource_id: str = Field(sa_column=Column(String(64), nullable=False)) + visibility_class: str = Field(sa_column=Column(String(64), nullable=False)) + projected_subject: str = Field(sa_column=Column(String(256), nullable=False)) + source_kind: str = Field(sa_column=Column(String(64), nullable=False)) + source_owner_key: str = Field(sa_column=Column(String(256), nullable=False)) + source_locator: str = Field(sa_column=Column(String(256), nullable=False)) + source_fingerprint: str = Field(sa_column=Column(CHAR(64), nullable=False)) + contribution_fingerprint: str = Field(sa_column=Column(CHAR(64), nullable=False)) + model_key: str | None = Field( + default=None, + sa_column=Column(String(64), nullable=True), + ) + source_version: int = Field(sa_column=Column(BigInteger, nullable=False)) + tuple_fingerprint: str = Field(sa_column=Column(CHAR(64), nullable=False)) + state: str = Field( + default=VisibleSourceProjectionState.PENDING.value, + sa_column=Column(String(64), nullable=False, server_default=text("'PENDING'")), + ) + operation_id: int | None = Field( + default=None, + sa_column=Column( + BigInteger, + ForeignKey("permission_projection_operation.id", ondelete="RESTRICT"), + nullable=True, + ), + ) + migration_item_id: int | None = Field( + default=None, + sa_column=Column( + BigInteger, + ForeignKey("permission_migration_item.id", ondelete="RESTRICT"), + nullable=True, + ), + ) + create_time: datetime | None = Field( + default=None, + sa_column=Column( + DateTime, + nullable=False, + server_default=text("CURRENT_TIMESTAMP"), + ), + ) + update_time: datetime | None = Field( + default=None, + sa_column=Column( + DateTime, + nullable=False, + server_default=UPDATE_TIME_SERVER_DEFAULT, + ), + ) diff --git a/src/backend/bisheng/permission/domain/repositories/interfaces.py b/src/backend/bisheng/permission/domain/repositories/interfaces.py index 410e2f7a69..6a81c42cee 100644 --- a/src/backend/bisheng/permission/domain/repositories/interfaces.py +++ b/src/backend/bisheng/permission/domain/repositories/interfaces.py @@ -15,6 +15,7 @@ PermissionMigrationRun, PermissionProjectionOperation, PermissionProjectionTuple, + PermissionVisibleSourceProjection, ResourcePermissionMode, ) @@ -236,6 +237,87 @@ async def aget_retry_cursor( async def aget_operation_checksum(self, operation_id: int) -> str | None: """Return a checksum rebuilt from normalized tuple rows.""" + @abstractmethod + async def aupsert_visible_source( + self, + source: PermissionVisibleSourceProjection, + ) -> PermissionVisibleSourceProjection: + """Persist one canonical visible contribution idempotently.""" + + @abstractmethod + async def aretire_visible_source( + self, + *, + projection_id: int, + expected_source_version: int, + operation_id: int | None, + ) -> bool: + """Retire one contribution without deleting its audit identity.""" + + @abstractmethod + async def acount_active_visible_sources( + self, + *, + resource_type: str, + resource_id: str, + visibility_class: str, + projected_subject: str, + ) -> int: + """Count active contributions to one flattened visible tuple.""" + + @abstractmethod + async def aget_visible_model_cursor( + self, + *, + model_key: str, + states: tuple[str, ...], + after_id: int, + limit: int, + ) -> tuple[list[PermissionVisibleSourceProjection], int | None]: + """Return the current tenant's model references in stable ID order.""" + + @abstractmethod + async def aget_visible_source_cursor( + self, + *, + source_kind: str, + source_owner_key: str, + states: tuple[str, ...], + after_id: int, + limit: int, + ) -> tuple[list[PermissionVisibleSourceProjection], int | None]: + """Return source-owner contributions in stable ID order.""" + + @abstractmethod + async def aget_visible_migration_cursor( + self, + *, + migration_item_id: int, + after_id: int, + limit: int, + ) -> tuple[list[PermissionVisibleSourceProjection], int | None]: + """Return contributions linked to one formal migration item.""" + + @abstractmethod + async def aget_visible_operation_sources( + self, + operation_id: int, + ) -> list[PermissionVisibleSourceProjection]: + """Return contributions frozen by one projection operation.""" + + @abstractmethod + async def aget_visible_operation_checksum(self, operation_id: int) -> str | None: + """Return the canonical source checksum for one operation.""" + + @abstractmethod + async def aget_visible_source_checksum( + self, + *, + states: tuple[str, ...], + model_key: str | None = None, + ) -> str | None: + """Return a tenant-scoped checksum for residual verification.""" + class PermissionMigrationRepositoryPort(PermissionRepositoryTransactionPort): @abstractmethod diff --git a/src/backend/bisheng/permission/domain/repositories/projection_repository.py b/src/backend/bisheng/permission/domain/repositories/projection_repository.py index 8aa3e25606..3680cfeba7 100644 --- a/src/backend/bisheng/permission/domain/repositories/projection_repository.py +++ b/src/backend/bisheng/permission/domain/repositories/projection_repository.py @@ -13,6 +13,8 @@ from bisheng.permission.domain.models import ( PermissionProjectionOperation, PermissionProjectionTuple, + PermissionVisibleSourceProjection, + VisibleSourceProjectionState, ) from bisheng.permission.domain.repositories.catalog_repository import ( SessionFactory, @@ -179,3 +181,246 @@ async def aget_operation_checksum(self, operation_id: int) -> str | None: for row in tuples ) return sha256(canonical.encode()).hexdigest() + + async def aupsert_visible_source( + self, + source: PermissionVisibleSourceProjection, + ) -> PermissionVisibleSourceProjection: + async with self._session(write=True) as session: + statement = ( + select(PermissionVisibleSourceProjection) + .where( + PermissionVisibleSourceProjection.resource_type == source.resource_type, + PermissionVisibleSourceProjection.resource_id == source.resource_id, + PermissionVisibleSourceProjection.visibility_class == source.visibility_class, + PermissionVisibleSourceProjection.projected_subject == source.projected_subject, + PermissionVisibleSourceProjection.contribution_fingerprint + == source.contribution_fingerprint, + ) + .with_for_update() + ) + existing = (await session.execute(statement)).scalars().first() + if existing is None: + session.add(source) + await session.flush() + return source + + collision_fields = ( + "source_kind", + "source_owner_key", + "source_locator", + "source_fingerprint", + "model_key", + ) + if any(getattr(existing, field) != getattr(source, field) for field in collision_fields): + raise PermissionVersionConflictError( + msg="Visible source contribution fingerprint collision" + ) + if source.source_version < existing.source_version: + raise PermissionVersionConflictError(msg="Visible source version is stale") + + mutable_fields = ( + "source_version", + "tuple_fingerprint", + "state", + "operation_id", + "migration_item_id", + ) + if source.source_version == existing.source_version: + if all(getattr(existing, field) == getattr(source, field) for field in mutable_fields): + return existing + raise PermissionVersionConflictError( + msg="Visible source version is bound to a different projection" + ) + + for field in mutable_fields: + setattr(existing, field, getattr(source, field)) + existing.update_time = datetime.now() + session.add(existing) + await session.flush() + return existing + + async def aretire_visible_source( + self, + *, + projection_id: int, + expected_source_version: int, + operation_id: int | None, + ) -> bool: + async with self._session(write=True) as session: + statement = ( + select(PermissionVisibleSourceProjection) + .where(PermissionVisibleSourceProjection.id == projection_id) + .with_for_update() + ) + existing = (await session.execute(statement)).scalars().first() + if existing is None or existing.source_version != expected_source_version: + return False + if existing.state == VisibleSourceProjectionState.RETIRED.value: + return True + existing.state = VisibleSourceProjectionState.RETIRED.value + if operation_id is not None: + existing.operation_id = operation_id + existing.update_time = datetime.now() + session.add(existing) + await session.flush() + return True + + async def acount_active_visible_sources( + self, + *, + resource_type: str, + resource_id: str, + visibility_class: str, + projected_subject: str, + ) -> int: + async with self._session() as session: + statement = select(func.count(PermissionVisibleSourceProjection.id)).where( + PermissionVisibleSourceProjection.resource_type == resource_type, + PermissionVisibleSourceProjection.resource_id == resource_id, + PermissionVisibleSourceProjection.visibility_class == visibility_class, + PermissionVisibleSourceProjection.projected_subject == projected_subject, + PermissionVisibleSourceProjection.state == VisibleSourceProjectionState.ACTIVE.value, + ) + return int((await session.execute(statement)).scalar_one()) + + async def aget_visible_model_cursor( + self, + *, + model_key: str, + states: tuple[str, ...], + after_id: int, + limit: int, + ) -> tuple[list[PermissionVisibleSourceProjection], int | None]: + return await self._aget_visible_source_cursor( + conditions=(PermissionVisibleSourceProjection.model_key == model_key,), + states=states, + after_id=after_id, + limit=limit, + ) + + async def aget_visible_source_cursor( + self, + *, + source_kind: str, + source_owner_key: str, + states: tuple[str, ...], + after_id: int, + limit: int, + ) -> tuple[list[PermissionVisibleSourceProjection], int | None]: + return await self._aget_visible_source_cursor( + conditions=( + PermissionVisibleSourceProjection.source_kind == source_kind, + PermissionVisibleSourceProjection.source_owner_key == source_owner_key, + ), + states=states, + after_id=after_id, + limit=limit, + ) + + async def aget_visible_migration_cursor( + self, + *, + migration_item_id: int, + after_id: int, + limit: int, + ) -> tuple[list[PermissionVisibleSourceProjection], int | None]: + return await self._aget_visible_source_cursor( + conditions=(PermissionVisibleSourceProjection.migration_item_id == migration_item_id,), + states=(), + after_id=after_id, + limit=limit, + require_states=False, + ) + + async def aget_visible_operation_sources( + self, + operation_id: int, + ) -> list[PermissionVisibleSourceProjection]: + async with self._session() as session: + statement = ( + select(PermissionVisibleSourceProjection) + .where(PermissionVisibleSourceProjection.operation_id == operation_id) + .order_by(PermissionVisibleSourceProjection.id) + ) + return list((await session.execute(statement)).scalars().all()) + + async def aget_visible_operation_checksum(self, operation_id: int) -> str | None: + rows = await self.aget_visible_operation_sources(operation_id) + return _visible_source_checksum(rows) + + async def aget_visible_source_checksum( + self, + *, + states: tuple[str, ...], + model_key: str | None = None, + ) -> str | None: + if not states: + return None + async with self._session() as session: + statement = select(PermissionVisibleSourceProjection).where( + PermissionVisibleSourceProjection.state.in_(states) + ) + if model_key is not None: + statement = statement.where(PermissionVisibleSourceProjection.model_key == model_key) + rows = list((await session.execute(statement)).scalars().all()) + return _visible_source_checksum(rows) + + async def _aget_visible_source_cursor( + self, + *, + conditions: tuple, + states: tuple[str, ...], + after_id: int, + limit: int, + require_states: bool = True, + ) -> tuple[list[PermissionVisibleSourceProjection], int | None]: + if limit <= 0 or (require_states and not states): + return [], None + async with self._session() as session: + statement = ( + select(PermissionVisibleSourceProjection) + .where( + *conditions, + PermissionVisibleSourceProjection.id > after_id, + ) + .order_by(PermissionVisibleSourceProjection.id) + .limit(limit + 1) + ) + if states: + statement = statement.where(PermissionVisibleSourceProjection.state.in_(states)) + rows = list((await session.execute(statement)).scalars().all()) + items = rows[:limit] + next_cursor = int(items[-1].id) if len(rows) > limit and items else None + return items, next_cursor + + +def _visible_source_checksum( + rows: list[PermissionVisibleSourceProjection], +) -> str | None: + if not rows: + return None + canonical_rows = sorted( + "\0".join( + ( + str(row.tenant_id), + row.resource_type, + row.resource_id, + row.visibility_class, + row.projected_subject, + row.source_kind, + row.source_owner_key, + row.source_locator, + row.source_fingerprint, + row.contribution_fingerprint, + row.model_key or "", + str(row.source_version), + row.tuple_fingerprint, + row.state, + str(row.operation_id or ""), + str(row.migration_item_id or ""), + ) + ) + for row in rows + ) + return sha256("\n".join(canonical_rows).encode()).hexdigest() diff --git a/src/backend/bisheng/permission/domain/schemas/__init__.py b/src/backend/bisheng/permission/domain/schemas/__init__.py index 81e108a1fa..270b3aa25c 100644 --- a/src/backend/bisheng/permission/domain/schemas/__init__.py +++ b/src/backend/bisheng/permission/domain/schemas/__init__.py @@ -30,6 +30,10 @@ PermissionTupleAction, ResourcePermissionModeDTO, VerifiedPermissionTarget, + VisibilityEnumerationStatus, + VisibleObjectEnumerationRequest, + VisibleObjectEnumerationResult, + VisibleSourceProjectionDTO, ) __all__ = [ @@ -64,4 +68,8 @@ "PermissionTupleAction", "ResourcePermissionModeDTO", "VerifiedPermissionTarget", + "VisibilityEnumerationStatus", + "VisibleObjectEnumerationRequest", + "VisibleObjectEnumerationResult", + "VisibleSourceProjectionDTO", ] diff --git a/src/backend/bisheng/permission/domain/schemas/f048.py b/src/backend/bisheng/permission/domain/schemas/f048.py index 6225874a7e..49267ffb9a 100644 --- a/src/backend/bisheng/permission/domain/schemas/f048.py +++ b/src/backend/bisheng/permission/domain/schemas/f048.py @@ -67,6 +67,12 @@ class CatalogChangeType(StrEnum): SET_ALLOW_SAME_LEVEL = "SET_ALLOW_SAME_LEVEL" +class VisibilityEnumerationStatus(StrEnum): + """Terminal states that are safe to hand to a business list caller.""" + + NORMAL = "NORMAL" + + class StrictRequestModel(BaseModel): """Base class that rejects undeclared, server-owned request fields.""" @@ -354,6 +360,63 @@ class PermissionMutation(BaseModel): fga_object: str = Field(min_length=1, max_length=256) +class VisibleObjectEnumerationRequest(BaseModel): + """Bounded internal request for one complete visible-object enumeration.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + tenant_id: int = Field(gt=0) + resource_type: str = Field(min_length=1, max_length=64) + fga_user: str = Field(min_length=1, max_length=256) + max_results: int = Field(gt=0, le=5_000) + + +class VisibleObjectEnumerationResult(BaseModel): + """Immutable result emitted only after the upstream stream ends normally.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + resource_type: str = Field(min_length=1, max_length=64) + object_ids: tuple[str, ...] = () + max_results: int = Field(gt=0, le=5_000) + status: VisibilityEnumerationStatus + + @model_validator(mode="after") + def validate_complete_set(self) -> VisibleObjectEnumerationResult: + if self.status is not VisibilityEnumerationStatus.NORMAL: + raise ValueError("only normally completed enumeration results may be delivered") + if len(self.object_ids) > self.max_results: + raise ValueError("enumeration result exceeds max_results") + if len(set(self.object_ids)) != len(self.object_ids): + raise ValueError("enumeration result must already be deduplicated") + if any(not object_id or len(object_id) > 64 for object_id in self.object_ids): + raise ValueError("enumeration result contains an invalid resource ID") + return self + + +class VisibleSourceProjectionDTO(BaseModel): + """Canonical source contribution used to compile one flattened visible tuple.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + tenant_id: int = Field(gt=0) + resource_type: str = Field(min_length=1, max_length=64) + resource_id: str = Field(min_length=1, max_length=64) + visibility_class: str = Field(min_length=1, max_length=64) + projected_subject: str = Field(min_length=1, max_length=256) + source_kind: str = Field(min_length=1, max_length=64) + source_owner_key: str = Field(min_length=1, max_length=256) + source_locator: str = Field(min_length=1, max_length=256) + source_fingerprint: str = Field(pattern=r"^[0-9a-f]{64}$") + contribution_fingerprint: str = Field(pattern=r"^[0-9a-f]{64}$") + model_key: str | None = Field(default=None, min_length=1, max_length=64) + source_version: int = Field(ge=0) + tuple_fingerprint: str = Field(pattern=r"^[0-9a-f]{64}$") + state: str = Field(min_length=1, max_length=64) + operation_id: int | None = Field(default=None, gt=0) + migration_item_id: int | None = Field(default=None, gt=0) + + class PermissionCheckRequest(StrictRequestModel): resource_type: str = Field(min_length=1, max_length=64) resource_id: str = Field(min_length=1, max_length=64) diff --git a/src/backend/bisheng/permission/domain/services/catalog_service.py b/src/backend/bisheng/permission/domain/services/catalog_service.py index 6cee2fcfb0..15e3342798 100644 --- a/src/backend/bisheng/permission/domain/services/catalog_service.py +++ b/src/backend/bisheng/permission/domain/services/catalog_service.py @@ -18,6 +18,7 @@ from bisheng.common.errcode.permission import ( AuthorizationModelMismatchError, PermissionImpactExpiredError, + PermissionModelStateConflictError, PermissionProjectionFailedError, PermissionPublishNotReadyError, ) @@ -30,10 +31,12 @@ ) from bisheng.permission.domain.services.model_policy import ( CustomModelSelection, + ModelReferenceSummary, PermissionModelImpact, PermissionModelRelease, calculate_model_impact, derive_permission_models, + ensure_model_deletable, ) @@ -68,6 +71,7 @@ class CatalogDraftBuildInput: custom_models: tuple[CustomModelSelection, ...] = () standard_allow_same_level: Mapping[str, bool] = field(default_factory=dict) grant_references: Mapping[str, tuple[str, ...]] = field(default_factory=dict) + model_reference_summaries: Mapping[str, ModelReferenceSummary] = field(default_factory=dict) draft_owner_id: int = 0 idempotency_key: str = "" expires_at: datetime | None = None @@ -259,6 +263,25 @@ async def build_draft( custom_models=build.custom_models, standard_allow_same_level=build.standard_allow_same_level, ) + before_by_key = {model.model_key: model for model in build.before_models.models} + after_keys = {model.model_key for model in model_release.models} + deleted_models = tuple( + before_by_key[model_key] + for model_key in sorted(set(before_by_key) - after_keys) + ) + for model in deleted_models: + references = build.model_reference_summaries.get(model.model_key) + if references is None: + raise PermissionModelStateConflictError( + msg=f"Model reference audit is missing: {model.model_key}" + ) + try: + ensure_model_deletable(model, references=references) + except ValueError as exc: + raise PermissionModelStateConflictError( + exception=exc, + msg=str(exc), + ) from exc custom_action_map = { model.model_key: frozenset(model.selected_action_codes) for model in build.before_models.models diff --git a/src/backend/bisheng/permission/domain/services/grant_service.py b/src/backend/bisheng/permission/domain/services/grant_service.py index cc969b4e9f..9e7c2932f4 100644 --- a/src/backend/bisheng/permission/domain/services/grant_service.py +++ b/src/backend/bisheng/permission/domain/services/grant_service.py @@ -19,18 +19,26 @@ PermissionProjectionOperation, ProjectionOperationStatus, ) -from bisheng.permission.domain.schemas import VerifiedPermissionTarget +from bisheng.permission.domain.schemas import ( + VerifiedPermissionTarget, + VisibleSourceProjectionDTO, +) from bisheng.permission.domain.services.grant_source_service import ( GrantModelSnapshot, GrantSnapshot, GrantSourceRecord, GrantSourceService, ) +from bisheng.permission.domain.services.projection_plan import merge_projection_deltas from bisheng.permission.domain.services.projection_service import ( ProjectionOutcome, ProjectionPlan, ProjectionTupleDelta, ) +from bisheng.permission.domain.services.visibility_projection_service import ( + VisibilityProjectionCompilation, + VisibilityProjectionCompiler, +) @dataclass(frozen=True, slots=True) @@ -67,6 +75,7 @@ class GrantMutationContext: capabilities: tuple[GrantCapability, ...] models: tuple[GrantModelSnapshot, ...] grants: tuple[GrantSnapshot, ...] + existing_visible_sources: tuple[VisibleSourceProjectionDTO, ...] = () @dataclass(frozen=True, slots=True) @@ -96,6 +105,7 @@ async def prepare( self, context: GrantMutationContext, grants: tuple[GrantSnapshot, ...], + visibility: VisibilityProjectionCompilation, *, idempotency_key: str, operation_id: int, @@ -105,6 +115,7 @@ async def finalize( self, context: GrantMutationContext, grants: tuple[GrantSnapshot, ...], + visibility: VisibilityProjectionCompilation, outcome: ProjectionOutcome, ) -> None: ... @@ -128,11 +139,13 @@ def __init__( projection: GrantProjectionPort, state: GrantMutationStatePort, events: GrantEventPort | None = None, + visibility_compiler: VisibilityProjectionCompiler | None = None, ) -> None: self._sources = source_service self._projection = projection self._state = state self._events = events or _NullEvents() + self._visibility = visibility_compiler or VisibilityProjectionCompiler() def grantable_models( self, @@ -188,8 +201,7 @@ def assert_roster_access(self, context: GrantMutationContext) -> None: if context.system_authorized: return if not any( - capability.model.active - and capability.model.derived_level is not None + capability.model.derived_level is not None and "manage_permission" in capability.model.action_codes for capability in context.capabilities ): @@ -230,7 +242,15 @@ async def mutate( compiled.extend(deltas) final_grants = tuple(grants[model_key] for model_key in grant_order) - net_deltas = self._net_deltas(tuple(compiled)) + visibility = self._visibility.compile( + tenant_id=context.target.tenant_id, + grants=final_grants, + existing_sources=context.existing_visible_sources, + ) + net_deltas = merge_projection_deltas( + self._net_deltas(tuple(compiled)), + visibility.deltas, + ) plan = ProjectionPlan( tenant_id=context.target.tenant_id, idempotency_key=idempotency_key, @@ -251,6 +271,7 @@ async def mutate( await self._state.prepare( context, final_grants, + visibility, idempotency_key=idempotency_key, operation_id=int(operation.id), ) @@ -258,7 +279,7 @@ async def mutate( await self._projection.abandon_prepared(plan, exc) raise outcome = await self._projection.execute(plan) - await self._state.finalize(context, final_grants, outcome) + await self._state.finalize(context, final_grants, visibility, outcome) await self._emit(context, outcome, len(changes), len(net_deltas)) return GrantMutationResult( grants=final_grants, @@ -273,8 +294,7 @@ def _capability_allows( ) -> bool: source = capability.model if ( - not source.active - or source.derived_level is None + source.derived_level is None or target.derived_level is None or "manage_permission" not in source.action_codes ): @@ -325,7 +345,7 @@ def _apply_change( raise PermissionVersionConflictError(msg="Grant assignee version changed") if source.protected: raise ProtectedAssignmentMutationError(msg="Protected permission source cannot be changed") - self.require_grantable_model(context, source_model_key) + self.require_manageable_existing_model(context, source_model_key) source_grant = grants[source_model_key] if operation == "REMOVE": @@ -352,6 +372,22 @@ def _apply_change( grants[target_model.model_key] = mutation.target_grant return mutation.deltas + def require_manageable_existing_model( + self, + context: GrantMutationContext, + model_key: str, + ) -> GrantModelSnapshot: + """Authorize cleanup of an existing binding even after model deactivation.""" + + model = next((row for row in context.models if row.model_key == model_key), None) + if model is None or model.derived_level is None or not model.action_codes: + raise PermissionModelStateConflictError(msg=f"Permission model is unavailable: {model_key}") + if not context.system_authorized and not any( + self._capability_allows(capability, model) for capability in context.capabilities + ): + raise GrantLevelForbiddenError(msg=f"Actor cannot manage permission model: {model_key}") + return model + @staticmethod def _target_grant( grants: dict[str, GrantSnapshot], diff --git a/src/backend/bisheng/permission/domain/services/grant_source_service.py b/src/backend/bisheng/permission/domain/services/grant_source_service.py index 6e94970e60..e20299618d 100644 --- a/src/backend/bisheng/permission/domain/services/grant_source_service.py +++ b/src/backend/bisheng/permission/domain/services/grant_source_service.py @@ -348,7 +348,7 @@ def effective_action_union( actions = { action for grant in grants - if grant.active and grant.model.active + if grant.active if any(source.active and source.projected_subject in projected_subjects for source in grant.sources) for action in grant.model.action_codes } diff --git a/src/backend/bisheng/permission/domain/services/model_policy.py b/src/backend/bisheng/permission/domain/services/model_policy.py index b44caab489..1e342655f4 100644 --- a/src/backend/bisheng/permission/domain/services/model_policy.py +++ b/src/backend/bisheng/permission/domain/services/model_policy.py @@ -74,6 +74,52 @@ class PermissionModelImpact: checksum: str +@dataclass(frozen=True, slots=True) +class ModelReferenceSummary: + """Cross-tenant deletion gate for one stable model key.""" + + active_grant_count: int = 0 + pending_grant_count: int = 0 + failed_grant_count: int = 0 + active_source_count: int = 0 + pending_source_count: int = 0 + failed_source_count: int = 0 + live_tuple_count: int = 0 + residual_checksum: str | None = None + + def __post_init__(self) -> None: + counts = ( + self.active_grant_count, + self.pending_grant_count, + self.failed_grant_count, + self.active_source_count, + self.pending_source_count, + self.failed_source_count, + self.live_tuple_count, + ) + if any(count < 0 for count in counts): + raise ValueError("model reference counts must be non-negative") + if self.residual_checksum is not None and ( + len(self.residual_checksum) != 64 + or any(character not in "0123456789abcdef" for character in self.residual_checksum) + ): + raise ValueError("model residual checksum must be lowercase SHA-256") + + @property + def reference_count(self) -> int: + return sum( + ( + self.active_grant_count, + self.pending_grant_count, + self.failed_grant_count, + self.active_source_count, + self.pending_source_count, + self.failed_source_count, + self.live_tuple_count, + ) + ) + + def _checksum(payload: object) -> str: serialized = json.dumps( payload, @@ -253,8 +299,6 @@ def effective_model_action_codes( ) -> tuple[str, ...]: """Intersect a model with active actions applicable to one resource.""" - if not model.active: - return () model_actions = set(model.action_codes) return tuple( action.code @@ -266,6 +310,13 @@ def effective_model_action_codes( ) +def ensure_model_assignable(model: DerivedPermissionModel) -> None: + """Reject inactive models only when selecting a target for ADD or MOVE.""" + + if not model.active: + raise ValueError(f"permission model {model.model_key} is inactive") + + def with_allow_same_level( model: DerivedPermissionModel, allow_same_level: bool, @@ -305,16 +356,14 @@ def validate_standard_model_update( def ensure_model_deletable( model: DerivedPermissionModel, *, - reference_count: int, + references: ModelReferenceSummary, ) -> None: - """Permit final deletion only for an inactive, unreferenced custom model.""" + """Permit final deletion only after every durable/live reference is zero.""" if model.kind == "STANDARD": raise ValueError("standard models cannot be deleted") - if model.active: - raise ValueError("custom model must be inactive before deletion") - if reference_count: - raise ValueError("custom model is still referenced by Grants") + if references.reference_count or references.residual_checksum is not None: + raise ValueError("custom model is still referenced by Grants, sources, or live tuples") def initialize_from_preset( diff --git a/src/backend/bisheng/permission/domain/services/owner_service.py b/src/backend/bisheng/permission/domain/services/owner_service.py index 2aff5b785b..eb9ff70aa4 100644 --- a/src/backend/bisheng/permission/domain/services/owner_service.py +++ b/src/backend/bisheng/permission/domain/services/owner_service.py @@ -123,7 +123,7 @@ async def project_created( grant = None source = None marker_relations = ( - "system_visible_marker", + "visible", *(f"system_{action}_marker" for action in context.system_action_codes if action != "visible"), ) protected_deltas = tuple( diff --git a/src/backend/bisheng/permission/domain/services/permission_action_service.py b/src/backend/bisheng/permission/domain/services/permission_action_service.py index 0258f9db43..a0b36a5828 100644 --- a/src/backend/bisheng/permission/domain/services/permission_action_service.py +++ b/src/backend/bisheng/permission/domain/services/permission_action_service.py @@ -10,11 +10,18 @@ from bisheng.common.errcode.permission import ( InvalidCatalogActionError, + PermissionEnumerationIncompleteError, PermissionFGAUnavailableError, PermissionProjectionFailedError, PermissionPublishNotReadyError, ) -from bisheng.permission.domain.schemas import VerifiedPermissionTarget +from bisheng.common.services.metric_log import emit_metric +from bisheng.permission.domain.schemas import ( + VerifiedPermissionTarget, + VisibilityEnumerationStatus, + VisibleObjectEnumerationRequest, + VisibleObjectEnumerationResult, +) from bisheng.permission.domain.services.catalog_policy import ( REGISTERED_ACTION_CODES, ) @@ -83,6 +90,15 @@ async def list_objects( consistency: str | None = None, ) -> list[str]: ... + async def stream_list_objects( + self, + *, + user: str, + relation: str, + type: str, + consistency: str | None = None, + ) -> tuple[str, ...]: ... + class PermissionListPolicyPort(Protocol): async def allows( @@ -173,23 +189,17 @@ async def check_visible( target: VerifiedPermissionTarget, ) -> bool: started = perf_counter() - shortcut = await self._identity_shortcut( - actor, - target, - action="visible", - ) - if shortcut is not None: - allowed, reason = shortcut + if target.tenant_id != actor.current_tenant_id: await self._emit_decision( actor, target, "visible", - allowed, - reason, + False, + "TENANT_MISMATCH", None, started, ) - return allowed + return False await self._catalog.ensure_runtime_ready() await self._scope_fence.ensure_readable(target) consistency = await self._consistency(target) @@ -280,13 +290,8 @@ async def batch_check_visible( unresolved: list[tuple[int, VerifiedPermissionTarget]] = [] consistency = None for index, target in enumerate(targets): - shortcut = await self._identity_shortcut( - actor, - target, - action="visible", - ) - if shortcut is not None: - results[index] = shortcut[0] + if target.tenant_id != actor.current_tenant_id: + results[index] = False continue await self._catalog.ensure_runtime_ready() await self._scope_fence.ensure_readable(target) @@ -321,6 +326,115 @@ async def batch_check_visible( results[index] = bool(allowed) return tuple(bool(value) for value in results) + async def list_visible_objects( + self, + actor: PermissionActor, + *, + resource_type: str, + max_results: int, + ) -> VisibleObjectEnumerationResult: + """Return a complete immutable visible ID set after normal stream EOF.""" + + if actor.current_tenant_id <= 0: + raise PermissionEnumerationIncompleteError(msg="Visible enumeration has no valid tenant fence") + request = VisibleObjectEnumerationRequest( + tenant_id=actor.current_tenant_id, + resource_type=resource_type, + fga_user=f"user:{actor.user_id}", + max_results=max_results, + ) + started = perf_counter() + await self._catalog.ensure_runtime_ready() + consistency = await self._scope_consistency( + request.tenant_id, + request.resource_type, + None, + ) + try: + fga_started = perf_counter() + objects = await self._fga.stream_list_objects( + user=request.fga_user, + relation="visible", + type=request.resource_type, + consistency=consistency, + ) + except Exception as exc: + emit_metric( + "permission_visible_list", + tenant=request.tenant_id, + resource_type=request.resource_type, + strategy="visible_ids_first", + candidate_count=0, + visible_count=0, + scanned_count=0, + scan_amplification=0, + stream_completed=False, + capacity=request.max_results, + db_elapsed_ms=0, + fga_elapsed_ms=(perf_counter() - fga_started) * 1000, + total_elapsed_ms=(perf_counter() - started) * 1000, + alert="stream_incomplete", + ) + raise PermissionEnumerationIncompleteError(exception=exc) from exc + + prefix = f"{request.resource_type}:" + if any(not value.startswith(prefix) for value in objects): + raise PermissionEnumerationIncompleteError( + msg="OpenFGA visible enumeration returned an unexpected object type", + ) + object_ids = tuple(sorted({value[len(prefix) :] for value in objects})) + if len(object_ids) > request.max_results: + self._emit_visible_list_metric( + request=request, + visible_count=len(object_ids), + fga_started=fga_started, + started=started, + alert="capacity_exceeded", + ) + raise PermissionEnumerationIncompleteError( + msg="Visible enumeration exceeded its reviewed capacity", + ) + capacity_ratio = len(object_ids) / request.max_results + self._emit_visible_list_metric( + request=request, + visible_count=len(object_ids), + fga_started=fga_started, + started=started, + alert="capacity_80_percent" if capacity_ratio >= 0.8 else None, + ) + return VisibleObjectEnumerationResult( + resource_type=request.resource_type, + object_ids=object_ids, + max_results=request.max_results, + status=VisibilityEnumerationStatus.NORMAL, + ) + + @staticmethod + def _emit_visible_list_metric( + *, + request: VisibleObjectEnumerationRequest, + visible_count: int, + fga_started: float, + started: float, + alert: str | None, + ) -> None: + emit_metric( + "permission_visible_list", + tenant=request.tenant_id, + resource_type=request.resource_type, + strategy="visible_ids_first", + candidate_count=0, + visible_count=visible_count, + scanned_count=visible_count, + scan_amplification=1 if visible_count else 0, + stream_completed=True, + capacity=request.max_results, + db_elapsed_ms=0, + fga_elapsed_ms=(perf_counter() - fga_started) * 1000, + total_elapsed_ms=(perf_counter() - started) * 1000, + alert=alert, + ) + async def list_action_objects( self, actor: PermissionActor, diff --git a/src/backend/bisheng/permission/domain/services/projection_plan.py b/src/backend/bisheng/permission/domain/services/projection_plan.py index 855e19f07d..5f7dce9d5e 100644 --- a/src/backend/bisheng/permission/domain/services/projection_plan.py +++ b/src/backend/bisheng/permission/domain/services/projection_plan.py @@ -161,6 +161,25 @@ def normalize_projection_plan(plan: ProjectionPlan) -> ProjectionPlan: return replace(plan, deltas=deltas) +def merge_projection_deltas( + *delta_groups: tuple[ProjectionTupleDelta, ...], +) -> tuple[ProjectionTupleDelta, ...]: + """Merge action and visibility deltas into one deterministic operation.""" + + ordered = sorted( + (delta for group in delta_groups for delta in group), + key=lambda row: ( + _PHASE_ORDER.get(row.phase.upper(), len(_PHASE_ORDER)), + row.user, + row.relation, + row.object, + row.action, + row.sequence, + ), + ) + return tuple(replace(delta, sequence=index) for index, delta in enumerate(ordered)) + + def projection_request_checksum(plan: ProjectionPlan) -> str: return _checksum( { diff --git a/src/backend/bisheng/permission/domain/services/visibility_projection_service.py b/src/backend/bisheng/permission/domain/services/visibility_projection_service.py new file mode 100644 index 0000000000..5392d16e88 --- /dev/null +++ b/src/backend/bisheng/permission/domain/services/visibility_projection_service.py @@ -0,0 +1,472 @@ +"""Pure compiler for flattened visible source contributions.""" + +from __future__ import annotations + +import json +from dataclasses import dataclass +from hashlib import sha256 +from time import perf_counter + +from bisheng.common.errcode.permission import PermissionPublishNotReadyError +from bisheng.common.services.metric_log import emit_metric +from bisheng.permission.domain.schemas import VisibleSourceProjectionDTO +from bisheng.permission.domain.services.grant_source_service import ( + GrantSnapshot, + GrantSourceRecord, +) +from bisheng.permission.domain.services.projection_plan import ProjectionTupleDelta + +_GRANT_SOURCE_TYPES = frozenset( + { + "DIRECT", + "DEPARTMENT", + "USER_GROUP", + "CREATOR", + "SPACE_MEMBERSHIP", + "CHANNEL_MEMBERSHIP", + "SNAPSHOT_FROM_PARENT", + "OTHER", + } +) + + +@dataclass(frozen=True, slots=True) +class VisibilityProjectionCompilation: + active_sources: tuple[VisibleSourceProjectionDTO, ...] + retired_sources: tuple[VisibleSourceProjectionDTO, ...] + deltas: tuple[ProjectionTupleDelta, ...] + source_checksum: str + aggregate_checksum: str + + +@dataclass(frozen=True, slots=True) +class VisibilityProjectionReconcilePlan: + """Exact source and aggregate differences for one fenced resource.""" + + upsert_sources: tuple[VisibleSourceProjectionDTO, ...] + retire_sources: tuple[VisibleSourceProjectionDTO, ...] + deltas: tuple[ProjectionTupleDelta, ...] + source_checksum: str + target_checksum: str + live_checksum: str + blockers: tuple[str, ...] = () + + +def _hash(value: str) -> str: + return sha256(value.encode()).hexdigest() + + +def _checksum(payload: object) -> str: + serialized = json.dumps( + payload, + ensure_ascii=True, + separators=(",", ":"), + sort_keys=True, + ) + return sha256(serialized.encode()).hexdigest() + + +def _aggregate_key( + source: VisibleSourceProjectionDTO, +) -> tuple[str, str, str]: + return ( + source.resource_type, + source.resource_id, + source.projected_subject, + ) + + +def _source_payload(source: VisibleSourceProjectionDTO) -> dict[str, object]: + return source.model_dump(mode="json") + + +class VisibilityProjectionCompiler: + """Compile canonical Grant sources without resolving organization members.""" + + def compile( + self, + *, + tenant_id: int, + grants: tuple[GrantSnapshot, ...], + existing_sources: tuple[VisibleSourceProjectionDTO, ...], + ) -> VisibilityProjectionCompilation: + started = perf_counter() + if tenant_id <= 0: + raise ValueError("visibility projection tenant_id must be positive") + if any(grant.tenant_id != tenant_id for grant in grants): + raise ValueError("visibility projection cannot mix tenants") + if any(source.tenant_id != tenant_id for source in existing_sources): + raise ValueError("visibility projection cannot mix existing tenants") + + desired = tuple( + sorted( + ( + self._compile_source(tenant_id, grant, source) + for grant in grants + if grant.active + for source in grant.sources + if source.active and source.source_type in _GRANT_SOURCE_TYPES + ), + key=lambda row: ( + row.resource_type, + row.resource_id, + row.visibility_class, + row.projected_subject, + row.contribution_fingerprint, + ), + ) + ) + desired_by_fingerprint = { + source.contribution_fingerprint: source for source in desired + } + if len(desired_by_fingerprint) != len(desired): + raise ValueError("visibility contribution fingerprints must be unique") + + existing_active = tuple(source for source in existing_sources if source.state == "ACTIVE") + existing_by_fingerprint = { + source.contribution_fingerprint: source for source in existing_active + } + active_sources: list[VisibleSourceProjectionDTO] = [] + for fingerprint, source in desired_by_fingerprint.items(): + previous = existing_by_fingerprint.get(fingerprint) + if previous is None: + active_sources.append(source) + continue + self._assert_same_canonical_source(previous, source) + active_sources.append( + source.model_copy( + update={ + "source_version": max(previous.source_version, source.source_version), + "operation_id": previous.operation_id, + "migration_item_id": previous.migration_item_id, + } + ) + ) + + retired_sources = tuple( + sorted( + ( + source.model_copy(update={"state": "RETIRED"}) + for fingerprint, source in existing_by_fingerprint.items() + if fingerprint not in desired_by_fingerprint + ), + key=lambda row: row.contribution_fingerprint, + ) + ) + active_sources_tuple = tuple( + sorted( + active_sources, + key=lambda row: ( + row.resource_type, + row.resource_id, + row.visibility_class, + row.projected_subject, + row.contribution_fingerprint, + ), + ) + ) + + before_aggregates = {_aggregate_key(source) for source in existing_active} + after_aggregates = {_aggregate_key(source) for source in active_sources_tuple} + deltas = self._aggregate_deltas( + before=before_aggregates, + after=after_aggregates, + ) + source_checksum = _checksum( + [_source_payload(source) for source in (*active_sources_tuple, *retired_sources)] + ) + aggregate_checksum = _checksum( + [list(key) for key in sorted(after_aggregates)] + ) + result = VisibilityProjectionCompilation( + active_sources=active_sources_tuple, + retired_sources=retired_sources, + deltas=deltas, + source_checksum=source_checksum, + aggregate_checksum=aggregate_checksum, + ) + emit_metric( + "permission_visibility_projection", + operation="project", + tenant=tenant_id, + source_count=len(result.active_sources), + unique_tuple_count=len(after_aggregates), + stale_count=len(result.retired_sources), + orphan_count=0, + source_checksum=result.source_checksum, + aggregate_checksum=result.aggregate_checksum, + elapsed_ms=(perf_counter() - started) * 1000, + alert=None, + ) + return result + + @staticmethod + def _compile_source( + tenant_id: int, + grant: GrantSnapshot, + source: GrantSourceRecord, + ) -> VisibleSourceProjectionDTO: + visibility_class = "protected" if source.protected else "ordinary" + relation = "visible" + source_owner_key = f"grant_assignee:{source.source_id}" + contribution_fingerprint = _hash( + "\0".join( + ( + "GRANT_ASSIGNEE", + source_owner_key, + source.source_fingerprint, + grant.model.model_key, + ) + ) + ) + fga_object = f"{grant.resource_type}:{grant.resource_id}" + tuple_fingerprint = _hash( + "\0".join( + ( + "WRITE", + source.projected_subject, + relation, + fga_object, + ) + ) + ) + return VisibleSourceProjectionDTO( + tenant_id=tenant_id, + resource_type=grant.resource_type, + resource_id=grant.resource_id, + visibility_class=visibility_class, + projected_subject=source.projected_subject, + source_kind="GRANT_ASSIGNEE", + source_owner_key=source_owner_key, + source_locator=source.source_locator, + source_fingerprint=source.source_fingerprint, + contribution_fingerprint=contribution_fingerprint, + model_key=grant.model.model_key, + source_version=source.version, + tuple_fingerprint=tuple_fingerprint, + state="ACTIVE", + ) + + @staticmethod + def _aggregate_deltas( + *, + before: set[tuple[str, str, str]], + after: set[tuple[str, str, str]], + ) -> tuple[ProjectionTupleDelta, ...]: + changes = ( + [("DELETE", key) for key in sorted(before - after)] + + [("WRITE", key) for key in sorted(after - before)] + ) + return tuple( + ProjectionTupleDelta( + phase="COMMIT", + sequence=index, + action=action, + user=projected_subject, + relation="visible", + object=f"{resource_type}:{resource_id}", + ) + for index, ( + action, + (resource_type, resource_id, projected_subject), + ) in enumerate(changes) + ) + + @staticmethod + def _assert_same_canonical_source( + previous: VisibleSourceProjectionDTO, + current: VisibleSourceProjectionDTO, + ) -> None: + immutable_fields = ( + "tenant_id", + "resource_type", + "resource_id", + "visibility_class", + "projected_subject", + "source_kind", + "source_owner_key", + "source_locator", + "source_fingerprint", + "contribution_fingerprint", + "model_key", + "tuple_fingerprint", + ) + if any(getattr(previous, field) != getattr(current, field) for field in immutable_fields): + raise ValueError("visibility contribution fingerprint collision") + + +class VisibilityProjectionReconciler: + """Plan fail-closed convergence from canonical sources to live tuples.""" + + def plan( + self, + *, + canonical_sources: tuple[VisibleSourceProjectionDTO, ...], + persisted_sources: tuple[VisibleSourceProjectionDTO, ...], + live_tuples: frozenset[tuple[str, str, str]], + ledger_complete: bool = True, + ) -> VisibilityProjectionReconcilePlan: + started = perf_counter() + blockers: list[str] = [] + if not ledger_complete: + blockers.append("visibility projection ledger is incomplete") + if any(source.state == "FAILED_CLOSED" for source in persisted_sources): + blockers.append("visibility source projection contains FAILED_CLOSED rows") + + canonical_by_fingerprint = self._unique_sources( + canonical_sources, + label="canonical", + blockers=blockers, + ) + persisted_by_fingerprint = self._unique_sources( + persisted_sources, + label="persisted", + blockers=blockers, + ) + self._validate_scope( + (*canonical_sources, *persisted_sources), + live_tuples, + blockers, + ) + + upsert: list[VisibleSourceProjectionDTO] = [] + for fingerprint, desired in canonical_by_fingerprint.items(): + existing = persisted_by_fingerprint.get(fingerprint) + if existing is None: + upsert.append(desired.model_copy(update={"state": "ACTIVE"})) + continue + try: + VisibilityProjectionCompiler._assert_same_canonical_source(existing, desired) + except ValueError: + blockers.append( + f"visibility contribution collision: {fingerprint}", + ) + continue + if existing.state != "ACTIVE" or existing.source_version != desired.source_version: + upsert.append(desired.model_copy(update={"state": "ACTIVE"})) + + retire = tuple( + sorted( + ( + source + for fingerprint, source in persisted_by_fingerprint.items() + if fingerprint not in canonical_by_fingerprint + and source.state in {"ACTIVE", "PENDING"} + ), + key=lambda row: row.contribution_fingerprint, + ) + ) + upsert_tuple = tuple( + sorted(upsert, key=lambda row: row.contribution_fingerprint) + ) + target_tuples = frozenset( + self._live_key(source) for source in canonical_by_fingerprint.values() + ) + changes = ( + [("DELETE", key) for key in sorted(live_tuples - target_tuples)] + + [("WRITE", key) for key in sorted(target_tuples - live_tuples)] + ) + deltas = tuple( + ProjectionTupleDelta( + phase="COMMIT", + sequence=index, + action=action, + user=key[0], + relation=key[1], + object=key[2], + ) + for index, (action, key) in enumerate(changes) + ) + active_canonical = tuple( + sorted( + (source.model_copy(update={"state": "ACTIVE"}) for source in canonical_by_fingerprint.values()), + key=lambda row: row.contribution_fingerprint, + ) + ) + result = VisibilityProjectionReconcilePlan( + upsert_sources=upsert_tuple, + retire_sources=retire, + deltas=deltas, + source_checksum=_checksum( + [_source_payload(source) for source in active_canonical] + ), + target_checksum=self._tuple_checksum(target_tuples), + live_checksum=self._tuple_checksum(live_tuples), + blockers=tuple(dict.fromkeys(blockers)), + ) + orphan_count = len(live_tuples - target_tuples) + emit_metric( + "permission_visibility_projection", + operation="reconcile", + source_count=len(active_canonical), + unique_tuple_count=len(target_tuples), + stale_count=len(retire), + orphan_count=orphan_count, + source_checksum=result.source_checksum, + aggregate_checksum=result.target_checksum, + live_checksum=result.live_checksum, + elapsed_ms=(perf_counter() - started) * 1000, + alert=( + "orphan_visible_tuple" + if orphan_count + else "checksum_mismatch" + if result.target_checksum != result.live_checksum + else "projection_blocked" + if result.blockers + else None + ), + ) + return result + + @staticmethod + def ensure_ready(plan: VisibilityProjectionReconcilePlan) -> None: + if plan.blockers: + raise PermissionPublishNotReadyError(msg="; ".join(plan.blockers)) + + @staticmethod + def _unique_sources( + sources: tuple[VisibleSourceProjectionDTO, ...], + *, + label: str, + blockers: list[str], + ) -> dict[str, VisibleSourceProjectionDTO]: + result: dict[str, VisibleSourceProjectionDTO] = {} + for source in sources: + fingerprint = source.contribution_fingerprint + if fingerprint in result: + blockers.append( + f"duplicate {label} visibility contribution: {fingerprint}", + ) + continue + result[fingerprint] = source + return result + + @staticmethod + def _live_key( + source: VisibleSourceProjectionDTO, + ) -> tuple[str, str, str]: + return ( + source.projected_subject, + "visible", + f"{source.resource_type}:{source.resource_id}", + ) + + @staticmethod + def _tuple_checksum(tuples: frozenset[tuple[str, str, str]]) -> str: + return _checksum([list(key) for key in sorted(tuples)]) + + @classmethod + def _validate_scope( + cls, + sources: tuple[VisibleSourceProjectionDTO, ...], + live_tuples: frozenset[tuple[str, str, str]], + blockers: list[str], + ) -> None: + scopes = {(source.tenant_id, source.resource_type, source.resource_id) for source in sources} + if len(scopes) > 1: + blockers.append("visibility reconcile cannot mix resource scopes") + resource_keys = {f"{resource_type}:{resource_id}" for _, resource_type, resource_id in scopes} + if resource_keys and any(key[2] not in resource_keys for key in live_tuples): + blockers.append("visibility reconcile live tuple is outside the fenced scope") + if any(key[1] != "visible" for key in live_tuples): + blockers.append("visibility reconcile received a non-flattened relation") diff --git a/src/backend/bisheng/permission/migration/f048_coordinator.py b/src/backend/bisheng/permission/migration/f048_coordinator.py index 5cba016a0d..fc9e004e56 100644 --- a/src/backend/bisheng/permission/migration/f048_coordinator.py +++ b/src/backend/bisheng/permission/migration/f048_coordinator.py @@ -15,10 +15,18 @@ authorization_model_checksum, build_authorization_model_f048, ) +from bisheng.permission.domain.services.grant_source_service import ( + GrantModelSnapshot, + GrantSnapshot, + GrantSourceRecord, +) from bisheng.permission.domain.services.model_policy import ( CustomModelSelection, derive_permission_models, ) +from bisheng.permission.domain.services.visibility_projection_service import ( + VisibilityProjectionCompiler, +) from bisheng.permission.migration.f048_mode_mapper import ( ModeMappingResult, map_resource_modes, @@ -421,6 +429,60 @@ def add(user: str, relation: str, object_key: str) -> None: ("protected_assignee" if assignee.protected else "ordinary_assignee"), grant_object, ) + + active_by_model = { + model.model_key: model.active for model in model_mapping.custom_models + } + grants_by_tenant: dict[int, list[GrantSnapshot]] = {} + source_id = 0 + for grant in tuple_mapping.grants: + sources: list[GrantSourceRecord] = [] + for assignee in grant.assignees: + source_id += 1 + projected_subject = f"{assignee.subject_type}:{assignee.subject_id}" + if assignee.userset_relation: + projected_subject += f"#{assignee.userset_relation}" + sources.append( + GrantSourceRecord( + source_id=source_id, + subject_type=assignee.subject_type, + subject_id=assignee.subject_id, + userset_relation=assignee.userset_relation, + include_children=assignee.include_children, + source_type=assignee.source_type, + source_ref=assignee.source_ref, + source_locator=(f"migration:{assignee.source_type}:{assignee.source_ref}")[:256], + source_fingerprint=assignee.source_checksum, + projected_subject=projected_subject, + protected=assignee.protected, + ) + ) + grants_by_tenant.setdefault(grant.tenant_id, []).append( + GrantSnapshot( + grant_id=grant.grant_key, + tenant_id=grant.tenant_id, + resource_type=grant.resource_type, + resource_id=grant.resource_id, + model=GrantModelSnapshot( + model_key=grant.model_key, + active=active_by_model.get(grant.model_key, True), + action_codes=(), + ), + active=True, + sources=tuple(sources), + ) + ) + visibility_compiler = VisibilityProjectionCompiler() + for tenant_id, grants in sorted(grants_by_tenant.items()): + visibility = visibility_compiler.compile( + tenant_id=tenant_id, + grants=tuple(grants), + existing_sources=(), + ) + for delta in visibility.deltas: + if delta.action != "WRITE": + raise ValueError("initial visibility compilation produced a delete") + add(delta.user, delta.relation, delta.object) for mode in mode_mapping.modes: resource_object = mode.resource_key add("user:*", f"{mode.mode.casefold()}_mode", resource_object) @@ -431,7 +493,7 @@ def add(user: str, relation: str, object_key: str) -> None: if resource.ownership_kind.upper() != "SYSTEM" or not resource.system_allowlisted: continue resource_object = f"{resource.resource_type}:{resource.resource_id}" - add("user:*", "system_visible_marker", resource_object) + add("user:*", "visible", resource_object) if resource.resource_type in { "knowledge_library", "workflow", diff --git a/src/backend/bisheng/permission/migration/f048_runtime_storage.py b/src/backend/bisheng/permission/migration/f048_runtime_storage.py index 55ed13cf83..abb6c74635 100644 --- a/src/backend/bisheng/permission/migration/f048_runtime_storage.py +++ b/src/backend/bisheng/permission/migration/f048_runtime_storage.py @@ -33,15 +33,24 @@ PermissionMigrationRun, PermissionModel, PermissionModelAction, + PermissionVisibleSourceProjection, ResourcePermissionMode, ) from bisheng.permission.domain.repositories.migration_repository import ( MigrationRepository, ) +from bisheng.permission.domain.services.grant_source_service import ( + GrantModelSnapshot, + GrantSnapshot, + GrantSourceRecord, +) from bisheng.permission.domain.services.model_policy import ( CustomModelSelection, derive_permission_models, ) +from bisheng.permission.domain.services.visibility_projection_service import ( + VisibilityProjectionCompiler, +) from bisheng.permission.migration.f048_coordinator import ( INITIAL_CATALOG_RELEASE_KEY, MigrationRunRequest, @@ -715,6 +724,21 @@ async def acontrol_plane_checksum(self) -> str: .scalars() .all() ) + visible_sources = list( + ( + await session.execute( + select(PermissionVisibleSourceProjection).order_by( + PermissionVisibleSourceProjection.tenant_id, + PermissionVisibleSourceProjection.resource_type, + PermissionVisibleSourceProjection.resource_id, + PermissionVisibleSourceProjection.projected_subject, + PermissionVisibleSourceProjection.contribution_fingerprint, + ) + ) + ) + .scalars() + .all() + ) action_code_by_id = {int(row.id): row.code for row in actions if row.id is not None} model_key_by_id = {int(row.id): row.model_key for row in models if row.id is not None} grant_key_by_id = { @@ -727,6 +751,23 @@ async def acontrol_plane_checksum(self) -> str: for row in grants if row.id is not None } + assignee_key_by_id = { + int(row.id): ( + grant_key_by_id[int(row.grant_id)], + row.source_fingerprint, + ) + for row in assignees + if row.id is not None + } + + def logical_visible_owner(row: PermissionVisibleSourceProjection): + if row.source_kind != "GRANT_ASSIGNEE": + return row.source_owner_key + _, separator, source_id = row.source_owner_key.partition(":") + if not separator or not source_id.isdigit(): + return row.source_owner_key + return ("GRANT_ASSIGNEE", assignee_key_by_id.get(int(source_id))) + payload = { "release": { "release_key": release.release_key, @@ -814,6 +855,24 @@ async def acontrol_plane_checksum(self) -> str: ) for row in modes ], + "visible_sources": [ + ( + row.tenant_id, + row.resource_type, + row.resource_id, + row.visibility_class, + row.projected_subject, + row.source_kind, + logical_visible_owner(row), + row.source_locator, + row.source_fingerprint, + row.model_key, + row.source_version, + row.tuple_fingerprint, + row.state, + ) + for row in visible_sources + ], } return _checksum(payload) @@ -1006,23 +1065,103 @@ async def _upsert_grants(self, grants: tuple[Any, ...]) -> None: .first() ) if existing is None: - session.add( - PermissionGrantAssignee( + existing = PermissionGrantAssignee( + tenant_id=mapped.tenant_id, + grant_id=grant.id, + subject_type=assignee.subject_type, + subject_id=assignee.subject_id, + userset_relation=assignee.userset_relation, + include_children=assignee.include_children, + source_type=assignee.source_type, + source_ref=assignee.source_ref[:256], + source_locator=source_locator, + source_fingerprint=assignee.source_checksum, + projected_subject=projected, + protected=assignee.protected, + state="ACTIVE", + ) + session.add(existing) + await session.flush() + if existing.id is None: + raise RuntimeError("Grant assignee row was not flushed") + + source = GrantSourceRecord( + source_id=int(existing.id), + subject_type=existing.subject_type, + subject_id=existing.subject_id, + userset_relation=existing.userset_relation, + include_children=existing.include_children, + source_type=existing.source_type, + source_ref=existing.source_ref, + source_locator=existing.source_locator, + source_fingerprint=existing.source_fingerprint, + projected_subject=existing.projected_subject, + protected=existing.protected, + active=existing.state == "ACTIVE", + version=existing.version, + ) + visibility = VisibilityProjectionCompiler().compile( + tenant_id=mapped.tenant_id, + grants=( + GrantSnapshot( + grant_id=str(grant.id), tenant_id=mapped.tenant_id, - grant_id=grant.id, - subject_type=assignee.subject_type, - subject_id=assignee.subject_id, - userset_relation=(assignee.userset_relation), - include_children=(assignee.include_children), - source_type=assignee.source_type, - source_ref=assignee.source_ref[:256], - source_locator=source_locator, - source_fingerprint=(assignee.source_checksum), - projected_subject=projected, - protected=assignee.protected, - state="ACTIVE", + resource_type=mapped.resource_type, + resource_id=mapped.resource_id, + model=GrantModelSnapshot( + model_key=mapped.model_key, + active=True, + action_codes=(), + ), + active=True, + sources=(source,), + ), + ), + existing_sources=(), + ) + projection = visibility.active_sources[0] + visible_row = ( + ( + await session.execute( + select(PermissionVisibleSourceProjection).where( + PermissionVisibleSourceProjection.tenant_id == projection.tenant_id, + PermissionVisibleSourceProjection.contribution_fingerprint + == projection.contribution_fingerprint, + ) ) ) + .scalars() + .first() + ) + if visible_row is None: + session.add( + PermissionVisibleSourceProjection( + **projection.model_dump(), + ) + ) + else: + immutable_fields = ( + "resource_type", + "resource_id", + "visibility_class", + "projected_subject", + "source_kind", + "source_owner_key", + "source_locator", + "source_fingerprint", + "contribution_fingerprint", + "model_key", + "tuple_fingerprint", + ) + if any( + getattr(visible_row, field) != getattr(projection, field) + for field in immutable_fields + ): + raise PermissionVersionConflictError( + msg="Visible source differs from migration checkpoint" + ) + visible_row.source_version = projection.source_version + visible_row.state = projection.state async def _upsert_modes(self, modes: tuple[Any, ...]) -> None: with bypass_tenant_filter(): diff --git a/src/backend/bisheng/permission/migration/f048_runtime_verification.py b/src/backend/bisheng/permission/migration/f048_runtime_verification.py index c480ed328c..96ed2732b1 100644 --- a/src/backend/bisheng/permission/migration/f048_runtime_verification.py +++ b/src/backend/bisheng/permission/migration/f048_runtime_verification.py @@ -24,6 +24,7 @@ PermissionGrant, PermissionGrantAssignee, PermissionMigrationItem, + PermissionVisibleSourceProjection, ) from bisheng.permission.domain.repositories.migration_repository import ( MigrationRepository, @@ -124,6 +125,27 @@ async def acollect( expected_tuples=target_tuples, consistency=consistency, ) + source_integrity = await self._visible_source_integrity() + expected_visible = { + _identity(row) for row in target_tuples if row["relation"] == "visible" + } + actual_visible = { + _identity(row) + for row in actual_rows + if row.get("relation") == "visible" + and str(row.get("object", "")).partition(":")[0] + in { + "knowledge_space", + "knowledge_library", + "folder", + "knowledge_file", + "workflow", + "assistant", + "tool", + "channel", + "dashboard", + } + } difference_types = [row.difference_type for row in items if row.difference_type] catalog, model_release = await self._release_rows( run.store_id, @@ -189,6 +211,13 @@ async def acollect( model_release=model_release, ), ), + visible_source_checksum_matches=source_integrity, + visible_aggregate_checksum_matches=(expected_visible == actual_visible), + unattributed_visible_count=len(actual_visible - expected_visible), + visible_stream_complete=semantic_results.get( + "visible_stream_oracle", + False, + ), ) async def _remote_model_checksum(self, model_id: str) -> str | None: @@ -294,10 +323,129 @@ async def _semantic_results( object=f"knowledge_file:{file_row['resource_id']}", consistency=consistency, ) + results.update( + await self._visible_oracle_semantics( + client, + expected_tuples, + consistency, + ) + ) return results finally: await client.close() + @staticmethod + async def _visible_oracle_semantics( + client: FGAClient, + tuples: tuple[dict[str, str], ...], + consistency: str, + ) -> dict[str, bool]: + users = sorted( + { + row["user"] + for row in tuples + if row["user"].startswith("user:") and "#" not in row["user"] + } + ) + objects_by_type: dict[str, list[str]] = {} + for row in tuples: + if row["relation"] != "visible": + continue + resource_type = row["object"].partition(":")[0] + objects_by_type.setdefault(resource_type, []).append(row["object"]) + if not users or not objects_by_type: + return { + "visible_single_batch_oracle": True, + "visible_stream_oracle": True, + } + + single_batch_matches = True + stream_matches = True + for user in users: + for resource_type, objects in sorted(objects_by_type.items()): + checks = [ + { + "user": user, + "relation": "visible", + "object": object_key, + } + for object_key in sorted(set(objects)) + ] + batch_results: list[bool] = [] + for offset in range(0, len(checks), 50): + batch_results.extend( + await client.batch_check( + checks[offset : offset + 50], + consistency=consistency, + ) + ) + single_results = [ + await client.check( + user=user, + relation="visible", + object=check["object"], + consistency=consistency, + ) + for check in checks + ] + single_batch_matches &= single_results == batch_results + expected = { + check["object"] + for check, allowed in zip(checks, batch_results, strict=True) + if allowed + } + try: + streamed = set( + await client.stream_list_objects( + user=user, + relation="visible", + type=resource_type, + consistency=consistency, + ) + ) + except Exception: + stream_matches = False + else: + stream_matches &= (streamed & set(objects)) == expected + return { + "visible_single_batch_oracle": single_batch_matches, + "visible_stream_oracle": stream_matches, + } + + @staticmethod + async def _visible_source_integrity() -> bool: + with bypass_tenant_filter(): + async with get_async_db_session() as session: + assignees = list( + ( + await session.execute( + select(PermissionGrantAssignee) + .join(PermissionGrant, PermissionGrant.id == PermissionGrantAssignee.grant_id) + .where( + PermissionGrant.state == "ACTIVE", + PermissionGrantAssignee.state == "ACTIVE", + ) + ) + ) + .scalars() + .all() + ) + sources = list( + ( + await session.execute( + select(PermissionVisibleSourceProjection).where( + PermissionVisibleSourceProjection.state == "ACTIVE", + PermissionVisibleSourceProjection.source_kind == "GRANT_ASSIGNEE", + ) + ) + ) + .scalars() + .all() + ) + expected = {f"grant_assignee:{row.id}" for row in assignees if row.id is not None} + actual = [row.source_owner_key for row in sources] + return len(actual) == len(set(actual)) and set(actual) == expected + @staticmethod async def _raw_tuple_checks( client: FGAClient, @@ -477,6 +625,4 @@ def _migration_target_pin( store_id=str(model_release.store_id if model_release else ""), model_id=str(model_release.model_id if model_release else ""), catalog_release_id=(int(catalog.id) if catalog and catalog.id else None), - dual_model_mode=False, - legacy_model_id=None, ) diff --git a/src/backend/bisheng/permission/migration/f048_verifier.py b/src/backend/bisheng/permission/migration/f048_verifier.py index e6d77670ee..75efdac2be 100644 --- a/src/backend/bisheng/permission/migration/f048_verifier.py +++ b/src/backend/bisheng/permission/migration/f048_verifier.py @@ -19,8 +19,6 @@ class InstancePinEvidence: store_id: str model_id: str catalog_release_id: int | None - dual_model_mode: bool - legacy_model_id: str | None @dataclass(frozen=True, slots=True) @@ -43,6 +41,10 @@ class MigrationVerificationEvidence: model_checksum_matches: bool semantic_results: Mapping[str, bool] instance_pins: tuple[InstancePinEvidence, ...] + visible_source_checksum_matches: bool = True + visible_aggregate_checksum_matches: bool = True + unattributed_visible_count: int = 0 + visible_stream_complete: bool = True class MigrationVerificationStorePort(Protocol): @@ -90,8 +92,6 @@ def _pin_reasons( not pin.ready or pin.store_id != run.store_id or pin.model_id != run.target_model_id - or pin.dual_model_mode - or pin.legacy_model_id is not None for pin in pins ) ) @@ -121,12 +121,19 @@ def _block_reasons( ("INVALID_OWNER_FACTS", evidence.invalid_owner_count), ("FAILED_TUPLES_REMAIN", evidence.failed_tuple_count), ("LEGACY_TUPLES_REMAIN", evidence.legacy_tuple_count), + ("UNATTRIBUTED_VISIBLE_TUPLES", evidence.unattributed_visible_count), ) reasons.extend(reason for reason, count in count_gates if count) if not evidence.preserved_tuple_checksum_matches: reasons.append("PRESERVED_TUPLE_CHECKSUM_MISMATCH") if not evidence.model_checksum_matches: reasons.append("MODEL_CHECKSUM_MISMATCH") + if not evidence.visible_source_checksum_matches: + reasons.append("VISIBLE_SOURCE_CHECKSUM_MISMATCH") + if not evidence.visible_aggregate_checksum_matches: + reasons.append("VISIBLE_AGGREGATE_CHECKSUM_MISMATCH") + if not evidence.visible_stream_complete: + reasons.append("VISIBLE_STREAM_INCOMPLETE") reasons.extend( f"SEMANTIC_CHECK_FAILED:{name}" for name, passed in sorted(evidence.semantic_results.items()) if not passed ) diff --git a/src/backend/bisheng/workstation/api/endpoints/config.py b/src/backend/bisheng/workstation/api/endpoints/config.py index 32305f2457..711937ca22 100644 --- a/src/backend/bisheng/workstation/api/endpoints/config.py +++ b/src/backend/bisheng/workstation/api/endpoints/config.py @@ -60,24 +60,29 @@ async def get_config(request: Request, login_user=LoginUserDep): ret["subscription"] = {"assistant_name": (sub_assistant_cfg.assistant_name or "") if sub_assistant_cfg else ""} # Sidebar entry names for the knowledge-space / subscription modules; the home and # app-center ones ride along in the daily config dump above. Empty => client i18n default. - ret["knowledge_space"]["menu_display_name"] = ( - (ks_assistant_cfg.menu_display_name or "") if ks_assistant_cfg else "" - ) - ret["subscription"]["menu_display_name"] = ( - (sub_assistant_cfg.menu_display_name or "") if sub_assistant_cfg else "" - ) + ret["knowledge_space"]["menu_display_name"] = (ks_assistant_cfg.menu_display_name or "") if ks_assistant_cfg else "" + ret["subscription"]["menu_display_name"] = (sub_assistant_cfg.menu_display_name or "") if sub_assistant_cfg else "" return resp_200(data=ret) @router.get("/config/daily", summary="Get daily workbench configuration", response_model=UnifiedResponseModel) async def get_daily_config(request: Request, login_user=LoginUserDep): - ret, inherited, source_tenant_id, has_override = await WorkStationService.get_daily_chat_config_with_meta() + ( + ret, + inherited, + source_tenant_id, + has_override, + is_fallback, + ) = await WorkStationService.get_daily_chat_config_with_meta() return resp_200( data={ "data": ret.model_dump(exclude_unset=True) if ret else None, "inherited_from_root": inherited, "source_tenant_id": source_tenant_id, "has_override": has_override, + # Nothing was stored — this payload is the built-in default, not the + # admin's settings. The config page confirms before persisting it. + "is_fallback": is_fallback, } ) diff --git a/src/backend/bisheng/workstation/domain/models/tenant_workstation_config.py b/src/backend/bisheng/workstation/domain/models/tenant_workstation_config.py index 90344c0bcd..ce3dbb23f5 100644 --- a/src/backend/bisheng/workstation/domain/models/tenant_workstation_config.py +++ b/src/backend/bisheng/workstation/domain/models/tenant_workstation_config.py @@ -1,6 +1,5 @@ import logging from datetime import datetime -from typing import Optional, Tuple from sqlalchemy import Column, DateTime, Integer, String, UniqueConstraint, text from sqlmodel import Field, select @@ -17,53 +16,60 @@ class TenantWorkstationConfigBase(SQLModelSerializable): tenant_id: int = Field( sa_column=Column( - Integer, nullable=False, index=True, - comment='Owner tenant; 1=Root, others=Child leaf', + Integer, + nullable=False, + index=True, + comment="Owner tenant; 1=Root, others=Child leaf", ), ) key: str = Field( sa_column=Column( - String(64), nullable=False, index=True, - comment='ConfigKeyEnum value: workstation/workstation_linsight/...', + String(64), + nullable=False, + index=True, + comment="ConfigKeyEnum value: workstation/workstation_linsight/...", ), ) - value: Optional[str] = Field( + value: str | None = Field( default=None, sa_column=Column( - LargeText, nullable=True, - comment='JSON-encoded workstation config payload', + LargeText, + nullable=True, + comment="JSON-encoded workstation config payload", ), ) - create_time: Optional[datetime] = Field( + create_time: datetime | None = Field( default=None, sa_column=Column( - DateTime, nullable=False, - server_default=text('CURRENT_TIMESTAMP'), + DateTime, + nullable=False, + server_default=text("CURRENT_TIMESTAMP"), ), ) - update_time: Optional[datetime] = Field( + update_time: datetime | None = Field( default=None, sa_column=Column( - DateTime, nullable=False, - server_default=text('CURRENT_TIMESTAMP'), - onupdate=text('CURRENT_TIMESTAMP'), + DateTime, + nullable=False, + server_default=text("CURRENT_TIMESTAMP"), + onupdate=text("CURRENT_TIMESTAMP"), ), ) class TenantWorkstationConfig(TenantWorkstationConfigBase, table=True): - __tablename__ = 'tenant_workstation_config' + __tablename__ = "tenant_workstation_config" __table_args__ = ( - UniqueConstraint('tenant_id', 'key', name='uq_tenant_workstation_tenant_key'), + UniqueConstraint("tenant_id", "key", name="uq_tenant_workstation_tenant_key"), {"mysql_charset": "utf8mb4", "mysql_collate": "utf8mb4_unicode_ci"}, ) - id: Optional[int] = Field(default=None, primary_key=True) + id: int | None = Field(default=None, primary_key=True) class TenantWorkstationConfigDao: @classmethod - async def aget(cls, tenant_id: int, key: str) -> Optional[TenantWorkstationConfig]: + async def aget(cls, tenant_id: int, key: str) -> TenantWorkstationConfig | None: async with get_async_db_session() as session: stmt = select(TenantWorkstationConfig).where( TenantWorkstationConfig.tenant_id == tenant_id, @@ -73,7 +79,7 @@ async def aget(cls, tenant_id: int, key: str) -> Optional[TenantWorkstationConfi return result.first() @classmethod - def get(cls, tenant_id: int, key: str) -> Optional[TenantWorkstationConfig]: + def get(cls, tenant_id: int, key: str) -> TenantWorkstationConfig | None: with get_sync_db_session() as session: stmt = select(TenantWorkstationConfig).where( TenantWorkstationConfig.tenant_id == tenant_id, @@ -83,7 +89,10 @@ def get(cls, tenant_id: int, key: str) -> Optional[TenantWorkstationConfig]: @classmethod async def aupsert( - cls, tenant_id: int, key: str, value: Optional[str], + cls, + tenant_id: int, + key: str, + value: str | None, ) -> TenantWorkstationConfig: async with get_async_db_session() as session: stmt = select(TenantWorkstationConfig).where( @@ -105,13 +114,27 @@ async def aupsert( @classmethod async def aresolve( - cls, tenant_id: int, key: str, - ) -> Tuple[Optional[str], bool, int, bool]: + cls, + tenant_id: int, + key: str, + ) -> tuple[str | None, bool, int, bool]: own = await cls.aget(tenant_id, key) if own is not None and own.value: return own.value, False, tenant_id, True if tenant_id == _ROOT_TENANT_ID: + # A filtered-out row and an absent row look identical from here: the + # tenant auto-filter narrows every SELECT by the request's + # visible-tenant set, so Root's own row reads back as "missing" + # whenever that set does not contain Root. Concluding "no config" + # from that is destructive — callers fabricate defaults from it and + # the admin UI persists whatever it was shown, which is how a whole + # workstation config got silently overwritten with defaults on + # 2026-08-13. Re-read unfiltered before deciding. + with bypass_tenant_filter(): + own = await cls.aget(_ROOT_TENANT_ID, key) + if own is not None and own.value: + return own.value, False, _ROOT_TENANT_ID, True return None, False, _ROOT_TENANT_ID, False with bypass_tenant_filter(): @@ -122,13 +145,21 @@ async def aresolve( @classmethod def resolve( - cls, tenant_id: int, key: str, - ) -> Tuple[Optional[str], bool, int, bool]: + cls, + tenant_id: int, + key: str, + ) -> tuple[str | None, bool, int, bool]: own = cls.get(tenant_id, key) if own is not None and own.value: return own.value, False, tenant_id, True if tenant_id == _ROOT_TENANT_ID: + # See ``aresolve``: for Root, "filtered out" is indistinguishable + # from "absent", so re-read unfiltered before reporting no config. + with bypass_tenant_filter(): + own = cls.get(_ROOT_TENANT_ID, key) + if own is not None and own.value: + return own.value, False, _ROOT_TENANT_ID, True return None, False, _ROOT_TENANT_ID, False with bypass_tenant_filter(): diff --git a/src/backend/bisheng/workstation/domain/services/chat_service.py b/src/backend/bisheng/workstation/domain/services/chat_service.py index 77cea4cf5a..222fe293b5 100644 --- a/src/backend/bisheng/workstation/domain/services/chat_service.py +++ b/src/backend/bisheng/workstation/domain/services/chat_service.py @@ -2177,13 +2177,15 @@ async def _task_mode_stream_completion(request: Request, data: APIChatCompletion ) # Enqueue HERE, not from the browser after it receives the handoff below. - # submit_user_question parses every attachment inline, so this request can run - # for minutes on a multi-file task; a user who stops waiting (refresh, closed - # tab, proxy timeout) never sends the follow-up start-execute, and the session - # is stranded at NOT_STARTED with no one to pick it up. Enqueueing server-side - # decouples "the task runs" from "the client is still listening". The client's - # start-execute remains as a late retry and is safe to arrive after this: the - # executor rejects re-entry on an already-running session. + # Attachment parsing no longer happens in this request (submit parks the raw + # refs in pending_files and the worker ingests them), so the request itself is + # fast — but the run must not depend on the browser coming back at all: a user + # who refreshes, closes the tab or hits a proxy timeout never sends the + # follow-up start-execute, and the session is stranded at NOT_STARTED with no + # one to pick it up. Enqueueing server-side decouples "the task runs" from + # "the client is still listening". The client's start-execute remains as a + # late retry and is safe to arrive after this: the executor rejects re-entry + # on an already-running session. from bisheng.linsight.domain import utils as linsight_execute_utils try: diff --git a/src/backend/bisheng/workstation/domain/services/workstation_service.py b/src/backend/bisheng/workstation/domain/services/workstation_service.py index f7c114f1ba..b2ee3b5a0a 100644 --- a/src/backend/bisheng/workstation/domain/services/workstation_service.py +++ b/src/backend/bisheng/workstation/domain/services/workstation_service.py @@ -242,17 +242,50 @@ async def _aupsert_tenant_config(cls, key: ConfigKeyEnum, payload: str) -> None: return await TenantWorkstationConfigDao.aupsert(cls._current_tenant_id(), key.value, payload) + @classmethod + def _read_tool_rows(cls, tool_type_ids: list) -> tuple[dict, dict]: + """Read the parent types + their children for ``tool_type_ids``.""" + tool_type_info = GptsToolsDao.get_all_tool_type(tool_type_ids) + exists_tool_type = {tool.id: tool for tool in tool_type_info} + tool_info = GptsToolsDao.get_list_by_type(list(exists_tool_type.keys())) + return exists_tool_type, {tool.id: tool for tool in tool_info} + @classmethod def sync_tool_info(cls, tools: list[dict]) -> list[dict]: - """Synchronize tool metadata from persistent storage.""" + """Synchronize tool metadata from persistent storage. + + Tools the lookup cannot find are dropped — that is how an admin's + deletion propagates into saved configs. But "not found" is also what a + narrowed visible-tenant IN-list produces for the config owner's own + rows, and the caller cannot tell the two apart: the config page + round-trips what it is shown, so a filtered read would persist as a + cleared tool pool. When anything fails to resolve, re-read pinned to the + tenant that owns the config (``strict_tenant_filter`` narrows to + ``tenant_id = current`` — it never widens) before believing the drop. + """ if not tools: return [] normalized_tools = [cls._to_plain_dict(tool) for tool in tools] tool_type_ids = [tool.get("id") for tool in normalized_tools if tool] - tool_type_info = GptsToolsDao.get_all_tool_type(tool_type_ids) - exists_tool_type = {tool.id: tool for tool in tool_type_info} - tool_info = GptsToolsDao.get_list_by_type(list(exists_tool_type.keys())) - exists_tool_info = {tool.id: tool for tool in tool_info} + exists_tool_type, exists_tool_info = cls._read_tool_rows(tool_type_ids) + + missing = [tid for tid in tool_type_ids if tid not in exists_tool_type] + if missing: + with strict_tenant_filter(): + strict_type, strict_info = cls._read_tool_rows(tool_type_ids) + # Keep whichever read resolved more; a genuine deletion resolves + # neither way and still drops out below. + if len(strict_type) > len(exists_tool_type): + logger.warning( + "workstation config tool sync: {} of {} tool group(s) unresolved under the request's " + "tenant filter, recovered {} with a strict re-read (tenant={})", + len(missing), + len(tool_type_ids), + len(strict_type) - len(exists_tool_type), + cls._current_tenant_id(), + ) + exists_tool_type, exists_tool_info = strict_type, strict_info + new_tools = [] for tool in normalized_tools: if not tool: @@ -691,18 +724,36 @@ async def update_daily_chat_config(cls, data: WorkstationConfig) -> WorkstationC return await cls.get_daily_chat_config() @classmethod - async def get_daily_chat_config_with_meta(cls) -> tuple[WorkstationConfig | None, bool, int, bool]: + async def get_daily_chat_config_with_meta(cls) -> tuple[WorkstationConfig | None, bool, int, bool, bool]: + """Resolve the daily config plus its provenance. + + The fifth element is ``is_fallback``: True when nothing was stored for + this tenant (or Root) and the returned config is the built-in default + rather than anything an admin saved. Callers that write the config back + must not treat a fallback as "the admin's current settings" — the admin + UI round-trips what it is given, so persisting a fallback silently + replaces a real config with defaults. + """ value, inherited, source_tenant_id, has_override = await cls._aresolve_tenant_config(ConfigKeyEnum.WORKSTATION) config = type("TenantConfigValue", (), {"value": value}) if value else None ret = cls.parse_config(config) + is_fallback = ret is None if ret is None: + logger.warning( + "daily workstation config falling back to built-in defaults: tenant={} source_tenant={} " + "inherited={} has_override={} (no stored value resolved)", + cls._current_tenant_id(), + source_tenant_id, + inherited, + has_override, + ) ret = await cls._abuild_default_daily_config() if ret and not inherited: ret.tools = cls.sync_tool_info(ret.tools) if inherited: ret = await cls._aproject_daily_config_for_current_tenant(ret, source_tenant_id) ret = cls._apply_workbench_models(ret, await LLMService.get_workbench_llm()) - return ret, inherited, source_tenant_id, has_override + return ret, inherited, source_tenant_id, has_override, is_fallback @classmethod async def get_linsight_config(cls) -> LinsightConfig | None: diff --git a/src/backend/scripts/README.md b/src/backend/scripts/README.md index cad94a9dad..b8546bf5ce 100644 --- a/src/backend/scripts/README.md +++ b/src/backend/scripts/README.md @@ -31,6 +31,52 @@ Options: ## Permission Scripts +### `reconcile_f048_visible_projection.py` + +Audit and repair environments that already completed an older F048 data +migration before the final flattened-visible design. The command is available +in production as well as development/test; safety comes from the same +maintenance and consistency gates, not from an environment-name allowlist. + +Run dry-run first from `src/backend/` with the live `config`: + +```bash +export config=config.yaml +PYTHONPATH=./ .venv/bin/python scripts/reconcile_f048_visible_projection.py +``` + +The JSON report includes canonical Grant/assignee source counts, persisted +source differences, and the deduplicated expected tuple count/checksum. The +script never scans or deletes existing visible tuples because system/public/ +shared visibility is not owned by the Grant source projection. + +For apply, stop ingress traffic and all API/Worker/Linsight processes, wait for +their F048 heartbeat TTL to expire, and copy the dry-run `store_id` into the +explicit confirmation: + +```bash +PYTHONPATH=./ .venv/bin/python scripts/reconcile_f048_visible_projection.py \ + --apply \ + --confirm-store-id \ + --operator-id \ + --allow-model-upgrade +``` + +`--allow-model-upgrade` is required only when dry-run reports that CURRENT +still points at an older F048 model. Apply exits with code `3` before writing +when Store confirmation differs, a runtime heartbeat or projection operation +is active, the CURRENT Catalog has an unrelated/non-resumable fence, canonical +SQL data is incomplete, or stale source projections would require a classified +revocation. It publishes/reuses the final immutable model in the same Store, +ensures every Grant-derived direct `visible` tuple in batches of at most 90 +with OpenFGA duplicate-ignore semantics, verifies them with higher consistency, +then activates the rebuilt Grant source rows and publishes a no-op Catalog +release bound to the new Authorization Model release. It does not create or modify a formal +`permission_migration_run`. Re-running after an interruption is forward-only +and idempotent. Restart all permission-using processes after success; they +discover the latest model through the stable Store name and validate the new +SQL CURRENT Catalog pin. + ### `reconcile_f048_projection_operations.py` Inspect and recover explicitly selected F048 permission projection ledger @@ -123,9 +169,11 @@ that needs an OpenFGA projection. PYTHONPATH=./ .venv/bin/alembic heads ``` - Both commands must identify `f048_migration_item_message_longtext` as the - head. This follow-up DDL widens the frozen source payload column before the - data script records large legacy Config values. + Both commands must identify `f048_visible_source_projection` as the head. + Its predecessor `f048_migration_item_message_longtext` widens the frozen + source payload column before the data script records large legacy Config + values; the head then creates the single-slot visible source projection + table required by migration and D4 verification. 4. Do not proceed if Redis is unavailable, a ready F048 runtime heartbeat remains, dashboard tenant attribution is ambiguous, an unresolved failed tuple exists, or the source watermark changes between scans. @@ -171,21 +219,23 @@ completed batch is idempotent. After migrate reports `VERIFYING`, leave the automatic migration gate in place and do not restart the processes yet. Do not query Store/model/Catalog tables -for values to copy into configuration. Ensure -`force_write_model=false`, `dual_model_mode=false`, and leave -`legacy_model_id` empty, then run: +for values to copy into configuration. The migration always reuses the durable +Store, source model and target model pinned by the original run; operators +cannot replace them with command-line parameters. Then run: ```bash PYTHONPATH=./ .venv/bin/python scripts/migrate_f048_permission_data.py \ verify --run-id ``` -Verification independently rebuilds source and target checksums, requires exact -target tuple counts, checks high-risk dashboard/download semantics, preserves -allowed Store facts, requires the run target release to be the one referenced -by the SQL CURRENT Catalog, and requires legacy tuple and blocker counts to be -zero. The report still records the retained legacy Config count for audit, but -that count does not block verification. Success moves the run to +Verification independently rebuilds source and target checksums, requires one +traceable visible contribution for every legal assignee, validates the +deduplicated direct `visible` aggregate and complete streamed enumeration, +checks high-risk dashboard/download semantics, preserves allowed Store facts, +requires the run target release to be the one referenced by the SQL CURRENT +Catalog, and requires unattributed visible tuples, legacy tuples and blockers +to be zero. The report still records the retained legacy Config count for +audit, but that count does not block verification. Success moves the run to `READY_TO_START`; restart API and Worker processes so they discover the new F048 model and automatically remove the migration gate. @@ -197,8 +247,9 @@ Store name or missing Store/model still fails startup. A predecessor checksum keeps the process alive but not ready for the explicit migration; after the post-migration restart, any model/Catalog mismatch fails readiness. -There is no preview, dry-run, cleanup, rollback, Store switch, dual-model -window, or automatic startup migration. A failure keeps maintenance active and +There is no preview, dry-run, cleanup, rollback, Store replacement, +intermediate permission model, relation-slot switch, second migration, or +automatic API/Celery startup migration. A failure keeps maintenance active and is repaired only by a forward fix against the same run and target model. Exit codes: diff --git a/src/backend/scripts/benchmark_f048_permission_paths.py b/src/backend/scripts/benchmark_f048_permission_paths.py index 7a1798f3f9..4a889f2d5c 100644 --- a/src/backend/scripts/benchmark_f048_permission_paths.py +++ b/src/backend/scripts/benchmark_f048_permission_paths.py @@ -1,24 +1,11 @@ -"""Prepare and run the F048 BENCH-01 permission performance contract. - -The script only accepts a non-production environment and an explicitly pinned -OpenFGA Store. It never creates or switches a Store, never runs the F048 data -migration, and never maintains an old/new authorization-model pair. Historical -latency samples are read from the signed fixture; every live request uses the -single F048 model supplied by ``--model-id``. - -Examples (run from ``src/backend``): - - PYTHONPATH=./ .venv/bin/python scripts/benchmark_f048_permission_paths.py \ - prepare --environment performance --api-url http://127.0.0.1:8080 \ - --store-id --apply - - PYTHONPATH=./ .venv/bin/python scripts/benchmark_f048_permission_paths.py \ - run --environment performance --api-url http://127.0.0.1:8080 \ - --store-id --model-id \ - --openfga-log /path/to/openfga-benchmark.jsonl --output report.json - -The bundled fixture is synthetic and can only exercise the harness. A formal -release run must supply a checksum-pinned, production-derived sanitized fixture. +#!/usr/bin/env python3 +"""Run the F048 single-slot visibility BENCH-01 contract. + +The harness writes only to an explicitly supplied non-production OpenFGA +Store. The compact fixture describes database cardinality independently from +the projected visibility tuples so 10k/100k business resource populations do +not require fake invisible OpenFGA objects. Every enumeration consumes +StreamedListObjects to a normal EOF before it records a successful sample. """ from __future__ import annotations @@ -28,14 +15,15 @@ import json import math import os +import re import sys from collections import defaultdict from collections.abc import Iterable -from dataclasses import dataclass +from dataclasses import asdict, dataclass from hashlib import sha256 from pathlib import Path from time import perf_counter -from typing import Any +from typing import Any, Protocol from uuid import uuid4 import httpx @@ -52,12 +40,22 @@ OPENFGA_VERSION = "1.15.1" WRITE_BATCH_SIZE = 90 BATCH_CHECK_SIZES = (20, 50, 100) -CONTRACT_VERSION = "f048-bench-01-v1" +OPENFGA_BATCH_CHECK_LIMIT = 50 +RESOURCE_SCALES = (10_000, 100_000) +VISIBLE_RESULT_SIZES = (10, 100, 1_000, 5_000) +SOURCE_KINDS = frozenset({"direct", "department", "group", "system", "multi_source"}) +CONTRACT_VERSION = "f048-bench-01-v2" EXIT_OK = 0 EXIT_INVALID_CONTRACT = 2 EXIT_GATE_FAILED = 3 EXIT_RUNTIME_ERROR = 4 -DEFAULT_FIXTURE = Path(_BACKEND_ROOT) / "test" / "permission" / "fixtures" / "f048_bench_contract.synthetic.json" +DEFAULT_FIXTURE = ( + Path(_BACKEND_ROOT) + / "test" + / "permission" + / "fixtures" + / "f048_bench_contract.synthetic.json" +) _PRODUCTION_ENVIRONMENTS = { "prod", "production", @@ -66,10 +64,11 @@ "正式", "生产", } +_AB_RELATION_PATTERN = re.compile(r"(?:^|_)(?:slot_[ab]|visible_[ab]|visibility_switch)(?:$|_)") class BenchmarkContractError(ValueError): - """The fixture, environment, or runtime pin is not safe to execute.""" + """The fixture, environment, or runtime pin is unsafe or incomplete.""" def canonical_json(value: Any) -> bytes: @@ -123,113 +122,77 @@ def summarize(values: Iterable[float]) -> dict[str, float | int]: def object_keys(scenario: dict[str, Any]) -> list[str]: resource_type = str(scenario["resource_type"]) prefix = str(scenario["resource_id_prefix"]) - return [f"{resource_type}:{prefix}{index:04d}" for index in range(1, int(scenario["result_count"]) + 1)] + return [ + f"{resource_type}:{prefix}{index:05d}" + for index in range(1, int(scenario["visible_count"]) + 1) + ] def object_set_checksum(scenario: dict[str, Any]) -> str: return checksum(sorted(object_keys(scenario))) -def _base_release_tuples(profile: dict[str, Any]) -> list[dict[str, str]]: - catalog = f"permission_catalog_release:{profile['catalog_release_id']}" - release = f"permission_model_release:{profile['model_release_id']}" - model = f"permission_model:{profile['permission_model_id']}" - actions = sorted({str(item["action"]) for item in profile["scenarios"]}) - tuples = [ - {"user": "user:*", "relation": "active", "object": catalog}, - {"user": catalog, "relation": "catalog", "object": release}, - {"user": "user:*", "relation": "enabled_marker", "object": release}, - {"user": release, "relation": "release", "object": model}, - ] - tuples.extend({"user": "user:*", "relation": f"{action}_marker", "object": release} for action in actions) - return tuples - - -def _grant_tuples( - *, - profile: dict[str, Any], - scenario: dict[str, Any], - resource: str, - actor: str, - suffix: str = "", -) -> list[dict[str, str]]: - model = f"permission_model:{profile['permission_model_id']}" - grant_id = resource.replace(":", "-") + suffix - grant = f"permission_grant:{grant_id}" - subject_kind = str(scenario["subject_kind"]) - tuples = [ - {"user": model, "relation": "model", "object": grant}, - {"user": grant, "relation": "grant", "object": resource}, - ] - if subject_kind == "department": - department = f"department:bench-{scenario['name']}" - tuples.extend( - ( - {"user": actor, "relation": "member", "object": department}, +def _source_tuples(scenario: dict[str, Any]) -> list[dict[str, str]]: + actor = f"user:{scenario['actor_id']}" + name = str(scenario["name"]) + source_kind = str(scenario["source_kind"]) + tuples: list[dict[str, str]] = [] + subject = actor + if source_kind in {"department", "multi_source"}: + department = f"department:bench-{name}" + tuples.append({"user": actor, "relation": "member", "object": department}) + if source_kind == "department": + subject = f"{department}#member" + if source_kind in {"group", "multi_source"}: + group = f"user_group:bench-{name}" + tuples.append({"user": actor, "relation": "member", "object": group}) + if source_kind == "group": + subject = f"{group}#member" + + for resource in object_keys(scenario): + if source_kind == "system": + tuples.append( { - "user": f"{department}#member", - "relation": "ordinary_assignee", - "object": grant, - }, - ) - ) - elif subject_kind == "group": - group = f"user_group:bench-{scenario['name']}" - tuples.extend( - ( - {"user": actor, "relation": "member", "object": group}, - { - "user": f"{group}#member", - "relation": "ordinary_assignee", - "object": grant, - }, - ) - ) - else: - tuples.append( - { - "user": actor, - "relation": "ordinary_assignee", - "object": grant, - } - ) - return tuples - - -def build_dataset_tuples(contract: dict[str, Any]) -> list[dict[str, str]]: - """Expand the compact fixture into deterministic OpenFGA tuple keys.""" - - profile = contract["dataset"]["profile"] - tuples = _base_release_tuples(profile) - for scenario in profile["scenarios"]: - actor = f"user:{scenario['actor_id']}" - resources = object_keys(scenario) - if scenario["subject_kind"] == "inherit": - parent = f"knowledge_space:bench-parent-{scenario['name']}" - parent_scenario = dict(scenario, subject_kind="direct") - tuples.extend( - _grant_tuples( - profile=profile, - scenario=parent_scenario, - resource=parent, - actor=actor, - ) + "user": "user:*", + "relation": "visible", + "object": resource, + } ) + elif source_kind == "multi_source": tuples.extend( ( + {"user": actor, "relation": "visible", "object": resource}, { - "user": "user:*", - "relation": "permission_enabled", - "object": parent, + "user": f"department:bench-{name}#member", + "relation": "visible", + "object": resource, }, { - "user": "user:*", - "relation": "custom_mode", - "object": parent, + "user": f"user_group:bench-{name}#member", + "relation": "visible", + "object": resource, }, ) ) - for resource in resources: + else: + tuples.append( + { + "user": subject, + "relation": "visible", + "object": resource, + } + ) + return tuples + + +def build_dataset_tuples(contract: dict[str, Any]) -> list[dict[str, str]]: + """Expand canonical shallow visibility sources into deterministic tuples.""" + + tuples: list[dict[str, str]] = [] + for scenario in contract["dataset"]["scenarios"]: + tuples.extend(_source_tuples(scenario)) + if scenario["source_kind"] != "system": + for resource in object_keys(scenario): tuples.extend( ( { @@ -239,55 +202,94 @@ def build_dataset_tuples(contract: dict[str, Any]) -> list[dict[str, str]]: }, { "user": "user:*", - "relation": "inherit_mode", + "relation": "custom_mode", "object": resource, }, - {"user": parent, "relation": "parent", "object": resource}, ) ) - continue - - for resource in resources: - tuples.extend( - _grant_tuples( - profile=profile, - scenario=scenario, - resource=resource, - actor=actor, - ) - ) - if scenario["subject_kind"] == "multi_grant": - tuples.extend( - _grant_tuples( - profile=profile, - scenario=dict(scenario, subject_kind="direct"), - resource=resource, - actor=actor, - suffix="-second", - ) - ) - tuples.extend( - ( + else: + for resource in object_keys(scenario): + tuples.append( { "user": "user:*", "relation": "permission_enabled", "object": resource, - }, - { - "user": "user:*", - "relation": "custom_mode", - "object": resource, - }, + } ) - ) - deduplicated = {(item["user"], item["relation"], item["object"]): item for item in tuples} + deduplicated = { + (item["user"], item["relation"], item["object"]): item for item in tuples + } return [deduplicated[key] for key in sorted(deduplicated)] +def source_checksum(contract: dict[str, Any]) -> str: + sources = [ + item + for item in build_dataset_tuples(contract) + if item["relation"] not in {"permission_enabled", "custom_mode"} + ] + return checksum(sources) + + +def visible_checksum(contract: dict[str, Any]) -> str: + return checksum( + { + str(item["name"]): object_set_checksum(item) + for item in contract["dataset"]["scenarios"] + } + ) + + def dataset_checksum(contract: dict[str, Any]) -> str: return checksum(build_dataset_tuples(contract)) +def model_has_ab_slots(model: dict[str, Any]) -> bool: + for definition in model["type_definitions"]: + for relation in definition.get("relations", {}): + if _AB_RELATION_PATTERN.search(relation): + return True + return False + + +def _validate_distribution(contract: dict[str, Any]) -> None: + scenarios = contract["dataset"]["scenarios"] + pairs = { + (int(item["resource_count"]), int(item["visible_count"])) + for item in scenarios + } + required_pairs = { + (resource_count, visible_count) + for resource_count in RESOURCE_SCALES + for visible_count in VISIBLE_RESULT_SIZES + } + if not required_pairs.issubset(pairs): + raise BenchmarkContractError( + f"fixture lacks N_db/V scenarios: {sorted(required_pairs - pairs)}" + ) + kinds = {str(item["source_kind"]) for item in scenarios} + if not SOURCE_KINDS.issubset(kinds): + raise BenchmarkContractError( + f"fixture lacks source kinds: {sorted(SOURCE_KINDS - kinds)}" + ) + if len({str(item["actor_id"]) for item in scenarios}) != len(scenarios): + raise BenchmarkContractError("benchmark scenario actors must be unique") + system_types = { + str(item["resource_type"]) + for item in scenarios + if item["source_kind"] == "system" + } + other_types = { + str(item["resource_type"]) + for item in scenarios + if item["source_kind"] != "system" + } + if system_types.intersection(other_types): + raise BenchmarkContractError( + "system wildcard scenarios must use isolated resource types" + ) + + def load_contract( path: Path, *, @@ -302,35 +304,43 @@ def load_contract( embedded = str(contract.get("contract_checksum", "")) calculated = contract_checksum(contract) if embedded != calculated: - raise BenchmarkContractError(f"fixture checksum mismatch: expected {embedded}, calculated {calculated}") + raise BenchmarkContractError( + f"fixture checksum mismatch: expected {embedded}, calculated {calculated}" + ) if expected_checksum and expected_checksum != calculated: - raise BenchmarkContractError("fixture does not match --expected-contract-checksum") - model_checksum = authorization_model_checksum(build_authorization_model_f048()) - if contract.get("authorization_model_checksum") != model_checksum: + raise BenchmarkContractError( + "fixture does not match --expected-contract-checksum" + ) + model = build_authorization_model_f048() + if model_has_ab_slots(model): + raise BenchmarkContractError("target model contains an A/B visibility relation") + if contract.get("authorization_model_checksum") != authorization_model_checksum(model): raise BenchmarkContractError("fixture authorization model checksum drift") + _validate_distribution(contract) + for scenario in contract["dataset"]["scenarios"]: + if int(scenario["visible_count"]) > int(scenario["resource_count"]): + raise BenchmarkContractError( + f"visible_count exceeds resource_count for {scenario['name']}" + ) + if scenario.get("expected_object_checksum") != object_set_checksum(scenario): + raise BenchmarkContractError( + f"object checksum drift for scenario {scenario['name']}" + ) calculated_dataset = dataset_checksum(contract) if contract["dataset"].get("dataset_checksum") != calculated_dataset: raise BenchmarkContractError("fixture expanded dataset checksum drift") - scenarios = contract["dataset"]["profile"]["scenarios"] - names = {str(item["name"]) for item in scenarios} - required = { - "direct", - "department", - "group", - "inherit", - "multi_grant", - "result_10", - "result_100", - "result_1000", - } - if not required.issubset(names): - raise BenchmarkContractError(f"fixture lacks required scenarios: {sorted(required - names)}") - for scenario in scenarios: - if scenario.get("expected_object_checksum") != object_set_checksum(scenario): - raise BenchmarkContractError(f"object checksum drift for scenario {scenario['name']}") - baseline_sizes = {int(value) for value in contract["baseline"]["batch_check_ms"]} - if baseline_sizes != set(BATCH_CHECK_SIZES): - raise BenchmarkContractError("baseline must contain BatchCheck 20/50/100") + if contract["dataset"].get("source_checksum") != source_checksum(contract): + raise BenchmarkContractError("fixture source checksum drift") + if contract["dataset"].get("visible_checksum") != visible_checksum(contract): + raise BenchmarkContractError("fixture visible checksum drift") + if {int(value) for value in contract["limits"]["batch_check_p95_ms"]} != set( + BATCH_CHECK_SIZES + ): + raise BenchmarkContractError("limits must contain BatchCheck 20/50/100") + if {int(value) for value in contract["limits"]["stream_p95_ms"]} != set( + VISIBLE_RESULT_SIZES + ): + raise BenchmarkContractError("limits must contain visible 10/100/1000/5000") return contract @@ -343,10 +353,61 @@ class RequestSample: error: str | None = None result_count: int | None = None result_checksum: str | None = None + stream_completed: bool | None = None + strategy: str | None = None + n_db: int | None = None + visible_total: int | None = None + selectivity: float | None = None + candidate_pass_rate: float | None = None + db_rows: int | None = None + scanned_count: int | None = None + scan_amplification: float | None = None + + +def strategy_metrics( + *, + n_db: int, + visible_total: int, + page_size: int, + scanned_count: int, +) -> dict[str, float | int]: + if n_db <= 0 or visible_total < 0 or page_size <= 0 or scanned_count < 0: + raise BenchmarkContractError("invalid list strategy cardinality") + selectivity = visible_total / n_db + returned = min(page_size, visible_total) + amplification = scanned_count / max(1, returned) + return { + "n_db": n_db, + "visible_total": visible_total, + "selectivity": round(selectivity, 8), + "scanned_count": scanned_count, + "scan_amplification": round(amplification, 8), + } + + +class BenchmarkClient(Protocol): + async def health(self) -> None: ... + + async def verify_model(self, expected_checksum: str) -> None: ... + + async def check(self, query: dict[str, str]) -> tuple[bool, str]: ... + + async def batch_check( + self, + queries: list[dict[str, str]], + ) -> tuple[list[bool], str]: ... + + async def stream_list_objects( + self, + *, + user: str, + relation: str, + resource_type: str, + ) -> tuple[list[str], str, bool]: ... class InstrumentedOpenFGAClient: - """Minimal benchmark-only REST client with request-id correlation.""" + """Minimal v1.15.1 REST client with request-id correlation.""" def __init__( self, @@ -373,7 +434,9 @@ async def health(self) -> None: raise BenchmarkContractError("OpenFGA health status is not SERVING") async def verify_model(self, expected_checksum: str) -> None: - response = await self._http.get(f"/stores/{self.store_id}/authorization-models/{self.model_id}") + response = await self._http.get( + f"/stores/{self.store_id}/authorization-models/{self.model_id}" + ) response.raise_for_status() value = response.json().get("authorization_model", response.json()) model = { @@ -383,7 +446,11 @@ async def verify_model(self, expected_checksum: str) -> None: if authorization_model_checksum(model) != expected_checksum: raise BenchmarkContractError("live OpenFGA model checksum mismatch") - async def _post(self, path: str, body: dict[str, Any]) -> tuple[dict, str]: + async def _post( + self, + path: str, + body: dict[str, Any], + ) -> tuple[dict[str, Any], str]: request_id = f"f048-bench-{uuid4().hex}" response = await self._http.post( path, @@ -391,7 +458,7 @@ async def _post(self, path: str, body: dict[str, Any]) -> tuple[dict, str]: headers={"X-Request-ID": request_id}, ) response.raise_for_status() - return response.json(), request_id + return response.json(), response.headers.get("x-request-id", request_id) async def check(self, query: dict[str, str]) -> tuple[bool, str]: data, request_id = await self._post( @@ -399,6 +466,7 @@ async def check(self, query: dict[str, str]) -> tuple[bool, str]: { "tuple_key": query, "authorization_model_id": self.model_id, + "consistency": "HIGHER_CONSISTENCY", }, ) return bool(data.get("allowed")), request_id @@ -411,29 +479,56 @@ async def batch_check( f"/stores/{self.store_id}/batch-check", { "authorization_model_id": self.model_id, - "checks": [{"tuple_key": query, "correlation_id": str(index)} for index, query in enumerate(queries)], + "consistency": "HIGHER_CONSISTENCY", + "checks": [ + {"tuple_key": query, "correlation_id": str(index)} + for index, query in enumerate(queries) + ], }, ) result = data.get("result", {}) - return [bool(result.get(str(index), {}).get("allowed")) for index in range(len(queries))], request_id + return [ + bool(result.get(str(index), {}).get("allowed")) + for index in range(len(queries)) + ], request_id - async def list_objects( + async def stream_list_objects( self, *, user: str, relation: str, resource_type: str, - ) -> tuple[list[str], str]: - data, request_id = await self._post( - f"/stores/{self.store_id}/list-objects", - { + ) -> tuple[list[str], str, bool]: + request_id = f"f048-bench-{uuid4().hex}" + objects: list[str] = [] + async with self._http.stream( + "POST", + f"/stores/{self.store_id}/streamed-list-objects", + json={ "user": user, "relation": relation, "type": resource_type, "authorization_model_id": self.model_id, + "consistency": "HIGHER_CONSISTENCY", }, - ) - return list(data.get("objects", ())), request_id + headers={ + "X-Request-ID": request_id, + "Accept": "application/x-ndjson", + }, + ) as response: + response.raise_for_status() + request_id = response.headers.get("x-request-id", request_id) + async for line in response.aiter_lines(): + if not line.strip(): + continue + payload = json.loads(line) + object_key = payload.get("result", {}).get("object") + if not isinstance(object_key, str): + raise BenchmarkContractError( + "StreamedListObjects returned an invalid item" + ) + objects.append(object_key) + return objects, request_id, True class DatasetPreparer: @@ -459,12 +554,13 @@ async def prepare(self, contract: dict[str, Any]) -> dict[str, Any]: model_id = response.json()["authorization_model_id"] tuples = build_dataset_tuples(contract) for offset in range(0, len(tuples), WRITE_BATCH_SIZE): - batch = tuples[offset : offset + WRITE_BATCH_SIZE] write = await self._http.post( f"/stores/{self.store_id}/write", json={ "authorization_model_id": model_id, - "writes": {"tuple_keys": batch}, + "writes": { + "tuple_keys": tuples[offset : offset + WRITE_BATCH_SIZE] + }, }, ) write.raise_for_status() @@ -473,40 +569,56 @@ async def prepare(self, contract: dict[str, Any]) -> dict[str, Any]: "model_id": model_id, "authorization_model_checksum": authorization_model_checksum(model), "dataset_checksum": dataset_checksum(contract), + "source_checksum": source_checksum(contract), + "visible_checksum": visible_checksum(contract), "tuple_count": len(tuples), } def _scenario_map(contract: dict[str, Any]) -> dict[str, dict[str, Any]]: - return {str(item["name"]): item for item in contract["dataset"]["profile"]["scenarios"]} + return { + str(item["name"]): item for item in contract["dataset"]["scenarios"] + } def _query(scenario: dict[str, Any], object_key: str) -> dict[str, str]: return { "user": f"user:{scenario['actor_id']}", - "relation": f"can_{scenario['action']}", + "relation": "visible", "object": object_key, } -async def _timed_call( +def _invisible_object(scenario: dict[str, Any], index: int) -> str: + return ( + f"{scenario['resource_type']}:{scenario['resource_id_prefix']}" + f"invisible-{index:05d}" + ) + + +async def _timed_value( *, operation: str, scenario: str, - call, - validate, + call: Any, + validate: Any, ) -> RequestSample: started = perf_counter() try: result, request_id = await call() validate(result) + request_ids = ( + (request_id,) + if isinstance(request_id, str) + else tuple(request_id) + ) return RequestSample( operation=operation, scenario=scenario, elapsed_ms=(perf_counter() - started) * 1000, - request_ids=(request_id,), - result_count=(len(result) if isinstance(result, list) else None), - result_checksum=(checksum(sorted(result)) if isinstance(result, list) else None), + request_ids=request_ids, + result_count=len(result) if isinstance(result, list) else None, + result_checksum=checksum(sorted(result)) if isinstance(result, list) else None, ) except Exception as exc: return RequestSample( @@ -518,8 +630,164 @@ async def _timed_call( ) +async def _bounded_batch_check( + client: BenchmarkClient, + queries: list[dict[str, str]], +) -> tuple[list[bool], tuple[str, ...]]: + allowed: list[bool] = [] + request_ids: list[str] = [] + for offset in range(0, len(queries), OPENFGA_BATCH_CHECK_LIMIT): + chunk_allowed, request_id = await client.batch_check( + queries[offset : offset + OPENFGA_BATCH_CHECK_LIMIT] + ) + allowed.extend(chunk_allowed) + request_ids.append(request_id) + return allowed, tuple(request_ids) + + +async def _stream_sample( + client: BenchmarkClient, + scenario: dict[str, Any], + *, + operation: str = "stream_list_objects", +) -> RequestSample: + started = perf_counter() + try: + objects, request_id, completed = await client.stream_list_objects( + user=f"user:{scenario['actor_id']}", + relation="visible", + resource_type=str(scenario["resource_type"]), + ) + expected = object_keys(scenario) + if not completed or sorted(set(objects)) != sorted(expected): + raise BenchmarkContractError( + "StreamedListObjects result set mismatch, truncation, or abnormal EOF" + ) + return RequestSample( + operation=operation, + scenario=str(scenario["name"]), + elapsed_ms=(perf_counter() - started) * 1000, + request_ids=(request_id,), + result_count=len(set(objects)), + result_checksum=checksum(sorted(set(objects))), + stream_completed=True, + ) + except Exception as exc: + return RequestSample( + operation=operation, + scenario=str(scenario["name"]), + elapsed_ms=(perf_counter() - started) * 1000, + request_ids=(), + error=f"{type(exc).__name__}: {exc}", + stream_completed=False, + ) + + +async def _candidate_chain_sample( + client: BenchmarkClient, + *, + name: str, + scenario: dict[str, Any], + n_db: int, + visible_total: int, + selectivity: float, + page_size: int, + batch_size: int, +) -> RequestSample: + started = perf_counter() + request_ids: list[str] = [] + visible: list[str] = [] + scanned = 0 + error: str | None = None + expected_visible = object_keys(scenario) + if not 0 < selectivity <= 1: + raise BenchmarkContractError("business path selectivity must be in (0, 1]") + try: + visible_index = 0 + invisible_index = 0 + while len(visible) < min(page_size, visible_total) and scanned < n_db: + candidates: list[str] = [] + for _ in range(min(batch_size, n_db - scanned)): + position = scanned + len(candidates) + should_be_visible = ( + visible_index < visible_total + and position % max(1, round(1 / selectivity)) == 0 + ) + if should_be_visible: + candidates.append(expected_visible[visible_index]) + visible_index += 1 + else: + invisible_index += 1 + candidates.append(_invisible_object(scenario, invisible_index)) + allowed, chunk_request_ids = await _bounded_batch_check( + client, + [_query(scenario, item) for item in candidates] + ) + request_ids.extend(chunk_request_ids) + scanned += len(candidates) + visible.extend( + item + for item, is_allowed in zip(candidates, allowed, strict=True) + if is_allowed + ) + visible = visible[:page_size] + if len(visible) != min(page_size, visible_total): + raise BenchmarkContractError("candidate-first path did not fill the page") + except Exception as exc: + error = f"{type(exc).__name__}: {exc}" + metrics = strategy_metrics( + n_db=n_db, + visible_total=visible_total, + page_size=page_size, + scanned_count=scanned, + ) + return RequestSample( + operation="business_path", + scenario=name, + elapsed_ms=(perf_counter() - started) * 1000, + request_ids=tuple(request_ids), + error=error, + result_count=len(visible), + result_checksum=checksum(visible), + strategy="candidate_first", + candidate_pass_rate=selectivity, + db_rows=scanned, + **metrics, + ) + + +async def _id_first_chain_sample( + client: BenchmarkClient, + *, + name: str, + scenario: dict[str, Any], + page_size: int, +) -> RequestSample: + sample = await _stream_sample(client, scenario, operation="business_path") + visible_total = int(scenario["visible_count"]) + n_db = int(scenario["resource_count"]) + db_rows = visible_total + metrics = strategy_metrics( + n_db=n_db, + visible_total=visible_total, + page_size=page_size, + scanned_count=db_rows, + ) + sample.scenario = name + sample.strategy = "visible_id_first" + sample.result_count = min(page_size, visible_total) + sample.db_rows = db_rows + sample.n_db = int(metrics["n_db"]) + sample.visible_total = int(metrics["visible_total"]) + sample.selectivity = float(metrics["selectivity"]) + sample.candidate_pass_rate = 1.0 + sample.scanned_count = int(metrics["scanned_count"]) + sample.scan_amplification = float(metrics["scan_amplification"]) + return sample + + async def run_workloads( - client: InstrumentedOpenFGAClient, + client: BenchmarkClient, contract: dict[str, Any], *, iterations: int, @@ -531,85 +799,86 @@ async def run_workloads( await client.verify_model(contract["authorization_model_checksum"]) async def run_once(record: bool) -> None: - for name, scenario in scenarios.items(): + for scenario in scenarios.values(): first = object_keys(scenario)[0] - sample = await _timed_call( + sample = await _timed_value( operation="check", - scenario=name, + scenario=str(scenario["name"]), call=lambda s=scenario, o=first: client.check(_query(s, o)), validate=lambda allowed: ( - None if allowed else (_ for _ in ()).throw(BenchmarkContractError("expected Check ALLOW")) + None + if allowed + else (_ for _ in ()).throw( + BenchmarkContractError("expected visible Check ALLOW") + ) ), ) if record: samples.append(sample) - batch_scenario = scenarios["result_1000"] + batch_scenario = max( + scenarios.values(), key=lambda item: int(item["visible_count"]) + ) batch_objects = object_keys(batch_scenario) for size in BATCH_CHECK_SIZES: - queries = [_query(batch_scenario, object_key) for object_key in batch_objects[:size]] - sample = await _timed_call( + queries = [ + _query(batch_scenario, object_key) + for object_key in batch_objects[:size] + ] + sample = await _timed_value( operation="batch_check", scenario=str(size), - call=lambda q=queries: client.batch_check(q), + call=lambda q=queries: _bounded_batch_check(client, q), validate=lambda allowed, expected=size: ( None if len(allowed) == expected and all(allowed) - else (_ for _ in ()).throw(BenchmarkContractError(f"BatchCheck {expected} result mismatch")) + else (_ for _ in ()).throw( + BenchmarkContractError( + f"visible BatchCheck {expected} result mismatch" + ) + ) ), ) if record: samples.append(sample) - for name, scenario in scenarios.items(): - expected_objects = object_keys(scenario) - sample = await _timed_call( - operation="list_objects", - scenario=name, - call=lambda s=scenario: client.list_objects( - user=f"user:{s['actor_id']}", - relation=f"can_{s['action']}", - resource_type=str(s["resource_type"]), - ), - validate=lambda objects, expected=expected_objects: ( - None - if sorted(objects) == sorted(expected) - else (_ for _ in ()).throw(BenchmarkContractError("ListObjects result set mismatch or truncation")) - ), - ) + for scenario in scenarios.values(): + sample = await _stream_sample(client, scenario) if record: samples.append(sample) - cursor_contract = contract["dataset"]["profile"]["business_cursor"] - cursor_scenario = scenarios[str(cursor_contract["scenario"])] - candidates = object_keys(cursor_scenario)[: int(cursor_contract["candidate_count"])] - page_size = int(cursor_contract["page_size"]) - started = perf_counter() - request_ids: list[str] = [] - visible: list[str] = [] - error: str | None = None - try: - for offset in range(0, len(candidates), page_size): - page = candidates[offset : offset + page_size] - allowed, request_id = await client.batch_check([_query(cursor_scenario, item) for item in page]) - request_ids.append(request_id) - visible.extend(item for item, is_allowed in zip(page, allowed, strict=True) if is_allowed) - if visible != candidates: - raise BenchmarkContractError("business cursor fingerprint mismatch") - except Exception as exc: - error = f"{type(exc).__name__}: {exc}" + paths = contract["dataset"]["business_paths"] + joined = paths["joined"] + joined_sample = await _id_first_chain_sample( + client, + name="joined", + scenario=scenarios[str(joined["scenario"])], + page_size=int(joined["page_size"]), + ) + department = paths["department"] + department_sample = await _candidate_chain_sample( + client, + name="department", + scenario=scenarios[str(department["scenario"])], + n_db=int(department["n_db"]), + visible_total=int(department["visible_total"]), + selectivity=float(department["selectivity"]), + page_size=int(department["page_size"]), + batch_size=int(department["batch_size"]), + ) + file_path = paths["file"] + file_sample = await _candidate_chain_sample( + client, + name="file", + scenario=scenarios[str(file_path["scenario"])], + n_db=int(file_path["n_db"]), + visible_total=int(file_path["visible_total"]), + selectivity=float(file_path["selectivity"]), + page_size=int(file_path["page_size"]), + batch_size=int(file_path["batch_size"]), + ) if record: - samples.append( - RequestSample( - operation="business_cursor", - scenario=str(cursor_contract["scenario"]), - elapsed_ms=(perf_counter() - started) * 1000, - request_ids=tuple(request_ids), - error=error, - result_count=len(visible), - result_checksum=checksum(visible), - ) - ) + samples.extend((joined_sample, department_sample, file_sample)) for _ in range(warmup): await run_once(False) @@ -648,11 +917,19 @@ def read_openfga_metrics( fields = _merged_log_fields(json.loads(line)) except (json.JSONDecodeError, TypeError): continue - request_id = str(fields.get("request_id") or fields.get("request.id") or fields.get("x-request-id") or "") + request_id = str( + fields.get("request_id") + or fields.get("request.id") + or fields.get("x-request-id") + or "" + ) if request_id not in request_ids: continue dispatch = _coerce_int(fields.get("dispatch_count")) - reads = _coerce_int(fields.get("datastore_query_count") or fields.get("datastore_read_count")) + reads = _coerce_int( + fields.get("datastore_query_count") + or fields.get("datastore_read_count") + ) if dispatch is not None and reads is not None: metrics[request_id] = { "dispatch_count": dispatch, @@ -661,16 +938,9 @@ def read_openfga_metrics( return metrics -def _sample_payload(sample: RequestSample) -> dict[str, Any]: - return { - "operation": sample.operation, - "scenario": sample.scenario, - "elapsed_ms": round(sample.elapsed_ms, 6), - "request_ids": list(sample.request_ids), - "error": sample.error, - "result_count": sample.result_count, - "result_checksum": sample.result_checksum, - } +def _summary_or_none(values: Iterable[float]) -> dict[str, float | int] | None: + collected = tuple(values) + return summarize(collected) if collected else None def evaluate( @@ -683,125 +953,140 @@ def evaluate( grouped[(sample.operation, sample.scenario)].append(sample) successful = [sample for sample in samples if sample.error is None] error_rate = 1 - (len(successful) / len(samples)) if samples else 1.0 - baseline = contract["baseline"] limits = contract["limits"] - check_values = [sample.elapsed_ms for sample in successful if sample.operation == "check"] - check_summary = summarize(check_values) - old_check = summarize(baseline["check_ms"]) - check_limit = max( - old_check["p95_ms"] * float(limits["check_p95_multiplier"]), - old_check["p95_ms"] + float(limits["check_p95_absolute_ms"]), + check = [sample for sample in successful if sample.operation == "check"] + check_summary = _summary_or_none(sample.elapsed_ms for sample in check) + check_passed = ( + check_summary is not None + and + len(check) == len(contract["dataset"]["scenarios"]) + * int(contract["run"]["iterations"]) + and check_summary["p95_ms"] <= float(limits["check_p95_ms"]) ) - check_gate = check_summary["p95_ms"] <= check_limit and error_rate < float(limits["max_error_rate"]) batch_report: dict[str, Any] = {} - batch_gate = True for size in BATCH_CHECK_SIZES: - values = [ - sample.elapsed_ms - for sample in successful - if sample.operation == "batch_check" and sample.scenario == str(size) - ] - summary = summarize(values) - old = summarize(baseline["batch_check_ms"][str(size)]) - limit = old["p95_ms"] * float(limits["batch_p95_multiplier"]) - passed = summary["p95_ms"] <= limit - batch_gate = batch_gate and passed - batch_report[str(size)] = { - "new": summary, - "baseline": old, - "p95_limit_ms": round(limit, 6), - "passed": passed, - } + group = grouped[("batch_check", str(size))] + summary = _summary_or_none( + sample.elapsed_ms for sample in group if sample.error is None + ) + passed = bool(group) and all(sample.error is None for sample in group) + passed = passed and summary is not None and summary["p95_ms"] <= float( + limits["batch_check_p95_ms"][str(size)] + ) + batch_report[str(size)] = {"latency": summary, "passed": passed} - scenario_by_name = _scenario_map(contract) - list_report: dict[str, Any] = {} - list_gate = True - for name, scenario in scenario_by_name.items(): - group = grouped[("list_objects", name)] + stream_report: dict[str, Any] = {} + for scenario in contract["dataset"]["scenarios"]: + name = str(scenario["name"]) + group = grouped[("stream_list_objects", name)] valid = bool(group) and all( sample.error is None - and sample.result_count == int(scenario["result_count"]) + and sample.stream_completed is True + and sample.result_count == int(scenario["visible_count"]) and sample.result_checksum == scenario["expected_object_checksum"] for sample in group ) - list_gate = list_gate and valid - list_report[name] = { - "latency": summarize(sample.elapsed_ms for sample in group if sample.error is None), - "expected_count": int(scenario["result_count"]), - "expected_checksum": scenario["expected_object_checksum"], - "passed": valid, + latency = _summary_or_none( + sample.elapsed_ms for sample in group if sample.error is None + ) + limit = float(limits["stream_p95_ms"][str(scenario["visible_count"])]) + passed = ( + valid + and latency is not None + and latency["p95_ms"] <= limit + ) + stream_report[name] = { + "resource_count": int(scenario["resource_count"]), + "visible_count": int(scenario["visible_count"]), + "source_kind": scenario["source_kind"], + "latency": latency, + "p95_limit_ms": limit, + "stream_completed": valid, + "passed": passed, } - cursor = [sample for sample in successful if sample.operation == "business_cursor"] - cursor_summary = summarize(sample.elapsed_ms for sample in cursor) - old_cursor = summarize(baseline["business_cursor_ms"]) - cursor_limit = old_cursor["p95_ms"] * float(limits["business_cursor_p95_multiplier"]) - cursor_contract = contract["dataset"]["profile"]["business_cursor"] - expected_cursor_objects = object_keys(scenario_by_name[str(cursor_contract["scenario"])])[ - : int(cursor_contract["candidate_count"]) - ] - cursor_gate = ( - cursor_summary["p95_ms"] <= cursor_limit - and bool(cursor) - and all( - sample.result_count == len(expected_cursor_objects) - and sample.result_checksum == checksum(expected_cursor_objects) - for sample in cursor - ) - ) + path_report: dict[str, Any] = {} + for name in ("joined", "department", "file"): + group = grouped[("business_path", name)] + valid = bool(group) and all(sample.error is None for sample in group) + path_report[name] = { + "strategy": group[0].strategy if group else None, + "latency": _summary_or_none( + sample.elapsed_ms for sample in group if sample.error is None + ), + "n_db": group[0].n_db if group else None, + "visible_total": group[0].visible_total if group else None, + "selectivity": group[0].selectivity if group else None, + "candidate_pass_rate": ( + group[0].candidate_pass_rate if group else None + ), + "db_rows": _summary_or_none( + float(sample.db_rows or 0) + for sample in group + if sample.error is None + ), + "scan_amplification": _summary_or_none( + float(sample.scan_amplification or 0) + for sample in group + if sample.error is None + ), + "passed": valid, + } - request_ids = {request_id for sample in samples for request_id in sample.request_ids} + request_ids = { + request_id for sample in samples for request_id in sample.request_ids + } observed_ids = request_ids.intersection(metrics) - observability_gate = bool(request_ids) and observed_ids == request_ids - metric_report = { + observability_passed = bool(request_ids) and observed_ids == request_ids + observability = { "request_count": len(request_ids), "observed_request_count": len(observed_ids), - "dispatch": (summarize(metrics[item]["dispatch_count"] for item in observed_ids) if observed_ids else None), - "datastore_reads": ( - summarize(metrics[item]["datastore_query_count"] for item in observed_ids) if observed_ids else None + "dispatch": _summary_or_none( + float(metrics[item]["dispatch_count"]) for item in observed_ids + ), + "datastore_reads": _summary_or_none( + float(metrics[item]["datastore_query_count"]) for item in observed_ids ), - "passed": observability_gate, + "passed": observability_passed, } - fixture_gate = bool(contract["dataset"]["production_derived"]) performance_passed = all( ( - check_gate, - batch_gate, - list_gate, - cursor_gate, - observability_gate, + check_passed, + all(item["passed"] for item in batch_report.values()), + all(item["passed"] for item in stream_report.values()), + all(item["passed"] for item in path_report.values()), + observability_passed, error_rate < float(limits["max_error_rate"]), ) ) + distribution_accepted = bool( + contract["dataset"].get("production_derived") + or contract["dataset"].get("representative_distribution") + ) return { "contract_version": contract["contract_version"], "contract_checksum": contract["contract_checksum"], - "dataset_checksum": contract["dataset"]["dataset_checksum"], "authorization_model_checksum": contract["authorization_model_checksum"], + "dataset_checksum": contract["dataset"]["dataset_checksum"], + "source_checksum": contract["dataset"]["source_checksum"], + "visible_checksum": contract["dataset"]["visible_checksum"], "openfga_version": OPENFGA_VERSION, "dataset_source": contract["dataset"]["source"], - "production_derived": fixture_gate, + "production_derived": bool(contract["dataset"].get("production_derived")), + "representative_distribution": bool( + contract["dataset"].get("representative_distribution") + ), "error_rate": round(error_rate, 8), - "check": { - "new": check_summary, - "baseline": old_check, - "p95_limit_ms": round(check_limit, 6), - "passed": check_gate, - }, + "check": {"latency": check_summary, "passed": check_passed}, "batch_check": batch_report, - "list_objects": list_report, - "business_cursor": { - "new": cursor_summary, - "baseline": old_cursor, - "p95_limit_ms": round(cursor_limit, 6), - "passed": cursor_gate, - }, - "openfga_observability": metric_report, + "streamed_list_objects": stream_report, + "business_paths": path_report, + "openfga_observability": observability, "performance_passed": performance_passed, - "release_ready": performance_passed and fixture_gate, - "samples": [_sample_payload(sample) for sample in samples], + "release_ready": performance_passed and distribution_accepted, + "samples": [asdict(sample) for sample in samples], } @@ -821,7 +1106,7 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace: "--apply", action="store_true", required=True, - help="Required acknowledgement for benchmark Store writes", + help="Required acknowledgement for writes to the dedicated benchmark Store", ) run = subparsers.choices["run"] run.add_argument("--model-id", required=True) @@ -832,7 +1117,7 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace: run.add_argument( "--allow-synthetic", action="store_true", - help="Harness smoke only; report remains release_ready=false", + help="Allow the checksum-pinned representative synthetic fixture", ) return parser.parse_args(argv) @@ -854,15 +1139,22 @@ async def execute(args: argparse.Namespace) -> tuple[int, dict[str, Any]]: finally: await preparer.close() - if not contract["dataset"]["production_derived"] and not args.allow_synthetic: + if not contract["dataset"].get("production_derived") and not args.allow_synthetic: raise BenchmarkContractError( - "formal BENCH-01 requires a production-derived sanitized fixture; " - "use --allow-synthetic only to smoke-test the harness" + "this BENCH-01 fixture is synthetic; use --allow-synthetic only in a dedicated benchmark environment" ) iterations = args.iterations or int(contract["run"]["iterations"]) - warmup = args.warmup if args.warmup is not None else int(contract["run"]["warmup"]) + warmup = ( + args.warmup + if args.warmup is not None + else int(contract["run"]["warmup"]) + ) if iterations < 1 or warmup < 0: - raise BenchmarkContractError("iterations must be positive and warmup non-negative") + raise BenchmarkContractError( + "iterations must be positive and warmup non-negative" + ) + runtime_contract = json.loads(json.dumps(contract)) + runtime_contract["run"]["iterations"] = iterations client = InstrumentedOpenFGAClient( api_url=args.api_url, store_id=args.store_id, @@ -872,22 +1164,26 @@ async def execute(args: argparse.Namespace) -> tuple[int, dict[str, Any]]: try: samples = await run_workloads( client, - contract, + runtime_contract, iterations=iterations, warmup=warmup, ) finally: await client.close() - request_ids = {request_id for sample in samples for request_id in sample.request_ids} + request_ids = { + request_id for sample in samples for request_id in sample.request_ids + } metrics = read_openfga_metrics(args.openfga_log, request_ids) - report = evaluate(contract, samples, metrics) + report = evaluate(runtime_contract, samples, metrics) if args.output: args.output.write_text( json.dumps(report, ensure_ascii=False, indent=2) + "\n", encoding="utf-8", ) - passed = report["performance_passed"] and (report["release_ready"] or args.allow_synthetic) - return (EXIT_OK if passed else EXIT_GATE_FAILED), report + return ( + EXIT_OK if report["release_ready"] else EXIT_GATE_FAILED, + report, + ) def main(argv: list[str] | None = None) -> int: diff --git a/src/backend/scripts/f048_migration_runtime.py b/src/backend/scripts/f048_migration_runtime.py index 04a36b2aa9..04d2ea98c4 100644 --- a/src/backend/scripts/f048_migration_runtime.py +++ b/src/backend/scripts/f048_migration_runtime.py @@ -20,8 +20,12 @@ ) from bisheng.common.errcode.permission import PermissionMigrationBlockedError from bisheng.common.services.config_service import settings +from bisheng.core.openfga.authorization_model_f048 import ( + authorization_model_checksum, + get_authorization_model_f048, +) from bisheng.core.openfga.client import FGAClient -from bisheng.core.openfga.discovery import discover_openfga_runtime +from bisheng.core.openfga.discovery import OpenFGARuntimePin, discover_openfga_runtime from bisheng.knowledge.domain.services.permission_migration_source import ( KnowledgePermissionMigrationSource, SqlKnowledgeMigrationRepository, @@ -75,6 +79,15 @@ async def aclose(self) -> None: await self.source_client.close() +def _require_predecessor_source(*, pin: OpenFGARuntimePin, run_id: int | None) -> None: + """Reject starting a second formal migration from the final F048 model.""" + + if run_id is None and pin.model_checksum == authorization_model_checksum( + get_authorization_model_f048() + ): + raise PermissionMigrationBlockedError(msg="F048_MIGRATION_ALREADY_COMPLETED") + + async def build_f048_migration_runtime( live_settings: Any = settings, *, @@ -96,6 +109,7 @@ async def build_f048_migration_runtime( required_store_id=run.store_id if run else None, required_model_id=run.source_model_id if run else None, ) + _require_predecessor_source(pin=pin, run_id=run_id) source_client = FGAClient( api_url=config.api_url, diff --git a/src/backend/scripts/reconcile_f048_projection_operations.py b/src/backend/scripts/reconcile_f048_projection_operations.py index 050d9ec65f..f55dd4c813 100644 --- a/src/backend/scripts/reconcile_f048_projection_operations.py +++ b/src/backend/scripts/reconcile_f048_projection_operations.py @@ -109,6 +109,9 @@ class OperationInspection: status: str tuple_count: int tuple_summary: dict[str, int] + visible_source_count: int + visible_source_summary: dict[str, int] + visible_source_checksum: str | None resource_mode: dict[str, Any] | None @@ -218,6 +221,12 @@ async def inspect_operation( with bypass_tenant_filter(): operation = await runtime.repository.aget_operation(operation_id) tuple_rows = await runtime.repository.aget_operation_tuples(operation_id) + visible_source_rows = await runtime.repository.aget_visible_operation_sources( + operation_id, + ) + visible_source_checksum = await runtime.repository.aget_visible_operation_checksum( + operation_id, + ) _require(operation is not None, f"operation {operation_id} does not exist") _require( @@ -237,6 +246,26 @@ async def inspect_operation( f"operation {operation_id} cannot be reconciled from status {operation.status}", ) _require(tuple_rows, f"operation {operation_id} has no durable tuple ledger") + if operation.operation_type == "GRANT_MUTATION": + _require( + bool(visible_source_rows), + f"operation {operation_id} has no frozen visible source after-state", + ) + visible_source_summary = Counter(row.state for row in visible_source_rows) + _require( + not visible_source_summary.get("FAILED_CLOSED"), + f"operation {operation_id} has FAILED_CLOSED visible sources", + ) + if operation.status == "FINALIZED": + _require( + not visible_source_summary.get("PENDING"), + f"operation {operation_id} is FINALIZED with pending visible sources", + ) + elif visible_source_rows: + _require( + set(visible_source_summary) == {"PENDING"}, + f"operation {operation_id} active visible source after-state is mixed", + ) try: restore_projection_plan(operation, tuple_rows) except ( @@ -283,6 +312,9 @@ async def inspect_operation( status=operation.status, tuple_count=len(tuple_rows), tuple_summary=dict(sorted(tuple_summary.items())), + visible_source_count=len(visible_source_rows), + visible_source_summary=dict(sorted(visible_source_summary.items())), + visible_source_checksum=visible_source_checksum, resource_mode=resource_mode, ) diff --git a/src/backend/scripts/reconcile_f048_visible_projection.py b/src/backend/scripts/reconcile_f048_visible_projection.py new file mode 100644 index 0000000000..905072c6e4 --- /dev/null +++ b/src/backend/scripts/reconcile_f048_visible_projection.py @@ -0,0 +1,810 @@ +#!/usr/bin/env python3 +"""Audit or repair the F048 flattened visible projection. + +The command is safe for production use when run in a maintenance window. It +uses PermissionGrant/PermissionGrantAssignee as the canonical authorization +source, rebuilds permission_visible_source_projection, idempotently ensures +all expected direct ``visible`` tuples, verifies them at higher consistency, +and can perform the immutable Authorization Model + Catalog forward cutover required after an +older F048 migration. + +Run from ``src/backend`` with the live ``config`` value:: + + PYTHONPATH=./ .venv/bin/python scripts/reconcile_f048_visible_projection.py + PYTHONPATH=./ .venv/bin/python scripts/reconcile_f048_visible_projection.py --apply --confirm-store-id --operator-id --allow-model-upgrade + +Dry-run is the default. Apply refuses active runtime heartbeats or in-flight +permission projection operations. It never scans or deletes existing +``visible`` tuples: system/public/shared sources are owned outside the Grant +source projection. +""" + +from __future__ import annotations + +import argparse +import asyncio +import json +import os +import sys +import traceback +from collections import defaultdict +from dataclasses import asdict, dataclass +from datetime import UTC, datetime, timedelta +from hashlib import sha256 +from typing import Any + +_BACKEND_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..")) +if _BACKEND_ROOT not in sys.path: + sys.path.insert(0, _BACKEND_ROOT) + +from sqlalchemy import func, update # noqa: E402 +from sqlmodel import col, select # noqa: E402 + +from bisheng.common.services.config_service import settings # noqa: E402 +from bisheng.core.context.manager import ( # noqa: E402 + close_app_context, + initialize_app_context, +) +from bisheng.core.context.tenant import bypass_tenant_filter # noqa: E402 +from bisheng.core.database import get_async_db_session # noqa: E402 +from bisheng.core.openfga.authorization_model_f048 import ( # noqa: E402 + MODEL_VERSION, + authorization_model_checksum, + get_authorization_model_f048, +) +from bisheng.core.openfga.client import ( # noqa: E402 + BUSINESS_BATCH_CHECK_LIMIT, + FGAClient, +) +from bisheng.core.openfga.discovery import discover_openfga_runtime # noqa: E402 +from bisheng.core.openfga.runtime_heartbeat import ( # noqa: E402 + list_runtime_heartbeats, +) +from bisheng.permission.application.catalog_api import ( # noqa: E402 + OpenFGACatalogProjector, + SqlCatalogImpact, + SqlCatalogState, +) +from bisheng.permission.application.sql_runtime import RedisConsistencyMarker # noqa: E402 +from bisheng.permission.domain.models import ( # noqa: E402 + AuthorizationModelRelease, + PermissionCatalogRelease, + PermissionGrant, + PermissionGrantAssignee, + PermissionProjectionOperation, + PermissionVisibleSourceProjection, +) +from bisheng.permission.domain.services.catalog_service import ( # noqa: E402 + CatalogDraftBuildInput, + CatalogService, +) +from bisheng.permission.domain.services.grant_source_service import ( # noqa: E402 + SOURCE_TYPES, + GrantModelSnapshot, + GrantSnapshot, + GrantSourceRecord, +) +from bisheng.permission.domain.services.model_policy import ( # noqa: E402 + CustomModelSelection, +) +from bisheng.permission.domain.services.visibility_projection_service import ( # noqa: E402 + VisibilityProjectionCompiler, +) +from bisheng.permission.migration.f048_runtime_storage import ( # noqa: E402 + OpenFGAMigrationModelPublisher, +) + +EXIT_OK = 0 +EXIT_BLOCKED = 3 +EXIT_RUNTIME_ERROR = 4 +HIGHER_CONSISTENCY = "HIGHER_CONSISTENCY" +ACTIVE_OPERATION_STATUSES = ("PREPARED", "STAGING", "COMMIT_UNKNOWN", "COMMITTED") + + +class VisibleReconcileBlockedError(RuntimeError): + """A safety or canonical-source invariant blocked reconciliation.""" + + +@dataclass(frozen=True, slots=True) +class CurrentRelease: + catalog_id: int + catalog_key: str + store_id: str + model_id: str + model_release_id: int + model_checksum: str + write_fenced: bool + + +@dataclass(frozen=True, slots=True) +class ReconcileReport: + mode: str + store_id: str + source_model_id: str + target_model_id: str | None + target_model_checksum: str + catalog_write_fenced: bool + grant_count: int + assignee_count: int + canonical_source_count: int + persisted_active_source_count: int + source_upsert_count: int + source_retire_count: int + expected_tuple_count: int + source_checksum: str + expected_tuple_checksum: str + + +def parse_args(argv: list[str] | None = None) -> argparse.Namespace: + parser = argparse.ArgumentParser( + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument( + "--apply", + action="store_true", + help="Repair SQL source projections, ensure expected tuples, and update the model/Catalog pin", + ) + parser.add_argument( + "--confirm-store-id", + default=None, + help="Required with --apply; must equal the discovered immutable Store ID", + ) + parser.add_argument( + "--operator-id", + type=int, + default=0, + help="Audit operator ID recorded on the no-op Catalog release", + ) + parser.add_argument( + "--allow-model-upgrade", + action="store_true", + help="Explicitly allow publishing and switching from an older F048 model", + ) + parser.add_argument( + "--batch-size", + type=int, + default=80, + help="OpenFGA write batch size, 1..90 (default: 80)", + ) + args = parser.parse_args(argv) + if not 1 <= args.batch_size <= 90: + parser.error("--batch-size must be between 1 and 90") + if args.operator_id < 0: + parser.error("--operator-id must not be negative") + if args.apply and not args.confirm_store_id: + parser.error("--apply requires --confirm-store-id") + if args.apply and args.operator_id <= 0: + parser.error("--apply requires a positive --operator-id") + return args + + +def _require(condition: bool, message: str) -> None: + if not condition: + raise VisibleReconcileBlockedError(message) + + +def _checksum(rows: Any) -> str: + payload = json.dumps(rows, ensure_ascii=True, sort_keys=True, separators=(",", ":")) + return sha256(payload.encode()).hexdigest() + + +def _environment_name(value: str | dict[str, Any]) -> str: + if isinstance(value, dict): + value = next( + (value[key] for key in ("name", "environment", "env", "mode") if value.get(key)), + "dev", + ) + return str(value or "dev")[:64] + + +def _offline_settings(live_settings: Any) -> Any: + openfga = getattr(live_settings, "openfga", None) + copy_settings = getattr(live_settings, "model_copy", None) + copy_openfga = getattr(openfga, "model_copy", None) + if not callable(copy_settings) or not callable(copy_openfga): + return live_settings + return copy_settings(update={"openfga": copy_openfga(update={"enabled": False})}) + + +async def _load_current_release() -> CurrentRelease: + with bypass_tenant_filter(): + async with get_async_db_session() as session: + rows = list( + ( + await session.execute( + select(PermissionCatalogRelease, AuthorizationModelRelease) + .join( + AuthorizationModelRelease, + AuthorizationModelRelease.id + == PermissionCatalogRelease.required_authorization_model_release_id, + ) + .where(PermissionCatalogRelease.status == "CURRENT") + ) + ).all() + ) + _require(len(rows) == 1, "Permission Catalog must have exactly one CURRENT release") + catalog, model = rows[0] + _require(model.status == "ACTIVE", "CURRENT Authorization Model release is not ACTIVE") + _require(catalog.id is not None and model.id is not None, "CURRENT release identity is incomplete") + return CurrentRelease( + catalog_id=int(catalog.id), + catalog_key=catalog.release_key, + store_id=model.store_id, + model_id=model.model_id, + model_release_id=int(model.id), + model_checksum=model.model_checksum, + write_fenced=bool(catalog.write_fenced), + ) + + +async def _assert_maintenance_window(*, apply: bool) -> None: + with bypass_tenant_filter(): + async with get_async_db_session() as session: + active_operations = int( + ( + await session.execute( + select(func.count(PermissionProjectionOperation.id)).where( + PermissionProjectionOperation.status.in_(ACTIVE_OPERATION_STATUSES) + ) + ) + ).scalar_one() + ) + _require(active_operations == 0, f"{active_operations} permission projection operations are active") + if apply: + heartbeats = await list_runtime_heartbeats() + _require(not heartbeats, f"{len(heartbeats)} F048 runtime heartbeats are still active") + + +async def _load_canonical_grants() -> tuple[tuple[GrantSnapshot, ...], int]: + with bypass_tenant_filter(): + async with get_async_db_session() as session: + grant_rows = list( + ( + await session.execute( + select(PermissionGrant) + .where(PermissionGrant.state == "ACTIVE") + .order_by(PermissionGrant.tenant_id, PermissionGrant.id) + ) + ) + .scalars() + .all() + ) + grant_ids = [int(row.id) for row in grant_rows if row.id is not None] + assignee_rows = ( + list( + ( + await session.execute( + select(PermissionGrantAssignee) + .where( + col(PermissionGrantAssignee.grant_id).in_(grant_ids), + PermissionGrantAssignee.state == "ACTIVE", + ) + .order_by(PermissionGrantAssignee.tenant_id, PermissionGrantAssignee.id) + ) + ) + .scalars() + .all() + ) + if grant_ids + else [] + ) + blockers = [ + f"grant:{row.id}:projection_state={row.projection_state}" + for row in grant_rows + if row.projection_state != "CURRENT" + ] + unknown_sources = sorted({row.source_type for row in assignee_rows} - set(SOURCE_TYPES)) + if unknown_sources: + blockers.append(f"unknown assignee source types: {unknown_sources}") + grant_by_id = {int(row.id): row for row in grant_rows if row.id is not None} + assignees_by_grant: dict[int, list[PermissionGrantAssignee]] = defaultdict(list) + for row in assignee_rows: + grant = grant_by_id.get(int(row.grant_id)) + if grant is None or int(row.tenant_id or 0) != int(grant.tenant_id or 0): + blockers.append(f"assignee:{row.id}:grant_or_tenant_mismatch") + continue + assignees_by_grant[int(row.grant_id)].append(row) + _require(not blockers, "; ".join(blockers[:20])) + + snapshots: list[GrantSnapshot] = [] + for row in grant_rows: + tenant_id = int(row.tenant_id or 0) + _require(tenant_id > 0 and row.id is not None, f"grant {row.id} has invalid identity") + sources = tuple( + GrantSourceRecord( + source_id=int(source.id or 0), + subject_type=source.subject_type, + subject_id=source.subject_id, + userset_relation=source.userset_relation, + include_children=bool(source.include_children), + source_type=source.source_type, + source_ref=source.source_ref, + source_locator=source.source_locator, + source_fingerprint=source.source_fingerprint, + projected_subject=source.projected_subject, + protected=bool(source.protected), + active=True, + version=int(source.version), + ) + for source in assignees_by_grant.get(int(row.id), ()) + ) + snapshots.append( + GrantSnapshot( + grant_id=str(row.id), + tenant_id=tenant_id, + resource_type=row.resource_type, + resource_id=row.resource_id, + model=GrantModelSnapshot( + model_key=row.model_key, + active=True, + action_codes=(), + ), + active=bool(sources), + sources=sources, + version=int(row.version), + ) + ) + return tuple(snapshots), len(assignee_rows) + + +async def _load_persisted_sources() -> tuple[PermissionVisibleSourceProjection, ...]: + with bypass_tenant_filter(): + async with get_async_db_session() as session: + return tuple( + ( + await session.execute( + select(PermissionVisibleSourceProjection).order_by( + PermissionVisibleSourceProjection.tenant_id, + PermissionVisibleSourceProjection.id, + ) + ) + ) + .scalars() + .all() + ) + + +def _compile_sources(grants: tuple[GrantSnapshot, ...]): + compiler = VisibilityProjectionCompiler() + grouped: dict[int, list[GrantSnapshot]] = defaultdict(list) + for grant in grants: + grouped[grant.tenant_id].append(grant) + return tuple( + source + for tenant_id in sorted(grouped) + for source in compiler.compile( + tenant_id=tenant_id, + grants=tuple(grouped[tenant_id]), + existing_sources=(), + ).active_sources + ) + + +def _source_key(source: Any) -> str: + return str(source.contribution_fingerprint) + + +def _tuple_key(source: Any) -> tuple[str, str, str]: + return ( + str(source.projected_subject), + "visible", + f"{source.resource_type}:{source.resource_id}", + ) + + +def _build_report( + *, + mode: str, + current: CurrentRelease, + target_model_id: str | None, + target_checksum: str, + grants: tuple[GrantSnapshot, ...], + assignee_count: int, + canonical_sources: tuple[Any, ...], + persisted: tuple[PermissionVisibleSourceProjection, ...], +) -> tuple[ReconcileReport, tuple[Any, ...], tuple[PermissionVisibleSourceProjection, ...], frozenset]: + desired = {_source_key(row): row for row in canonical_sources} + persisted_active = {_source_key(row): row for row in persisted if row.state == "ACTIVE"} + upserts = tuple( + row + for key, row in sorted(desired.items()) + if key not in persisted_active + or persisted_active[key].source_version != row.source_version + or persisted_active[key].tuple_fingerprint != row.tuple_fingerprint + ) + retires = tuple(row for key, row in sorted(persisted_active.items()) if key not in desired) + expected = frozenset(_tuple_key(row) for row in canonical_sources) + report = ReconcileReport( + mode=mode, + store_id=current.store_id, + source_model_id=current.model_id, + target_model_id=target_model_id, + target_model_checksum=target_checksum, + catalog_write_fenced=current.write_fenced, + grant_count=len(grants), + assignee_count=assignee_count, + canonical_source_count=len(canonical_sources), + persisted_active_source_count=len(persisted_active), + source_upsert_count=len(upserts), + source_retire_count=len(retires), + expected_tuple_count=len(expected), + source_checksum=_checksum([row.model_dump(mode="json") for row in sorted(canonical_sources, key=_source_key)]), + expected_tuple_checksum=_checksum(sorted(expected)), + ) + return report, upserts, retires, expected + + +async def _apply_source_rows(upserts: tuple[Any, ...]) -> None: + with bypass_tenant_filter(): + async with get_async_db_session() as session: + async with session.begin(): + for source in upserts: + existing = ( + ( + await session.execute( + select(PermissionVisibleSourceProjection) + .where( + PermissionVisibleSourceProjection.tenant_id == source.tenant_id, + PermissionVisibleSourceProjection.contribution_fingerprint + == source.contribution_fingerprint, + ) + .with_for_update() + ) + ) + .scalars() + .first() + ) + if existing is None: + session.add( + PermissionVisibleSourceProjection( + **source.model_dump(exclude={"operation_id", "migration_item_id"}), + ) + ) + continue + immutable = ( + "resource_type", + "resource_id", + "visibility_class", + "projected_subject", + "source_kind", + "source_owner_key", + "source_locator", + "source_fingerprint", + "model_key", + ) + _require( + all(getattr(existing, field) == getattr(source, field) for field in immutable), + f"visible source fingerprint collision: {source.contribution_fingerprint}", + ) + existing.source_version = source.source_version + existing.tuple_fingerprint = source.tuple_fingerprint + existing.state = "ACTIVE" + + +async def _ensure_expected_tuples( + client: FGAClient, + expected: frozenset[tuple[str, str, str]], + *, + batch_size: int, +) -> None: + rows = sorted(expected) + for offset in range(0, len(rows), batch_size): + batch = rows[offset : offset + batch_size] + client.validate_business_mutation_size(len(batch)) + await client.write_tuples( + writes=[{"user": user, "relation": relation, "object": object_key} for user, relation, object_key in batch], + ignore_duplicate_writes=True, + ) + + +async def _verify_expected_tuples( + client: FGAClient, + expected: frozenset[tuple[str, str, str]], +) -> None: + rows = sorted(expected) + for offset in range(0, len(rows), BUSINESS_BATCH_CHECK_LIMIT): + batch = rows[offset : offset + BUSINESS_BATCH_CHECK_LIMIT] + allowed = await client.batch_check( + [{"user": user, "relation": relation, "object": object_key} for user, relation, object_key in batch], + consistency=HIGHER_CONSISTENCY, + ) + _require( + len(allowed) == len(batch) and all(allowed), + "higher-consistency visible tuple verification failed", + ) + + +async def _authorization_release_id(store_id: str, model_id: str) -> int: + with bypass_tenant_filter(): + async with get_async_db_session() as session: + row = ( + ( + await session.execute( + select(AuthorizationModelRelease).where( + AuthorizationModelRelease.store_id == store_id, + AuthorizationModelRelease.model_id == model_id, + ) + ) + ) + .scalars() + .first() + ) + _require(row is not None and row.id is not None, "target Authorization Model release is missing") + return int(row.id) + + +async def _is_resumable_upgrade_draft( + *, + current_catalog_id: int, + target_checksum: str, +) -> bool: + idempotency_key = f"f048-visible-{target_checksum[:32]}" + with bypass_tenant_filter(): + async with get_async_db_session() as session: + count = int( + ( + await session.execute( + select(func.count(PermissionCatalogRelease.id)).where( + PermissionCatalogRelease.predecessor_id == current_catalog_id, + PermissionCatalogRelease.idempotency_key == idempotency_key, + PermissionCatalogRelease.status.in_(("DRAFT", "PROJECTING", "COMMITTED")), + ) + ) + ).scalar_one() + ) + return count == 1 + + +async def _activate_model_release(release_id: int) -> None: + with bypass_tenant_filter(): + async with get_async_db_session() as session: + async with session.begin(): + await session.execute( + update(AuthorizationModelRelease) + .where(AuthorizationModelRelease.id == release_id) + .values(status="ACTIVE", activated_at=func.now(), retired_at=None) + ) + + +async def _publish_noop_catalog_cutover( + *, + current: CurrentRelease, + target_client: FGAClient, + target_release_id: int, + operator_id: int, + target_checksum: str, +) -> int: + state = SqlCatalogState() + before = await state.load_snapshot(current.catalog_id) + _require( + before.action_release is not None and before.model_release is not None, + "CURRENT Catalog snapshot is incomplete", + ) + idempotency_key = f"f048-visible-{target_checksum[:32]}" + reservation = await state.reserve_draft( + base_release_id=current.catalog_id, + operator_id=operator_id, + idempotency_key=idempotency_key, + expires_at=datetime.now(UTC).replace(tzinfo=None) + timedelta(minutes=30), + ) + impact = SqlCatalogImpact() + marker = RedisConsistencyMarker() + await marker.initialize() + service = CatalogService( + state=state, + impact=impact, + projector=OpenFGACatalogProjector(client=target_client, marker=marker), + ) + if not reservation.complete: + customs = tuple( + CustomModelSelection( + model_key=model.model_key, + name=model.name, + action_codes=model.selected_action_codes, + active=model.active, + allow_same_level=model.allow_same_level, + config_scope=model.config_scope, + ) + for model in before.model_release.models + if model.kind == "CUSTOM" + ) + standard_policy = { + model.model_key: model.allow_same_level for model in before.model_release.models if model.kind == "STANDARD" + } + await service.build_draft( + CatalogDraftBuildInput( + release_id=reservation.release_id, + release_key=reservation.release_key, + predecessor_release_id=reservation.predecessor_id, + predecessor_release_key=reservation.predecessor_key, + before_actions=before.action_release, + before_models=before.model_release, + actions=before.action_release.actions, + custom_models=customs, + standard_allow_same_level=standard_policy, + grant_references=await state.grant_references(), + draft_owner_id=operator_id, + idempotency_key=idempotency_key, + expires_at=datetime.now(UTC).replace(tzinfo=None) + timedelta(minutes=30), + ) + ) + await state.bind_draft_authorization_release( + draft_id=reservation.release_id, + authorization_release_id=target_release_id, + ) + outcome = await service.publish( + draft_id=reservation.release_id, + expected_current_release_id=current.catalog_id, + idempotency_key=idempotency_key, + ) + _require(outcome.status == "CURRENT", "Authorization Model Catalog cutover did not finalize") + return outcome.release_id + + +async def _retire_other_active_models(current: CurrentRelease) -> None: + with bypass_tenant_filter(): + async with get_async_db_session() as session: + async with session.begin(): + await session.execute( + update(AuthorizationModelRelease) + .where( + AuthorizationModelRelease.store_id == current.store_id, + AuthorizationModelRelease.id != current.model_release_id, + AuthorizationModelRelease.status == "ACTIVE", + ) + .values(status="RETIRED", retired_at=func.now()) + ) + + +async def execute(args: argparse.Namespace, *, live_settings: Any = settings) -> int: + await initialize_app_context(config=_offline_settings(live_settings)) + source_client: FGAClient | None = None + target_client: FGAClient | None = None + try: + current = await _load_current_release() + config = live_settings.openfga + pin = await discover_openfga_runtime( + config, + expected_model=None, + allow_bootstrap=False, + required_store_id=current.store_id, + required_model_id=current.model_id, + ) + _require(pin.store_id == current.store_id, "OpenFGA Store differs from CURRENT Catalog") + _require( + pin.model_checksum == current.model_checksum, + "OpenFGA model checksum differs from the CURRENT SQL release", + ) + if args.apply: + _require( + args.confirm_store_id == current.store_id, + "--confirm-store-id does not match the discovered Store", + ) + await _assert_maintenance_window(apply=args.apply) + source_client = FGAClient( + api_url=config.api_url, + store_id=pin.store_id, + model_id=pin.model_id, + timeout=config.timeout, + ) + target_model = get_authorization_model_f048() + target_checksum = authorization_model_checksum(target_model) + target_model_id: str | None = current.model_id if current.model_checksum == target_checksum else None + target_client = source_client + + grants, assignee_count = await _load_canonical_grants() + canonical_sources = _compile_sources(grants) + persisted = await _load_persisted_sources() + report, upserts, retires, expected = _build_report( + mode="apply" if args.apply else "dry-run", + current=current, + target_model_id=target_model_id, + target_checksum=target_checksum, + grants=grants, + assignee_count=assignee_count, + canonical_sources=canonical_sources, + persisted=persisted, + ) + print(json.dumps(asdict(report), ensure_ascii=False, sort_keys=True)) + if not args.apply: + print("[dry-run] no SQL, OpenFGA, Authorization Model, or Catalog mutations were requested") + return EXIT_OK + + _require( + not retires, + f"{len(retires)} stale Grant source projections require classified removal; no writes applied", + ) + if current.write_fenced: + _require( + await _is_resumable_upgrade_draft( + current_catalog_id=current.catalog_id, + target_checksum=target_checksum, + ), + "CURRENT Catalog is fenced by an unrelated or non-resumable publication", + ) + if target_model_id is None: + _require( + args.allow_model_upgrade, + "CURRENT model differs; re-run with --allow-model-upgrade after reviewing dry-run", + ) + publisher = OpenFGAMigrationModelPublisher( + source_client=source_client, + environment=_environment_name(live_settings.environment), + predecessor_model_id=current.model_id, + ) + target_model_id = await publisher.aget_or_publish( + store_id=current.store_id, + model=target_model, + checksum=target_checksum, + ) + target_client = source_client.for_model(target_model_id) + _require(target_model_id is not None, "target Authorization Model was not published") + await _ensure_expected_tuples(target_client, expected, batch_size=args.batch_size) + await _verify_expected_tuples(target_client, expected) + await _apply_source_rows(upserts) + + new_catalog_id = current.catalog_id + if current.model_id != target_model_id: + target_release_id = await _authorization_release_id( + current.store_id, + target_model_id, + ) + await _activate_model_release(target_release_id) + new_catalog_id = await _publish_noop_catalog_cutover( + current=current, + target_client=target_client, + target_release_id=target_release_id, + operator_id=args.operator_id, + target_checksum=target_checksum, + ) + + final = await _load_current_release() + _require( + final.catalog_id == new_catalog_id + and final.store_id == current.store_id + and final.model_id == target_model_id + and final.model_checksum == target_checksum, + "final CURRENT Catalog/model pin verification failed", + ) + await _retire_other_active_models(final) + print( + json.dumps( + { + "event": "reconciled", + "catalog_release_id": final.catalog_id, + "store_id": final.store_id, + "model_id": final.model_id, + "model_version": MODEL_VERSION, + "source_upserts": len(upserts), + "source_retires": len(retires), + "visible_tuples_ensured": len(expected), + "visible_tuples_verified": len(expected), + }, + ensure_ascii=False, + sort_keys=True, + ) + ) + return EXIT_OK + finally: + if target_client is not None and target_client is not source_client: + await target_client.close() + if source_client is not None: + await source_client.close() + await close_app_context() + + +def main(argv: list[str] | None = None) -> int: + args = parse_args(argv) + try: + return asyncio.run(execute(args)) + except VisibleReconcileBlockedError as exc: + print(f"F048 visible reconcile blocked: {exc}", file=sys.stderr) + return EXIT_BLOCKED + except Exception: + traceback.print_exc() + return EXIT_RUNTIME_ERROR + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/backend/test/department/test_department_knowledge_space_service.py b/src/backend/test/department/test_department_knowledge_space_service.py index 73707bdd5b..5ccb07541d 100644 --- a/src/backend/test/department/test_department_knowledge_space_service.py +++ b/src/backend/test/department/test_department_knowledge_space_service.py @@ -509,7 +509,7 @@ async def test_get_user_department_spaces_uses_department_binding_without_member DepartmentKnowledgeSpaceService = _load_service_class() login_user = _make_login_user(is_admin=False) binding = SimpleNamespace(space_id=101, department_id=10) - expected = [SimpleNamespace(id=101, space_kind="department")] + expected = [SimpleNamespace(id=101)] with ( patch( @@ -517,18 +517,13 @@ async def test_get_user_department_spaces_uses_department_binding_without_member new_callable=AsyncMock, return_value=[binding], ), - patch( - "bisheng.knowledge.domain.services.department_knowledge_space_service.SpaceChannelMemberDao.async_get_user_space_members", - new_callable=AsyncMock, - return_value=[], - ), patch( "bisheng.knowledge.domain.services.department_knowledge_space_service.batch_check_business_actions", new_callable=AsyncMock, return_value={"101": frozenset({"visible"})}, ), patch( - "bisheng.knowledge.domain.services.department_knowledge_space_service.KnowledgeSpaceService._format_accessible_spaces", + "bisheng.knowledge.domain.services.department_knowledge_space_service.KnowledgeSpaceService._format_basic_spaces", new_callable=AsyncMock, return_value=expected, ) as mock_format, @@ -543,8 +538,6 @@ async def test_get_user_department_spaces_uses_department_binding_without_member mock_format.assert_awaited_once() assert mock_format.await_args.args[0] == [101] assert mock_format.await_args.args[1] == "name" - assert mock_format.await_args.kwargs["memberships"] == [] - assert mock_format.await_args.kwargs["required_action"] == "visible" @pytest.mark.asyncio @@ -560,10 +553,7 @@ async def test_get_user_department_spaces_super_admin_sees_all_departments(): SimpleNamespace(space_id=101, department_id=10), SimpleNamespace(space_id=202, department_id=20), ] - expected = [ - SimpleNamespace(id=101, space_kind="department"), - SimpleNamespace(id=202, space_kind="department"), - ] + expected = [SimpleNamespace(id=101), SimpleNamespace(id=202)] with ( patch( @@ -571,11 +561,6 @@ async def test_get_user_department_spaces_super_admin_sees_all_departments(): new_callable=AsyncMock, return_value=all_bindings, ), - patch( - "bisheng.knowledge.domain.services.department_knowledge_space_service.SpaceChannelMemberDao.async_get_user_space_members", - new_callable=AsyncMock, - return_value=[], - ), patch( "bisheng.knowledge.domain.services.department_knowledge_space_service.batch_check_business_actions", new_callable=AsyncMock, @@ -585,7 +570,7 @@ async def test_get_user_department_spaces_super_admin_sees_all_departments(): }, ) as mock_batch_check, patch( - "bisheng.knowledge.domain.services.department_knowledge_space_service.KnowledgeSpaceService._format_accessible_spaces", + "bisheng.knowledge.domain.services.department_knowledge_space_service.KnowledgeSpaceService._format_basic_spaces", new_callable=AsyncMock, return_value=expected, ) as mock_format, @@ -606,7 +591,6 @@ async def test_get_user_department_spaces_super_admin_sees_all_departments(): mock_format.assert_awaited_once() assert sorted(mock_format.await_args.args[0]) == [101, 202] assert mock_format.await_args.args[1] == "name" - assert mock_format.await_args.kwargs["required_action"] == "visible" def _make_binding(space_id, department_id, *, is_hidden=False): diff --git a/src/backend/test/knowledge/test_file_visible_candidate_pagination.py b/src/backend/test/knowledge/test_file_visible_candidate_pagination.py new file mode 100644 index 0000000000..d0f9650e09 --- /dev/null +++ b/src/backend/test/knowledge/test_file_visible_candidate_pagination.py @@ -0,0 +1,156 @@ +"""F048 candidate-first visibility pagination for space files and folders.""" + +from __future__ import annotations + +from datetime import datetime, timedelta +from unittest.mock import AsyncMock, MagicMock, patch + +import pytest + +from bisheng.common.cursor import decode_cursor +from bisheng.knowledge.domain.models.knowledge_file import FileType, KnowledgeFile +from bisheng.knowledge.domain.services.knowledge_space_service import KnowledgeSpaceService + +_SERVICE = "bisheng.knowledge.domain.services.knowledge_space_service" + + +class _User: + user_id = 41 + tenant_id = 7 + is_global_super = False + + def is_admin(self) -> bool: + return False + + +def _item(file_id: int, *, file_type: int = FileType.FILE.value) -> KnowledgeFile: + return KnowledgeFile( + id=file_id, + user_id=90, + tenant_id=7, + knowledge_id=9, + file_name=f"item-{file_id}.pdf", + file_type=file_type, + file_level_path="", + update_time=datetime(2026, 8, 13, 8, 0) + timedelta(seconds=file_id), + ) + + +async def test_candidate_batches_use_final_child_visible_not_parent_allow() -> None: + service = KnowledgeSpaceService(request=None, login_user=_User()) + folder = _item(1, file_type=FileType.DIR.value) + file_row = _item(2) + visible_check = AsyncMock(return_value={"1": False, "2": True}) + + with ( + patch( + f"{_SERVICE}.batch_check_business_visible", + new=visible_check, + create=True, + ), + patch.object( + service, + "_batch_actions", + new=AsyncMock(side_effect=AssertionError("visible must use the no-admin-shortcut facade")), + ), + ): + result = await service._filter_visible_child_items( + [folder, file_row], + space_id=9, + context={"permissions": {("knowledge_space", "9"): {"visible"}}}, + ) + + assert [row.id for row in result] == [2] + assert visible_check.await_count == 2 + assert {call.kwargs["resource_type"] for call in visible_check.await_args_list} == { + "folder", + "knowledge_file", + } + + +@pytest.mark.parametrize( + ("allowed_ids", "page_size", "expected_ids", "expected_has_more", "expected_cursor_id"), + [ + (set(range(1, 101)) - {5, 17}, 20, list(range(1, 5)) + list(range(6, 17)) + list(range(18, 23)), True, 22), + ({60, 120, 180, 240}, 3, [60, 120, 180], True, 239), + ({60, 120}, 3, [60, 120], False, None), + ], +) +async def test_candidate_scan_refills_page_and_tracks_last_scanned_candidate( + allowed_ids: set[int], + page_size: int, + expected_ids: list[int], + expected_has_more: bool, + expected_cursor_id: int | None, +) -> None: + service = KnowledgeSpaceService(request=None, login_user=_User()) + candidates = [_item(file_id) for file_id in range(1, 251)] + fetch_cursors: list[list | None] = [] + + async def _fetch(*_args, cursor=None, page_size=100, **_kwargs): + fetch_cursors.append(cursor) + start_id = int(cursor[-1]) + 1 if cursor else 1 + return [row for row in candidates if row.id >= start_id][:page_size] + + async def _visible(items, **_kwargs): + return [row for row in items if row.id in allowed_ids] + + metric = MagicMock() + with ( + patch(f"{_SERVICE}.SpaceFileDao.async_list_children", new=AsyncMock(side_effect=_fetch)), + patch.object(service, "_filter_visible_child_items", new=AsyncMock(side_effect=_visible)), + patch(f"{_SERVICE}.emit_metric", new=metric, create=True), + ): + rows, has_more, scan_cursor = await service._scan_visible_child_items( + space_id=9, + parent_id=None, + file_ids=None, + order_field="file_type", + order_sort="asc", + file_status=None, + file_type=None, + page_size=page_size, + ) + + assert [row.id for row in rows] == expected_ids + assert has_more is expected_has_more + assert (scan_cursor[-1] if scan_cursor else None) == expected_cursor_id + assert all(cursor is None or len(cursor) == 4 for cursor in fetch_cursors) + metric.assert_called_once() + assert metric.call_args.kwargs["scanned_candidates"] >= len(rows) + assert metric.call_args.kwargs["returned_items"] == len(rows) + + +async def test_list_cursor_uses_scan_boundary_and_is_retry_stable() -> None: + service = KnowledgeSpaceService(request=None, login_user=_User()) + visible = _item(20) + scan_boundary = [visible.file_type, 1, visible.update_time, 39] + + async def _call() -> str: + with ( + patch.object(service, "_require_read_permission", new=AsyncMock()), + patch.object(service, "_require_action", new=AsyncMock()), + patch.object( + service, + "_scan_visible_child_items", + new=AsyncMock(return_value=([visible], True, scan_boundary)), + ), + patch.object(service, "_enrich_with_version_info", new=AsyncMock()), + patch.object( + service, + "_handle_file_folder_extra_info", + new=AsyncMock(return_value=[{"id": visible.id}]), + ), + ): + page = await service.list_space_children(space_id=9, page_size=1) + assert page.next_cursor is not None + return page.next_cursor + + first = await _call() + second = await _call() + assert first == second + assert decode_cursor( + first, + expected_key_len=4, + expected_context="space_children|order=file_type_asc", + ) == [visible.file_type, 1, visible.update_time.isoformat(), 39] diff --git a/src/backend/test/knowledge/test_knowledge_space_children_cursor.py b/src/backend/test/knowledge/test_knowledge_space_children_cursor.py index 9382921679..909145e4c1 100644 --- a/src/backend/test/knowledge/test_knowledge_space_children_cursor.py +++ b/src/backend/test/knowledge/test_knowledge_space_children_cursor.py @@ -17,13 +17,11 @@ from __future__ import annotations import ast -import inspect import re from pathlib import Path import pytest - -from sqlalchemy import literal, select +from sqlalchemy import select from sqlalchemy.dialects import sqlite from bisheng.knowledge.domain.models.knowledge_space_file import ( @@ -99,11 +97,11 @@ def test_python_and_sql_ext_rank_agree_for_all_extensions(): assert _compute_ext_rank_python(file_name) == expected_rank -def test_ext_priority_table_length_is_15(): - """Spec §6.2: pdf=1 .. html=15, exactly 15 ranked extensions.""" - assert len(_EXT_PRIORITIES) == 15 +def test_ext_priority_table_length_is_16(): + """The SQL/Python cursor rank table includes OFD after the original 15.""" + assert len(_EXT_PRIORITIES) == 16 assert _EXT_PRIORITIES[0] == ("pdf", 1) - assert _EXT_PRIORITIES[-1] == ("html", 15) + assert _EXT_PRIORITIES[-1] == ("ofd", 16) # --------------------------------------------------------------------------- @@ -142,8 +140,8 @@ def test_scan_visible_child_items_uses_cursor_fetch_until_enough(): assert "batch_cursor" in func_src assert "_compute_ext_rank_python" in func_src - # Fetch-until-enough: break when visible exceeds page_size+1 probe - assert re.search(r"len\(visible_page_items\)\s*>\s*page_size", func_src) + # Fetch-until-enough: the next visible candidate is the has_more probe. + assert re.search(r"len\(visible_page_items\)\s*==\s*page_size", func_src) def test_scan_visible_child_items_returns_data_and_has_more(): @@ -192,8 +190,9 @@ def test_list_space_children_returns_page_infinite_cursor_data(): func_src = ast.get_source_segment(src, func) assert "PageInfiniteCursorData(" in func_src assert "encode_cursor(" in func_src - # Cursor key shape: 4-tuple including ext_rank - assert "_compute_ext_rank_python" in func_src + # Cursor key is the 4-element last-scanned candidate boundary returned by the scanner. + assert "scan_cursor" in func_src + assert "encode_cursor(scan_cursor" in func_src # --------------------------------------------------------------------------- diff --git a/src/backend/test/knowledge/test_space_joined_visible_ids.py b/src/backend/test/knowledge/test_space_joined_visible_ids.py new file mode 100644 index 0000000000..e4fbcb3bfe --- /dev/null +++ b/src/backend/test/knowledge/test_space_joined_visible_ids.py @@ -0,0 +1,270 @@ +"""F048 joined-space listing uses the complete personal visible-ID set.""" + +from __future__ import annotations + +from contextlib import asynccontextmanager +from datetime import datetime, timedelta +from types import SimpleNamespace +from unittest.mock import AsyncMock, patch + +import pytest +from sqlalchemy.dialects import sqlite + +from bisheng.common.errcode.permission import PermissionEnumerationIncompleteError +from bisheng.knowledge.domain.models import knowledge as knowledge_module +from bisheng.knowledge.domain.models.knowledge import ( + Knowledge, + KnowledgeDao, + KnowledgeState, + KnowledgeTypeEnum, +) +from bisheng.knowledge.domain.services.knowledge_space_service import KnowledgeSpaceService +from bisheng.permission.domain.schemas import ( + VisibilityEnumerationStatus, + VisibleObjectEnumerationResult, +) + +_SERVICE = "bisheng.knowledge.domain.services.knowledge_space_service" + + +class _User: + user_id = 41 + tenant_id = 7 + is_global_super = False + + def is_admin(self) -> bool: + return False + + +def _space(space_id: int, *, creator: int, minutes: int) -> Knowledge: + timestamp = datetime(2026, 8, 13, 8, 0) + timedelta(minutes=minutes) + return Knowledge( + id=space_id, + user_id=creator, + tenant_id=7, + name=f"space-{space_id}", + type=KnowledgeTypeEnum.SPACE.value, + state=KnowledgeState.PUBLISHED.value, + create_time=timestamp, + update_time=timestamp, + ) + + +def _visible(*object_ids: str) -> VisibleObjectEnumerationResult: + return VisibleObjectEnumerationResult( + resource_type="knowledge_space", + object_ids=object_ids, + max_results=5_000, + status=VisibilityEnumerationStatus.NORMAL, + ) + + +def _actor() -> SimpleNamespace: + return SimpleNamespace( + user_id=_User.user_id, + current_tenant_id=_User.tenant_id, + super_admin=False, + tenant_admin_tenant_ids=frozenset(), + ) + + +async def test_joined_repository_applies_tenant_active_type_creator_and_stable_order_filters( + monkeypatch: pytest.MonkeyPatch, +) -> None: + session = SimpleNamespace() + captured = None + + class _Rows: + @staticmethod + def all() -> list[Knowledge]: + return [] + + async def _exec(statement): + nonlocal captured + captured = statement + return _Rows() + + session.exec = _exec + + @asynccontextmanager + async def _session_context(): + yield session + + monkeypatch.setattr(knowledge_module, "get_async_db_session", _session_context) + await KnowledgeDao.async_get_joined_spaces_by_visible_ids( + [10, 20], + tenant_id=7, + exclude_creator_id=41, + order_by="update_time", + ) + + sql = str( + captured.compile( + dialect=sqlite.dialect(), + compile_kwargs={"literal_binds": True}, + ) + ).lower() + compact = " ".join(sql.split()) + assert "knowledge.id in (10, 20)" in compact + assert "knowledge.tenant_id = 7" in compact + assert f"knowledge.type = {KnowledgeTypeEnum.SPACE.value}" in compact + assert f"knowledge.state = {KnowledgeState.PUBLISHED.value}" in compact + assert "knowledge.user_id != 41" in compact + assert "order by knowledge.update_time desc, knowledge.id desc" in compact + + +async def test_joined_uses_all_visible_sources_and_only_lightweight_db_results() -> None: + """The source kind is irrelevant once every source projects canonical visible.""" + + service = KnowledgeSpaceService(request=None, login_user=_User()) + runtime = SimpleNamespace( + list_visible_objects=AsyncMock( + return_value=_visible("101", "102", "103", "104", "105") + ) + ) + rows = [ + _space(105, creator=99, minutes=5), # another valid projected source + _space(104, creator=99, minutes=4), # manual subscription + _space(103, creator=99, minutes=3), # group grant + _space(102, creator=99, minutes=2), # department grant + _space(101, creator=99, minutes=1), # direct grant + ] + + with ( + patch.object(service, "_permission_actor", new=AsyncMock(return_value=_actor())), + patch(f"{_SERVICE}.get_f048_runtime", new=AsyncMock(return_value=runtime)), + patch( + f"{_SERVICE}.KnowledgeDao.async_get_joined_spaces_by_visible_ids", + new=AsyncMock(return_value=rows), + ) as list_rows, + patch( + f"{_SERVICE}.KnowledgeSpaceUserPinDao.list_pinned_space_ids", + new=AsyncMock(return_value=set()), + ), + patch( + f"{_SERVICE}.SpaceChannelMemberDao.async_get_user_followed_members", + new=AsyncMock(side_effect=AssertionError("joined must not read membership")), + ), + patch.object( + service, + "_batch_actions", + new=AsyncMock(side_effect=AssertionError("joined must not repeat visible or read actions")), + ), + patch.object( + service, + "_populate_root_file_counts", + new=AsyncMock(side_effect=AssertionError("joined must not count files")), + ), + patch.object( + service, + "_decorate_department_metadata", + new=AsyncMock(side_effect=AssertionError("joined must not load department metadata")), + ), + ): + result = await service.get_my_followed_spaces("update_time") + + runtime.list_visible_objects.assert_awaited_once() + call = runtime.list_visible_objects.await_args + assert call.kwargs["resource_type"] == "knowledge_space" + assert call.kwargs["max_results"] == 5_000 + assert call.args[0].user_id == _User.user_id + assert call.args[0].current_tenant_id == _User.tenant_id + list_rows.assert_awaited_once_with( + [101, 102, 103, 104, 105], + tenant_id=7, + exclude_creator_id=41, + order_by="update_time", + ) + assert [item.id for item in result] == [105, 104, 103, 102, 101] + assert all( + key not in item.model_dump() + for item in result + for key in ("user_role", "file_num", "department_id", "department_name") + ) + assert all(item.actions is None for item in result) + + +async def test_joined_chunks_visible_ids_and_excludes_canonical_creator() -> None: + service = KnowledgeSpaceService(request=None, login_user=_User()) + visible_ids = tuple(str(index) for index in range(1, 1_002)) + runtime = SimpleNamespace(list_visible_objects=AsyncMock(return_value=_visible(*visible_ids))) + owned = _space(1, creator=_User.user_id, minutes=20) + other_old = _space(500, creator=88, minutes=10) + other_new = _space(1_001, creator=88, minutes=30) + db_calls: list[list[int]] = [] + + async def _list_chunk( + ids: list[int], + *, + tenant_id: int, + exclude_creator_id: int, + order_by: str, + ) -> list[Knowledge]: + assert tenant_id == 7 + assert exclude_creator_id == 41 + assert order_by == "update_time" + db_calls.append(ids) + return [row for row in (owned, other_old, other_new) if row.id in ids and row.user_id != 41] + + with ( + patch.object(service, "_permission_actor", new=AsyncMock(return_value=_actor())), + patch(f"{_SERVICE}.get_f048_runtime", new=AsyncMock(return_value=runtime)), + patch( + f"{_SERVICE}.KnowledgeDao.async_get_joined_spaces_by_visible_ids", + new=AsyncMock(side_effect=_list_chunk), + ), + patch( + f"{_SERVICE}.KnowledgeSpaceUserPinDao.list_pinned_space_ids", + new=AsyncMock(return_value={500}), + ), + ): + result = await service.get_my_followed_spaces("update_time") + + assert [len(chunk) for chunk in db_calls] == [500, 500, 1] + assert [item.id for item in result] == [500, 1_001] + assert [item.is_pinned for item in result] == [True, False] + + +@pytest.mark.parametrize("identity_kind", ["ordinary", "super_admin", "tenant_admin"]) +async def test_joined_admin_identities_use_the_same_personal_visible_enumeration( + identity_kind: str, +) -> None: + user = _User() + if identity_kind == "super_admin": + user.is_global_super = True + runtime = SimpleNamespace(list_visible_objects=AsyncMock(return_value=_visible())) + actor = SimpleNamespace( + user_id=user.user_id, + current_tenant_id=user.tenant_id, + super_admin=identity_kind == "super_admin", + tenant_admin_tenant_ids=(frozenset({user.tenant_id}) if identity_kind == "tenant_admin" else frozenset()), + ) + service = KnowledgeSpaceService(request=None, login_user=user) + + with ( + patch.object(service, "_permission_actor", new=AsyncMock(return_value=actor)), + patch(f"{_SERVICE}.get_f048_runtime", new=AsyncMock(return_value=runtime)), + ): + assert await service.get_my_followed_spaces() == [] + + runtime.list_visible_objects.assert_awaited_once_with( + actor, + resource_type="knowledge_space", + max_results=5_000, + ) + + +@pytest.mark.parametrize("reason", ["stream failed", "5,001 visible objects"]) +async def test_joined_propagates_incomplete_visible_enumeration_as_25014(reason: str) -> None: + service = KnowledgeSpaceService(request=None, login_user=_User()) + failure = PermissionEnumerationIncompleteError(msg=reason) + runtime = SimpleNamespace(list_visible_objects=AsyncMock(side_effect=failure)) + + with ( + patch.object(service, "_permission_actor", new=AsyncMock(return_value=_actor())), + patch(f"{_SERVICE}.get_f048_runtime", new=AsyncMock(return_value=runtime)), + ): + with pytest.raises(PermissionEnumerationIncompleteError) as exc_info: + await service.get_my_followed_spaces() + + assert exc_info.value.Code == 25014 diff --git a/src/backend/test/knowledge/test_space_listing_pin_source.py b/src/backend/test/knowledge/test_space_listing_pin_source.py index 05acb70377..20de1eb564 100644 --- a/src/backend/test/knowledge/test_space_listing_pin_source.py +++ b/src/backend/test/knowledge/test_space_listing_pin_source.py @@ -91,15 +91,29 @@ async def test_member_spaces_pin_decoupled_from_member_row(): f"{_KS}.KnowledgeSpaceUserPinDao.list_pinned_space_ids", new=AsyncMock(return_value={300}), ), - patch.object( - service, - "_populate_root_file_counts", - new_callable=AsyncMock, - ), - patch.object(service, "_decorate_department_metadata", new=AsyncMock(side_effect=lambda x: x)), ): result = await service._format_member_spaces([member], "name") assert len(result) == 1 assert result[0].id == 300 assert result[0].is_pinned is True + assert "file_num" not in result[0].model_dump() + assert "space_kind" not in result[0].model_dump() + + +async def test_basic_spaces_only_adds_pin_state_and_preserves_database_order(): + service = _service() + spaces = [_space(100, "alpha", user_id=9), _space(200, "beta", user_id=9)] + + with ( + patch(f"{_KS}.KnowledgeDao.async_get_spaces_by_ids", new=AsyncMock(return_value=spaces)), + patch( + f"{_KS}.KnowledgeSpaceUserPinDao.list_pinned_space_ids", + new=AsyncMock(return_value={200}), + ), + ): + result = await service._format_basic_spaces([100, 200], "name") + + assert [item.id for item in result] == [200, 100] + assert all("file_num" not in item.model_dump() for item in result) + assert all("space_kind" not in item.model_dump() for item in result) diff --git a/src/backend/test/linsight/_e2e_abc_runner.py b/src/backend/test/linsight/_e2e_abc_runner.py index 2fdf66a97f..dbb3007293 100644 --- a/src/backend/test/linsight/_e2e_abc_runner.py +++ b/src/backend/test/linsight/_e2e_abc_runner.py @@ -153,7 +153,7 @@ async def track_a() -> None: backend = WorkspaceBackend(svid=svid, minio=get_minio_storage_sync(), file_dir=tmp) with patch( "bisheng.linsight.domain.services.agent_factory._resolve_model", - return_value=GenericFakeChatModel(messages=iter([])), + return_value=(GenericFakeChatModel(messages=iter([])), True), ): agent = await create_linsight_agent( session_model=session_model, diff --git a/src/backend/test/linsight/_e2e_llm_resilience_runner.py b/src/backend/test/linsight/_e2e_llm_resilience_runner.py index fc94ad2bd0..5d6f9c43ae 100644 --- a/src/backend/test/linsight/_e2e_llm_resilience_runner.py +++ b/src/backend/test/linsight/_e2e_llm_resilience_runner.py @@ -161,7 +161,7 @@ async def _build_linsight_agent(model): # middleware, astream — is the real path. (settings is a pydantic model, so we # patch the class method, not the instance attribute.) with ( - patch.object(agent_factory, "_resolve_model", return_value=model), + patch.object(agent_factory, "_resolve_model", return_value=(model, True)), patch.object(ConfigService, "get_all_config", return_value={}), ): agent = await agent_factory.create_linsight_agent( diff --git a/src/backend/test/linsight/test_binary_content_guard.py b/src/backend/test/linsight/test_binary_content_guard.py index 4ccd9e8848..3dc24ebd07 100644 --- a/src/backend/test/linsight/test_binary_content_guard.py +++ b/src/backend/test/linsight/test_binary_content_guard.py @@ -179,7 +179,7 @@ async def handler(req): async def test_image_blocks_preserved(): - mw = ModelContentGuardMiddleware() + mw = ModelContentGuardMiddleware(supports_vision=True) image_block = {"type": "image_url", "image_url": {"url": "data:image/png;base64,AA"}} request = FakeModelRequest(messages=[HumanMessage(content=[image_block])]) seen = {} @@ -260,7 +260,7 @@ async def test_backend_binary_error_without_code_interpreter(): async def test_ordinary_tool_error_passes_through(): """Only OUR binary marker is upgraded; a plain not-found must stay verbatim.""" msg = ToolMessage(content="Error: File '/output/nope.md' not found", name="read_file", tool_call_id="c1") - mw = BinaryReadGuardMiddleware(has_code_interpreter=True) + mw = BinaryReadGuardMiddleware(has_code_interpreter=True, supports_vision=True) result = await mw.awrap_tool_call(read_call("/output/nope.md"), handler_returning(msg)) assert result.content == "Error: File '/output/nope.md' not found" @@ -277,7 +277,7 @@ async def test_valid_image_block_survives_read_guard(): name="read_file", tool_call_id="c1", ) - mw = BinaryReadGuardMiddleware(has_code_interpreter=True) + mw = BinaryReadGuardMiddleware(has_code_interpreter=True, supports_vision=True) result = await mw.awrap_tool_call(read_call("/output/chart.png"), handler_returning(msg)) assert isinstance(result.content, list) @@ -300,7 +300,7 @@ async def test_corrupt_image_block_replaced_by_read_guard(): async def test_corrupt_image_stripped_before_model_call(): - mw = ModelContentGuardMiddleware(has_code_interpreter=False) + mw = ModelContentGuardMiddleware(has_code_interpreter=False, supports_vision=True) request = FakeModelRequest( messages=[HumanMessage(content=[{"type": "image", "base64": "���", "mime_type": "image/png"}])] ) @@ -321,7 +321,7 @@ async def handler(req): async def test_http_image_url_is_not_flagged(): """No inline payload to validate — a plain URL must pass untouched.""" - mw = ModelContentGuardMiddleware() + mw = ModelContentGuardMiddleware(supports_vision=True) block = {"type": "image_url", "image_url": {"url": "https://example.com/a.png"}} request = FakeModelRequest(messages=[HumanMessage(content=[block])]) seen = {} @@ -334,11 +334,234 @@ async def handler(req): assert seen["messages"][0].content == [block] +# -------------------------------------------------------------------------- +# Layer 2 — tool-message images are relocated into a user turn +# -------------------------------------------------------------------------- + + +def image_tool_message(path="/scratch/S1_p5.png", tool_call_id="c1", payload="aGk="): + """The exact shape deepagents' `read_file` returns for an image + (`deepagents/middleware/filesystem.py`): a multimodal block in the TOOL role.""" + return ToolMessage( + content_blocks=[{"type": "image", "base64": payload, "mime_type": "image/png"}], + name="read_file", + tool_call_id=tool_call_id, + additional_kwargs={"read_file_path": path, "read_file_media_type": "image/png"}, + ) + + +def ai_tool_call(*ids): + return AIMessage( + content="", + tool_calls=[{"name": "read_file", "args": {"file_path": f"/scratch/{i}.png"}, "id": i} for i in ids], + ) + + +async def collect(mw, messages): + request = FakeModelRequest(messages=messages) + seen = {} + + async def handler(req): + seen["messages"] = req.messages + return AIMessage(content="ok") + + await mw.awrap_model_call(request, handler) + return request, seen["messages"] + + +async def test_tool_image_relocated_into_a_user_turn(): + """Regression: Kimi K3 answers `image_url parts are supported only in user + messages` with a 400 that fails the whole session. Every mainstream endpoint + accepts an image in the user role, so the block is MOVED rather than dropped + (dropping it would regress the scanned-page workflow this feature exists for).""" + mw = ModelContentGuardMiddleware(supports_vision=True) + request, out = await collect(mw, [ai_tool_call("c1"), image_tool_message()]) + + assert len(out) == 3 + + tool_payload = _convert_message_to_dict(out[1]) + assert tool_payload["role"] == "tool" + # the tool result survives as non-empty text, and names the file it read + assert all(b["type"] == "text" for b in tool_payload["content"]) + assert "/scratch/S1_p5.png" in tool_payload["content"][0]["text"] + + carrier = _convert_message_to_dict(out[2]) + assert carrier["role"] == "user" + assert [b["type"] for b in carrier["content"]] == ["text", "text", "image_url"] + assert carrier["content"][2]["image_url"]["url"] == "data:image/png;base64,aGk=" + + # request-only: the state/checkpoint copy keeps the original shape, so this is + # reversible and a rollback cannot leave a rewritten history behind. + assert request.messages[1].content[0]["type"] == "image" + + +async def test_carrier_goes_after_the_whole_tool_batch(): + """Ordering rule: every tool message answering one `tool_calls` batch must be + contiguous and precede any other role. Inserting the carrier directly after the + image-bearing tool message would split the batch — trading a 400 for a 400.""" + mw = ModelContentGuardMiddleware(supports_vision=True) + _, out = await collect( + mw, + [ + ai_tool_call("c1", "c2"), + image_tool_message(tool_call_id="c1"), + ToolMessage(content="plain text result", name="read_file", tool_call_id="c2"), + AIMessage(content="done"), + ], + ) + + assert [_convert_message_to_dict(m)["role"] for m in out] == ["assistant", "tool", "tool", "user", "assistant"] + + +async def test_each_tool_batch_gets_its_own_carrier(): + mw = ModelContentGuardMiddleware(supports_vision=True) + _, out = await collect( + mw, + [ + ai_tool_call("c1"), + image_tool_message(tool_call_id="c1"), + ai_tool_call("c2"), + image_tool_message(path="/scratch/b.png", tool_call_id="c2"), + ], + ) + + assert [_convert_message_to_dict(m)["role"] for m in out] == [ + "assistant", + "tool", + "user", + "assistant", + "tool", + "user", + ] + + +async def test_multiple_relocated_images_keep_order(): + mw = ModelContentGuardMiddleware(supports_vision=True) + _, out = await collect( + mw, + [ + ai_tool_call("c1", "c2"), + image_tool_message(path="/scratch/a.png", tool_call_id="c1", payload="YQ=="), + image_tool_message(path="/scratch/b.png", tool_call_id="c2", payload="Yg=="), + ], + ) + + carrier = out[3] + labels = [b["text"] for b in carrier.content if b["type"] == "text"] + assert "/scratch/a.png" in labels[1] + assert "/scratch/b.png" in labels[2] + assert [b["base64"] for b in carrier.content if b["type"] == "image"] == ["YQ==", "Yg=="] + + +async def test_openai_native_image_url_in_a_tool_message_is_relocated(): + """The other shape an image arrives in — same role problem, same fix.""" + mw = ModelContentGuardMiddleware(supports_vision=True) + block = {"type": "image_url", "image_url": {"url": "data:image/png;base64,aGk="}} + _, out = await collect( + mw, + [ai_tool_call("c1"), ToolMessage(content=[block], name="read_file", tool_call_id="c1")], + ) + + assert _convert_message_to_dict(out[1])["role"] == "tool" + assert all(b["type"] == "text" for b in out[1].content) + assert out[2].content[-1] == block + + +async def test_user_images_are_not_relocated(): + """Only the tool role is the problem — a human attachment is already legal and + must not gain a spurious carrier turn.""" + mw = ModelContentGuardMiddleware(supports_vision=True) + block = {"type": "image", "base64": "aGk=", "mime_type": "image/png"} + _, out = await collect(mw, [HumanMessage(content=[block])]) + + assert len(out) == 1 + assert out[0].content == [block] + + +async def test_corrupt_tool_image_is_stripped_not_relocated(): + """A payload that is not real base64 stays useless in any role: moving it would + only relocate the 400. It keeps the strip path, and adds no carrier turn.""" + mw = ModelContentGuardMiddleware(has_code_interpreter=False, supports_vision=True) + _, out = await collect( + mw, + [ + ai_tool_call("c1"), + ToolMessage( + content=[{"type": "image", "base64": "���", "mime_type": "image/png"}], + name="read_file", + tool_call_id="c1", + ), + ], + ) + + assert len(out) == 2 + assert out[1].content[0]["type"] == "text" + assert "base64" in out[1].content[0]["text"] + + +# -------------------------------------------------------------------------- +# The `visual` gate — a model with no declared vision capability gets no image +# -------------------------------------------------------------------------- + + +async def test_image_read_refused_when_the_model_has_no_vision(): + """`WSModel.visual` off means the endpoint would reject the image (or the model + would see nothing). Refuse at the TOOL layer, where the hint can name why.""" + mw = BinaryReadGuardMiddleware(has_code_interpreter=True, supports_vision=False) + msg = image_tool_message(path="/scratch/page5.png") + result = await mw.awrap_tool_call(read_call("/scratch/page5.png"), handler_returning(msg)) + + assert isinstance(result.content, str) + assert "/scratch/page5.png" in result.content + assert "视觉" in result.content # names the checkbox the admin has to tick + assert "bisheng_code_interpreter" in result.content # the route that still works + assert result.additional_kwargs == {} + + +async def test_no_vision_hint_forbids_guessing(): + """Honesty over a plausible answer: a model told nothing will describe the page + anyway. The hint must demand it says the check did not happen.""" + mw = BinaryReadGuardMiddleware(has_code_interpreter=True, supports_vision=False) + result = await mw.awrap_tool_call(read_call("/scratch/p.png"), handler_returning(image_tool_message())) + + assert "不要凭猜测描述图片内容" in result.content + assert "never guess" in result.content + + +async def test_no_vision_hint_omits_unbound_code_interpreter(): + """prompt ⟺ tool lockstep, same rule as the binary hint.""" + mw = BinaryReadGuardMiddleware(has_code_interpreter=False, supports_vision=False) + result = await mw.awrap_tool_call(read_call("/scratch/p.png"), handler_returning(image_tool_message())) + + assert "bisheng_code_interpreter" not in result.content + + +async def test_no_vision_model_call_strips_images_in_any_role(): + """Backstop for images the tool guard never saw — the real case is a replayed + checkpoint from a turn that ran on a vision model before the admin switched it.""" + mw = ModelContentGuardMiddleware(supports_vision=False) + block = {"type": "image", "base64": "aGk=", "mime_type": "image/png"} + _, out = await collect(mw, [HumanMessage(content=[block]), ai_tool_call("c1"), image_tool_message()]) + + assert len(out) == 3 # no carrier turn was added + assert out[0].content[0]["type"] == "text" + assert out[2].content[0]["type"] == "text" + assert all("image_url" not in str(_convert_message_to_dict(m)) for m in out) + + +def test_guards_fail_closed_on_vision(): + """Both guards default to no-vision: a call site that forgets the flag must lose + image reads, never leak a payload the endpoint would reject.""" + assert BinaryReadGuardMiddleware()._supports_vision is False + assert ModelContentGuardMiddleware()._supports_vision is False + + def test_build_binary_guards_pairs_both_layers(): """A new subagent must get both guards by construction — its subgraph is never wrapped by the parent's middleware.""" from bisheng.linsight.domain.services.binary_content_guard import build_binary_guards - guards = build_binary_guards(has_code_interpreter=True) + guards = build_binary_guards(has_code_interpreter=True, supports_vision=True) assert [type(g).__name__ for g in guards] == ["BinaryReadGuardMiddleware", "ModelContentGuardMiddleware"] assert all(g._has_code_interpreter for g in guards) + assert all(g._supports_vision for g in guards) diff --git a/src/backend/test/linsight/test_call_id_uniqueness.py b/src/backend/test/linsight/test_call_id_uniqueness.py new file mode 100644 index 0000000000..6b0a2b9fae --- /dev/null +++ b/src/backend/test/linsight/test_call_id_uniqueness.py @@ -0,0 +1,136 @@ +"""Regression: providers whose tool_call ids are only unique WITHIN one response +must not lose execution history. + +Session ``8a570723…`` on 114 (kimi-k3 via tokenrouter) issued 156 +``bisheng_code_interpreter`` calls, 9 ``read_file`` and 2 ``write_file`` across 171 +model turns — yet ``linsight_execute_task.history`` ended up holding 4 tool rows, +and the step flow read "运行代码 1 次 · 编辑 1 个文件". + +Cause: tokenrouter mints ``:`` and restarts the index at 0 on +every response (probing its ``/chat/completions`` returns ``read_file:0`` / +``read_file:1``). That is legal — the OpenAI contract only requires the id to be +the handle a ToolMessage refers back to, never a cross-request guarantee. But +``add_execution_task_step`` upserts by call_id across the WHOLE history, so each +call silently overwrote its predecessor. + +Fix: the mapper mints the id it emits (``#:``) while its own +bookkeeping dicts stay keyed by the provider's value — that is what comes back on +``ToolMessage.tool_call_id``. These tests drive the real mapper into the real +persistence upsert, which nothing covered end-to-end before. +""" + +from langchain_core.messages import AIMessage, ToolMessage + +from bisheng.linsight.domain.services.stream_event_mapper import StreamEventMapper +from bisheng_langchain.linsight.event import ExecStep +from test.linsight.test_step_persistence import _make_manager + +SVID = "8a570723-0838-47a7-b2c0-df2a3be9879f" + + +def _steps(mapper: StreamEventMapper, message) -> list[ExecStep]: + return [e for e in mapper.normalize("messages", (message, {})) if isinstance(e, ExecStep)] + + +def _call_round(mapper: StreamEventMapper, *, tool: str, raw_id: str, args: dict, output: str) -> list[ExecStep]: + """One model turn: the AIMessage announcing a tool call, then its result.""" + start = AIMessage(content="", tool_calls=[{"id": raw_id, "name": tool, "args": args}]) + end = ToolMessage(content=output, tool_call_id=raw_id, name=tool) + return _steps(mapper, start) + _steps(mapper, end) + + +async def _persist(mgr, steps: list[ExecStep]) -> None: + for step in steps: + await mgr.add_execution_task_step("t1", step) + + +async def test_repeated_provider_id_keeps_one_row_per_call(monkeypatch): + """The exact 114 failure: N turns reusing ``bisheng_code_interpreter:0``. + + Each turn is a start+end pair that must fold into ONE row (that part always + worked); what regressed is that turn N+1 used to overwrite turn N's row. + """ + mgr, store = _make_manager(monkeypatch) + mapper = StreamEventMapper(svid=SVID) + + rounds = 6 + for i in range(rounds): + await _persist( + mgr, + _call_round( + mapper, + tool="bisheng_code_interpreter", + raw_id="bisheng_code_interpreter:0", # provider restarts at 0 each turn + args={"python_code": f"print({i})"}, + output=f"out-{i}", + ), + ) + + assert len(store["history"]) == rounds, "each tool call must own a history row" + # ...and in call order, with nothing overwritten. + assert [h["output"] for h in store["history"]] == [f"out-{i}" for i in range(rounds)] + + +async def test_ids_do_not_collide_across_mapper_instances(monkeypatch): + """A mapper is built fresh per RUN of one session version (first execute / + ask_user resume / follow-up turn), all sharing one svid. A per-instance + counter alone would restart at 1 and overwrite the earlier run's rows, so the + id also carries ``run_token`` — same reasoning as thinking segments. + """ + mgr, store = _make_manager(monkeypatch) + + for run, output in ((StreamEventMapper(svid=SVID), "first-run"), (StreamEventMapper(svid=SVID), "resumed-run")): + await _persist( + mgr, + _call_round( + run, tool="read_file", raw_id="read_file:0", args={"file_path": "/uploads/a.md"}, output=output + ), + ) + + assert len(store["history"]) == 2 + assert [h["output"] for h in store["history"]] == ["first-run", "resumed-run"] + + +async def test_parallel_calls_in_one_response_stay_separate(monkeypatch): + """Two calls inside a SINGLE response: here the provider's ids do differ + (``read_file:0`` / ``read_file:1``), and each must still end its own row — + this is the case that rules out "only merge against history[-1]" as a fix. + """ + mgr, store = _make_manager(monkeypatch) + mapper = StreamEventMapper(svid=SVID) + + start = AIMessage( + content="", + tool_calls=[ + {"id": "read_file:0", "name": "read_file", "args": {"file_path": "/uploads/a.md"}}, + {"id": "read_file:1", "name": "read_file", "args": {"file_path": "/uploads/b.md"}}, + ], + ) + await _persist(mgr, _steps(mapper, start)) + # Results come back interleaved-last, as they do on the wire. + for raw_id, output in (("read_file:0", "内容A"), ("read_file:1", "内容B")): + await _persist(mgr, _steps(mapper, ToolMessage(content=output, tool_call_id=raw_id, name="read_file"))) + + assert len(store["history"]) == 2 + assert sorted(h["output"] for h in store["history"]) == ["内容A", "内容B"] + assert all(h["status"] == "end" for h in store["history"]), "end frame must supersede its start" + + +async def test_globally_unique_provider_ids_are_unaffected(monkeypatch): + """Guard on the promise made when choosing this fix: a provider that already + returns globally-unique ids (OpenAI's ``call_xxx``) behaves exactly as before — + one row per call, start/end still merged. Only the stored id string changes. + """ + mgr, store = _make_manager(monkeypatch) + mapper = StreamEventMapper(svid=SVID) + + for i in range(3): + await _persist( + mgr, + _call_round(mapper, tool="search", raw_id=f"call_abc{i}", args={"q": str(i)}, output=f"r-{i}"), + ) + + assert len(store["history"]) == 3 + assert [h["output"] for h in store["history"]] == ["r-0", "r-1", "r-2"] + # The provider's id stays greppable as the prefix. + assert [h["call_id"].split("#")[0] for h in store["history"]] == ["call_abc0", "call_abc1", "call_abc2"] diff --git a/src/backend/test/linsight/test_deferred_ingest.py b/src/backend/test/linsight/test_deferred_ingest.py new file mode 100644 index 0000000000..742e83d1a5 --- /dev/null +++ b/src/backend/test/linsight/test_deferred_ingest.py @@ -0,0 +1,507 @@ +"""Attachment ingestion left the submit request; only the metadata gate stayed. + +Parsing one attachment runs the full ETL (600s timeout per file). A production +task with 12 PDFs spent 19 minutes inside ``submit_user_question`` — trace +8467e765, 12:37→12:56, the whole window inside ``knowledge.rag.pipeline`` — and +nginx cut the HTTP request at 300s, so the browser never got the handoff and the +session sat at NOT_STARTED with nobody to pick it up. Submit now parks the raw +refs in ``pending_files`` and the worker materializes them before the run. + +What these tests hold in place, request side: + + * submit touches NO storage — no MinIO, no Redis, no ETL — and ``files`` stays + None, because its contract ("ingested and usable") is what every downstream + reader depends on; + * the pure-metadata checks did NOT move: 11021/11022/11023 and the "file still + parsing" rejection are what the frontend branches on, and minutes later they + would only be a generic task failure; + * ``ingest_pending_files`` REBINDS ``files`` — ``JsonType`` carries no + ``MutableList`` wrapper, so an in-place mutation emits no UPDATE and the + whole ingest evaporates on commit; + * ``amget`` drops misses, so its result is never paired positionally. + +Worker-side ordering / degradation lives in ``test_deferred_ingest_worker.py``. +External services are faked; no live middleware required. +""" + +from __future__ import annotations + +import sys +from types import ModuleType, SimpleNamespace +from unittest.mock import AsyncMock, MagicMock, patch + +import pytest + +from bisheng.api.v1.schema.chat_schema import APIChatCompletion +from bisheng.common.errcode.linsight import ( + LinsightFolderDepthExceededError, + LinsightFolderFileCountExceededError, + LinsightFolderTotalSizeExceededError, +) +from bisheng.database.models.message import ChatMessage, ChatMessageDao +from bisheng.database.models.session import MessageSession, MessageSessionDao +from bisheng.linsight.domain import utils as linsight_execute_utils +from bisheng.linsight.domain.models.linsight_session_version import LinsightSessionVersionDao +from bisheng.linsight.domain.schemas.linsight_schema import LinsightQuestionSubmitSchema, SubmitFileSchema +from bisheng.linsight.domain.services import workbench_impl +from bisheng.linsight.domain.services.workbench_impl import LinsightWorkbenchImpl + +# The tenant the fake DAO stamps on insert, standing in for the multi-tenant +# listener: the queue item is the worker's ONLY source of tenant context. +FAKE_TENANT_ID = 42 + + +class FakeMinio: + """Object store with the surface ingestion uses — and a readable ``store``.""" + + def __init__(self) -> None: + self.bucket = "bisheng" + self.tmp_bucket = "tmp-dir" + self.store: dict[tuple[str, str], bytes] = {} + + async def object_exists(self, bucket_name=None, object_name=None): + return (bucket_name or self.bucket, object_name) in self.store + + async def copy_object(self, source_bucket=None, source_object=None, dest_bucket=None, dest_object=None): + src = self.store.get((source_bucket or self.tmp_bucket, source_object), b"parsed-md") + self.store[(dest_bucket or self.bucket, dest_object)] = src + + async def get_object(self, bucket_name=None, object_name=None): + return self.store.get((bucket_name or self.bucket, object_name)) + + async def put_object(self, *, bucket_name=None, object_name, file, **kwargs): + self.store[(bucket_name or self.bucket, object_name)] = file if isinstance(file, bytes) else bytes(file) + + +def _legacy_file(file_id: str = "f1", name: str = "a.pdf", status: str = "completed") -> SubmitFileSchema: + """A linsight-pipeline upload: its parsed markdown is looked up in Redis.""" + return SubmitFileSchema(file_id=file_id, file_name=name, parsing_status=status) + + +def _daily_file(file_id: str = "d1", name: str = "report.pdf") -> SubmitFileSchema: + """A daily-bucket upload: this is the one that drags the whole ETL in.""" + return SubmitFileSchema( + file_id=file_id, + file_name=name, + parsing_status="completed", + file_url=f"/tmp-dir/{file_id}.pdf?X-Amz-Algorithm=AWS4", + ) + + +def _temp_info(file_id: str, name: str) -> dict: + return { + "file_id": file_id, + "original_filename": name, + "parsing_status": "completed", + "markdown_filename": f"{file_id}.md", + "markdown_file_path": f"{file_id}.md", + } + + +@pytest.fixture +def submit_env(monkeypatch: pytest.MonkeyPatch): + """Isolate submit from the DB and wire up probes on every ingestion door. + + The probes are the assertion surface: a regression that puts parsing back in + the request shows up as bytes in ``minio.store``, a Redis read, a download or + a constructed ``TempFilePipeline`` — not as a mock call count on a seam that + may well have been renamed by then. + """ + env: dict = {"messages": [], "redis_reads": [], "downloads": [], "pipelines": []} + minio = FakeMinio() + env["minio"] = minio + + async def _fake_insert_session(_cls, data: MessageSession): + env["session"] = data + return data + + async def _fake_insert_version(_cls, data): + # The tenant listener stamps this on the real INSERT; the worker reads it + # back off the queue item, so the fake has to stamp it too. + data.tenant_id = FAKE_TENANT_ID + env["version"] = data + return data + + async def _fake_insert_message(_cls, data: ChatMessage): + env["messages"].append(data) + return data + + fake_redis = AsyncMock() + + async def _record_amget(keys): + env["redis_reads"].extend(keys) + return [] + + async def _record_aget(key): + env["redis_reads"].append(key) + return None + + fake_redis.amget = AsyncMock(side_effect=_record_amget) + fake_redis.aget = AsyncMock(side_effect=_record_aget) + env["redis"] = fake_redis + + async def _record_download(url, *args, **kwargs): + env["downloads"].append(url) + return ("/tmp/never-used", "never-used") + + class _RecordingPipeline: + def __init__(self, *args, **kwargs): + env["pipelines"].append(kwargs.get("file_name")) + + async def arun(self): + return SimpleNamespace(documents=[]) + + monkeypatch.setattr(MessageSessionDao, "async_insert_one", classmethod(_fake_insert_session)) + monkeypatch.setattr(LinsightSessionVersionDao, "insert_one", classmethod(_fake_insert_version)) + monkeypatch.setattr(ChatMessageDao, "ainsert_one", classmethod(_fake_insert_message)) + monkeypatch.setattr(workbench_impl.telemetry_service, "log_event", AsyncMock(return_value=None)) + # Promoting the display attachments (temp bucket -> permanent) is a cheap + # copy that has always been in the request; stub it so the storage probes + # only ever see ingestion traffic. + monkeypatch.setattr(workbench_impl, "promote_chat_attachments", AsyncMock(side_effect=lambda files, _uid: files)) + monkeypatch.setattr(workbench_impl, "get_minio_storage", AsyncMock(return_value=minio)) + monkeypatch.setattr(LinsightWorkbenchImpl, "_get_redis", AsyncMock(return_value=fake_redis)) + monkeypatch.setattr("bisheng.core.cache.utils.async_file_download", _record_download) + monkeypatch.setattr("bisheng.knowledge.rag.temp_file_pipeline.TempFilePipeline", _RecordingPipeline) + return env + + +def _login_user(user_id: int = 7): + login_user = MagicMock() + login_user.user_id = user_id + return login_user + + +def _assert_storage_untouched(env: dict) -> None: + assert env["minio"].store == {} + assert env["redis_reads"] == [] + assert env["downloads"] == [] + assert env["pipelines"] == [] + + +@pytest.fixture +def queue_env(monkeypatch: pytest.MonkeyPatch): + """Stub the worker queue behind the unified task-mode entry; hand back ``put``.""" + from bisheng.workstation.domain.services import chat_service + + put = AsyncMock() + # LinsightQueue / encode_queue_item are imported function-locally from + # bisheng.linsight.worker; a stub module keeps the heavy worker import chain + # out of the test, and it must export BOTH names or the ImportError escapes. + fake_worker = ModuleType("bisheng.linsight.worker") + fake_worker.LinsightQueue = lambda *a, **k: SimpleNamespace(put=put) + fake_worker.encode_queue_item = lambda session_version_id, **kwargs: { + "session_version_id": session_version_id, + **kwargs, + } + monkeypatch.setitem(sys.modules, "bisheng.linsight.worker", fake_worker) + # Enqueueing lives in linsight_execute_utils, so the Redis stub belongs on + # THAT module — stubbing only the caller's namespace lets a real client + # through and the call reaches the DB. + monkeypatch.setattr(linsight_execute_utils, "get_redis_client", AsyncMock(return_value=SimpleNamespace())) + monkeypatch.setattr(linsight_execute_utils, "persist_task_turn_message", AsyncMock()) + monkeypatch.setattr(chat_service.LLMService, "get_bisheng_llm", AsyncMock(return_value=MagicMock())) + return chat_service, put + + +def _task_mode_request(files: list[dict] | None) -> APIChatCompletion: + return APIChatCompletion( + clientTimestamp="1755000000", + model="m1", + text="总结下附件", + task_mode=True, + files=files, + ) + + +# --------------------------------------------------------------------------- +# Submit: park the refs, touch nothing +# --------------------------------------------------------------------------- +async def test_submit_parks_the_refs_and_does_no_ingestion_io(submit_env): + """The 19-minute submit: three attachments must cost zero storage round-trips. + + Both upload shapes are present because they fail differently — a daily file + drags in ``TempFilePipeline``, a legacy one a Redis lookup plus a bucket copy. + """ + files = [_legacy_file("f1", "a.pdf"), _legacy_file("f2", "b.pdf"), _daily_file("d1", "c.pdf")] + submit_obj = LinsightQuestionSubmitSchema(question="总结下附件", files=files) + + await LinsightWorkbenchImpl.submit_user_question(submit_obj, _login_user()) + + _assert_storage_untouched(submit_env) + version = submit_env["version"] + assert version.files is None + assert [item["file_id"] for item in version.pending_files] == ["f1", "f2", "d1"] + + +async def test_submit_without_files_parks_nothing(submit_env): + """An empty batch must leave the staging column NULL, not an empty list — a + ``[]`` there would send the worker into the ingest path for every text-only + task ever submitted.""" + await LinsightWorkbenchImpl.submit_user_question(LinsightQuestionSubmitSchema(question="写个周报"), _login_user()) + + version = submit_env["version"] + assert version.pending_files is None + assert version.files is None + + +async def test_submit_with_defer_disabled_still_ingests_inline(submit_env, monkeypatch: pytest.MonkeyPatch): + """The inline path stays reachable for a caller that needs the bytes now.""" + process = AsyncMock(return_value=[{"file_id": "f1", "valid": True}]) + monkeypatch.setattr(LinsightWorkbenchImpl, "_process_submitted_files", process) + submit_obj = LinsightQuestionSubmitSchema(question="总结下附件", files=[_legacy_file()]) + + await LinsightWorkbenchImpl.submit_user_question(submit_obj, _login_user(), defer_ingest=False) + + process.assert_awaited_once() + version = submit_env["version"] + assert version.files == [{"file_id": "f1", "valid": True}] + assert version.pending_files is None + + +async def test_task_mode_submit_enqueues_once_carrying_the_tenant(submit_env, queue_env): + """End of the request: the row is parked and handed to the worker exactly once. + + The queue item is the worker's only source of tenant context (it runs outside + any request), and the enqueue is what makes the run independent of the + browser ever coming back — the two halves that together replace the old + "parse inline, then wait for start-execute". + """ + chat_service, put = queue_env + data = _task_mode_request( + [{"file_id": "d1", "filename": "report.pdf", "filepath": "/tmp-dir/d1.pdf?X-Amz-Algorithm=AWS4"}] + ) + + await chat_service._task_mode_stream_completion(MagicMock(), data, _login_user()) + + _assert_storage_untouched(submit_env) + assert submit_env["version"].files is None + assert submit_env["version"].pending_files + put.assert_awaited_once() + queued = put.await_args.kwargs["data"] + assert queued["session_version_id"] == submit_env["version"].id + assert queued["tenant_id"] == FAKE_TENANT_ID + + +# --------------------------------------------------------------------------- +# Validation: the metadata gate did NOT move +# --------------------------------------------------------------------------- +async def test_submit_still_rejects_a_file_that_is_still_parsing(submit_env): + """Deferring the BYTES must not defer this: the request is where the user is + still watching, and the worker could only report it as a dead task.""" + submit_obj = LinsightQuestionSubmitSchema(question="q", files=[_legacy_file(status="parsing")]) + + with pytest.raises(LinsightWorkbenchImpl.LinsightError): + await LinsightWorkbenchImpl.submit_user_question(submit_obj, _login_user()) + + _assert_storage_untouched(submit_env) + assert "version" not in submit_env + + +def _folder_batch(kind: str) -> list[SubmitFileSchema]: + if kind == "count": + return [ + SubmitFileSchema( + file_id=f"f{i}", file_name=f"{i}.pdf", parsing_status="completed", relative_path=f"docs/{i}.pdf" + ) + for i in range(LinsightWorkbenchImpl._FOLDER_MAX_FILES + 1) + ] + if kind == "size": + half = LinsightWorkbenchImpl._FOLDER_MAX_TOTAL_BYTES // 2 + 1 + return [ + SubmitFileSchema( + file_id=f"f{i}", + file_name=f"{i}.pdf", + parsing_status="completed", + relative_path=f"docs/{i}.pdf", + size=half, + ) + for i in range(2) + ] + deep = "/".join(f"d{i}" for i in range(LinsightWorkbenchImpl._FOLDER_MAX_DEPTH + 1)) + "/f.pdf" + return [SubmitFileSchema(file_id="f1", file_name="f.pdf", parsing_status="completed", relative_path=deep)] + + +@pytest.mark.parametrize( + ("kind", "error", "code"), + [ + ("count", LinsightFolderFileCountExceededError, 11021), + ("size", LinsightFolderTotalSizeExceededError, 11022), + ("depth", LinsightFolderDepthExceededError, 11023), + ], +) +async def test_folder_limits_raise_their_typed_code_before_any_byte_moves(submit_env, kind, error, code): + """These three codes have frontend copy hanging off them. Sunk into the + worker they would surface as "task failed" minutes after the user dropped the + folder, with nothing to retry against.""" + submit_obj = LinsightQuestionSubmitSchema(question="q", files=_folder_batch(kind)) + + with pytest.raises(error) as exc_info: + await LinsightWorkbenchImpl.submit_user_question(submit_obj, _login_user()) + + assert exc_info.value.Code == code + _assert_storage_untouched(submit_env) + assert "version" not in submit_env + + +async def test_a_rejected_batch_never_reaches_the_worker_queue(submit_env, queue_env): + """A rejected submit must leave nothing behind for the worker to pick up. + + Enqueueing is server-side now, so "the request failed" and "the task never + runs" are no longer the same statement: a queue item written next to a + refused batch would be a ghost task nobody asked for, and the user has + already been told the upload was rejected. + """ + chat_service, put = queue_env + over_count = [ + {"file_id": f"f{i}", "filename": f"{i}.pdf", "relative_path": f"docs/{i}.pdf"} + for i in range(LinsightWorkbenchImpl._FOLDER_MAX_FILES + 1) + ] + + with pytest.raises(LinsightFolderFileCountExceededError): + await chat_service._task_mode_stream_completion(MagicMock(), _task_mode_request(over_count), _login_user()) + + put.assert_not_awaited() + _assert_storage_untouched(submit_env) + assert "version" not in submit_env + + +# --------------------------------------------------------------------------- +# ingest_pending_files: the worker-side entry point +# --------------------------------------------------------------------------- +async def test_ingest_rebinds_files_and_clears_pending(monkeypatch: pytest.MonkeyPatch): + """Whole-attribute rebind, deliberately: ``JsonType`` is a bare JSON column + with no ``MutableList`` wrapper anywhere in this repo, so a ``.extend()`` on + the loaded list leaves SQLAlchemy with no attribute change and the UPDATE is + never emitted — the ingest would vanish at commit.""" + processed = [{"file_id": "f1", "valid": True}] + monkeypatch.setattr(LinsightWorkbenchImpl, "_process_submitted_files", AsyncMock(return_value=processed)) + session_model = SimpleNamespace(id="svid1", user_id=7, files=None, pending_files=[_legacy_file().model_dump()]) + + assert await LinsightWorkbenchImpl.ingest_pending_files(session_model) is True + assert session_model.files == processed + assert session_model.pending_files is None + + +async def test_ingest_without_pending_files_is_a_no_op(): + """Every non-deferred session (resume, continue, a pre-upgrade row) reaches + this with pending_files NULL — it must not clear or re-write ``files``.""" + session_model = SimpleNamespace(id="svid1", user_id=7, files=[{"file_id": "old"}], pending_files=None) + + assert await LinsightWorkbenchImpl.ingest_pending_files(session_model) is False + assert session_model.files == [{"file_id": "old"}] + + +async def test_ingest_reports_progress_per_file_and_honours_abort(monkeypatch: pytest.MonkeyPatch): + """A stop request must not have to wait out a 600s-per-file batch, and the + timeline needs a frame per file or the panel sits empty for the whole ingest.""" + seen: list[str] = [] + progress: list[tuple[int, int, str]] = [] + + async def _fake_ingest_one(cls, submit_file, *args, **kwargs): + seen.append(submit_file.file_id) + return {"file_id": submit_file.file_id} + + async def _on_progress(done, total, name): + progress.append((done, total, name)) + + monkeypatch.setattr(LinsightWorkbenchImpl, "_ingest_one_file", classmethod(_fake_ingest_one)) + monkeypatch.setattr(LinsightWorkbenchImpl, "_get_redis", AsyncMock(return_value=AsyncMock(amget=AsyncMock()))) + monkeypatch.setattr(workbench_impl, "get_minio_storage", AsyncMock(return_value=MagicMock())) + + result = await LinsightWorkbenchImpl._process_submitted_files( + [_legacy_file("f1", "a.pdf"), _legacy_file("f2", "b.pdf"), _legacy_file("f3", "c.pdf")], + "svid1", + 7, + on_progress=_on_progress, + should_abort=lambda: len(seen) >= 2, + ) + + assert seen == ["f1", "f2"] + assert [entry["file_id"] for entry in result] == ["f1", "f2"] + assert progress == [(1, 3, "a.pdf"), (2, 3, "b.pdf")] + + +async def test_one_file_blowing_up_costs_only_that_file(monkeypatch: pytest.MonkeyPatch): + """The linsight branch makes bare ``copy_object`` calls with no guard of its + own, so one MinIO hiccup used to abandon every remaining attachment. In the + request that surfaced as a failed submit; from the worker it is silent — the + run simply proceeds with fewer files than the user attached, and nothing says + so. Degrade the one file, keep the batch.""" + seen: list[str] = [] + + async def _flaky_ingest_one(cls, submit_file, *args, **kwargs): + seen.append(submit_file.file_id) + if submit_file.file_id == "f2": + raise RuntimeError("S3 connection reset") + return {"file_id": submit_file.file_id, "valid": True} + + monkeypatch.setattr(LinsightWorkbenchImpl, "_ingest_one_file", classmethod(_flaky_ingest_one)) + monkeypatch.setattr(LinsightWorkbenchImpl, "_get_redis", AsyncMock(return_value=AsyncMock(amget=AsyncMock()))) + monkeypatch.setattr(workbench_impl, "get_minio_storage", AsyncMock(return_value=MagicMock())) + + result = await LinsightWorkbenchImpl._process_submitted_files( + [_legacy_file("f1", "a.pdf"), _legacy_file("f2", "b.pdf"), _legacy_file("f3", "c.pdf")], "svid1", 7 + ) + + assert seen == ["f1", "f2", "f3"] + by_id = {entry["file_id"]: entry for entry in result} + assert by_id["f1"]["valid"] is True + assert by_id["f3"]["valid"] is True + assert by_id["f2"]["valid"] is False + assert "S3 connection reset" in by_id["f2"]["error_message"] + + +async def test_the_staging_column_never_leaves_the_server(submit_env): + """``pending_files`` carries the presigned temp-bucket link the browser + uploaded with (7-day validity), and the version list is reachable through a + share link — so a plain ``model_dump`` would hand a share recipient direct + URLs to the submitter's originals. Both public read surfaces go through + ``public_dump``; the worker reads the column off the row, never off a + response.""" + submit_obj = LinsightQuestionSubmitSchema(question="总结下附件", files=[_daily_file("d1", "报告.pdf")]) + + await LinsightWorkbenchImpl.submit_user_question(submit_obj, _login_user()) + + version = submit_env["version"] + assert version.pending_files[0]["file_url"].startswith("/tmp-dir/") + assert "pending_files" not in version.public_dump() + # Everything else still ships, including the (empty) ingested list. + assert version.public_dump()["files"] is None + assert version.public_dump()["id"] == version.id + + +# --------------------------------------------------------------------------- +# amget misalignment (pre-existing bug, widened by the longer deferral window) +# --------------------------------------------------------------------------- +async def test_an_expired_temp_key_cannot_hand_its_neighbour_the_wrong_markdown(): + """``amget`` is ``[loads(v) for v in values if v is not None]`` — a miss makes + the returned list SHORTER than the keys, so pairing it positionally with the + file list handed file A the markdown that belongs to file B. Deferring the + ingest by minutes only makes an expired temp key likelier. + """ + minio = FakeMinio() + info2 = _temp_info("f2", "b.pdf") + minio.store[(minio.tmp_bucket, info2["markdown_file_path"])] = b"# B\nbody of b\n" + + fake_redis = AsyncMock() + # f1's temp key expired; only f2's value survives the mget. + fake_redis.amget = AsyncMock(return_value=[info2]) + fake_redis.aget = AsyncMock(side_effect=lambda key: None if key.endswith("f1") else info2) + + with ( + patch.object(LinsightWorkbenchImpl, "_get_redis", return_value=fake_redis), + patch.object(workbench_impl, "get_minio_storage", new=AsyncMock(return_value=minio)), + ): + result = await LinsightWorkbenchImpl._process_submitted_files( + [_legacy_file("f1", "a.pdf"), _legacy_file("f2", "b.pdf")], "svid1", 7 + ) + + by_id = {entry["file_id"]: entry for entry in result} + # The expired one is reported as expired instead of quietly wearing b.pdf's body. + assert by_id["f1"]["valid"] is False + assert by_id["f1"].get("parsing_status") in ("expired", "invalid") + assert by_id["f2"]["valid"] is True + assert by_id["f2"]["workspace_path"] == "/uploads/b.md" + assert minio.store[("bisheng", "workspace/svid1/uploads/b.md")] == b"# B\nbody of b\n" + assert not [key for (_bucket, key) in minio.store if key.startswith("workspace/svid1/uploads/a")] diff --git a/src/backend/test/linsight/test_deferred_ingest_worker.py b/src/backend/test/linsight/test_deferred_ingest_worker.py new file mode 100644 index 0000000000..93f4bf2537 --- /dev/null +++ b/src/backend/test/linsight/test_deferred_ingest_worker.py @@ -0,0 +1,517 @@ +"""Worker side of the deferred attachment ingest: when it runs, and what it may cost. + +Submit now parks the raw file refs in ``pending_files`` (see +``test_deferred_ingest.py`` for why) and the worker materializes them at the top +of the run. That moved a multi-minute, failure-prone step into the middle of the +execution lifecycle, which is where every constraint below comes from: + + * it runs INSIDE the restored tenant context and BEFORE anything reads + ``files`` — the local prefetch, the code interpreter's ``os.walk`` snapshot, + the 可用文件 pointer block, and WorkspaceBackend's ``read_file``, which can + only serve ``workspace/{svid}/uploads/*`` once the ingest has PUT them; + * the IN_PROGRESS claim moved ahead of it, so the duplicate queue item that + server-side enqueue made routine (submit enqueues, the browser's + start-execute lands anyway) cannot re-enter a batch already being parsed; + * the DB write is followed by a Redis snapshot refresh, or the snapshot cached + by that same IN_PROGRESS flip shadows the ingested files for the whole run; + * one unreadable attachment costs its own entry (``valid=False``) and nothing + more, but a SYSTEMIC failure fails the run — an agent answering a question + about "the attached report" with no attachment is worse than an error; + * a stop pressed inside that window wins, even though the very next statement + of ``_execute_workflow`` writes IN_PROGRESS unconditionally. + +External services are faked; no live middleware required. +""" + +from __future__ import annotations + +import asyncio +import copy +from contextlib import suppress +from types import SimpleNamespace +from unittest.mock import AsyncMock, MagicMock, patch + +import pytest + +from bisheng.core.context.tenant import current_tenant_id +from bisheng.linsight.domain import task_exec as te +from bisheng.linsight.domain.models.linsight_session_version import ( + LinsightSessionVersion, + SessionVersionStatusEnum, +) +from bisheng.linsight.domain.schemas.linsight_schema import SubmitFileSchema +from bisheng.linsight.domain.services import workbench_impl +from bisheng.linsight.domain.services.workbench_impl import LinsightWorkbenchImpl +from bisheng.linsight.domain.task_exec import LinsightWorkflowTask + +TENANT_ID = 3 + + +@pytest.fixture(autouse=True) +def _no_question_row_annotation(monkeypatch: pytest.MonkeyPatch): + """Keep the post-ingest question-row back-fill off the DB. + + It is a real DAO round-trip on the success path; ``test_unified_task_turn_write`` + owns its behaviour. Here it would only be a connection error swallowed by the + best-effort guard, plus the wait for it. + """ + annotate = AsyncMock(return_value=True) + monkeypatch.setattr(te.linsight_execute_utils, "annotate_task_user_turn_files", annotate) + return annotate + + +class FakeMinio: + def __init__(self) -> None: + self.bucket = "bisheng" + self.tmp_bucket = "tmp-dir" + self.store: dict[tuple[str, str], bytes] = {} + + async def object_exists(self, bucket_name=None, object_name=None): + return (bucket_name or self.bucket, object_name) in self.store + + async def copy_object(self, source_bucket=None, source_object=None, dest_bucket=None, dest_object=None): + src = self.store.get((source_bucket or self.tmp_bucket, source_object), b"parsed-md") + self.store[(dest_bucket or self.bucket, dest_object)] = src + + async def get_object(self, bucket_name=None, object_name=None): + return self.store.get((bucket_name or self.bucket, object_name)) + + async def put_object(self, *, bucket_name=None, object_name, file, **kwargs): + self.store[(bucket_name or self.bucket, object_name)] = file if isinstance(file, bytes) else bytes(file) + + +class FakeStateManager: + """Mirrors the three properties of the real manager this change depends on. + + ``set_session_version_info`` writes the DB row and THEN caches a snapshot; + ``get_session_version_info`` answers from that snapshot and only falls back + to the row when there is none (``state_message_manager.py``). Reproducing the + order is the whole point — a test that reads back the live object would pass + against a missing refresh. + + The third is ``insert_one``'s closing ``session.refresh()``: a column this + worker did NOT assign is re-read from the row and lands back in the model, so + a terminate written by the endpoint mid-ingest arrives in memory. A column it + DID assign wins over the row instead — which is precisely how + ``_execute_workflow``'s blind IN_PROGRESS write can undo that terminate. + ``row_status`` is what an outside writer (the endpoint) pokes. + """ + + def __init__(self, session_model) -> None: + self.db_row = session_model + self.snapshot: dict | None = None + self.steps: list = [] + self.messages: list = [] + self.row_status = session_model.status + self._model_status = session_model.status + + async def set_session_version_info(self, session_model) -> None: + if session_model.status == self._model_status: + session_model.status = self.row_status + else: + self.row_status = session_model.status + self._model_status = session_model.status + self.db_row = session_model + self.snapshot = copy.deepcopy(session_model.model_dump()) + + async def get_session_version_info(self): + if self.snapshot is None: + return self.db_row + return LinsightSessionVersion.model_validate(self.snapshot) + + async def add_execution_task_step(self, svid, step) -> None: + self.steps.append(step) + + async def push_message(self, message) -> None: + self.messages.append(message) + + +def _pending(file_id: str = "f1", name: str = "a.pdf") -> dict: + return SubmitFileSchema(file_id=file_id, file_name=name, parsing_status="completed").model_dump() + + +def _daily_pending(file_id: str, name: str) -> dict: + return SubmitFileSchema( + file_id=file_id, + file_name=name, + parsing_status="completed", + file_url=f"/tmp-dir/{file_id}.pdf?X-Amz-Algorithm=AWS4", + ).model_dump() + + +def _session(pending: list[dict] | None, status=SessionVersionStatusEnum.NOT_STARTED) -> LinsightSessionVersion: + return LinsightSessionVersion( + id="svid1", + session_id="chat1", + user_id=7, + question="总结下附件", + status=status, + tenant_id=TENANT_ID, + pending_files=pending, + ) + + +def _task(session_model) -> tuple[LinsightWorkflowTask, FakeStateManager]: + task = LinsightWorkflowTask() + task.session_version_id = session_model.id + state = FakeStateManager(session_model) + task._state_manager = state + return task, state + + +# --------------------------------------------------------------------------- +# Ordering inside the run +# --------------------------------------------------------------------------- +async def test_ingest_runs_in_tenant_context_before_anything_reads_the_files(monkeypatch: pytest.MonkeyPatch): + """The ingest sits between the tenant restore and every reader of ``files``. + + Moving it later (or earlier, into the queue consumer) breaks two different + things at once: the workspace PUTs and DB reads would run with no tenant + bound, and the local prefetch / tool construction / pointer block would all + snapshot an empty ``uploads/``. + """ + session_model = _session([_pending()]) + task, state = _task(session_model) + order: list[str] = [] + observed: dict = {} + + async def _fake_ingest(model, **kwargs): + observed["tenant_id"] = current_tenant_id.get() + observed["status"] = model.status + order.append("ingest") + model.files = [{"file_id": "f1", "valid": True}] + model.pending_files = None + return True + + async def _fake_init_dir(model): + observed["files_at_prefetch"] = list(model.files or []) + order.append("init_file_directory") + return "/tmp/linsight-svid1" + + async def _fake_execute(model): + order.append("execute_workflow") + + monkeypatch.setattr(te.LinsightSessionVersionDao, "get_by_id", AsyncMock(return_value=session_model)) + monkeypatch.setattr(te, "LinsightStateMessageManager", lambda _svid: state) + monkeypatch.setattr(te, "ensure_linsight_permission_runtime", AsyncMock(return_value={})) + monkeypatch.setattr(LinsightWorkbenchImpl, "ingest_pending_files", _fake_ingest) + task._ensure_session_pseudo_task = AsyncMock(side_effect=lambda _m: order.append("pseudo_task")) + task._init_file_directory = _fake_init_dir + task._execute_workflow = _fake_execute + + await task.async_run("svid1", tenant_id=TENANT_ID) + + assert order == ["pseudo_task", "ingest", "init_file_directory", "execute_workflow"] + # The worker owns no request context, so a missing restore reads as None here. + assert observed["tenant_id"] == TENANT_ID + # The pseudo task row must already exist: the progress steps hang off it. + assert observed["status"] == SessionVersionStatusEnum.IN_PROGRESS + # Whatever runs next sees the materialized list, not the staging column. + assert observed["files_at_prefetch"] == [{"file_id": "f1", "valid": True}] + assert session_model.pending_files is None + + +async def test_worker_ingest_rebinds_files_so_the_next_reader_sees_them(monkeypatch: pytest.MonkeyPatch): + """``files`` is replaced wholesale, never appended to in place: ``JsonType`` + has no ``MutableList`` wrapper, so an in-place mutation emits no UPDATE and + the run would still be the only thing that ever saw the attachments.""" + session_model = _session([_pending("f1"), _pending("f2", "b.pdf")]) + # Anything already on the row (a HITL-added file, say) has to survive, which + # is exactly the case an ``.extend()`` would look correct on in memory. + session_model.files = [{"file_id": "old", "valid": True}] + loaded_list = session_model.files + processed = [{"file_id": "f1", "valid": True}, {"file_id": "f2", "valid": True}] + monkeypatch.setattr(LinsightWorkbenchImpl, "_process_submitted_files", AsyncMock(return_value=processed)) + + await _task(session_model)[0]._ingest_pending_attachments(session_model) + + assert [entry["file_id"] for entry in session_model.files] == ["old", "f1", "f2"] + # A new list object, and the one SQLAlchemy loaded was never touched. + assert session_model.files is not loaded_list + assert loaded_list == [{"file_id": "old", "valid": True}] + assert session_model.pending_files is None + + +async def test_ingested_files_are_not_shadowed_by_the_earlier_redis_snapshot(): + """``get_session_version_info`` reads Redis first, and the IN_PROGRESS flip + already cached a snapshot carrying ``files=None``. Without a refresh right + after the ingest, every reader for the rest of the run gets that stale + snapshot and the task behaves as if nothing was ever uploaded.""" + session_model = _session([_pending()]) + task, state = _task(session_model) + + # The claim at the top of _managed_execution: this is what caches files=None. + await task._update_session_status(session_model, SessionVersionStatusEnum.IN_PROGRESS) + assert (await state.get_session_version_info()).files is None + + with patch.object( + LinsightWorkbenchImpl, "_process_submitted_files", new=AsyncMock(return_value=[{"file_id": "f1"}]) + ): + await task._ingest_pending_attachments(session_model) + + cached = await state.get_session_version_info() + assert cached.files == [{"file_id": "f1"}] + assert cached.pending_files is None + + +# --------------------------------------------------------------------------- +# Degradation: a bad attachment costs its own entry, nothing more +# --------------------------------------------------------------------------- +async def test_one_unreadable_attachment_does_not_fail_the_session(monkeypatch: pytest.MonkeyPatch): + """A ten-file task must not die because one file's download failed. The bad + entry is marked ``valid=False`` for the chip, the others ingest normally, and + the session status is left exactly where the run put it.""" + session_model = _session([_daily_pending("good", "ok.pdf"), _daily_pending("bad", "broken.pdf")]) + task, state = _task(session_model) + await task._update_session_status(session_model, SessionVersionStatusEnum.IN_PROGRESS) + + class _Doc: + page_content = "# parsed\nbody\n" + + class _OkPipeline: + def __init__(self, *args, **kwargs): + pass + + async def arun(self): + return SimpleNamespace(documents=[_Doc()]) + + async def _download(url, *args, **kwargs): + if "bad" in url: + raise ValueError("minio download boom") + return ("/tmp/ok.pdf", "ok.pdf") + + with ( + patch.object(workbench_impl, "get_minio_storage", new=AsyncMock(return_value=FakeMinio())), + patch.object(LinsightWorkbenchImpl, "_get_redis", return_value=AsyncMock()), + patch("bisheng.core.cache.utils.async_file_download", new=_download), + patch("bisheng.knowledge.rag.temp_file_pipeline.TempFilePipeline", _OkPipeline), + ): + await task._ingest_pending_attachments(session_model) + + by_id = {entry["file_id"]: entry for entry in session_model.files} + assert by_id["good"]["valid"] is True + assert by_id["bad"]["valid"] is False + assert "boom" in by_id["bad"]["error_message"] + assert session_model.pending_files is None + assert session_model.status == SessionVersionStatusEnum.IN_PROGRESS + assert (await state.get_session_version_info()).status == SessionVersionStatusEnum.IN_PROGRESS + + +async def test_a_systemic_ingest_failure_fails_the_run_instead_of_answering_blind( + monkeypatch: pytest.MonkeyPatch, +): + """Per-file failures never reach here — ``_process_submitted_files`` degrades + them. What does reach here is systemic (storage unreachable, a malformed ref), + and swallowing it leaves ``files`` empty while the run carries on: the model + is asked about "the attached report" with no attachment and answers anyway, + confidently, about a document it never saw. Fail loudly instead, and leave + ``pending_files`` for /workbench/continue to retry. + """ + session_model = _session([_pending()]) + task, state = _task(session_model) + monkeypatch.setattr( + LinsightWorkbenchImpl, "ingest_pending_files", AsyncMock(side_effect=RuntimeError("minio down")) + ) + + with pytest.raises(te.TaskExecutionError, match="ingest uploaded attachments"): + await task._ingest_pending_attachments(session_model) + + assert session_model.files is None + assert session_model.pending_files == [_pending()] + # And it is visible in the timeline, not only in the worker log. Asserted on + # the structured payload rather than the label: the row carries no prose at + # all, because the copy is the client's (a backend-formatted string would show + # Chinese to a Japanese user and would be frozen wrong in persisted history). + assert state.steps and state.steps[-1].status == "end" + assert state.steps[-1].name == te._INGEST_STEP_NAME + assert state.steps[-1].extra_info["ingest_progress"]["phase"] == "failed" + assert "minio down" in state.steps[-1].output + + +async def test_a_failed_ingest_reaches_the_users_error_instead_of_the_stranded_net( + monkeypatch: pytest.MonkeyPatch, +): + """``TaskExecutionError`` out of ``_managed_execution`` is the controlled + route: ``async_run`` hands it to ``_handle_execution_error``, which writes a + FAILED session carrying the reason. The alternative is the startup sweep's + "Worker node crash detected", minutes later and about the wrong thing.""" + session_model = _session([_pending()]) + task, state = _task(session_model) + failures: list[str] = [] + + monkeypatch.setattr(te.LinsightSessionVersionDao, "get_by_id", AsyncMock(return_value=session_model)) + monkeypatch.setattr(te, "LinsightStateMessageManager", lambda _svid: state) + monkeypatch.setattr(te, "ensure_linsight_permission_runtime", AsyncMock(return_value={})) + monkeypatch.setattr( + LinsightWorkbenchImpl, "ingest_pending_files", AsyncMock(side_effect=RuntimeError("minio down")) + ) + task._ensure_session_pseudo_task = AsyncMock() + task._init_file_directory = AsyncMock(side_effect=AssertionError("prefetch must not run without the files")) + task._execute_workflow = AsyncMock(side_effect=AssertionError("the agent must not run without the files")) + task._handle_execution_error = AsyncMock(side_effect=lambda err: failures.append(str(err))) + + await task.async_run("svid1", tenant_id=TENANT_ID) + + assert failures and "minio down" in failures[0] + + +# --------------------------------------------------------------------------- +# Double pick-up and termination +# --------------------------------------------------------------------------- +async def test_a_duplicate_queue_item_cannot_re_enter_the_ingest(monkeypatch: pytest.MonkeyPatch): + """Two queue items for one svid is the NORMAL case since submit enqueues + server-side and the browser's start-execute still lands. The IN_PROGRESS + claim therefore had to move ahead of the ingest: with the claim still inside + ``_execute_workflow``, the second worker walked straight into the same batch + and parsed every attachment a second time, on top of the first. + """ + session_model = _session([_pending()]) + first, state = _task(session_model) + second, _ = _task(session_model) + second._state_manager = state + ingesting = asyncio.Event() + release = asyncio.Event() + entered: list[str] = [] + + async def _slow_ingest(model, **kwargs): + entered.append(model.id) + ingesting.set() + # Bounded on purpose: if the claim ever slips back behind the ingest the + # second worker lands here too, and an unbounded wait would hang the + # suite instead of failing the assertion below. + with suppress(TimeoutError): + await asyncio.wait_for(release.wait(), timeout=2) + model.files = [{"file_id": "f1"}] + model.pending_files = None + return True + + monkeypatch.setattr(te.LinsightSessionVersionDao, "get_by_id", AsyncMock(return_value=session_model)) + # _managed_execution mints its own state manager, so both workers must be + # given the shared fake or they talk to a real Redis. + monkeypatch.setattr(te, "LinsightStateMessageManager", lambda _svid: state) + monkeypatch.setattr(LinsightWorkbenchImpl, "ingest_pending_files", _slow_ingest) + for task in (first, second): + task._ensure_session_pseudo_task = AsyncMock() + task._init_file_directory = AsyncMock(return_value="/tmp/linsight-svid1") + + async def _run(task): + async with task._managed_execution(): + pass + + running = asyncio.create_task(_run(first)) + await asyncio.wait_for(ingesting.wait(), timeout=5) + + with pytest.raises(te.TaskAlreadyInProgressError): + await _run(second) + + release.set() + await asyncio.wait_for(running, timeout=5) + + assert entered == ["svid1"] + + +async def test_a_stop_request_lands_between_files_instead_of_after_the_batch(monkeypatch: pytest.MonkeyPatch): + """Stop used to be answerable within a step; a deferred ingest can hold the + task for minutes before the agent ever starts. The abort flag is polled + before each file so the user's stop costs at most one file, not the batch.""" + session_model = _session([_pending("f1", "a.pdf"), _pending("f2", "b.pdf"), _pending("f3", "c.pdf")]) + task, _state = _task(session_model) + ingested: list[str] = [] + + async def _fake_ingest_one(cls, submit_file, *args, **kwargs): + ingested.append(submit_file.file_id) + # The user hits stop while the first file is being parsed. + task._is_terminated = True + return {"file_id": submit_file.file_id, "valid": True} + + monkeypatch.setattr(LinsightWorkbenchImpl, "_ingest_one_file", classmethod(_fake_ingest_one)) + monkeypatch.setattr(LinsightWorkbenchImpl, "_get_redis", AsyncMock(return_value=AsyncMock(amget=AsyncMock()))) + monkeypatch.setattr(workbench_impl, "get_minio_storage", AsyncMock(return_value=MagicMock())) + + await task._ingest_pending_attachments(session_model) + + assert ingested == ["f1"] + assert [entry["file_id"] for entry in session_model.files] == ["f1"] + + +async def test_a_stop_during_the_ingest_is_not_undone_by_the_run(monkeypatch: pytest.MonkeyPatch): + """The stop has to survive the very next statement of the run. + + ``_execute_workflow`` opens with an UNCONDITIONAL IN_PROGRESS write, and the + ingest's own DB write refreshes the model straight from the row — so a + terminate that landed mid-ingest arrives in memory just in time to be + overwritten. Nothing writes a terminal status after that: the session shows + as running forever and pressing 停止 again is the only way out, if it wins the + race at all. ``_managed_execution`` therefore re-reads the authoritative + status once the ingest returns and bails before any of that. + """ + session_model = _session([_pending()]) + task, state = _task(session_model) + ran: list[str] = [] + + async def _terminating_ingest(model, **kwargs): + # What terminate_execute does: write TERMINATED straight to the row while + # this worker is still parsing and has not polled the monitor yet. + state.row_status = SessionVersionStatusEnum.TERMINATED + model.files = [{"file_id": "f1", "valid": True}] + model.pending_files = None + return True + + async def _blind_claim(model): + # The first statement of the real _execute_workflow, verbatim in effect. + ran.append("execute_workflow") + await task._update_session_status(model, SessionVersionStatusEnum.IN_PROGRESS) + + monkeypatch.setattr(te.LinsightSessionVersionDao, "get_by_id", AsyncMock(return_value=session_model)) + monkeypatch.setattr(te, "LinsightStateMessageManager", lambda _svid: state) + monkeypatch.setattr(te, "ensure_linsight_permission_runtime", AsyncMock(return_value={})) + monkeypatch.setattr(LinsightWorkbenchImpl, "ingest_pending_files", _terminating_ingest) + task._ensure_session_pseudo_task = AsyncMock() + task._init_file_directory = AsyncMock(return_value="/tmp/linsight-svid1") + task._execute_workflow = _blind_claim + + # async_run swallows UserTerminationError by design (the endpoint already + # wrote the terminal state); the assertion is that nothing resurrected it. + await task.async_run("svid1", tenant_id=TENANT_ID) + + assert ran == [] + assert (await state.get_session_version_info()).status == SessionVersionStatusEnum.TERMINATED + + +async def test_resume_ingests_a_batch_a_crashed_worker_left_behind(monkeypatch: pytest.MonkeyPatch): + """The resume/continue path is the column's only second chance. + + A worker killed mid-ingest leaves the row IN_PROGRESS with ``pending_files`` + intact; the startup sweep then force-writes FAILED, and FAILED is one of the + two statuses /workbench/continue accepts. Without an ingest here the + follow-up turn runs with no attachments at all and the refs are stranded on + the row forever. + """ + session_model = _session([_pending()], status=SessionVersionStatusEnum.FAILED) + task, state = _task(session_model) + order: list[str] = [] + + async def _fake_ingest(model, **kwargs): + order.append("ingest") + model.files = [{"file_id": "f1", "valid": True}] + model.pending_files = None + return True + + async def _fake_init_dir(model): + order.append("init_file_directory") + return "/tmp/linsight-svid1" + + monkeypatch.setattr(te.LinsightSessionVersionDao, "get_by_id", AsyncMock(return_value=session_model)) + monkeypatch.setattr(te, "LinsightStateMessageManager", lambda _svid: state) + monkeypatch.setattr(LinsightWorkbenchImpl, "ingest_pending_files", _fake_ingest) + task._start_termination_monitor = AsyncMock() + task._ensure_session_pseudo_task = AsyncMock() + task._init_file_directory = _fake_init_dir + + async with task._managed_resume() as resumed: + pass + + # Still ahead of the local prefetch, same as the fresh path. + assert order == ["ingest", "init_file_directory"] + assert resumed.files == [{"file_id": "f1", "valid": True}] + assert resumed.pending_files is None diff --git a/src/backend/test/linsight/test_incomplete_stream_guard.py b/src/backend/test/linsight/test_incomplete_stream_guard.py new file mode 100644 index 0000000000..4de88f2d2b --- /dev/null +++ b/src/backend/test/linsight/test_incomplete_stream_guard.py @@ -0,0 +1,202 @@ +"""Unit tests for the incomplete-stream guard in LinsightModelResilienceMiddleware. + +A provider that closes the SSE stream mid-turn raises nothing: langchain hands back +whatever it accumulated. When that partial message carries no tool call, deepagents +reads it as "the agent is done" — which is how a 114 run (2026-08-13) with 3 of its +5 steps still pending completed "successfully", its final answer being the one line +of narration the model had emitted before the stream died. + +The guard detects that shape, re-sends the call a bounded number of times, and — if +it keeps coming back incomplete — fails the main graph / degrades the subagent +instead of passing the cut-off narration off as an answer. + +``asyncio_mode = auto`` — async tests need no decorator. +""" + +import pytest +from langchain.agents.middleware.types import ModelRequest, ModelResponse +from langchain_core.messages import AIMessage, HumanMessage + +from bisheng.common.services.llm_error_classifier import ( + Behavior, + ErrorType, + classify_behavior, + classify_for_event, + label_error, +) +from bisheng.linsight.domain.services.resilience_middleware import ( + _DEGRADE_MESSAGE, + _INCOMPLETE_STREAM_RETRY_LIMIT, + IncompleteStreamError, + LinsightModelResilienceMiddleware, + _is_incomplete_stream_response, +) +from bisheng.linsight.domain.task_exec import TaskExecutionError + +# The exact shape observed on 114: metadata from the first chunk, one line of +# narration, no finish_reason, no usage, no tool call. +_METADATA = {"model_name": "grok4.6", "model_provider": "openai"} + + +def _incomplete_ai(content="正在撰写完整参数清单与 D2 数据表。"): + return AIMessage( + content=content, + response_metadata=dict(_METADATA), + usage_metadata={"input_tokens": 0, "output_tokens": 69, "total_tokens": 69}, + ) + + +def _ok_ai(content="done"): + return AIMessage( + content=content, + response_metadata={**_METADATA, "finish_reason": "stop"}, + usage_metadata={"input_tokens": 147642, "output_tokens": 2423, "total_tokens": 150065}, + ) + + +def _resp(msg): + return ModelResponse(result=[msg]) + + +def _req(): + return ModelRequest(model=None, messages=[HumanMessage(content="写一份报告")]) + + +def _mw(*, is_subagent=False): + return LinsightModelResilienceMiddleware(max_retries=3, initial_delay=0.0, is_subagent=is_subagent) + + +# --------------------------------------------------------------------------- detection + + +def test_detects_the_114_shape(): + assert _is_incomplete_stream_response(_resp(_incomplete_ai())) is True + + +def test_detect_accepts_bare_aimessage(): + assert _is_incomplete_stream_response(_incomplete_ai()) is True + + +def test_completed_stream_is_not_incomplete(): + assert _is_incomplete_stream_response(_resp(_ok_ai())) is False + + +def test_anthropic_stop_reason_counts_as_finished(): + ai = AIMessage(content="done", response_metadata={**_METADATA, "stop_reason": "end_turn"}) + assert _is_incomplete_stream_response(_resp(ai)) is False + + +def test_usage_alone_proves_the_stream_completed(): + """A provider that omits finish_reason but reported real input usage is done.""" + ai = AIMessage( + content="done", + response_metadata=dict(_METADATA), + usage_metadata={"input_tokens": 1234, "output_tokens": 56, "total_tokens": 1290}, + ) + assert _is_incomplete_stream_response(_resp(ai)) is False + + +def test_stream_cut_mid_tool_call_is_out_of_scope(): + """Deliberately excluded: the graph keeps running, so it is never silent.""" + ai = AIMessage( + content="", + tool_calls=[{"name": "write_file", "args": {"file_path": "/output/r.md"}, "id": "t", "type": "tool_call"}], + response_metadata=dict(_METADATA), + ) + assert _is_incomplete_stream_response(_resp(ai)) is False + + +def test_synthetic_message_without_provider_metadata_is_ignored(): + """A degraded/synthetic AIMessage is not a provider stream — never retried.""" + assert _is_incomplete_stream_response(AIMessage(content=_DEGRADE_MESSAGE)) is False + + +# --------------------------------------------------------------------------- behaviour + + +async def test_retries_then_returns_the_recovered_response(): + mw = _mw() + calls = [] + + async def handler(_request): + calls.append(1) + return _resp(_incomplete_ai()) if len(calls) == 1 else _resp(_ok_ai()) + + out = await mw.awrap_model_call(_req(), handler) + assert len(calls) == 2 # exactly one retry + assert out.result[0].content == "done" + + +async def test_main_graph_fails_instead_of_passing_off_a_cut_off_answer(): + mw = _mw() + calls = [] + + async def handler(_request): + calls.append(1) + return _resp(_incomplete_ai()) + + with pytest.raises(IncompleteStreamError): + await mw.awrap_model_call(_req(), handler) + assert len(calls) == _INCOMPLETE_STREAM_RETRY_LIMIT + 1 # initial + bounded retries + + +async def test_subagent_degrades_so_the_parent_task_continues(): + mw = _mw(is_subagent=True) + + async def handler(_request): + return _resp(_incomplete_ai()) + + out = await mw.awrap_model_call(_req(), handler) + assert isinstance(out, AIMessage) + assert out.content == _DEGRADE_MESSAGE + + +async def test_retry_budget_is_separate_from_the_exception_budget(): + """A transient exception first must not eat the incomplete-stream allowance.""" + mw = LinsightModelResilienceMiddleware(max_retries=1, initial_delay=0.0) + calls = [] + + async def handler(_request): + calls.append(1) + if len(calls) == 1: + raise TimeoutError("transient") + return _resp(_incomplete_ai()) if len(calls) <= 2 else _resp(_ok_ai()) + + out = await mw.awrap_model_call(_req(), handler) + assert len(calls) == 3 # exception retry + incomplete retry, both spent + assert out.result[0].content == "done" + + +def test_sync_wrapper_guards_the_same_way(): + mw = _mw() + calls = [] + + def handler(_request): + calls.append(1) + return _resp(_incomplete_ai()) if len(calls) == 1 else _resp(_ok_ai()) + + out = mw.wrap_model_call(_req(), handler) + assert len(calls) == 2 + assert out.result[0].content == "done" + + +# --------------------------------------------------------------------------- classification + + +def test_error_is_classified_as_transient_network(): + """Subclassing ConnectionError is what buys the right bucket + user-facing copy.""" + exc = IncompleteStreamError("stream closed early") + assert classify_behavior(exc) is Behavior.RETRYABLE + assert label_error(exc) is ErrorType.NETWORK_TIMEOUT + + +def test_wrapped_error_still_reaches_the_network_card(): + """task_exec re-raises as ``TaskExecutionError(...) from e`` — the card must survive.""" + cause = IncompleteStreamError("stream closed early") + try: + try: + raise cause + except IncompleteStreamError as e: + raise TaskExecutionError("Agent task execution failed") from e + except TaskExecutionError as wrapper: + assert classify_for_event(wrapper).error_type == ErrorType.NETWORK_TIMEOUT.value diff --git a/src/backend/test/linsight/test_model_vision_flag.py b/src/backend/test/linsight/test_model_vision_flag.py new file mode 100644 index 0000000000..062b4516c6 --- /dev/null +++ b/src/backend/test/linsight/test_model_vision_flag.py @@ -0,0 +1,78 @@ +"""The 视觉 checkbox has to reach the Linsight guards. + +``WSModel.visual`` is set per model in 系统模型设置 → 工作台模型 and was, until now, +read only by daily chat (``chat_service._process_agent_files``). Linsight picks its +model from that SAME list, so the flag is resolvable there too — these tests pin the +mapping, because a silent False turns image reads off platform-wide and a silent +True hands a text-only endpoint a payload it answers with a 400. + +``asyncio_mode = auto`` — async tests need no decorator. +""" + +from unittest.mock import AsyncMock, patch + +from bisheng.api.v1.schemas import WSModel +from bisheng.linsight.domain.services import agent_factory +from bisheng.llm.domain.schemas import WorkbenchModelConfig + + +def session(tenant_id=1, user_id=1): + return type("S", (), {"id": "svid", "tenant_id": tenant_id, "user_id": user_id})() + + +def workbench(models, default_id=None): + return WorkbenchModelConfig( + models=[WSModel(id=str(mid), visual=visual) for mid, visual in models], + linsight_default_model_id=default_id, + ) + + +async def resolve(conf, model_id): + """Run ``_resolve_model`` with only the two external services faked out.""" + with ( + patch.object(agent_factory.LLMService, "get_workbench_llm", AsyncMock(return_value=conf)), + patch.object(agent_factory.LLMService, "get_bisheng_linsight_llm", AsyncMock(return_value="LLM")), + # settings is a pydantic model -> patch the CLASS method, not the instance + # attribute (pydantic rejects the assignment). + patch.object( + type(agent_factory.settings), "get_linsight_conf", return_value=type("C", (), {"default_temperature": 0})() + ), + ): + return await agent_factory._resolve_model(session(), model_id) + + +async def test_ticked_model_reports_vision(): + model, supports_vision = await resolve(workbench([("7", True), ("8", False)]), "7") + assert model == "LLM" + assert supports_vision is True + + +async def test_unticked_model_reports_no_vision(): + _, supports_vision = await resolve(workbench([("7", True), ("8", False)]), "8") + assert supports_vision is False + + +async def test_default_model_is_looked_up_too(): + """Per-task model omitted -> the tenant default id, whose OWN flag decides.""" + conf = workbench([("7", True), ("8", False)], default_id="7") + _, supports_vision = await resolve(conf, None) + assert supports_vision is True + + +async def test_unknown_model_id_fails_closed(): + """A model id absent from the list (config drift) must read as "no vision", + matching the field's own default — never as "assume it can see".""" + _, supports_vision = await resolve(workbench([("7", True)]), "99") + assert supports_vision is False + + +async def test_empty_model_list_fails_closed(): + _, supports_vision = await resolve(WorkbenchModelConfig(models=None, linsight_default_model_id="7"), "7") + assert supports_vision is False + + +async def test_id_comparison_survives_int_vs_str(): + """``WSModel.id`` is a str while callers pass the per-task model id straight + through; an int on either side must still match its row.""" + _, supports_vision = await resolve(workbench([("7", True)]), 7) + assert supports_vision is True diff --git a/src/backend/test/linsight/test_step_persistence.py b/src/backend/test/linsight/test_step_persistence.py index 05d1b6ae1f..74afc45718 100644 --- a/src/backend/test/linsight/test_step_persistence.py +++ b/src/backend/test/linsight/test_step_persistence.py @@ -442,6 +442,62 @@ async def fake_get(svid, *a, **k): assert "aaaa" in ids +def _chain(*ids): + """Rows wired into one previous/next linked chain, in the given order.""" + rows = [_real_task(tid) for tid in ids] + for i in range(len(rows) - 1): + rows[i].next_task_id = rows[i + 1].id + rows[i + 1].previous_task_id = rows[i].id + return rows + + +async def test_detail_does_not_replay_an_overlapping_chain(monkeypatch): + """Two chain heads running into a shared tail must not duplicate that tail. + + ``_save_task_info`` only inserts rows it has not seen; it never rewrites an + existing row's previous/next pointers. So when the model reshapes its plan + mid-run, the new rows form a second chain stitched onto the old one's head + while that head keeps ``previous_task_id = None`` — two heads, one tail. + Session 8a570723 on 114 held 11 rows and this endpoint returned 20. + """ + from bisheng.linsight.domain.services import workbench_impl as wi + + rows = _chain("aaaa", "bbbb", "cccc") + newcomer = _real_task("dddd") + newcomer.next_task_id = "aaaa" # stitched on; rows[0].previous_task_id stays None + tasks = [*rows, newcomer] + + async def fake_get(svid, *a, **k): + return tasks + + monkeypatch.setattr(wi.LinsightExecuteTaskDao, "get_by_session_version_id", fake_get) + + result = await wi.LinsightWorkbenchImpl.get_execute_task_detail("svid") + ids = [node["id"] for node in result] + assert len(ids) == len(tasks), f"every row exactly once, got {ids}" + assert sorted(ids) == ["aaaa", "bbbb", "cccc", "dddd"] + + +async def test_detail_survives_a_cyclic_chain(monkeypatch): + """A cycle must terminate instead of spinning the request forever. + + Defensive: nothing writes a cycle today, but the walk had no visited guard at + all, so a single bad pointer pair would have hung the endpoint. + """ + from bisheng.linsight.domain.services import workbench_impl as wi + + a, b = _real_task("aaaa"), _real_task("bbbb") + a.next_task_id, b.next_task_id = "bbbb", "aaaa" + + async def fake_get(svid, *a_, **k): + return [a, b] + + monkeypatch.setattr(wi.LinsightExecuteTaskDao, "get_by_session_version_id", fake_get) + + result = await wi.LinsightWorkbenchImpl.get_execute_task_detail("svid") + assert sorted(node["id"] for node in result) == ["aaaa", "bbbb"] + + # --------------------------------------------------------------------------- # F035 reload parity: clarify answers survive resume re-stream # --------------------------------------------------------------------------- diff --git a/src/backend/test/linsight/test_stream_event_mapper.py b/src/backend/test/linsight/test_stream_event_mapper.py index c155e4ee79..89e5eb9516 100644 --- a/src/backend/test/linsight/test_stream_event_mapper.py +++ b/src/backend/test/linsight/test_stream_event_mapper.py @@ -203,13 +203,17 @@ def test_tool_start_then_end_two_frames_same_call_id(self, mapper: StreamEventMa start_events = [e for e in mapper.normalize("messages", (start_msg, {})) if isinstance(e, ExecStep)] assert len(start_events) == 1 assert start_events[0].status == "start" - assert start_events[0].call_id == "call_x" + # The emitted id is MINTED from the provider's one (StreamContext.tool_seq), + # because a provider id is only unique within a single response. Assert the + # relationship, not a literal — the raw id stays as the prefix. + assert start_events[0].call_id.startswith("call_x#") end_msg = ToolMessage(content="结果数据", tool_call_id="call_x", name="search") end_events = [e for e in mapper.normalize("messages", (end_msg, {})) if isinstance(e, ExecStep)] assert len(end_events) == 1 assert end_events[0].status == "end" - assert end_events[0].call_id == "call_x" + # Start and end MUST keep the same id — that is what folds them into one row. + assert end_events[0].call_id == start_events[0].call_id assert end_events[0].output == "结果数据" # merged: name/params carried from the start frame assert end_events[0].name == "search" @@ -653,11 +657,15 @@ def _arg_delta(name: str, args: str, cid: str, index: int = 0) -> AIMessageChunk tool_call_chunks=[{"name": name, "args": args, "id": cid, "index": index, "type": "tool_call_chunk"}], ) + @staticmethod + def _steps(mapper: StreamEventMapper, chunk: AIMessageChunk) -> list[ExecStep]: + return [e for e in mapper.normalize("messages", (chunk, {})) if isinstance(e, ExecStep)] + def test_streamed_task_args_recover_goal(self, mapper: StreamEventMapper): # chunk 1: name+id, args still empty -> argless start frame (goal '') c1 = self._arg_delta("task", "", "call_s1") s1 = [e for e in mapper.normalize("messages", (c1, {})) if isinstance(e, ExecStep)] - assert s1 and s1[0].step_type == "subagent" and s1[0].call_id == "call_s1" + assert s1 and s1[0].step_type == "subagent" and s1[0].call_id.startswith("call_s1#") assert s1[0].extra_info.get("delegate_goal") == "" # goal not known yet # chunk 2: partial args JSON — buffer still incomplete, nothing emitted @@ -670,7 +678,7 @@ def test_streamed_task_args_recover_goal(self, mapper: StreamEventMapper): assert len(s3) == 1 ev = s3[0] # same call_id -> persistence upserts it over the argless start frame - assert ev.call_id == "call_s1" + assert ev.call_id == s1[0].call_id assert ev.step_type == "subagent" assert ev.status == "start" assert ev.name == "general-purpose" # subagent_type recovered from args @@ -702,9 +710,32 @@ def test_parallel_streamed_delegations_stay_separated(self, mapper: StreamEventM if isinstance(e, ExecStep) ] assert len(ev_b) == 1 - assert ev_b[0].call_id == "call_b" + assert ev_b[0].call_id.startswith("call_b#") assert ev_b[0].extra_info.get("delegate_goal") == "任务B" + def test_abandoned_call_does_not_leak_its_goal_into_an_id_reuse(self, mapper: StreamEventMapper): + """A call that never gets an end frame must not poison the next call that + reuses its provider id. + + ``tool_arg_buffers`` is keyed by the provider's id and is normally cleared + by the end frame. When a call is abandoned (HITL park / recursion cut / + budget refusal) its partial JSON stays behind, and a provider that recycles + ids — ``:``, restarting at 0 every response — hands that + id straight to the next call. Here the leftover is one character short of + closing, so the newcomer's first fragment completes it and the delegation + would report the ABANDONED call's goal as its own. + """ + # Call 1: args start streaming, then the call is abandoned (no end frame). + self._steps(mapper, self._arg_delta("task", '{"description": "stal', "task:0")) + + # Call 2 reuses the id; its own args stream in from scratch. + self._steps(mapper, self._arg_delta("task", "", "task:0")) + events = self._steps(mapper, self._arg_delta("", 'e"}', "")) + + assert not any(e.extra_info.get("delegate_goal") == "stale" for e in events), ( + "the abandoned call's args must not close against the new call's fragment" + ) + def test_complete_args_in_one_chunk_still_works(self, mapper: StreamEventMapper): # Backward-compat: a provider that ships full args in one AIMessage (no # streaming) gets the goal at the start frame and triggers no refresh. diff --git a/src/backend/test/linsight/test_subagent_reintroduction.py b/src/backend/test/linsight/test_subagent_reintroduction.py index 7ddfd24778..be6d855750 100644 --- a/src/backend/test/linsight/test_subagent_reintroduction.py +++ b/src/backend/test/linsight/test_subagent_reintroduction.py @@ -281,7 +281,9 @@ def test_stream_mapper_namespaced_tool_keeps_real_step_type(): # B1: namespace no longer rewrites step_type — knowledge inference wins assert step.step_type == "knowledge" assert step.extra_info.get("namespace") == sub_ns[0] - assert step.call_id == "call_sub_1" + # Emitted ids are minted from the provider's (StreamContext.tool_seq); the raw + # id stays as the prefix. + assert step.call_id.startswith("call_sub_1#") # sanity contrast: the SAME tool with ns=None is the same (knowledge) step main_mapper = StreamEventMapper(svid=SVID) diff --git a/src/backend/test/linsight/test_task_exec_llm_resolution.py b/src/backend/test/linsight/test_task_exec_llm_resolution.py index 87d27ea83e..832c5af2a6 100644 --- a/src/backend/test/linsight/test_task_exec_llm_resolution.py +++ b/src/backend/test/linsight/test_task_exec_llm_resolution.py @@ -53,7 +53,7 @@ async def test_get_llm_uses_per_task_model(monkeypatch: pytest.MonkeyPatch, patc monkeypatch.setattr( LLMService, "get_workbench_llm", - AsyncMock(return_value=SimpleNamespace(linsight_default_model_id=None)), + AsyncMock(return_value=SimpleNamespace(linsight_default_model_id=None, models=[])), ) fake_llm = object() get_llm_mock = AsyncMock(return_value=fake_llm) @@ -74,7 +74,7 @@ async def test_get_llm_preserves_original_exception(monkeypatch: pytest.MonkeyPa monkeypatch.setattr( LLMService, "get_workbench_llm", - AsyncMock(return_value=SimpleNamespace(linsight_default_model_id="tenant-default")), + AsyncMock(return_value=SimpleNamespace(linsight_default_model_id="tenant-default", models=[])), ) original = RuntimeError("model offline") monkeypatch.setattr(LLMService, "get_bisheng_linsight_llm", AsyncMock(side_effect=original)) diff --git a/src/backend/test/linsight/test_task_submit_server_side_enqueue.py b/src/backend/test/linsight/test_task_submit_server_side_enqueue.py index 45d61db4c6..b2129f8cb1 100644 --- a/src/backend/test/linsight/test_task_submit_server_side_enqueue.py +++ b/src/backend/test/linsight/test_task_submit_server_side_enqueue.py @@ -4,11 +4,14 @@ a ``linsight_task_handoff`` event, and only then did the browser POST ``/workbench/start-execute``. Everything between those two steps was a window in which the task could be lost — and it was not a narrow one, because -``submit_user_question`` parses every attachment inline. A production task with -12 attachments spent minutes in that call; the user stopped waiting, so the -second request never came and the session sat at NOT_STARTED forever. The +``submit_user_question`` used to parse every attachment inline. A production task +with 12 attachments spent 19 minutes in that call; the user stopped waiting, so +the second request never came and the session sat at NOT_STARTED forever. The conversation lost its task row too, which is why even the task-mode badge -vanished on reload. +vanished on reload. Ingestion now runs in the worker, so the window is short — +but a short window is still a window, and the browser is still optional. (Where +the parsing went, and what deliberately stayed behind in the request, is pinned +in ``test_deferred_ingest.py``.) So: submit enqueues server-side, and start-execute degrades to a late retry. That makes double-enqueue the normal case (server + client), which is safe diff --git a/src/backend/test/linsight/test_unified_submit_session.py b/src/backend/test/linsight/test_unified_submit_session.py index ae8f962826..f45309780a 100644 --- a/src/backend/test/linsight/test_unified_submit_session.py +++ b/src/backend/test/linsight/test_unified_submit_session.py @@ -7,7 +7,7 @@ as flow_type=15 — matching how ``workstation/chat_service`` mints daily chats — so the conversation reads as one stream regardless of task mode (C8, design §3.1). -DAO / telemetry / file-processing are patched; the unit under test is the +DAO / telemetry / attachment-promotion are patched; the unit under test is the session-creation branch of submit_user_question, not DB or MinIO access. """ @@ -45,7 +45,12 @@ async def _fake_insert_message(_cls, data: ChatMessage): monkeypatch.setattr(MessageSessionDao, "async_insert_one", classmethod(_fake_insert_session)) monkeypatch.setattr(LinsightSessionVersionDao, "insert_one", classmethod(_fake_insert_version)) monkeypatch.setattr(ChatMessageDao, "ainsert_one", classmethod(_fake_insert_message)) - monkeypatch.setattr(LinsightWorkbenchImpl, "_process_submitted_files", AsyncMock(return_value=None)) + # Attachment parsing left this request (submit only parks the refs in + # pending_files now), so stubbing ``_process_submitted_files`` isolated + # nothing. What still reaches storage is the temp -> permanent promotion of + # the display attachments, and that is what has to be stubbed instead — the + # display-files test below carries a tmp-bucket link. + monkeypatch.setattr(workbench_impl, "promote_chat_attachments", AsyncMock(side_effect=lambda files, _uid: files)) monkeypatch.setattr(workbench_impl.telemetry_service, "log_event", AsyncMock(return_value=None)) return captured diff --git a/src/backend/test/linsight/test_unified_task_turn_write.py b/src/backend/test/linsight/test_unified_task_turn_write.py index 2f58a914cf..30a5826a3f 100644 --- a/src/backend/test/linsight/test_unified_task_turn_write.py +++ b/src/backend/test/linsight/test_unified_task_turn_write.py @@ -94,6 +94,75 @@ async def test_user_turn_written_as_question(capture_message): assert "帮我写周报" in msg.message +def _question_row(svid: str, chat_id: str, files: list[dict]) -> ChatMessage: + return ChatMessage( + id=11, + is_bot=False, + chat_id=chat_id, + user_id=1, + flow_id="", + type="over", + category="question", + sender="User", + message=json.dumps({"query": "总结下附件", "files": files}, ensure_ascii=False), + files=json.dumps(files), + extra=json.dumps({"linsight_session_version_id": svid}), + tenant_id=1, + ) + + +def _ingested_session(svid: str, chat_id: str, files: list[dict]) -> LinsightSessionVersion: + return LinsightSessionVersion( + id=svid, session_id=chat_id, user_id=1, question="总结下附件", files=files, tenant_id=1 + ) + + +async def test_question_row_carries_the_sv_pointer_so_the_worker_can_find_it(capture_message): + """With the ingest deferred, everything the parse learns about the files + arrives minutes after this row is written. The pointer is how the worker gets + back to this exact row — the bot task turn has used the same one all along.""" + await linsight_execute_utils.persist_task_user_turn( + chat_id="chat-6", user_id=1, question="总结下附件", files=[{"file_id": "v1"}], session_version_id="SV-6" + ) + + assert json.loads(capture_message["message"].extra)["linsight_session_version_id"] == "SV-6" + + +async def test_worker_back_fills_the_video_cover_onto_the_question_row(capture_message): + """The poster frame is produced by the ingest, which now runs in the worker — + so the chip that was persisted at submit has no cover to render and a + task-mode video shows as a blank card on every reload. (The live session only + looks right because the client stamps its own copy in memory.)""" + row = _question_row("SV-6", "chat-6", [{"file_id": "v1", "filename": "clip.mp4"}]) + capture_message["existing"] = [row] + session = _ingested_session( + "SV-6", + "chat-6", + [{"file_id": "v1", "valid": True, "parsing_status": "completed", "cover_filepath": "chat/1/cover.jpg"}], + ) + + assert await linsight_execute_utils.annotate_task_user_turn_files(session) is True + + assert capture_message.get("updated") is True + persisted = json.loads(capture_message["message"].message)["files"][0] + assert persisted["cover_filepath"] == "chat/1/cover.jpg" + assert persisted["parsing_status"] == "completed" + # The mirrored column has to move with it — history rendering reads it too. + assert json.loads(capture_message["message"].files)[0]["cover_filepath"] == "chat/1/cover.jpg" + + +async def test_back_fill_never_touches_another_turns_question_row(capture_message): + """One conversation holds every turn's question. Matching on "the newest + question row" would stamp turn N's parse result onto turn N+1's attachments; + the SV pointer is what keeps them apart.""" + other = _question_row("SV-OTHER", "chat-6", [{"file_id": "v1", "filename": "clip.mp4"}]) + capture_message["existing"] = [other] + session = _ingested_session("SV-6", "chat-6", [{"file_id": "v1", "cover_filepath": "chat/1/cover.jpg"}]) + + assert await linsight_execute_utils.annotate_task_user_turn_files(session) is False + assert capture_message.get("updated") is None + + async def test_failed_task_turn_falls_back_to_error_message(capture_message): """A failed task (no answer) still writes a task turn, using the error message.""" session = LinsightSessionVersion( @@ -117,8 +186,15 @@ async def test_persist_upserts_existing_placeholder_row(capture_message): """If a task-turn row for this SV already exists (placeholder written at start), completion UPDATES it in place rather than inserting a duplicate.""" placeholder = ChatMessage( - id=42, is_bot=True, chat_id="chat-7", user_id=1, flow_id="", type="over", - category="task", message="", extra=json.dumps({"linsight_session_version_id": "SV-9"}), + id=42, + is_bot=True, + chat_id="chat-7", + user_id=1, + flow_id="", + type="over", + category="task", + message="", + extra=json.dumps({"linsight_session_version_id": "SV-9"}), tenant_id=1, ) capture_message["existing"] = [placeholder] diff --git a/src/backend/test/linsight/test_workbench_attachments.py b/src/backend/test/linsight/test_workbench_attachments.py index 3db29291d9..38c413c078 100644 --- a/src/backend/test/linsight/test_workbench_attachments.py +++ b/src/backend/test/linsight/test_workbench_attachments.py @@ -417,7 +417,7 @@ def test_annotate_display_files_stamps_parse_result(): "cover_filepath": "tmp/cover.jpg", }, ] - out = LinsightWorkbenchImpl._annotate_display_files(display, processed) + out = LinsightWorkbenchImpl.annotate_display_files(display, processed) by_id = {f["file_id"]: f for f in out} assert by_id["ok"]["valid"] is True assert by_id["ok"]["parsing_status"] == "completed" @@ -431,8 +431,28 @@ def test_annotate_display_files_stamps_parse_result(): def test_annotate_display_files_handles_empty(): - assert LinsightWorkbenchImpl._annotate_display_files(None, []) is None - assert LinsightWorkbenchImpl._annotate_display_files([], None) == [] + assert LinsightWorkbenchImpl.annotate_display_files(None, []) is None + assert LinsightWorkbenchImpl.annotate_display_files([], None) == [] + + +def test_annotate_display_files_keeps_an_object_name_that_is_already_set(): + """With the ingest deferred, submit promotes each attachment out of the temp + bucket itself and stamps that key — and THAT is the copy conversation + deletion sweeps. The worker's later back-fill must not repoint it at the + workspace original, which would orphan the promoted object.""" + display = [ + {"file_id": "promoted", "filename": "a.png", "object_name": "chat/7/abc.png"}, + {"file_id": "fresh", "filename": "b.png"}, + ] + processed = [ + {"file_id": "promoted", "valid": True, "original_file_path": "linsight/sv1/promoted_original.png"}, + {"file_id": "fresh", "valid": True, "original_file_path": "linsight/sv1/fresh_original.png"}, + ] + + by_id = {f["file_id"]: f for f in LinsightWorkbenchImpl.annotate_display_files(display, processed)} + + assert by_id["promoted"]["object_name"] == "chat/7/abc.png" + assert by_id["fresh"]["object_name"] == "linsight/sv1/fresh_original.png" async def test_expired_temp_no_formal_marks_invalid(): diff --git a/src/backend/test/permission/fixtures/f048_bench_contract.synthetic.json b/src/backend/test/permission/fixtures/f048_bench_contract.synthetic.json index 1605c486b0..428ffc3bff 100644 --- a/src/backend/test/permission/fixtures/f048_bench_contract.synthetic.json +++ b/src/backend/test/permission/fixtures/f048_bench_contract.synthetic.json @@ -1,229 +1,143 @@ { - "contract_version": "f048-bench-01-v1", - "authorization_model_checksum": "edf2c67ebabf5cd24bd9d520181e50ee12c9cf667061488c6886db2253042ee4", + "contract_version": "f048-bench-01-v2", + "authorization_model_checksum": "cdcb8c337e756305966082e0d9cd5540e794692e614a048de9f8efbbf168350f", "dataset": { - "name": "f048-bench-synthetic-contract", - "source": "synthetic-ci-harness-only", + "name": "f048-single-slot-visible-synthetic-v2", + "source": "checksum-pinned-representative-synthetic-not-production", "production_derived": false, - "dataset_checksum": "ad302adc12b1807080f85c238b36c7e93a75ef1c39165e6250a3138710f6e54b", - "profile": { - "catalog_release_id": "bench-catalog-v1", - "model_release_id": "bench-model-release-v1", - "permission_model_id": "bench-standard-v1", - "scenarios": [ - { - "name": "direct", - "subject_kind": "direct", - "actor_id": 1001, - "resource_type": "workflow", - "resource_id_prefix": "f048-direct-", - "action": "use", - "result_count": 10, - "expected_object_checksum": "75abb7f47147ba1673c9819045c42d00266c028f8ffe10b038a0517e0552259a" - }, - { - "name": "department", - "subject_kind": "department", - "actor_id": 1002, - "resource_type": "workflow", - "resource_id_prefix": "f048-department-", - "action": "use", - "result_count": 10, - "expected_object_checksum": "b435f1c1ee6607b56900e829f1592782f031dacc991585727b3b3bebdb4db9dd" - }, - { - "name": "group", - "subject_kind": "group", - "actor_id": 1003, - "resource_type": "workflow", - "resource_id_prefix": "f048-group-", - "action": "use", - "result_count": 10, - "expected_object_checksum": "68a008fca9dd7f4c0db965e0d8f1e9215b95df85a8dc770a36d1cca88ea859fe" - }, - { - "name": "inherit", - "subject_kind": "inherit", - "actor_id": 1004, - "resource_type": "knowledge_file", - "resource_id_prefix": "f048-inherit-", - "action": "download", - "result_count": 10, - "expected_object_checksum": "cc27cc709d27c66a1dd15d0fbb815115d8905b594ac9710424006076e9e5c2e7" - }, - { - "name": "multi_grant", - "subject_kind": "multi_grant", - "actor_id": 1005, - "resource_type": "workflow", - "resource_id_prefix": "f048-multi-grant-", - "action": "use", - "result_count": 10, - "expected_object_checksum": "8160f323d3b98ad435cd69bb400a85fe8e81566aeabd322c7776cb8bab35e65c" - }, - { - "name": "result_10", - "subject_kind": "direct", - "actor_id": 1010, - "resource_type": "workflow", - "resource_id_prefix": "f048-result-10-", - "action": "use", - "result_count": 10, - "expected_object_checksum": "e2a1f753e092b10af46c9759056b922e3147a90706216b5af34433cdc53781a8" - }, - { - "name": "result_100", - "subject_kind": "direct", - "actor_id": 1100, - "resource_type": "workflow", - "resource_id_prefix": "f048-result-100-", - "action": "use", - "result_count": 100, - "expected_object_checksum": "1c718ed1c394291b91ed4135831fe66d2cb14200ca4e431b1698b719d0359c90" - }, - { - "name": "result_1000", - "subject_kind": "direct", - "actor_id": 2000, - "resource_type": "workflow", - "resource_id_prefix": "f048-result-1000-", - "action": "use", - "result_count": 1000, - "expected_object_checksum": "7da7c02aeb5f11c7ce10c70f48959aa718a3f674091a214823c10c0f1bdeb6d6" - } - ], - "business_cursor": { - "scenario": "result_100", - "candidate_count": 100, + "representative_distribution": true, + "representative_reason": "Covers the design-mandated N_db/V matrix, every canonical subject kind, multi-source overlap, and all three list path shapes without copying production rows.", + "dataset_checksum": "d083c56febda9eb055e9ae5356f800ca837a1e534628fdeda623dc4b8063ab22", + "source_checksum": "9e2e7699dbe65851de0b04651ee8bdb009e2a85ac16e8c12a5b9e5092c220df0", + "visible_checksum": "f185c204418fbd98d41c15b0bac7759060c2df73da8c8c4fcf381f0a02c1994e", + "scenarios": [ + { + "name": "n10000_v10_direct", + "source_kind": "direct", + "actor_id": 1010, + "resource_type": "workflow", + "resource_id_prefix": "bench-n10k-v10-", + "resource_count": 10000, + "visible_count": 10, + "expected_object_checksum": "276af3de69921d262e61bd168ab288d0609f685dd2547e9a1a618a414bec633d" + }, + { + "name": "n10000_v100_department", + "source_kind": "department", + "actor_id": 1100, + "resource_type": "assistant", + "resource_id_prefix": "bench-n10k-v100-", + "resource_count": 10000, + "visible_count": 100, + "expected_object_checksum": "cd02436e0256be2f7897168846ab8d3c33eb5324b2d47bea912fd083bf0e4ca6" + }, + { + "name": "n10000_v1000_group", + "source_kind": "group", + "actor_id": 2000, + "resource_type": "tool", + "resource_id_prefix": "bench-n10k-v1000-", + "resource_count": 10000, + "visible_count": 1000, + "expected_object_checksum": "2e083800048bf32787bbae4a55d692985e236a313fe95bbf1c69ab043bc08090" + }, + { + "name": "n10000_v5000_system", + "source_kind": "system", + "actor_id": 6000, + "resource_type": "dashboard", + "resource_id_prefix": "bench-n10k-v5000-", + "resource_count": 10000, + "visible_count": 5000, + "expected_object_checksum": "39774222a1afd329db8148b2fe00bb9f466b22cfccec1d1c30f71049bf3fe4fa" + }, + { + "name": "n100000_v10_direct", + "source_kind": "direct", + "actor_id": 10010, + "resource_type": "channel", + "resource_id_prefix": "bench-n100k-v10-", + "resource_count": 100000, + "visible_count": 10, + "expected_object_checksum": "59c5930799bb1af3c8261121b5fbb73860ee779f65e09067ff47a0c34724f76c" + }, + { + "name": "n100000_v100_department", + "source_kind": "department", + "actor_id": 10100, + "resource_type": "knowledge_space", + "resource_id_prefix": "bench-n100k-v100-", + "resource_count": 100000, + "visible_count": 100, + "expected_object_checksum": "b3cd3850fc6ddcc900320c8f7152e33bdc985fde3a1bcef5b83d62cc01cc93a6" + }, + { + "name": "n100000_v1000_group", + "source_kind": "group", + "actor_id": 11000, + "resource_type": "knowledge_library", + "resource_id_prefix": "bench-n100k-v1000-", + "resource_count": 100000, + "visible_count": 1000, + "expected_object_checksum": "5a3da0068b3ab75fb0d61e1ab2332ea114182f72b945f1900a2f8f7b85488ed0" + }, + { + "name": "n100000_v5000_multi", + "source_kind": "multi_source", + "actor_id": 15000, + "resource_type": "knowledge_file", + "resource_id_prefix": "bench-n100k-v5000-", + "resource_count": 100000, + "visible_count": 5000, + "expected_object_checksum": "b489f002d624e9e36aa3b925c1af6b8795edc1b521cf60f5fb6ebd2617123162" + } + ], + "business_paths": { + "joined": { + "scenario": "n100000_v100_department", + "strategy": "visible_id_first", "page_size": 20 + }, + "department": { + "scenario": "n100000_v100_department", + "strategy": "candidate_first", + "n_db": 200, + "visible_total": 100, + "selectivity": 0.5, + "page_size": 20, + "batch_size": 50 + }, + "file": { + "scenario": "n100000_v5000_multi", + "strategy": "candidate_first", + "n_db": 100000, + "visible_total": 5000, + "selectivity": 0.95, + "page_size": 50, + "batch_size": 100 } } }, - "baseline": { - "source": "synthetic-contract-values-not-a-production-baseline", - "check_ms": [ - 20.1, - 20.4, - 20.8, - 21.0, - 21.3, - 21.7, - 22.0, - 22.2, - 22.4, - 22.8, - 23.0, - 23.3, - 23.5, - 23.8, - 24.0, - 24.2, - 24.5, - 24.8, - 25.0, - 25.4 - ], - "batch_check_ms": { - "20": [ - 30.0, - 30.5, - 31.0, - 31.5, - 32.0, - 32.5, - 33.0, - 33.5, - 34.0, - 34.5, - 35.0, - 35.5, - 36.0, - 36.5, - 37.0, - 37.5, - 38.0, - 38.5, - 39.0, - 39.5 - ], - "50": [ - 45.0, - 45.8, - 46.6, - 47.4, - 48.2, - 49.0, - 49.8, - 50.6, - 51.4, - 52.2, - 53.0, - 53.8, - 54.6, - 55.4, - 56.2, - 57.0, - 57.8, - 58.6, - 59.4, - 60.2 - ], - "100": [ - 70.0, - 71.0, - 72.0, - 73.0, - 74.0, - 75.0, - 76.0, - 77.0, - 78.0, - 79.0, - 80.0, - 81.0, - 82.0, - 83.0, - 84.0, - 85.0, - 86.0, - 87.0, - 88.0, - 89.0 - ] - }, - "business_cursor_ms": [ - 80.0, - 81.0, - 82.0, - 83.0, - 84.0, - 85.0, - 86.0, - 87.0, - 88.0, - 89.0, - 90.0, - 91.0, - 92.0, - 93.0, - 94.0, - 95.0, - 96.0, - 97.0, - 98.0, - 99.0 - ] - }, "limits": { - "check_p95_multiplier": 1.2, - "check_p95_absolute_ms": 5.0, - "batch_p95_multiplier": 1.25, - "business_cursor_p95_multiplier": 1.2, + "check_p95_ms": 50, + "batch_check_p95_ms": { + "20": 100, + "50": 200, + "100": 300 + }, + "stream_p95_ms": { + "10": 50, + "100": 100, + "1000": 300, + "5000": 1000 + }, "max_error_rate": 0.001 }, "run": { "warmup": 2, - "iterations": 30 + "iterations": 30, + "concurrency": 1, + "consistency": "HIGHER_CONSISTENCY" }, - "contract_checksum": "12654227efbcdb9ebc2effdd1da5c668723cdcd96c4333eb621f0229b190dc14" + "contract_checksum": "80a80489bf4f3ae01f5b1f3519cd275b10a92f16dd58df5fa1378de48c608912" } diff --git a/src/backend/test/permission/test_f027_rebac_static_grep.py b/src/backend/test/permission/test_f027_rebac_static_grep.py index 80b9815eb3..70062c4c70 100644 --- a/src/backend/test/permission/test_f027_rebac_static_grep.py +++ b/src/backend/test/permission/test_f027_rebac_static_grep.py @@ -16,7 +16,6 @@ import subprocess from pathlib import Path - _BISHENG = Path(__file__).resolve().parents[2] / "bisheng" @@ -32,7 +31,7 @@ def _grep_count(pattern: str, *paths: str, extra_args: list[str] | None = None) proc = subprocess.run(cmd, capture_output=True, text=True) if proc.returncode == 1: # 1 = no matches, normal return 0 - lines = [l for l in proc.stdout.splitlines() if l.strip()] + lines = [line for line in proc.stdout.splitlines() if line.strip()] return len(lines) @@ -110,7 +109,7 @@ def test_scan_visible_child_items_has_cursor_loop_invariants(): body = m.group(0) assert "batch_cursor" in body assert "_compute_ext_rank_python" in body - assert re.search(r"len\(visible_page_items\)\s*>\s*page_size", body) + assert re.search(r"len\(visible_page_items\)\s*==\s*page_size", body) # The old OFFSET-style scan_page state is gone assert "scan_page" not in body diff --git a/src/backend/test/permission/test_f048_authorization_model.py b/src/backend/test/permission/test_f048_authorization_model.py index 9dcf69b09b..8934785fc4 100644 --- a/src/backend/test/permission/test_f048_authorization_model.py +++ b/src/backend/test/permission/test_f048_authorization_model.py @@ -2,7 +2,7 @@ 覆盖 AC: AC-02, AC-04, AC-07, AC-08, AC-15, AC-19, AC-20, AC-21, AC-22, AC-26, AC-28, AC-29, AC-33, AC-36, AC-37, AC-38, AC-39, -AC-40, AC-41, AC-45, AC-46, AC-47 +AC-40, AC-41, AC-45, AC-46, AC-47, AC-159, AC-161, AC-163, AC-164 """ from __future__ import annotations @@ -144,6 +144,7 @@ def _resource_grant_tuples( subject: str, protected: bool = False, custom: bool = True, + flattened_visible: bool = True, ) -> set[TupleKey]: relation = "protected_assignee" if protected else "ordinary_assignee" tuples: set[TupleKey] = { @@ -154,6 +155,8 @@ def _resource_grant_tuples( } if custom: tuples.add(("user:*", "custom_mode", resource)) + if flattened_visible: + tuples.add((subject, "visible", resource)) return tuples @@ -208,6 +211,7 @@ def test_catalog_model_and_permission_enabled_are_all_required() -> None: tuples - {required_tuple}, ) assert not without_required.check("user:7", "can_edit", resource) + assert without_required.check("user:7", "visible", resource) def test_custom_gate_blocks_ordinary_but_not_protected_assignment() -> None: @@ -235,7 +239,7 @@ def test_custom_gate_blocks_ordinary_but_not_protected_assignment() -> None: assert evaluator.check("user:8", "can_edit", resource) -def test_inherit_uses_only_canonical_parent_and_system_visibility_ignores_mode() -> None: +def test_inherit_uses_only_canonical_parent_and_projected_system_visibility_ignores_mode() -> None: parent = "knowledge_space:s1" child = "knowledge_file:f1" grant = "permission_grant:g1" @@ -253,12 +257,14 @@ def test_inherit_uses_only_canonical_parent_and_system_visibility_ignores_mode() } evaluator = ModelEvaluator(build_authorization_model_f048(), tuples) assert evaluator.check("user:7", "can_download", child) + assert evaluator.check("user:7", "visible", child) custom_child = ModelEvaluator( build_authorization_model_f048(), tuples - {("user:*", "inherit_mode", child)} | {("user:*", "custom_mode", child)}, ) assert not custom_child.check("user:7", "can_download", child) + assert not custom_child.check("user:7", "visible", child) system_custom_child = ModelEvaluator( build_authorization_model_f048(), @@ -272,6 +278,21 @@ def test_inherit_uses_only_canonical_parent_and_system_visibility_ignores_mode() assert system_custom_child.check("user:99", "can_download", child) +def test_only_top_level_resources_use_pure_flat_visible() -> None: + model = build_authorization_model_f048() + types = {definition["type"]: definition for definition in model["type_definitions"]} + + for resource_type in ("knowledge_space", "workflow"): + visible = types[resource_type]["relations"]["visible"] + assert visible["union"]["child"][0] == {"this": {}} + assert visible["union"]["child"][1] == {"computedUserset": {"relation": "system_visible"}} + for resource_type in ("folder", "knowledge_file"): + relations = types[resource_type]["relations"] + assert relations["visible"] != {"this": {}} + assert "inherited_visible" in relations + assert "system_visible" in relations + + def test_department_subtree_and_user_group_usersets_are_not_expanded() -> None: resource = "channel:c1" grant = "permission_grant:g1" @@ -341,3 +362,83 @@ def test_shared_system_read_is_resource_specific_and_read_only() -> None: assert not evaluator.check("user:7", "can_edit", "knowledge_library:k1") assert evaluator.check("user:7", "can_download", "knowledge_file:f1") assert not evaluator.check("user:7", "can_use", "knowledge_file:f1") + + +def test_visible_is_single_slot_shallow_and_has_no_ab_switch() -> None: + model = build_authorization_model_f048() + types = {definition["type"]: definition for definition in model["type_definitions"]} + for type_name in ("knowledge_space", "dashboard"): + relations = types[type_name]["relations"] + assert "visible" in relations + assert relations["visible"]["union"]["child"][0] == {"this": {}} + assert "ordinary_visible" not in relations + assert "protected_visible" not in relations + assert all("visible_a" not in relation and "visible_b" not in relation for relation in relations) + assert all("visibility_switch" not in relation for relation in relations) + + for type_name in ("folder", "knowledge_file"): + relations = types[type_name]["relations"] + assert relations["visible"] != {"this": {}} + assert "inherited_visible" in relations + assert "system_visible" in relations + assert all("visible_a" not in relation and "visible_b" not in relation for relation in relations) + assert all("visibility_switch" not in relation for relation in relations) + + grant_relations = types["permission_grant"]["relations"] + assert "ordinary_visible" not in grant_relations + assert "protected_visible" not in grant_relations + + resource = "knowledge_space:s1" + evaluator = ModelEvaluator( + model, + { + ("department:17#member", "visible", resource), + ("user:7", "member", "department:17"), + }, + ) + assert evaluator.check("user:7", "visible", resource) + + +def test_inactive_model_keeps_existing_visible_actions_and_manage() -> None: + resource = "workflow:w-inactive" + grant = "permission_grant:g-inactive" + release = "permission_model_release:catalog-1~editor" + tuples = _active_model_tuples(actions=("edit", "manage_permission")) + tuples.remove(("user:*", "enabled_marker", release)) + tuples |= _resource_grant_tuples( + resource=resource, + grant=grant, + model_key="editor", + subject="user:7", + ) + + evaluator = ModelEvaluator(build_authorization_model_f048(), tuples) + assert evaluator.check("user:7", "visible", resource) + assert evaluator.check("user:7", "can_edit", resource) + assert evaluator.check("user:7", "can_manage_permission", resource) + + missing_model = ModelEvaluator( + build_authorization_model_f048(), + tuples + - { + (release, "release", "permission_model:editor"), + ("user:7", "visible", resource), + }, + ) + assert not missing_model.check("user:7", "visible", resource) + assert not missing_model.check("user:7", "can_edit", resource) + assert not missing_model.check("user:7", "can_manage_permission", resource) + + +def test_admin_identity_does_not_expand_personal_visible_set() -> None: + resource = "knowledge_space:private" + evaluator = ModelEvaluator( + build_authorization_model_f048(), + { + ("user:1", "super_admin", "system:root"), + ("user:1", "admin", "tenant:1"), + ("user:*", "permission_enabled", resource), + ("user:*", "custom_mode", resource), + }, + ) + assert not evaluator.check("user:1", "visible", resource) diff --git a/src/backend/test/permission/test_f048_catalog_runtime.py b/src/backend/test/permission/test_f048_catalog_runtime.py index 8abea92f4f..6ab50a91ec 100644 --- a/src/backend/test/permission/test_f048_catalog_runtime.py +++ b/src/backend/test/permission/test_f048_catalog_runtime.py @@ -4,6 +4,7 @@ from collections.abc import AsyncIterator, Callable from contextlib import AbstractAsyncContextManager, asynccontextmanager +from datetime import UTC, datetime, timedelta import pytest import sqlalchemy as sa @@ -82,6 +83,9 @@ "permission_catalog_projection_tuple", "permission_projection_operation", "permission_projection_tuple", + "permission_migration_run", + "permission_migration_item", + "permission_visible_source_projection", "permission_grant", "permission_grant_assignee", "resource_permission_mode", @@ -306,6 +310,50 @@ def _api( ) +async def test_catalog_draft_can_bind_one_same_store_authorization_release( + session_factory: SessionFactory, +) -> None: + fga = InMemoryCatalogFGA() + current = await _seed_current(session_factory, fga) + async with session_factory() as session: + async with session.begin(): + target = AuthorizationModelRelease( + environment="test", + store_id=fga.store_id, + model_version="f048-v2", + model_id="model-v2", + predecessor_model_id=fga.model_id, + model_checksum="a" * 64, + required_relations_checksum="b" * 64, + openfga_version="1.15.1", + status="STAGED", + ) + session.add(target) + await session.flush() + target_id = int(target.id) + + state = SqlCatalogState(session_factory=session_factory) + reservation = await state.reserve_draft( + base_release_id=int(current.id), + operator_id=7, + idempotency_key="authorization-model-upgrade", + expires_at=datetime.now(UTC).replace(tzinfo=None) + timedelta(minutes=10), + ) + await state.bind_draft_authorization_release( + draft_id=reservation.release_id, + authorization_release_id=target_id, + ) + await state.bind_draft_authorization_release( + draft_id=reservation.release_id, + authorization_release_id=target_id, + ) + + async with session_factory() as session: + draft = await session.get(PermissionCatalogRelease, reservation.release_id) + assert draft is not None + assert draft.required_authorization_model_release_id == target_id + + async def test_decision_runtime_rejects_catalog_model_pin_drift( session_factory: SessionFactory, monkeypatch, @@ -980,29 +1028,35 @@ async def test_deactivate_and_delete_in_one_batch( assert "collaborator" not in {row.model_key for row in rows} -async def test_an_active_model_alone_is_still_refused( +async def test_active_model_can_be_deleted_when_reference_audit_is_zero( session_factory: SessionFactory, ) -> None: - """The guard itself stays: deleting a live model needs the deactivation.""" - - from bisheng.common.errcode.permission import PermissionModelStateConflictError + """Active controls future assignment, not whether a zero-ref model can delete.""" fga = InMemoryCatalogFGA() marker = FakeCatalogMarker() current = await _seed_current(session_factory, fga) api = _api(session_factory, fga, marker) - with pytest.raises(PermissionModelStateConflictError): - await api.create_draft( - request=CatalogDraftRequest( - idempotency_key="delete-while-active", - base_release_id=int(current.id), - changes=( - CatalogChangeRequest( - type=CatalogChangeType.DELETE_MODEL, - model_key="collaborator", - ), + draft = await api.create_draft( + request=CatalogDraftRequest( + idempotency_key="delete-while-active", + base_release_id=int(current.id), + changes=( + CatalogChangeRequest( + type=CatalogChangeType.DELETE_MODEL, + model_key="collaborator", ), ), - operator_id=7, + ), + operator_id=7, + ) + async with session_factory() as session: + rows = list( + ( + await session.execute( + select(PermissionModel).where(PermissionModel.catalog_release_id == draft["draft_id"]) + ) + ).scalars() ) + assert "collaborator" not in {row.model_key for row in rows} diff --git a/src/backend/test/permission/test_f048_catalog_service.py b/src/backend/test/permission/test_f048_catalog_service.py index c0bbe527c2..c3de5efc44 100644 --- a/src/backend/test/permission/test_f048_catalog_service.py +++ b/src/backend/test/permission/test_f048_catalog_service.py @@ -1,7 +1,7 @@ """F048 Catalog publisher crash-matrix contracts. 覆盖 AC: AC-03, AC-06, AC-13, AC-14, AC-16, AC-17, AC-18, -AC-66, AC-67, AC-68, AC-69, AC-143, AC-156 +AC-27, AC-66, AC-67, AC-68, AC-69, AC-143, AC-156, AC-164, AC-165, AC-167 """ from __future__ import annotations @@ -13,6 +13,7 @@ from bisheng.common.errcode.permission import ( AuthorizationModelMismatchError, PermissionImpactExpiredError, + PermissionModelStateConflictError, PermissionProjectionFailedError, PermissionPublishNotReadyError, PermissionVersionConflictError, @@ -33,6 +34,7 @@ ) from bisheng.permission.domain.services.model_policy import ( CustomModelSelection, + ModelReferenceSummary, derive_permission_models, ) @@ -342,6 +344,88 @@ async def test_build_draft_recomputes_complete_release_and_cross_tenant_impact() assert state.log == ["save_draft"] +@pytest.mark.asyncio +async def test_delete_model_accepts_active_zero_reference_and_preserves_history() -> None: + actions = derive_action_release(_assigned_actions()) + custom = CustomModelSelection( + model_key="delete-directly", + name="直接删除", + action_codes=("edit", "manage_permission"), + active=True, + ) + before_models = derive_permission_models(actions, custom_models=(custom,)) + analysis = CatalogImpactSummary( + checksum="8" * 64, + resource_count=0, + grant_count=0, + assignee_count=0, + expansion_count=0, + revocation_count=0, + ) + service, state, _projector, _events = _service( + _draft(), + impact=FakeImpact(analysis=analysis), + ) + + draft = await service.build_draft( + CatalogDraftBuildInput( + release_id=2, + release_key="catalog-v2", + predecessor_release_id=1, + predecessor_release_key="catalog-v1", + before_actions=actions, + before_models=before_models, + actions=_assigned_actions(), + custom_models=(), + model_reference_summaries={ + "delete-directly": ModelReferenceSummary(), + }, + ) + ) + + assert draft.model_release is not None + assert "delete-directly" not in {model.model_key for model in draft.model_release.models} + assert "delete-directly" in {model.model_key for model in before_models.models} + assert state.log == ["save_draft"] + + +@pytest.mark.asyncio +async def test_delete_model_fails_closed_for_unknown_or_nonzero_references() -> None: + actions = derive_action_release(_assigned_actions()) + custom = CustomModelSelection( + model_key="blocked-delete", + name="阻断删除", + action_codes=("edit",), + ) + before_models = derive_permission_models(actions, custom_models=(custom,)) + + for summaries in ( + {}, + { + "blocked-delete": ModelReferenceSummary( + failed_source_count=1, + residual_checksum="a" * 64, + ) + }, + ): + service, state, _projector, _events = _service(_draft()) + with pytest.raises(PermissionModelStateConflictError): + await service.build_draft( + CatalogDraftBuildInput( + release_id=2, + release_key="catalog-v2", + predecessor_release_id=1, + predecessor_release_key="catalog-v1", + before_actions=actions, + before_models=before_models, + actions=_assigned_actions(), + custom_models=(), + model_reference_summaries=summaries, + ) + ) + assert state.log == [] + + @pytest.mark.asyncio async def test_publish_orders_fence_stage_marker_two_tuple_commit_and_finalize() -> None: service, state, projector, events = _service(_draft()) diff --git a/src/backend/test/permission/test_f048_database_integration.py b/src/backend/test/permission/test_f048_database_integration.py index 1c9f9ef567..1254ce52b7 100644 --- a/src/backend/test/permission/test_f048_database_integration.py +++ b/src/backend/test/permission/test_f048_database_integration.py @@ -30,6 +30,9 @@ from bisheng.core.database.alembic.versions import ( f048_permission_model_grants as revision, ) +from bisheng.core.database.alembic.versions import ( + v3_0_0_f048_visible_source_projection as visible_revision, +) REVISION_PATH = Path(revision.__file__) F048_TABLES = ( @@ -47,30 +50,39 @@ "resource_permission_mode", "permission_migration_run", "permission_migration_item", + "permission_visible_source_projection", ) LIVE_ENABLED = os.environ.get("F048_DATABASE_INTEGRATION") == "1" _DML = re.compile(r"\b(SELECT|INSERT|UPDATE|DELETE|MERGE)\s", re.IGNORECASE) def test_alembic_revision_is_schema_only() -> None: - source = REVISION_PATH.read_text(encoding="utf-8") - tree = ast.parse(source) - imported_modules = {node.module for node in ast.walk(tree) if isinstance(node, ast.ImportFrom) and node.module} - assert imported_modules <= { - "collections.abc", - "alembic", - "bisheng.core.database.alembic_helpers.online", - } - assert not _DML.search(source) - assert "session" not in {node.id for node in ast.walk(tree) if isinstance(node, ast.Name)} + for path in (REVISION_PATH, Path(visible_revision.__file__)): + source = path.read_text(encoding="utf-8") + tree = ast.parse(source) + imported_modules = { + node.module + for node in ast.walk(tree) + if isinstance(node, ast.ImportFrom) and node.module + } + assert imported_modules <= { + "collections.abc", + "alembic", + "bisheng.core.database.alembic_helpers.online", + } + assert not _DML.search(source) + assert "session" not in {node.id for node in ast.walk(tree) if isinstance(node, ast.Name)} assert revision.revision == "f048_permission_grants" assert revision.down_revision == "f044_llm_status_time" + assert visible_revision.revision == "f048_visible_source_projection" def test_revision_declares_expected_schema_and_resume_constraints() -> None: source = REVISION_PATH.read_text(encoding="utf-8") - for table in F048_TABLES: + visible_source = Path(visible_revision.__file__).read_text(encoding="utf-8") + for table in F048_TABLES[:-1]: assert f'"{table}"' in source + assert "permission_visible_source_projection" in visible_source for constraint in ( "uq_auth_model_release", "uq_perm_catalog_version", @@ -80,8 +92,10 @@ def test_revision_declares_expected_schema_and_resume_constraints() -> None: "uq_perm_migration_environment", "uq_perm_migration_item_source", "ix_perm_migration_item_resume", + "uq_perm_visible_source_contribution", + "ix_perm_visible_source_owner", ): - assert constraint in source + assert constraint in source or constraint in visible_source assert "dashboard" in source assert "tenant_id" in source @@ -140,6 +154,14 @@ def _apply_revision( lambda table, index: any(item["name"] == index for item in inspector.get_indexes(table)), ) revision.upgrade() + inspector = sa.inspect(connection) + monkeypatch.setattr(visible_revision, "op", operations) + monkeypatch.setattr( + visible_revision, + "table_exists", + lambda table: inspector.has_table(table), + ) + visible_revision.upgrade() def _assert_schema(engine: Engine) -> None: @@ -151,18 +173,35 @@ def _assert_schema(engine: Engine) -> None: "resource_permission_mode": {"uq_resource_permission_mode"}, "permission_migration_run": {"uq_perm_migration_environment"}, "permission_migration_item": {"uq_perm_migration_item_source"}, + "permission_visible_source_projection": { + "uq_perm_visible_source_contribution" + }, } for table, required in expected_uniques.items(): names = {str(item.get("name")) for item in inspector.get_unique_constraints(table)} assert required.issubset(names) resume_indexes = {str(item.get("name")) for item in inspector.get_indexes("permission_migration_item")} assert "ix_perm_migration_item_resume" in resume_indexes + visible_indexes = { + str(item.get("name")) + for item in inspector.get_indexes("permission_visible_source_projection") + } + assert { + "ix_perm_visible_resource_subject", + "ix_perm_visible_model_state", + "ix_perm_visible_source_owner", + }.issubset(visible_indexes) def _exercise_checkpoint_contract(engine: Engine) -> None: metadata = sa.MetaData() run = sa.Table("permission_migration_run", metadata, autoload_with=engine) item = sa.Table("permission_migration_item", metadata, autoload_with=engine) + visible = sa.Table( + "permission_visible_source_projection", + metadata, + autoload_with=engine, + ) fingerprint = "1" * 64 source_checksum = "2" * 64 target_checksum = "3" * 64 @@ -207,6 +246,24 @@ def _exercise_checkpoint_contract(engine: Engine) -> None: ) ], ) + connection.execute( + visible.insert().values( + tenant_id=1, + resource_type="workflow", + resource_id="1", + visibility_class="ordinary", + projected_subject="user:7", + source_kind="GRANT_ASSIGNEE", + source_owner_key="grant_assignee:11", + source_locator="migration:DIRECT:grant:1", + source_fingerprint="4" * 64, + contribution_fingerprint="5" * 64, + model_key="viewer", + source_version=1, + tuple_fingerprint="6" * 64, + state="ACTIVE", + ) + ) with engine.begin() as connection: first_page = connection.execute( @@ -295,6 +352,26 @@ def _exercise_checkpoint_contract(engine: Engine) -> None: severity="INFO", ) ) + with pytest.raises(sa.exc.IntegrityError): + with engine.begin() as connection: + connection.execute( + visible.insert().values( + tenant_id=1, + resource_type="workflow", + resource_id="1", + visibility_class="ordinary", + projected_subject="user:7", + source_kind="GRANT_ASSIGNEE", + source_owner_key="grant_assignee:12", + source_locator="migration:DIRECT:grant:2", + source_fingerprint="7" * 64, + contribution_fingerprint="5" * 64, + model_key="viewer", + source_version=1, + tuple_fingerprint="6" * 64, + state="ACTIVE", + ) + ) def _run_live_contract( diff --git a/src/backend/test/permission/test_f048_fga_client.py b/src/backend/test/permission/test_f048_fga_client.py index b57570b98e..68792151ee 100644 --- a/src/backend/test/permission/test_f048_fga_client.py +++ b/src/backend/test/permission/test_f048_fga_client.py @@ -1,10 +1,12 @@ """F048 OpenFGA client pin, consistency, and atomic limit contracts. -覆盖 AC: AC-30, AC-31, AC-32, AC-34, AC-69, AC-109, AC-111, AC-112 +覆盖 AC: AC-30, AC-31, AC-32, AC-34, AC-69, AC-109, AC-111, AC-112, +AC-160, AC-161, AC-162, AC-163, AC-168, AC-169, AC-170, AC-171 """ from __future__ import annotations +import asyncio from unittest.mock import AsyncMock import pytest @@ -17,7 +19,11 @@ OPENFGA_WRITE_TUPLE_LIMIT, FGAClient, ) -from bisheng.core.openfga.exceptions import FGAClientError, FGAWriteError +from bisheng.core.openfga.exceptions import ( + FGAClientError, + FGAConnectionError, + FGAWriteError, +) @pytest.fixture @@ -79,6 +85,82 @@ async def test_check_batch_and_list_are_model_scoped_with_consistency( assert body["consistency"] == "HIGHER_CONSISTENCY" +@pytest.mark.asyncio +async def test_stream_list_objects_consumes_normal_end_and_preserves_model_pin( + client: FGAClient, +) -> None: + captured = {} + + async def stream(path, body): + captured.update(path=path, body=body) + yield {"result": {"object": "workflow:w1"}} + yield {"result": {"object": "workflow:w1"}} + yield {"result": {"object": "workflow:w2"}} + + client._streamed_post = stream + + assert await client.stream_list_objects( + user="user:7", + relation="visible", + type="workflow", + consistency="HIGHER_CONSISTENCY", + ) == ("workflow:w1", "workflow:w1", "workflow:w2") + assert captured["path"] == "/stores/store-1/streamed-list-objects" + assert captured["body"] == { + "user": "user:7", + "relation": "visible", + "type": "workflow", + "authorization_model_id": "model-f048", + "consistency": "HIGHER_CONSISTENCY", + } + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "error", + ( + FGAConnectionError("deadline"), + FGAClientError("service error"), + ), +) +async def test_stream_list_objects_never_returns_prefix_after_error( + client: FGAClient, + error: Exception, +) -> None: + async def stream(path, body): + del path, body + yield {"object": "workflow:prefix"} + raise error + + client._streamed_post = stream + + with pytest.raises(type(error), match=str(error)): + await client.stream_list_objects( + user="user:7", + relation="visible", + type="workflow", + ) + + +@pytest.mark.asyncio +async def test_stream_list_objects_propagates_cancellation_without_prefix( + client: FGAClient, +) -> None: + async def stream(path, body): + del path, body + yield {"object": "workflow:prefix"} + raise asyncio.CancelledError + + client._streamed_post = stream + + with pytest.raises(asyncio.CancelledError): + await client.stream_list_objects( + user="user:7", + relation="visible", + type="workflow", + ) + + @pytest.mark.asyncio async def test_batch_check_splits_business_batch_at_openfga_limit( client: FGAClient, @@ -229,12 +311,36 @@ async def test_write_is_single_model_and_enforces_openfga_limit( client._post.assert_called_once() body = client._post.call_args.args[1] assert body["authorization_model_id"] == "model-f048" + assert "on_duplicate" not in body["writes"] with pytest.raises(FGAWriteError): await client.write_tuples(writes=tuples, deletes=[tuples[0]]) assert client._post.call_count == 1 +@pytest.mark.asyncio +async def test_write_can_ignore_duplicate_tuples_for_reconciliation( + client: FGAClient, +) -> None: + client._post = AsyncMock(return_value={}) + relationship = { + "user": "department:7#member", + "relation": "visible", + "object": "knowledge_space:42", + } + + await client.write_tuples( + writes=[relationship], + ignore_duplicate_writes=True, + ) + + body = client._post.call_args.args[1] + assert body["writes"] == { + "tuple_keys": [relationship], + "on_duplicate": "ignore", + } + + def test_business_atomic_limit_is_stricter_than_service_limit() -> None: assert BUSINESS_ATOMIC_TUPLE_LIMIT == 90 assert OPENFGA_WRITE_TUPLE_LIMIT == 100 diff --git a/src/backend/test/permission/test_f048_grant_policy.py b/src/backend/test/permission/test_f048_grant_policy.py index 34e4e1fbab..da911213a3 100644 --- a/src/backend/test/permission/test_f048_grant_policy.py +++ b/src/backend/test/permission/test_f048_grant_policy.py @@ -1,7 +1,7 @@ """F048 grant-level and protected-assignment orchestration contracts. -覆盖 AC: AC-36, AC-37, AC-38, AC-39, AC-40, AC-41, AC-42, AC-43, -AC-44, AC-157 +覆盖 AC: AC-15, AC-36, AC-37, AC-38, AC-39, AC-40, AC-41, AC-42, +AC-43, AC-44, AC-164, AC-166, AC-167, AC-170, AC-157 """ from __future__ import annotations @@ -30,6 +30,9 @@ GrantSourceService, ) from bisheng.permission.domain.services.projection_service import ProjectionOutcome +from bisheng.permission.domain.services.visibility_projection_service import ( + VisibilityProjectionCompiler, +) class FakeProjection: @@ -64,14 +67,15 @@ async def prepare( self, context, grants, + visibility, *, idempotency_key: str, operation_id: int, ) -> None: - self.prepared.append((context, grants, idempotency_key, operation_id)) + self.prepared.append((context, grants, visibility, idempotency_key, operation_id)) - async def finalize(self, context, grants, outcome) -> None: - self.finalized.append((context, grants, outcome)) + async def finalize(self, context, grants, visibility, outcome) -> None: + self.finalized.append((context, grants, visibility, outcome)) class FakeEvents: @@ -154,6 +158,7 @@ def _context( capabilities=capabilities, models=models, grants=grants or tuple(_grant(model) for model in models), + existing_visible_sources=(), ) @@ -412,3 +417,145 @@ def test_inactive_target_model_is_not_grantable() -> None: context = replace(context, models=models) with pytest.raises(PermissionModelStateConflictError): service.require_grantable_model(context, "viewer") + + +@pytest.mark.asyncio +async def test_add_commits_action_and_single_slot_visible_in_one_operation() -> None: + service, source_service, projection, state, _ = _service() + capability = GrantCapability( + model=_model("owner", 4, "manage_permission", allow_same_level=True), + source_key="owner-source", + ) + context = _context(capabilities=(capability,)) + source = source_service.canonicalize_source( + source_id=40, + subject_type="user", + subject_id="400", + source_type="DIRECT", + ) + + await service.mutate( + context, + changes=(CanonicalGrantChange(operation="ADD", model_key="viewer", source=source),), + expected_resource_version=4, + expected_catalog_release_id=12, + idempotency_key="add-visible", + ) + + plan = projection.plans[0] + assert {row.relation for row in plan.deltas} >= { + "ordinary_assignee", + "visible", + } + assert all("slot" not in row.relation and not row.relation.endswith(("_a", "_b")) for row in plan.deltas) + prepared_visibility = state.prepared[0][2] + assert [row.model_key for row in prepared_visibility.active_sources] == ["viewer"] + + +@pytest.mark.asyncio +async def test_inactive_source_model_allows_precise_remove_and_visible_revoke() -> None: + service, source_service, projection, state, _ = _service() + inactive_viewer = _model("viewer", 1, "download", active=False) + source = source_service.canonicalize_source( + source_id=41, + subject_type="user", + subject_id="401", + source_type="DIRECT", + ) + seeded = source_service.add_source( + _grant(replace(inactive_viewer, active=True)), + source, + ).grant + seeded = replace(seeded, model=inactive_viewer) + base = _context( + capabilities=( + GrantCapability( + model=_model("owner", 4, "manage_permission", allow_same_level=True), + source_key="owner-source", + ), + ) + ) + models = (inactive_viewer, *base.models[1:]) + grants = (seeded, *base.grants[1:]) + existing = VisibilityProjectionCompiler().compile( + tenant_id=7, + grants=grants, + existing_sources=(), + ).active_sources + context = replace( + base, + models=models, + grants=grants, + existing_visible_sources=existing, + ) + + await service.mutate( + context, + changes=( + CanonicalGrantChange( + operation="REMOVE", + assignee_id=41, + expected_assignee_version=1, + ), + ), + expected_resource_version=4, + expected_catalog_release_id=12, + idempotency_key="remove-inactive-visible", + ) + + assert {row.relation for row in projection.plans[0].deltas} >= { + "ordinary_assignee", + "visible", + } + prepared_visibility = state.prepared[0][2] + assert prepared_visibility.active_sources == () + assert len(prepared_visibility.retired_sources) == 1 + + +@pytest.mark.asyncio +async def test_move_freezes_action_and_single_slot_contribution_in_same_operation() -> None: + service, source_service, projection, state, _ = _service() + source = source_service.canonicalize_source( + source_id=42, + subject_type="user", + subject_id="402", + source_type="DIRECT", + ) + base = _context( + capabilities=( + GrantCapability( + model=_model("owner", 4, "manage_permission", allow_same_level=True), + source_key="owner-source", + ), + ) + ) + viewer = source_service.add_source(base.grants[0], source).grant + grants = (viewer, *base.grants[1:]) + existing = VisibilityProjectionCompiler().compile( + tenant_id=7, + grants=grants, + existing_sources=(), + ).active_sources + context = replace(base, grants=grants, existing_visible_sources=existing) + + await service.mutate( + context, + changes=( + CanonicalGrantChange( + operation="MOVE", + assignee_id=42, + expected_assignee_version=1, + target_model_key="editor", + ), + ), + expected_resource_version=4, + expected_catalog_release_id=12, + idempotency_key="move-visible", + ) + + assert projection.plans[0].operation_type == "GRANT_MUTATION" + assert "ordinary_assignee" in {row.relation for row in projection.plans[0].deltas} + visibility = state.prepared[0][2] + assert [row.model_key for row in visibility.active_sources] == ["editor"] + assert [row.model_key for row in visibility.retired_sources] == ["viewer"] + assert visibility.deltas == () diff --git a/src/backend/test/permission/test_f048_grant_sources.py b/src/backend/test/permission/test_f048_grant_sources.py index 9f27e12c87..dbec815da7 100644 --- a/src/backend/test/permission/test_f048_grant_sources.py +++ b/src/backend/test/permission/test_f048_grant_sources.py @@ -1,6 +1,7 @@ """F048 Grant source identity and reference-count contracts. -覆盖 AC: AC-19, AC-20, AC-21, AC-22, AC-23, AC-24, AC-25, AC-26, AC-27 +覆盖 AC: AC-15, AC-19, AC-20, AC-21, AC-22, AC-23, AC-24, AC-25, +AC-26, AC-27, AC-164, AC-166, AC-167 """ from __future__ import annotations @@ -181,7 +182,40 @@ def test_different_models_coexist_and_effective_actions_are_a_union() -> None: assert service.effective_action_union( (editor, inactive), projected_subjects=frozenset({"user:100", "department:17#member"}), - ) == ("edit",) + ) == ("download", "edit") + + +def test_inactive_source_model_keeps_existing_binding_and_allows_precise_revoke() -> None: + service = GrantSourceService() + direct = service.canonicalize_source( + source_id=1, + subject_type="user", + subject_id="100", + source_type="DIRECT", + ) + membership = service.canonicalize_source( + source_id=2, + subject_type="user", + subject_id="100", + source_type="SPACE_MEMBERSHIP", + source_ref="573", + ) + grant = service.add_source( + _grant("g-custom", "custom", ("download",)), + direct, + ).grant + grant = service.add_source(grant, membership).grant + inactive = replace(grant, model=replace(grant.model, active=False)) + + assert service.effective_action_union( + (inactive,), + projected_subjects=frozenset({"user:100"}), + ) == ("download",) + first_revoke = service.remove_source(inactive, source_id=1) + assert first_revoke.deltas == () + last_revoke = service.remove_source(first_revoke.grant, source_id=2) + assert last_revoke.grant.active is False + assert [row.action for row in last_revoke.deltas] == ["DELETE", "DELETE", "DELETE"] def test_remove_one_source_does_not_remove_other_model_or_source() -> None: diff --git a/src/backend/test/permission/test_f048_list_action_cost.py b/src/backend/test/permission/test_f048_list_action_cost.py index 5da8e90fe7..1653a40a2d 100644 --- a/src/backend/test/permission/test_f048_list_action_cost.py +++ b/src/backend/test/permission/test_f048_list_action_cost.py @@ -120,3 +120,27 @@ async def test_an_ordinary_user_still_goes_through_resolution(monkeypatch, count ) assert counted.resolutions == 3 + + +@pytest.mark.parametrize( + "actor", + [ + PermissionActor(user_id=1, current_tenant_id=1, super_admin=True), + PermissionActor(user_id=2, current_tenant_id=7, tenant_admin_tenant_ids=frozenset({7})), + ], +) +async def test_visible_batch_never_expands_admin_identity( + monkeypatch, + counted, + actor: PermissionActor, +) -> None: + _actor_resolver(monkeypatch, actor) + + granted = await business_authorization.batch_check_business_visible( + object(), + resource_type="knowledge_file", + resource_ids=["1", "2", "3"], + ) + + assert counted.resolutions == 3 + assert granted == {"1": True, "2": True, "3": True} diff --git a/src/backend/test/permission/test_f048_migration_cli.py b/src/backend/test/permission/test_f048_migration_cli.py index f0c8956e7d..af0960381c 100644 --- a/src/backend/test/permission/test_f048_migration_cli.py +++ b/src/backend/test/permission/test_f048_migration_cli.py @@ -3,10 +3,20 @@ from __future__ import annotations from dataclasses import dataclass +from pathlib import Path import pytest +from bisheng.common.errcode.permission import PermissionMigrationBlockedError +from bisheng.core.openfga.authorization_model_f048 import ( + authorization_model_checksum, + get_authorization_model_f048, +) +from bisheng.core.openfga.discovery import OpenFGARuntimePin from scripts import migrate_f048_permission_data as cli +from scripts.f048_migration_runtime import _require_predecessor_source + +BACKEND_ROOT = Path(__file__).resolve().parents[2] def test_migrate_requires_apply_before_runtime_initialization(): @@ -24,6 +34,47 @@ def test_unsupported_preview_or_rollback_commands_are_rejected(command): assert exc_info.value.code == 2 +@pytest.mark.parametrize( + "arguments", + [ + ["migrate", "--apply", "--store-id", "replacement"], + ["migrate", "--apply", "--model-id", "intermediate-model"], + ["migrate", "--apply", "--visible-slot", "b"], + ["migrate", "--apply", "--dual-model-mode"], + ["verify", "--run-id", "7", "--legacy-model-id", "legacy"], + ], +) +def test_store_model_and_ab_override_parameters_are_rejected(arguments): + with pytest.raises(SystemExit) as exc_info: + cli.parse_args(arguments) + + assert exc_info.value.code == 2 + + +def test_new_migration_is_rejected_when_store_already_uses_final_f048_model(): + pin = OpenFGARuntimePin( + store_id="durable-store", + model_id="final-model", + model_checksum=authorization_model_checksum(get_authorization_model_f048()), + ) + + with pytest.raises(PermissionMigrationBlockedError, match="F048_MIGRATION_ALREADY_COMPLETED"): + _require_predecessor_source(pin=pin, run_id=None) + + _require_predecessor_source(pin=pin, run_id=7) + + +def test_formal_migration_is_not_called_from_api_or_celery_startup(): + entrypoint = "migrate_f048_permission_data" + startup_files = ( + BACKEND_ROOT / "bisheng/main.py", + BACKEND_ROOT / "bisheng/run_celery.py", + BACKEND_ROOT / "bisheng/worker/main.py", + ) + + assert all(entrypoint not in path.read_text(encoding="utf-8") for path in startup_files) + + @dataclass class FakeRuntime: coordinator: object @@ -144,6 +195,7 @@ async def close_context(): async def test_verify_only_reads_existing_formal_run_and_always_closes(): events = [] + runtime_run_ids = [] coordinator = FakeCoordinator() verifier = FakeVerifier() runtime = FakeRuntime( @@ -161,7 +213,7 @@ async def close_context(): args = cli.parse_args(["verify", "--run-id", "7"]) exit_code = await cli.execute( args, - runtime_factory=lambda **_: runtime, + runtime_factory=lambda **kwargs: (runtime_run_ids.append(kwargs["run_id"]) or runtime), initialize_context=initialize_context, close_context=close_context, live_settings="live-settings", @@ -170,6 +222,7 @@ async def close_context(): assert exit_code == cli.EXIT_OK assert verifier.calls == [{"run_id": 7}] assert coordinator.calls == [] + assert runtime_run_ids == [7] assert events == ["initialize", "close"] assert runtime.closed is True @@ -205,3 +258,10 @@ async def close_context(): ) assert coordinator.calls[0]["run_id"] == 9 + + +def test_runtime_resume_pins_durable_store_and_source_model(): + source = (BACKEND_ROOT / "scripts/f048_migration_runtime.py").read_text(encoding="utf-8") + + assert "required_store_id=run.store_id if run else None" in source + assert "required_model_id=run.source_model_id if run else None" in source diff --git a/src/backend/test/permission/test_f048_migration_coordinator.py b/src/backend/test/permission/test_f048_migration_coordinator.py index c7e238c55a..a4400ae7a1 100644 --- a/src/backend/test/permission/test_f048_migration_coordinator.py +++ b/src/backend/test/permission/test_f048_migration_coordinator.py @@ -334,6 +334,81 @@ async def test_formal_migration_uses_same_store_batches_and_delete_after_verify( assert store.items +async def test_formal_migration_compiles_one_single_slot_visible_aggregate() -> None: + writer = FakeTargetWriter() + coordinator = F048MigrationCoordinator( + source_provider=FakeSourceProvider(_snapshot()), + run_store=FakeRunStore(), + model_publisher=FakeModelPublisher(), + target_writer=writer, + ) + + await coordinator.migrate( + expected_store_id="store-live", + lock_token="operator-1", + ) + + visible = [ + row + for row in writer.written_tuples + if row == { + "user": "user:11", + "relation": "visible", + "object": "workflow:wf-1", + } + ] + assert len(visible) == 1 + assert all( + "slot" not in row["relation"].casefold() + and "switch" not in row["relation"].casefold() + and row["relation"] not in {"visiblea", "visibleb"} + for row in writer.written_tuples + ) + + +async def test_inactive_custom_binding_keeps_existing_visible_contribution() -> None: + base = _snapshot() + snapshot = replace( + base, + config_sources=( + LegacyConfigSource( + key="permission_relation_models_v1", + row_version="2", + raw_value=( + '[{"id":"custom-inactive","name":"legacy",' + '"permissions":["edit"],"active":false}]' + ), + ), + LegacyConfigSource( + key="permission_relation_model_bindings_v1", + row_version="2", + raw_value=( + '[{"binding_key":"binding-inactive","tenant_id":7,' + '"resource_type":"workflow","resource_id":"wf-1",' + '"relation":"editor","model_id":"custom-inactive",' + '"subject_type":"user","subject_id":"22"}]' + ), + ), + ), + tuples=(*base.tuples, LegacyTupleSource(tenant_id=7, user="user:22", relation="editor", object="workflow:wf-1")), + ) + writer = FakeTargetWriter() + coordinator = F048MigrationCoordinator( + source_provider=FakeSourceProvider(snapshot), + run_store=FakeRunStore(frozen_snapshot=snapshot), + model_publisher=FakeModelPublisher(), + target_writer=writer, + ) + + await coordinator.migrate(expected_store_id="store-live", lock_token="operator-1") + + assert { + "user": "user:22", + "relation": "visible", + "object": "workflow:wf-1", + } in writer.written_tuples + + async def test_formal_migration_writes_department_child_mirror() -> None: base = _snapshot() snapshot = replace( diff --git a/src/backend/test/permission/test_f048_migration_runtime.py b/src/backend/test/permission/test_f048_migration_runtime.py index 10a6288523..734201096a 100644 --- a/src/backend/test/permission/test_f048_migration_runtime.py +++ b/src/backend/test/permission/test_f048_migration_runtime.py @@ -30,6 +30,7 @@ PermissionMigrationRun, PermissionModel, PermissionModelAction, + PermissionVisibleSourceProjection, ResourcePermissionMode, ) from bisheng.permission.domain.repositories.migration_repository import ( @@ -598,8 +599,11 @@ async def discover(*args, **kwargs): "permission_model", "permission_model_action", "permission_projection_operation", + "permission_migration_run", + "permission_migration_item", "permission_grant", "permission_grant_assignee", + "permission_visible_source_projection", "resource_permission_mode", ) @@ -726,6 +730,23 @@ async def session_factory() -> AsyncIterator[AsyncSession]: mode="CUSTOM", projection_state="CURRENT", ), + PermissionVisibleSourceProjection( + id=id_offset + 10, + tenant_id=7, + resource_type="workflow", + resource_id="wf-1", + visibility_class="ordinary", + projected_subject="user:11", + source_kind="GRANT_ASSIGNEE", + source_owner_key=f"grant_assignee:{id_offset + 8}", + source_locator="migration:DIRECT:legacy", + source_fingerprint="d" * 64, + contribution_fingerprint="e" * 64, + model_key="viewer", + source_version=1, + tuple_fingerprint="f" * 64, + state="ACTIVE", + ), ) ) diff --git a/src/backend/test/permission/test_f048_migration_verifier.py b/src/backend/test/permission/test_f048_migration_verifier.py index cf56b4710a..119a561cb6 100644 --- a/src/backend/test/permission/test_f048_migration_verifier.py +++ b/src/backend/test/permission/test_f048_migration_verifier.py @@ -65,8 +65,6 @@ def _evidence(**overrides) -> MigrationVerificationEvidence: store_id="store-live", model_id="new-model", catalog_release_id=9, - dual_model_mode=False, - legacy_model_id=None, ), InstancePinEvidence( role="worker", @@ -74,10 +72,12 @@ def _evidence(**overrides) -> MigrationVerificationEvidence: store_id="store-live", model_id="new-model", catalog_release_id=9, - dual_model_mode=False, - legacy_model_id=None, ), ), + "visible_source_checksum_matches": True, + "visible_aggregate_checksum_matches": True, + "unattributed_visible_count": 0, + "visible_stream_complete": True, } values.update(overrides) return MigrationVerificationEvidence(**values) @@ -150,6 +150,10 @@ async def test_verifier_allows_retained_legacy_config_audit_rows(): ("unapproved_manual_count", 1, "UNAPPROVED_MANUAL_ITEMS"), ("cross_tenant_count", 1, "CROSS_TENANT_FACTS"), ("invalid_owner_count", 1, "INVALID_OWNER_FACTS"), + ("unattributed_visible_count", 1, "UNATTRIBUTED_VISIBLE_TUPLES"), + ("visible_source_checksum_matches", False, "VISIBLE_SOURCE_CHECKSUM_MISMATCH"), + ("visible_aggregate_checksum_matches", False, "VISIBLE_AGGREGATE_CHECKSUM_MISMATCH"), + ("visible_stream_complete", False, "VISIBLE_STREAM_INCOMPLETE"), ("actual_target_checksum", "x" * 64, "TARGET_CHECKSUM_MISMATCH"), ], ) @@ -176,8 +180,6 @@ async def test_verifier_blocks_semantic_failure_or_mixed_runtime_pins(): store_id="store-live", model_id="legacy", catalog_release_id=9, - dual_model_mode=True, - legacy_model_id="legacy", ), ) verifier = F048MigrationVerifier( diff --git a/src/backend/test/permission/test_f048_model_policy.py b/src/backend/test/permission/test_f048_model_policy.py index f35e1458ef..ab66ad11c8 100644 --- a/src/backend/test/permission/test_f048_model_policy.py +++ b/src/backend/test/permission/test_f048_model_policy.py @@ -1,7 +1,7 @@ """Pure standard/custom permission-model contracts. 覆盖 AC: AC-07, AC-08, AC-09, AC-10, AC-11, AC-12, AC-13, AC-14, -AC-15, AC-16, AC-17, AC-18, AC-39, AC-156 +AC-15, AC-16, AC-17, AC-18, AC-27, AC-39, AC-156, AC-164, AC-165, AC-167 """ from __future__ import annotations @@ -19,9 +19,11 @@ from bisheng.permission.domain.services.model_policy import ( CustomModelSelection, ModelPreset, + ModelReferenceSummary, calculate_model_impact, derive_permission_models, effective_model_action_codes, + ensure_model_assignable, ensure_model_deletable, initialize_from_preset, validate_standard_model_update, @@ -200,6 +202,29 @@ def test_inactive_model_keeps_selection_but_never_grants_actions() -> None: ) +def test_inactive_model_is_not_assignable_but_keeps_existing_actions() -> None: + model = _by_key( + derive_permission_models( + _action_release(), + custom_models=( + CustomModelSelection( + model_key="inactive-editor", + name="已停用编辑者", + action_codes=("edit", "manage_permission"), + active=False, + ), + ), + ) + )["inactive-editor"] + + with pytest.raises(ValueError, match="inactive"): + ensure_model_assignable(model) + assert set(effective_model_action_codes(model, _action_release(), "workflow")) == { + "edit", + "manage_permission", + } + + def test_shared_model_change_reports_all_grant_references_once() -> None: before = derive_permission_models( _action_release(), @@ -230,7 +255,23 @@ def test_shared_model_change_reports_all_grant_references_once() -> None: assert impact.affected_grant_refs == ("grant-1", "grant-2", "grant-3") -def test_model_delete_requires_inactive_unreferenced_custom() -> None: +@pytest.mark.parametrize( + ("field", "value"), + ( + ("active_grant_count", 1), + ("pending_grant_count", 1), + ("failed_grant_count", 1), + ("active_source_count", 1), + ("pending_source_count", 1), + ("failed_source_count", 1), + ("live_tuple_count", 1), + ("residual_checksum", "a" * 64), + ), +) +def test_model_delete_requires_zero_references_and_residuals( + field: str, + value: int | str, +) -> None: model = _by_key( derive_permission_models( _action_release(), @@ -244,14 +285,22 @@ def test_model_delete_requires_inactive_unreferenced_custom() -> None: ), ) )["retired"] - ensure_model_deletable(model, reference_count=0) + ensure_model_deletable(model, references=ModelReferenceSummary()) + ensure_model_deletable( + replace(model, active=True), + references=ModelReferenceSummary(), + ) with pytest.raises(ValueError, match="referenced"): - ensure_model_deletable(model, reference_count=1) - with pytest.raises(ValueError, match="inactive"): - ensure_model_deletable(replace(model, active=True), reference_count=0) + ensure_model_deletable( + model, + references=replace(ModelReferenceSummary(), **{field: value}), + ) standard = _by_key(derive_permission_models(_action_release()))["viewer"] with pytest.raises(ValueError, match="standard"): - ensure_model_deletable(standard, reference_count=0) + ensure_model_deletable( + standard, + references=ModelReferenceSummary(), + ) def test_preset_only_initializes_an_independent_selection() -> None: diff --git a/src/backend/test/permission/test_f048_openfga_integration.py b/src/backend/test/permission/test_f048_openfga_integration.py index 1df0d2e7b9..82b5654ab6 100644 --- a/src/backend/test/permission/test_f048_openfga_integration.py +++ b/src/backend/test/permission/test_f048_openfga_integration.py @@ -83,7 +83,7 @@ async def _create_store(http: httpx.AsyncClient) -> str: return str(response.json()["id"]) -@pytest.fixture(scope="module") +@pytest.fixture async def openfga_runtime(): metadata = _runtime_metadata() api_url = os.environ.get("F048_OPENFGA_API_URL", "http://127.0.0.1:8080") @@ -227,6 +227,12 @@ async def test_same_store_single_new_model_and_model_checksum( } assert authorization_model_checksum(live_model) == authorization_model_checksum(build_authorization_model_f048()) + serialized = json.dumps(live_model, sort_keys=True) + assert '"visible"' in serialized + assert "visible_a" not in serialized + assert "visible_b" not in serialized + assert "visible_switch" not in serialized + async def test_real_check_batch_list_and_higher_consistency_semantics( openfga_runtime: OpenFGARuntime, @@ -242,6 +248,35 @@ async def test_real_check_batch_list_and_higher_consistency_semantics( ) tuples.extend( ( + { + "user": "user:1", + "relation": "visible", + "object": "workflow:direct", + }, + { + "user": "department:visible-engineering#member", + "relation": "visible", + "object": "workflow:department", + }, + { + "user": "user_group:reviewers#member", + "relation": "visible", + "object": "workflow:group", + }, + { + "user": "user:*", + "relation": "visible", + "object": "workflow:system", + }, + ) + ) + tuples.extend( + ( + { + "user": "user:1", + "relation": "member", + "object": "department:visible-engineering", + }, { "user": "user:2", "relation": "member", @@ -299,6 +334,16 @@ async def test_real_check_batch_list_and_higher_consistency_semantics( ) ) await client.write_tuples(writes=tuples) + await client.write_tuples( + writes=[ + { + "user": "department:visible-engineering#member", + "relation": "visible", + "object": "workflow:department", + } + ], + ignore_duplicate_writes=True, + ) assert await client.check( "user:1", @@ -346,6 +391,52 @@ async def test_real_check_batch_list_and_higher_consistency_semantics( consistency="HIGHER_CONSISTENCY", ) == ["workflow:direct"] + expected_visible = { + "workflow:department", + "workflow:direct", + "workflow:system", + } + assert await client.check( + "user:1", + "visible", + "workflow:direct", + consistency="HIGHER_CONSISTENCY", + ) + assert await client.batch_check( + [ + { + "user": "department:visible-engineering#member", + "relation": "visible", + "object": "workflow:department", + }, + { + "user": "user_group:reviewers#member", + "relation": "visible", + "object": "workflow:group", + }, + ], + consistency="HIGHER_CONSISTENCY", + ) == [True, True] + assert await client.batch_check( + [ + { + "user": "user:1", + "relation": "visible", + "object": object_key, + } + for object_key in sorted(expected_visible | {"workflow:group"}) + ], + consistency="HIGHER_CONSISTENCY", + ) == [True, True, False, True] + assert set( + await client.stream_list_objects( + "user:1", + "visible", + "workflow", + consistency="HIGHER_CONSISTENCY", + ) + ) == expected_visible + async def test_store_scoped_legacy_delete_with_new_model( openfga_runtime: OpenFGARuntime, @@ -356,12 +447,16 @@ async def test_store_scoped_legacy_delete_with_new_model( "object": "workflow:legacy", } await openfga_runtime.old_client.write_tuples(writes=[legacy]) - assert await openfga_runtime.old_client.read_tuples( + stored = await openfga_runtime.old_client.read_tuples( user=legacy["user"], relation=legacy["relation"], object=legacy["object"], consistency="HIGHER_CONSISTENCY", - ) == [legacy] + ) + assert [ + {key: row[key] for key in ("user", "relation", "object")} + for row in stored + ] == [legacy] await openfga_runtime.client.delete_tuples_store_scoped([legacy]) assert ( await openfga_runtime.old_client.read_tuples( diff --git a/src/backend/test/permission/test_f048_owner_projection.py b/src/backend/test/permission/test_f048_owner_projection.py index 73eea3ff2d..e24a8065f4 100644 --- a/src/backend/test/permission/test_f048_owner_projection.py +++ b/src/backend/test/permission/test_f048_owner_projection.py @@ -446,7 +446,7 @@ async def test_system_owned_requires_code_allowlist_and_business_predicate() -> result = await service.project_created(valid) assert result.grant is None - assert any(delta.relation == "system_visible_marker" for delta in projection.plans[0].deltas) + assert any(delta.relation == "visible" for delta in projection.plans[0].deltas) assert any(delta.relation == "system_use_marker" for delta in projection.plans[0].deltas) for invalid in ( diff --git a/src/backend/test/permission/test_f048_performance_contract.py b/src/backend/test/permission/test_f048_performance_contract.py index 3c3ea5b58e..d4e7718763 100644 --- a/src/backend/test/permission/test_f048_performance_contract.py +++ b/src/backend/test/permission/test_f048_performance_contract.py @@ -1,6 +1,6 @@ -"""BENCH-01 fixed-fixture, workload, threshold, and safety contracts. +"""BENCH-01 single-slot visibility performance contracts. -覆盖 AC: AC-28, AC-29, AC-30, AC-31, AC-32, AC-33, AC-34, AC-35, AC-69 +覆盖 AC: AC-160, AC-161, AC-162, AC-163, AC-168, AC-175, AC-176 """ from __future__ import annotations @@ -10,23 +10,28 @@ import pytest +from bisheng.core.openfga.authorization_model_f048 import build_authorization_model_f048 from scripts.benchmark_f048_permission_paths import ( - BATCH_CHECK_SIZES, CONTRACT_VERSION, DEFAULT_FIXTURE, + RESOURCE_SCALES, + SOURCE_KINDS, + VISIBLE_RESULT_SIZES, BenchmarkContractError, RequestSample, build_dataset_tuples, - checksum, contract_checksum, dataset_checksum, evaluate, load_contract, + model_has_ab_slots, nearest_rank_percentile, object_keys, - read_openfga_metrics, run_workloads, + source_checksum, + strategy_metrics, validate_environment, + visible_checksum, ) @@ -35,37 +40,54 @@ def contract() -> dict: return load_contract(DEFAULT_FIXTURE) -def test_fixed_fixture_checksums_and_required_distribution(contract: dict) -> None: +def test_fixed_fixture_covers_resource_visibility_and_source_matrix( + contract: dict, +) -> None: assert contract["contract_version"] == CONTRACT_VERSION - assert contract_checksum(contract) == ("12654227efbcdb9ebc2effdd1da5c668723cdcd96c4333eb621f0229b190dc14") - assert dataset_checksum(contract) == ("ad302adc12b1807080f85c238b36c7e93a75ef1c39165e6250a3138710f6e54b") - assert len(build_dataset_tuples(contract)) == 5823 - scenarios = {item["name"]: item for item in contract["dataset"]["profile"]["scenarios"]} + assert contract_checksum(contract) == ( + "80a80489bf4f3ae01f5b1f3519cd275b10a92f16dd58df5fa1378de48c608912" + ) + assert dataset_checksum(contract) == ( + "d083c56febda9eb055e9ae5356f800ca837a1e534628fdeda623dc4b8063ab22" + ) + assert source_checksum(contract) == ( + "9e2e7699dbe65851de0b04651ee8bdb009e2a85ac16e8c12a5b9e5092c220df0" + ) + assert visible_checksum(contract) == ( + "f185c204418fbd98d41c15b0bac7759060c2df73da8c8c4fcf381f0a02c1994e" + ) + assert len(build_dataset_tuples(contract)) == 41_666 + + scenarios = contract["dataset"]["scenarios"] assert { - "direct", - "department", - "group", - "inherit", - "multi_grant", - "result_10", - "result_100", - "result_1000", - } == set(scenarios) - assert [scenarios[f"result_{size}"]["result_count"] for size in (10, 100, 1000)] == [ - 10, - 100, - 1000, - ] - assert set(map(int, contract["baseline"]["batch_check_ms"])) == set(BATCH_CHECK_SIZES) + (int(item["resource_count"]), int(item["visible_count"])) + for item in scenarios + } == { + (resource_count, visible_count) + for resource_count in RESOURCE_SCALES + for visible_count in VISIBLE_RESULT_SIZES + } + assert {item["source_kind"] for item in scenarios} == SOURCE_KINDS assert contract["dataset"]["production_derived"] is False + assert contract["dataset"]["representative_distribution"] is True + + +def test_target_model_and_dataset_have_no_ab_visibility_slots( + contract: dict, +) -> None: + assert model_has_ab_slots(build_authorization_model_f048()) is False + assert not any( + relation in {"visible_a", "visible_b", "slot_a", "slot_b"} + for relation in (item["relation"] for item in build_dataset_tuples(contract)) + ) -def test_fixture_checksum_drift_fails_closed( +def test_fixture_checksum_and_distribution_drift_fail_closed( contract: dict, tmp_path: Path, ) -> None: tampered = json.loads(json.dumps(contract)) - tampered["dataset"]["profile"]["scenarios"][0]["actor_id"] = 9999 + tampered["dataset"]["scenarios"][0]["actor_id"] = 9999 path = tmp_path / "tampered.json" path.write_text(json.dumps(tampered), encoding="utf-8") with pytest.raises(BenchmarkContractError, match="fixture checksum mismatch"): @@ -97,128 +119,46 @@ def test_nearest_rank_percentiles_are_deterministic() -> None: nearest_rank_percentile([], 95) -def _passing_samples(contract: dict) -> list[RequestSample]: - samples: list[RequestSample] = [] - scenarios = contract["dataset"]["profile"]["scenarios"] - for index, scenario in enumerate(scenarios): - samples.append( - RequestSample( - operation="check", - scenario=scenario["name"], - elapsed_ms=1, - request_ids=(f"check-{index}",), - ) - ) - objects = object_keys(scenario) - samples.append( - RequestSample( - operation="list_objects", - scenario=scenario["name"], - elapsed_ms=2, - request_ids=(f"list-{index}",), - result_count=len(objects), - result_checksum=checksum(sorted(objects)), - ) - ) - for size in BATCH_CHECK_SIZES: - samples.append( - RequestSample( - operation="batch_check", - scenario=str(size), - elapsed_ms=3, - request_ids=(f"batch-{size}",), - result_count=size, - ) - ) - cursor_contract = contract["dataset"]["profile"]["business_cursor"] - scenario = next(item for item in scenarios if item["name"] == cursor_contract["scenario"]) - visible = object_keys(scenario)[: cursor_contract["candidate_count"]] - samples.append( - RequestSample( - operation="business_cursor", - scenario=scenario["name"], - elapsed_ms=4, - request_ids=("cursor-1", "cursor-2"), - result_count=len(visible), - result_checksum=checksum(visible), - ) +def test_strategy_metrics_record_n_db_v_p_and_scan_amplification() -> None: + sparse = strategy_metrics( + n_db=100_000, + visible_total=100, + page_size=20, + scanned_count=20, ) - return samples - - -def _metrics(samples: list[RequestSample]) -> dict[str, dict[str, int]]: - return { - request_id: {"dispatch_count": 2, "datastore_query_count": 1} - for sample in samples - for request_id in sample.request_ids + assert sparse == { + "n_db": 100_000, + "visible_total": 100, + "selectivity": 0.001, + "scanned_count": 20, + "scan_amplification": 1.0, } - - -def test_threshold_report_passes_performance_but_not_release_fixture( - contract: dict, -) -> None: - samples = _passing_samples(contract) - report = evaluate(contract, samples, _metrics(samples)) - assert report["check"]["passed"] is True - assert all(item["passed"] for item in report["batch_check"].values()) - assert all(item["passed"] for item in report["list_objects"].values()) - assert report["business_cursor"]["passed"] is True - assert report["openfga_observability"]["passed"] is True - assert report["performance_passed"] is True - assert report["production_derived"] is False - assert report["release_ready"] is False - - -def test_missing_openfga_metrics_and_list_truncation_fail_gate( - contract: dict, -) -> None: - samples = _passing_samples(contract) - result_1000 = next(item for item in samples if item.operation == "list_objects" and item.scenario == "result_1000") - result_1000.result_count = 999 - result_1000.result_checksum = checksum(["workflow:truncated"]) - report = evaluate(contract, samples, {}) - assert report["list_objects"]["result_1000"]["passed"] is False - assert report["openfga_observability"]["passed"] is False - assert report["performance_passed"] is False - - -def test_openfga_json_log_correlates_dispatch_and_datastore_reads( - tmp_path: Path, -) -> None: - path = tmp_path / "openfga.jsonl" - path.write_text( - "\n".join( - ( - json.dumps( - { - "fields": { - "request_id": "wanted", - "dispatch_count": 17, - "datastore_query_count": 3, - } - } - ), - json.dumps( - { - "request_id": "unrelated", - "dispatch_count": 99, - "datastore_query_count": 99, - } - ), - "not-json", - ) - ), - encoding="utf-8", + candidate = strategy_metrics( + n_db=100_000, + visible_total=5_000, + page_size=50, + scanned_count=100, ) - assert read_openfga_metrics(path, {"wanted"}) == {"wanted": {"dispatch_count": 17, "datastore_query_count": 3}} + assert candidate["selectivity"] == 0.05 + assert candidate["scan_amplification"] == 2.0 class FakeBenchmarkClient: - def __init__(self, contract: dict) -> None: + def __init__(self, contract: dict, *, truncate: bool = False) -> None: self.contract = contract + self.truncate = truncate self.batch_sizes: list[int] = [] + self.stream_relations: list[str] = [] self._sequence = 0 - self._by_actor = {f"user:{item['actor_id']}": item for item in contract["dataset"]["profile"]["scenarios"]} + self._by_actor = { + f"user:{item['actor_id']}": item + for item in contract["dataset"]["scenarios"] + } + self._visible = { + object_key + for item in contract["dataset"]["scenarios"] + for object_key in object_keys(item) + } def _request_id(self) -> str: self._sequence += 1 @@ -231,39 +171,129 @@ async def verify_model(self, expected_checksum: str) -> None: assert expected_checksum == self.contract["authorization_model_checksum"] async def check(self, query: dict) -> tuple[bool, str]: - assert query["relation"].startswith("can_") - return True, self._request_id() + assert query["relation"] == "visible" + return query["object"] in self._visible, self._request_id() async def batch_check(self, queries: list[dict]) -> tuple[list[bool], str]: self.batch_sizes.append(len(queries)) - assert len(queries) <= 100 - return [True] * len(queries), self._request_id() + assert len(queries) <= 50 + assert all(query["relation"] == "visible" for query in queries) + return [query["object"] in self._visible for query in queries], self._request_id() - async def list_objects( + async def stream_list_objects( self, *, user: str, relation: str, resource_type: str, - ) -> tuple[list[str], str]: + ) -> tuple[list[str], str, bool]: scenario = self._by_actor[user] - assert relation == f"can_{scenario['action']}" + self.stream_relations.append(relation) + assert relation == "visible" assert resource_type == scenario["resource_type"] - return object_keys(scenario), self._request_id() + objects = object_keys(scenario) + if self.truncate: + objects = objects[:-1] + return objects, self._request_id(), True + + +def _with_one_iteration(contract: dict) -> dict: + value = json.loads(json.dumps(contract)) + value["run"]["iterations"] = 1 + value["run"]["warmup"] = 0 + return value + + +def _metrics(samples: list[RequestSample]) -> dict[str, dict[str, int]]: + return { + request_id: {"dispatch_count": 2, "datastore_query_count": 1} + for sample in samples + for request_id in sample.request_ids + } -async def test_workload_executes_check_batch_list_and_bounded_business_cursor( +async def test_workload_uses_single_slot_stream_batch_and_business_paths( contract: dict, ) -> None: - client = FakeBenchmarkClient(contract) - samples = await run_workloads(client, contract, iterations=1, warmup=0) + runtime_contract = _with_one_iteration(contract) + client = FakeBenchmarkClient(runtime_contract) + samples = await run_workloads( + client, + runtime_contract, + iterations=1, + warmup=0, + ) assert len([item for item in samples if item.operation == "check"]) == 8 - assert len([item for item in samples if item.operation == "list_objects"]) == 8 - assert [item.scenario for item in samples if item.operation == "batch_check"] == [ - "20", - "50", - "100", - ] - # Three explicit BatchCheck samples plus five pages of 20 candidates. - assert client.batch_sizes == [20, 50, 100, 20, 20, 20, 20, 20] + assert len( + [item for item in samples if item.operation == "stream_list_objects"] + ) == 8 + assert [ + item.scenario for item in samples if item.operation == "batch_check" + ] == ["20", "50", "100"] + assert {item.scenario for item in samples if item.operation == "business_path"} == { + "joined", + "department", + "file", + } + assert set(client.stream_relations) == {"visible"} assert all(item.error is None for item in samples) + assert all( + item.stream_completed is True + for item in samples + if item.operation == "stream_list_objects" + ) + + paths = { + item.scenario: item + for item in samples + if item.operation == "business_path" + } + assert paths["joined"].strategy == "visible_id_first" + assert paths["joined"].n_db == 100_000 + assert paths["joined"].visible_total == 100 + assert paths["department"].strategy == "candidate_first" + assert paths["department"].selectivity == 0.5 + assert paths["department"].candidate_pass_rate == 0.5 + assert paths["file"].strategy == "candidate_first" + assert paths["file"].selectivity == 0.05 + assert paths["file"].candidate_pass_rate == 0.95 + assert paths["file"].scan_amplification is not None + assert {20, 50}.issubset(set(client.batch_sizes)) + assert client.batch_sizes.count(50) >= 2 + assert max(client.batch_sizes) == 50 + + +async def test_complete_stream_and_metrics_are_release_gate( + contract: dict, +) -> None: + runtime_contract = _with_one_iteration(contract) + samples = await run_workloads( + FakeBenchmarkClient(runtime_contract), + runtime_contract, + iterations=1, + warmup=0, + ) + report = evaluate(runtime_contract, samples, _metrics(samples)) + assert report["performance_passed"] is True + assert report["release_ready"] is True + assert report["production_derived"] is False + assert report["representative_distribution"] is True + assert all( + item["stream_completed"] and item["passed"] + for item in report["streamed_list_objects"].values() + ) + assert all(item["passed"] for item in report["business_paths"].values()) + + truncated = await run_workloads( + FakeBenchmarkClient(runtime_contract, truncate=True), + runtime_contract, + iterations=1, + warmup=0, + ) + truncated_report = evaluate(runtime_contract, truncated, _metrics(truncated)) + assert truncated_report["performance_passed"] is False + assert truncated_report["release_ready"] is False + + missing_metrics = evaluate(runtime_contract, samples, {}) + assert missing_metrics["openfga_observability"]["passed"] is False + assert missing_metrics["performance_passed"] is False diff --git a/src/backend/test/permission/test_f048_permission_service.py b/src/backend/test/permission/test_f048_permission_service.py index b50deb0403..fa371cfb8d 100644 --- a/src/backend/test/permission/test_f048_permission_service.py +++ b/src/backend/test/permission/test_f048_permission_service.py @@ -1,7 +1,8 @@ """F048 concrete-action decision facade contracts. 覆盖 AC: AC-28, AC-29, AC-30, AC-31, AC-32, AC-33, AC-34, AC-35, -AC-63, AC-65, AC-69, AC-155 +AC-63, AC-65, AC-69, AC-155, AC-160, AC-161, AC-162, AC-163, +AC-168, AC-169, AC-170, AC-171 """ from __future__ import annotations @@ -10,11 +11,15 @@ from bisheng.common.errcode.permission import ( InvalidCatalogActionError, + PermissionEnumerationIncompleteError, PermissionFGAUnavailableError, PermissionProjectionFailedError, PermissionPublishNotReadyError, ) -from bisheng.permission.domain.schemas import VerifiedPermissionTarget +from bisheng.permission.domain.schemas import ( + VerifiedPermissionTarget, + VisibilityEnumerationStatus, +) from bisheng.permission.domain.services.permission_service import ( F048PermissionService, PermissionActor, @@ -88,13 +93,20 @@ async def check(self, *, user, relation, object, consistency=None): self.checks.append((user, relation, object, consistency)) if self.fail: raise RuntimeError("openfga down") + if relation == "visible" and self.objects: + return object in self.objects return self.allowed async def batch_check(self, checks, consistency=None): self.batches.append((checks, consistency)) if self.fail: raise RuntimeError("openfga down") - return [row["object"].endswith(":allow") for row in checks] + return [ + row["object"] in self.objects + if row["relation"] == "visible" and self.objects + else row["object"].endswith(":allow") + for row in checks + ] async def list_objects( self, @@ -109,6 +121,19 @@ async def list_objects( raise RuntimeError("openfga down") return self.objects + async def stream_list_objects( + self, + *, + user, + relation, + type, + consistency=None, + ): + self.lists.append((user, relation, type, consistency)) + if self.fail: + raise RuntimeError("openfga stream failed") + return tuple(self.objects) + class FakeListPolicy: def __init__(self) -> None: @@ -304,6 +329,108 @@ async def test_visible_batch_uses_one_openfga_batch_without_action_alias() -> No assert all(row["relation"] == "visible" for row in checks) +@pytest.mark.asyncio +async def test_visible_checks_never_expand_super_or_tenant_admin_scope() -> None: + service, _, _, _, fga, _, _ = _service() + fga.allowed = False + + assert not await service.check_visible(_actor(super_admin=True), _target()) + assert not await service.check_visible( + _actor(tenant_admin_ids=frozenset({7})), + _target(), + ) + assert [row[1] for row in fga.checks] == ["visible", "visible"] + + results = await service.batch_check_visible( + _actor(super_admin=True), + (_target("allow"), _target("deny")), + ) + assert results == (True, False) + assert len(fga.batches) == 1 + + +@pytest.mark.asyncio +async def test_complete_visible_enumeration_is_deduplicated_and_consistent() -> None: + service, _, _, marker, fga, _, _ = _service() + marker.higher = True + fga.objects = ["workflow:allow", "workflow:allow", "workflow:second"] + + result = await service.list_visible_objects( + _actor(super_admin=True), + resource_type="workflow", + max_results=5_000, + ) + + assert result.status is VisibilityEnumerationStatus.NORMAL + assert result.object_ids == ("allow", "second") + assert fga.lists[-1] == ( + "user:100", + "visible", + "workflow", + "HIGHER_CONSISTENCY", + ) + candidate_ids = ("allow", "second", "deny") + single = tuple( + [await service.check_visible(_actor(), _target(resource_id)) for resource_id in candidate_ids] + ) + batch = await service.batch_check_visible( + _actor(), + tuple(_target(resource_id) for resource_id in candidate_ids), + ) + assert single == batch == (True, True, False) + assert tuple(resource_id for resource_id, allowed in zip(candidate_ids, batch, strict=True) if allowed) == ( + result.object_ids + ) + + +@pytest.mark.asyncio +async def test_visible_enumeration_capacity_error_and_no_sql_allow_fallback() -> None: + service, _, _, _, fga, _, _ = _service() + fga.objects = [f"workflow:{index}" for index in range(5_000)] + accepted = await service.list_visible_objects( + _actor(), + resource_type="workflow", + max_results=5_000, + ) + assert len(accepted.object_ids) == 5_000 + + fga.objects.append("workflow:5000") + with pytest.raises(PermissionEnumerationIncompleteError): + await service.list_visible_objects( + _actor(), + resource_type="workflow", + max_results=5_000, + ) + + fga.objects = [] + denied = await service.list_visible_objects( + _actor(), + resource_type="workflow", + max_results=5_000, + ) + assert denied.object_ids == () + + +@pytest.mark.asyncio +async def test_visible_enumeration_tenant_fence_and_stream_error_are_explicit() -> None: + service, _, _, _, fga, _, _ = _service() + with pytest.raises(PermissionEnumerationIncompleteError): + await service.list_visible_objects( + _actor(tenant_id=0), + resource_type="workflow", + max_results=5_000, + ) + assert fga.lists == [] + + fga.fail = True + with pytest.raises(PermissionEnumerationIncompleteError): + await service.list_visible_objects( + _actor(), + resource_type="workflow", + max_results=5_000, + ) + + @pytest.mark.asyncio async def test_list_objects_is_allowlisted_bounded_and_never_generic_paging() -> None: service, _, _, _, fga, list_policy, _ = _service() diff --git a/src/backend/test/permission/test_f048_projection_reconcile_cli.py b/src/backend/test/permission/test_f048_projection_reconcile_cli.py index a28f0f54ab..c97c62850a 100644 --- a/src/backend/test/permission/test_f048_projection_reconcile_cli.py +++ b/src/backend/test/permission/test_f048_projection_reconcile_cli.py @@ -12,9 +12,10 @@ class FakeRepository: - def __init__(self, operation, tuples) -> None: + def __init__(self, operation, tuples, visible_sources=()) -> None: self.operation = operation self.tuples = tuples + self.visible_sources = tuple(visible_sources) async def aget_operation(self, operation_id: int): if self.operation is None or self.operation.id != operation_id: @@ -26,6 +27,16 @@ async def aget_operation_tuples(self, operation_id: int): return [] return self.tuples + async def aget_visible_operation_sources(self, operation_id: int): + if self.operation is None or self.operation.id != operation_id: + return [] + return list(self.visible_sources) + + async def aget_visible_operation_checksum(self, operation_id: int): + if self.operation is None or self.operation.id != operation_id or not self.visible_sources: + return None + return "v" * 64 + class FakeProjection: def __init__(self, statuses: dict[int, str]) -> None: @@ -59,6 +70,10 @@ def _tuple(phase: str, action: str): return SimpleNamespace(phase=phase, action=action) +def _visible_source(state: str = "PENDING"): + return SimpleNamespace(state=state) + + def _inspection(operation_id: int, status: str) -> cli.OperationInspection: return cli.OperationInspection( operation_id=operation_id, @@ -73,6 +88,9 @@ def _inspection(operation_id: int, status: str) -> cli.OperationInspection: status=status, tuple_count=4, tuple_summary={"COMMIT:WRITE": 1, "STAGE:WRITE": 3}, + visible_source_count=1, + visible_source_summary={"PENDING" if status != "FINALIZED" else "ACTIVE": 1}, + visible_source_checksum="v" * 64, resource_mode={ "mode": "INHERIT" if status != "FINALIZED" else "CUSTOM", "version": 4 if status != "FINALIZED" else 5, @@ -106,6 +124,7 @@ async def test_inspect_operation_validates_pin_ledger_and_resource_fence(monkeyp _tuple("COMMIT", "DELETE"), _tuple("COMMIT", "WRITE"), ], + visible_sources=(), ), projection=None, ) @@ -135,6 +154,7 @@ async def load_mode(**kwargs): "COMMIT:WRITE": 1, "STAGE:WRITE": 2, } + assert inspected.visible_source_count == 0 assert inspected.resource_mode == { "mode": "INHERIT", "version": 4, @@ -179,6 +199,37 @@ async def test_inspect_operation_blocks_live_pin_mismatch() -> None: ) +async def test_inspect_grant_operation_requires_complete_visible_source_after_state() -> None: + operation = _operation(operation_type="GRANT_MUTATION") + runtime = cli.ProjectionReconcileRuntime( + client=SimpleNamespace(store_id="store-live", model_id="model-live"), + repository=FakeRepository(operation, [_tuple("COMMIT", "WRITE")]), + projection=None, + ) + + with pytest.raises(cli.ProjectionReconcileBlockedError, match="visible source"): + await cli.inspect_operation( + runtime, + operation_id=11, + tenant_id=1, + ) + + runtime = replace( + runtime, + repository=FakeRepository( + operation, + [_tuple("COMMIT", "WRITE")], + visible_sources=(_visible_source("FAILED_CLOSED"),), + ), + ) + with pytest.raises(cli.ProjectionReconcileBlockedError, match="FAILED_CLOSED"): + await cli.inspect_operation( + runtime, + operation_id=11, + tenant_id=1, + ) + + async def test_execute_dry_run_never_builds_apply_runtime_or_reconciles(capsys) -> None: events: list[object] = [] args = cli.parse_args(["--tenant-id", "1", "11"]) diff --git a/src/backend/test/permission/test_f048_projection_service.py b/src/backend/test/permission/test_f048_projection_service.py index dd509ad312..5918324431 100644 --- a/src/backend/test/permission/test_f048_projection_service.py +++ b/src/backend/test/permission/test_f048_projection_service.py @@ -1,6 +1,7 @@ """Durable F048 projection-ledger contracts. -覆盖 AC: AC-16, AC-54, AC-66, AC-67, AC-68, AC-69, AC-70, AC-143 +覆盖 AC: AC-15, AC-16, AC-54, AC-66, AC-67, AC-68, AC-69, AC-70, +AC-143, AC-164, AC-166, AC-167, AC-170 """ from __future__ import annotations @@ -296,6 +297,45 @@ async def test_prepare_stage_recent_commit_verify_and_finalize_order() -> None: assert events.rows[-1][0] == "permission_projection" +@pytest.mark.asyncio +async def test_action_and_single_slot_visible_are_one_atomic_commit() -> None: + plan = _plan( + ProjectionTupleDelta( + phase="COMMIT", + sequence=0, + action="WRITE", + user="user:100", + relation="ordinary_assignee", + object="permission_grant:g-viewer", + ), + ProjectionTupleDelta( + phase="COMMIT", + sequence=1, + action="WRITE", + user="user:100", + relation="visible", + object="workflow:42", + ), + ) + service, repository, marker, scope, fga, finalizer, _ = _service(plan) + + outcome = await service.execute(plan) + + assert outcome.status == ProjectionOperationStatus.FINALIZED + assert repository.operation.status == ProjectionOperationStatus.FINALIZED + assert marker.log == ["recent"] + assert scope.fenced is False + assert finalizer.calls == 1 + assert len(fga.calls) == 1 + writes, deletes = fga.calls[0] + assert deletes == () + assert {row.relation for row in writes} == { + "ordinary_assignee", + "visible", + } + assert all("slot" not in row.relation and not row.relation.endswith(("_a", "_b")) for row in writes) + + @pytest.mark.asyncio async def test_move_plan_verifies_final_state_when_enabled_is_restored() -> None: plan = _move_plan() diff --git a/src/backend/test/permission/test_f048_projection_sql_runtime.py b/src/backend/test/permission/test_f048_projection_sql_runtime.py index ae91a52d2c..425a03a817 100644 --- a/src/backend/test/permission/test_f048_projection_sql_runtime.py +++ b/src/backend/test/permission/test_f048_projection_sql_runtime.py @@ -34,15 +34,21 @@ PermissionGrant, PermissionGrantAssignee, PermissionProjectionOperation, + PermissionVisibleSourceProjection, ResourcePermissionMode, ) from bisheng.permission.domain.services.grant_source_service import ( + GrantModelSnapshot, + GrantSnapshot, GrantSourceService, ) from bisheng.permission.domain.services.projection_plan import ( ProjectionPlan, ProjectionTupleDelta, ) +from bisheng.permission.domain.services.visibility_projection_service import ( + VisibilityProjectionCompiler, +) @pytest.fixture(autouse=True) @@ -64,6 +70,9 @@ async def session_factory( metadata = sa.MetaData() for name in ( "permission_projection_operation", + "permission_migration_run", + "permission_migration_item", + "permission_visible_source_projection", "permission_grant", "permission_grant_assignee", "resource_permission_mode", @@ -193,6 +202,44 @@ async def _seed_projecting_state(session_factory) -> int: ), ) ) + session.add_all( + ( + PermissionVisibleSourceProjection( + tenant_id=7, + resource_type="folder", + resource_id="42", + visibility_class="ordinary", + projected_subject="user:11", + source_kind="GRANT_ASSIGNEE", + source_owner_key="grant_assignee:101", + source_locator="direct:user:11", + source_fingerprint="d" * 64, + contribution_fingerprint="1" * 64, + model_key="viewer", + source_version=1, + tuple_fingerprint="2" * 64, + state="PENDING", + operation_id=int(operation.id), + ), + PermissionVisibleSourceProjection( + tenant_id=7, + resource_type="folder", + resource_id="42", + visibility_class="ordinary", + projected_subject="user:12", + source_kind="GRANT_ASSIGNEE", + source_owner_key="grant_assignee:102", + source_locator="direct:user:12", + source_fingerprint="e" * 64, + contribution_fingerprint="3" * 64, + model_key="viewer", + source_version=4, + tuple_fingerprint="4" * 64, + state="PENDING", + operation_id=int(operation.id), + ), + ) + ) return int(operation.id) @@ -227,6 +274,15 @@ async def test_resource_finalizer_atomically_converges_and_replays( assignees = list( (await session.execute(select(PermissionGrantAssignee).order_by(PermissionGrantAssignee.id))).scalars() ) + visible_sources = list( + ( + await session.execute( + select(PermissionVisibleSourceProjection).order_by( + PermissionVisibleSourceProjection.id + ) + ) + ).scalars() + ) assert (mode.version, mode.projection_state, mode.mode) == ( 4, @@ -238,6 +294,7 @@ async def test_resource_finalizer_atomically_converges_and_replays( ("ACTIVE", 1), ("INACTIVE", 5), ] + assert [row.state for row in visible_sources] == ["ACTIVE", "RETIRED"] @pytest.mark.asyncio @@ -304,6 +361,110 @@ async def test_assignee_move_preserves_identity_and_advances_version( assert moved.version == 2 +@pytest.mark.asyncio +async def test_visible_source_after_state_is_frozen_then_finalized( + session_factory, +) -> None: + source = GrantSourceService().canonicalize_source( + source_id=201, + subject_type="user", + subject_id="21", + source_type="DIRECT", + ) + grant = GrantSnapshot( + grant_id="g-viewer", + tenant_id=7, + resource_type="folder", + resource_id="42", + model=GrantModelSnapshot( + model_key="viewer", + active=True, + action_codes=("download",), + derived_level=1, + ), + active=True, + sources=(source,), + ) + compiler = VisibilityProjectionCompiler() + added = compiler.compile( + tenant_id=7, + grants=(grant,), + existing_sources=(), + ) + + with bypass_tenant_filter(): + async with session_factory() as session: + async with session.begin(): + operation = PermissionProjectionOperation( + tenant_id=7, + idempotency_key="visible-add", + request_checksum="a" * 64, + operation_type="GRANT_MUTATION", + scope_type="resource", + scope_key="folder:42", + expected_version=3, + target_version=4, + store_id="store", + model_id="model", + status="PREPARED", + before_checksum="b" * 64, + after_checksum="c" * 64, + operator_id=9, + ) + session.add(operation) + await session.flush() + operation_id = int(operation.id) + await SqlPermissionControlState._prepare_visible_sources( + session, + tenant_id=7, + visibility=added, + operation_id=operation_id, + ) + async with session_factory() as session: + pending = (await session.execute(select(PermissionVisibleSourceProjection))).scalars().one() + assert (pending.state, pending.operation_id) == ("PENDING", operation_id) + async with session_factory() as session: + async with session.begin(): + await SqlPermissionControlState._finalize_visible_sources( + session, + tenant_id=7, + visibility=added, + operation_id=operation_id, + ) + async with session_factory() as session: + active = (await session.execute(select(PermissionVisibleSourceProjection))).scalars().one() + assert active.state == "ACTIVE" + + removed = compiler.compile( + tenant_id=7, + grants=(replace(grant, active=False, sources=()),), + existing_sources=added.active_sources, + ) + with bypass_tenant_filter(): + async with session_factory() as session: + async with session.begin(): + await SqlPermissionControlState._prepare_visible_sources( + session, + tenant_id=7, + visibility=removed, + operation_id=operation_id, + ) + async with session_factory() as session: + pending = (await session.execute(select(PermissionVisibleSourceProjection))).scalars().one() + assert pending.state == "PENDING" + async with session_factory() as session: + async with session.begin(): + await SqlPermissionControlState._finalize_visible_sources( + session, + tenant_id=7, + visibility=removed, + operation_id=operation_id, + ) + async with session_factory() as session: + retired = (await session.execute(select(PermissionVisibleSourceProjection))).scalars().one() + assert retired.state == "RETIRED" + + def test_resource_claim_rejects_competing_same_version_operation() -> None: row = ResourcePermissionMode( tenant_id=7, diff --git a/src/backend/test/permission/test_f048_repositories.py b/src/backend/test/permission/test_f048_repositories.py index 8c8f8b0271..8b703f2d28 100644 --- a/src/backend/test/permission/test_f048_repositories.py +++ b/src/backend/test/permission/test_f048_repositories.py @@ -1,6 +1,7 @@ """F048 repository idempotency, version, cursor, and tenant contracts. -覆盖 AC: AC-19, AC-25, AC-27, AC-68, AC-93, AC-94, AC-143, AC-147 +覆盖 AC: AC-19, AC-25, AC-27, AC-68, AC-93, AC-94, AC-143, AC-147, +AC-165, AC-166, AC-168, AC-169, AC-171 """ from __future__ import annotations @@ -28,6 +29,7 @@ PermissionMigrationRun, PermissionProjectionOperation, PermissionProjectionTuple, + PermissionVisibleSourceProjection, ) from bisheng.permission.domain.repositories.catalog_repository import CatalogRepository from bisheng.permission.domain.repositories.grant_repository import GrantRepository @@ -54,6 +56,7 @@ "resource_permission_mode", "permission_migration_run", "permission_migration_item", + "permission_visible_source_projection", ) @@ -79,6 +82,7 @@ async def session_factory() -> AsyncIterator[SessionFactory]: "permission_projection_operation", "permission_projection_tuple", "permission_migration_item", + "permission_visible_source_projection", } ) tenant_filter.register_tenant_filter_events() @@ -333,6 +337,215 @@ async def test_projection_operation_idempotency_checksum_and_status_cas( assert await repository.aget_operation_checksum(stored.id) is not None +def _visible_source( + *, + owner: str, + fingerprint_char: str, + model_key: str = "viewer", + operation_id: int | None = None, + migration_item_id: int | None = None, +) -> PermissionVisibleSourceProjection: + return PermissionVisibleSourceProjection( + resource_type="knowledge_space", + resource_id="100", + visibility_class="ordinary", + projected_subject="user:7", + source_kind="GRANT_ASSIGNEE", + source_owner_key=owner, + source_locator=f"direct:user:{owner}", + source_fingerprint=fingerprint_char * 64, + contribution_fingerprint=fingerprint_char * 64, + model_key=model_key, + source_version=1, + tuple_fingerprint="f" * 64, + state="ACTIVE", + operation_id=operation_id, + migration_item_id=migration_item_id, + ) + + +@pytest.mark.asyncio +async def test_visible_source_contribution_uniqueness_reference_count_and_retire( + session_factory: SessionFactory, +) -> None: + repository = ProjectionRepository(session_factory) + first = await repository.aupsert_visible_source( + _visible_source(owner="grant_assignee:1", fingerprint_char="a") + ) + duplicate = await repository.aupsert_visible_source( + _visible_source(owner="grant_assignee:1", fingerprint_char="a") + ) + second = await repository.aupsert_visible_source( + _visible_source(owner="grant_assignee:2", fingerprint_char="b") + ) + + assert duplicate.id == first.id + assert second.id != first.id + assert await repository.acount_active_visible_sources( + resource_type="knowledge_space", + resource_id="100", + visibility_class="ordinary", + projected_subject="user:7", + ) == 2 + + checksum_before = await repository.aget_visible_source_checksum(states=("ACTIVE",)) + assert checksum_before is not None + assert await repository.aretire_visible_source( + projection_id=first.id, + expected_source_version=1, + operation_id=None, + ) + assert await repository.aretire_visible_source( + projection_id=first.id, + expected_source_version=1, + operation_id=None, + ) + assert await repository.acount_active_visible_sources( + resource_type="knowledge_space", + resource_id="100", + visibility_class="ordinary", + projected_subject="user:7", + ) == 1 + assert await repository.aget_visible_source_checksum(states=("ACTIVE",)) != checksum_before + + +@pytest.mark.asyncio +async def test_visible_source_model_owner_cursor_and_tenant_isolation( + session_factory: SessionFactory, +) -> None: + repository = ProjectionRepository(session_factory) + first = await repository.aupsert_visible_source( + _visible_source(owner="grant_assignee:1", fingerprint_char="a", model_key="editor") + ) + second = await repository.aupsert_visible_source( + _visible_source(owner="grant_assignee:2", fingerprint_char="b", model_key="editor") + ) + + model_page, model_cursor = await repository.aget_visible_model_cursor( + model_key="editor", + states=("ACTIVE",), + after_id=0, + limit=1, + ) + assert [row.id for row in model_page] == [first.id] + assert model_cursor == first.id + owner_page, owner_cursor = await repository.aget_visible_source_cursor( + source_kind="GRANT_ASSIGNEE", + source_owner_key="grant_assignee:2", + states=("ACTIVE",), + after_id=0, + limit=10, + ) + assert [row.id for row in owner_page] == [second.id] + assert owner_cursor is None + + token = set_current_tenant_id(2) + try: + tenant_two = await repository.aupsert_visible_source( + _visible_source(owner="grant_assignee:3", fingerprint_char="c", model_key="editor") + ) + tenant_two_page, _ = await repository.aget_visible_model_cursor( + model_key="editor", + states=("ACTIVE",), + after_id=0, + limit=10, + ) + assert [row.id for row in tenant_two_page] == [tenant_two.id] + assert await repository.aget_visible_source_checksum(states=("ACTIVE",)) is not None + finally: + current_tenant_id.reset(token) + + tenant_one_page, _ = await repository.aget_visible_model_cursor( + model_key="editor", + states=("ACTIVE",), + after_id=0, + limit=10, + ) + assert [row.id for row in tenant_one_page] == [first.id, second.id] + + +@pytest.mark.asyncio +async def test_visible_source_operation_and_migration_item_traceability( + session_factory: SessionFactory, +) -> None: + projection_repository = ProjectionRepository(session_factory) + migration_repository = MigrationRepository(session_factory) + operation = await projection_repository.acreate_operation( + PermissionProjectionOperation( + idempotency_key="visible-op", + request_checksum="a" * 64, + operation_type="VISIBLE_SOURCE_RECONCILE", + scope_type="RESOURCE", + scope_key="knowledge_space:100", + expected_version=1, + target_version=2, + store_id="store", + model_id="model", + before_checksum="b" * 64, + after_checksum="c" * 64, + operator_id=7, + ), + [], + ) + run = await migration_repository.aget_or_create_run( + PermissionMigrationRun( + environment_fingerprint="d" * 64, + phase="D2", + store_id="store", + source_model_id="old", + target_model_id="new", + ) + ) + item = await migration_repository.aupsert_item( + PermissionMigrationItem( + run_id=run.id, + tenant_id=1, + source_kind="MODEL_MAPPING", + source_locator="mapping:model:viewer", + source_checksum="e" * 64, + status="MIGRATED", + severity="INFO", + ) + ) + stored = await projection_repository.aupsert_visible_source( + _visible_source( + owner="grant_assignee:4", + fingerprint_char="d", + operation_id=operation.id, + migration_item_id=item.id, + ) + ) + + operation_rows = await projection_repository.aget_visible_operation_sources(operation.id) + assert [row.id for row in operation_rows] == [stored.id] + assert await projection_repository.aget_visible_operation_checksum(operation.id) is not None + migration_rows, cursor = await projection_repository.aget_visible_migration_cursor( + migration_item_id=item.id, + after_id=0, + limit=10, + ) + assert [row.id for row in migration_rows] == [stored.id] + assert cursor is None + + +def test_visible_source_schema_has_no_ab_slot() -> None: + table = SQLModel.metadata.tables["permission_visible_source_projection"] + assert "visibility_slot" not in table.c + unique_columns = { + tuple(column.name for column in constraint.columns) + for constraint in table.constraints + if isinstance(constraint, sa.UniqueConstraint) + } + assert ( + "tenant_id", + "resource_type", + "resource_id", + "visibility_class", + "projected_subject", + "contribution_fingerprint", + ) in unique_columns + + @pytest.mark.asyncio async def test_migration_environment_lease_item_and_checkpoint_resume( session_factory: SessionFactory, diff --git a/src/backend/test/permission/test_f048_schema_contract.py b/src/backend/test/permission/test_f048_schema_contract.py index ce60da636c..eeb6446020 100644 --- a/src/backend/test/permission/test_f048_schema_contract.py +++ b/src/backend/test/permission/test_f048_schema_contract.py @@ -24,6 +24,9 @@ from bisheng.core.database.alembic.versions import ( v3_0_0_f048_migration_item_message_longtext as message_revision, ) +from bisheng.core.database.alembic.versions import ( + v3_0_0_f048_visible_source_projection as visible_revision, +) from bisheng.core.database.dialect_helpers import LargeText from bisheng.permission.domain import models as permission_models @@ -32,6 +35,9 @@ MESSAGE_REVISION_PATH = ( BACKEND_ROOT / "bisheng/core/database/alembic/versions/v3_0_0_f048_migration_item_message_longtext.py" ) +VISIBLE_REVISION_PATH = ( + BACKEND_ROOT / "bisheng/core/database/alembic/versions/v3_0_0_f048_visible_source_projection.py" +) F048_TABLES = { "authorization_model_release", @@ -46,9 +52,11 @@ "resource_permission_mode", "permission_projection_operation", "permission_projection_tuple", + "permission_visible_source_projection", "permission_migration_run", "permission_migration_item", } +BASE_REVISION_TABLES = F048_TABLES - {"permission_visible_source_projection"} TENANT_TABLES = { "permission_grant", @@ -56,6 +64,7 @@ "resource_permission_mode", "permission_projection_operation", "permission_projection_tuple", + "permission_visible_source_projection", } @@ -117,6 +126,7 @@ def test_f048_unique_and_foreign_key_contract() -> None: "uq_resource_permission_mode", "uq_perm_projection_idempotency", "uq_perm_projection_tuple", + "uq_perm_visible_source_contribution", "uq_perm_migration_environment", "uq_perm_migration_item_source", } <= unique_names @@ -139,7 +149,32 @@ def test_f048_revision_is_the_single_alembic_head() -> None: "script_location", str(BACKEND_ROOT / "bisheng/core/database/alembic"), ) - assert ScriptDirectory.from_config(config).get_heads() == ["f048_migration_item_message_longtext"] + assert ScriptDirectory.from_config(config).get_heads() == ["f048_visible_source_projection"] + + +def test_f048_visible_projection_revision_is_static_ddl_only() -> None: + source = VISIBLE_REVISION_PATH.read_text(encoding="utf-8") + tree = ast.parse(source) + called_names = { + node.func.attr for node in ast.walk(tree) if isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute) + } + + assert visible_revision.down_revision == "linsight_pending_files" + assert visible_revision.revision == "f048_visible_source_projection" + assert {"create_table", "create_index", "drop_table"} <= called_names + assert not {"execute", "bulk_insert"} & called_names + + +def test_f048_visible_projection_revision_is_idempotent_and_downgrades() -> None: + engine = sa.create_engine("sqlite://") + with engine.connect() as connection: + context = MigrationContext.configure(connection) + with Operations.context(context): + visible_revision.upgrade() + assert "permission_visible_source_projection" in inspect(connection).get_table_names() + visible_revision.upgrade() + visible_revision.downgrade() + assert "permission_visible_source_projection" not in inspect(connection).get_table_names() def test_f048_message_revision_is_static_ddl_only() -> None: @@ -189,10 +224,10 @@ def test_f048_revision_upgrade_is_idempotent_and_downgrades() -> None: context = MigrationContext.configure(connection) with Operations.context(context): revision.upgrade() - assert F048_TABLES <= set(inspect(connection).get_table_names()) + assert BASE_REVISION_TABLES <= set(inspect(connection).get_table_names()) revision.upgrade() revision.downgrade() - assert not (F048_TABLES & set(inspect(connection).get_table_names())) + assert not (BASE_REVISION_TABLES & set(inspect(connection).get_table_names())) def test_f048_tables_compile_for_mysql_without_native_enum_or_json() -> None: diff --git a/src/backend/test/permission/test_f048_tuple_mapper.py b/src/backend/test/permission/test_f048_tuple_mapper.py index c26bbae78d..286b981808 100644 --- a/src/backend/test/permission/test_f048_tuple_mapper.py +++ b/src/backend/test/permission/test_f048_tuple_mapper.py @@ -219,6 +219,53 @@ def test_duplicate_tuple_is_idempotently_deduplicated(): assert result.deduplicated_count == 1 +def test_direct_and_membership_sources_share_grant_but_keep_provenance(): + tuples = ( + _tuple(user="user:11", relation="viewer"), + _tuple(user="user:11", relation="editor"), + ) + bindings = ( + LegacyGrantBinding( + binding_key="direct-binding", + tenant_id=7, + resource_type="workflow", + resource_id="wf-1", + relation="viewer", + model_source_key="custom-combined", + subject_type="user", + subject_id="11", + source_type="DIRECT", + source_ref="direct:11", + ), + LegacyGrantBinding( + binding_key="membership-binding", + tenant_id=7, + resource_type="workflow", + resource_id="wf-1", + relation="editor", + model_source_key="custom-combined", + subject_type="user", + subject_id="11", + source_type="SPACE_MEMBERSHIP", + source_ref="membership:99", + ), + ) + + result = map_legacy_tuples( + tuples, + bindings, + model_key_by_source={"custom-combined": "custom-combined"}, + ) + + assert len(result.grants) == 1 + assert len(result.grants[0].assignees) == 2 + assert {row.source_type for row in result.grants[0].assignees} == { + "DIRECT", + "SPACE_MEMBERSHIP", + } + assert len({row.source_checksum for row in result.grants[0].assignees}) == 2 + + def test_system_shared_and_parent_facts_are_preserved_not_converted_to_grants(): tuples = ( _tuple(user="system:public", relation="shared_with"), diff --git a/src/backend/test/permission/test_f048_visibility_observability.py b/src/backend/test/permission/test_f048_visibility_observability.py new file mode 100644 index 0000000000..deb035917c --- /dev/null +++ b/src/backend/test/permission/test_f048_visibility_observability.py @@ -0,0 +1,172 @@ +"""Observability contract for flattened visibility and list strategies.""" + +from __future__ import annotations + +from dataclasses import dataclass + +import pytest + +from bisheng.common.errcode.permission import PermissionEnumerationIncompleteError +from bisheng.permission.domain.services.grant_source_service import ( + GrantModelSnapshot, + GrantSnapshot, + GrantSourceRecord, +) +from bisheng.permission.domain.services.permission_action_service import ( + F048PermissionService, + PermissionActor, +) +from bisheng.permission.domain.services.visibility_projection_service import ( + VisibilityProjectionCompiler, + VisibilityProjectionReconciler, +) + + +class _ReadyCatalog: + async def ensure_runtime_ready(self): + return None + + async def is_action_effective(self, resource_type, action): + return True + + +class _Fence: + async def ensure_readable(self, target): + return None + + +class _Marker: + async def consistency_for(self, **kwargs): + return "HIGHER_CONSISTENCY" + + +class _Policy: + async def allows(self, resource_type, action, max_results): + return True + + +@dataclass +class _FGA: + objects: tuple[str, ...] = () + error: Exception | None = None + + async def stream_list_objects(self, **kwargs): + if self.error: + raise self.error + return self.objects + + +def _service(fga: _FGA) -> F048PermissionService: + return F048PermissionService( + catalog=_ReadyCatalog(), + scope_fence=_Fence(), + marker=_Marker(), + fga=fga, + list_policy=_Policy(), + ) + + +async def test_visible_list_metric_reports_complete_stream_and_capacity_alert(monkeypatch): + metrics = [] + monkeypatch.setattr( + "bisheng.permission.domain.services.permission_action_service.emit_metric", + lambda domain, **fields: metrics.append((domain, fields)), + ) + + result = await _service( + _FGA(tuple(f"knowledge_space:{index}" for index in range(8))) + ).list_visible_objects( + PermissionActor(user_id=7, current_tenant_id=3), + resource_type="knowledge_space", + max_results=10, + ) + + assert len(result.object_ids) == 8 + domain, fields = metrics[-1] + assert domain == "permission_visible_list" + assert fields["strategy"] == "visible_ids_first" + assert fields["visible_count"] == fields["scanned_count"] == 8 + assert fields["stream_completed"] is True + assert fields["capacity"] == 10 + assert fields["fga_elapsed_ms"] >= 0 + assert fields["total_elapsed_ms"] >= fields["fga_elapsed_ms"] + assert fields["alert"] == "capacity_80_percent" + assert not {"user_name", "resource_name", "token", "config"}.intersection(fields) + + +async def test_visible_list_metric_marks_incomplete_stream(monkeypatch): + metrics = [] + monkeypatch.setattr( + "bisheng.permission.domain.services.permission_action_service.emit_metric", + lambda domain, **fields: metrics.append((domain, fields)), + ) + + with pytest.raises(PermissionEnumerationIncompleteError): + await _service(_FGA(error=RuntimeError("transport closed"))).list_visible_objects( + PermissionActor(user_id=7, current_tenant_id=3), + resource_type="knowledge_space", + max_results=5000, + ) + + assert metrics[-1][1]["stream_completed"] is False + assert metrics[-1][1]["alert"] == "stream_incomplete" + + +def _grant() -> GrantSnapshot: + return GrantSnapshot( + grant_id="11", + tenant_id=3, + resource_type="knowledge_space", + resource_id="42", + model=GrantModelSnapshot(model_key="viewer", active=False, action_codes=()), + active=True, + sources=( + GrantSourceRecord( + source_id=19, + subject_type="user", + subject_id="7", + userset_relation=None, + include_children=False, + source_type="DIRECT", + source_ref="grant:11", + source_locator="grant:11:assignee:19", + source_fingerprint="a" * 64, + projected_subject="user:7", + protected=False, + active=True, + version=2, + ), + ), + ) + + +def test_projection_metrics_report_sources_checksums_and_orphan_alert(monkeypatch): + metrics = [] + monkeypatch.setattr( + "bisheng.permission.domain.services.visibility_projection_service.emit_metric", + lambda domain, **fields: metrics.append((domain, fields)), + ) + compiled = VisibilityProjectionCompiler().compile( + tenant_id=3, + grants=(_grant(),), + existing_sources=(), + ) + plan = VisibilityProjectionReconciler().plan( + canonical_sources=compiled.active_sources, + persisted_sources=compiled.active_sources, + live_tuples=frozenset( + { + ("user:7", "visible", "knowledge_space:42"), + ("user:9", "visible", "knowledge_space:42"), + } + ), + ) + + assert plan.deltas[0].action == "DELETE" + project = next(fields for domain, fields in metrics if fields["operation"] == "project") + reconcile = next(fields for domain, fields in metrics if fields["operation"] == "reconcile") + assert project["source_count"] == project["unique_tuple_count"] == 1 + assert len(project["source_checksum"]) == len(project["aggregate_checksum"]) == 64 + assert reconcile["orphan_count"] == 1 + assert reconcile["alert"] == "orphan_visible_tuple" + assert not {"user_name", "resource_name", "token", "config"}.intersection(reconcile) diff --git a/src/backend/test/permission/test_f048_visibility_projection.py b/src/backend/test/permission/test_f048_visibility_projection.py new file mode 100644 index 0000000000..1649f39abc --- /dev/null +++ b/src/backend/test/permission/test_f048_visibility_projection.py @@ -0,0 +1,232 @@ +"""Single-slot visible source projection compiler contracts. + +覆盖 AC: AC-159, AC-164, AC-165, AC-166, AC-168, AC-169, AC-171 +""" + +from __future__ import annotations + +from dataclasses import replace + +from bisheng.permission.domain.services.grant_source_service import ( + GrantModelSnapshot, + GrantSnapshot, + GrantSourceRecord, + GrantSourceService, +) +from bisheng.permission.domain.services.visibility_projection_service import ( + VisibilityProjectionCompiler, +) + + +def _grant( + *, + grant_id: str, + model_key: str, + sources: tuple[GrantSourceRecord, ...], + model_active: bool = True, + actions: tuple[str, ...] = ("edit",), +) -> GrantSnapshot: + return GrantSnapshot( + grant_id=grant_id, + tenant_id=7, + resource_type="knowledge_space", + resource_id="100", + model=GrantModelSnapshot( + model_key=model_key, + active=model_active, + action_codes=actions, + ), + active=True, + sources=sources, + ) + + +def test_direct_department_subtree_group_and_protected_compile_shallow_sources() -> None: + source_service = GrantSourceService() + sources = ( + source_service.canonicalize_source( + source_id=1, + subject_type="user", + subject_id="10", + source_type="DIRECT", + ), + source_service.canonicalize_source( + source_id=2, + subject_type="department", + subject_id="17", + source_type="DEPARTMENT", + ), + source_service.canonicalize_source( + source_id=3, + subject_type="department", + subject_id="18", + source_type="DEPARTMENT", + include_children=True, + ), + source_service.canonicalize_source( + source_id=4, + subject_type="user_group", + subject_id="8", + source_type="USER_GROUP", + userset_relation="admin", + ), + source_service.canonicalize_source( + source_id=5, + subject_type="user", + subject_id="11", + source_type="CREATOR", + source_ref="knowledge_space:100", + protected=True, + ), + ) + + result = VisibilityProjectionCompiler().compile( + tenant_id=7, + grants=(_grant(grant_id="g1", model_key="manager", sources=sources),), + existing_sources=(), + ) + + assert {row.projected_subject for row in result.active_sources} == { + "user:10", + "department:17#member", + "department:18#subtree_member", + "user_group:8#admin", + "user:11", + } + assert {row.visibility_class for row in result.active_sources} == { + "ordinary", + "protected", + } + assert {row.source_owner_key for row in result.active_sources} == { + f"grant_assignee:{source.source_id}" for source in sources + } + assert {(row.action, row.relation) for row in result.deltas} == { + ("WRITE", "visible"), + } + assert all(row.object == "knowledge_space:100" for row in result.deltas) + + +def test_multiple_models_and_sources_share_one_aggregate_until_last_revoke() -> None: + source_service = GrantSourceService() + direct = source_service.canonicalize_source( + source_id=10, + subject_type="user", + subject_id="42", + source_type="DIRECT", + ) + membership = source_service.canonicalize_source( + source_id=11, + subject_type="user", + subject_id="42", + source_type="SPACE_MEMBERSHIP", + source_ref="573", + ) + compiler = VisibilityProjectionCompiler() + initial = compiler.compile( + tenant_id=7, + grants=( + _grant(grant_id="g-editor", model_key="editor", sources=(direct,)), + _grant(grant_id="g-viewer", model_key="viewer", sources=(membership,)), + ), + existing_sources=(), + ) + assert len(initial.active_sources) == 2 + assert len({row.contribution_fingerprint for row in initial.active_sources}) == 2 + assert [(row.action, row.relation) for row in initial.deltas] == [ + ("WRITE", "visible") + ] + + one_remaining = compiler.compile( + tenant_id=7, + grants=(_grant(grant_id="g-viewer", model_key="viewer", sources=(membership,)),), + existing_sources=initial.active_sources, + ) + assert len(one_remaining.active_sources) == 1 + assert len(one_remaining.retired_sources) == 1 + assert one_remaining.deltas == () + + none_remaining = compiler.compile( + tenant_id=7, + grants=(), + existing_sources=one_remaining.active_sources, + ) + assert none_remaining.active_sources == () + assert len(none_remaining.retired_sources) == 1 + assert [(row.action, row.relation) for row in none_remaining.deltas] == [ + ("DELETE", "visible") + ] + + +def test_inactive_and_visibility_only_existing_grants_still_contribute() -> None: + source = GrantSourceService().canonicalize_source( + source_id=20, + subject_type="user", + subject_id="42", + source_type="DIRECT", + ) + result = VisibilityProjectionCompiler().compile( + tenant_id=7, + grants=( + _grant( + grant_id="g-inactive", + model_key="inactive-custom", + sources=(source,), + model_active=False, + actions=(), + ), + ), + existing_sources=(), + ) + assert len(result.active_sources) == 1 + assert result.active_sources[0].model_key == "inactive-custom" + assert [(row.action, row.relation) for row in result.deltas] == [ + ("WRITE", "visible") + ] + + +def test_system_public_and_shared_sources_do_not_enter_grant_projection() -> None: + direct = GrantSourceService().canonicalize_source( + source_id=30, + subject_type="user", + subject_id="42", + source_type="DIRECT", + ) + system_source = replace( + direct, + source_id=31, + source_type="PUBLIC", + source_locator="public_policy:1", + source_fingerprint="a" * 64, + ) + result = VisibilityProjectionCompiler().compile( + tenant_id=7, + grants=(_grant(grant_id="g1", model_key="viewer", sources=(system_source,)),), + existing_sources=(), + ) + assert result.active_sources == () + assert result.retired_sources == () + assert result.deltas == () + + +def test_compiler_is_deterministic_and_has_no_visibility_slot() -> None: + source = GrantSourceService().canonicalize_source( + source_id=40, + subject_type="user", + subject_id="42", + source_type="DIRECT", + ) + compiler = VisibilityProjectionCompiler() + first = compiler.compile( + tenant_id=7, + grants=(_grant(grant_id="g1", model_key="viewer", sources=(source,)),), + existing_sources=(), + ) + second = compiler.compile( + tenant_id=7, + grants=(_grant(grant_id="g1", model_key="viewer", sources=(source,)),), + existing_sources=(), + ) + assert first == second + assert first.source_checksum == second.source_checksum + assert first.aggregate_checksum == second.aggregate_checksum + assert all(not hasattr(row, "visibility_slot") for row in first.active_sources) diff --git a/src/backend/test/permission/test_f048_visibility_reconcile.py b/src/backend/test/permission/test_f048_visibility_reconcile.py new file mode 100644 index 0000000000..190f9a6e30 --- /dev/null +++ b/src/backend/test/permission/test_f048_visibility_reconcile.py @@ -0,0 +1,190 @@ +"""F048 flattened visibility reconciliation contracts. + +覆盖 AC: AC-165, AC-167, AC-168, AC-170, AC-171 +""" + +from __future__ import annotations + +import pytest + +from bisheng.common.errcode.permission import ( + PermissionPublishNotReadyError, +) +from bisheng.permission.domain.services.grant_source_service import ( + GrantModelSnapshot, + GrantSnapshot, + GrantSourceService, +) +from bisheng.permission.domain.services.model_policy import ( + DerivedPermissionModel, + ModelReferenceSummary, + ensure_model_deletable, +) +from bisheng.permission.domain.services.visibility_projection_service import ( + VisibilityProjectionCompiler, + VisibilityProjectionReconciler, +) + + +def _sources(*, second_model: bool = False): + service = GrantSourceService() + direct = service.canonicalize_source( + source_id=1, + subject_type="user", + subject_id="100", + source_type="DIRECT", + ) + membership = service.canonicalize_source( + source_id=2, + subject_type="user", + subject_id="100", + source_type="SPACE_MEMBERSHIP", + source_ref="space:42:user:100", + ) + grants = [] + for index, source in enumerate((direct, membership)): + model_key = "editor" if second_model and index else "viewer" + grants.append( + GrantSnapshot( + grant_id=f"g-{model_key}-{index}", + tenant_id=7, + resource_type="workflow", + resource_id="42", + model=GrantModelSnapshot( + model_key=model_key, + active=True, + action_codes=("download",), + derived_level=1, + ), + active=True, + sources=(source,), + ) + ) + return VisibilityProjectionCompiler().compile( + tenant_id=7, + grants=tuple(grants), + existing_sources=(), + ).active_sources + + +def _live_key(source): + return ( + source.projected_subject, + "visible", + f"{source.resource_type}:{source.resource_id}", + ) + + +def test_multiple_sources_and_models_keep_one_live_tuple_and_repair_only_missing_source() -> None: + canonical = _sources(second_model=True) + live = frozenset({_live_key(canonical[0])}) + + plan = VisibilityProjectionReconciler().plan( + canonical_sources=canonical, + persisted_sources=(canonical[0],), + live_tuples=live, + ) + + assert [row.contribution_fingerprint for row in plan.upsert_sources] == [ + canonical[1].contribution_fingerprint, + ] + assert plan.retire_sources == () + assert plan.deltas == () + assert plan.blockers == () + + +def test_missing_and_orphan_live_tuples_produce_only_exact_aggregate_deltas() -> None: + canonical = _sources() + desired_key = _live_key(canonical[0]) + orphan_key = ("user:999", "visible", "workflow:42") + + missing = VisibilityProjectionReconciler().plan( + canonical_sources=canonical, + persisted_sources=canonical, + live_tuples=frozenset(), + ) + assert [(row.action, row.key) for row in missing.deltas] == [("WRITE", desired_key)] + + orphan = VisibilityProjectionReconciler().plan( + canonical_sources=canonical, + persisted_sources=canonical, + live_tuples=frozenset({desired_key, orphan_key}), + ) + assert [(row.action, row.key) for row in orphan.deltas] == [("DELETE", orphan_key)] + + +def test_mixed_source_and_live_drift_does_not_touch_valid_contribution() -> None: + canonical = _sources(second_model=True) + valid = canonical[0] + missing = canonical[1] + orphan = valid.model_copy( + update={ + "projected_subject": "user:999", + "contribution_fingerprint": "a" * 64, + "tuple_fingerprint": "b" * 64, + "source_owner_key": "grant_assignee:999", + "source_locator": "direct:user:999", + "source_fingerprint": "c" * 64, + } + ) + valid_key = _live_key(valid) + orphan_key = _live_key(orphan) + + plan = VisibilityProjectionReconciler().plan( + canonical_sources=canonical, + persisted_sources=(valid, orphan), + live_tuples=frozenset({valid_key, orphan_key}), + ) + + assert plan.upsert_sources == (missing,) + assert plan.retire_sources == (orphan,) + assert {(row.action, row.key) for row in plan.deltas} == { + ("DELETE", orphan_key), + } + assert all(row.key != valid_key for row in plan.deltas) + assert plan.target_checksum != plan.live_checksum + + +def test_failed_closed_or_incomplete_snapshot_is_never_guessed_ready() -> None: + canonical = _sources() + failed = canonical[0].model_copy(update={"state": "FAILED_CLOSED"}) + reconciler = VisibilityProjectionReconciler() + + plan = reconciler.plan( + canonical_sources=canonical, + persisted_sources=(failed,), + live_tuples=frozenset(), + ledger_complete=False, + ) + + assert plan.blockers + with pytest.raises(PermissionPublishNotReadyError): + reconciler.ensure_ready(plan) + + +def test_model_delete_remains_blocked_until_source_and_live_residuals_are_zero() -> None: + model = DerivedPermissionModel( + model_key="custom", + name="Custom", + kind="CUSTOM", + config_scope="PLATFORM", + derived_level=1, + active=True, + allow_same_level=False, + selected_action_codes=("download",), + action_codes=("download",), + ) + with pytest.raises(ValueError, match="still referenced"): + ensure_model_deletable( + model, + references=ModelReferenceSummary( + pending_source_count=1, + live_tuple_count=1, + residual_checksum="a" * 64, + ), + ) + + ensure_model_deletable( + model, + references=ModelReferenceSummary(), + ) diff --git a/src/backend/test/permission/test_f048_visible_reconcile_cli.py b/src/backend/test/permission/test_f048_visible_reconcile_cli.py new file mode 100644 index 0000000000..d21a8fe28f --- /dev/null +++ b/src/backend/test/permission/test_f048_visible_reconcile_cli.py @@ -0,0 +1,147 @@ +"""Contracts for the production F048 visible reconciliation command.""" + +from __future__ import annotations + +from types import SimpleNamespace + +import pytest + +from bisheng.permission.domain.schemas import VisibleSourceProjectionDTO +from scripts import reconcile_f048_visible_projection as cli + + +def _source(*, fingerprint: str, subject: str = "user:7") -> VisibleSourceProjectionDTO: + return VisibleSourceProjectionDTO( + tenant_id=1, + resource_type="knowledge_space", + resource_id="42", + visibility_class="ordinary", + projected_subject=subject, + source_kind="GRANT_ASSIGNEE", + source_owner_key=f"grant_assignee:{fingerprint[0]}", + source_locator=f"direct:{subject}", + source_fingerprint=fingerprint, + contribution_fingerprint=fingerprint, + model_key="viewer", + source_version=1, + tuple_fingerprint="f" * 64, + state="ACTIVE", + ) + + +def test_parse_defaults_to_dry_run_and_apply_requires_store_confirmation() -> None: + args = cli.parse_args([]) + assert args.apply is False + assert args.batch_size == 80 + + with pytest.raises(SystemExit) as exc_info: + cli.parse_args(["--apply"]) + assert exc_info.value.code == 2 + + args = cli.parse_args( + [ + "--apply", + "--confirm-store-id", + "store-1", + "--operator-id", + "7", + "--batch-size", + "90", + ] + ) + assert args.apply is True + assert args.confirm_store_id == "store-1" + assert args.batch_size == 90 + + +def test_report_deduplicates_only_the_same_projected_subject_tuple() -> None: + first = _source(fingerprint="a" * 64) + second = _source(fingerprint="b" * 64) + department = _source( + fingerprint="c" * 64, + subject="department:7#member", + ) + user_group = _source( + fingerprint="d" * 64, + subject="user_group:9#member", + ) + current = cli.CurrentRelease( + catalog_id=1, + catalog_key="catalog-v1", + store_id="store-1", + model_id="model-old", + model_release_id=2, + model_checksum="c" * 64, + write_fenced=False, + ) + + report, upserts, retires, expected = cli._build_report( + mode="dry-run", + current=current, + target_model_id=None, + target_checksum="d" * 64, + grants=(SimpleNamespace(),), + assignee_count=4, + canonical_sources=(first, second, department, user_group), + persisted=(), + ) + + assert report.canonical_source_count == 4 + assert report.expected_tuple_count == 3 + assert upserts == (first, second, department, user_group) + assert retires == () + assert expected == { + ("user:7", "visible", "knowledge_space:42"), + ("department:7#member", "visible", "knowledge_space:42"), + ("user_group:9#member", "visible", "knowledge_space:42"), + } + + +class _FGAClient: + def __init__(self) -> None: + self.writes = [] + self.checks = [] + + @staticmethod + def validate_business_mutation_size(operation_count: int) -> None: + assert operation_count <= 90 + + async def write_tuples(self, *, writes, ignore_duplicate_writes=False): + self.writes.append((writes, ignore_duplicate_writes)) + + async def batch_check(self, checks, consistency=None): + self.checks.append((checks, consistency)) + return [True] * len(checks) + + +@pytest.mark.asyncio +async def test_reconcile_ensures_without_live_scan_and_verifies_usersets() -> None: + client = _FGAClient() + expected = frozenset( + { + ("department:7#member", "visible", "knowledge_space:42"), + ("user_group:9#member", "visible", "knowledge_space:42"), + } + ) + + await cli._ensure_expected_tuples(client, expected, batch_size=80) + await cli._verify_expected_tuples(client, expected) + + assert client.writes == [ + ( + [ + { + "user": "department:7#member", + "relation": "visible", + "object": "knowledge_space:42", + }, + { + "user": "user_group:9#member", + "relation": "visible", + "object": "knowledge_space:42", + }, + ], + True, + ) + ] + assert client.checks[0][1] == cli.HIGHER_CONSISTENCY diff --git a/src/backend/test/workstation/test_config_resolve_root_bypass.py b/src/backend/test/workstation/test_config_resolve_root_bypass.py new file mode 100644 index 0000000000..12928905bb --- /dev/null +++ b/src/backend/test/workstation/test_config_resolve_root_bypass.py @@ -0,0 +1,108 @@ +"""Root's stored config must survive a narrowed tenant filter. + +Every ``SELECT`` goes through the tenant auto-filter, which narrows by the +request's visible-tenant set. When that set does not contain Root, Root's own +``tenant_workstation_config`` row reads back as missing — indistinguishable from +"this deployment has never saved a config". Treating that as "no config" is +destructive: ``get_daily_chat_config_with_meta`` fabricates built-in defaults +from it, and the 工作台配置 page round-trips whatever it was shown, so the next +保存 replaces the real config (welcome message, icons, org KBs, tool pool) with +defaults. That is exactly what happened on 2026-08-13. + +``aresolve`` / ``resolve`` therefore re-read unfiltered before concluding that +Root has no config, and the service reports ``is_fallback`` so callers can tell a +fabricated default from a saved config. +""" + +from __future__ import annotations + +from types import SimpleNamespace + +import pytest + +from bisheng.core.context.tenant import is_tenant_filter_bypassed +from bisheng.workstation.domain.models.tenant_workstation_config import ( + TenantWorkstationConfigDao, +) + +ROOT = 1 +CHILD = 36 +KEY = "workstation" +STORED = '{"welcomeMessage": "\\u6211\\u662f BISHENG"}' + + +def _filtered_reader(visible: set[int] | None): + """Stand-in for ``aget``/``get`` under a tenant filter. + + Rows exist only for Root. ``visible`` is the request's visible-tenant set: + a read is served only when the filter is bypassed or Root is visible — + mirroring how the real listener rewrites the statement. + """ + + def read(tenant_id: int, key: str): + if tenant_id != ROOT or key != KEY: + return None + if is_tenant_filter_bypassed() or visible is None or ROOT in visible: + return SimpleNamespace(tenant_id=ROOT, key=key, value=STORED) + return None + + return read + + +@pytest.mark.parametrize("visible", [None, {ROOT}, {CHILD}, set()]) +async def test_aresolve_returns_root_config_whatever_the_visible_set(monkeypatch, visible): + reader = _filtered_reader(visible) + monkeypatch.setattr( + TenantWorkstationConfigDao, + "aget", + classmethod(lambda cls, tenant_id, key: _async(reader(tenant_id, key))), + ) + + value, inherited, source_tenant_id, has_override = await TenantWorkstationConfigDao.aresolve(ROOT, KEY) + + assert value == STORED, "Root's saved config must not read back as absent" + assert inherited is False + assert source_tenant_id == ROOT + assert has_override is True + + +@pytest.mark.parametrize("visible", [None, {ROOT}, {CHILD}, set()]) +def test_resolve_sync_matches_async(monkeypatch, visible): + reader = _filtered_reader(visible) + monkeypatch.setattr( + TenantWorkstationConfigDao, "get", classmethod(lambda cls, tenant_id, key: reader(tenant_id, key)) + ) + + assert TenantWorkstationConfigDao.resolve(ROOT, KEY) == (STORED, False, ROOT, True) + + +async def test_aresolve_reports_no_config_when_root_row_really_is_absent(monkeypatch): + """The fallback path stays intact for a genuinely fresh deployment.""" + monkeypatch.setattr( + TenantWorkstationConfigDao, + "aget", + classmethod(lambda cls, tenant_id, key: _async(None)), + ) + + assert await TenantWorkstationConfigDao.aresolve(ROOT, KEY) == (None, False, ROOT, False) + + +async def test_child_tenant_still_inherits_from_root(monkeypatch): + """A Child with no override keeps reading Root's config, marked inherited.""" + reader = _filtered_reader({CHILD, ROOT}) + monkeypatch.setattr( + TenantWorkstationConfigDao, + "aget", + classmethod(lambda cls, tenant_id, key: _async(reader(tenant_id, key) if tenant_id == ROOT else None)), + ) + + value, inherited, source_tenant_id, has_override = await TenantWorkstationConfigDao.aresolve(CHILD, KEY) + + assert value == STORED + assert inherited is True + assert source_tenant_id == ROOT + assert has_override is False + + +async def _async(value): + return value diff --git a/src/backend/test/workstation/test_daily_config_is_fallback.py b/src/backend/test/workstation/test_daily_config_is_fallback.py new file mode 100644 index 0000000000..628fd8dd94 --- /dev/null +++ b/src/backend/test/workstation/test_daily_config_is_fallback.py @@ -0,0 +1,76 @@ +"""``get_daily_chat_config_with_meta`` must say when it fabricated a default. + +The 工作台配置 page round-trips whatever the GET hands it, so it needs to tell a +saved config from the built-in default the service invents when nothing +resolves. Without that signal a single failed read turns into a permanent +overwrite the moment an admin hits 保存 (2026-08-13). ``is_fallback`` is the +contract the page's save guard is built on — keep it honest. +""" + +from __future__ import annotations + +import pytest + +from bisheng.api.v1.schemas import WorkstationConfig +from bisheng.llm.domain.services.llm import LLMService +from bisheng.workstation.domain.services.workstation_service import WorkStationService + +STORED = '{"welcomeMessage": "saved"}' + + +@pytest.fixture(autouse=True) +def _neutralize_side_lookups(monkeypatch: pytest.MonkeyPatch): + """Keep the test on the resolve path: no DB, no tool sync, no LLM config.""" + monkeypatch.setattr(WorkStationService, "sync_tool_info", classmethod(lambda cls, tools: tools)) + monkeypatch.setattr(LLMService, "get_workbench_llm", staticmethod(_none)) + monkeypatch.setattr( + WorkStationService, + "_abuild_default_daily_config", + classmethod(lambda cls: _config(WorkstationConfig(welcomeMessage="built-in default"))), + ) + + +async def test_saved_config_is_not_flagged_as_fallback(monkeypatch): + monkeypatch.setattr( + WorkStationService, + "_aresolve_tenant_config", + classmethod(lambda cls, key: _tuple((STORED, False, 1, True))), + ) + + ( + ret, + inherited, + source_tenant_id, + has_override, + is_fallback, + ) = await WorkStationService.get_daily_chat_config_with_meta() + + assert ret.welcomeMessage == "saved" + assert (inherited, source_tenant_id, has_override) == (False, 1, True) + assert is_fallback is False + + +async def test_missing_config_is_flagged_as_fallback(monkeypatch): + monkeypatch.setattr( + WorkStationService, + "_aresolve_tenant_config", + classmethod(lambda cls, key: _tuple((None, False, 1, False))), + ) + + ret, _inherited, _source, has_override, is_fallback = await WorkStationService.get_daily_chat_config_with_meta() + + assert ret.welcomeMessage == "built-in default" + assert has_override is False + assert is_fallback is True, "a fabricated default must never look like the admin's saved config" + + +async def _none(): + return None + + +async def _tuple(value): + return value + + +async def _config(value): + return value diff --git a/src/backend/test/workstation/test_sync_tool_info_strict_recovery.py b/src/backend/test/workstation/test_sync_tool_info_strict_recovery.py new file mode 100644 index 0000000000..2bb713b674 --- /dev/null +++ b/src/backend/test/workstation/test_sync_tool_info_strict_recovery.py @@ -0,0 +1,109 @@ +"""A narrowed tenant filter must not read back as "the admin deleted the tools". + +``sync_tool_info`` drops tool groups it cannot find — that is how a real +deletion reaches saved configs. The trap is that the tenant auto-filter produces +the same "not found" for the config owner's own rows whenever the request's +visible-tenant IN-list excludes that tenant. The 工作台配置 page round-trips what +it is shown, so one filtered read would persist as a cleared tool pool — the +narrower survivor of the 2026-08-13 incident, after the config body itself was +protected by the resolve-level fix. + +So an unresolved group triggers a re-read pinned to the config's own tenant +(``strict_tenant_filter`` narrows to ``tenant_id = current``; it never widens), +and only a drop that survives *that* is believed. +""" + +from __future__ import annotations + +from types import SimpleNamespace + +import pytest + +from bisheng.core.context.tenant import is_strict_tenant_filter +from bisheng.tool.domain.models.gpts_tools import GptsToolsDao +from bisheng.workstation.domain.services.workstation_service import WorkStationService + +ROOT = 1 +OTHER = 36 + +TYPES = { + 16: SimpleNamespace(id=16, name="联网搜索", is_preset=1, description="搜索互联网信息"), + 6: SimpleNamespace(id=6, name="代码执行器", is_preset=1, description="执行代码"), +} +CHILDREN = { + 38: SimpleNamespace(id=38, type=16, name="联网搜索", tool_key="web_search", desc="联网检索"), + 6: SimpleNamespace(id=6, type=6, name="代码执行器", tool_key="bisheng_code_interpreter", desc="执行 Python"), +} + +STORED = [ + {"id": 16, "name": "联网搜索", "default_checked": True, "children": [{"id": 38, "tool_key": "web_search"}]}, + { + "id": 6, + "name": "代码执行器", + "default_checked": False, + "children": [{"id": 6, "tool_key": "bisheng_code_interpreter"}], + }, +] + + +def _patch_dao(monkeypatch: pytest.MonkeyPatch, visible: set[int] | None, types: dict = TYPES) -> None: + """DAO stand-ins that honour a visible-tenant IN-list unless pinned strict. + + All rows belong to Root, mirroring the incident: the request's IN-list did + not contain Root, so Root's own tools read back empty. + """ + + def _readable() -> bool: + return is_strict_tenant_filter() or visible is None or ROOT in visible + + monkeypatch.setattr( + GptsToolsDao, + "get_all_tool_type", + staticmethod(lambda type_ids: [types[t] for t in type_ids if t in types] if _readable() else []), + ) + monkeypatch.setattr( + GptsToolsDao, + "get_list_by_type", + staticmethod(lambda type_ids: [c for c in CHILDREN.values() if c.type in type_ids] if _readable() else []), + ) + + +def _group_ids(tools: list[dict]) -> list[int]: + return [t["id"] for t in tools] + + +def test_visible_set_containing_the_owner_resolves_normally(monkeypatch): + _patch_dao(monkeypatch, visible={ROOT}) + assert _group_ids(WorkStationService.sync_tool_info(STORED)) == [16, 6] + + +def test_visible_set_excluding_the_owner_no_longer_clears_the_pool(monkeypatch): + """The incident shape: current tenant is Root, the IN-list is not.""" + _patch_dao(monkeypatch, visible={OTHER}) + + synced = WorkStationService.sync_tool_info(STORED) + + assert _group_ids(synced) == [16, 6], "a filtered read must not look like a deleted tool pool" + assert [c["tool_key"] for t in synced for c in t["children"]] == [ + "web_search", + "bisheng_code_interpreter", + ] + + +def test_genuinely_deleted_group_is_still_dropped(monkeypatch): + """The strict re-read is a recovery path, not a resurrection path.""" + _patch_dao(monkeypatch, visible=None, types={16: TYPES[16]}) # tool type 6 really is gone + + assert _group_ids(WorkStationService.sync_tool_info(STORED)) == [16] + + +def test_everything_deleted_returns_empty(monkeypatch): + _patch_dao(monkeypatch, visible=None, types={}) + + assert WorkStationService.sync_tool_info(STORED) == [] + + +def test_empty_input_is_untouched(monkeypatch): + _patch_dao(monkeypatch, visible=None) + + assert WorkStationService.sync_tool_info([]) == [] diff --git a/src/frontend/client/eslint-suppressions.json b/src/frontend/client/eslint-suppressions.json index d5401cd71c..e1e0861e06 100644 --- a/src/frontend/client/eslint-suppressions.json +++ b/src/frontend/client/eslint-suppressions.json @@ -741,11 +741,6 @@ "count": 1 } }, - "src/components/Chat/Messages/DeepThinkingGroup.tsx": { - "no-restricted-syntax": { - "count": 6 - } - }, "src/components/Chat/Messages/HoverButtons.tsx": { "@typescript-eslint/no-explicit-any": { "count": 2 diff --git a/src/frontend/client/src/api/knowledge.test.ts b/src/frontend/client/src/api/knowledge.test.ts new file mode 100644 index 0000000000..0985fad438 --- /dev/null +++ b/src/frontend/client/src/api/knowledge.test.ts @@ -0,0 +1,59 @@ +import request from "~/api/request"; +import { + getDepartmentSpacesApi, + getJoinedSpacesApi, + getMineSpacesApi, +} from "~/api/knowledge"; + +jest.mock("~/api/request", () => ({ + __esModule: true, + default: { + get: jest.fn(), + }, +})); + +const mockedRequest = request as jest.Mocked; + +function spacePayload(spaceKind?: string) { + return { + status_code: 200, + status_message: "success", + data: [ + { + id: 7, + name: "space-7", + auth_type: "private", + space_kind: spaceKind, + }, + ], + }; +} + +describe("knowledge space list entry mapping", () => { + it("marks department entries without backend department metadata", async () => { + mockedRequest.get.mockResolvedValueOnce(spacePayload()); + + const spaces = await getDepartmentSpacesApi({ order_by: "update_time" }); + + expect(spaces).toHaveLength(1); + expect(spaces[0].spaceKind).toBe("department"); + expect(spaces[0].departmentId).toBeUndefined(); + expect(spaces[0].departmentName).toBeUndefined(); + expect(mockedRequest.get).toHaveBeenCalledWith( + "/api/v1/knowledge/space/department", + { params: { order_by: "update_time" } }, + ); + }); + + it("keeps mine and joined entries normal regardless of raw defaults", async () => { + mockedRequest.get + .mockResolvedValueOnce(spacePayload("department")) + .mockResolvedValueOnce(spacePayload()); + + const mine = await getMineSpacesApi(); + const joined = await getJoinedSpacesApi(); + + expect(mine[0].spaceKind).toBe("normal"); + expect(joined[0].spaceKind).toBe("normal"); + }); +}); diff --git a/src/frontend/client/src/api/knowledge.ts b/src/frontend/client/src/api/knowledge.ts index 72be8e5be8..d439029302 100644 --- a/src/frontend/client/src/api/knowledge.ts +++ b/src/frontend/client/src/api/knowledge.ts @@ -752,7 +752,10 @@ export async function getMineSpacesApi(params?: { order_by: params?.order_by, }, }); - return extractKnowledgeSpaceList(res).map(mapSpace); + return extractKnowledgeSpaceList(res).map((raw) => ({ + ...mapSpace(raw), + spaceKind: "normal", + })); } /** @@ -766,7 +769,10 @@ export async function getJoinedSpacesApi(params?: { order_by: params?.order_by, }, }); - return extractKnowledgeSpaceList(res).map(mapSpace); + return extractKnowledgeSpaceList(res).map((raw) => ({ + ...mapSpace(raw), + spaceKind: "normal", + })); } /** @@ -794,7 +800,10 @@ export async function getDepartmentSpacesApi(params?: { order_by: params?.order_by, }, }); - return extractKnowledgeSpaceList(res).map(mapSpace); + return extractKnowledgeSpaceList(res).map((raw) => ({ + ...mapSpace(raw), + spaceKind: "department", + })); } /** diff --git a/src/frontend/client/src/components/Chat/Messages/DeepThinkingGroup.tsx b/src/frontend/client/src/components/Chat/Messages/DeepThinkingGroup.tsx index 3145eec50c..773d48fb71 100644 --- a/src/frontend/client/src/components/Chat/Messages/DeepThinkingGroup.tsx +++ b/src/frontend/client/src/components/Chat/Messages/DeepThinkingGroup.tsx @@ -1,9 +1,19 @@ /** * DeepThinkingGroup — outer collapsible wrapper around a contiguous run of - * thinking + tool_call events. Header reads "已深度思考（用时 N 秒）" once - * the run is closed by a following text block (or stream end), or - * "正在深度思考（已用 N 秒）..." while still open. Collapsing the wrapper - * hides everything inside, including any inner ThinkingContent state. + * thinking + tool_call events. Header reads "已深度思考" once the run is closed + * by a following text block (or stream end), or "正在深度思考…" while still + * open. Collapsing the wrapper hides everything inside, including any inner + * ThinkingContent state. + * + * **No duration (2026-08-13).** The header used to end in "（用时 N 秒）", driven + * by a 100ms ticker. A live counter promises the number matters, and for model + * reasoning it does not — it only measures how long you have waited, with no + * denominator to reason against, at a precision that implies the work should + * have been quick. On a long run it made the product look broken rather than + * busy. Liveness is carried by the streaming glyph and the reasoning text. + * Removed here in lockstep with task mode's GroupHeaderLabel, which the two + * surfaces are deliberately isomorphic with — see Linsight/Execution/ + * GroupHeaderLabel.tsx for the full reasoning. */ import { Outlined } from "bisheng-icons"; import { @@ -17,7 +27,8 @@ import { type MouseEvent, } from "react"; import type { AgentEvent } from "~/api/chatApi"; -import { cn, formatSeconds } from "~/utils"; +import { useLocalize } from "~/hooks"; +import { cn } from "~/utils"; import ThinkingContent from "./ThinkingContent"; import ToolCallDisplay from "./ToolCallDisplay"; @@ -33,45 +44,9 @@ export interface DeepThinkingGroupProps { isStreaming: boolean; } -/** Pick the earliest started_at across events; fall back to undefined. */ -function groupStart(events: AgentEvent[]): number | undefined { - let earliest: number | undefined; - for (const ev of events) { - if (ev.type === "thinking" || ev.type === "tool_call") { - if (ev.started_at != null && (earliest == null || ev.started_at < earliest)) { - earliest = ev.started_at; - } - } - } - return earliest; -} - -/** Pick the latest ended_at across events. */ -function groupEnd(events: AgentEvent[]): number | undefined { - let latest: number | undefined; - for (const ev of events) { - if (ev.type === "thinking" || ev.type === "tool_call") { - if (ev.ended_at != null && (latest == null || ev.ended_at > latest)) { - latest = ev.ended_at; - } - } - } - return latest; -} - -/** Fallback: sum duration_ms when wall-clock fields aren't on legacy rows. */ -function durationFallback(events: AgentEvent[]): number { - let sum = 0; - for (const ev of events) { - if (ev.type === "thinking" || ev.type === "tool_call") { - sum += ev.duration_ms ?? 0; - } - } - return sum; -} - const DeepThinkingGroup: FC = memo( ({ events, isStreaming }) => { + const localize = useLocalize(); // Open while the run is live so the user can watch it; closed for // already-finished groups (history rows mount with isStreaming false). const [isExpanded, setIsExpanded] = useState(isStreaming); @@ -88,42 +63,13 @@ const DeepThinkingGroup: FC = memo( wasStreamingRef.current = isStreaming; }, [isStreaming]); - const start = groupStart(events); - const end = groupEnd(events); - - // Live-tick while streaming so the header counter advances every 100ms. - const [tick, setTick] = useState(0); - useEffect(() => { - if (!isStreaming) return; - const id = window.setInterval(() => setTick((t) => t + 1), 100); - return () => window.clearInterval(id); - }, [isStreaming]); - - const elapsedMs = (() => { - if (start == null) return durationFallback(events); - // For closed groups with no end_at, fall back to the per-event sum - // so the label doesn't creep upward against Date.now(). - if (!isStreaming && end == null) return durationFallback(events); - const stop = isStreaming ? Date.now() : end!; - return Math.max(0, stop - start); - })(); - // `tick` is read here so the IIFE re-runs on every interval render. - void tick; - - const label = (() => { - // Hide the duration entirely when it's 0 — happens on legacy - // history rows (no started_at/ended_at/duration_ms) and the - // brief moment before any tick lands. - const showDuration = elapsedMs > 0; - if (isStreaming) { - return showDuration - ? `正在深度思考（已用 ${formatSeconds(elapsedMs)} 秒）...` - : `正在深度思考...`; - } - return showDuration - ? `已深度思考（用时 ${formatSeconds(elapsedMs)} 秒）` - : `已深度思考`; - })(); + // Shared with task mode on purpose: the two 深度思考 surfaces are + // explicitly isomorphic, so one key pair keeps their wording from + // drifting apart. (The `_compact` name is historical — it was the + // duration-free variant back when a duration variant existed.) + const label = localize( + isStreaming ? "com_linsight_deep_thinking_running_compact" : "com_linsight_deep_thinking_done_compact", + ); const handleClick = useCallback((e: MouseEvent) => { e.preventDefault(); diff --git a/src/frontend/client/src/components/Linsight/Execution/BreathingRow.tsx b/src/frontend/client/src/components/Linsight/Execution/BreathingRow.tsx index 63935d1118..7248dc3afe 100644 --- a/src/frontend/client/src/components/Linsight/Execution/BreathingRow.tsx +++ b/src/frontend/client/src/components/Linsight/Execution/BreathingRow.tsx @@ -7,7 +7,8 @@ * carriers (ExecutionFlow full-page view + TaskTurnPanel chat-embedded view), * which keeps the live feedback identical across surfaces. * - * State → copy mapping (all keys pre-existing): + * State → copy mapping: + * - preparing → com_linsight_preparing ("Preparing the task") * - planning → com_linsight_planning ("Planning tasks") * - researching → com_linsight_executing ("Working") * - generating → com_linsight_generating ("Generating results, please wait…") @@ -15,10 +16,11 @@ import { Outlined } from 'bisheng-icons'; import { useLocalize } from '~/hooks'; -type BreathingState = 'planning' | 'researching' | 'generating'; +type BreathingState = 'preparing' | 'planning' | 'researching' | 'generating'; // state → localize key (single source of truth for the activity copy). const STATE_I18N: Record = { + preparing: 'com_linsight_preparing', planning: 'com_linsight_planning', researching: 'com_linsight_executing', generating: 'com_linsight_generating', diff --git a/src/frontend/client/src/components/Linsight/Execution/DeepStepGroup.tsx b/src/frontend/client/src/components/Linsight/Execution/DeepStepGroup.tsx index b143466613..fa0f6e94d0 100644 --- a/src/frontend/client/src/components/Linsight/Execution/DeepStepGroup.tsx +++ b/src/frontend/client/src/components/Linsight/Execution/DeepStepGroup.tsx @@ -40,7 +40,9 @@ import { ACCENT, ACTIVITY_I18N, BODY, INK } from './execTokens'; // the single Accent (blue) highlight; the chevron is muted and darkens on hover; // the title + narration sit lighter as quiet meta. const NODE_TEXT = '#999999'; +import { readIngestProgress } from './execTypes'; import { GroupHeaderLabel } from './GroupHeaderLabel'; +import { IngestPhaseRow } from './IngestPhaseRow'; import { KnowledgeRow } from './KnowledgeRow'; import { NarrationTicker } from './NarrationTicker'; import ToolRowLite from './ToolRowLite'; @@ -163,10 +165,8 @@ const DeepStepGroupBase: FC = ({ group, compact = false, sub return () => io.disconnect(); }, [open]); - // Timestamps on MergedStep are second-level ints (BaseEvent.timestamp); the - // ticker math is in milliseconds, so scale up here. - const startMs = group.startedAt != null ? group.startedAt * 1000 : null; - const endMs = group.endedAt != null ? group.endedAt * 1000 : null; + // (`group.startedAt` / `endedAt` are still built and still order the timeline; + // they simply no longer surface as a header duration — see GroupHeaderLabel.) // R1 (段流重构 2026-06): the segment header is the ACTIVITY SUMMARY of what // this episode did ("检索知识库 3 次 · 读 2 文件") — built from summarizeActivity, @@ -208,11 +208,15 @@ const DeepStepGroupBase: FC = ({ group, compact = false, sub

); } - // Knowledge steps keep their richer hit-list row; everything else is a - // lite tool row. Both preserve original timeline order. + // Knowledge steps keep their richer hit-list row; the attachment ingest is + // preparation, not an agent action, and gets its own counted phase row; + // everything else is a lite tool row. All preserve original timeline order. if (seg.step.stepType === 'knowledge') { return ; } + if (readIngestProgress(seg.step)) { + return ; + } return ; }); @@ -271,14 +275,7 @@ const DeepStepGroupBase: FC = ({ group, compact = false, sub running && 'animate-pulse group-hover:animate-none', )} > - + {/* Single chevron rotates right→down (collapsed → expanded), matching the StepRow / daily "深度思考" toggle; muted, darkening on hover. */} diff --git a/src/frontend/client/src/components/Linsight/Execution/ExecutionFlow.tsx b/src/frontend/client/src/components/Linsight/Execution/ExecutionFlow.tsx index 7b1746b8f7..e98d389a2c 100644 --- a/src/frontend/client/src/components/Linsight/Execution/ExecutionFlow.tsx +++ b/src/frontend/client/src/components/Linsight/Execution/ExecutionFlow.tsx @@ -109,10 +109,18 @@ export function ExecutionFlow({ versionId, conversationId, isSharePage = false, [sessionSteps.length, sessionSteps[sessionSteps.length - 1]?.status, sessionSteps[sessionSteps.length - 1]?.call_id], ); + // preparing row: the worker has claimed the session (IN_PROGRESS ⇒ running, + // queue position gone) but has emitted nothing yet, because the attachment + // ingest moved off the submit request and now runs here — minutes on a large + // batch. Same window, same copy as TaskTurnPanel: both carriers must give the + // identical live signal. Nothing is being planned during it, so the planning + // row must not claim it (the two are mutually exclusive by construction). + const preparing = running && noProgressYet && !queueing; // planning row: running, todo list not generated yet, nothing else pending, and // the session timeline has no content yet (deep-thinking hasn't started — else // it would render concurrently with "正在深度思考"). - const planning = running && !queueing && !pendingInput && !realTasks.length && !hasSessionTimeline; + const planning = + running && !queueing && !pendingInput && !realTasks.length && !hasSessionTimeline && !preparing; // generating row: todos exist but no task is actively streaming a spinner // right now — bridges the gaps before the first task, between tasks, AND the // final report-generation phase (status stays Running with no step events @@ -191,6 +199,11 @@ export function ExecutionFlow({ versionId, conversationId, isSharePage = false, inline IntentRow at its chronological position here. */} + {/* pre-output breathing row: the worker holds us but has + emitted nothing yet (deferred attachment ingest / run + startup) */} + {preparing && } + {/* planning breathing row */} {planning && } diff --git a/src/frontend/client/src/components/Linsight/Execution/ExecutionTimeline.tsx b/src/frontend/client/src/components/Linsight/Execution/ExecutionTimeline.tsx index 0a7daa8bf3..d3fb6c2df9 100644 --- a/src/frontend/client/src/components/Linsight/Execution/ExecutionTimeline.tsx +++ b/src/frontend/client/src/components/Linsight/Execution/ExecutionTimeline.tsx @@ -9,11 +9,13 @@ * * It does NOT import or touch any Chat/Messages (daily /c) component — the shared * visual tokens live in the task-mode primitives (CollapsibleTimelineItem / - * TimelineRail / useElapsedTicker) and the Wave2 group components below. + * TimelineRail) and the Wave2 group components below. */ import { useMemo } from 'react'; import { DeepStepGroup } from './DeepStepGroup'; import { useExecutionLive } from './executionLive'; +import { readIngestProgress } from './execTypes'; +import { IngestPhaseRow } from './IngestPhaseRow'; import { IntentRow } from './IntentRow'; import { KnowledgeRow } from './KnowledgeRow'; import { ToolRowLite } from './ToolRowLite'; @@ -37,8 +39,8 @@ export function ExecutionTimeline({ history }: ExecutionTimelineProps) { const nodes = useMemo(() => buildTimelineGroups(mergeStepFrames(history)), [sig]); // Container liveness (session turn / running task), provided by the carrier via // ExecutionLiveContext. The ACTIVE episode is the LAST node while the container - // is live — that one carries the live facets (正在 label, header pulse, ticking - // clock); every earlier node is done. (The fold itself opens collapsed for all + // is live — that one carries the live facets (正在 label, header pulse); every + // earlier node is done. (The fold itself opens collapsed for all // groups — see DeepStepGroup.) Passing this stable `active` down (instead of // letting each group read the volatile per-tool group.running) is what stops the // label/pulse from flickering on every tool call within one episode. @@ -101,6 +103,9 @@ export function ExecutionTimeline({ history }: ExecutionTimelineProps) { if (step.stepType === 'knowledge') { return ; } + if (readIngestProgress(step)) { + return ; + } return ; })} diff --git a/src/frontend/client/src/components/Linsight/Execution/GroupHeaderLabel.tsx b/src/frontend/client/src/components/Linsight/Execution/GroupHeaderLabel.tsx index 16b7880bea..cf36e06b82 100644 --- a/src/frontend/client/src/components/Linsight/Execution/GroupHeaderLabel.tsx +++ b/src/frontend/client/src/components/Linsight/Execution/GroupHeaderLabel.tsx @@ -1,24 +1,27 @@ /** - * GroupHeaderLabel — the live "用时 N 秒" header text for a DeepStepGroup, split - * out as its OWN component so the 100ms elapsed ticker re-renders ONLY this label, - * never the group body (thinking passages + tool rows). + * GroupHeaderLabel — the header text for a DeepStepGroup. * - * Why the split (perf): useElapsedTicker fires a 100ms setInterval while the group - * is the live tail. When the ticker lived in DeepStepGroup, every tick re-rendered - * the whole group — its thinking

blocks and every ToolRowLite — ten times a - * second, on top of the per-WS-frame timeline rebuild. That main-thread pressure - * starved the timer callback and made the counter advance unevenly / skip seconds. - * Isolating the ticker here means a tick touches only this one-line label; the - * group body re-renders solely when its steps actually change (a real WS frame). + * **No duration (2026-08-13).** This used to end in "（用时 N 秒）", driven by a + * 100ms ticker. It was removed on purpose, and the reasoning is worth keeping: * - * The label math is unchanged from the old in-group useMemo (whole-second format, - * the subagent-goal / activity-summary / pure-reasoning branches, the noDuration - * gate), so the rendered text — and the DeepStepGroup label tests — are identical. + * A live counter is a promise that the number matters. For agent reasoning it + * does not — nobody decides anything on "1416". What it does do is measure how + * long you have been waiting, with no denominator to reason against, at a + * precision (seconds) that implies the operation should have been quick. On a + * 20-minute run the header read "读取 6 个文件 · 执行 1 步操作（用时 1416 秒）" + * and the product looked broken rather than busy. Liveness is carried by the + * narration line and the running glyph instead, which say what is happening + * rather than how long it has hurt. + * + * Dropping it also removed the component's original reason to exist: it was split + * out of DeepStepGroup so a 100ms setInterval would re-render one line instead of + * the whole group (thinking passages + every tool row) ten times a second. That + * timer is gone, so the surface displaying "how slow this is" is no longer itself + * a source of slowness. The split is kept because the branch logic below is worth + * isolating and testing on its own. */ import { useLocalize } from '~/hooks'; -import { formatSeconds } from '~/utils'; import { firstLine } from './stepUtils'; -import { useElapsedTicker } from './useElapsedTicker'; /** * Subagent header budget: the delegation goal is the `task` tool's `description` @@ -33,50 +36,30 @@ export interface GroupHeaderLabelProps { activityText: string; /** Subagent context when this group is an exploded subagent segment. */ subagent?: { goal: string; idx: number }; - /** Drilldown (inside a subagent card): drop the "（用时 N 秒）" clause. */ - compact: boolean; - /** Group clock start/end in ms (null ⇒ no clock; caller scaled second→ms). */ - startMs: number | null; - endMs: number | null; - /** True while this group is the live tail episode (drives 正在/已 + the ticker). */ + /** True while this group is the live tail episode (drives 正在/已). */ running: boolean; } -export function GroupHeaderLabel({ activityText, subagent, compact, startMs, endMs, running }: GroupHeaderLabelProps) { +export function GroupHeaderLabel({ activityText, subagent, running }: GroupHeaderLabelProps) { const localize = useLocalize(); - // Owns the 100ms live ticker. setTick re-renders THIS component only, so a - // running group's "用时" advances without re-rendering the group body. - const { elapsedMs } = useElapsedTicker(startMs, endMs, running); - // No useMemo: elapsedMs advances on every 100ms tick (and reads Date.now() - // fresh each render), so a memo keyed on it would never hit — the label is - // recomputed every render regardless. Inline string-building is cheaper. - const seconds = formatSeconds(elapsedMs); - // Drop the duration clause when nested (compact) OR when the measured span is 0 - // (a single second-level frame would read a misleading "用时 0 秒"). - const noDuration = compact || elapsedMs <= 0; let label: string; if (subagent) { - // R3 完全拆平: a subagent segment is headed by its delegation GOAL + 用时. - // The goal is the subagent's identity, so it OWNS the header line — show - // only its GIST (firstLine), falling back to the activity summary and - // finally the "子智能体 N" label for a goal-less (degraded) subagent. + // R3 完全拆平: a subagent segment is headed by its delegation GOAL. The goal + // is the subagent's identity, so it OWNS the header line — show only its + // GIST (firstLine), falling back to the activity summary and finally the + // "子智能体 N" label for a goal-less (degraded) subagent. const goalGist = firstLine(subagent.goal, SUBAGENT_GOAL_TITLE_MAX); - const core = goalGist || activityText || localize('com_linsight_subagent_track', { 0: String(subagent.idx) }); - label = noDuration ? core : localize('com_linsight_act_summary', { 0: core, 1: seconds }); + label = goalGist || activityText || localize('com_linsight_subagent_track', { 0: String(subagent.idx) }); } else if (activityText) { // Activity-summary header (verbs + counts), the primary case. - label = noDuration ? activityText : localize('com_linsight_act_summary', { 0: activityText, 1: seconds }); - } else if (noDuration) { - // Pure-reasoning fallback (no measurable span): the compact 深度思考 label. - label = localize( - running ? 'com_linsight_deep_thinking_running_compact' : 'com_linsight_deep_thinking_done_compact', - ); + label = activityText; } else { - // Pure-reasoning fallback with a duration. + // Pure-reasoning fallback: the compact 深度思考 label. The `_compact` + // variants are the duration-free wording, which is now the only wording — + // the "（用时 N 秒）" pair they were the counterpart to no longer renders. label = localize( - running ? 'com_linsight_deep_thinking_running' : 'com_linsight_deep_thinking_done', - { 0: seconds }, + running ? 'com_linsight_deep_thinking_running_compact' : 'com_linsight_deep_thinking_done_compact', ); } diff --git a/src/frontend/client/src/components/Linsight/Execution/IngestPhaseRow.tsx b/src/frontend/client/src/components/Linsight/Execution/IngestPhaseRow.tsx new file mode 100644 index 0000000000..7d72840997 --- /dev/null +++ b/src/frontend/client/src/components/Linsight/Execution/IngestPhaseRow.tsx @@ -0,0 +1,78 @@ +/** + * IngestPhaseRow — the attachment-ingest phase, rendered as PREPARATION rather + * than as an agent tool call. + * + * Deferring the ingest into the worker moved a multi-minute parse (12 bid PDFs + * measured at ~20 min) into the visible timeline. It first shipped through the + * generic tool row, which was wrong three ways at once: + * + * 1. **Vocabulary.** "已使用 ingest_uploads" names an internal function. The user + * uploaded attachments; that is what the row must talk about. + * 2. **Attribution.** As a tool row it counted into the group's activity tally, + * so the system's parse time was billed to the agent's thinking — the header + * read "执行 1 步操作（用时 1416 秒）" and the product looked catastrophically + * slow when it was simply parsing a lot of paper. (Fixed in activity.ts.) + * 3. **Silence.** `done/total` and the current filename were already on the wire + * in `extra_info.ingest_progress` and nothing rendered them, so twenty minutes + * passed with a static line — indistinguishable from a hang. + * + * So this row leads with the COUNT (the only phase of a run with a true a-priori + * denominator) and carries the current filename underneath. No duration: a + * stopwatch measures how long you have waited, never how much is left, and the + * whole surface dropped elapsed-time headers for that reason. + */ +import { Outlined } from 'bisheng-icons'; +import type { FC } from 'react'; +import { useLocalize } from '~/hooks'; +import { ACCENT, MUTED } from './execTokens'; +import { INGEST_PHASE_I18N, readIngestProgress } from './execTypes'; +import { NarrationTicker } from './NarrationTicker'; +import type { MergedStep } from './stepUtils'; + +export interface IngestPhaseRowProps { + step: MergedStep; +} + +export const IngestPhaseRow: FC = ({ step }) => { + const localize = useLocalize(); + const progress = readIngestProgress(step); + // Defensive: callers gate on the same predicate, but an older worker emits the + // step name with no payload — falling through to nothing beats rendering a row + // that says "undefined/undefined". + if (!progress) return null; + + const running = progress.phase === 'running'; + const title = localize(INGEST_PHASE_I18N[progress.phase], { + 0: String(progress.done), + 1: String(progress.total), + }); + + return ( +

+ + {running ? ( + + ) : ( + // A file glyph, not the generic tool wrench: the icon is the + // fastest signal that this row is about the user's uploads. + + )} + +
+ + {title} + + {/* The moving part. NarrationTicker owns the one-line-at-a-time + crossfade + reduced-motion handling; reusing it keeps this line + behaving exactly like the reasoning aside above it. Failure text + rides `output` and takes the same slot, since a failed ingest + ends the run and the reason has to be readable inline. */} + +
+
+ ); +}; + +IngestPhaseRow.displayName = 'IngestPhaseRow'; + +export default IngestPhaseRow; diff --git a/src/frontend/client/src/components/Linsight/Execution/TaskTurnPanel.tsx b/src/frontend/client/src/components/Linsight/Execution/TaskTurnPanel.tsx index c69c9a068b..bafde4c76b 100644 --- a/src/frontend/client/src/components/Linsight/Execution/TaskTurnPanel.tsx +++ b/src/frontend/client/src/components/Linsight/Execution/TaskTurnPanel.tsx @@ -138,7 +138,14 @@ export function TaskTurnPanel({ versionId, liked, allowFeedback = true, conversa [sessionSteps.length, sessionSteps[sessionSteps.length - 1]?.status, sessionSteps[sessionSteps.length - 1]?.call_id], ); - const planning = running && !queueing && !tasks.length && !pendingInput && !hasSessionTimeline; + // Attachment ingest was moved off the submit request into the worker, which + // claims the session (IN_PROGRESS ⇒ running) BEFORE parsing uploads/*. For a + // large batch that is minutes with no task, no step and no queue position — + // the panel had nothing to render and the run looked hung. Own that window + // explicitly: nothing is being planned yet, so the planning row must not + // claim it (the two are mutually exclusive by construction). + const preparing = running && noProgressYet && !queueing; + const planning = running && !queueing && !tasks.length && !pendingInput && !hasSessionTimeline && !preparing; const handleClarifySubmit = (taskId: string, ans: string) => { sendInput({ task_id: taskId || versionId, user_input: ans, files: [] }); @@ -202,6 +209,10 @@ export function TaskTurnPanel({ versionId, liked, allowFeedback = true, conversa (时序内联) instead of being hoisted above the timeline. */} + {/* pre-output breathing row: the worker holds us but has emitted + nothing yet (deferred attachment ingest / run startup) */} + {preparing && } + {/* planning breathing row */} {planning && } diff --git a/src/frontend/client/src/components/Linsight/Execution/ToolRow.tsx b/src/frontend/client/src/components/Linsight/Execution/ToolRow.tsx deleted file mode 100644 index 7143d4aaa4..0000000000 --- a/src/frontend/client/src/components/Linsight/Execution/ToolRow.tsx +++ /dev/null @@ -1,34 +0,0 @@ -/** - * F035 Track H (P3): tool-call step row (step_type=tool, spec §3). - * Header: icon + tool name; expanded: call reason + input params + output. - */ -import { useLocalize } from '~/hooks'; -import { cn } from '~/utils'; -import { detailTextCls, formatStepParams, RunningSpinner, StepRow, stepTypeIcon } from './StepRow'; -import type { MergedStep } from './stepUtils'; - -export function ToolRow({ step }: { step: MergedStep }) { - const localize = useLocalize(); - const paramsText = formatStepParams(step.params); - return ( - : stepTypeIcon(step.name)} - title={step.name} - running={step.running} - > - {step.callReason &&

{step.callReason}

} - {paramsText && ( -
-

{localize('com_linsight_step_input')}

-

{paramsText}

-
- )} - {step.output && ( -
-

{localize('com_linsight_step_output')}

-

{step.output}

-
- )} - - ); -} diff --git a/src/frontend/client/src/components/Linsight/Execution/activity.ts b/src/frontend/client/src/components/Linsight/Execution/activity.ts index b9afdde1db..5c19f641a4 100644 --- a/src/frontend/client/src/components/Linsight/Execution/activity.ts +++ b/src/frontend/client/src/components/Linsight/Execution/activity.ts @@ -2,6 +2,7 @@ * Activity summary: classify a group's tool calls into readable buckets. * Split out of stepUtils.ts. */ +import { INGEST_STEP_NAME } from './execTypes'; import type { MergedStep } from './execTypes'; /** @@ -38,6 +39,12 @@ function classifyActivity(name: string): ActivityCategory | null { if (!n) return 'other'; // never-count noise (defensive — callers already drop these) if (n === 'thinking' || n === 'ls' || n === 'write_todos' || n === 'ask_user') return null; + // Attachment ingest is the SYSTEM preparing the user's uploads, not the agent + // acting. Counting it charged the agent for the whole parse — a 12-PDF batch + // turned the header into "读取 6 个文件 · 执行 1 步操作（用时 1416 秒）", i.e. + // 20 minutes of file parsing billed to the model's thinking. It renders as its + // own phase row (IngestPhaseRow) and is deliberately absent from the tally. + if (n === INGEST_STEP_NAME) return null; // knowledge before web_search: search_knowledge_base must not match web_search if (n.includes('knowledge') || n.includes('search_knowledge')) return 'knowledge'; if (n.includes('web_search') || n.includes('search')) return 'web_search'; diff --git a/src/frontend/client/src/components/Linsight/Execution/execTypes.ts b/src/frontend/client/src/components/Linsight/Execution/execTypes.ts index e4949d929f..3c31232f05 100644 --- a/src/frontend/client/src/components/Linsight/Execution/execTypes.ts +++ b/src/frontend/client/src/components/Linsight/Execution/execTypes.ts @@ -21,6 +21,57 @@ export type ExecStepType = 'tool' | 'thinking' | 'knowledge' | 'subagent' | 'ui_card' | 'call_user_input'; +/** + * Attachment ingest moved off the submit request into the worker, so the parse + * now happens while the task is already claimed. The worker reports it as an + * ordinary tool row carrying only DATA — the wording lives here, because a + * backend-formatted label would show Chinese to a Japanese user and would stay + * wrong in persisted history after a language switch. + * + * ⚠️ Contract with the backend: this name is `_INGEST_STEP_NAME` in + * bisheng/linsight/domain/task_exec.py. Change one side alone and the row falls + * back to rendering the raw token. + */ +export const INGEST_STEP_NAME = 'ingest_uploads'; + +export type IngestPhase = 'running' | 'done' | 'failed' | 'aborted'; + +export interface IngestProgress { + phase: IngestPhase; + done: number; + total: number; + fileName?: string; +} + +/** phase → i18n key, same shape as ACTIVITY_I18N in execTokens. */ +export const INGEST_PHASE_I18N: Record = { + running: 'com_linsight_ingest_running', + done: 'com_linsight_ingest_done', + failed: 'com_linsight_ingest_failed', + aborted: 'com_linsight_ingest_aborted', +}; + +const INGEST_PHASES = Object.keys(INGEST_PHASE_I18N) as IngestPhase[]; + +/** Pull the ingest payload off a step, or null when it is not an ingest row. + Defensive about every field: this crosses a process boundary, and an older + worker emits nothing here at all. */ +export function readIngestProgress(step: { + name?: string; + extraInfo?: Record; +}): IngestProgress | null { + if (step.name !== INGEST_STEP_NAME) return null; + const raw = step.extraInfo?.ingest_progress; + if (!raw || typeof raw !== 'object') return null; + const payload = raw as Record; + return { + phase: INGEST_PHASES.find((p) => p === payload.phase) ?? 'running', + done: Number(payload.done) || 0, + total: Number(payload.total) || 0, + fileName: typeof payload.file_name === 'string' ? payload.file_name : undefined, + }; +} + /** Raw `task_execute_step.data` frame (contract C1). */ export interface ExecStepEventData { call_id?: string; diff --git a/src/frontend/client/src/components/Linsight/Execution/useElapsedTicker.ts b/src/frontend/client/src/components/Linsight/Execution/useElapsedTicker.ts deleted file mode 100644 index fb7cfb5a76..0000000000 --- a/src/frontend/client/src/components/Linsight/Execution/useElapsedTicker.ts +++ /dev/null @@ -1,63 +0,0 @@ -/** - * useElapsedTicker — group-level elapsed-time hook for task-mode timeline groups. - * - * Extracted from the daily /c DeepThinkingGroup timer so task-mode groups tick - * with the *same* cadence and label math (zero drift from the North Star). While - * `streaming` is true a 100ms interval re-renders the consumer so a live "用时 N - * 秒" counter advances; once streaming stops the interval is cleared and the - * elapsed value freezes at `endMs - startMs`. - * - * Wall-clock contract (mirrors DeepThinkingGroup): - * - `startMs == null` → no clock; elapsedMs = 0 (caller hides 用时). - * - streaming → elapsedMs = now - startMs (live). - * - closed with `endMs != null` → elapsedMs = endMs - startMs (frozen). - * - closed with `endMs == null` → elapsedMs = 0 (no end stamp; avoid creeping - * the label upward against Date.now()). - * - * `Date.now()` is only ever read while the component is mounted and rendering, - * which is valid at runtime (the surrounding workflow-script restriction does - * not apply to component code). - */ -import { useEffect, useState } from 'react'; - -export interface ElapsedTicker { - /** Elapsed milliseconds for the group (0 when unknown — caller hides 用时). */ - elapsedMs: number; - /** Mirrors `streaming`; convenient for "正在…" vs "已…" label branching. */ - running: boolean; -} - -/** - * @param startMs group start wall-clock (ms); null/undefined ⇒ no clock. - * @param endMs group end wall-clock (ms); read only when not streaming. - * @param streaming true while the group is still open/active. - */ -export function useElapsedTicker( - startMs: number | null | undefined, - endMs: number | null | undefined, - streaming: boolean, -): ElapsedTicker { - // Live-tick while streaming so the consumer re-renders every 100ms and the - // header counter advances. The tick value itself is unused — it exists only - // to trigger re-render; the elapsed math below reads Date.now() fresh. - const [, setTick] = useState(0); - useEffect(() => { - if (!streaming) return; - const id = window.setInterval(() => setTick((t) => t + 1), 100); - return () => window.clearInterval(id); - }, [streaming]); - - let elapsedMs = 0; - if (startMs != null) { - if (streaming) { - elapsedMs = Math.max(0, Date.now() - startMs); - } else if (endMs != null) { - // Closed group with a real end stamp: freeze at the measured span. - elapsedMs = Math.max(0, endMs - startMs); - } - // Closed with no end stamp ⇒ leave elapsedMs at 0 so the label hides the - // 用时 clause rather than creeping upward against Date.now(). - } - - return { elapsedMs, running: streaming }; -} diff --git a/src/frontend/client/src/components/permission/PermissionGrantTab.test.tsx b/src/frontend/client/src/components/permission/PermissionGrantTab.test.tsx index ac252eb6a3..84d8a744cd 100644 --- a/src/frontend/client/src/components/permission/PermissionGrantTab.test.tsx +++ b/src/frontend/client/src/components/permission/PermissionGrantTab.test.tsx @@ -207,13 +207,13 @@ describe("F048 Client PermissionGrantTab", () => { changes: [ { op: "MOVE", - assignee_id: 1, + assignee_id: "1", expected_assignee_version: 2, target_model_key: "editor", }, { op: "REMOVE", - assignee_id: 2, + assignee_id: "2", expected_assignee_version: 2, }, { @@ -258,6 +258,54 @@ describe("F048 Client PermissionGrantTab", () => { ).toBeDisabled(); }); + it("keeps an inactive existing row but excludes it from ADD and MOVE targets", async () => { + render( + , + ); + + const rowModel = await screen.findByLabelText( + "f048_permission.grant.model.8", + ); + expect(rowModel).toHaveValue("inactive"); + expect(rowModel).toHaveTextContent("Inactive"); + expect( + screen.getByLabelText("f048_permission.grant.add_model"), + ).not.toHaveTextContent("Inactive"); + + fireEvent.change(rowModel, { target: { value: "editor" } }); + fireEvent.click( + screen.getByRole("button", { name: "f048_permission.grant.submit" }), + ); + await waitFor(() => expect(mockedMutate).toHaveBeenCalledTimes(1)); + expect(mockedMutate.mock.calls[0][2]).toEqual( + expect.objectContaining({ + changes: [ + { + op: "MOVE", + assignee_id: "8", + expected_assignee_version: 2, + target_model_key: "editor", + }, + ], + }), + ); + }); + it("locks a subject who already holds a grant, whatever model is selected", async () => { // Granting the same person under a second model put two rows in the roster, // of which only the higher one means anything. diff --git a/src/frontend/client/src/components/permission/PermissionGrantTab.tsx b/src/frontend/client/src/components/permission/PermissionGrantTab.tsx index 3f4e17c76a..d824ba7b29 100644 --- a/src/frontend/client/src/components/permission/PermissionGrantTab.tsx +++ b/src/frontend/client/src/components/permission/PermissionGrantTab.tsx @@ -405,6 +405,9 @@ export function PermissionGrantTab({ {assignees.map((assignee) => { const editable = isEditable(assignee, context); const removed = removedIds.has(assignee.assignee_id); + const currentIsGrantable = models.some( + (model) => model.key === assignee.model.key, + ); return (
+ {!currentIsGrantable && ( + + )} {models.map((model) => (
)} {!isStandard && !createMode && ( - - - - {/* A disabled button swallows pointer events, so the tooltip - needs a wrapper to hear the hover. */} - - - - - {active && ( - {t("model.disableBeforeDelete")} - )} - - +
+ + + {deleteBlocked + ? t("model.deleteBlocked") + : t("model.deleteRequirement")} + +
)} @@ -339,12 +345,9 @@ describe("F048 RolesAndPermissions", () => { }) it("falls back to the plain failure when the server sends no reason", async () => { - vi.mocked(publishPermissionCatalogDraftApi).mockRejectedValueOnce( + vi.mocked(createPermissionCatalogDraftApi).mockRejectedValueOnce( "some other failure", ) - vi.mocked(createPermissionCatalogDraftApi).mockResolvedValueOnce({ - draft_id: 31, - } as never) renderWithUser({ role: "admin", is_global_super: true }) await waitFor(() => expect(getPermissionCatalogApi).toHaveBeenCalled())