Skip to content

Commit 4b1d635

Browse files
Add read_agent_trace fleet verb (orchestrator tiers only) (#599)
* Add read_agent_trace fleet verb for orchestrator tiers Lets an orchestrator or nested orchestrator read a worker's on-disk turns.jsonl directly, so a cancelled or interrupted worker's completed work is no longer invisible once its in-memory session record is gone. Every response is bounded (turn window, entry count, per-entry chars), tolerates partially written/malformed lines, and reports a clean error for an unknown target. Fixes a latest-symlink double-count in directory enumeration by resolving symlinks and de-duping by real path. progress_note for leaf workers is a separate follow-up, not included here. * Fix cross-subtree trace leak and add an aggregate output cap Review on #599 found two real issues: - findAgentTraceDir walked the whole (flat, shared) subagents/ tree from the root workdirBase, so a Tier-2 nested orchestrator could read any worker's trace, not just its own descendants. Wired assertCanTargetAgent (authority.ts's first live call site) using the worker's own SubAgentSessionStore id and the store's existing parentSessionId chain, rather than reshaping the on-disk layout — the disk tree is deliberately flat across the whole fleet (shared by worktrees and intervention logs too), so a structural per-subtree root would be a much larger change. To make that id available, run.ts now names a worker's trace directory after its session-store id when one is supplied (task-tool.ts passes it), instead of always minting a fresh disk-only id. - The per-entry, entry-count, and turn-window caps multiply (500 * 4,000 = 2,000,000 chars). Added a total-output character cap that stops filling entries once reached and reports the remainder via the existing `omitted` block. Noted but not changed: readAllTurns loads each full segment before bounds apply. Segments are already bounded to ~256KB by the writer, so this isn't unbounded, but avoiding the read entirely needs a line-count index or a streaming reader — left as a follow-up rather than expanding this fix.
1 parent 8af97ac commit 4b1d635

10 files changed

Lines changed: 1007 additions & 5 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,22 @@ parallel copies under `docs/` or `scripts/notes/`. At cut time: rename
3939
own), so the two layers no longer multiply. Attempt counts are now logged
4040
on each recovery so retry storms are visible in traces.
4141

42+
- **`read_agent_trace` lets an orchestrator inspect a worker's on-disk trace
43+
directly**, so a cancelled or interrupted worker's completed work is no
44+
longer invisible just because its in-memory session record is gone. Reads
45+
turns, tool calls, and tool errors straight from the worker's
46+
`turns.jsonl`, tolerating a partially written or malformed line without
47+
failing. Every response is bounded on four independent axes — turn window,
48+
entry count, per-entry characters, and total output characters (the first
49+
three multiply, so a total-output ceiling caps them together) — each with
50+
a hard maximum the caller cannot exceed, and a truncated response says
51+
exactly what was left out and how to page for the rest. A Tier 2 nested
52+
orchestrator can only read its own descendants' traces, enforced by
53+
reusing `SubAgentSessionStore`'s existing parentSessionId chain
54+
(`assertCanTargetAgent`'s first live call site); leaf directors never see
55+
the tool at all. `progress_note` for leaf workers is a separate,
56+
not-yet-implemented follow-up.
57+
4258
### Fixed
4359

4460
- **Interrupting a turn no longer risks a startup crash.** If an interrupt hit

‎src/agent/tools.ts‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ import { createWebSearchTool, disposeWebSearchClients } from "../tools/web-searc
4646
import { createUseSkillTool } from "./use-skill.js";
4747
import { createToolIndex, createToolSearchTool } from "./tool-search.js";
4848
import { createSearchAgentsTool } from "./agent-search.js";
49+
import { createReadAgentTraceTool } from "../subagent/trace-tool.js";
4950
import {
5051
createCodexToolProxies,
5152
type CodexRunManageTasks,
@@ -309,6 +310,11 @@ export async function createAgentToolset(args: AgentToolsetArgs): Promise<AgentT
309310
}),
310311
]
311312
: []),
313+
// Tier 1: the primary session is always an orchestrator and may
314+
// target any worker (assertCanTargetAgent's rule), so no authority
315+
// context is passed here — omitting it is treated as unrestricted,
316+
// matching Tier 1's actual authority.
317+
createReadAgentTraceTool(args.subAgent.getWorkdirBase),
312318
]
313319
: []),
314320
stringTool({

‎src/subagent/authority.ts‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,10 @@
55
* in a prompt. This module owns two checks:
66
*
77
* - assertTierMayMountFleetVerb: a Tier 3 leaf may never mount a fleet verb
8-
* (today: task, search_agents; the spawn_agent/wait_agents/list_agents/
9-
* send_input/interrupt_agent/close_agent/resume_agent/read_agent_trace
10-
* verbs land in later child issues against this same gate).
8+
* (today: task, search_agents, read_agent_trace; the spawn_agent/
9+
* wait_agents/list_agents/send_input/interrupt_agent/close_agent/
10+
* resume_agent/followup_task verbs land in later child issues against
11+
* this same gate).
1112
* - assertCanTargetAgent: a Tier 2 nested orchestrator may act only on its
1213
* own descendants, never a sibling or anything above it in the tree.
1314
* Tier 1 (the primary orchestrator) may target anyone. Callers pass the

‎src/subagent/run.ts‎

Lines changed: 23 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,7 @@ import {
105105
} from "./stop-policy.js";
106106
import { SubAgentDirector } from "./nudge-director.js";
107107
import { assertTierMayMountFleetVerb } from "./authority.js";
108+
import { createReadAgentTraceTool } from "./trace-tool.js";
108109
import {
109110
abortError,
110111
createSubAgentSpawnRegistryPlugin,
@@ -435,7 +436,7 @@ export async function runSubAgent(params: RunSubAgentParams): Promise<string> {
435436
// AgentProfile with orchestrator: true from mounting task/search_agents
436437
// just because it is outside the closed director set.
437438
const tier = params.orchestratorTier ?? "leaf";
438-
for (const verb of ["task", "search_agents"]) {
439+
for (const verb of ["task", "search_agents", "read_agent_trace"]) {
439440
assertTierMayMountFleetVerb(tier, verb);
440441
}
441442
if (params.nestedDispatch === undefined) {
@@ -481,6 +482,19 @@ export async function runSubAgent(params: RunSubAgentParams): Promise<string> {
481482
}),
482483
]
483484
: []),
485+
// Every worker at every nesting depth is created under the same root
486+
// workdirBase (nestedDispatch.getWorkdirBase is threaded through
487+
// unchanged, never rebound to this worker's own dir), so the trace
488+
// reader's search root is that same function. Descendant-only
489+
// scoping is enforced inside the tool via assertCanTargetAgent,
490+
// reusing the fleet nodes SubAgentSessionStore already tracks and
491+
// this worker's own store id (params.id) — not the disk layout,
492+
// which is intentionally flat across the whole fleet.
493+
createReadAgentTraceTool(nd.getWorkdirBase, {
494+
actorId: params.id,
495+
tier,
496+
getNodes: () => nd.sessions?.list() ?? [],
497+
}),
484498
];
485499
}
486500

@@ -578,7 +592,14 @@ export async function runSubAgent(params: RunSubAgentParams): Promise<string> {
578592
},
579593
});
580594

581-
const workdir = join(params.workdirBase, "subagents", generateSessionId());
595+
// Reuse the caller's session-store id as the on-disk directory name
596+
// when it is safe as a path segment, so read_agent_trace's descendant
597+
// check can walk the same parentSessionId chain SubAgentSessionStore
598+
// already tracks instead of needing a second, disk-only identity
599+
// scheme.
600+
const safeRequestedId =
601+
params.id !== undefined && /^[A-Za-z0-9_-]+$/.test(params.id) ? params.id : undefined;
602+
const workdir = join(params.workdirBase, "subagents", safeRequestedId ?? generateSessionId());
582603
await mkdir(workdir, { recursive: true });
583604
// One record per stop/nudge, with its measured value beside its threshold,
584605
// written into this leaf's own trace dir (CL-6938).

‎src/subagent/task-tool.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -804,6 +804,10 @@ export function createTaskTool(deps: TaskToolDeps): AgentTool {
804804
...sandbox,
805805
cwd: worktreeCwd ?? deps.cwd,
806806
workdirBase: deps.getWorkdirBase(),
807+
// Same id as the SubAgentSessionStore record so read_agent_trace's
808+
// descendant check (authority.ts assertCanTargetAgent) can reuse the
809+
// store's parentSessionId chain instead of a second identity scheme.
810+
...(session !== undefined ? { id: session.id } : {}),
807811
provider,
808812
...(settings !== undefined ? { settings } : {}),
809813
...(catalog !== undefined ? { catalog } : {}),

‎src/subagent/trace-reader.test.ts‎

Lines changed: 250 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,250 @@
1+
import { describe, test, expect } from "bun:test";
2+
import fs from "node:fs";
3+
import os from "node:os";
4+
import path from "node:path";
5+
6+
import {
7+
AgentTraceNotFoundError,
8+
findAgentTraceDir,
9+
listUniqueSubdirs,
10+
readAgentTrace,
11+
MAX_TRACE_ENTRY_LIMIT,
12+
MAX_TRACE_TOTAL_CHARS,
13+
MAX_TRACE_TURN_WINDOW,
14+
} from "./trace-reader.js";
15+
16+
function tempDir(): string {
17+
return fs.mkdtempSync(path.join(os.tmpdir(), "trace-reader-"));
18+
}
19+
20+
function writeTurns(dir: string, turns: unknown[]): void {
21+
fs.mkdirSync(dir, { recursive: true });
22+
const text = turns.map((t) => JSON.stringify(t)).join("\n") + (turns.length > 0 ? "\n" : "");
23+
fs.writeFileSync(path.join(dir, "turns.jsonl"), text);
24+
}
25+
26+
describe("listUniqueSubdirs", () => {
27+
test("a directory containing latest plus its target enumerates the session exactly once", async () => {
28+
const root = tempDir();
29+
const real = path.join(root, "01234567-89ab-7def-8123-456789abcdef");
30+
fs.mkdirSync(real);
31+
fs.symlinkSync(path.basename(real), path.join(root, "latest"));
32+
33+
const entries = await listUniqueSubdirs(root);
34+
expect(entries).toHaveLength(1);
35+
expect(entries[0]!.path).toBe(fs.realpathSync(real));
36+
});
37+
38+
test("two distinct real directories are both listed", async () => {
39+
const root = tempDir();
40+
fs.mkdirSync(path.join(root, "a"));
41+
fs.mkdirSync(path.join(root, "b"));
42+
const entries = await listUniqueSubdirs(root);
43+
expect(entries).toHaveLength(2);
44+
});
45+
46+
test("a broken symlink is skipped, not thrown", async () => {
47+
const root = tempDir();
48+
fs.symlinkSync(path.join(root, "does-not-exist"), path.join(root, "dangling"));
49+
const entries = await listUniqueSubdirs(root);
50+
expect(entries).toHaveLength(0);
51+
});
52+
53+
test("missing directory returns empty rather than throwing", async () => {
54+
const entries = await listUniqueSubdirs(path.join(tempDir(), "nope"));
55+
expect(entries).toHaveLength(0);
56+
});
57+
});
58+
59+
describe("findAgentTraceDir", () => {
60+
test("finds a direct child under root/subagents", async () => {
61+
const root = tempDir();
62+
const childDir = path.join(root, "subagents", "child-1");
63+
writeTurns(childDir, []);
64+
const found = await findAgentTraceDir(root, "child-1");
65+
expect(found).toBe(fs.realpathSync(childDir));
66+
});
67+
68+
test("finds a nested descendant several levels deep", async () => {
69+
const root = tempDir();
70+
const grandchildDir = path.join(root, "subagents", "child-1", "subagents", "grandchild-1");
71+
writeTurns(grandchildDir, []);
72+
const found = await findAgentTraceDir(root, "grandchild-1");
73+
expect(found).toBe(fs.realpathSync(grandchildDir));
74+
});
75+
76+
test("returns null for an unknown id", async () => {
77+
const root = tempDir();
78+
writeTurns(path.join(root, "subagents", "child-1"), []);
79+
const found = await findAgentTraceDir(root, "does-not-exist");
80+
expect(found).toBeNull();
81+
});
82+
83+
test("is not confused by a latest symlink alongside the real worker dir", async () => {
84+
const root = tempDir();
85+
const childDir = path.join(root, "subagents", "child-1");
86+
writeTurns(childDir, []);
87+
fs.symlinkSync("child-1", path.join(root, "subagents", "latest"));
88+
const found = await findAgentTraceDir(root, "child-1");
89+
expect(found).toBe(fs.realpathSync(childDir));
90+
});
91+
});
92+
93+
describe("readAgentTrace", () => {
94+
test("throws a clean error for a missing target", async () => {
95+
const root = tempDir();
96+
await expect(readAgentTrace(root, "ghost")).rejects.toBeInstanceOf(AgentTraceNotFoundError);
97+
});
98+
99+
test("reads turns, tool calls, and tool errors", async () => {
100+
const root = tempDir();
101+
const childDir = path.join(root, "subagents", "worker-1");
102+
writeTurns(childDir, [
103+
{ role: "user", content: [{ type: "text", text: "do the thing" }] },
104+
{
105+
role: "assistant",
106+
content: [{ type: "tool_call", id: "call-1", name: "run_shell", arguments: { cmd: "ls" } }],
107+
},
108+
{
109+
role: "user",
110+
content: [
111+
{
112+
type: "tool_result",
113+
callId: "call-1",
114+
content: [{ type: "text", text: "boom" }],
115+
isError: true,
116+
},
117+
],
118+
},
119+
]);
120+
121+
const result = await readAgentTrace(root, "worker-1");
122+
expect(result.totalTurns).toBe(3);
123+
expect(result.entries.map((e) => e.kind)).toEqual(["text", "tool_call", "error"]);
124+
expect(result.entries[2]!.isError).toBe(true);
125+
expect(result.omitted).toBeNull();
126+
});
127+
128+
test("skips a malformed trailing line instead of throwing", async () => {
129+
const root = tempDir();
130+
const childDir = path.join(root, "subagents", "worker-1");
131+
fs.mkdirSync(childDir, { recursive: true });
132+
const good = JSON.stringify({ role: "user", content: [{ type: "text", text: "hi" }] });
133+
fs.writeFileSync(path.join(childDir, "turns.jsonl"), `${good}\n{"role":"assistant","cont`);
134+
135+
const result = await readAgentTrace(root, "worker-1");
136+
expect(result.totalTurns).toBe(1);
137+
expect(result.parseWarnings).toBe(1);
138+
expect(result.entries).toHaveLength(1);
139+
});
140+
141+
test("bounds the entry count to the requested limit and reports omission", async () => {
142+
const root = tempDir();
143+
const childDir = path.join(root, "subagents", "worker-1");
144+
const turns = Array.from({ length: 5 }, (_, i) => ({
145+
role: "assistant",
146+
content: [{ type: "text", text: `turn ${i}` }],
147+
}));
148+
writeTurns(childDir, turns);
149+
150+
const result = await readAgentTrace(root, "worker-1", { limit: 2 });
151+
expect(result.entries).toHaveLength(2);
152+
expect(result.entriesTruncated).toBe(true);
153+
expect(result.omitted).not.toBeNull();
154+
expect(result.omitted!.hint.length).toBeGreaterThan(0);
155+
});
156+
157+
test("never exceeds the total-output character cap regardless of entry/window caps", async () => {
158+
const root = tempDir();
159+
const childDir = path.join(root, "subagents", "worker-1");
160+
const turns = Array.from({ length: 600 }, (_, i) => ({
161+
role: "assistant",
162+
content: [{ type: "text", text: `turn ${i} `.repeat(1000) }], // ~5,000 chars each
163+
}));
164+
writeTurns(childDir, turns);
165+
166+
const result = await readAgentTrace(root, "worker-1", {
167+
fromTurn: 0,
168+
toTurn: 600,
169+
limit: MAX_TRACE_ENTRY_LIMIT,
170+
});
171+
const totalChars = result.entries.reduce((sum, e) => sum + e.content.length, 0);
172+
expect(totalChars).toBeLessThanOrEqual(MAX_TRACE_TOTAL_CHARS);
173+
expect(result.entriesTruncated).toBe(true);
174+
expect(result.omitted).not.toBeNull();
175+
expect(result.omitted!.reason).toContain("total output cap");
176+
});
177+
178+
test("never exceeds the hard entry-limit cap regardless of requested limit", async () => {
179+
const root = tempDir();
180+
const childDir = path.join(root, "subagents", "worker-1");
181+
const turns = Array.from({ length: 10 }, (_, i) => ({
182+
role: "assistant",
183+
content: [{ type: "text", text: `turn ${i}` }],
184+
}));
185+
writeTurns(childDir, turns);
186+
187+
const result = await readAgentTrace(root, "worker-1", { limit: 1_000_000 });
188+
expect(result.entries.length).toBeLessThanOrEqual(MAX_TRACE_ENTRY_LIMIT);
189+
});
190+
191+
test("never exceeds the hard turn-window cap regardless of requested range", async () => {
192+
const root = tempDir();
193+
const childDir = path.join(root, "subagents", "worker-1");
194+
const turns = Array.from({ length: 500 }, (_, i) => ({
195+
role: "assistant",
196+
content: [{ type: "text", text: `turn ${i}` }],
197+
}));
198+
writeTurns(childDir, turns);
199+
200+
const result = await readAgentTrace(root, "worker-1", {
201+
fromTurn: 0,
202+
toTurn: 500,
203+
limit: MAX_TRACE_ENTRY_LIMIT,
204+
});
205+
expect(result.toTurn - result.fromTurn).toBeLessThanOrEqual(MAX_TRACE_TURN_WINDOW);
206+
});
207+
208+
test("filters entries by kind", async () => {
209+
const root = tempDir();
210+
const childDir = path.join(root, "subagents", "worker-1");
211+
writeTurns(childDir, [
212+
{
213+
role: "assistant",
214+
content: [
215+
{ type: "thinking", thinking: "hmm" },
216+
{ type: "text", text: "hello" },
217+
],
218+
},
219+
]);
220+
221+
const result = await readAgentTrace(root, "worker-1", { kinds: ["text"] });
222+
expect(result.entries.map((e) => e.kind)).toEqual(["text"]);
223+
});
224+
225+
test("truncates an oversized entry body and marks it truncated", async () => {
226+
const root = tempDir();
227+
const childDir = path.join(root, "subagents", "worker-1");
228+
writeTurns(childDir, [
229+
{ role: "assistant", content: [{ type: "text", text: "x".repeat(10_000) }] },
230+
]);
231+
232+
const result = await readAgentTrace(root, "worker-1");
233+
expect(result.entries[0]!.truncated).toBe(true);
234+
expect(result.entries[0]!.content.length).toBeLessThan(10_000);
235+
});
236+
237+
test("a partially written trace (worker still running) reads what exists so far", async () => {
238+
const root = tempDir();
239+
const childDir = path.join(root, "subagents", "worker-1");
240+
fs.mkdirSync(childDir, { recursive: true });
241+
fs.writeFileSync(
242+
path.join(childDir, "turns.jsonl"),
243+
`${JSON.stringify({ role: "user", content: [{ type: "text", text: "go" }] })}\n`,
244+
);
245+
246+
const result = await readAgentTrace(root, "worker-1");
247+
expect(result.totalTurns).toBe(1);
248+
expect(result.entries).toHaveLength(1);
249+
});
250+
});

0 commit comments

Comments
 (0)