Repository navigation
feat(runner): stream fullsend's console live under the progress prefix - #286
astefanutti wants to merge 6 commits into
Conversation
With a log prefix (batch mode's "eval", "eval:<case>" per case) the fullsend
runner printed only "Running: <cmd>" and "OK/FAIL (Ns)", so a CI job log was
silent for the whole run: 13 minutes for two cases, up to two hours for a
full batch. The claude-code runner streams its progress under the same
" <prefix> | " lines.
_run now reads fullsend's stdout and stderr concurrently (one daemon thread
per pipe), keeps the full text for RunResult.stderr as communicate()
returned it (stdout, then stderr; the provenance parse of the Image: line
reads the same text), and forwards a filtered subset as it arrives: the
section and step lines (→ • ✓ ✗ !) with the detail lines indented under
them (plan block, → Results), the agent activity lines (💬 ⚙ ⏳), plain
Error/Warning/fatal lines and Claude Code's "Ignoring N permissions.allow
entries" line (the one behind the eval permission denials), each cut at 200
characters with credential assignments (…token=, …secret=, …password=,
…api_key=) shown as KEY=***. The per-finding "! Sanitized […]" lines, the
assistant's thinking (🧠, which the claude-code runner does not stream
either) and multi-line text and the pre/post script output are not
forwarded; the capture keeps everything, unredacted. The real two-case
console (1,096 lines) forwards 360 lines, 30 KB.
Draining never depends on printing: a forward that fails (the harness's
stdout closed, or unable to encode a marker) stops the forwarding and the
reader keeps emptying the pipe, so fullsend cannot stall on a full pipe
until the timeout. Each reader closes its pipe at EOF, as communicate() did.
The wait keeps communicate(timeout)'s contract (both pipes at EOF and the
process exited within the timeout, else the group is torn down with the
leader still unreaped), and terminate_process_group grows a keyword-only
capture_output=False for a caller whose own threads read the pipes, so the
teardown only reaps instead of competing for them. A descendant outside the
group still holding a pipe after the bounded drain is noted in the case's
stderr ("Console pipes still open after the group kill"), as the
communicate(timeout=5) it replaces reported it. Undecodable console bytes
become U+FFFD instead of raising out of the case.
No config knob: streaming follows the harness's progress logging, as it
does for the claude-code runner; no runner has a console-verbosity key.
Signed-off-by: Antonin Stefanutti <antonin@stefanutti.fr>
PR Summary by QodoStream fullsend console progress in runner logs
AI Description
Diagram
High-Level Assessment
Files changed (6)
|
|
🤖 Finished Review · ✅ Success · Started 4:03 PM UTC · Completed 4:10 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $1.83 |
Code Review by Qodo
1.
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
|
<!-- fullsend:risk-assessment --> DetailsModerate risk change expanding live console streaming across 6 files and 1,652 lines with solid test coverage, zero protected or security-sensitive paths, and moderate churn keeping the assessment anchored at moderate. Previous run<!-- fullsend:risk-assessment --> DetailsModerate risk change expanding live console streaming across 6 files and 1,647 lines with solid test coverage, zero protected or security-sensitive paths, and moderate churn keeping the assessment anchored at moderate. Previous run (2)<!-- fullsend:risk-assessment --> DetailsModerate risk change expanding live console streaming across 6 files and 1,049 lines with solid test coverage, zero protected or security-sensitive paths, and moderate churn keeping the assessment anchored at moderate. Previous run (3)<!-- fullsend:risk-assessment --> DetailsModerate change adding live console streaming across 6 files and ~700 lines with solid test coverage, zero protected or security-sensitive paths, and moderate churn. |
ReviewFindingsMedium
Low
Earlier findings
Next steps:
1 finding(s) below the Previous run<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJyZXF1ZXN0LWNoYW5nZXMiLCJmaW5kaW5ncyI6W3sic2V2ZXJpdHkiOiJtZWRpdW0iLCJjYXRlZ29yeSI6InNlY3JldC1leHBvc3VyZSIsImZpbGUiOiJhZ2VudF9ldmFsL2FnZW50L2Z1bGxzZW5kLnB5IiwiaWQiOiJmXzRlNzY2Y2Q5NTYxN2E1ZWMiLCJsaW5lIjo4NTUsImFjdGlvbmFibGUiOnRydWV9LHsic2V2ZXJpdHkiOiJsb3ciLCJjYXRlZ29yeSI6ImVkZ2UtY2FzZSIsImZpbGUiOiJhZ2VudF9ldmFsL2FnZW50L2Z1bGxzZW5kLnB5IiwiaWQiOiJmXzk1MWUxZDM4Y2E0YzRhYzEiLCJsaW5lIjo3NDgsImFjdGlvbmFibGUiOmZhbHNlfSx7InNldmVyaXR5IjoiaW5mbyIsImNhdGVnb3J5IjoiaW5qZWN0aW9uLXZ1bG4iLCJmaWxlIjoiYWdlbnRfZXZhbC9hZ2VudC9mdWxsc2VuZC5weSIsImlkIjoiZl8wZTgwZWJiZTZkNWMyNDEzIiwibGluZSI6NzQ4LCJhY3Rpb25hYmxlIjpmYWxzZX0seyJzZXZlcml0eSI6ImxvdyIsImNhdGVnb3J5IjoibG9naWMtZXJyb3IiLCJmaWxlIjoiYWdlbnRfZXZhbC9hZ2VudC9mdWxsc2VuZC5weSIsImlkIjoiZl8xOTVhZDg0Y2I0YWVkNjVkIiwibGluZSI6MTQxLCJhY3Rpb25hYmxlIjp0cnVlfV0sImRpc3Bvc2l0aW9ucyI6W3siaWQiOiJmXzRlNzY2Y2Q5NTYxN2E1ZWMiLCJzdGF0dXMiOiJvcGVuIn0seyJpZCI6ImZfMTk1YWQ4NGNiNGFlZDY1ZCIsInN0YXR1cyI6InJlc29sdmVkX2J5X2NoYW5nZSJ9LHsiaWQiOiJmXzBlODBlYmJlNmQ1YzI0MTMiLCJzdGF0dXMiOiJyZWNsYXNzaWZpZWQifV19 --> ReviewFindingsMedium
Low
Earlier findings
Labels: The PR adds a new live-streaming capability to the fullsend runner. Next steps:
1 finding(s) below the Previous run (2)<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJyZXF1ZXN0LWNoYW5nZXMiLCJmaW5kaW5ncyI6W3sic2V2ZXJpdHkiOiJtZWRpdW0iLCJjYXRlZ29yeSI6InNlY3JldC1leHBvc3VyZSIsImZpbGUiOiJhZ2VudF9ldmFsL2FnZW50L2Z1bGxzZW5kLnB5IiwiaWQiOiJmXzRlNzY2Y2Q5NTYxN2E1ZWMiLCJsaW5lIjo3NTQsImFjdGlvbmFibGUiOnRydWV9LHsic2V2ZXJpdHkiOiJsb3ciLCJjYXRlZ29yeSI6ImxvZ2ljLWVycm9yIiwiZmlsZSI6ImFnZW50X2V2YWwvYWdlbnQvZnVsbHNlbmQucHkiLCJpZCI6ImZfMTk1YWQ4NGNiNGFlZDY1ZCIsImxpbmUiOjE0MSwiYWN0aW9uYWJsZSI6dHJ1ZX0seyJzZXZlcml0eSI6ImxvdyIsImNhdGVnb3J5IjoiaW5qZWN0aW9uLXZ1bG4iLCJmaWxlIjoiYWdlbnRfZXZhbC9hZ2VudC9mdWxsc2VuZC5weSIsImlkIjoiZl8wZTgwZWJiZTZkNWMyNDEzIiwibGluZSI6NjYzLCJhY3Rpb25hYmxlIjp0cnVlfV0sImRpc3Bvc2l0aW9ucyI6W3siaWQiOiJmXzE5NWFkODRjYjRhZWQ2NWQiLCJzdGF0dXMiOiJyZWNsYXNzaWZpZWQifSx7ImlkIjoiZl80ZTc2NmNkOTU2MTdhNWVjIiwic3RhdHVzIjoib3BlbiJ9LHsiaWQiOiJmXzBlODBlYmJlNmQ1YzI0MTMiLCJzdGF0dXMiOiJvcGVuIn1dfQ== --> ReviewFindingsMedium
Low
Earlier findings
Next steps:
Previous run (3)<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJyZXF1ZXN0LWNoYW5nZXMiLCJmaW5kaW5ncyI6W3sic2V2ZXJpdHkiOiJtZWRpdW0iLCJjYXRlZ29yeSI6ImxvZ2ljLWVycm9yIiwiZmlsZSI6ImFnZW50X2V2YWwvYWdlbnQvZnVsbHNlbmQucHkiLCJpZCI6ImZfMTk1YWQ4NGNiNGFlZDY1ZCIsImxpbmUiOjEyOSwiYWN0aW9uYWJsZSI6dHJ1ZX0seyJzZXZlcml0eSI6Im1lZGl1bSIsImNhdGVnb3J5Ijoic2VjcmV0LWV4cG9zdXJlIiwiZmlsZSI6ImFnZW50X2V2YWwvYWdlbnQvZnVsbHNlbmQucHkiLCJpZCI6ImZfNGU3NjZjZDk1NjE3YTVlYyIsImxpbmUiOjY4OSwiYWN0aW9uYWJsZSI6dHJ1ZX0seyJzZXZlcml0eSI6ImxvdyIsImNhdGVnb3J5IjoiaW5qZWN0aW9uLXZ1bG4iLCJmaWxlIjoiYWdlbnRfZXZhbC9hZ2VudC9mdWxsc2VuZC5weSIsImlkIjoiZl8wZTgwZWJiZTZkNWMyNDEzIiwibGluZSI6NjY1LCJhY3Rpb25hYmxlIjp0cnVlfV19 --> ReviewFindingsMedium
Low
Next steps:
|
![fullsend-ai-review[bot]](https://avatars.githubusercontent.com/in/3478501?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
…cut quoted credential whole, strip terminal control Review findings on the live fullsend console (#286). A pipe a descendant outside fullsend's process group still held after the bounded drain kept its reader forwarding under the finished case's prefix: the descendant's later lines landed after the case's FAIL line and among the next cases' progress. _ConsoleCapture.detach() now ends the forwarding and the keeping of lines once the run is over; it is taken under the print lock the readers print under and the caller prints its status line under, so a forwarded line in flight has landed before detach() returns and none follows. The reader goes on emptying the pipe (a read blocked on it cannot be interrupted) and closes it at EOF, keeping nothing. The timeout path detaches after its drain; the interrupt path, which re-raised without touching the readers, now gets the same bounded drain and detach before the interrupt propagates. _SECRET_ASSIGNMENT_RE redacted an unterminated quoted value only up to its first blank, so a command fullsend cut at 120 runes inside the value was forwarded as PASSWORD=*** second third. A quoted value now runs over escaped quotes (\", \') to its closing quote and, with none on the line, to the end of the line; a bare value is unchanged, as is a non-credential KEY=value. _ANSI_RE stripped only CSI sequences from a forwarded line. _CONTROL_RE drops CSI, OSC (BEL or ST terminated), DCS/SOS/PM/APC strings, the single-character and nF escapes, their 8-bit C1 forms, and the other C0/C1 control characters except the tab; a string with no terminator runs to the end of the line. The capture, stderr.log and the provenance parse keep the console as it was written. Tests: unterminated double and single quotes, escaped quotes in a quoted value, a command cut mid-value with blanks after the cut, non-secret assignments untouched, control sequences in the filter and on both pipes of a run, a descendant outside the group writing after the bounded drain of a timeout and of an interrupt (nothing forwarded or kept, the pipe closed at its EOF). Live progress in runner.md states the redaction rule, the control stripping and what the timeout note now means. Signed-off-by: Antonin Stefanutti <antonin@stefanutti.fr>
…detach the console readers on every exit Follow-up to the #286 review of the live fullsend console. A displayed command cut right after a backslash inside a quoted credential (or a shell line continuation) left the quoted alternatives of _SECRET_ASSIGNMENT_RE unable to reach the end of the line: a lone trailing backslash matched neither branch, so the bare-value fallback redacted only to the first blank and PASSWORD="abc def\ was forwarded as PASSWORD=*** def\. The escape in both quoted branches is now optional, so the value runs through the end of the line as the rule says; every other input of the redaction tests, escaped backslashes included, renders as before. The comment also says what the quoted rule does not cover: a quote further in or around the whole assignment is a bare value, redacted to the first blank. A Ctrl-C landing inside the timeout handler (the term-grace wait, the pipe drain) propagated past the sibling interrupt clause and skipped the readers' detach, so a reader could still print under the finished case's prefix while the interrupt unwound. One outer try/finally now detaches on every exit; the inner try/finally on the interrupt path goes. The holder tests give the leader three seconds (not one) to start Python, spawn its holder and print the pid, and _holder_pid fails with a message instead of an IndexError when it did not get to. Signed-off-by: Antonin Stefanutti <antonin@stefanutti.fr>
|
🤖 Finished Review · ✅ Success · Started 5:30 PM UTC · Completed 5:44 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $2.61 |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
… after the clip, strip bidi controls Second review round of the live fullsend console (#286). A ⚙ line forwarded its raw argument, so a command or a prompt fullsend displayed — "⚙ Bash: curl --password swordfish …" — reached the CI log with its credential, and the assignment rule covered one shape only: TOKEN=ab"c d", TOKEN=abc\ def and "api_key=ab cd" leaked the part after the first blank; --password x, Authorization: Bearer x, "token": "x", URL userinfo and AWS_ACCESS_KEY_ID= were not redacted at all. _present now renders a ⚙ line without its raw argument: a file tool (Read, Write, Edit, MultiEdit, Glob, Grep, NotebookEdit, LS) keeps its path, Bash shows its program name alone (the first shell word after any leading NAME=value assignments, when it is a plain program name) and every other tool — Agent, Task, Skill, WebFetch, WebSearch, an MCP tool, an unknown name — its name alone. Every forwarded line then has the values of the credential-named variables of fullsend's environment masked (os.environ with execution.env, runner.env and the hook env over it; a name matching _SECRET_KEY_RE, a value of eight characters or more, longest first; read once per run in _run, kept on the console filter, never printed) and is redacted by pattern: KEY=value, --flag value and key: value / "key": value whose key names a credential (a credential word of its own, a name ending in key, token, secret or password, or containing credential — fullsend's Tokens: line is none of these), the value being one whole shell word (quoted parts, escaped blanks and concatenations included, an unterminated quote to the end of the line, an assignment opened inside a quoted word through that word), Authorization: Bearer|Basic|token <x> and a bare Bearer <x>, and URL userinfo (user:***@, ***@ for a bare token). _SECRET_ASSIGNMENT_RE backtracked quadratically on one long word repeating a credential name with no "=", and ran before the clip (1.3 s at 50 KB, 5.4 s at 100 KB, holding the GIL). The line is now clipped first, and each keyed form is found with one forward scan — \b(\w+)=, a flag or a field key anchored at a word start — whose capture is tested in a callback, the value consumed from there; the URL pattern is anchored the same way. Clipping first is safe: a bare value cut short is the rest of its word, a quoted one runs to the end of the line, a userinfo cut before its @ is taken for one (the cut form of the pattern), and a line the redaction lengthened is cut again. 100 KB of any shape clears select() in milliseconds. _CONTROL_RE left the Unicode bidi controls (U+061C, U+200E/F, U+202A–E, U+2066–9), the line and paragraph separators (U+2028/9) and the zero-width characters (U+200B–D, U+2060–4, U+FEFF) in a forwarded line, which can reorder or hide text in a log viewer; they are stripped now, ordinary non-ASCII text and the markers' U+FE0F kept. The capture (RunResult.stderr, stderr.log, the provenance parse) is still byte for byte what communicate() would have returned; timeout, interrupt and detach are untouched; nothing streams without a log prefix; no new dependency (shlex for the program name). Tests: the sample console gains Write/Agent/Skill calls and a long 💬 line (the ⚙ Bash expectations become "⚙ Bash: python3"); tool-call rendering of every kind; shell-concatenated values; flags, headers, fields and URLs; environment values masked, as a unit and on both pipes of a run; clip-then-redact leaving no fragment of a cut credential; a 100 KB pathological line of ten shapes under a second, clipped and unclipped; bidi and zero-width characters in the filter and on both pipes. The existing redaction tests move from ⚙ Bash lines to the 💬/Error lines that keep their text. Live progress in runner.md states what a ⚙ line shows, the redaction rules, the environment masking and the clip-then-redact order; the module docstring follows. Signed-off-by: Antonin Stefanutti <antonin@stefanutti.fr>
|
🤖 Finished Review · ✅ Success · Started 11:41 AM UTC · Completed 11:52 AM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $2.71 |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
… after the clip, strip bidi controls Second review round of the live fullsend console (#286). A ⚙ line forwarded its raw argument, so a command or a prompt fullsend displayed — "⚙ Bash: curl --password <pw> …" — reached the CI log with its credential, and the assignment rule covered one shape only: TOKEN=ab"c d", TOKEN=abc\ def and "api_key=ab cd" leaked the part after the first blank; --password x, Authorization: Bearer x, "token": "x", URL userinfo and AWS_ACCESS_KEY_ID= were not redacted at all. _present now renders a ⚙ line without its raw argument: a file tool (Read, Write, Edit, MultiEdit, Glob, Grep, NotebookEdit, LS) keeps its path, Bash shows its program name alone (the first shell word after any leading NAME=value assignments, when it is a plain program name) and every other tool — Agent, Task, Skill, WebFetch, WebSearch, an MCP tool, an unknown name — its name alone. Every forwarded line then has the values of the credential-named variables of fullsend's environment masked (os.environ with execution.env, runner.env and the hook env over it; a name matching _SECRET_KEY_RE, a value of eight characters or more, longest first; read once per run in _run, kept on the console filter, never printed) and is redacted by pattern: KEY=value, --flag value and key: value / "key": value whose key names a credential (a credential word of its own, a name ending in key, token, secret or password, or containing credential — fullsend's Tokens: line is none of these), the value being one whole shell word (quoted parts, escaped blanks and concatenations included, an unterminated quote to the end of the line, an assignment opened inside a quoted word through that word), Authorization: Bearer|Basic|token <x> and a bare Bearer <x>, and URL userinfo (user:***@, ***@ for a bare token). _SECRET_ASSIGNMENT_RE backtracked quadratically on one long word repeating a credential name with no "=", and ran before the clip (1.3 s at 50 KB, 5.4 s at 100 KB, holding the GIL). The line is now clipped first, and each keyed form is found with one forward scan — \b(\w+)=, a flag or a field key anchored at a word start — whose capture is tested in a callback, the value consumed from there; the URL pattern is anchored the same way. Clipping first is safe: a bare value cut short is the rest of its word, a quoted one runs to the end of the line, a userinfo cut before its @ is taken for one (the cut form of the pattern), and a line the redaction lengthened is cut again. 100 KB of any shape clears select() in milliseconds. _CONTROL_RE left the Unicode bidi controls (U+061C, U+200E/F, U+202A–E, U+2066–9), the line and paragraph separators (U+2028/9) and the zero-width characters (U+200B–D, U+2060–4, U+FEFF) in a forwarded line, which can reorder or hide text in a log viewer; they are stripped now, ordinary non-ASCII text and the markers' U+FE0F kept. The capture (RunResult.stderr, stderr.log, the provenance parse) is still byte for byte what communicate() would have returned; timeout, interrupt and detach are untouched; nothing streams without a log prefix; no new dependency (shlex for the program name). Tests: the sample console gains Write/Agent/Skill calls and a long 💬 line (the ⚙ Bash expectations become "⚙ Bash: python3"); tool-call rendering of every kind; shell-concatenated values; flags, headers, fields and URLs; environment values masked, as a unit and on both pipes of a run; clip-then-redact leaving no fragment of a cut credential; a 100 KB pathological line of ten shapes under a second, clipped and unclipped; bidi and zero-width characters in the filter and on both pipes. The existing redaction tests move from ⚙ Bash lines to the 💬/Error lines that keep their text. Live progress in runner.md states what a ⚙ line shows, the redaction rules, the environment masking and the clip-then-redact order; the module docstring follows. The credentials in the test fixtures are placeholders (<pw>, <jwt>, <b64>, <pat>): a value a secret scanner reads as a credential, a base64 user:pass pair or a provider's key prefix, is an alert even in a test. Signed-off-by: Antonin Stefanutti <antonin@stefanutti.fr>
…from the cut command
Second-round review of the console redaction.
A bare Bearer value ran to the next blank regardless of quotes, so in
compact JSON it swallowed the following field's key and left that field's
value past its first blank outside any key context:
{"Authorization":"Bearer <t>","password":"p w"} forwarded "w". The
bare alternative now ends at a quote as well as a blank; a quoted
Authorization header keeps its closing quote (the one previous expectation
that changes).
A ⚙ Bash line's program name was read by shlex, a per-character lexer, from
the whole command before the clip (6 s at 1 MB in the reader thread), which
contradicted the "cost bounded by the cut" documentation. It is read from
the command's first 200 characters: the first word of a command displayed
whole is the same, a split word is cut again, a split quote is no program.
Smaller review notes taken: passphrase and signature are credential words
(--passphrase x, PASSPHRASE=x, X-Amz-Signature=…); an environment variable
named by auth/oauth is masked too (DOCKER_AUTH_CONFIG's auth field is none
of the key rule's words — masking only ever hides); a credential field's
bare value runs to the next , ; } ] or the end of the line, so a multi-word
password in prose is hidden whole; the linear-time test measures CPU time
and feeds the ⚙ Bash shapes. The joiner strip (U+200C/D) stays: it runs
before the redaction and defeats a credential name split by a joiner,
which the tests and runner.md now say.
The captured console is untouched; the forwarded copy only.
Signed-off-by: Antonin Stefanutti <antonin@stefanutti.fr>
|
Force-pushed the two round-2 commits (4f6622c → ff49fa2, 7336d04 → fa500c0). GitHub secret scanning raised an alert on the fixture of Only |
7336d04 to
fa500c0
Compare
|
🤖 Finished Review · ✅ Success · Started 12:18 PM UTC · Completed 12:28 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $2.83 |
The review's secret-exposure finding: a quoted header name was not
redacted ({"Authorization":"Basic x"} forwarded x) and a URL's username
was kept, so https://<token>:x-oauth-basic@github.com forwarded the
token.
_redact_authorization replaces the Authorization regex with a small
parser. The name is found bare (after a dot, a dash, a Markdown mark or
an escaped line break; after a "/" it is a path), quoted as a JSON or
dict key (a bytes key too) or opening a quoted word, followed by ":",
"=" or "=>". A quoted value, or the value of a header that opened a
quoted word, is hidden through the closing quote, the scheme kept when
known; a string holding the scheme alone hides the operand concatenated
after it (a word, a quoted string, a $(...) substitution, after a "+"
too, a quoted scheme alone chaining once more). A bare value is hidden
as one word after a token68 scheme (or its name=value list, Rails'
Token token="..."), as its name=value list after a parameter scheme
(Digest, OAuth, AWS4-HMAC-SHA256, Hawk, SCRAM; a list that does not
parse whole), inside its brackets when it opens some (a Go header dump,
a JSON list), and whole, through the next delimiter or a quote glued to
a word, otherwise: a scheme the rule does not know may carry a
multi-word credential. The bare Bearer rule leaves a *** already there
with its delimiter, and a quote a blank follows opens no credential.
The URL userinfo is masked whole. The field rule reads a bytes key.
Five adversarial rounds drove the shapes the test pins: a "="-padded
token in a quoted header taken for a name="value" parameter that
swallowed the next field's quote, map[Authorization:[Bearer x]],
unknown schemes, f-strings, a Digest list inside a quoted shell word, a
shell-concatenated credential, {"Authorization":["Basic x"]}, blanks
around "=", a partly env-masked token, http.client's
send: b'...\r\nAuthorization: Basic x\r\n' dump. Every pattern stays
linear (the separator regex that went quadratic on a blank run is
unambiguous now): 100 KB shapes redact in under 40 ms.
Known residue, outside the forms the rule claims: a credential-named
key after "=>" or in Markdown bold is not a key for the keyed rules;
dotted or call forms (headers.set('Authorization', 'Basic x')), other
languages' concatenation operators and double-encoded JSON are not
read; a prose "authorization:" loses the rest of its clause, the safe
side.
Signed-off-by: Antonin Stefanutti <antonin@stefanutti.fr>
|
🤖 Finished Review · ❌ Failure (provider "openai": no OpenAI credential: set FULLSEND_OPENAI_AUDIENCE, FULLSEND_OPENAI_IDENTITY_PROVIDER_ID and FULLSEND_OPENAI_SERVICE_ACCOUNT_ID (or inference.openai in config.yaml) for Workload Ide…) · Started 4:34 PM UTC · Completed 4:34 PM UTC Commit: Effort: high |
Why
With a log prefix (batch mode's
eval,eval:<case>per case) the fullsend runner printed onlyRunning: <cmd>andOK/FAIL (Ns), so a CI job log is silent for the whole run: 13 minutes for two cases, up to two hours for a full batch, indistinguishable from a stalled barrier or a hung sandbox until the eval timeout. The claude-code runner streams its progress under the same<prefix> |lines (_extract_progress); this gives the fullsend runner the same behaviour.What
FullsendRunner._runreads fullsend's stdout and stderr concurrently (one daemon thread per pipe), keeps the full text forRunResult.stderrexactly ascommunicate()returned it (stdout, then stderr; the provenance parse of theImage:line reads the same text), and forwards a filtered subset live: section and step lines (→ • ✓ ✗ !) with their indented detail lines (plan block,→ Results), agent activity lines (💬 ⚙ ⏳), plainError/Warning/fatallines and Claude Code'sIgnoring N permissions.allow entries …line, each cut at 200 characters, with credential assignments (…token=,…secret=,…password=,…api_key=) shown asKEY=***. Not forwarded: the per-finding! Sanitized […]lines, the assistant's thinking (🧠, which the claude-code runner does not stream either) and multi-line text, and pre/post-script output. The real two-case console (1,096 lines) forwards 360 lines, 30 KB, so a 26-case batch stays well inside a GitLab runner's default 4 MB output limit.communicate(timeout)'s contract (both pipes at EOF and the process exited within the timeout, else the group is torn down with the leader still unreaped).terminate_process_groupgrows a keyword-onlycapture_output=Falsefor a caller whose own threads read the pipes. A descendant outside the group still holding a pipe after the bounded drain is noted in the case's stderr (Console pipes still open after the group kill). Undecodable console bytes become U+FFFD instead of raising out of the case.website/reference/config/runner.md"Live progress",website/concepts/runners.md.Tests
tests/test_fullsend_runner.py: the fake binary replays a recorded fullsend console; newTestConsoleFilterandTestFullsendConsoleStreamingcover lines forwarded under the prefix in order (stdout and stderr), dropped lines, byte-for-byte capture with provenance still parsingImage:, no streaming without a prefix, both pipes drained past the pipe buffer (1.8 MB each), a forward failure that keeps draining (ASCII sink, broken pipe), the timeout path (streams, then terminates; pipes held past the group kill), pipes closed once drained, thinking not forwarded, credential redaction.tests/test_config.py:terminate_process_group(capture_output=False). Full suite: 2788 passed, 8 skipped.mkdocs build --strictpasses.Not exercised against a live fullsend binary or a GitLab job yet; the line shapes come from a fullsend 0.51 console. A shape fullsend changes later falls back to "dropped" (still captured), never to an error. The first consumer is the rfe-creator-eval fullsend job (fullsend-ai/rfe-creator
eval-profiles/fullsend.yaml).