Ran the new eval suite in evals/ (Sonnet agent + judge, 3 runs/case, mocked uploads MCP, no Bash).
Five cases are rebuilt from real transcripts of UI changes where a screenshot was warranted but the prompt never asks for one — the skill has to fire off its description's "having just captured or changed something visual" clause. That clause is the load-bearing trigger for the staging-as-you-go workflow, and it's the weakest part.
With the plugin loaded, github-screenshots fired:
| case |
fired |
| docs content width |
1/2 (one run aborted by the mock) |
| thumbnails larger |
2/3 |
| sidebar design tweaks (3-part visual tweak list) |
0/3 |
| thumbnail link bug |
2/3 |
| usage alert email |
2/2 |
| neg: CLI shell-completion debugging |
1/3 (should be 0) |
Same prompt, same model, different run, different outcome.
Firing isn't the whole story, and I don't want to oversell it: two runs captured a screenshot without the skill firing, and two fired it and then captured nothing. So post-fire guidance needs a look too.
Deliberately not citing the scored deltas. The without-plugin arm has no uploads MCP tools at all, so a positive delta is guaranteed by construction and measures tool presence, not steering. Two graders also referenced the wrong MCP namespace in this run. The fire counts above are unaffected by both.
Ask: decide whether description wording can carry an "unprompted, after a visual edit" trigger at all, or whether it needs a harness-level nudge (e.g. a PostToolUse hook on edits to UI files).
Repro: evals/run.sh. n=3 — worth re-running the two headline cases at --runs 5 before acting.
Ran the new eval suite in
evals/(Sonnet agent + judge, 3 runs/case, mocked uploads MCP, no Bash).Five cases are rebuilt from real transcripts of UI changes where a screenshot was warranted but the prompt never asks for one — the skill has to fire off its description's "having just captured or changed something visual" clause. That clause is the load-bearing trigger for the staging-as-you-go workflow, and it's the weakest part.
With the plugin loaded,
github-screenshotsfired:Same prompt, same model, different run, different outcome.
Firing isn't the whole story, and I don't want to oversell it: two runs captured a screenshot without the skill firing, and two fired it and then captured nothing. So post-fire guidance needs a look too.
Deliberately not citing the scored deltas. The without-plugin arm has no uploads MCP tools at all, so a positive delta is guaranteed by construction and measures tool presence, not steering. Two graders also referenced the wrong MCP namespace in this run. The fire counts above are unaffected by both.
Ask: decide whether description wording can carry an "unprompted, after a visual edit" trigger at all, or whether it needs a harness-level nudge (e.g. a PostToolUse hook on edits to UI files).
Repro:
evals/run.sh. n=3 — worth re-running the two headline cases at--runs 5before acting.