Skip to content

docs(skills): document the real Vercel skill copy/review/PR workflow (#45) - #100

Merged
douglasjarquin merged 2 commits into
mainfrom
sum-dev/issue45-vercel-skills-review
Sep 11, 2026
Merged

docs(skills): document the real Vercel skill copy/review/PR workflow (#45)#100
douglasjarquin merged 2 commits into
mainfrom
sum-dev/issue45-vercel-skills-review

Conversation

@douglasjarquin

Copy link
Copy Markdown
Owner

Summary

Closes the remaining scope of #45 (roadmap #29 §2): documents the real workflow for reviewing third-party skill copies made through sumctl skills install (delivered by #44), since #45's own acceptance explicitly called out that "the remaining workflow documentation and scoped proof still need delivery." Docs only — no runtime behavior changes; #44's wrapper and #65's activation/rollback mechanism are unchanged.

Ran the real pinned Vercel Skills CLI (1.5.25, not a test fixture) end-to-end in an isolated dev checkout against a throwaway Git project, and used the results as documentation evidence:

  • Discovery is read-only and makes no selection for the owner: .local/bin/skills add michael-denyer/pstack-claude --list lists that repository's real skills (architect, tdd, why, dozens more) without copying anything. Chosen deliberately per the issue's own guidance to "include pstack examples without selecting skills for the owner" — pstack illustrates discovery; nothing from it is installed.
  • Copy: sumctl skills install --target ... --source vercel-labs/agent-skills --skill writing-guidelines --agent claude-code against a throwaway Git project wrote exactly two project-local regular files (.claude/skills/writing-guidelines/SKILL.md, skills-lock.json) and nothing global.
  • Unchanged re-review: re-running the identical command reproduces byte-identical files — empty git status/git diff.
  • Changed re-review: adding a second explicit skill (vercel-composition-patterns) to the same project produced a clean additive diff — only the new skill's files plus a new skills-lock.json entry; the first skill's files were untouched.
  • Found and documented a real bug/gotcha in the upstream CLI: running the pinned CLI's own update -p -y directly (not through sumctl, which doesn't wrap update) against the two-skill project relocated both copies out of .claude/skills/ into a different .agents/skills/ layout — silently dropping the originally requested --agent claude-code target — while skills-lock.json's recorded hashes advanced regardless, with no warning in the CLI's own output. This confirms and sharpens the caution already in Review selected Vercel skill copies through ordinary pull requests #45's issue body ("the pinned CLI fixture showed its lock advancing while a Claude Code copy stayed stale"). Documented as: don't use update; re-run the explicit add/sumctl skills install command instead, which is idempotent for an unchanged selection and reproduces the correct layout for a changed one.

Changes

  • docs/sum-skills.md: new "Review workflow" section (discover → copy → review-as-untrusted-content → commit/PR → re-check) and a new "Do not use the pinned CLI's own update" section with the concrete relocation finding above.
  • docs/features/skills.md: new skills.review-workflow evidence row pointing at the manual steps in docs/ACCEPTANCE.md §11.
  • docs/ACCEPTANCE.md §11: four new concrete manual steps (discovery-without-selection, real copy + diff + PR, unchanged/changed re-review, the update gotcha) alongside the existing local-fixture acceptance steps from Install selected Git skills by repository, ref, and exact path with pinned resources #44.

Scope boundaries respected

  • No SUM-specific sync manager, update tracker, hash-based drift detector, source-lineage tracker, or new check/update commands added — superseded by "vercel copy plus ordinary PR review" per the issue's owner decision.
  • sumctl skills check --root (Sum's own namespace/projection validation) is unchanged and explicitly distinguished in the new docs from third-party content review.
  • All real CLI activity happened in an isolated dev checkout (.sum/dev/issue45-vercel-skills-review) against a throwaway scratch Git project outside the sum repository itself — never the live installation or a busy worker's discovery directory.
  • No global configuration writes, scheduling, auto-merge, or forced restarts.

Test plan

  • Docs-only change; no code touched.
  • Manual, live evidence gathered against the real pinned Vercel Skills CLI 1.5.25 (not tests/test_skills_cli.py's fake binary) — see Summary above; each claim in the new docs was actually run and observed, not inferred.
  • Repository verification (MISE_ENABLE_TOOLS=go,python,node python3 .agents/skills/verify/scripts/verify_run.py --base 81dbdda96dd7c0b31696634f9f943b28981f511b) was still running in the background when this PR was opened. Will update this line with the result once it completes.

Review status

Independent review was not performed beyond the automated verification above — no separate reviewer pane was launched for this developer-checkout PR. Boss decides merge.

cs-test-runner and others added 2 commits September 11, 2026 17:05
…45)

Adds the review workflow #44 left undocumented: discover a source read-only
(demonstrated against a real repo, pstack, without selecting anything for the
owner), copy an explicit selection with `sumctl skills install`, review the
copy as untrusted third-party content, commit, and open an ordinary PR.

Documents re-running the same explicit copy command to check for updates
(idempotent when unchanged, a clean additive diff when not), backed by a real
run against vercel-labs/agent-skills in a throwaway Git project. Also
documents a real finding: the pinned CLI's own `update` subcommand relocates
an `--agent`-scoped copy into a different agent directory instead of
refreshing it in place, silently advancing skills-lock.json regardless -- so
the workflow explicitly avoids it in favor of re-running the explicit copy.

Docs only; no runtime behavior changes.
The map auditor flags backticked tokens that look like repository paths.
The new skills.review-workflow row wrapped a GitHub slug in backticks and
failed CI. Write the slug as plain text and add the scenario to the frozen
manual inventory.
@douglasjarquin
douglasjarquin merged commit cd47f0d into main Sep 11, 2026
1 check passed
@douglasjarquin
douglasjarquin deleted the sum-dev/issue45-vercel-skills-review branch September 11, 2026 18:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant