Agent-scoped submission tools for seeding reference data and testing forms by travist · Pull Request #82 · formio/ai

travist · 2026-09-29T13:59:37Z

Summary

The skills.sh remediation (#80 and the Sep 9 pass) stated that the agent never touches submission data, and the toolset backed that by having no submission tool at all. That over-corrected. A select that reads a Resource shows an empty dropdown until someone adds rows, and nothing could check a form's validation, conditional fields, calculated values, or actions with a real submission.

This PR adds that capability, with the boundary enforced by the MCP server. The skills don't just ask the agent to behave: every submission_* tool reaches only submissions this server created and signed for the calling working directory.

OpenSpec change: openspec/changes/agent-scoped-submission-tools/ (proposal, specs, design with resolved open questions, tasks — 66/66).

How the boundary works

Signed tag. Every agent write carries a server-written metadata.agent tag with exactly four fields: source (picks out agent rows, for the index query and in the portal), session (a label derived from the directory's key, so list queries return only that directory's rows and other directories can't crowd them off a page), purpose (reference-data or test, which test-row cleanup filters on), and sig. The sig is a deterministic HMAC-SHA256 over the project, form, owner, session, purpose, and the full stored data. Anyone who submits to a form can set metadata, so a label alone would be a claim; the signature is the proof. A copied tag verifies only on a byte-identical copy of the agent's own row, which is also why no per-tag nonce is needed.
Dry run first. Each write first sends ?dryrun=1, which returns the data as Form.io will store it (defaults, server-calculated values). That data is what gets signed. The dry run saves nothing and runs no action, so there is no second write and no duplicate email or webhook. Create-then-PATCH was rejected because PATCH fires update actions.
Per-directory key. Each working directory gets its own 32-byte key, stored in ~/.formio/mcp-submission-keys.json with mode 0600. An unreadable store fails loudly, and the key and store path never appear in a tool result.
Verify every record. Each returned record is re-verified, and failures are dropped and counted. A get treats a 404 and a submission the agent didn't create the same way: not found, with none of its content.
Writes check ownership first. Update and delete verify the target before writing. Update keeps the tag's identity and re-signs over the new data. Every write result names the form actions it triggered.

Tools

Tool	Purpose
`submission_create`	Create a signed `reference-data` or `test` submission
`submission_list`	List this directory's signed submissions, by purpose and `data` filters
`submission_get`	Get one signed submission
`submission_update`	Replace a signed submission's data
`submission_delete`	Delete a signed submission

The server instructions also state the scope and the action_list check, for clients without the skills. FormioApiError now carries status and the response body, so a dry-run 400 reports Form.io's validation messages.

Skills

New guideline at formio-mcp-setup/references/agent-submissions.md (other skills link to it rather than restate it). It covers:
- the two purposes, reference-data and test;
- no reference rows in user-type Resources or in forms with a Login, Role Assignment, or Group Assignment action;
- invented values only;
- action_list + preview + approval before every write, with a warning for live projects;
- test-row cleanup.
formio-form-builder offers to seed an empty dropdown source and to write a test submission after SAVE.
formio-application gains Step 3.6, which seeds the Resources the planner marks with a new Seed: reference-data line.
formio-actions documents testing an action with a test submission.
Over-corrected wording revised. Places that said "no submission tool", "no tool returns submission data", or "only an admin seeds reference data" now name the scoped tools. The ban on reading an end user's submission by any route stays.

Live verification

live-submission-behavior.test.ts is opt-in (it skips unless FORMIO_IT_PROJECT_URL / FORMIO_IT_ADMIN_TOKEN are set). Run against Form.io Enterprise 9.9.2 (form.test/sandbox), 5/5 passed, and its scratch forms and user were removed afterwards. It confirmed:

a dry-run POST fills defaults and server-calculated values without saving or running an action;
the real write stores exactly the dry-run data, with the same owner;
a dry-run PUT changes nothing;
metadata.agent is stored verbatim;
a non-admin who sets owner to the admin's id is still stored as the owner.

Test plan

Red → Green → Refactor per task group; every new behavior has a failing test first
pnpm test: 1200 mcp-server tests (+5 live, skipped without env) and 1093 skill-tests pass
pnpm lint and pnpm format clean; openspec validate --strict passes
Live suite 5/5 against form.test/sandbox
After release, check the skills.sh Snyk / Agent Trust Hub / Socket results for the skills whose submission prose changed

Changeset: .changeset/agent-scoped-submission-tools.md (@formio/mcp minor, @formio/ai minor).

🤖 Generated with Claude Code

…sting forms The skills.sh remediation stated that the agent never touches submission data, and the toolset backed that with the absence of any submission tool. That over-corrected: a select reading a Resource shows an empty dropdown until someone adds rows, and nothing could check a form with a real submission. This adds that capability with the boundary enforced by the server rather than by prose. MCP server - submission_create / _list / _get / _update / _delete, registered through registerAllTools, each reaching only submissions this server created and signed for the calling working directory. - Every agent write carries a server-written metadata.agent tag. Its HMAC covers projectUrl, form, owner, session, purpose, nonce, and the full stored data; the stored data comes from a ?dryrun=1 pass, which normalizes (defaults, calculated values) without saving or running actions. - Per-directory 32-byte key in ~/.formio/mcp-submission-keys.json (0600); an unreadable store fails loudly and never names its path. - Reads force the tag filters and build the query from structured data.* filters with an operator allow-list. resourcejs splits keys on the first `__`, so `__` inside a field name is refused rather than reaching Mongo as an arbitrary $ selector. Every returned record is re-verified; failures are dropped and counted, and a get answers a 404 and an unowned record the same way. - Update and delete verify first; update keeps the tag identity and re-signs. Write results name the form actions that ran. - FormioApiError carries status and response body; server instructions state the scope and the action_list check. - Opt-in live suite (live-submission-behavior.test.ts) confirmed the dry-run and owner behavior against Form.io Enterprise 9.9.2. Skills - Canonical guideline formio-mcp-setup/references/agent-submissions.md: the two purposes, no reference rows in auth forms, invented values, action_list + preview + approval before every write, test cleanup. - formio-form-builder seeds an empty dropdown source and writes a test submission after SAVE; formio-application gains Step 3.6 for planner- marked `Seed: reference-data` Resources; formio-actions documents testing an action. - Prose that said no submission tool exists, no tool returns submission data, or only an admin seeds reference data now names the scoped tools; the ban on reading an end user's submission by any route stays. OpenSpec change agent-scoped-submission-tools carries the proposal, specs, design (with the resolved open questions), and tasks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The tag's signature already binds the project, form, owner, session, purpose, and the full stored data, so a copied tag verifies only on a byte-identical copy of the agent's own row. A random per-tag nonce added no protection on top of that, and two identical seed rows sharing a signature is harmless. metadata.agent is now exactly { source, session, purpose, sig }, and the signature is deterministic. session stays because Mongo cannot verify an HMAC and the index query needs it to return only this directory's rows; purpose stays because test-row cleanup filters on it. Update no longer carries a nonce forward. Spec, design (with the rationale), tasks, and the changeset follow. Live suite re-run against form.test/sandbox: 5/5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

blakekrammes · 2026-10-01T20:35:19Z

@@ -0,0 +1,95 @@
+import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';


The first time I ran the submission_create tool I got this message indicating it couldn't read the submission back.

"The submission was written but can't be read back, so the test didn't fully work. Form.io stored it as submission 6abec34cba9cae5e41036401 on the Contact Us form. The server then found its signature doesn't match the stored record, so the submission tools can't fetch it, show it, or delete it.

Likely cause. The fields came back identical. The error's "fields: unknown" means no data field differed. The signature is built in src/tools/submission_create.ts from project, form, owner, session, purpose and data. With data matching, the likeliest culprits are:

Owner (most likely). owner is taken from the dry run (dryRun.owner, line 65). If the dry run returns no owner, or a different one from the real save, the record is signed with owner: null but stored with your admin user's ID, and the check fails.

The tag itself. If the server drops or rewrites metadata.agent on save, verification fails the same way.

I haven't confirmed which. Confirming means reading the raw stored record, and the agent-submission rules forbid me reading a submission any way other than these tools. You can check it in the portal: compare the record's owner with what a ?dryrun=1 POST returns.

Cleanup. Delete submission 6abec34cba9cae5e41036401 in the Form.io portal. Filtering on metadata.agent.source = agent finds it.

I can write a failing test that reproduces the owner mismatch and fix submission_create. One option is to sign using the owner the real save returns, though that changes what the signature guarantees."

I noticed just now that the sigs are different in the verifyRecord fn.

I believe reason 1 from the first comment checks out. I removed the owner field and verification started working.

"Owner (most likely). owner is taken from the dry run (dryRun.owner, line 65). If the dry run returns no owner, or a different one from the real save, the record is signed with owner: null but stored with your admin user's ID, and the check fails."

blakekrammes · 2026-10-01T20:43:11Z

+export function signTag({ key, projectUrl, formId, owner, purpose, data }: SignTagInput): AgentTag {
+  const session = sessionLabel(key);
+  const sig = computeSig(key, { projectUrl, formId, owner, session, purpose, data });
+  return { source: 'agent', session, purpose, sig };


Since the object is named metadata.agent, I would remove the additional source property here.

Suggested change

return { source: 'agent', session, purpose, sig };

return { session, purpose, sig };

blakekrammes · 2026-10-01T21:41:57Z

+          method: 'POST',
+          body: { data },
+        })) as DryRun;
+        const owner = typeof dryRun.owner === 'string' ? dryRun.owner : null;


It looks like it's not intended for the MCP server to be able to override owner on a submission. In what case would the dry run return an owner?

blakekrammes

It works really well! This is an exciting feature that I enjoyed testing.

I had some issues with the owner not coming back from the dry run (see comments), which caused all the submission verifications to fail.

I manually reviewed all the files with the exception of:

The skills tests
The openspec artifacts

I also was able to configure the MCP server to run locally with a debugger, which was really helpful for understanding the tool calls and logic. It could be nice to commit that as a dev config.

blakekrammes · 2026-10-02T18:08:13Z

+        formIdOrPath: z.string().describe('Form ID (_id) or path'),
+        data: z
+          .record(z.string(), z.unknown())
+          .describe('The submission data, keyed by component key; invented values only'),


'invented values only' wouldn't apply to reference data though.

blakekrammes · 2026-10-02T20:22:25Z

+ * so it reveals neither the key nor the directory's path; it is not a secret — the
+ * signature, not the label, is what a record must carry to verify.
+ */
+export function sessionLabel(key: Buffer): string {


It seems like the session label is really a directory label. Starting a new session in the same directory would yield the same dir key and session label.

Maybe dirLabel?

blakekrammes · 2026-10-02T20:52:20Z

+  server.registerTool(
+    'submission_update',
+    {
+      description: `Replace the data of one submission the agent created; its purpose is kept. ${SCOPE_SENTENCE} Updating runs the form's update actions; the result names the ones that ran. ${RULES_SENTENCE}`,


My agent noted that the actions that are listed as having run include actions having conditions that were not met.

i.e. an action that fires after create that is conditionally run if, say data.priority is high, would be listed in actionTitlesFor even if it didn't execute (if data.priority was low).

A simple fix would just be to alter the language a bit. The agent may figure out which actions ran and which didn't on its own after the fact.

travist and others added 2 commits September 29, 2026 08:59

travist requested a review from blakekrammes September 29, 2026 16:58

blakekrammes reviewed Oct 1, 2026

View reviewed changes

blakekrammes requested changes Oct 2, 2026

View reviewed changes

Navigation Menu

Repository navigation

Agent-scoped submission tools for seeding reference data and testing forms - #82

Agent-scoped submission tools for seeding reference data and testing forms#82
travist wants to merge 2 commits into
mainfrom
feat/agent-submission-tools

travist commented Sep 29, 2026 •

edited

Loading

Uh oh!

blakekrammes Oct 1, 2026 •

edited

Loading

Uh oh!

blakekrammes Oct 1, 2026

Uh oh!

blakekrammes Oct 1, 2026

Uh oh!

blakekrammes Oct 1, 2026 •

edited

Loading

Uh oh!

blakekrammes Oct 1, 2026

Uh oh!

blakekrammes left a comment

Uh oh!

blakekrammes Oct 2, 2026

Uh oh!

blakekrammes Oct 2, 2026 •

edited

Loading

Uh oh!

blakekrammes Oct 2, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -0,0 +1,95 @@
		import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';

	return { source: 'agent', session, purpose, sig };
	return { session, purpose, sig };

Conversation

travist commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How the boundary works

Tools

Skills

Live verification

Test plan

Uh oh!

blakekrammes Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

blakekrammes Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

blakekrammes Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

blakekrammes Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

blakekrammes Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

blakekrammes left a comment

Choose a reason for hiding this comment

Uh oh!

blakekrammes Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

blakekrammes Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

blakekrammes Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

travist commented Sep 29, 2026 •

edited

Loading

blakekrammes Oct 1, 2026 •

edited

Loading

blakekrammes Oct 1, 2026 •

edited

Loading

blakekrammes Oct 2, 2026 •

edited

Loading

blakekrammes Oct 2, 2026 •

edited

Loading