Skip to content

feat: external volume seeded per actor from a CSI snapshot handle - #231

Closed
teemow wants to merge 9 commits into
giantswarmfrom
feat/seeded-external-volume-v3
Closed

teemow wants to merge 9 commits into
giantswarmfrom
feat/seeded-external-volume-v3

Conversation

@teemow

@teemow teemow commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

Problem

A session that starts in a prepared workspace needs its own read-write volume whose content comes from a snapshot taken outside the actor, chosen when the actor is created. An ActorTemplate's external volume was created empty for every actor of the template, and an Actor carried no volume of its own.

Proposed solution

  • ExternalVolumeTemplate.seeded (field 10001) makes a template's external volume optional and seeded per actor.
  • Actor.volume_seeds (field 10001, immutable): VolumeSeed = volume name, CSI driver, snapshot handle (the status.snapshotHandle of a VolumeSnapshotContent), and an optional capacity that sizes this actor's volume instead of the template's (the caller computes it; substrate bakes in no default).
  • CreateActor refuses, with the reason, a seed whose volume is not a seeded external volume of the template, whose driver is not the volume's StorageClass provisioner or has no CSIDriverConfig, or whose snapshot the driver does not list as ready (CSI ListSnapshots), and a capacity (the seed's or the template's) below the snapshot's reported size.
  • At the first resume the volume is created through CSI CreateVolume with the snapshot as volume_content_source. A response that does not report that source deletes the volume and fails the resume, so a driver that ignores content sources never hands an actor an empty volume.
  • An actor created without a seed gets neither the volume nor its mounts; its workload is exactly that of the template without the volume.
  • A seeded actor boots from its image instead of restoring the template's golden snapshot: the golden actor has no seed, so its guest state was captured without the seeded mount. An explicit source_tag together with seeds stays refused, as tag cloning of templates with external volumes already is: a tag does not record which seeded mounts its guest state was captured with (upstream's volume-snapshot work, Add snapshotting support for external volumes agent-substrate/substrate#1951).
  • Pause, resume (on another node too) and delete treat the seeded volume like any external volume.
  • The control-plane plugin takes upstream's CreateVolumeRequest/CreateVolumeResponse shape with SourceSnapshotID/ContentSourceSnapshotID from Add volume snapshot support to volume plugins agent-substrate/substrate#1849, and gains GetSnapshot. Unlike Add volume snapshot support to volume plugins agent-substrate/substrate#1849, ContentSourceSnapshotID is the source the driver's response reports, not the one requested, so the data-loss check can fire. This seed shape, with that difference, is proposed upstream in Add snapshotting support for external volumes agent-substrate/substrate#1951 (Add snapshotting support for external volumes agent-substrate/substrate#1951 (comment)).
  • ate-setup setup csi hostpath installs the volume snapshot CRDs and snapshot-controller (external-snapshotter v8.6.0, vendored under hack/third_party/external-snapshotter). pr-workflow sets up both CSI drivers and runs the new seededvolumes suite with E2E_CSI_HOSTPATH=1, where a missing driver fails instead of skipping.
  • resources.DeepEqual, the unchanged-value check of the generated validation, compares a repeated message field with proto.Equal. It used reflect.DeepEqual, which compares the size cache a marshal fills, so the immutable volume_seeds failed every CreateActor with "field is immutable" (found by this PR's e2e; a functional test covers it now). Upstream has the same code.
  • FORK.md row naming the upstream series it follows (Add volume snapshot support to volume plugins agent-substrate/substrate#1849, Add snapshotting support for external volumes agent-substrate/substrate#1951, Add a "preview gate" concept, use it for external volumes agent-substrate/substrate#1994) and its exit; docs/csi-volumes.md section.

Acceptance criteria

  • e2e on the kind CSI hostpath setup: an actor created with a snapshot handle starts with that snapshot's files mounted read-write at the declared path (TestSeededVolumes/StartsWithSnapshotFilesReadWrite)
  • Two actors seeded from the same handle have independent volumes: a file written in one is absent in the other (ActorsFromOneHandleAreIndependent)
  • An actor seeded with a capacity larger than the template's gets a volume of that capacity; one without a capacity gets the template's (SeedCapacityReplacesTheTemplates: the driver's recorded volume size; unit TestCreateActorVolumes_Seeded, TestValidateVolumeSeeds, TestValidateCreateActorRequest). Provisioned size proven on the kind hostpath driver; filesystem size needs a block-backed CSI driver, followed up in test: prove a seeded volume's filesystem size on a block-backed CSI driver #232.
  • Such an actor pauses and resumes with its content intact; deleting it deletes its volume (no PV left) (PauseResumeKeepsContent, DeleteDeletesTheVolume: the driver's data directory no longer holds the volume)
  • An actor of the same template created without a seed has no such volume or mount, and its create request is unchanged (unit tests TestInitialActorVolumes_Seeded, TestWorkloadSpecFromActorTemplateSeededVolume, TestPlugin_CreateVolume; e2e UnseededActorHasNoVolumeOrMount)
  • A seed naming an unknown driver or a missing handle is refused at create with that reason (unit TestValidateVolumeSeeds; e2e RefusesBadSeedsAtCreate)
  • FORK.md row naming the upstream series it follows; released as a line release candidate

Closes #227. Replaces #229 and #230 (same change, split into an upstream-ready plugin commit and the seed API commit, DCO-signed).


This pull request was written by an agent.

teemow added 3 commits October 9, 2026 14:04
The control-plane volume plugin can now seed a new volume from an
existing CSI snapshot and look a snapshot up by its handle.

CreateVolume takes a CreateVolumeRequest and returns a
CreateVolumeResponse. A request's SourceSnapshotID becomes the CSI
CreateVolume volume_content_source. The response's
ContentSourceSnapshotID is the snapshot the driver's response reports as
the volume's content source, not the one requested: a driver that
ignores the content source answers with an empty volume and success, and
only the response tells a restore from that silent data loss.

GetSnapshot maps to ListSnapshots filtered to one handle, the CSI spec
having no single-snapshot read. A handle the driver does not list is
not found; a driver that does not implement ListSnapshots is an error
with code Unimplemented, since it cannot tell either way.

Signed-off-by: Timo Derstappen <teemow@gmail.com>
A session that starts in a prepared workspace needs its own read-write
copy of a snapshot taken outside the actor, chosen when the actor is
created. An ActorTemplate's external volume was created empty for every
actor of the template.

ExternalVolumeTemplate.seeded makes a template volume optional and
seeded: an actor gets it only when CreateActor names a seed for it in
Actor.volume_seeds (volume, CSI driver, snapshot handle), and the volume
is then created from that snapshot. CreateActor refuses a seed for a
volume that is not seeded, a driver that is not the volume's
StorageClass provisioner or is not registered, and a snapshot the
driver does not list as ready. A volume the driver does not report as
restored from the seed is deleted and fails the resume, so an actor is
never handed an empty volume in place of its seed. An actor created
without a seed has neither the volume nor its mounts, and its workload
is that of the template without the volume. A seeded actor boots from
its image rather than restoring the template's golden snapshot, which
was built without the seeded volume's mount.

ate-setup's hostpath CSI setup installs the volume snapshot CRDs and the
snapshot-controller from external-snapshotter v8.6.0. pr-workflow sets
up both CSI drivers and runs the new seededvolumes suite, which takes a
VolumeSnapshot of a filled PVC and seeds actors from its handle.

Signed-off-by: Timo Derstappen <teemow@gmail.com>
teemow added 6 commits October 9, 2026 14:25
DeepEqual handed a slice of proto messages to reflect.DeepEqual, which
compares each message's internal state as well as its fields. The RPC
logger's marshal fills the size cache of a request's messages, so a
repeated message field and its clone compared unequal, and declarative
validation's unchanged-value shortcut then ran the field's update
checks: an immutable repeated field failed every create, because the
create validates the stored object as an update of the request.

A slice of messages is now compared element by element with
proto.Equal, a nil and an empty one being the same field value.

Signed-off-by: Timo Derstappen <teemow@gmail.com>
Signed-off-by: Timo Derstappen <teemow@gmail.com>
Signed-off-by: Timo Derstappen <teemow@gmail.com>
Later subtests use the actor the first one started, so its deletion is registered on the suite's test rather than the subtest's.

Signed-off-by: Timo Derstappen <teemow@gmail.com>
A seeded volume's content differs per actor, so one template capacity cannot fit them all. VolumeSeed.capacity sizes that actor's volume and falls back to the template's when empty; CreateActor refuses a capacity below the size the driver reports for the snapshot.

Signed-off-by: Timo Derstappen <teemow@gmail.com>
@teemow

teemow commented Oct 9, 2026

Copy link
Copy Markdown
Member Author

Closed as superseded: the snapshot-seeded per-actor volume this draft implemented was replaced by the existing-volume-at-a-sub-path design, which landed in #234 (released as v1.7.0-rc.1). Written by an agent.

@teemow teemow closed this Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: mount an existing read-write-many volume at a sub-path

1 participant