Batch dataset import and manifest workbench

[Andesha]

Goal

Allow users to recursively discover and describe a whole dataset, a dataset chunk, or a single participant without manually renaming source files or authoring a manifest first. The batch workbench produces a reusable batch manifest; Configuration loads that manifest and the existing Recording to BIDS workflow consumes it.

This supersedes #86. The broader scope and implementation will be tracked here through sub-issues so the original reporter is not notified for every planning update.

Validated direction

A throwaway prototype compared three workflows. The preferred direction is the dataset workbench (variant B) on branch prototype/issue-86-batch-import, refined around a strict responsibility boundary:

• The batch manifest is the workbench's canonical output and the validated contract passed to the rest of the application.
• The workbench authors recording assignments; it does not own conversion configuration, execution, results, or retry.
• Users can save the manifest to a versioned file or load the same serialized representation directly into Configuration.
• Configuration owns dataset-wide conversion settings and combines them with the active manifest.
• Recording to BIDS owns destination preview, preflight, conversion, per-recording results, and retry.
• Canonical BIDS entities may prefill empty fields, but arbitrary filename or folder semantics are never guessed.
• Required ambiguous values remain unresolved until explicitly assigned.
• Recordings expose participant assignment directly; renaming a participant updates linked recordings.
• Recording metadata and participant demographics are reviewed separately.
• Users can bulk-edit selected recordings, define explicit token mappings, and handle individual exceptions.
• Existing outputs are detected during preflight and never silently overwritten.
• Deliberate replacement of existing recordings is out of scope initially.

Intended workflow

1. Select a potentially nested root folder in the Batch workbench.
2. Recursively discover supported recordings and companion files.
3. Review canonical prefills and their provenance.
4. Assign or rename participants and bulk-correct session/task/run values, including explicit user-defined token mappings for noncanonical names.
5. Review participant-level metadata separately.
6. Save the versioned batch manifest file or load that same manifest directly into Configuration.
7. Adjust dataset-wide settings in Configuration.
8. Use Recording to BIDS to preview destinations, run output preflight, convert ready recordings, and selectively retry failures.

Responsibility and data boundary

The saved manifest file is the canonical handoff contract:

Folder/files
    ↓
Batch workbench
    ├── Save manifest file
    └── Load manifest into Configuration
                         ↓
              Manifest + configuration
                         ↓
                 Recording to BIDS

The manifest describes source recordings, companion files, explicit participant/session/task/run assignments, participant linkage and demographics, inclusion/readiness, and optional assignment provenance. Configuration remains separate and describes how recordings are converted. Loading a manifest into Configuration must use the same versioned serialized representation as saving and reopening it; there must not be a second in-memory-only contract.

Changing configuration must not silently rewrite the source manifest. A future resumable job representation may reference both manifest and configuration, but global configuration values should not be duplicated into every recording row.

Assignment boundary

Automatic prefilling and user-owned mappings must remain deliberately conservative:

1. Parse explicit canonical BIDS entities and prefill only empty fields.
2. Display the provenance of every prefilled value inline.
3. For noncanonical names, let users define and preview filename or relative-path token mappings before applying them.
4. Do not assign semantic meaning to arbitrary tokens or abbreviations, strip prefixes, or treat repeated folder names as metadata without explicit user confirmation.
5. Save the resulting explicit assignments in the manifest so reopening does not require prefilling or rerunning mappings.

A poor automatic guess must create review work, never incorrectly named output. User-defined mappings make human knowledge explicit rather than disguising it as inference.

Planned sub-issues

• Build a manually populated batch workbench #193 — Build a manually populated batch workbench
• Discover recordings recursively from a selected folder #194 — Discover recordings recursively from a selected folder
• Prefill metadata from canonical BIDS entities #195 — Prefill metadata from canonical BIDS entities
• Correct batches through bulk editing and participant review #196 — Correct batches through bulk editing and participant review
• Save and load a versioned batch manifest #199 — Save and load a versioned batch manifest
• Validate and preview the batch in the conversion workflow #197 — Validate and preview the proposed BIDS batch in the conversion workflow
• Convert the active batch with isolated results and retry #198 — Convert ready recordings with isolated results and retry

Prototype verdict

The prototype answered: a dataset workbench can make a large, arbitrarily nested folder manageable without requiring users to manually author a manifest. The production UI should follow variant B, including its explicit participant-to-recording linkage, while treating the workbench as a manifest author rather than a replacement conversion workflow. Prototype code is throwaway and should be rewritten with production state management, validation, and tests.