Skip to content

feat(audio-datasets): make dataset manifest stages agent-ready - #2341

Open
shubhamNvidia wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
shubhamNvidia:agent/datasets
Open

feat(audio-datasets): make dataset manifest stages agent-ready#2341
shubhamNvidia wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
shubhamNvidia:agent/datasets

Conversation

@shubhamNvidia

Copy link
Copy Markdown
Contributor

Contracts only — no behavior change. Adds describe() + *_key params to the FLEURS and ReadSpeech initial-manifest source stages.

These are sources (they produce audio paths + text from nothing), and their contracts declare the two facts a planner has to know up front: they write to disk and they need internet on the first run (when auto_download is on) — so an offline or read-only environment is caught before the run starts rather than partway through a download.

Depends on the agent-ready foundation, #2332 — that must merge first (this branch imports nemo_curator/stages/audio/_agent/, so CI here stays red until it lands).

Adds describe() contracts to the FLEURS and ReadSpeech initial-manifest stages so a planner can tell what each produces before running it.

Depends on the agent-ready foundation (nemo_curator/stages/audio/_agent/), which must merge first.

Signed-off-by: Shubham Bhawsar <shbhawsar@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 25, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@shubhamNvidia
shubhamNvidia marked this pull request as ready for review August 25, 2026 11:19
@shubhamNvidia
shubhamNvidia requested a review from a team as a code owner August 25, 2026 11:19
@shubhamNvidia
shubhamNvidia requested review from weijiac0619 and removed request for a team August 25, 2026 11:19
@greptile-apps

greptile-apps Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Adds agent-facing contracts to the FLEURS and ReadSpeech initial-manifest source stages without changing their dataset-processing paths.

  • Marks both stages as disk-writing, first-run internet-dependent fan-out sources.
  • Exposes configurable audio-path and transcript keys in each stage contract.
  • Declares emitted rows independently reusable for planner purposes.

Confidence Score: 5/5

The contract changes appear safe to merge once the explicitly required agent-ready foundation dependency is present.

The new declarations align with the stages’ emitted keys and existing download and disk-staging behavior, and no unacknowledged actionable defect remains.

Important Files Changed

Filename Overview
nemo_curator/stages/audio/datasets/fleurs/create_initial_manifest.py Adds AgentReady inheritance and a contract matching the stage’s configurable output keys, persistent disk staging, conditional first-run download, and one-to-many output.
nemo_curator/stages/audio/datasets/readspeech/create_initial_manifest.py Adds configurable output keys and an agent contract describing disk staging, conditional download requirements, fan-out cardinality, and the intentionally accepted row-independence policy.

Reviews (1): Last reviewed commit: "feat(audio-datasets): make dataset manif..." | Re-trigger Greptile

@shubhamNvidia
shubhamNvidia requested review from ayushdg, mohammadaaftabv and sarahyurick and removed request for weijiac0619 August 25, 2026 12:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant