Add capacity-aware federated media providers for queued remote generation

[atomantic]

Problem / Goal

PortOS instances can register federated peers and expose /sdapi/v1/* for A1111-compatible image generation, but federation does not currently support remote PortOS-managed audio, video, or image jobs.

The shared media queue already supports audio jobs and Creative Director audio generation, but standalone music generation remains a long-running synchronous request. There is also no provider capability or capacity contract, so a Creative Commission running on one instance cannot safely use a CUDA-only peer such as a MiniMax Music 3 host without risking overload.

For this issue, “audio” means queued music/audio synthesis. Existing voice/TTS APIs remain separate.

Context

• server/services/instances.js probes peer health and server/lib/peerHttpClient.js already provides authenticated peer HTTP transport.
• server/routes/instances.js has a generic read-only query proxy, but no allowlisted state-changing media-job protocol.
• server/services/mediaJobQueue/index.js provides persistence, queue positions, cancellation, watchdogs, and SSE-compatible lifecycle events for image, video, and audio jobs.
• server/routes/music.js currently awaits generateMusic() directly, while server/services/pipeline/musicGen.js already advertises CUDA readiness and the MiniMax Music 3 engine.
• server/services/creative/tools/media.js already exposes queued image, video, and audio tools used by Creative Director and Creative Commission flows.
• server/routes/systemHealth.js reports CPU, memory, disk, PM2, and CoS state, but not GPU/media-lane capacity. Instances.jsx only displays those general health fields.
• server/lib/apiRegistry.js provides the existing opt-in API exposure and authentication model.
• Related but distinct: #4224 covers a native Apple-Silicon MiniMax Music 3 runtime, not federated provider routing.

Proposed approach

1. Add a first-class federated media-provider contract.

   • Add a dedicated, allowlisted API for provider capabilities, capacity/status, job submission, job lookup, cancellation, and result download.
   • Keep it disabled by default and require authenticated peer access for job mutation.
   • Reuse peerFetch, instance IDs, per-peer credentials, Zod validation, and a versioned wire contract.
   • Do not use the arbitrary peer query proxy for media execution.

2. Add provider and consumer configuration.

   • A provider instance can enable media sharing and choose supported kinds/models, queue limits, and admission policy.
   • A consuming instance can enable a peer as a media provider, select allowed kinds/models, and choose local versus peer execution.
   • Provider selection must be explicit and server-validated; an agent must not be able to route arbitrary jobs to arbitrary peers.
   • Keep commission schedules, assignments, run history, CoS ledgers, and other machine-local records local.

3. Extend the media queue with remote execution.

   • Preserve the existing local job shape: queued, running, completed, failed, and canceled.
   • Create a local proxy job on the requesting instance and a normal queued job on the provider instance.
   • Propagate queue position, progress, status, ETA, cancellation, errors, and result provenance.
   • Reconcile remote jobs after restart or temporary peer loss instead of assuming an in-flight job failed.
   • Use idempotency keys so retries cannot duplicate provider jobs.
   • Transfer input/output assets through authenticated, hash-verified asset endpoints; never send filesystem paths across peers.

4. Make music generation use the durable queue.

   • Change /api/music/generate to return a queued 202 response with jobId, position, and status.
   • Persist track/album results from the queue completion path rather than holding the HTTP request open.
   • Update MusicGenPanel to use the shared job progress/cancellation UI.
   • Allow MiniMax Music 3 and other audio models to select a configured remote provider.
   • Keep the existing Creative Director audio lane and music-bed completion hooks working for local and remote jobs.

5. Add capacity-aware admission and reporting.

   • Add local and provider status reporting for:
     • supported media kinds and model IDs;
     • runtime/platform/model readiness;
     • CUDA state, with available, absent, and unknown kept distinct;
     • running and queued jobs by lane/kind;
     • configured concurrency and queue limits;
     • stale timestamps and estimated wait where available.
   • The provider remains authoritative: preflight status is advisory, and every submission rechecks authorization and capacity.
   • A provider must reject jobs with a typed reason when disabled, stale, unavailable, over queue limits, or unsupported.
   • Unknown or stale capacity must block new remote submissions rather than being treated as available.

6. Add UI and integration coverage.

   • Extend the Instances peer card and System Health page with media-provider readiness and queue state.
   • Add provider/model selection and capacity messaging to image, video, music, and Creative Commission flows.
   • Show clear blocked/busy/unavailable states and avoid silently falling back to another provider.
   • Add server, client, queue, authentication, cancellation, restart-reconciliation, asset-transfer, and mixed-version peer tests.
   • Document provider setup, security boundaries, model capability reporting, and migration/backward-compatibility behavior.

Acceptance criteria

• A PortOS instance can be explicitly configured as an authenticated image, video, and/or audio provider.
• Another federated instance can discover supported models, runtime readiness, queue state, and capacity.
• Remote submissions are rejected safely when the provider is disabled, unsupported, stale, unavailable, or over its configured queue limit.
• Remote image, video, and music jobs use the shared queued lifecycle with progress, cancellation, retry/reconciliation, and terminal status.
• No filesystem paths, credentials, prompts in status payloads, private record bodies, or machine-local CoS/commission ledgers cross federation implicitly.
• Input and output assets are transferred through authenticated, integrity-checked APIs.
• /api/music/generate no longer holds a long synchronous request open and supports remote MiniMax Music 3 execution.
• Creative Commission and Creative Director media tools can use the configured remote provider without duplicating scheduled commissions on peers.
• Instances and System Health UIs show provider readiness, active jobs, queue depth, capacity limits, and stale/unknown states.
• Existing /sdapi/v1/*, voice APIs, local generation, record synchronization, and older peers continue to work unchanged.
• No live DB-backed tests are run against the real portos database.

Out of scope

• Automatic weighted load balancing across multiple providers.
• Arbitrary remote shell execution or a general-purpose peer API proxy.
• Remote LLM/CoS task execution.
• Federating Creative Commission schedules, prompts, run history, or CoS ledgers.
• Remote model installation or automatic model downloads.
• Replacing the existing A1111-compatible SDAPI.

Open questions

No blocking questions. Initial decisions:

• First release uses explicit provider/default routing; automatic load balancing can follow once routing policy and idempotent failover are established.
• The new provider API is the PortOS-to-PortOS contract; SDAPI remains for external A1111-compatible clients.
• Provider result retention follows the existing media-job archive window; the requesting instance owns the imported result.

[atomantic]

Swarm implementation scope: this issue spans several independently shippable workstreams. This PR will deliver the provider-side queued-audio foundation for the CUDA/MiniMax use case: a versioned PortOS-to-PortOS contract, disabled-by-default authenticated provider configuration, live model/runtime/CUDA and queue-capacity reporting, registered-peer authorization, idempotent submission, owner-scoped lookup/cancellation, and integrity-described result download, with focused UI, docs, and tests.

The consumer-side proxy executor and restart reconciliation, /api/music/generate queue conversion, remote image/video and input-asset transfer, peer selection, and Instances/System Health integration remain on #4348. The PR will use Refs #4348 and include a ## Remaining section rather than closing the issue.

[atomantic]

Done: provider-side queued audio provider API shipped in PR #4437. Remaining: consumer-side peer proxying and restart reconciliation, provider selection/failover plus fairness/quota policy, durable /api/music/generate queue semantics, remote image/video jobs with versioned asset transfer, and aggregate provider health UI.