Skip to content

API.md

Latest commit

 

History

History
885 lines (719 loc) · 56 KB

API.md

File metadata and controls

885 lines (719 loc) · 56 KB
Raw

API Reference

All workflows accept an optional credentials object for runtime credential injection. This is inherited from the base MuxAIOptions interface and is not repeated for each workflow below.

Scoped execution

Workflows that analyze asset content accept an optional asset-relative scope:

interface WorkflowScope {
  startTime?: number; // Inclusive, in seconds
  endTime?: number; // Exclusive, in seconds
}

Both boundaries are optional. { startTime: 30 } analyzes from 30 seconds through the end of the asset, while { endTime: 90 } analyzes from the beginning up to 90 seconds. Times remain relative to the original asset in workflow outputs.

scope is supported by getSummaryAndTags, getModerationScores, hasBurnedInCaptions, askQuestions, generateChapters, generateEmbeddings, and generateText. It bounds storyboard or thumbnail selection and transcript cues as applicable. The workflow rejects negative, non-finite, empty, reversed, or out-of-asset ranges.

getSummaryAndTags(assetId, options?)

Analyzes a Mux video or audio asset and returns AI-generated metadata.

Parameters:

  • assetId (string) - Mux asset ID (video or audio-only)
  • options (optional) - Configuration options

Options:

  • provider?: 'openai' | 'anthropic' | 'google' | 'baseten' | 'openai-compatible' - AI provider (default: 'openai')
  • tone?: 'neutral' | 'playful' | 'professional' - Analysis tone (default: 'neutral')
  • model?: string - AI model to use. Defaults per provider: gpt-5.6-luna at medium reasoning (OpenAI), claude-sonnet-4-5 (Anthropic), gemini-3-flash-preview (Google). Baseten and OpenAI-compatible endpoints have no default — pass model or set the BASETEN_MODEL / OPENAI_COMPATIBLE_MODEL environment variable.
  • languageCode?: string - Language code for transcript track selection (e.g., 'en', 'fr'). When omitted, prefers English if available.
  • outputLanguageCode?: string - BCP 47 language code (e.g., 'en', 'fr', 'ja') for the generated title, description, and tags. When omitted or set to 'auto', auto-detects from the selected transcript track's language. Falls back to unconstrained (LLM decides) if no language metadata is available.
  • includeTranscript?: boolean - Include transcript in analysis (default: true)
  • cleanTranscript?: boolean - Remove VTT timestamps and formatting from transcript (default: true)
  • imageSubmissionMode?: 'url' | 'base64' - How to submit storyboard to AI providers (default: 'url')
  • imageDownloadOptions?: object - Options for image download when using base64 mode
    • timeout?: number - Request timeout in milliseconds (default: 10000)
    • retries?: number - Maximum retry attempts (default: 3)
    • retryDelay?: number - Base delay between retries in milliseconds (default: 1000)
    • maxRetryDelay?: number - Maximum delay between retries in milliseconds (default: 10000)
    • exponentialBackoff?: boolean - Whether to use exponential backoff (default: true)
  • promptOverrides?: object - Override specific sections of the prompt for custom use cases
    • task?: string - Override the main task instruction
    • title?: string - Override title generation guidance
    • description?: string - Override description generation guidance
    • keywords?: string - Override keywords generation guidance
    • qualityGuidelines?: string - Override quality guidelines

Returns:

interface SummaryAndTagsResult {
  assetId: string;
  title: string; // Short title
  description: string; // Detailed description
  tags: string[]; // Up to 10 relevant keywords
  storyboardUrl?: string; // Video storyboard URL (undefined for audio-only assets)
  usage?: TokenUsage; // Token usage from the AI provider
  transcriptText?: string; // Raw transcript text (when includeTranscript is true)
}

getModerationScores(assetId, options?)

Analyzes a Mux asset for inappropriate content using OpenAI's Moderation API, Hive's Moderation API, or Google Cloud Vision SafeSearch.

By default it moderates everything that's available on the asset and reports what it skipped:

  • Thumbnails — sampled storyboard frames (image moderation). Skipped for audio-only assets.
  • Transcript — the caption track's text, in time windows (text moderation). Skipped when the asset has no ready caption track, the track has nothing to moderate, or includeTranscript is false on a video asset. Only openai supports text moderation; hive and google-vision-api are image-only and report the transcript as skipped.

Parameters:

  • assetId (string) - Mux asset ID (video or audio-only)
  • options (optional) - Configuration options

Options:

  • provider?: 'openai' | 'hive' | 'google-vision-api' - Moderation provider (default: 'openai')
  • model?: string - OpenAI moderation model to use (default: omni-moderation-latest); ignored for hive and google-vision-api
  • languageCode?: string - Language code of the caption track to moderate. When omitted, an English subtitles track is preferred and otherwise the first ready text track is used.
  • includeTranscript?: boolean - Moderate the caption transcript alongside thumbnails (default: true). When false, video assets are moderated on thumbnails only and transcriptModeration is skipped with skipReason: 'disabled'. Ignored for audio-only assets, where the transcript is the only surface that can be moderated.
  • thresholds?: { sexual?: number; violence?: number } - Custom thresholds (default: {sexual: 0.7, violence: 0.8})
  • thumbnailInterval?: number - Seconds between thumbnails for long videos (default: 10)
  • thumbnailWidth?: number - Thumbnail width in pixels (default: 640)
  • maxSamples?: number - Maximum number of thumbnails to sample. Acts as a cap: if thumbnailInterval produces fewer samples than this limit the interval is respected; otherwise samples are evenly distributed with first and last frames pinned. (default: unlimited)
  • maxConcurrent?: number - Maximum concurrent API requests (default: 5)
  • imageSubmissionMode?: 'url' | 'base64' - How to submit images to AI providers (default: 'url')
  • imageDownloadOptions?: object - Options for image download when using base64 mode
    • timeout?: number - Request timeout in milliseconds (default: 10000)
    • retries?: number - Maximum retry attempts (default: 3)
    • retryDelay?: number - Base delay between retries in milliseconds (default: 1000)
    • maxRetryDelay?: number - Maximum delay between retries in milliseconds (default: 10000)
    • exponentialBackoff?: boolean - Whether to use exponential backoff (default: true)

Hive note: this workflow uses Hive for image moderation only. Transcript text is never sent to Hive; with provider: "hive" the transcript surface is reported as skipped (unsupported_provider), and an audio-only asset therefore throws because nothing can be moderated.

Google Vision note: SafeSearch returns the adult, violence, racy, spoof, and medical Likelihood enum values (UNKNOWN..VERY_LIKELY). @mux/ai consumes only adult (mapped to sexual) and violence, and converts the enum onto a 0..1 scale linearly: UNKNOWN=0, VERY_UNLIKELY=0.2, UNLIKELY=0.4, POSSIBLE=0.6, LIKELY=0.8, VERY_LIKELY=1.0. Because exceedsThreshold uses strict >, the default 0.8 threshold treats only VERY_LIKELY as exceeding — drop the threshold to e.g. 0.7 if you want LIKELY to flag. This mapping may change in future versions of @mux/ai.

Returns:

{
  assetId: string;
  // Which surfaces were moderated (derived from thumbnailModeration / transcriptModeration):
  // 'thumbnails' = images only; 'transcript' = transcript text only; 'combined' = both.
  mode: 'thumbnails' | 'transcript' | 'combined';
  isAudioOnly: boolean;
  thumbnailScores: Array<{ // Image (thumbnail) results; empty when thumbnailModeration.status === 'skipped'
    url: string;
    time?: number; // Time in seconds of the thumbnail within the video
    sexual: number; // 0-1 score
    violence: number; // 0-1 score
    error: boolean;
    errorMessage?: string;
  }>;
  transcriptScores: Array<{ // Time-windowed transcript results; empty when transcriptModeration.status === 'skipped'
    startTime: number; // Seconds — start of the moderated time window. Ranges may overlap between consecutive entries by design.
    endTime: number;   // Seconds — end of the moderated time window
    sexual: number; // 0-1 score
    violence: number; // 0-1 score
    error: boolean;
    errorMessage?: string;
  }>;
  thumbnailModeration: { // Audit trail for the thumbnail surface
    // 'completed' — ran and scored at least one thumbnail (others may carry a per-item `error`).
    // 'skipped'   — nothing to moderate (see `skipReason` / `skipMessage`).
    // 'failed'    — ran but every thumbnail errored (see `errorMessage`); contributes no scores.
    status: 'completed' | 'skipped' | 'failed';
    // Present only when status === 'skipped':
    //   'no_video_track'    — the asset has no video track, so there are no thumbnails.
    //   'no_video_in_scope' — the requested `scope` lies entirely past the end of the video track.
    skipReason?: 'no_video_track' | 'no_video_in_scope';
    skipMessage?: string; // Present only when status === 'skipped'. Human-readable explanation.
    errorMessage?: string; // Present only when status === 'failed'. Summarizes the per-item errors.
  };
  transcriptModeration: { // Audit trail for the transcript surface
    // 'completed' — ran and scored at least one window (others may carry a per-item `error`).
    // 'skipped'   — nothing to moderate (see `skipReason` / `skipMessage`).
    // 'failed'    — ran but every window errored (see `errorMessage`); contributes no scores.
    status: 'completed' | 'skipped' | 'failed';
    // Present only when status === 'skipped':
    //   'no_ready_text_track'  — no ready caption/subtitle track (or none matching languageCode).
    //   'no_cues'              — a track exists but produced nothing to moderate: empty VTT body,
    //                            no parseable cues, or no cue inside `scope`. Valid-but-empty, not an error.
    //   'unsupported_provider' — the provider is image-only (hive / google-vision-api).
    //   'disabled'             — `includeTranscript: false` was passed for a video asset.
    skipReason?: 'no_ready_text_track' | 'no_cues' | 'unsupported_provider' | 'disabled';
    skipMessage?: string; // Present only when status === 'skipped'. Human-readable explanation.
    errorMessage?: string; // Present only when status === 'failed'. Summarizes the per-item errors.
  };
  maxScores: { // Highest scores across thumbnails AND all transcript time windows
    sexual: number;
    violence: number;
  };
  coverage: {
    // Counts describe thumbnail sampling only (transcript windows are excluded):
    requestedSampleCount: number;
    successfulSampleCount: number;
    failedSampleCount: number;
    sampleCoverage: number; // 0-1 fraction of requested samples that succeeded
    // The booleans cover both surfaces:
    isPartial: boolean; // true when any thumbnail or transcript window failed but the workflow still returned a result
    isLowConfidence: boolean; // true when thumbnail coverage is thin or a surface that ran failed entirely
  };
  exceedsThreshold: boolean; // true if content should be flagged
  thresholds: { // Threshold values used
    sexual: number;
    violence: number;
  };
  usage?: TokenUsage; // Workflow usage metadata
}

Skipped surfaces are auditable, not silent. The result always carries thumbnailModeration and transcriptModeration, each describing whether that surface was completed or skipped — and, when skipped, a machine-readable skipReason plus a human-readable skipMessage. This lets callers tell "moderated cleanly with no findings" apart from "nothing was moderated" without inspecting an empty scores array. Skips are for expected conditions only (no caption track, an empty track, an image-only provider, includeTranscript: false); a caption track that exists but cannot be fetched is an error, not a skip. If both surfaces end up skipped, the call throws rather than returning exceedsThreshold: false for an asset nothing was checked on.

Failed surfaces are reported, not folded into completed. If a surface ran but every item errored (for example every transcript window was rejected by the provider), its status is failed with an errorMessage, and coverage.isLowConfidence is set because maxScores and exceedsThreshold then reflect only the other surface. If every item on every surface failed, the call throws.

Transcript moderation uses dynamic, overlapping time windows. Each transcriptScores entry is a moderated time window carrying startTime/endTime (in seconds) — directly analogous to how a flagged thumbnail carries its time — so consumers can locate flagged speech on the timeline rather than receiving a single verdict for the whole transcript. Window size scales with the asset's duration (windowSeconds = clamp(duration / 40, 20s, 120s)), and consecutive windows overlap (~15% of the window size, at least 5 seconds) so content that straddles a window boundary is still scored intact in at least one window. Windows are aligned to caption-cue boundaries (a single cue is never split across windows) so the reported timecodes stay accurate. Because windows overlap by design, consecutive transcriptScores entries' [startTime, endTime] ranges may overlap by roughly the overlap amount. Windows are sent to OpenAI as batched array requests (multiple window texts per /v1/moderations call via the array input, with results[] mapped back index-aligned); if a batch is rejected as too large it is split in half and retried down to a single window. When both thumbnails and transcript windows are moderated, mode is 'combined'; maxScores/exceedsThreshold aggregate the highest sexual/violence across thumbnails and all transcript windows.

Completeness and coverage. Before you rely on a result, understand exactly what was scored and how completely:

  • All scored units are returned; scoring never stops at the first flag. Every sampled thumbnail appears in thumbnailScores and every transcript window in transcriptScores, whether or not it crosses a threshold. maxScores and exceedsThreshold are a max-based rollup across all of them (exceedsThreshold is true if any single unit's sexual/violence exceeds its threshold). To find which region triggered a flag, scan the arrays for entries above your threshold.
  • Thumbnail (visual) coverage is sampled, not exhaustive. Frames are sampled at thumbnailInterval (default every 10s), capped by maxSamples. Content between sampled frames is not inspected, so a result with no visual flag does not guarantee every frame is clean; lower thumbnailInterval or raise maxSamples for denser coverage. The coverage counts (requestedSampleCount, successfulSampleCount, failedSampleCount, sampleCoverage) describe thumbnail sampling only — transcript windows are excluded from the denominator. isPartial and isLowConfidence cover both surfaces: isPartial is true if any thumbnail or transcript window failed, and isLowConfidence signals thin thumbnail sampling or a surface that failed entirely, without making thumbnail coverage exhaustive.
  • Transcript (text) coverage is exhaustive. Every caption cue falls in at least one time window and windows overlap, so all spoken content is moderated. Because windows overlap, the same flagged utterance near a boundary can appear in two adjacent transcriptScores entries with overlapping [startTime, endTime] ranges; merge/deduplicate overlapping flagged ranges if you want one finding per utterance.

hasBurnedInCaptions(assetId, options?)

Analyzes video frames to detect burned-in captions (hardcoded subtitles) that are permanently embedded in the video image.

Parameters:

  • assetId (string) - Mux video asset ID
  • options (optional) - Configuration options

Options:

  • provider?: 'openai' | 'anthropic' | 'google' | 'baseten' | 'openai-compatible' - AI provider (default: 'openai')
  • model?: string - AI model to use. Defaults per provider: gpt-5.6-luna at medium reasoning (OpenAI), claude-sonnet-4-5 (Anthropic), gemini-3-flash-preview (Google). Baseten and OpenAI-compatible endpoints have no default — pass model or set the BASETEN_MODEL / OPENAI_COMPATIBLE_MODEL environment variable.
  • imageSubmissionMode?: 'url' | 'base64' - How to submit storyboard to AI providers (default: 'url')
  • imageDownloadOptions?: object - Options for image download when using base64 mode
    • timeout?: number - Request timeout in milliseconds (default: 10000)
    • retries?: number - Maximum retry attempts (default: 3)
    • retryDelay?: number - Base delay between retries in milliseconds (default: 1000)
    • maxRetryDelay?: number - Maximum delay between retries in milliseconds (default: 10000)
    • exponentialBackoff?: boolean - Whether to use exponential backoff (default: true)
  • promptOverrides?: object - Override specific sections of the detection prompt
    • task?: string - Override the main analysis task instruction
    • analysisSteps?: string - Override the step-by-step analysis procedure
    • positiveIndicators?: string - Override criteria for classifying text as captions
    • negativeIndicators?: string - Override criteria for ruling out captions

Returns:

{
  assetId: string;
  hasBurnedInCaptions: boolean; // Whether burned-in captions were detected
  confidence: number; // Confidence score (0.0-1.0)
  detectedLanguage: string | null; // Language of detected captions, or null
  storyboardUrl: string; // URL to analyzed storyboard
  usage?: TokenUsage; // Token usage from the AI provider
}

Detection Logic:

  • Analyzes video storyboard frames to identify text overlays
  • Distinguishes between actual captions and marketing/end-card text
  • Text appearing only in final 1-2 frames is classified as marketing copy
  • Caption text must appear across multiple frames throughout the timeline
  • Optimized prompts minimize false positives

askQuestions(assetId, questions, options?)

Answer questions about asset content by analyzing storyboard frames and optional transcripts. For audio-only assets, this workflow analyzes transcript text only. By default, answers are "yes"/"no", but you can override the allowed responses.

Parameters:

  • assetId (string) - Mux asset ID (video or audio-only)
  • questions (array) - Array of question objects
    • Each question object must have a question field (string)
    • Each question may optionally include answerOptions?: string[] (defaults to ["yes", "no"])
    • Each question may optionally include freeFormReply?: boolean (experimental — see below)
    • Example: [{ question: "What is the production quality?", answerOptions: ["amateur", "semi-pro", "professional"] }]
  • options (optional) - Configuration options

Options:

  • provider?: 'openai' | 'anthropic' | 'google' | 'baseten' | 'openai-compatible' - AI provider (default: 'openai')
  • model?: string - AI model to use. Defaults per provider: gpt-5.6-luna at medium reasoning (OpenAI), claude-sonnet-4-5 (Anthropic), gemini-3-flash-preview (Google). Baseten and OpenAI-compatible endpoints have no default — pass model or set the BASETEN_MODEL / OPENAI_COMPATIBLE_MODEL environment variable.
  • languageCode?: string - Language code for transcript track selection (e.g., 'en', 'fr'). When omitted, prefers English if available.
  • includeTranscript?: boolean - Include transcript in analysis (default: true, required for audio-only assets)
  • cleanTranscript?: boolean - Remove VTT timestamps and formatting from transcript (default: true)
  • imageSubmissionMode?: 'url' | 'base64' - How to submit storyboard to AI providers (default: 'url')
  • imageDownloadOptions?: object - Options for image download when using base64 mode
    • timeout?: number - Request timeout in milliseconds (default: 10000)
    • retries?: number - Maximum retry attempts (default: 3)
    • retryDelay?: number - Base delay between retries in milliseconds (default: 1000)
    • maxRetryDelay?: number - Maximum delay between retries in milliseconds (default: 10000)
    • exponentialBackoff?: boolean - Whether to use exponential backoff (default: true)
  • storyboardWidth?: number - Storyboard resolution in pixels (default: 640)
  • maxFreeFormAnswerLength?: number - Experimental. Maximum character length for free-form answers when a question sets freeFormReply: true (default: 500). Keep low to bound the open-ended output channel.

Returns:

interface AskQuestionsResult {
  assetId: string;
  answers: Array<{
    question: string; // The original question
    answer: string | null; // Answer from allowed options (null when skipped)
    confidence: number; // Confidence score (0.0-1.0)
    reasoning: string; // AI's explanation based on observable evidence or why the question was skipped
    skipped: boolean; // True when the question was not answerable from the asset content
  }>;
  storyboardUrl?: string; // URL to analyzed storyboard (undefined for audio-only assets)
  usage?: TokenUsage; // Token usage from the AI provider
  transcriptText?: string; // Raw transcript (when includeTranscript is true)
}

Examples:

// Single question
const result = await askQuestions("asset-id", [
  { question: "Does this video contain cooking?" }
]);

console.log(result.answers[0].answer); // "yes" or "no" by default
console.log(result.answers[0].confidence); // 0.95
console.log(result.answers[0].reasoning); // "A chef prepares ingredients..."

// Multiple questions (efficient single API call)
const result = await askQuestions("asset-id", [
  { question: "Does this video contain people?" },
  { question: "Is this video in color?" },
  { question: "Does this video contain violence?" }
]);

// Without transcript (visual-only analysis)
const result = await askQuestions("asset-id", questions, {
  includeTranscript: false
});

// Per-question answer options — mix yes/no with classification scales
const result = await askQuestions("asset-id", [
  { question: "Does this contain cooking?" }, // answer options default to yes/no
  { question: "What is the production quality?", answerOptions: ["amateur", "semi-pro", "professional"] },
  { question: "What is the primary content type?", answerOptions: ["tutorial", "entertainment", "news", "advertisement"] },
  { question: "What is the overall sentiment?", answerOptions: ["positive", "neutral", "negative"] },
]);

Tips for Effective Questions:

  • Be specific and focused on observable evidence
  • Frame questions positively (prefer "Is X present?" over "Is X not present?")
  • Avoid ambiguous or subjective questions
  • Questions should have clear answers that map to your allowed options
  • The AI prioritizes visual evidence when transcript and visuals conflict

⚠️ Experimental: freeFormReply enables open-ended answers for a given question. This bypasses the enum schema and allows the model to reply with prose instead of yes/no or answerOptions — use it only when your use case genuinely needs open-ended answers (e.g. describing a scene, extracting a quoted line) and treat the answer as untrusted model output. Mutually exclusive with answerOptions (setting both throws). The length cap (maxFreeFormAnswerLength, default 500 chars) and the output-safety scrubber still apply. API shape may change.

// Experimental: free-form answers for a single question
const result = await askQuestions("asset-id", [
  { question: "Is this video about glasses?" }, // defaults to yes/no
  {
    question: "Describe the primary subject of this video in one sentence.",
    freeFormReply: true,
  },
], { maxFreeFormAnswerLength: 300 });

console.log(result.answers[1].answer); // e.g. "A pair of tortoiseshell glasses..."

generateEngagementInsights(assetId, options?)

Generate AI-powered insights explaining viewer engagement patterns by analyzing hotspot data, heatmap statistics, visual frames, and transcripts.

Parameters:

  • assetId (string) - Mux asset ID
  • options (optional) - Configuration options

Options:

  • provider?: 'openai' | 'anthropic' | 'google' | 'baseten' | 'openai-compatible' - AI provider (default: 'openai')

  • model?: string - AI model to use. Defaults per provider: gpt-5.6-luna at medium reasoning (OpenAI), claude-sonnet-4-5 (Anthropic), gemini-3-flash-preview (Google). Baseten and OpenAI-compatible endpoints have no default — pass model or set the BASETEN_MODEL / OPENAI_COMPATIBLE_MODEL environment variable.

  • hotspotLimit?: number - Number of engagement moments to analyze per direction (default: 5, range: 1-10). Note: actual moment count may be up to 2x this value since both peaks and valleys are fetched.

  • timeframe?: string - Engagement data timeframe (default: '7:days')

    • Examples: '60:minutes', '24:hours', '7:days', '30:days'

  • skipShots?: boolean - Skip shots integration, use thumbnails instead (default: false). Recommended for latency-sensitive use cases.

Returns:

interface EngagementInsightsResult {
  assetId: string;
  momentInsights: Array<{
    startMs: number; // Start time in milliseconds
    endMs: number; // End time in milliseconds
    timestamp: string; // Human-readable timestamp (e.g., "2:15")
    engagementScore: number; // Normalized score (0.0-1.0)
    insight: string; // Explanation of engagement pattern
  }>;
  overallInsight: {
    summary: string; // Overall engagement summary
    trends: string[]; // Key trends identified
  };
  usage?: { // Token usage statistics
    inputTokens: number;
    outputTokens: number;
    totalTokens: number;
  };
}

Examples:

// Basic usage - informational insights
const result = await generateEngagementInsights("asset-id");

result.momentInsights.forEach(m => {
  console.log(`${m.timestamp}: ${m.insight}`);
});

// Custom timeframe
const result = await generateEngagementInsights("asset-id", {
  timeframe: "30:days",
  hotspotLimit: 5,
});

console.log(result.overallInsight.summary);
console.log("Trends:", result.overallInsight.trends);

// Low-latency mode (skip shots polling)
const result = await generateEngagementInsights("asset-id", {
  skipShots: true,
});

Requirements:

  • Newer or low-view videos may not have sufficient engagement data
  • Works with both video and audio-only assets (audio-only skips visual analysis)

Use Cases:

  • Content optimization based on viewer behavior
  • Understanding what drives re-watching and engagement
  • Identifying pacing issues and drop-off points
  • A/B testing video variations
  • Providing engagement feedback to content creators

translateCaptions(assetId, trackId, toLanguageCode, options?)

Translates existing captions from one language to another and optionally adds them as a new track to the Mux asset. The source language is inferred from the track's metadata.

Parameters:

  • assetId (string) - Mux asset ID (video or audio-only)
  • trackId (string) - ID of the source caption track to translate
  • toLanguageCode (string) - Target language code (e.g., 'es', 'fr', 'de')
  • options - Configuration options

Options:

  • provider: 'openai' | 'anthropic' | 'google' | 'baseten' | 'openai-compatible' - AI provider (required)
  • model?: string - Model to use (defaults to the provider's chat model if omitted)
  • uploadToMux?: boolean - Whether to upload translated track to Mux (default: true)
  • replaceExistingTracks?: 'fail' | 'replace_all' | 'replace_generated' - What to do when the asset already has a text track in the target language or with the target name (default: 'fail', which rejects before any translation runs). 'replace_all' deletes every such track first; 'replace_generated' deletes only Mux-generated (ASR) tracks and rejects if anything else is in the way.
  • trackName?: string - Name for the created Mux text track (default: " (Auto-translated)", e.g. "Spanish (Auto-translated)")
  • trackPassthrough?: string - passthrough written on the created track, max 255 characters (default: {"mux_ai":{"workflow":"translate-captions"}})
  • s3Endpoint?: string - S3-compatible storage endpoint
  • s3Region?: string - S3 region (default: 'auto')
  • s3Bucket?: string - S3 bucket name
  • storageAdapter?: StorageAdapter - Optional adapter with putObject and createPresignedGetUrl methods
  • s3SignedUrlExpirySeconds?: number - Expiry duration in seconds for S3 presigned GET URLs (default: 86400 / 24 hours)
  • chunking?: object - Optional VTT-aware chunking controls for large caption translations
    • enabled?: boolean - Set to false to translate all cues in a single structured request (default: true)
    • minimumAssetDurationSeconds?: number - Prefer a single request until the asset is at least this long (default: 1800)
    • targetChunkDurationSeconds?: number - Soft target for chunk duration once chunking starts (default: 1800)
    • maxConcurrentTranslations?: number - Max number of concurrent translation requests when chunking (default: 4)
    • maxCuesPerChunk?: number - Hard cap for cues included in a single AI translation chunk (default: 80)
    • maxCueTextTokensPerChunk?: number - Approximate cap for cue text tokens included in a single AI translation chunk (default: 2000)
  • neverTranslate?: string[] - Terms (brand names, proper nouns) to preserve verbatim in the translated output; max 100 terms of 100 characters each, < and > not allowed. Compliance is verified and reported on result.neverTranslateTermsPreserved, not guaranteed.

Returns:

interface TranslationResult {
  assetId: string;
  trackId: string; // Source track ID
  sourceLanguageCode: string; // Inferred from track metadata
  targetLanguageCode: string;
  sourceLanguage: LanguageCodePair; // { iso639_1: string; iso639_3: string }
  targetLanguage: LanguageCodePair; // { iso639_1: string; iso639_3: string }
  originalVtt: string; // Original VTT content
  translatedVtt: string; // Translated VTT content
  uploadedTrackId?: string; // Mux track ID (if uploaded)
  replacedTracks?: TextTrackSummary[]; // Existing tracks deleted before the upload ({ id, name, languageCode, status, textSource, passthrough })
  presignedUrl?: string; // S3 presigned URL (default expiry: 24 hours)
  usage?: TokenUsage; // Token usage from the AI provider
  neverTranslateTermsPreserved?: boolean; // Present when neverTranslate terms were supplied; false if any term was not preserved verbatim
}

Supported Languages: All ISO 639-1 language codes are automatically supported using Intl.DisplayNames. Examples: Spanish (es), French (fr), German (de), Italian (it), Portuguese (pt), Polish (pl), Japanese (ja), Korean (ko), Chinese (zh), Russian (ru), Arabic (ar), Hindi (hi), Thai (th), Swahili (sw), and many more.

Chunking Behavior:

  • Chunking is enabled by default for translateCaptions
  • Shorter assets are translated in a single request until minimumAssetDurationSeconds is reached
  • When chunking is active, requests stay aligned to VTT cues and the final VTT is rebuilt locally
  • Chunk size is bounded by both cue count and approximate cue text token budget

editCaptions(assetId, trackId, options)

Edits a caption track using LLM-powered profanity censorship, static find/replace, speaker-label replacement, or a combination of these operations. Optionally uploads the edited track to Mux.

Parameters:

  • assetId (string) - Mux asset ID (video or audio-only)
  • trackId (string) - ID of the caption track to edit
  • options - Configuration options

Options:

  • provider?: 'openai' | 'anthropic' | 'google' | 'baseten' | 'openai-compatible' - AI provider (required when autoCensorProfanity is set)
  • model?: string - Model to use (defaults to the provider's chat model if omitted)
  • autoCensorProfanity?: object - LLM-powered profanity censorship (optional)
    • mode?: 'blank' | 'remove' | 'mask' - Replacement strategy (default: 'blank')
      • 'blank': shit → [____] (bracketed underscores matching word length)
      • 'remove': word removed entirely
      • 'mask': shit → ???? (question marks matching word length)
    • alwaysCensor?: string[] - Words to always censor regardless of LLM output
    • neverCensor?: string[] - Words to never censor even if the LLM flags them (takes precedence over alwaysCensor)
  • replacements?: Array<{ find: string; replace: string; caseSensitive?: boolean }> - Static find/replace pairs (optional, no LLM needed). Each entry matches case-sensitively by default; set caseSensitive: false to match regardless of case.
  • speakerReplacements?: Array<{ find: string; replace: string }> - Replaces bracketed speaker labels at the start of cues without changing matching words in spoken caption content.
  • uploadToMux?: boolean - Whether to upload edited track to Mux (default: true)
  • replaceExistingTracks?: 'replace' | 'fail' - What to do with the source track (default: 'replace': the source is deleted and the edited track takes its place under the source's name; no other track is touched). 'fail' deletes nothing and adds the edited track alongside the source, so it requires a trackName different from the source's. With either value, editCaptions rejects before editing if a track other than the source already has the edited track's name, since Mux requires unique track names. Other tracks in the same language are never touched.
  • trackName?: string - Name for the edited Mux text track (default: the source track's name)
  • trackPassthrough?: string - passthrough written on the edited track, max 255 characters (default: {"mux_ai":{"workflow":"edit-captions"}})
  • deleteOriginalTrack?: boolean - Deprecated, use replaceExistingTracks. When set, the previous behaviour applies: create <source name> (<trackNameSuffix>), then delete the source if true. Cannot be combined with replaceExistingTracks or trackName.
  • trackNameSuffix?: string - Deprecated, use trackName. Selects the previous naming scheme (default suffix 'edited', e.g. "Subtitles (edited)"). Cannot be combined with replaceExistingTracks or trackName.
  • s3Endpoint?: string - S3-compatible storage endpoint
  • s3Region?: string - S3 region (default: 'auto')
  • s3Bucket?: string - S3 bucket name
  • storageAdapter?: StorageAdapter - Optional adapter with putObject and createPresignedGetUrl methods
  • s3SignedUrlExpirySeconds?: number - Expiry duration in seconds for S3 presigned GET URLs (default: 86400 / 24 hours)

At least one of autoCensorProfanity, replacements, or speakerReplacements must be provided.

Returns:

interface ReplacementRecord {
  cueStartTime: number; // Start time of the cue where the replacement occurred (seconds)
  before: string; // Original word/phrase
  after: string; // Replacement text
}

interface EditCaptionsResult {
  assetId: string;
  trackId: string;
  originalVtt: string; // Original VTT content
  editedVtt: string; // Edited VTT content
  totalReplacementCount: number; // Total replacements across all operations
  autoCensorProfanity?: { // Present when autoCensorProfanity was used
    replacements: ReplacementRecord[]; // Each censored word with cue timing
  };
  replacements?: { // Present when replacements were used
    replacements: ReplacementRecord[]; // Each static replacement with cue timing
  };
  speakerReplacements?: { // Present when speakerReplacements were used
    replacements: ReplacementRecord[]; // Each speaker-label replacement with cue timing
  };
  uploadedTrackId?: string; // Mux track ID (if uploaded)
  replacedTracks?: TextTrackSummary[]; // Existing tracks deleted before the upload, including the source when replaced in place
  presignedUrl?: string; // S3 presigned URL (default expiry: 24 hours)
  usage?: TokenUsage; // Token usage (only present if LLM was used)
}

generateChapters(assetId, options?)

Generates AI-powered chapter markers by analyzing video or audio transcripts. Creates logical chapter breaks based on topic changes and content transitions.

Parameters:

  • assetId (string) - Mux asset ID (video or audio-only)
  • options (optional) - Configuration options

Options:

  • languageCode?: string - Language code for captions (e.g., 'en', 'es', 'fr'). When omitted, prefers English if available.
  • outputLanguageCode?: string - BCP 47 language code (e.g., 'en', 'fr', 'ja') for the generated chapter titles. When omitted or set to 'auto', auto-detects from the selected transcript track's language. Falls back to unconstrained (LLM decides) if no language metadata is available.
  • provider?: 'openai' | 'anthropic' | 'google' | 'baseten' | 'openai-compatible' - AI provider (default: 'openai')
  • model?: string - AI model to use. Defaults per provider: gpt-5.6-luna at medium reasoning (OpenAI), claude-sonnet-4-5 (Anthropic), gemini-3-flash-preview (Google). Baseten and OpenAI-compatible endpoints have no default — pass model or set the BASETEN_MODEL / OPENAI_COMPATIBLE_MODEL environment variable.
  • promptOverrides?: object - Override specific sections of the chaptering prompt
    • task?: string - Override the main task instruction
    • outputFormat?: string - Override the expected output format description
    • chapterGuidelines?: string - Override chapter count and formatting guidelines
    • titleGuidelines?: string - Override chapter title style guidelines
  • minChaptersPerHour?: number - Minimum chapters to generate per hour of content (default: 3)
  • maxChaptersPerHour?: number - Maximum chapters to generate per hour of content (default: 8)

Returns:

{
  assetId: string;
  languageCode?: string; // Resolved from input or track metadata
  chapters: Array<{
    startTime: number; // Chapter start time in seconds
    title: string; // Descriptive chapter title
  }>;
  usage?: TokenUsage; // Token usage from the AI provider
}

Requirements:

  • Asset must have a ready caption/transcript track
  • When languageCode is omitted, prefers an English track if available
  • Uses existing auto-generated or uploaded captions/transcripts

Example Output:

// Perfect format for Mux Player
player.addChapters([
  { startTime: 0, title: "Introduction and Setup" },
  { startTime: 45, title: "Main Content Discussion" },
  { startTime: 120, title: "Conclusion" }
]);

generateText(assetId, options)

Writes a set of source-grounded text artifacts from a video or audio asset. Extracts one editorial brief from the transcript (plus a scoped storyboard for video assets), then writes every variant × artifact combination from that brief in parallel.

Parameters:

  • assetId (string) - Mux asset ID (video or audio-only, must have a ready caption track)
  • options - Configuration options

Options:

  • artifacts: GenerateTextArtifact[] - Deliverables to write for every variant (1-5, unique lowercase snake_case keys). Each artifact accepts optional instructions (up to 500 characters).
    • { key, channel?, maxLength?, format?, instructions? }
    • channel is one of generic (default), x, linkedin, facebook, instagram, tiktok, youtube. It guides the writing conventions and sets the default cap (300 words for generic).
    • maxLength is { unit: "characters", value: 10-20000 } or { unit: "words", value: 5-3000 }. x artifacts are always held to 280 characters, even when maxLength is expressed in words.
    • format is "plain" (default, no Markdown syntax) or "markdown".
  • variants?: Array<{ key: string; instructions?: string }> - Named versions of the complete artifact set (1-5, unique keys). Defaults to [{ key: "default" }]. Omit instructions for an independent take on the same brief.
  • provider?: 'openai' | 'anthropic' | 'google' | 'baseten' | 'openai-compatible' - AI provider (default: 'openai')
  • model?: string - AI model to use. Defaults per provider: gpt-5.6-luna at medium reasoning (OpenAI), claude-sonnet-4-5 (Anthropic), gemini-3-flash-preview (Google). Baseten and OpenAI-compatible endpoints have no default — pass model or set the BASETEN_MODEL / OPENAI_COMPATIBLE_MODEL environment variable.
  • audience?: string - Intended reader, used as best-effort guidance (up to 160 characters)
  • voice?: 'conversational' | 'editorial' | 'playful' | 'professional' - Best-effort writing voice for every artifact
  • callToAction?: 'none' | 'soft' | 'direct' - Whether and how the text should invite the reader to engage further
  • brandTerms?: string[] - Brand or domain terms to use exactly when the source supports them (1-10 terms, 40 characters each, 240 combined)
  • useShots?: boolean - Attach up to 24 evenly sampled shot frames as extra visual evidence when the asset's shots are already generated. When they are not ready, the workflow uses the storyboard alone. Generate shots beforehand with the shots primitives if you need them (default: false, video assets only)
  • languageCode?: string - BCP 47 language code for transcript track selection. When omitted, prefers English if available.
  • outputLanguageCode?: string - BCP 47 language code for the generated text. When omitted or 'auto', follows the selected transcript track's language when it is reliable.
  • scope?: WorkflowScope - Asset-relative execution window. Bounds transcript cues, the storyboard, and shot selection.

Returns:

interface GenerateTextResult {
  assetId: string;
  variants: Array<{
    key: string;
    artifacts: Array<{
      key: string;
      content: string; // Empty when suppressed by the output-safety scrubber
    }>;
  }>;
  storyboardUrl?: string; // Storyboard attached to the brief (undefined for audio-only assets)
  usage?: TokenUsage; // Aggregate usage across the brief and every artifact
  safety?: SafetyReport; // Output-side scrubbing report
}

Errors:

  • validation_error - Invalid option bounds, malformed language tags, missing transcript, empty scope, or useShots on an audio-only asset. Thrown before any model call.
  • processing_error - An artifact was still empty after its generation retry (retryable), bounded paragraph repair failed to meet all length caps within two calls (retryable), the editorial brief was suppressed by the output-safety scrubber (retryable), or the model stopped before producing a complete response. Output-limit truncation is non-retryable; provider failures retain their retryability.
  • content_policy_error - The provider blocked the transcript or an image under its content policy, or refused a paragraph repair (non-retryable).

Paragraph repair preserves typed errors' public classification and retryability, as well as attempt usage. It disables AI SDK transport retries and stops on provider failure; callers own transient retry/backoff at the job orchestration level and must respect the failure's retryability without expanding the two-call repair budget.

translateAudio(assetId, toLanguageCode, options?)

Creates dubbed audio from a Mux asset using ElevenLabs voice cloning and translation. The workflow uses the asset's default audio track and ElevenLabs' dubbing_v1 model. Source language is detected automatically unless fromLanguageCode is provided.

Downloaded audio keeps the format supplied by ElevenLabs, such as FLAC. The stored file's extension and Content-Type match that format. The result's dubbingId is an ElevenLabs project ID and can be used with the ElevenLabs project API.

ElevenLabs charges for the first target language when a project is created. The workflow does not automatically retry project creation. If a creation request times out or its response is lost, check your ElevenLabs account before starting another run to avoid creating and paying for a second project. Status checks and downloads retain workflow retries. Individual ElevenLabs API requests have a four-minute timeout; dubbingPollTimeoutSeconds controls the separate wait for dubbing to finish.

Parameters:

  • assetId (string) - Mux asset ID (video or audio-only; must contain audio). The workflow requests an audio static rendition if needed, or uses existing files for assets with the deprecated mp4_support option.
  • toLanguageCode (string) - Target language tag, such as es, ur, yue, or pt-BR. Known ISO 639-3 aliases, such as spa and urd, are also accepted. Region and script subtags are retained on Mux tracks; dubbing uses the primary language. See supported languages below.
  • options (optional) - Configuration options

Options:

  • provider?: 'elevenlabs' - AI provider (default: 'elevenlabs')
  • fromLanguageCode?: string - Source language tag, such as en or en-US. Known ISO 639-3 aliases, such as eng, are also accepted. Region and script subtags are dropped for ElevenLabs. Omit or use "auto" for automatic detection.
  • numSpeakers?: number - Must be 0 for automatic speaker detection (default). Explicit speaker counts are not supported.
  • uploadToMux?: boolean - Whether to upload dubbed track to Mux (default: true). Mux does not support adding audio tracks to clips, assets from a live stream (including finished recordings), or audio-only assets, so with true those are rejected with a validation_error before dubbing starts; pass uploadToMux: false, uploadToS3: true to dub them and use the file from presignedUrl instead.
  • uploadCaptionsToMux?: boolean - Also attach the dub's translated transcript as a subtitles text track (default: false). Independent of uploadToMux, which controls audio only; caption upload implies S3 staging even when audio upload is disabled.
  • captionsTrackName?: string - Subtitle name override; defaults to trackName, then the generated audio track name.
  • captionsReplaceExistingTracks?: 'fail' | 'replace_all' | 'replace_generated' - Subtitle conflict policy override; defaults to replaceExistingTracks. Conflicts fail only subtitle attachment and are returned in captionsOutcome; the dubbed audio and staged downloads remain available.
  • replaceExistingTracks?: 'fail' | 'replace_all' | 'replace_generated' - What to do when the asset already has an audio track in the target language or with the target name (default: 'fail', which rejects before dubbing starts). 'replace_all' deletes them first. Audio tracks are never Mux-generated, so 'replace_generated' only ever removes ASR captions. captionsReplaceExistingTracks can override this policy for subtitles. The asset's primary audio track is never deleted under any policy.
  • trackName?: string - Name for the created Mux tracks; audio and text tracks are separate name groups so both get the same name unless captionsTrackName overrides the subtitle name (default: "<Language> (Auto-dubbed)")
  • trackPassthrough?: string - passthrough written on the created tracks, max 255 characters (default: {"mux_ai":{"workflow":"translate-audio"}})
  • s3Endpoint?: string - S3-compatible storage endpoint
  • s3Region?: string - S3 region (default: 'auto')
  • s3Bucket?: string - S3 bucket name
  • storageAdapter?: StorageAdapter - Optional adapter with putObject and createPresignedGetUrl methods
  • s3SignedUrlExpirySeconds?: number - Expiry duration in seconds for S3 presigned GET URLs (default: 86400 / 24 hours)
  • dubbingPollTimeoutSeconds?: number - Max time to wait for ElevenLabs to finish dubbing before timing out (default: 7200 / 2 hours). Raise for long-form content or when jobs queue behind the concurrency limit.
  • staticRenditionCleanup?: 'delete' | 'keep' - What to do with an audio.m4a static rendition the workflow created as dubbing input (default: 'delete'). A rendition that already existed on the asset is never deleted. When dubbing multiple languages concurrently on one asset, create the rendition before fanning out or pass 'keep' — the creating run's delete can otherwise race a concurrent run's source fetch.

Returns:

interface TranslateAudioResult {
  assetId: string;
  targetLanguageCode: string;
  targetLanguage: LanguageCodePair; // { iso639_1: string; iso639_3: string }
  dubbingId: string; // ElevenLabs project ID
  uploadedTrackId?: string; // Mux audio track ID (if uploaded)
  presignedUrl?: string; // S3 presigned audio URL (default expiry: 24 hours)
  replacedTracks?: TextTrackSummary[]; // Existing audio/text tracks deleted before the uploads; `type` says which group
  captionsTrackId?: string; // Mux text track ID for the dubbed captions (when uploadCaptionsToMux is true)
  captionsOutcome?: DubbedCaptionsOutcome; // available, unavailable, created, or failed
  captionsPresignedUrl?: string; // S3 presigned URL for the dub transcript VTT
  createdStaticRenditionId?: string; // Static rendition ID this run created (undefined if one already existed)
  staticRenditionCleanup: "deleted" | "delete_failed" | "kept" | "not_created";
  usage?: TokenUsage; // Workflow usage metadata
}

translateAudio reports transcript staging and attachment through captionsOutcome whenever staging is attempted. unavailable includes message and retryable; created includes trackId and replacedTracks; failed includes message, retryable, and replacedTracks, including deletions before failed creation. Subtitle failures preserve the completed dub and any staged download URLs. Outcome messages are safe for callers; provider and storage diagnostics are logged separately.

Requirements:

  • Asset must have an audio.m4a static rendition (auto-requested if missing, and deleted again afterwards by default — see staticRenditionCleanup)
  • Assets on the deprecated mp4_support option are dubbed from their existing files, never modified: audio.m4a for audio-only and audio-only,capped-1080p, capped-1080p.mp4 for capped-1080p, and low.mp4 for standard. A file over 1 GB is rejected before dubbing; move the asset to the Static Renditions API and add an audio-only rendition instead.
  • An audio track to dub (video-only assets are rejected with a validation_error)
  • ElevenLabs API key with access to Dubbing v1
  • S3-compatible storage for Mux ingestion

Supported Languages:

See the Dubbing v1 language list for available targets. Language tags include es (Spanish), ur (Urdu), and yue (Cantonese). BCP-47 primary language subtags can have two or three letters. Known ISO 639-3 aliases are normalized automatically, for example urd becomes ur.

The workflow drops region and script subtags for ElevenLabs while retaining the requested tag on Mux tracks. For example, pt-BR requests Portuguese dubbing and produces a Mux track tagged pt-br. These qualifiers do not select a regional dialect in the dubbing_v1 model. ElevenLabs validates the primary target language when the project is created, after any required Mux rendition preparation.

targetLanguageCode contains the normalized requested tag, including any region or script subtags. The existing targetLanguage.iso639_1 result field also contains that tag, including three-letter values such as yue when no two-letter code exists. Track names are generated from the language name, such as "Spanish (Auto-dubbed)", unless trackName is provided.

generateEmbeddings(assetId, options?)

Generate vector embeddings for transcript chunks from video or audio assets for semantic search.

Deprecated: generateVideoEmbeddings is deprecated. Use generateEmbeddings instead.

Parameters:

  • assetId (string) - Mux asset ID (video or audio-only)
  • options (optional) - Configuration options

Options:

  • provider?: 'openai' | 'google' | 'baseten' | 'openai-compatible' - Embedding provider (default: 'openai')
  • model?: string - Embedding model to use. Defaults per provider: text-embedding-3-small (OpenAI), gemini-embedding-001 (Google). Baseten and OpenAI-compatible endpoints have no default — pass model or set the BASETEN_EMBEDDING_MODEL / OPENAI_COMPATIBLE_EMBEDDING_MODEL environment variable.
  • chunkingStrategy?: object - How to chunk the transcript
    • type: 'token' | 'vtt' - Chunking method
    • maxTokens?: number - Maximum tokens per chunk (default: 500)
    • overlap?: number - Token overlap between chunks (for type: 'token', default: 100)
    • overlapCues?: number - VTT cue overlap between chunks (for type: 'vtt', default: 2)
  • languageCode?: string - Language code for transcript track selection. When omitted, prefers English if available.
  • batchSize?: number - Maximum number of chunks to process concurrently (default: 5)

Returns:

{
  assetId: string;
  chunks: Array<{
    chunkId: string;
    embedding: number[]; // Vector embedding
    metadata: {
      startTime?: number; // Chunk start time in seconds
      endTime?: number; // Chunk end time in seconds
      tokenCount: number;
    };
  }>;
  averagedEmbedding: number[]; // Single embedding for entire transcript
  provider: string;
  model: string;
  metadata: {
    totalChunks: number;
    totalTokens: number;
    chunkingStrategy: string;
    embeddingDimensions: number;
    generatedAt: string;
  };
  usage?: TokenUsage; // Workflow usage metadata
}

Custom Prompts with promptOverrides

Customize specific sections of the summarization prompt for different use cases like SEO, social media, or technical analysis. See the Prompt Customization guide for a full overview of the prompt builder pattern.

Tip: Before adding overrides, read through the default summarization prompt template in src/workflows/summarization.ts (the summarizationPromptBuilder config) so that you have clear context on what each section does and what you're changing.

import { getSummaryAndTags } from "@mux/ai/workflows";

// SEO-optimized metadata
const seoResult = await getSummaryAndTags(assetId, {
  tone: "professional",
  promptOverrides: {
    task: "Generate SEO-optimized metadata that maximizes discoverability.",
    title: "Create a search-optimized title (50-60 chars) with primary keyword front-loaded.",
    keywords: "Focus on high search volume terms and long-tail keywords.",
  },
});

// Social media optimized for engagement
const socialResult = await getSummaryAndTags(assetId, {
  promptOverrides: {
    title: "Create a scroll-stopping headline using emotional triggers or curiosity gaps.",
    description: "Write shareable copy that creates FOMO and works without watching the video.",
    keywords: "Generate hashtag-ready keywords for trending and niche community tags.",
  },
});

// Technical/production analysis
const technicalResult = await getSummaryAndTags(assetId, {
  tone: "professional",
  promptOverrides: {
    task: "Analyze cinematography, lighting, and production techniques.",
    title: "Describe the production style or filmmaking technique.",
    description: "Provide a technical breakdown of camera work, lighting, and editing.",
    keywords: "Use industry-standard production terminology.",
  },
});

Available override sections:

Section Description
task Main instruction for what to analyze
title Guidance for generating the title
description Guidance for generating the description
keywords Guidance for generating keywords/tags
qualityGuidelines General quality instructions

Each override can be a simple string (replaces the section content) or a full PromptSection object for advanced control over XML tag names and attributes.

Common Types

TokenUsage

Returned by all workflows in the usage field:

interface TokenUsage {
  inputTokens?: number; // Tokens in the input prompt
  outputTokens?: number; // Tokens generated in the output
  totalTokens?: number; // Total tokens consumed
  reasoningTokens?: number; // Chain-of-thought reasoning tokens
  cachedInputTokens?: number; // Input tokens served from cache
  metadata?: {
    assetDurationSeconds?: number;
    thumbnailCount?: number;
  };
}

LanguageCodePair

Returned by translateCaptions and translateAudio:

interface LanguageCodePair {
  iso639_1: string; // Two-letter code (e.g., "en", "es") — use for Mux/browser players
  iso639_3: string; // Three-letter code (e.g., "eng", "spa") — use for ElevenLabs
}