feat: add AI transcript imports with YouTube captions and browser downloads

[annieappflowy]

Description

Add audio/video file imports and YouTube transcript imports to the AI transcript block. Users can insert AI transcript from the slash menu, upload a file, or paste a YouTube URL. YouTube imports now try existing captions first and offer an explicit browser audio download when captions are unavailable. AppFlowy servers do not download or proxy YouTube audio/video.

• Import captions: preview the video, select an available manual or automatic caption language, and save text with timestamps directly into the document. This creates no transcription job and consumes no transcription minutes. Summary generation uses the normal AI-response allowance.
• Download audio and transcribe: the optional Chrome/Edge extension opens a temporary YouTube tab, briefly plays its native player muted, and obtains playback metadata. The AppFlowy page then downloads only the audio track directly from Googlevideo on the user's device. The playable file enters the existing signed-upload and transcription flow, including workspace duration, file-size, and usage limits.
• Upload audio/video: users can choose an existing file without installing the extension. This remains available when captions or browser downloads fail. Missing captions never trigger an automatic media download.
• Retain transcription progress, Stop, Retry, saved-job recovery, quota guidance, speaker/title preprocessing, summary template/detail/language options, and numbered timestamp citations. Preserve saved summaries on cancellation or failure and retain citations through block moves, duplication, undo/redo, and reopening.
• Resolve the review findings: late transcription results recheck live Yjs state before applying, polling resumes when editing is restored, and intentional AI Writer cancellation preserves its normal output controls.

Why use a browser extension?

An ordinary React page is subject to browser cross-origin restrictions. A React downloader library cannot grant access to YouTube resources or alter the browser's network-header rules. YouTube downloads also need current playback metadata from its native player. The helper prepares that playback on YouTube's origin and applies narrowly scoped network-header rules so the audio can be fetched directly on the user's device. This meets the requirement to keep YouTube media downloading off AppFlowy servers. See Chrome's cross-origin request guidance and network-rule API.

Transcription needs the audio track, so the download omits video frames. Captions already contain text and timestamps; caption import and manual file upload require no extension. The architecture follows the local-extension/network-rule strategy demonstrated by Kira, with an original helper implementation.

Extension and deployment

• Ship the helper source and downloadable ZIP, plus installation guidance in the dialog. It currently requires unzip → Developer mode → Load unpacked → reload AppFlowy; it is not published to an extension store. This package targets Chrome/Edge. Self-hosted domains require a package configured for their exact origin.
• Restrict the bridge to AppFlowy beta/test and loopback origins, and to availability checks and playback preparation. Validate the owned YouTube tab, video, and playback session to exclude ads. The helper exposes no general HTTP proxy, forwards no cookies or authorization headers, and does not persist playback tokens.
• Execute YouTube's player on YouTube's origin. Stop, navigation, loss of editing/transcription availability, or preparation timeout cleans up the helper tab and cancels pending work. The downloader checks file completeness and playability and enforces the workspace file limit and a 256 MiB browser cap.
• Keep /api/youtube-audio for deployment compatibility, but restrict it to authenticated metadata/caption GET requests; the old audio-download POST is rejected. Vite, Node, Bun, and SSR deployments use the shared handler. Static-only deployments must route the endpoint to Node/Bun. No AppFlowy Cloud Premium or desktop changes are included.

See the AI transcript documentation and helper installation/permissions.

Validation

• 677 tests passed in 38 focused Jest suites, with one opt-in live caption test skipped in the default run. Coverage includes caption parsing/import, download transport and limits, cancellation, stale responses, collaborator edits, recovery, AI Writer Stop, summary generation, and citations.
• 9 extension tests passed, covering origin/permission boundaries, playback-session selection, forged messages, cancellation, and cleanup.
• pnpm lint passed, including the shared error catalog, TypeScript, and ESLint. pnpm build and git diff --check passed.
• A separate real caption test successfully retrieved the supplied video's English automatic captions.
• A real browser download through the production downloader and this extension successfully fetched W2zbKF5z1cs: 2,112,472 bytes of playable MP4 audio, 130.45 seconds. The check observed direct Googlevideo requests, zero AppFlowy server requests, and zero remaining helper tabs. It used a disposable browser profile without account sign-in and submitted no paid transcription request or document edits.

Reproduce the real browser download with:

RUN_YOUTUBE_DOWNLOAD_LIVE_TEST=1 node scripts/test-youtube-browser-download.mjs

This requires a desktop display and installed Playwright Chromium. Automated unit tests use mocked services; they do not validate the live transcription or summary provider.

Remaining limitations

The helper is a development distribution pending an extension-store release. YouTube access restrictions, consent pages, unavailable videos, or player changes may still prevent downloading; file upload remains available. Current live streams are unsupported.

The last recorded full summary request to test.appflowy.cloud returned AI provider authentication failed, while small completion requests succeeded. That upstream issue remains unresolved and needs AI-service investigation. The successful YouTube download does not establish that full live transcription and summary generation succeed; the client preserves the existing summary when generation fails.

Feature Preview

Run pnpm storybook and open the AI transcript import stories for welcome, progress, quota, error, transcript, and summary states.

For local testing, run pnpm dev --port 3001, open a document, and insert AI transcript. Paste a YouTube URL to import captions, or follow the dialog's helper installation steps and select Download audio and transcribe. Upload audio/video is also available.

Checklist

General

• I've included relevant documentation or comments for the changes introduced.
• I've tested the changes in multiple environments (e.g., different browsers, operating systems).

Testing

• I've added or updated tests to validate the changes introduced for AppFlowy Web.

Feature-Specific

• I've added a preview in the Feature Preview section.
• I've verified that this feature integrates seamlessly with existing functionality. Full live transcription and summary generation still need verification after the service issue above is resolved.

[sourcery-ai]

Sorry @annieappflowy, your pull request is larger than the review limit of 150,000 diff characters