EchoMimicV3 talking head: ship it (#77) + Modal Volume weights (#76) - #79
Merged
Conversation
Candidate replacement for SadTalker, which is built on a model unmaintained since 2023. EchoMimicV3-Flash (Ant Group, Apache 2.0, AAAI 2026) preserves the input aspect ratio, so a 16:9 presenter fills the 16:9 NarratorPiP box instead of letterboxing a square crop — the framing problem pluribus-sprint hit. Measured against a real SadTalker render (same face, same audio): mouth-region motion 3.488 vs 2.185, identity indistinguishable from the source still. Costs roughly 6.5x SadTalker (28.9x realtime at 5 steps/640, 47.8x at 8 steps/768). Deploys either with weights baked into the image or from a modal.Volume, via ECHOMIMIC_WEIGHTS at deploy time; see #76 for that trade-off. Load-bearing details, all found the hard way: - diffusers and transformers are pinned, not floored. Newer transformers drives output_hidden_states from config rather than the kwarg EchoMimic's Wav2Vec2 subclass passes, so the encoder returns hidden_states=None and there is no lip sync at all — a silent failure, not an error. - enable_model_cpu_offload() is required; the model does not fit 24GB with every component resident. - chinese-wav2vec2-base ships a .bin and transformers >=4.51.3 refuses torch.load below torch 2.6, so it is converted to safetensors at fetch time. - Upstream infer_flash.py generates one 81-frame clip and silently truncates longer audio, so the segment loop lives here, re-anchoring on the previous segment's tail frames and cross-fading the seam. Known bug tracked in #77: that re-anchoring can latch a blink, holding the eyes closed across a segment boundary. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…y SHA (#76) Closes #76. Every other docker/modal-* app bakes its weights into the image, a habit inherited from RunPod where serverless genuinely pulls the image. Modal recommends the opposite, and the benchmark backs it: rebuild after a dependency change 79-385s baked 1.8-8.2s volume cold start 57-94s 60-67s generation speed identical identical Cold start and generation are a wash; the rebuild gap is 20-100x. Getting this model working took four dependency changes, each of which re-downloaded 26GB under the baked scheme, so that is the cost that actually bites. Storage is free at this scale — Modal includes 1 TiB/month and this app stores 26.6GB — which is what makes the choice an engineering one rather than a cost trade-off. The settled apps (upscale, image-edit) change rarely and gain nothing by moving, so they stay baked. Baking remains fully supported here too, as a separate app: ECHOMIMIC_WEIGHTS=image uv run modal deploy docker/modal-echomimic3/app.py The real cost of a volume is reproducibility: weights are no longer tied to the image, so nothing else stops the two drifting apart. Hence REPO_REF moves off `main` to a commit SHA (an unpinned ref silently re-resolving on rebuild is how #71/#74 happened to flux2) and all four model revisions are pinned too. Fetching by SHA needs git init + fetch --depth 1; clone --branch only takes a ref name. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
#77) The Flash pipeline generates one 81-frame clip and silently truncates longer audio, so the segment loop lives in app.py: each continuation segment re-anchors on the last `overlap` frames of the previous one. Whatever pose those frames hold therefore becomes the next segment's opening pose — and when they landed mid-blink the model started closed-eyed and *held* it, giving a prolonged closure straddling the boundary. Observed across frames 72-80 with the seam at 73. Detecting eyes would mean adding the face-landmark stack this image deliberately omits (tensorflow + retina-face, neither reachable from the Flash path). Instead --anchor-retreat scores candidate anchor windows by how much motion they contain in the upper half of the frame — where blinks live and mouth movement does not — and backs off up to N frames to anchor on the calmest one. A blink is the largest short transient up there, so it scores worst and gets skipped. Anchoring on a settled pose is the better default anyway: a continuation has to extrapolate from whatever frames it is handed. Retreating means regenerating the dropped frames, so it only fires on a clear improvement (under 0.8x the score at retreat 0). Clean seams cost nothing. --anchor-retreat 0 restores the previous behaviour. Verified twice. Offline against synthetic blinks: detects one inside the anchor window, leaves clean windows alone, picks a window excluding the blink frames, and clamps to 0 on clips too short to retreat. Then end-to-end on the model with --audio-guidance-scale 1.8, the config that originally latched — it fired on one seam of three (frames 219 -> 213) and left the other two untouched, and the repaired join shows open eyes throughout with no pop. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
objectPosition was declared in NarratorPiPProps, documented with a default of 'center top', and destructured in the component — then never used, because the style block hardcoded 'center bottom'. Anyone passing the prop got silence. Honoured now, with the default corrected to 'center bottom' to match what the component actually did, so no existing project re-renders differently. Also adds an objectFit prop (default 'contain', unchanged). 'contain' suits both generators: every SIZE_PRESET is 16:9, so a 16:9 source fills the box exactly — which is what echomimic3 produces, and what sadtalker produces with --preprocess full. SadTalker's default square crop letterboxes instead, and can now opt into 'cover' to fill the box at the cost of the sides. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ision rule (#77) Adds docs/echomimic3.md and a registry entry, and rewrites CLAUDE.md's talking head section around a decision table now that there are two generators. The rule: EchoMimicV3 when the narrator is large in frame or the shot is held long enough to watch; SadTalker for small overlays, drafts, and generating several takes to choose between. Cost is ~$0.009/second of output against ~$0.0014, but wall clock is the sharper constraint — 22.8-47.8x realtime means a 3-minute narrator is 1.5-2.4 hours, so per-scene clips beat one long render. That split was measured, not assumed. Controlled A/B, same still and same audio, both downscaled to NarratorPiP 'sm' (240x135, ~3% of a 1080p frame): EchoMimicV3 carries 3.5x the whole-frame motion and 6.2x in the mouth band, so the gap survives the shrink rather than washing out. On review it is clearly the better picture while SadTalker remains perfectly usable at that size — hence two tools rather than a replacement. (The SadTalker side was rendered --preprocess full --still, so this compared articulation, not framing.) Also carries the traps forward into user-facing docs, since they cost hours and are invisible from the outside: the transformers==4.49.0 pin is load-bearing and a newer version silently removes all lip sync; --wav2vec english is worse than chinese even for English audio; and a mouth-crop sync metric is not a quality score — it ranked highest the one tuning variant with a visible eye defect. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 30, 2026
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Takes the EchoMimicV3 spike from "works on my Modal account" to shippable, and settles the
weight-storage question it raised.
Builds on
d8b1a7c, which addedtools/echomimic3.pyanddocker/modal-echomimic3/.Closes #76 — weights go in a Modal Volume
Cold start and generation are a wash; the rebuild gap is 20-100x. Getting this model working
took four dependency changes, each re-downloading 26GB under the baked scheme. Storage is free
at this scale (Modal includes 1 TiB/month; this stores 26.6GB), so the decision is an
engineering one, not a cost one.
Scope: this app only. The other six stay baked — they change rarely and gain nothing.
Baking stays fully supported here too via
ECHOMIMIC_WEIGHTS=image, which deploys as aseparate app. Documented as a real choice rather than a migration.
The volume's one genuine cost is reproducibility: weights are no longer tied to the image.
So
REPO_REFmoves offmainto a SHA — the release blocker the app README had flagged,and the same failure mode as #71/#74 on flux2 — and all four model revisions are pinned too.
Fixes the #77 seam bug
Each continuation segment re-anchors on the previous segment's last
overlapframes, sotheir pose becomes the next segment's opening pose. Landing mid-blink made the model start
closed-eyed and hold it.
--anchor-retreatscores candidate anchor windows by motion in the upper half of theframe — where blinks live and mouth movement doesn't — and backs off up to N frames to the
calmest one. Eye detection would have meant adding the face-landmark stack the image
deliberately omits. It only fires on a clear improvement, so clean seams cost nothing;
--anchor-retreat 0restores the old behaviour.Verified offline against synthetic blinks, then on the model with the config that originally
latched: fired on one seam of three (219 -> 213), left the others alone, repaired join shows
open eyes throughout.
Does SadTalker stay? Yes
#77 item 4 called this untested. Controlled A/B — same still, same audio, both downscaled to
NarratorPiP
sm(240x135, ~3% of a 1080p frame): EchoMimicV3 carries 3.5x the whole-framemotion and 6.2x in the mouth band, so the gap survives the shrink. On review it's clearly
better, while SadTalker is still perfectly usable at that size. Two tools, not a replacement —
CLAUDE.md gets a decision table.
Also
docs/echomimic3.md, registry entry,docs/modal-setup.md, CHANGELOGNarratorPiPbug unrelated to either issue:objectPositionwas declared, documentedand destructured, then ignored by a hardcoded value. Now honoured, default corrected to what
the component actually did so nothing re-renders differently. Adds
objectFit.Not in this PR
Deferred #77 items, to be filed as a follow-up: per-scene narrator generation, replacing
project.json'ssadtalkerJobswith ageneratorfield, and presets.Still open and documented as such:
--audio-guidance-scaleis unsettled (upstream suggests1.8-2.0, default here is 3.0, and the metric that would decide it is the discredited one), and
full-length narration — drift over 60+ segments — has never been run.
Verification
Deployed and smoke-tested against the live endpoint. Volume redeploy took 11s. Clone-by-SHA
confirmed in the build log (
HEAD is now at 7e89489).populate_weightsre-verified thevolume at 26.6GB against the pinned revisions.