SoulX-FlashHead replaces EchoMimicV3 as the talking head generator - #82
Merged
Conversation
SoulX-FlashHead (Soul AI Lab, Apache 2.0, 1.3B) becomes the toolkit's default talking head generator. It preserves the input aspect ratio, so 16:9 presenter images come back 16:9 with no --preprocess workaround, and --size snaps to the model's latent grid while keeping the aspect. It is the default because identity holds over a long take. Segment-chained talking heads re-anchor each segment on the previous segment's output, which makes the failure absorbing rather than gradual. SoulX is trained with Oracle-Guided Bidirectional Distillation against exactly that, and it measures out: 97% of frame-zero sharpness at 70s, flat across all 72 segments, verified on two subjects at three resolutions. There is no short-render ceiling to design around, so per-scene generation is a choice rather than a workaround. ~$0.0024 per second of output against SadTalker's ~$0.0014, and ~6.4-7.9x realtime on A10G. Both figures are a floor: flash-attn and sageattention are optional here and neither is installed, while upstream's quoted FPS assumes one. Notes for whoever touches this next: - There is no resolution argument anywhere upstream. height/width live in a module-global loaded at import from a relative path, so the app chdirs into the repo root and mutates that global. Nothing validates the value either -- an off-grid size floors silently and renders wrong rather than raising, so _check_size guards it on both sides. - torch.compile is on upstream, costs ~600s per container and recompiles on every resolution change. scaledown_window is 600s so a batch at one size amortises it; SOULX_COMPILE=0 disables it for odd one-offs. - Weights live in a Modal Volume, not baked: redeploy after a code change is ~2.3s. Repo ref and both model revisions pinned by SHA, since volume weights are not tied to the image. - modal run cancels the call when its client dies (a sleeping laptop counts, and --detach does not save it). Renders are written to the soulx-out volume before being returned. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
EchoMimicV3 was added earlier in this same unreleased cycle and never reached a tagged release, so this removes it rather than deprecating it — no public API changes. Identity drift is why. On a controlled A/B (same photo, same 80s audio, same 544x736) it fell to 50% of frame-zero sharpness by 70s, glasses dissolving around 55s and no recognisable face by 67s, where SoulX-FlashHead held 97%. The failure is absorbing rather than gradual — each segment re-anchors on the previous segment's output, so one bad segment poisons everything after it — and colour correction recovers none of it because the damage is structural. That implied a ~30s render ceiling which shaped the surrounding design. The one capability SoulX lacks is text-prompt and CFG steering, and our own docs already recorded from testing that the prompt is close to inert. Everything else EchoMimicV3 offered, SoulX matches or beats: aspect preservation, motion quality, ~3.7x cheaper, ~3-6x faster. The genuine loss is gesture and upper-body motion, which nothing in the toolkit currently uses. Also in this commit: - Talking head docs, registry entries (tools and modal endpoints), cloud_gpu dispatch, modal-setup, NarratorPiP's comment and both env files repointed to soulx. cloud_gpu also gains a GPU tier for it, which echomimic3 never had -- its jobs silently reported no cost estimate. - The Unreleased Kiro changelog entry is dropped; it shipped in v0.19.0 and was flagged in-file for removal when this section was cut. Findings worth keeping outlived the tool and moved into docs/soulx.md: the warning against scoring talking heads with similarity metrics (two have now misled — one ranked highest a variant with a visible eye defect, the other plateaued straight through a total collapse), the volume-weights rationale, and the image guidelines. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces EchoMimicV3 with SoulX-FlashHead (Soul AI Lab, Apache 2.0, 1.3B) as the toolkit's talking head generator.
Demo: 80 seconds from a single still — one photo plus audio, rendered at 640x640 in one pass.
Why
EchoMimicV3 drifts. Segment-chained talking heads re-anchor each segment on the previous segment's output, so the failure is absorbing rather than gradual — one bad segment poisons everything after it, and colour correction recovers none of it because the damage is structural.
Controlled A/B, same photo, same 80s audio, same 544x736 output, as a share of frame-zero sharpness:
At 70s EchoMimicV3 is a featureless smear with no eyes; SoulX is still a sharp, correctly framed, unmistakably identical face. Repeated on a second subject at 640x640 — different colouring, no glasses, busy background — which held 97-100% across the full 80s, so it is a property of the model rather than of one photograph.
SoulX is also
3.7x cheaper ($0.0024 vs ~$0.009 per second of output) and ~3-6x faster. Both figures are a floor: flash-attn and sageattention are optional and neither is installed, while upstream's quoted FPS assumes one of them.EchoMimicV3 never reached a tagged release — it was added earlier in this same unreleased cycle — so this removes it rather than deprecating it. No public API changes.
The only capability it had that SoulX lacks is text-prompt/CFG steering, and
docs/echomimic3.mdalready recorded from our own testing that the prompt is "close to inert". The genuine loss is gesture and upper-body motion, which nothing in the toolkit currently uses.Notes for review
height/widthlive in a module-global loaded at import from a relative path, so the appchdirs into the repo root and mutates it. Nothing validates the value either — an off-grid size floors silently and renders wrong rather than raising — so_check_sizeguards both sides. Pro uses the Wan2.1 VAE (grid 16); Lite uses LTX-Video (grid 64), so768x432is legal for one and illegal for the other.torch.compileis on upstream, costs ~600s per container and recompiles per resolution.scaledown_windowis 600s so a batch at one size amortises it;SOULX_COMPILE=0disables it.modal runcancels the call when its client dies — a sleeping laptop counts, and--detachdoes not save the in-flight call. Renders are written to thesoulx-outvolume before being returned.cloud_gpu.pygains a GPU tier for the tool, whichechomimic3never had — its jobs silently reported no cost estimate.UnreleasedKiro changelog entry is dropped; it shipped in v0.19.0 and was flagged in-file for removal when the section was cut.Closes #81
Closes #80
🤖 Generated with Claude Code