Repository navigation
fix(video): allow slow Seedance 2.5 jobs to finish - #110
Conversation
VickyXAI
left a comment
There was a problem hiding this comment.
Thanks for chasing this down with real production numbers — the 479s/400s/275s measurements are exactly the right evidence, and the diagnosis that the failure was client-side is correct. One problem with the margin arithmetic before this lands.
The 60s margin doesn't hold for the last poll
The 540s budget sits 60s inside the 600s authorization only at loop entry. The loop body then adds two costs that aren't in the budget:
video.ts:541 while (Date.now() - startedAt < VIDEO_TOTAL_BUDGET_MS) // may enter at 539.999s
video.ts:542 await new Promise(r => setTimeout(r, POLL_INTERVAL_MS)); +5s -> 545s
video.ts:547 }, 90_000); // poll fetch timeout +90s -> 635s
validBefore is startedAt + 600s (@blockrun/llm/dist/index.js:109: const validBefore = now + (options.maxTimeoutSeconds || 300)), with maxTimeoutSeconds: Math.max(..., 600) at video.ts:487. And fetchWithTimeout deliberately never clears its abort timer (src/utils/http.ts:6-12), so 90s bounds headers plus body read.
So a poll can be in flight at 635s — 35s after the authorization expires, rather than 60s before it.
This already shipped once, on the sibling tool
CHANGELOG.md:1351:
fix(music)— keep the payment authorization valid through the poll window. ... A slow MiniMax track completing aftervalidBeforefailed settlement server-side and surfaced as a wrong "fund your wallet" error for a track that actually generated.
That fix raised music to 600s to stay inside the window. This change pushes video back out past it:
| budget | +interval | +poll timeout | worst case | vs 600s auth | |
|---|---|---|---|---|---|
| video (before this PR) | 300s | 5s | 90s | 395s | +205s safe |
music (music.ts:164-170) |
240s | 5s | 90s | 335s | +265s safe |
| video (this PR) | 540s | 5s | 90s | 635s | -35s expired |
There's a secondary effect worth noting: video.ts:571-576 books local spend the instant it observes status === "completed", without checking that settlement succeeded. That was safe before because expiry was unreachable; in the 600–635s tail it would record a ledger charge for USDC that never moved — the mirror image of the bug the comment directly above it describes fixing.
Lowering the budget won't work
Worth flagging because it's the tempting fix: staying inside 600s unclamped requires budget <= 445s, which fails the 479s job this PR exists to support. The budget number isn't the lever.
Suggested fix: clamp the last poll to the remaining budget
Bounds worst-case wall time at exactly 540s, so the 60s margin becomes real and the 479s job still finishes:
export const VIDEO_POLL_TIMEOUT_MS = 90_000;
export const VIDEO_PAYMENT_AUTH_SECONDS = 600;
// How long the next poll may block, given the budget that is left. Returns 0
// when the budget is spent, which the caller treats as "stop".
export function pollTimeoutFor(deadlineMs: number, nowMs: number): number {
const remainingMs = deadlineMs - nowMs;
if (remainingMs <= 0) return 0;
return Math.min(VIDEO_POLL_TIMEOUT_MS, remainingMs);
}const deadline = startedAt + VIDEO_TOTAL_BUDGET_MS;
while (Date.now() < deadline) {
await new Promise((r) => setTimeout(r, POLL_INTERVAL_MS));
// Checking the deadline only at the top of the loop bounds when a poll may
// START, not when it finishes — an unclamped 90s poll entered just under the
// wire stays in flight well past validBefore.
const pollTimeoutMs = pollTimeoutFor(deadline, Date.now());
if (pollTimeoutMs === 0) break;
const pollResp = await fetchWithTimeout(pollAbsoluteUrl, {
method: "GET",
headers: { "PAYMENT-SIGNATURE": paymentPayload },
}, pollTimeoutMs);Also worth using VIDEO_PAYMENT_AUTH_SECONDS at video.ts:487 — right now 600 is a magic literal there, while the budget it has to be reconciled against is an exported constant.
Extracting pollTimeoutFor rather than inlining the clamp is deliberate: inline, it sits inside the handler behind a 402 + payment + submit chain that test/video-models.test.ts blocks at the network by design (the NETWORK_ESCAPE sentinel), so its removal would go uncaught.
On the new test
assert.equal(VIDEO_TOTAL_BUDGET_MS, 9 * 60 * 1000);
assert.equal(10 * 60 * 1000 - VIDEO_TOTAL_BUDGET_MS, 60 * 1000);Two issues: it restates the authorization window as a literal instead of deriving it from the 600 the request actually sends, so changing that value leaves the test green while the invariant breaks; and it omits POLL_INTERVAL_MS and the poll timeout — the two terms that actually blow the margin. As written it asserts 600000 - 540000 === 60000, which certifies a margin that doesn't exist.
Suggested replacement — asserts both the capability and the invariant, and sweeps the property that matters:
test("video polling allows slow 30s jobs while staying inside payment authorization", () => {
// The capability this budget buys: production jobs measured at 479s/400s/275s.
const SLOWEST_OBSERVED_MS = 479_000;
assert.ok(VIDEO_TOTAL_BUDGET_MS > SLOWEST_OBSERVED_MS);
const authMs = VIDEO_PAYMENT_AUTH_SECONDS * 1000;
assert.ok(VIDEO_TOTAL_BUDGET_MS + 60_000 <= authMs);
// The clamp is load-bearing: unclamped, worst case overruns the window.
assert.ok(VIDEO_TOTAL_BUDGET_MS + VIDEO_POLL_TIMEOUT_MS > authMs);
});
test("the last poll is clamped to the remaining budget, never left in flight past the deadline", () => {
const start = 1_000_000;
const deadline = start + VIDEO_TOTAL_BUDGET_MS;
assert.equal(pollTimeoutFor(deadline, start), VIDEO_POLL_TIMEOUT_MS);
assert.equal(pollTimeoutFor(deadline, deadline - 1_000), 1_000);
assert.equal(pollTimeoutFor(deadline, deadline), 0);
// A poll started at any reachable instant finishes on or before the deadline.
for (let elapsed = 0; elapsed <= VIDEO_TOTAL_BUDGET_MS; elapsed += 4_999) {
const now = start + elapsed;
assert.ok(now + pollTimeoutFor(deadline, now) <= deadline);
}
});Verified locally
I applied the above on top of your branch: 343/343 pass, tsc --noEmit clean, npm run build clean. I also confirmed the new test is load-bearing — reverting the Math.min to a bare VIDEO_POLL_TIMEOUT_MS produces not ok 15 - the last poll is clamped..., so it catches the regression it's meant to catch.
One last thing: the description's "9 min hard cap" becomes accurate on its own once the clamp lands, so no change needed there.
Happy to push the commit to this branch if you'd rather not hand-apply it — just say the word.
|
Merging this as-is — the budget extension is right and the production data backs it. I'm landing the polling-window clamp from my review as an immediate follow-up PR rather than asking you to round-trip it. To be explicit about the risk window: nothing ships from this merge. Thanks for the live Seedance measurements — that's what made the client-side diagnosis unambiguous. |
…t the poll budget (#113) blockrun_video got this in #111; music has the same shape and the same latent bug, one budget bump away from firing. music measures its 240s poll budget from AFTER submit, but validBefore is counted from signing — and submit is allowed 95s. So the real worst case from the authorization's own origin is 95 + 240 + 5 + 90 = 430s. That fits inside 600s today, which is exactly why it looks fine and why the next person to raise MUSIC_POLL_BUDGET_MS the way #110 raised video's would walk straight into it: at 420s the sum is 610s and settlement starts failing for tracks that rendered. Naively copying video's fix (stamp startedAt before signing) would have been worse than the bug: the 240s budget would then absorb up to 95s of submit, leaving as little as 145s of polling for MiniMax tracks the code documents as taking 60-180s. So the loop takes the EARLIER of two deadlines instead — the poll budget, and the authorization minus settle-side slack. A fast submit keeps the full 240s window; a slow one shortens it rather than silently pushing polls past validBefore. - share pollTimeoutFor from src/utils/poll.ts, used by both tools - deliberately not in http.ts: four test files replace that module with a network sentinel, so a pure function there forces them to stub arithmetic - video.ts: repair the indentation break at the spend-booking site and drop the redundant inner !spendBooked guard already implied by the enclosing condition - cover the clamp in test/poll.test.ts and the music invariant in music-cost Co-authored-by: VickyXAI <vicky.fuyu@gmail.com>
Summary
Why
Production tests of Seedance 2.5 30-second clips completed in 479s, 275s, and 400s. All three succeeded after increasing the local MCP budget; two exceeded the previous 300-second hard cap. The gateway already uses async submit plus short polling requests, so the old failure was client-side.
Validation
No gateway protocol or settlement behavior changes.