feat(m3u): a parser that never holds the playlist - #6
Merged
Conversation
genrewatch and tipoffwatch each carry a copy of this parser, for the same reason they each carried a copy of the codec table, and both copies buffer the file they read. That is not a tidiness problem. A reader's catalogue is 300,000 entries; `parseM3u(await res.text())` holds it as one several-hundred-megabyte string, hashes it into a second copy, and splits it into an array with one string per line -- and doing that in the same process as the HTTP server put genrewatch into a garbage collection spiral that pegged three cores, filled the accept queue to 513 connections and stopped it answering at all. Every five minutes, one refresh interval after each boot, while the deployment went on reporting SUCCESS. So the lookahead becomes a two-state machine that needs no more than the line in front of it, and `parseM3uStream` consumes a fetch body directly: chunks in, entries out, nothing whole held anywhere. Each chunk goes back to the caller through `onChunk` so the file can still be hashed by someone who never sees it -- which is what the unchanged-poll short circuit needs, and the reason the stream is always read to the end even once `max` is reached. A digest over most of a file is worth nothing. A subpath, and deliberately runtime-neutral: no DOM, no node: imports, no Buffer. It is imported by a Bun server, and dist/m3u.js compiles to 3.4KB with no imports at all, so nothing drags a player or an engine in behind it. The parsing rules are unchanged and the tests say so, including the two that look like bugs and are not: of two #EXTINF lines sharing a URL the first wins, and an empty #EXTGRP clears the group unless an entry is waiting. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JPR9SzQVWd1HS4MREpskKu
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
@profullstack/player/m3usubpath export, and bumps to 0.5.0.genrewatch and tipoffwatch each carry a copy of this parser — the same duplication
codecs.tswas created to end — and both copies buffer the file they read.Why it matters
A reader's catalogue is 300,000 entries.
parseM3u(await res.text())holds it as one several-hundred-megabyte string, hashes it into a second copy, and splits it into an array with one string per line. Done in the same process as the HTTP server, that put genrewatch into a GC spiral: ~285% CPU almost entirely in kernel time, threads pegged indo_madviseandJITWorker, the HTTP thread idle inepoll_wait, 513 connections banked up in the accept queue and the edge reportingconnection dial timeout. Every five minutes — one refresh interval after each boot — while the deployment reportedSUCCESSthroughout.What's here
createM3uParser({max})— the#EXTINFlookahead rewritten as a two-state machine, so it needs no more than the line in front of itparseM3u(text, {max})— unchanged behaviour, for callers that genuinely hold the file already (a form paste, a fixture)parseM3uStream(chunks, {max, onChunk})— takes a fetch body directly; chunks in, entries out, nothing whole held anywhereentryKind— moved along with it, since parsing is the only place it can be computed (both sites seal the URL at rest)onChunkhands every chunk back so the file can still be hashed by someone who never sees it — which is exactly what the unchanged-poll short circuit needs. It is also why the stream is read to the end even aftermaxis reached: a digest over most of a file is worth nothing. Past that point decoding and splitting stop, so the tail of an oversized list costs only the read.Throwing from
onChunkaborts and cancels the stream — that is where a size ceiling belongs, because the policy and its wording are the caller's.Runtime-neutral, deliberately
No DOM, no
node:imports, noBuffer. It is imported by a Bun server ingesting into Postgres, which is the opposite end of the stack from the rest of this package.dist/m3u.jscompiles to 3.4KB with no imports at all, so nothing drags a player or a lazy engine chunk in behind it.Testing
pnpm vitest run— 115 pass (23 new)\rand the\n, and a multi-byte character split at every byte offsetonChunkreceives the whole body, in order, including pastmaxpnpm typecheck,pnpm build,pnpm lintall cleanThe parsing rules are unchanged, including the two that look like bugs and are not: of two
#EXTINFlines sharing one URL the first wins, and an empty#EXTGRPclears the group unless an entry is waiting for its URL.🤖 Generated with Claude Code
https://claude.ai/code/session_01JPR9SzQVWd1HS4MREpskKu