Take subscription lists as they come: 200k lines, and .txt everywhere (#134)

Three things stopped a hundred-and-ten-thousand-line list of feeds from
being an ordinary thing to submit.

The cap that was read as one was never a cap on a submission at all.
RAW_INPUT_LINE_LIMIT bounds the copy of an upload kept to name it on its
status page — the ceiling on a submission is MAX_UPLOAD_FEEDS and it is
in the tens of millions — but "only the first 50,000 lines of the upload
are kept" reads as a limit however it is meant, and fifty thousand is
below the size of the lists people actually bring. Raised to 200,000, so
a real export is recorded whole and the note is one almost nobody sees.

A plain list of feed URLs uploaded as a file was silently thrown away by
every client that is not our uploader. The field is named `opml` and the
endpoint took the name for the truth, so a .txt of one feed per line was
parsed as XML, matched no outlines, and came back `no-feeds-in-opml`
having imported nothing — with JavaScript off, from curl, or from an
agent. It is sniffed now, by the same call the browser has always made.
sniffKind reads the file before the name while it is here: `<opml` in
the head means OPML whatever the file is called, which is the case its
own comment promised and the old order lost.

And a large submission arriving in one piece was crawled inside the
request that carried it — importFeeds reads every feed URL and slug in
the directory, then a round trip per five hundred rows, which for a
hundred and eight thousand entries is over two hundred of them under a
five-minute ceiling. Past five thousand entries it is staged instead and
released to the poller, exactly as the batched uploader stages it: one
bulk insert per two thousand, nothing that scales with the directory.
Measured on that file: 1.0s to hand over, all 108,000 recorded. Below
the threshold nothing changes, and one URL still redirects to the blog
it just added.

Two things the staged path exposed once everything took it:

- The progress bar counted only queued feeds, so the whole handover —
  now most of a large import's life — showed a full bar and the word
  "finished" under a heading saying "Import in progress". Staged entries
  count towards the total in both the page and the stream, and the tick
  interval is sized from them too, which had put the biggest uploads on
  the fastest poll.
- The uploader switched its progress panel on the sniffed kind, so a
  .txt chosen in the file form drew its bar under the paste box. It
  switches on which form started it now.

Verified end to end against a 108,000-line, 5.4 MB file: staged in 1.0s
over multipart, 2.3s in the browser (54 batches), 50,000 feeds drained
in 6.1s, unique slugs throughout, and OPML — including OPML living in a
.txt — still read as OPML with its titles and entities intact.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: ralyodio

apps/web/src/app/api/submissions/[id]/stream/route.js

@@ -42,7 +42,14 @@ export async function GET(req, { params }) {
   // uploader made a file that size importable. Such a queue also takes days to
   // drain, so polling it every second buys no visible smoothness at all: the
   // bar advances by a fraction of a percent an hour either way.
-  const tick = pollInterval(Number(submission.queued_count ?? 0));
+  //
+  // Sized from the staged total as well as the queued one. A large import
+  // spends its first stretch entirely staged — `queued_count` is still zero —
+  // so reading that alone put the biggest uploads on the fastest tick, which is
+  // precisely backwards and is the case this whole calculation exists for.
+  const tick = pollInterval(
+    Math.max(Number(submission.queued_count ?? 0), Number(submission.entries_total ?? 0)),
+  );
 
   return stream(async (first) => {
     const [progressRow, events] = await Promise.all([
@@ -51,15 +58,20 @@ export async function GET(req, { params }) {
     ]);
 
     const frames = [];
-    const total = progressRow.queued;
+
+    // Staged entries count towards both numbers. They are feeds this import
+    // will crawl that simply have no row yet, so leaving them out reported a
+    // handover as a finished crawl — a full bar, the word "finished", and the
+    // page above it still saying how many were waiting to be added.
+    const total = progressRow.queued + progressRow.pending;
     const settled = progressRow.crawled + progressRow.failed;
-    const done = progressRow.waiting === 0;
+    const done = progressRow.waiting === 0 && progressRow.pending === 0;
 
     const progress = {
       total,
       settled,
-      // A submission with nothing queued finished the moment it was made, and
-      // an empty bar reading 0% would be a lie about that.
+      // A submission with nothing queued and nothing staged finished the moment
+      // it was made, and an empty bar reading 0% would be a lie about that.
       percent: total === 0 ? 100 : Math.floor((settled / total) * 100),
       crawled: progressRow.crawled,
       failed: progressRow.failed,

apps/web/src/app/api/submit/route.js

@@ -1,7 +1,9 @@
-import { submitCatalogue, submitOpml, hashIp } from '@rssamplifier/ingest';
+import { submitCatalogue, hashIp } from '@rssamplifier/ingest';
+import { parseOpml } from '@rssamplifier/feed';
 import { q, newId } from '@rssamplifier/db';
 
 import { db, siteUrl } from '../../../lib/db.js';
+import { sniffKind } from '../../../lib/opml-scan.js';
 import { clampRawInput } from '../../../lib/submitted.js';
 
 export const dynamic = 'force-dynamic';
@@ -27,6 +29,47 @@ const RATE_LIMIT = 20;
  */
 const INLINE_UPLOAD_LIMIT = Number(process.env['SUBMIT_INLINE_BYTES'] ?? 10_000_000) || 10_000_000;
 
+/**
+ * How much of a file is read before deciding what kind of file it is.
+ *
+ * The signature that settles it — the `<opml>` root, or the first `<outline>` —
+ * is in the first line or two of any real export, so this only has to be past
+ * the XML declaration and any comment somebody put above it.
+ */
+const SNIFF_CHARS = 4096;
+
+/**
+ * Entries above which a submission is handed over rather than imported here.
+ *
+ * Importing means `importFeeds`: one read of every feed URL and slug in the
+ * directory, then a round trip per five hundred rows. That is seconds for a
+ * paste and minutes for a subscription export — a hundred and ten thousand
+ * entries is over two hundred sequential round trips inside a single request
+ * with a five-minute ceiling on it, which is a coin toss at best and loses the
+ * whole upload when it comes up wrong.
+ *
+ * Past this the entries are staged instead, exactly as the batched uploader
+ * stages them: one bulk insert per couple of thousand, no lookups, no slugs,
+ * nothing that scales with the size of the directory, and the poller drains it
+ * afterwards. The submitter gets the status page, which is where an import of
+ * that size was always going to end up.
+ *
+ * Below it nothing changes, and that matters more than it sounds: a small
+ * submission is resolved over the network while the submitter waits, and one
+ * URL still redirects to the blog it just added.
+ */
+const STAGE_ABOVE = 5_000;
+
+/**
+ * Entries written per staging statement.
+ *
+ * `stageImportEntries` binds four parameters per row, and SQLite's ceiling on
+ * bound parameters is what decides this rather than the size of the payload —
+ * two thousand rows is eight thousand of them, comfortably under it, and the
+ * same slice the uploader and the drainer both work in.
+ */
+const STAGE_CHUNK = 2_000;
+
 /**
  * Split a paste into candidate URLs.
  *
@@ -107,9 +150,24 @@ export async function POST(req) {
 
       if (file && typeof file !== 'string' && file.size > 0) {
         if (file.size > INLINE_UPLOAD_LIMIT) return tooLarge(req, file.size);
-        kind = 'opml';
         raw = await file.text();
-        opml = raw;
+
+        // What the file is, rather than what the field is called. The input is
+        // named `opml` because that is what it was built for, but a plain list
+        // of feed URLs is the other half of what this page accepts and arrives
+        // through the same field — and reading one as the other finds no
+        // outlines at all, so a perfectly good subscription list was answered
+        // with `no-feeds-in-opml` and nothing was imported. The browser has
+        // sniffed its own uploads since the batched uploader landed; this is
+        // the same call, for the clients that do not run it.
+        if (sniffKind({ name: file.name }, raw.slice(0, SNIFF_CHARS)) === 'opml') {
+          kind = 'opml';
+          opml = raw;
+        } else {
+          const urls = splitUrls(raw);
+          kind = urls.length > 1 ? 'list' : 'url';
+          entries = urls.map((url) => ({ url }));
+        }
       } else {
         raw = String(form.get('input') ?? '');
         const urls = splitUrls(raw);
@@ -135,15 +193,83 @@ export async function POST(req) {
 
   const browser = (req.headers.get('accept') ?? '').includes('text/html');
 
+  // An OPML document becomes its entries here rather than inside `submitOpml`,
+  // because how many of them there are is what decides what happens next.
+  const catalogue = opml === null ? entries : parseOpml(opml);
+
+  // A file that parsed as OPML and holds no outlines is the one failure worth
+  // naming: it is almost always the wrong file, and "nothing happened" is a
+  // much worse answer than saying so.
+  if (opml !== null && catalogue.length === 0) {
+    const rejected = [{ url: '', error: 'no-feeds-in-opml' }];
+
+    await q.completeSubmission(client, submissionId, {
+      accepted_count: 0,
+      rejected_count: 1,
+      queued_count: 0,
+      notify_email: null,
+      errors: rejected,
+    });
+
+    if (browser) {
+      return new Response(null, { status: 303, headers: { location: '/submit?error=1' } });
+    }
+
+    return json({
+      ok: false,
+      accepted: [],
+      rejected,
+      queued: 0,
+      total: 0,
+      submissionId,
+      statusUrl: `${siteUrl()}/submissions/${submissionId}`,
+    });
+  }
+
+  // Too big to crawl inside this request, so it is handed over instead: staged
+  // in bulk and released to the poller, which is what the batched uploader has
+  // done since it landed. This is the same handover for everyone who is not
+  // running it — a client with JavaScript off, curl, an agent posting JSON.
+  if (catalogue.length > STAGE_ABOVE) {
+    const staged = await stageAll(client, submissionId, catalogue);
+
+    await q.markImportReady(client, submissionId, {
+      entries_total: staged,
+      rejected_count: catalogue.length - staged,
+      // Nobody is owed a notification about an upload that staged nothing, for
+      // the reason `completeSubmission` documents: an address on a submission
+      // with no pending work reads as a finished import and is mailed at once.
+      notify_email: staged > 0 ? email : null,
+    });
+
+    if (browser) {
+      return new Response(null, {
+        status: 303,
+        headers: { location: `/submissions/${submissionId}` },
+      });
+    }
+
+    return json({
+      ok: staged > 0,
+      accepted: [],
+      rejected: [],
+      queued: 0,
+      // Staged, not queued: these are recorded and waiting for the poller to
+      // turn them into feeds, and the status page reports them as such.
+      pending: staged,
+      total: catalogue.length,
+      submissionId,
+      statusUrl: `${siteUrl()}/submissions/${submissionId}`,
+    });
+  }
+
   let resolveQueued;
   const queuedCount = new Promise((resolve) => {
     resolveQueued = resolve;
   });
 
   const opts = { submissionId, onQueued: (n) => resolveQueued(n) };
-  const work = (
-    opml === null ? submitCatalogue(client, entries, opts) : submitOpml(client, opml, opts)
-  ).then(async (result) => {
+  const work = submitCatalogue(client, catalogue, opts).then(async (result) => {
     await q.completeSubmission(client, submissionId, {
       accepted_count: result.accepted.length,
       rejected_count: result.rejected.length,
@@ -191,6 +317,38 @@ export async function POST(req) {
   });
 }
 
+/**
+ * Write a catalogue into staging, one statement at a time.
+ *
+ * Entries with nothing url-shaped in them are dropped rather than stored, the
+ * same way `/api/submit/stage` drops them: the drainer would only throw them
+ * out later, and they would sit in the table until it got round to it. What is
+ * dropped is still counted, by the caller, so the totals add up to what arrived.
+ *
+ * @param {import('@libsql/client').Client} client
+ * @param {string} submissionId
+ * @param {Array<{ url?: string, title?: string, siteUrl?: string|null }>} catalogue
+ * @returns {Promise<number>} rows written
+ */
+async function stageAll(client, submissionId, catalogue) {
+  let staged = 0;
+
+  for (let at = 0; at < catalogue.length; at += STAGE_CHUNK) {
+    const slice = catalogue
+      .slice(at, at + STAGE_CHUNK)
+      .map((entry) => ({
+        url: String(entry?.url ?? '').trim(),
+        title: entry?.title ?? null,
+        siteUrl: entry?.siteUrl ?? null,
+      }))
+      .filter((entry) => entry.url);
+
+    if (slice.length > 0) staged += await q.stageImportEntries(client, submissionId, slice);
+  }
+
+  return staged;
+}
+
 /**
  * Refuse an upload this endpoint cannot read in one piece.
  *

apps/web/src/app/submissions/[id]/page.jsx

@@ -57,7 +57,14 @@ export default async function SubmissionPage({ params }) {
   const staged = Number(progress.pending ?? 0);
   const done = progress.waiting === 0 && staged === 0;
   const settled = progress.crawled + progress.failed;
-  const percent = progress.queued === 0 ? 100 : Math.floor((settled / progress.queued) * 100);
+
+  // Everything this import will crawl, staged entries included. Counting only
+  // what is already queued made the bar read 100% and "finished" for the whole
+  // handover — which is now most of a large import's life, and is exactly the
+  // stretch where nothing has been crawled at all. A full bar over a heading
+  // that says "Import in progress" is worse than no bar.
+  const total = progress.queued + staged;
+  const percent = total === 0 ? 100 : Math.floor((settled / total) * 100);
 
   return (
     <>
@@ -76,7 +83,7 @@ export default async function SubmissionPage({ params }) {
         lines={lines}
         unit="feeds"
         verb="Crawl"
-        initial={{ total: progress.queued, settled, percent, done }}
+        initial={{ total, settled, percent, done }}
       />
 
       <dl className="stats">

apps/web/src/app/submit/Uploader.jsx

@@ -67,6 +67,7 @@ export default function Uploader({ shared = '' }) {
   /**
    * @type {[null | {
    *   phase: 'reading'|'saving'|'done'|'empty'|'error',
+   *   via: 'paste'|'file',
    *   name: string,
    *   percent: number,
    *   read: number,
@@ -89,13 +90,19 @@ export default function Uploader({ shared = '' }) {
   /**
    * Drive one import from beginning to end.
    *
-   * @param {{ kind: 'opml'|'list', source: File|string, size: number, name: string, email: string }} job
+   * @param {{ via: 'paste'|'file', kind: 'opml'|'list', source: File|string, size: number, name: string, email: string }} job
    */
   async function start(job) {
     /** @type {Array<{ at: number, text: string }>} */
     const lines = [];
     let state = {
       phase: /** @type {'reading'} */ ('reading'),
+      // Which form started this, kept apart from `kind` on purpose. `kind` is
+      // what the file turned out to hold and can change under us the moment it
+      // has been sniffed — a subscription list saved as .txt is uploaded
+      // through the file form and scanned as a list — and switching the panel
+      // on it put the progress bar for a file upload underneath the paste box.
+      via: job.via,
       kind: job.kind,
       name: job.name,
       percent: 0,
@@ -228,7 +235,14 @@ export default function Uploader({ shared = '' }) {
     if (value.length < PASTE_OBVIOUS && scanUrls(value).length <= PASTE_DIRECT) return;
 
     event.preventDefault();
-    start({ kind: 'list', source: value, size: value.length, name: 'your list', email: '' });
+    start({
+      via: 'paste',
+      kind: 'list',
+      source: value,
+      size: value.length,
+      name: 'your list',
+      email: '',
+    });
   }
 
   /**
@@ -243,6 +257,10 @@ export default function Uploader({ shared = '' }) {
 
     event.preventDefault();
     start({
+      via: 'file',
+      // Provisional: `sniffKind` decides for real once the head of the file has
+      // been read, because the field takes an OPML export and a plain list of
+      // URLs alike and the name on it is not always the truth.
       kind: 'opml',
       source: file,
       size: file.size,
@@ -272,7 +290,7 @@ export default function Uploader({ shared = '' }) {
         </div>
       </form>
 
-      {run?.kind === 'list' && <Progress run={run} onReset={() => setRun(null)} />}
+      {run?.via === 'paste' && <Progress run={run} onReset={() => setRun(null)} />}
 
       <form
         className="submit-box"
@@ -281,17 +299,18 @@ export default function Uploader({ shared = '' }) {
         encType="multipart/form-data"
         onSubmit={onUpload}
       >
-        <p className="eyebrow">Or upload an OPML file</p>
+        <p className="eyebrow">Or upload a file — OPML, or one feed per line</p>
         <input
           ref={fileRef}
           type="file"
           name="opml"
           accept=".opml,.xml,.txt,text/xml,text/plain"
-          aria-label="OPML file"
+          aria-label="OPML or list of feeds"
           disabled={busy}
         />
         <p className="hint">
-          Any size. The file is read here in your browser and the feeds are sent a few thousand at a
+          An OPML export from any reader, or a plain text file with one feed URL on each line. Any
+          size: the file is read here in your browser and the feeds are sent a few thousand at a
           time, so a 100 MB subscription list imports without the upload ever timing out — you will
           see it go, and get a status page to watch the crawl.
         </p>
@@ -310,7 +329,7 @@ export default function Uploader({ shared = '' }) {
         </div>
       </form>
 
-      {run?.kind === 'opml' && <Progress run={run} onReset={() => setRun(null)} />}
+      {run?.via === 'file' && <Progress run={run} onReset={() => setRun(null)} />}
     </>
   );
 }

apps/web/src/app/submit/page.jsx

@@ -6,7 +6,7 @@ import Uploader from './Uploader.jsx';
 export const metadata = {
   title: 'Submit a blog',
   description:
-    'Add a URL, a playlist, a list of URLs or an OPML file to the directory. No account needed.',
+    'Add a URL, a playlist, a list of URLs, an OPML export or a text file of feed URLs to the directory. No account needed.',
 };
 
 /**
@@ -33,8 +33,9 @@ export default async function SubmitPage({ searchParams }) {
         <p className="notice">
           That file is too big to send in one piece. The uploader below reads it in your browser and
           sends the feeds a few thousand at a time instead, which has no size limit worth the name —
-          but it needs JavaScript. With JavaScript off, split the file, or post the feeds to{' '}
-          <code>/api/submit</code> yourself.
+          but it needs JavaScript. With JavaScript off, split the file into pieces under 10 MB —
+          each one may hold as many feeds as it likes — or post them to <code>/api/submit</code>{' '}
+          yourself.
         </p>
       ) : (
         params.error && (