From 5e0eed2a18f20336d9d9f747259a2cdefcca9439 Mon Sep 17 00:00:00 2001 From: Logan Nguyen Date: Mon, 20 Jul 2026 12:47:33 -0700 Subject: [PATCH 1/6] feat(search): add scheduled credibility-list refresh pipeline Signed-off-by: Logan Nguyen --- .../workflows/credibility-list-refresh.yml | 104 ++++ docs/built-in-web-search.md | 2 + scripts/refresh-credibility-list.ts | 560 ++++++++++++++++++ .../src/websearch/credibility_domains.txt | 80 ++- 4 files changed, 714 insertions(+), 32 deletions(-) create mode 100644 .github/workflows/credibility-list-refresh.yml create mode 100644 scripts/refresh-credibility-list.ts diff --git a/.github/workflows/credibility-list-refresh.yml b/.github/workflows/credibility-list-refresh.yml new file mode 100644 index 00000000..e3aa8179 --- /dev/null +++ b/.github/workflows/credibility-list-refresh.yml @@ -0,0 +1,104 @@ +name: Credibility List Refresh + +# Scheduled maintenance for the keyless search engine's domain-credibility list +# (src-tauri/src/websearch/credibility.rs). Regenerates the autogenerated +# regions of credibility_domains.txt from upstream license-clean sources and +# opens a reviewable PR when the diff is non-empty. Mirrors the existing +# Renovate engine-bump pattern (#238): automation proposes, a human reviews and +# merges, updates ship with releases. The app itself never fetches these lists +# at runtime; this is dev/CI tooling only. +# +# Note: this PR is opened with the default GITHUB_TOKEN, so per +# https://github.com/peter-evans/create-pull-request/blob/main/docs/concepts-guidelines.md#triggering-further-workflow-runs +# it will not trigger this repo's other pull_request-triggered CI workflows. +# A reviewer must push an empty commit, or close and reopen the PR, to get CI +# to run on it before merging. + +on: + schedule: + # Monthly, 06:17 UTC on the 1st. Off-the-hour minute to avoid the + # scheduled-workflow thundering herd at :00. + - cron: '17 6 1 * *' + workflow_dispatch: + +permissions: + contents: write + pull-requests: write + +jobs: + refresh: + name: Refresh credibility list + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - name: Checkout repository + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + with: + fetch-depth: 1 + + - name: Setup Bun + uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0 + with: + bun-version: 1.3.11 + + - name: Install dependencies + run: bun install --frozen-lockfile + + - name: Run credibility list refresh + id: refresh + run: bun scripts/refresh-credibility-list.ts --summary refresh-summary.md + + - name: Append summary to job summary + if: always() + run: | + if [ -f refresh-summary.md ]; then + cat refresh-summary.md >> "$GITHUB_STEP_SUMMARY" + fi + + - name: Check for changes + id: diff + run: | + if git diff --quiet -- src-tauri/src/websearch/credibility_domains.txt; then + echo "changed=false" >> "$GITHUB_OUTPUT" + else + echo "changed=true" >> "$GITHUB_OUTPUT" + fi + + - name: Open pull request + if: steps.diff.outputs.changed == 'true' + env: + GH_TOKEN: ${{ github.token }} + run: | + set -euo pipefail + branch=chore/credibility-list-refresh + git config user.name 'github-actions[bot]' + git config user.email '41898282+github-actions[bot]@users.noreply.github.com' + git checkout -B "$branch" + git add src-tauri/src/websearch/credibility_domains.txt + git commit -s -m 'chore(search): refresh domain-credibility list' + git push --force origin "$branch" + + { + echo "Automated refresh of the autogenerated regions in" + echo "\`src-tauri/src/websearch/credibility_domains.txt\`" + echo "from upstream license-clean sources. See the summary below for" + echo "per-region counts and the added/removed domains." + echo "" + echo "This PR was opened with the default \`GITHUB_TOKEN\`, so it will" + echo "not trigger this repo's other pull_request-triggered CI" + echo "workflows. Push an empty commit, or close and reopen this PR," + echo "to run CI before merging." + echo "" + cat refresh-summary.md + } > pr-body.md + + existing=$(gh pr list --head "$branch" --state open --json number --jq '.[0].number // empty') + if [ -n "$existing" ]; then + gh pr edit "$existing" --body-file pr-body.md + else + gh pr create \ + --base main \ + --head "$branch" \ + --title 'chore(search): refresh domain-credibility list' \ + --body-file pr-body.md + fi diff --git a/docs/built-in-web-search.md b/docs/built-in-web-search.md index 9b412997..cf8b58b3 100644 --- a/docs/built-in-web-search.md +++ b/docs/built-in-web-search.md @@ -559,6 +559,8 @@ Code: `judge.rs`, `orchestrator.rs`. **Example.** Query `openai ceo` → DuckDuckGo ranks A,B,C; Mojeek ranks B,D → B’s RRF score wins → fetch B first. +**Maintenance.** The bulk-imported penalize clusters in `credibility_domains.txt` are refreshed by a scheduled GitHub Action that pulls the latest upstream license-clean spam/copycat lists and opens a reviewable PR when the diff is non-empty; a human still reviews and merges every change. The app itself never fetches these lists at runtime, so the no-phone-home privacy stance is unchanged; updates only ship inside a release. + Code: `engine.rs`, `credibility.rs`. --- diff --git a/scripts/refresh-credibility-list.ts b/scripts/refresh-credibility-list.ts new file mode 100644 index 00000000..b98ec6c8 --- /dev/null +++ b/scripts/refresh-credibility-list.ts @@ -0,0 +1,560 @@ +// Refreshes the autogenerated regions of src-tauri/src/websearch/credibility_domains.txt +// from upstream license-clean ad/SEO-spam-blocklist sources. Run manually with +// `bun scripts/refresh-credibility-list.ts` or via the scheduled +// credibility-list-refresh GitHub Action (see .github/workflows/credibility-list-refresh.yml). +// +// Only text between "# BEGIN AUTOGEN " / "# END AUTOGEN " +// sentinel comments is rewritten; every other byte in the file (the drop section, +// the boost section, and the hand-curated penalize clusters) is left untouched. +// This mirrors the existing Renovate engine-bump pattern (#238): automation +// proposes a diff, a human reviews and merges it. The app itself never fetches +// these lists at runtime; this script is dev/CI tooling only. + +import { promises as dns } from 'node:dns'; +import { mkdtemp, readFile, rename, rm, writeFile } from 'node:fs/promises'; +import { dirname, join, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +/** Byte cap on any single fetched upstream response, a guard against a hung or hostile server. */ +const MAX_RESPONSE_BYTES = 5 * 1024 * 1024; + +/** Minimum domains a parsed region must yield; fewer trips the truncation/hijack guard. */ +const MIN_REGION_DOMAINS = 10; + +/** Domain count above which a region is flagged in the summary as suspiciously large. */ +const WARN_REGION_MAX_DOMAINS = 5000; + +/** DNS lookup timeout for the drop-section liveness check, per domain. */ +const DNS_TIMEOUT_MS = 4000; + +const repoRoot = fileURLToPath(new URL('../', import.meta.url)); +const CREDIBILITY_FILE = resolve( + repoRoot, + 'src-tauri/src/websearch/credibility_domains.txt', +); + +/** The two upstream line formats an autogen source can be published in. */ +type SourceFormat = 'ublacklist' | 'domain-list'; + +/** One upstream source feeding a single autogen sentinel region. */ +interface Source { + /** Sentinel id, must match "# BEGIN/END AUTOGEN " in the txt file. */ + id: string; + /** Raw-content URL fetched over HTTPS. */ + url: string; + /** Line format the fetched text is in. */ + format: SourceFormat; + /** One-line courtesy credit written at the top of the rewritten region. */ + credit: string; +} + +const SOURCES: readonly Source[] = [ + { + id: 'quenhus-seo-spam', + url: 'https://raw.githubusercontent.com/quenhus/uBlock-Origin-dev-filter/main/dist/other_format/domains/seo_spam.txt', + format: 'domain-list', + credit: + '# source: quenhus/uBlock-Origin-dev-filter (dist/other_format/domains/seo_spam.txt), Unlicense', + }, + { + id: 'quenhus-wikipedia-copycats', + url: 'https://raw.githubusercontent.com/quenhus/uBlock-Origin-dev-filter/main/dist/other_format/domains/wikipedia_copycats.txt', + format: 'domain-list', + credit: + '# source: quenhus/uBlock-Origin-dev-filter (dist/other_format/domains/wikipedia_copycats.txt), Unlicense', + }, + { + id: 'arosh-stackoverflow-copycats', + url: 'https://raw.githubusercontent.com/arosh/ublacklist-stackoverflow-translation/master/uBlacklist.txt', + format: 'ublacklist', + credit: + '# source: arosh/ublacklist-stackoverflow-translation (uBlacklist.txt), CC0-1.0', + }, +]; + +/** Verdict for one drop-list domain's liveness probe. */ +type LivenessVerdict = 'resolves' | 'possibly dead' | 'inconclusive'; + +/** + * Fetches `url` over HTTPS as text, aborting if the response body exceeds + * `maxBytes`. Streams the body rather than trusting a Content-Length header, + * since that header can be absent or wrong. Throws on a non-2xx status, a + * missing body, or the size cap being exceeded. + */ +async function fetchWithCap(url: string, maxBytes: number): Promise { + const response = await fetch(url); + if (!response.ok) { + throw new Error(`fetch ${url} failed: HTTP ${response.status}`); + } + const body = response.body; + if (!body) { + throw new Error(`fetch ${url} returned no response body`); + } + const reader = body.getReader(); + const chunks: Uint8Array[] = []; + let total = 0; + for (;;) { + const { done, value } = await reader.read(); + if (done) { + break; + } + total += value.byteLength; + if (total > maxBytes) { + await reader.cancel(); + throw new Error( + `fetch ${url} exceeded ${maxBytes} byte cap (truncated read at ${total} bytes)`, + ); + } + chunks.push(value); + } + const buffer = Buffer.concat(chunks.map((c) => Buffer.from(c))); + return buffer.toString('utf8'); +} + +/** + * Validates and normalizes one extracted domain token. Lowercases the token, + * strips any character outside `[a-z0-9.-]` (defense against comment injection + * or control characters riding in on a hijacked upstream file), drops a + * leading `www.` label (the file's own header contract is "no scheme or + * www"; a `www.`-only entry would not cover the bare domain via the parser's + * host-to-parent suffix walk, silently weakening coverage), then requires the + * result to look like a registrable domain: at least two dot-separated + * labels, each label alphanumeric-with-internal-hyphens only. Only the + * cosmetic `www.` prefix is stripped, not reduced to eTLD+1 in general: the + * list intentionally carries meaningful subdomains (e.g. + * `agecalculator.iamrohit.in`), which a general collapse would over-widen. + * Returns null for anything that fails the shape check, so the caller can + * drop it silently. + */ +function normalizeDomainToken(raw: string): string | null { + const lowered = raw.toLowerCase(); + const stripped = lowered.replace(/[^a-z0-9.-]/g, ''); + const withoutWww = stripped.replace(/^www\./, ''); + const labels = withoutWww.split('.'); + if (labels.length < 2) { + return null; + } + const labelPattern = /^[a-z0-9]([a-z0-9-]*[a-z0-9])?$/; + if (!labels.every((label) => label.length > 0 && labelPattern.test(label))) { + return null; + } + return withoutWww; +} + +/** + * Parses the arosh uBlacklist match-pattern format: lines like + * `*://code-examples.net/*` or a wildcard-subdomain form `*://*.voidcc.com/*`. + * Strips the `*://` scheme prefix, an optional leading `*.` wildcard label, + * and everything from the first `/` onward (path-scoped entries collapse to + * their host), then normalizes what remains as a domain token. + */ +function parseUblacklistFormat(text: string): string[] { + const domains: string[] = []; + for (const rawLine of text.split('\n')) { + const line = rawLine.trim(); + if (line.length === 0) { + continue; + } + const withoutScheme = line.replace(/^\*:\/\//, ''); + const withoutWildcardSubdomain = withoutScheme.replace(/^\*\./, ''); + const host = withoutWildcardSubdomain.split('/')[0] ?? ''; + const normalized = normalizeDomainToken(host); + if (normalized) { + domains.push(normalized); + } + } + return domains; +} + +/** + * Parses the quenhus `dist/other_format/domains/*.txt` format: `#`-prefixed + * comments and blank lines are skipped, and each remaining line is a bare + * domain, optionally followed by a path (`brianlovin.com/hn`), which is + * dropped by keeping only the portion before the first `/`. + */ +function parseDomainListFormat(text: string): string[] { + const domains: string[] = []; + for (const rawLine of text.split('\n')) { + const line = rawLine.trim(); + if (line.length === 0 || line.startsWith('#')) { + continue; + } + const host = line.split('/')[0] ?? ''; + const normalized = normalizeDomainToken(host); + if (normalized) { + domains.push(normalized); + } + } + return domains; +} + +/** Parses `text` per `format` into a raw (unsorted, possibly duplicated) domain list. */ +function parseSource(text: string, format: SourceFormat): string[] { + return format === 'ublacklist' + ? parseUblacklistFormat(text) + : parseDomainListFormat(text); +} + +/** Lowercases, deduplicates, and lexicographically sorts a domain list for deterministic output. */ +function dedupeSort(domains: readonly string[]): string[] { + return Array.from(new Set(domains.map((d) => d.toLowerCase()))).sort(); +} + +/** One sentinel-delimited region located inside the credibility txt file. */ +interface Region { + /** Line index of the "# BEGIN AUTOGEN " marker. */ + beginIndex: number; + /** Line index of the "# END AUTOGEN " marker. */ + endIndex: number; + /** Domain lines currently between the markers (excludes credit comment lines). */ + domains: string[]; +} + +/** + * Locates the sentinel region for `sourceId` inside `lines` (the credibility + * file split on newlines). Returns the marker line indices plus the domain + * lines currently inside the region (any line starting with `#` between the + * markers is a courtesy-credit comment, not a domain, and is excluded). + * Throws if the markers are missing or malformed, since a missing sentinel + * means the file was hand-edited out of sync with this script's expectations. + */ +function findRegion(lines: readonly string[], sourceId: string): Region { + const beginMarker = `# BEGIN AUTOGEN ${sourceId}`; + const endMarker = `# END AUTOGEN ${sourceId}`; + const beginIndex = lines.indexOf(beginMarker); + const endIndex = lines.indexOf(endMarker); + if (beginIndex === -1 || endIndex === -1 || endIndex < beginIndex) { + throw new Error( + `could not locate sentinel region for "${sourceId}" (expected "${beginMarker}" / "${endMarker}")`, + ); + } + const domains = lines + .slice(beginIndex + 1, endIndex) + .map((l) => l.trim()) + .filter((l) => l.length > 0 && !l.startsWith('#')); + return { beginIndex, endIndex, domains }; +} + +/** + * Rewrites the sentinel region for `source` inside `lines`, replacing + * everything between its BEGIN/END markers with the courtesy credit line + * followed by the sorted `domains`. The markers themselves, and every line + * outside them, are left untouched. Returns a new line array. + */ +function replaceRegion( + lines: readonly string[], + source: Source, + domains: readonly string[], +): string[] { + const { beginIndex, endIndex } = findRegion(lines, source.id); + const replacement = [source.credit, ...domains]; + return [ + ...lines.slice(0, beginIndex + 1), + ...replacement, + ...lines.slice(endIndex), + ]; +} + +/** + * Parses the credibility file's `# drop` section into a domain list, mirroring + * the section-header handling in src-tauri/src/websearch/credibility.rs: a + * line reading exactly `# drop`, `# penalize`, or `# boost` switches section; + * any other `#` line (including a BEGIN/END sentinel) is a comment; a blank + * line is skipped; anything else inside the `# drop` section is a domain. + */ +function extractDropSection(fileText: string): string[] { + const domains: string[] = []; + let inDrop = false; + for (const rawLine of fileText.split('\n')) { + const line = rawLine.trim(); + if (line.length === 0) { + continue; + } + if (line.startsWith('#')) { + const header = line.slice(1).trim(); + if (header === 'drop' || header === 'penalize' || header === 'boost') { + inDrop = header === 'drop'; + } + continue; + } + if (inDrop) { + domains.push(line.toLowerCase()); + } + } + return domains; +} + +/** + * Resolves `domain` with a bounded timeout to classify it as still live, + * possibly dead, or inconclusive. A successful lookup is "resolves"; an + * ENOTFOUND/ENODATA error (the DNS server affirmatively has no record) is + * "possibly dead"; anything else (timeout, network failure, other resolver + * errors) is "inconclusive" because it does not prove the domain is gone. + */ +async function checkLiveness(domain: string): Promise { + let timeoutHandle: ReturnType | undefined; + const timeout = new Promise<'timeout'>((resolve) => { + timeoutHandle = setTimeout(() => resolve('timeout'), DNS_TIMEOUT_MS); + }); + try { + const outcome = await Promise.race([ + dns.lookup(domain).then(() => 'resolved' as const), + timeout, + ]); + if (outcome === 'timeout') { + return 'inconclusive'; + } + return 'resolves'; + } catch (err) { + const code = (err as NodeJS.ErrnoException).code; + if (code === 'ENOTFOUND' || code === 'ENODATA') { + return 'possibly dead'; + } + return 'inconclusive'; + } finally { + clearTimeout(timeoutHandle); + } +} + +/** One row of the drop-list liveness report. */ +interface LivenessResult { + domain: string; + verdict: LivenessVerdict; +} + +/** Runs `checkLiveness` over every drop-section domain, in parallel. */ +async function checkDropListLiveness( + domains: readonly string[], +): Promise { + return Promise.all( + domains.map(async (domain) => ({ + domain, + verdict: await checkLiveness(domain), + })), + ); +} + +/** Writes `content` to `path` atomically: write to a sibling temp file, then rename over the target. */ +async function writeFileAtomic(path: string, content: string): Promise { + const tmpDir = await mkdtemp(join(dirname(path), '.refresh-tmp-')); + const tmpPath = join(tmpDir, 'credibility_domains.txt'); + try { + await writeFile(tmpPath, content, 'utf8'); + await rename(tmpPath, path); + } finally { + await rm(tmpDir, { recursive: true, force: true }); + } +} + +/** Per-region before/after domain counts and set diff, for the summary. */ +interface RegionReport { + sourceId: string; + previousCount: number; + newCount: number; + added: string[]; + removed: string[]; + warnings: string[]; +} + +/** Diffs `previous` against `next`, returning domains only in one side. */ +function diffDomains( + previous: readonly string[], + next: readonly string[], +): { added: string[]; removed: string[] } { + const previousSet = new Set(previous); + const nextSet = new Set(next); + return { + added: next.filter((d) => !previousSet.has(d)).sort(), + removed: previous.filter((d) => !nextSet.has(d)).sort(), + }; +} + +/** + * Renders the markdown maintenance summary: per-section counts (drop / + * penalize / boost), per-region before/after counts with added/removed + * domains, any size-anomaly warnings, and the drop-list liveness report. + */ +function renderSummary( + sectionCounts: { drop: number; penalize: number; boost: number }, + regionReports: readonly RegionReport[], + liveness: readonly LivenessResult[], +): string { + const lines: string[] = []; + lines.push('# Credibility list refresh summary'); + lines.push(''); + lines.push('## Section counts'); + lines.push(`- drop: ${sectionCounts.drop}`); + lines.push(`- penalize: ${sectionCounts.penalize}`); + lines.push(`- boost: ${sectionCounts.boost}`); + lines.push(''); + lines.push('## Autogen regions'); + for (const report of regionReports) { + lines.push(`### ${report.sourceId}`); + lines.push( + `- domains: ${report.previousCount} -> ${report.newCount} (added ${report.added.length}, removed ${report.removed.length})`, + ); + for (const warning of report.warnings) { + lines.push(`- warning: ${warning}`); + } + if (report.added.length > 0) { + lines.push('
Added domains'); + lines.push(''); + lines.push(report.added.map((d) => `- ${d}`).join('\n')); + lines.push(''); + lines.push('
'); + } + if (report.removed.length > 0) { + lines.push('
Removed domains'); + lines.push(''); + lines.push(report.removed.map((d) => `- ${d}`).join('\n')); + lines.push(''); + lines.push('
'); + } + lines.push(''); + } + lines.push('## Drop-list liveness (report only, list never modified)'); + const possiblyDead = liveness.filter((r) => r.verdict === 'possibly dead'); + const inconclusive = liveness.filter((r) => r.verdict === 'inconclusive'); + lines.push( + `- ${liveness.length} domains checked: ${liveness.length - possiblyDead.length - inconclusive.length} resolve, ${possiblyDead.length} possibly dead, ${inconclusive.length} inconclusive`, + ); + if (possiblyDead.length > 0) { + lines.push( + '- possibly dead: ' + possiblyDead.map((r) => r.domain).join(', '), + ); + } + if (inconclusive.length > 0) { + lines.push( + '- inconclusive: ' + inconclusive.map((r) => r.domain).join(', '), + ); + } + lines.push(''); + return lines.join('\n'); +} + +/** Reads `--summary ` from argv, if present. */ +function parseArgs(argv: readonly string[]): { summaryPath: string | null } { + const idx = argv.indexOf('--summary'); + if (idx === -1 || idx + 1 >= argv.length) { + return { summaryPath: null }; + } + return { summaryPath: argv[idx + 1] ?? null }; +} + +/** + * Entry point: fetches every upstream source, parses and validates it, + * rewrites only the matching sentinel regions of the credibility file, runs + * the drop-list liveness check, and writes the markdown summary (to stdout + * always, and to `--summary ` if given). Exits nonzero without touching + * the credibility file if any region parses to fewer than MIN_REGION_DOMAINS + * domains, since that indicates upstream truncation or hijack. + */ +async function main(): Promise { + const { summaryPath } = parseArgs(process.argv.slice(2)); + const originalText = await readFile(CREDIBILITY_FILE, 'utf8'); + const originalLines = originalText.split('\n'); + + const regionReports: RegionReport[] = []; + let updatedLines = originalLines; + + for (const source of SOURCES) { + const previousRegion = findRegion(originalLines, source.id); + const fetched = await fetchWithCap(source.url, MAX_RESPONSE_BYTES); + const parsed = parseSource(fetched, source.format); + const domains = dedupeSort(parsed); + if (domains.length < MIN_REGION_DOMAINS) { + throw new Error( + `source "${source.id}" parsed to only ${domains.length} domains (minimum ${MIN_REGION_DOMAINS}); ` + + 'refusing to write, this looks like upstream truncation or a hijacked response', + ); + } + const warnings: string[] = []; + if (domains.length > WARN_REGION_MAX_DOMAINS) { + warnings.push( + `region has ${domains.length} domains, above the ${WARN_REGION_MAX_DOMAINS} sanity threshold`, + ); + } + if ( + previousRegion.domains.length > 0 && + domains.length < previousRegion.domains.length / 2 + ) { + warnings.push( + `region shrank from ${previousRegion.domains.length} to ${domains.length} domains (more than half)`, + ); + } + const { added, removed } = diffDomains(previousRegion.domains, domains); + regionReports.push({ + sourceId: source.id, + previousCount: previousRegion.domains.length, + newCount: domains.length, + added, + removed, + warnings, + }); + updatedLines = replaceRegion(updatedLines, source, domains); + } + + const updatedText = updatedLines.join('\n'); + const dropDomains = extractDropSection(updatedText); + const penalizeSectionCount = countSectionDomains(updatedText, 'penalize'); + const boostSectionCount = countSectionDomains(updatedText, 'boost'); + + const liveness = await checkDropListLiveness(dropDomains); + + const summary = renderSummary( + { + drop: dropDomains.length, + penalize: penalizeSectionCount, + boost: boostSectionCount, + }, + regionReports, + liveness, + ); + + await writeFileAtomic(CREDIBILITY_FILE, updatedText); + + process.stdout.write(summary + '\n'); + if (summaryPath) { + await writeFile(resolve(summaryPath), summary, 'utf8'); + } +} + +/** + * Counts domains in a named section (`drop` / `penalize` / `boost`), the same + * way `extractDropSection` does for `drop`, generalized to any of the three. + */ +function countSectionDomains(fileText: string, section: string): number { + let count = 0; + let inSection = false; + for (const rawLine of fileText.split('\n')) { + const line = rawLine.trim(); + if (line.length === 0) { + continue; + } + if (line.startsWith('#')) { + const header = line.slice(1).trim(); + if (header === 'drop' || header === 'penalize' || header === 'boost') { + inSection = header === section; + } + continue; + } + if (inSection) { + count += 1; + } + } + return count; +} + +main().catch(async (err) => { + const message = err instanceof Error ? err.message : String(err); + process.stderr.write(`refresh-credibility-list: ${message}\n`); + const { summaryPath } = parseArgs(process.argv.slice(2)); + const failureSummary = `# Credibility list refresh summary\n\nRefresh failed, credibility_domains.txt was not modified:\n\n\`\`\`\n${message}\n\`\`\`\n`; + process.stdout.write(failureSummary); + if (summaryPath) { + await writeFile(resolve(summaryPath), failureSummary, 'utf8').catch( + () => undefined, + ); + } + process.exitCode = 1; +}); diff --git a/src-tauri/src/websearch/credibility_domains.txt b/src-tauri/src/websearch/credibility_domains.txt index 7bbb035f..7cff449f 100644 --- a/src-tauri/src/websearch/credibility_domains.txt +++ b/src-tauri/src/websearch/credibility_domains.txt @@ -7,6 +7,10 @@ # Lines beginning with # other than the three section headers are courtesy source # credits and are ignored by the parser. Every domain traces to a CC0/Unlicense # source or individual editorial verification; no attribution is required. +# Regions between "# BEGIN AUTOGEN " and "# END AUTOGEN " are +# rewritten verbatim by scripts/refresh-credibility-list.ts; do not hand-edit domains +# inside them, hand-edit the upstream list instead. Everything outside those regions +# is hand-curated and left untouched by the script. # drop @@ -38,12 +42,14 @@ kulturellen.net # penalize -# cluster: quenhus/uBlock-Origin-dev-filter (dist/duckduckgo/seo_spam.txt) -- Unlicense +# BEGIN AUTOGEN quenhus-seo-spam +# source: quenhus/uBlock-Origin-dev-filter (dist/other_format/domains/seo_spam.txt), Unlicense 900913.ru actingcollegeses.com answerforyou.net azazworld.com bong-faq.com +brianlovin.com britguidenewyork.net code-discuss.com codertw.com @@ -77,10 +83,13 @@ thesassway.com topcode.in unbate.com worldgrowthtoday.com +# END AUTOGEN quenhus-seo-spam -# cluster: quenhus/uBlock-Origin-dev-filter (dist/duckduckgo/wikipedia_copycats.txt) -- Unlicense +# BEGIN AUTOGEN quenhus-wikipedia-copycats +# source: quenhus/uBlock-Origin-dev-filter (dist/other_format/domains/wikipedia_copycats.txt), Unlicense 360wiki.ru accordeonmuseum.nl +algebra.com buildwiki.ru cyclowiki.org datewiki.ru @@ -98,7 +107,9 @@ hmong.ru livepcwiki.ru mediawiki.feverous.co.uk ru-wiki.ru +scholarship.edu.vn second.wiki +secret-bases.co.uk static.hlt.bme.hu sv.abcdef.wiki th.hmong.wiki @@ -122,51 +133,77 @@ wikiwand.com wikizero.com wiwa.wiki zxc.wiki +# END AUTOGEN quenhus-wikipedia-copycats -# cluster: arosh/ublacklist-stackoverflow-translation (uBlacklist.txt) -- CC0-1.0 +# BEGIN AUTOGEN arosh-stackoverflow-copycats +# source: arosh/ublacklist-stackoverflow-translation (uBlacklist.txt), CC0-1.0 16892.net 1r1g.com 55276.net +5axxw.com 711web.com 9ishenzhen.com 9to5answer.com ajaxhispano.com answer-id.com +answerlib.com +anycodings.com +appsloveworld.com arip-photo.org ask-dev.ru askcodez.com +binarydevelop.com bitcoden.com cainiaojiaocheng.com code-examples.net +codebaoku.com codegrepr.com codeguides.site +codenong.com +coder.work coderoad.ru codeutility.org copyprogramming.com daplus.net de-vraag.com +debugcn.com +debugko.com +desenv-web-rp.com developreference.com digitrain.ru doraprojects.net dovov.com +edureka.co errorsfixing.com exchangetuts.com +fixes.pub fluffyfables.com flutterhq.com fmihm.org fullstackuser.com +generacodice.com +intellipaat.com iquestion.pro isolution.pro +it-swarm-fr.com +it-swarm-ja.com +it-swarm-ja.tech +it-swarm.jp.net itecnote.com itecnotes.com itectec.com iteramos.com +javaer101.com +javafixing.com jike.in jonic.cn jtuto.com +kutombawewe.net kzen.dev +linuxfixes.com living-sun.com mediatagtw.com +mejorcodigo.com microeducate.tech mlink.in narkive.jp @@ -196,9 +233,11 @@ qastack.net.bd qastack.ru qastack.vn qi-u.com +querythreads.com question-it.com questu.ru routinepanic.com +semicolonworld.com shenghuobao.net shenzhenjia.cn shenzhenjia.net @@ -207,6 +246,7 @@ softwareuser.asklobster.com solveforum.com splunktool.com sqlite.in +stackfinder.ru stackguides.com stackoom.com stackovergo.com @@ -221,45 +261,21 @@ vigge.cn vigge.net vigges.net voidcc.com -web-dev-qa-db-fra.com -webdevask.com -webdevdesigner.com -wujigu.com -5axxw.com -answerlib.com -anycodings.com -appsloveworld.com -binarydevelop.com -codebaoku.com -codenong.com -coder.work -debugcn.com -debugko.com -desenv-web-rp.com -fixes.pub -generacodice.com -it-swarm-fr.com -it-swarm-ja.com -it-swarm-ja.tech -it-swarm.jp.net -javaer101.com -javafixing.com -kutombawewe.net -linuxfixes.com -mejorcodigo.com -querythreads.com -semicolonworld.com -stackfinder.ru wake-up-neo.net web-dev-qa-db-fr.com +web-dev-qa-db-fra.com web-dev-qa-db-ja.com web-dev-qa-db-pt.com web-dev-qa.com +webdevask.com +webdevdesigner.com +wujigu.com xstack.ru xstack.us yaoply.com zaizhele.cn zaizhele.net +# END AUTOGEN arosh-stackoverflow-copycats # cluster: editorial judgment -- thin finance/ranking-content aggregators (user-observed offenders; only 2 independently confirmed, category needs expansion -- see report) wealthrank.in From 702ec3307152233a27a58ccdf8d6ebca21aee50f Mon Sep 17 00:00:00 2001 From: Logan Nguyen Date: Mon, 20 Jul 2026 12:57:27 -0700 Subject: [PATCH 2/6] feat(search): add manual eval answer-capture workflow Signed-off-by: Logan Nguyen --- .github/workflows/search-eval-capture.yml | 168 ++++++++++++++++++++++ docs/search-eval.md | 15 ++ 2 files changed, 183 insertions(+) create mode 100644 .github/workflows/search-eval-capture.yml diff --git a/.github/workflows/search-eval-capture.yml b/.github/workflows/search-eval-capture.yml new file mode 100644 index 00000000..10c41d0b --- /dev/null +++ b/.github/workflows/search-eval-capture.yml @@ -0,0 +1,168 @@ +name: Search Eval Capture + +# Manual (phase 1) eval-run workflow for the built-in search pipeline, per +# issue #309. Runs the hermetic eval-corpus unit tests unconditionally, and +# optionally the live answer-capture harness (tests/live_answer_capture.rs) +# against the real internet. +# +# Search engines heavily rate-limit GitHub-hosted runner IPs. Before spending +# time on an engine build, the live-capture job first sends one cheap request +# to each keyless engine the pipeline actually uses (DuckDuckGo's html +# endpoint, Mojeek) and checks for HTTP 200 with no captcha/block markers. If +# either engine is blocked, the job fails fast with a step-summary explanation +# instead of burning the rest of the timeout on a build that would only +# produce empty capture rows. See docs/search-eval.md for what each mode does. +# +# Phase 2 (a live-model judge step over two capture runs) is documented as +# future work in docs/search-eval.md and is not built here; this workflow only +# covers phase 1, the capture harness. + +on: + workflow_dispatch: + inputs: + mode: + description: 'Which checks to run' + type: choice + options: + - live + - hermetic + default: live + +permissions: + contents: read + +jobs: + hermetic-corpus-checks: + name: Hermetic corpus checks + runs-on: macos-15 + timeout-minutes: 30 + steps: + - name: Checkout repository + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + with: + fetch-depth: 1 + + - name: Setup Bun + uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0 + with: + bun-version: 1.3.11 + + - name: Install dependencies + run: bun install --frozen-lockfile + + - name: Cache llama.cpp sidecar + uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + with: + path: src-tauri/binaries + key: llama-cpp-${{ runner.os }}-${{ hashFiles('scripts/ensure-llama-server.ts') }} + + - name: Build llama-server sidecar + run: bun run engine:ensure + + - name: Run eval-corpus unit tests (no network, no model) + working-directory: src-tauri + run: cargo test --test live_answer_quality_eval + + live-capture: + name: Live answer capture + needs: hermetic-corpus-checks + if: ${{ github.event.inputs.mode == 'live' }} + runs-on: macos-15 + timeout-minutes: 60 + steps: + - name: Checkout repository + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + with: + fetch-depth: 1 + + - name: Probe search-engine reachability + id: probe + run: | + set -uo pipefail + UA="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" + CAPTCHA_MARKERS=(anomaly-modal challenge-form cf-challenge hcaptcha recaptcha captcha-wrap altcha-widget) + + # Sends one real query to `name`'s endpoint (the exact requests the + # pipeline's engine.rs builds), and fails if the response isn't a + # clean 200 with no bot-challenge markup. + check_engine() { + name="$1" + shift + body="/tmp/probe-${name}.html" + http_code=$(curl -s -o "$body" -w '%{http_code}' "$@") + if [ "$http_code" != "200" ]; then + echo "blocked (${name}): HTTP ${http_code}" + return 1 + fi + for marker in "${CAPTCHA_MARKERS[@]}"; do + if grep -qi "$marker" "$body"; then + echo "blocked (${name}): captcha marker '${marker}' in response body" + return 1 + fi + done + echo "reachable (${name}): HTTP 200, no captcha markers" + return 0 + } + + ddg_ok=0 + check_engine ddg \ + -A "$UA" \ + -H "Accept: text/html,application/xhtml+xml" \ + --data "q=rust+programming+language&kl=wt-wt&b=" \ + https://html.duckduckgo.com/html/ || ddg_ok=1 + + mojeek_ok=0 + check_engine mojeek \ + -A "$UA" \ + -H "Accept: text/html,application/xhtml+xml" \ + -G --data-urlencode "q=rust programming language" \ + https://www.mojeek.com/search || mojeek_ok=1 + + if [ "$ddg_ok" -ne 0 ] || [ "$mojeek_ok" -ne 0 ]; then + echo "::error::Search-engine reachability probe failed from this GitHub-hosted runner. Per issue #309's documented caveat, keyless search engines heavily rate-limit runner IPs; the live answer-capture harness stays a dev-machine tool. Failing early instead of spending the rest of the timeout on an engine build." + { + echo "## Live capture: search-engine probe failed" + echo "" + echo "DuckDuckGo and/or Mojeek returned a non-200 response or a captcha/block marker to this runner's IP." + echo "" + echo "This confirms the caveat in issue #309: GitHub-hosted runner IPs are rate-limited or blocked by the keyless search engines the pipeline uses. \`live_answer_capture.rs\` stays a dev-machine tool; run it locally instead:" + echo "" + echo '```sh' + echo "cargo test --test live_answer_capture -- --ignored --nocapture --test-threads=1" + echo '```' + echo "" + echo "\`hermetic-corpus-checks\` still ran and passed; only this job is affected." + } >> "$GITHUB_STEP_SUMMARY" + exit 1 + fi + + echo "Both engines reachable from this runner; continuing to engine build and live capture." >> "$GITHUB_STEP_SUMMARY" + + - name: Setup Bun + uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0 + with: + bun-version: 1.3.11 + + - name: Install dependencies + run: bun install --frozen-lockfile + + - name: Cache llama.cpp sidecar + uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + with: + path: src-tauri/binaries + key: llama-cpp-${{ runner.os }}-${{ hashFiles('scripts/ensure-llama-server.ts') }} + + - name: Build llama-server sidecar + run: bun run engine:ensure + + - name: Run live answer-capture harness + working-directory: src-tauri + run: cargo test --test live_answer_capture -- --ignored --nocapture --test-threads=1 + + - name: Upload captured answers + if: always() + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: eval-answers + path: src-tauri/target/eval/answers-*.jsonl + retention-days: 90 diff --git a/docs/search-eval.md b/docs/search-eval.md index 480e0360..eda86dde 100644 --- a/docs/search-eval.md +++ b/docs/search-eval.md @@ -118,6 +118,21 @@ Deterministic-first, never pairwise, never a rubric score: Before trusting the gate, hand-label a sample of 30-40 rows spanning all three graded sources and volatility buckets with your own CORRECT/INCORRECT/NOT_ATTEMPTED judgment against the same predicted answers the harness generated, then run the live judge (majority-of-3) over that identical sample and compute agreement: the fraction of rows where the judge's majority verdict matches your hand label. This is the only way to know whether the local judge model is reliable enough for `CONFIDENTLY_WRONG_GATE` to mean anything, and is a prerequisite for tightening it. It needs a live model and a human rater (Logan), so it is a documented manual procedure, not code in this repository. +## CI workflows + +`.github/workflows/search-eval-capture.yml` is a `workflow_dispatch`-only workflow, triggered by hand from the Actions tab, with a `mode` choice input (`live`, the default, or `hermetic`). + +Two jobs: + +- **`hermetic-corpus-checks`** always runs, regardless of `mode`. It builds the sidecar (needed for the crate to compile at all) and runs `cargo test --test live_answer_quality_eval`, which is every non-ignored test in that file: the corpus-loading, grading-logic, and composition unit tests. No network, no live model; this is the closest thing this repo has to a CI gate on the eval corpus itself. +- **`live-capture`** runs only when `mode` is `live`, and only after `hermetic-corpus-checks` passes. Before building anything, it sends one real request to each keyless engine the pipeline uses (DuckDuckGo's `html` endpoint, Mojeek) and checks for a clean HTTP 200 with no captcha/block markers. GitHub-hosted runner IPs are heavily rate-limited by keyless search engines; if either engine blocks the probe, the job fails immediately with a `::error::` annotation and a step-summary explanation, without spending the rest of the timeout on an engine build that would only produce empty capture rows. If the probe passes, it builds the engine sidecar and runs `cargo test --test live_answer_capture -- --ignored --nocapture --test-threads=1`, then uploads `src-tauri/target/eval/answers-*.jsonl` as a 90-day artifact (uploaded even on a failed test step, so a partial capture is still inspectable). + +A probe failure is not itself a bug: it means the harness has to stay a dev-machine tool for now, run by hand as documented above, rather than something CI can execute unattended. If the probe consistently passes over time, that is the "proves viable" signal this doc's phase 2 note below is gated on. + +A live-model judge step over two capture runs (a baseline and a candidate) is documented as future work, gated on phase 1's reachability probe proving reliable in practice; it is not built in this workflow. See "Judging: pairwise superseded by absolute SimpleQA grading" below for why the judge design for that step, if it is ever automated, would not be the pairwise comparison this doc's history once sketched. + +The domain-credibility list has its own scheduled refresh workflow (`.github/workflows/credibility-list-refresh.yml`); see [built-in-web-search.md](./built-in-web-search.md) for that automation. + ## Judging: pairwise superseded by absolute SimpleQA grading An earlier revision of this doc sketched pairwise, position-swapped LLM-as-judge scoring, following [Brave's published search-eval methodology](https://brave.com/blog/), as the intended next step once two capture runs existed to compare: an LLM judge shown both runs' answers in one order, then the swapped order, with a majority vote across both orderings deciding the winner or a tie. From 5ecb32acaf9ac9acf4b880028b6a01a5add64f43 Mon Sep 17 00:00:00 2001 From: Logan Nguyen Date: Mon, 20 Jul 2026 13:09:27 -0700 Subject: [PATCH 3/6] fix(search): harden maintenance pipeline per review Signed-off-by: Logan Nguyen --- scripts/refresh-credibility-list.ts | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/scripts/refresh-credibility-list.ts b/scripts/refresh-credibility-list.ts index b98ec6c8..b5511c95 100644 --- a/scripts/refresh-credibility-list.ts +++ b/scripts/refresh-credibility-list.ts @@ -152,7 +152,7 @@ function parseUblacklistFormat(text: string): string[] { const domains: string[] = []; for (const rawLine of text.split('\n')) { const line = rawLine.trim(); - if (line.length === 0) { + if (line.length === 0 || line.startsWith('#')) { continue; } const withoutScheme = line.replace(/^\*:\/\//, ''); @@ -297,8 +297,13 @@ async function checkLiveness(domain: string): Promise { timeoutHandle = setTimeout(() => resolve('timeout'), DNS_TIMEOUT_MS); }); try { + // Held separately from the race so a late rejection (arriving after the + // timeout branch has already won) still has a handler attached and does + // not surface as an unhandledRejection once main() has moved on. + const lookup = dns.lookup(domain); + lookup.catch(() => {}); const outcome = await Promise.race([ - dns.lookup(domain).then(() => 'resolved' as const), + lookup.then(() => 'resolved' as const), timeout, ]); if (outcome === 'timeout') { From 3cc119f3281b34bb911e772bed5f2bf2e6e5b49b Mon Sep 17 00:00:00 2001 From: Logan Nguyen Date: Mon, 20 Jul 2026 15:45:44 -0700 Subject: [PATCH 4/6] refactor(search): drop eval capture workflow and make list refresh weekly Signed-off-by: Logan Nguyen --- .../workflows/credibility-list-refresh.yml | 4 +- .github/workflows/search-eval-capture.yml | 168 ------------------ docs/built-in-web-search.md | 112 ++++++------ docs/search-eval.md | 49 ++--- 4 files changed, 77 insertions(+), 256 deletions(-) delete mode 100644 .github/workflows/search-eval-capture.yml diff --git a/.github/workflows/credibility-list-refresh.yml b/.github/workflows/credibility-list-refresh.yml index e3aa8179..800b6b43 100644 --- a/.github/workflows/credibility-list-refresh.yml +++ b/.github/workflows/credibility-list-refresh.yml @@ -16,9 +16,9 @@ name: Credibility List Refresh on: schedule: - # Monthly, 06:17 UTC on the 1st. Off-the-hour minute to avoid the + # Weekly, 06:17 UTC on Mondays. Off-the-hour minute to avoid the # scheduled-workflow thundering herd at :00. - - cron: '17 6 1 * *' + - cron: '17 6 * * 1' workflow_dispatch: permissions: diff --git a/.github/workflows/search-eval-capture.yml b/.github/workflows/search-eval-capture.yml deleted file mode 100644 index 10c41d0b..00000000 --- a/.github/workflows/search-eval-capture.yml +++ /dev/null @@ -1,168 +0,0 @@ -name: Search Eval Capture - -# Manual (phase 1) eval-run workflow for the built-in search pipeline, per -# issue #309. Runs the hermetic eval-corpus unit tests unconditionally, and -# optionally the live answer-capture harness (tests/live_answer_capture.rs) -# against the real internet. -# -# Search engines heavily rate-limit GitHub-hosted runner IPs. Before spending -# time on an engine build, the live-capture job first sends one cheap request -# to each keyless engine the pipeline actually uses (DuckDuckGo's html -# endpoint, Mojeek) and checks for HTTP 200 with no captcha/block markers. If -# either engine is blocked, the job fails fast with a step-summary explanation -# instead of burning the rest of the timeout on a build that would only -# produce empty capture rows. See docs/search-eval.md for what each mode does. -# -# Phase 2 (a live-model judge step over two capture runs) is documented as -# future work in docs/search-eval.md and is not built here; this workflow only -# covers phase 1, the capture harness. - -on: - workflow_dispatch: - inputs: - mode: - description: 'Which checks to run' - type: choice - options: - - live - - hermetic - default: live - -permissions: - contents: read - -jobs: - hermetic-corpus-checks: - name: Hermetic corpus checks - runs-on: macos-15 - timeout-minutes: 30 - steps: - - name: Checkout repository - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 - with: - fetch-depth: 1 - - - name: Setup Bun - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0 - with: - bun-version: 1.3.11 - - - name: Install dependencies - run: bun install --frozen-lockfile - - - name: Cache llama.cpp sidecar - uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: src-tauri/binaries - key: llama-cpp-${{ runner.os }}-${{ hashFiles('scripts/ensure-llama-server.ts') }} - - - name: Build llama-server sidecar - run: bun run engine:ensure - - - name: Run eval-corpus unit tests (no network, no model) - working-directory: src-tauri - run: cargo test --test live_answer_quality_eval - - live-capture: - name: Live answer capture - needs: hermetic-corpus-checks - if: ${{ github.event.inputs.mode == 'live' }} - runs-on: macos-15 - timeout-minutes: 60 - steps: - - name: Checkout repository - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 - with: - fetch-depth: 1 - - - name: Probe search-engine reachability - id: probe - run: | - set -uo pipefail - UA="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" - CAPTCHA_MARKERS=(anomaly-modal challenge-form cf-challenge hcaptcha recaptcha captcha-wrap altcha-widget) - - # Sends one real query to `name`'s endpoint (the exact requests the - # pipeline's engine.rs builds), and fails if the response isn't a - # clean 200 with no bot-challenge markup. - check_engine() { - name="$1" - shift - body="/tmp/probe-${name}.html" - http_code=$(curl -s -o "$body" -w '%{http_code}' "$@") - if [ "$http_code" != "200" ]; then - echo "blocked (${name}): HTTP ${http_code}" - return 1 - fi - for marker in "${CAPTCHA_MARKERS[@]}"; do - if grep -qi "$marker" "$body"; then - echo "blocked (${name}): captcha marker '${marker}' in response body" - return 1 - fi - done - echo "reachable (${name}): HTTP 200, no captcha markers" - return 0 - } - - ddg_ok=0 - check_engine ddg \ - -A "$UA" \ - -H "Accept: text/html,application/xhtml+xml" \ - --data "q=rust+programming+language&kl=wt-wt&b=" \ - https://html.duckduckgo.com/html/ || ddg_ok=1 - - mojeek_ok=0 - check_engine mojeek \ - -A "$UA" \ - -H "Accept: text/html,application/xhtml+xml" \ - -G --data-urlencode "q=rust programming language" \ - https://www.mojeek.com/search || mojeek_ok=1 - - if [ "$ddg_ok" -ne 0 ] || [ "$mojeek_ok" -ne 0 ]; then - echo "::error::Search-engine reachability probe failed from this GitHub-hosted runner. Per issue #309's documented caveat, keyless search engines heavily rate-limit runner IPs; the live answer-capture harness stays a dev-machine tool. Failing early instead of spending the rest of the timeout on an engine build." - { - echo "## Live capture: search-engine probe failed" - echo "" - echo "DuckDuckGo and/or Mojeek returned a non-200 response or a captcha/block marker to this runner's IP." - echo "" - echo "This confirms the caveat in issue #309: GitHub-hosted runner IPs are rate-limited or blocked by the keyless search engines the pipeline uses. \`live_answer_capture.rs\` stays a dev-machine tool; run it locally instead:" - echo "" - echo '```sh' - echo "cargo test --test live_answer_capture -- --ignored --nocapture --test-threads=1" - echo '```' - echo "" - echo "\`hermetic-corpus-checks\` still ran and passed; only this job is affected." - } >> "$GITHUB_STEP_SUMMARY" - exit 1 - fi - - echo "Both engines reachable from this runner; continuing to engine build and live capture." >> "$GITHUB_STEP_SUMMARY" - - - name: Setup Bun - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0 - with: - bun-version: 1.3.11 - - - name: Install dependencies - run: bun install --frozen-lockfile - - - name: Cache llama.cpp sidecar - uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: src-tauri/binaries - key: llama-cpp-${{ runner.os }}-${{ hashFiles('scripts/ensure-llama-server.ts') }} - - - name: Build llama-server sidecar - run: bun run engine:ensure - - - name: Run live answer-capture harness - working-directory: src-tauri - run: cargo test --test live_answer_capture -- --ignored --nocapture --test-threads=1 - - - name: Upload captured answers - if: always() - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 - with: - name: eval-answers - path: src-tauri/target/eval/answers-*.jsonl - retention-days: 90 diff --git a/docs/built-in-web-search.md b/docs/built-in-web-search.md index cf8b58b3..d4557c34 100644 --- a/docs/built-in-web-search.md +++ b/docs/built-in-web-search.md @@ -135,8 +135,8 @@ These principles drive the design. Everything later is an application of them. | **Evidence over confidence** | “Today” and price questions drop stale archive URLs and numberless marketing pages so the model cannot sound sure on empty chrome. | | **Bounded work** | Fixed budgets: pages, tokens, repair rounds, at most one requery. Fixed stage order in the **orchestrator** (the module that runs stages in order and handles cancel/outcomes). | | **Honest when empty** | If search was needed but nothing usable returned, disclose that; do not fake “current” knowledge. | -| **Privacy of cache** | Conversation page cache and scrape caches live in process memory only; process exit wipes them. | -| **Reuse is gated** | Classifier `cached` only hints; re-rank stored **pages** for the new question and run a buffered writer check; escalate to a full search if pages cannot ground the answer. | +| **Privacy of cache** | Conversation page cache and scrape caches live in process memory only; process exit wipes them. | +| **Reuse is gated** | Classifier `cached` only hints; re-rank stored **pages** for the new question and run a buffered writer check; escalate to a full search if pages cannot ground the answer. | --- @@ -185,24 +185,24 @@ flowchart TD **Stages and helpers** (code under `src-tauri/src/websearch/`): -| Order | Stage | One-line role | -| ------ | ------------------------------- | ----------------------------------------------------------------------- | -| 1 | `prefilter` | Code rules: must search / must not / unsure | -| 2 | `prepass` | Classifier model: `no` / `cached` / `web` + queries (+ optional `lang`) | -| (side) | ForceWeb SERP race | On engine-shaped ForceWeb, raw-query SERP runs **with** the classifier | -| (side) | `clock` | Place-time for clock questions (not a search) | -| 3 | `lang` | Resolve language once from the user message; shape every channel | -| 4 | `cache` / `serp_cache` | Conversation page cache (reuse) + process SERP/page scrape cache | -| 5 | Verticals | Weather, news, Wikipedia, sports APIs | -| 6 | `judge` | Did the vertical actually answer? Else escalate | -| 7 | `engine` | Scrape keyless SERPs, fuse ranks (RRF + credibility list) | -| 8 | `fetch` | Download pages, extract readable text | -| 9 | `rank` + `recency` + `evidence` | Passages, freshness prior, price/freshness filters | -| 10 | `assemble` | Numbered source blocks under a token budget | +| Order | Stage | One-line role | +| ------ | ------------------------------- | ------------------------------------------------------------------------- | +| 1 | `prefilter` | Code rules: must search / must not / unsure | +| 2 | `prepass` | Classifier model: `no` / `cached` / `web` + queries (+ optional `lang`) | +| (side) | ForceWeb SERP race | On engine-shaped ForceWeb, raw-query SERP runs **with** the classifier | +| (side) | `clock` | Place-time for clock questions (not a search) | +| 3 | `lang` | Resolve language once from the user message; shape every channel | +| 4 | `cache` / `serp_cache` | Conversation page cache (reuse) + process SERP/page scrape cache | +| 5 | Verticals | Weather, news, Wikipedia, sports APIs | +| 6 | `judge` | Did the vertical actually answer? Else escalate | +| 7 | `engine` | Scrape keyless SERPs, fuse ranks (RRF + credibility list) | +| 8 | `fetch` | Download pages, extract readable text | +| 9 | `rank` + `recency` + `evidence` | Passages, freshness prior, price/freshness filters | +| 10 | `assemble` | Numbered source blocks under a token budget | | 11 | `writer` | Grounded answer stream; cache tier may buffer for `INSUFFICIENT_EVIDENCE` | -| 12 | `cite_check` | Mechanical citation support + optional repair | -| glue | `orchestrator` | Fixed order, cancellation, outcomes, timings | -| glue | `stage_timing` | Per-stage wall times → stderr + chat trace | +| 12 | `cite_check` | Mechanical citation support + optional repair | +| glue | `orchestrator` | Fixed order, cancellation, outcomes, timings | +| glue | `stage_timing` | Per-stage wall times → stderr + chat trace | Outbound HTTP goes through `src-tauri/src/net/`: SSRF-safe transport (every fetch re-checks that the target is a public internet address, not private/LAN). @@ -447,10 +447,10 @@ Code: `clock.rs`, `prefilter` helpers, `commands.rs`. **What.** Two different in-memory caches. Do not mix them up. -| Cache | Holds | Scope | Why | -| --- | --- | --- | --- | -| **Conversation page cache** (`cache`) | Full **fetched page texts** from recent successful **engine-tier** searches (up to **4** entries), not pre-assembled `[n]` blocks | Per conversation (epoch scope) + per-entry **TTL** (~10 min) | Follow-ups can re-answer from pages already downloaded without re-hitting the open web | -| **Web scrape cache** (`serp_cache`) | Per-engine SERP hit lists + extracted page bodies keyed by query | Process-wide; SERP ~5 min, pages ~15 min; **FIFO** max entries | Same scrape twice soon: save latency and rate-limit budget | +| Cache | Holds | Scope | Why | +| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | -------------------------------------------------------------------------------------- | +| **Conversation page cache** (`cache`) | Full **fetched page texts** from recent successful **engine-tier** searches (up to **4** entries), not pre-assembled `[n]` blocks | Per conversation (epoch scope) + per-entry **TTL** (~10 min) | Follow-ups can re-answer from pages already downloaded without re-hitting the open web | +| **Web scrape cache** (`serp_cache`) | Per-engine SERP hit lists + extracted page bodies keyed by query | Process-wide; SERP ~5 min, pages ~15 min; **FIFO** max entries | Same scrape twice soon: save latency and rate-limit budget | **Rule:** memory only. Process exit wipes both. Queries and page text stay off disk. @@ -794,39 +794,39 @@ How to run: [search-eval.md](./search-eval.md). ## Glossary -| Term | Plain meaning | -| -------------------- | ------------------------------------------------------------------ | -| Auto search | Setting that allows plain turns to open the web when needed | -| Builtin | Thuki’s bundled llama.cpp / `llama-server` provider | -| Pipeline | Fixed multi-step stages run in order | -| Keyless | No user-supplied API key for those sources | -| Classifier / prepass | Short model call that only routes search (JSON form) | -| Prefilter | Code-only search / must-not / unsure rules | -| ForceWeb / ForceNo | Prefilter: force web / force no web | -| Cached (classifier) | Hint to try conversation page reuse; still gated | -| Page cache reuse | Re-rank stored engine pages for a follow-up; escalate if thin | -| INSUFFICIENT_EVIDENCE | Buffered-writer sentinel: reuse failed grounding → fresh search | -| Vertical | Specialized API path (weather, news, wiki, sports) | -| SERP | **S**earch **E**ngine **R**esults **P**age (hit list + snippets) | -| RRF | **R**eciprocal **R**ank **F**usion: merge lists by `1/(k+rank)` | -| BM25 | Classic keyword relevance score (Okapi BM25) | -| DDG | DuckDuckGo HTML search | -| SSRF | **S**erver-**s**ide **r**equest **f**orgery risk; blocked by `net` | -| Nonce | One-time random fence token around untrusted page text | -| TTL | Time-to-live (cache expiry) | -| FIFO | First-in first-out eviction when a cache is full | -| Readability | Extract main article text from HTML | -| num_ctx | Model context window size (tokens) | -| IPC | Inter-process messages (Rust ↔ UI) | -| TTFT | Time to first streamed answer token | -| Punycode | ASCII encoding of international domain names | -| User-Agent | Client identity string on HTTP requests | -| Citation audit | Post-check that `[n]` claims match source text | -| Attribution | Licence/provider credit on a source (UI + metadata) | -| Language parity | Retrieval and answers follow the user’s language | -| ForceWeb race | Parallel raw SERP while the classifier rewrites | -| Evidence filters | Post-rank drops for stale archives / bad prices | -| Orchestrator | Module that runs stages, cancel, and outcomes | +| Term | Plain meaning | +| --------------------- | ------------------------------------------------------------------ | +| Auto search | Setting that allows plain turns to open the web when needed | +| Builtin | Thuki’s bundled llama.cpp / `llama-server` provider | +| Pipeline | Fixed multi-step stages run in order | +| Keyless | No user-supplied API key for those sources | +| Classifier / prepass | Short model call that only routes search (JSON form) | +| Prefilter | Code-only search / must-not / unsure rules | +| ForceWeb / ForceNo | Prefilter: force web / force no web | +| Cached (classifier) | Hint to try conversation page reuse; still gated | +| Page cache reuse | Re-rank stored engine pages for a follow-up; escalate if thin | +| INSUFFICIENT_EVIDENCE | Buffered-writer sentinel: reuse failed grounding → fresh search | +| Vertical | Specialized API path (weather, news, wiki, sports) | +| SERP | **S**earch **E**ngine **R**esults **P**age (hit list + snippets) | +| RRF | **R**eciprocal **R**ank **F**usion: merge lists by `1/(k+rank)` | +| BM25 | Classic keyword relevance score (Okapi BM25) | +| DDG | DuckDuckGo HTML search | +| SSRF | **S**erver-**s**ide **r**equest **f**orgery risk; blocked by `net` | +| Nonce | One-time random fence token around untrusted page text | +| TTL | Time-to-live (cache expiry) | +| FIFO | First-in first-out eviction when a cache is full | +| Readability | Extract main article text from HTML | +| num_ctx | Model context window size (tokens) | +| IPC | Inter-process messages (Rust ↔ UI) | +| TTFT | Time to first streamed answer token | +| Punycode | ASCII encoding of international domain names | +| User-Agent | Client identity string on HTTP requests | +| Citation audit | Post-check that `[n]` claims match source text | +| Attribution | Licence/provider credit on a source (UI + metadata) | +| Language parity | Retrieval and answers follow the user’s language | +| ForceWeb race | Parallel raw SERP while the classifier rewrites | +| Evidence filters | Post-rank drops for stale archives / bad prices | +| Orchestrator | Module that runs stages, cancel, and outcomes | --- diff --git a/docs/search-eval.md b/docs/search-eval.md index eda86dde..048c9e01 100644 --- a/docs/search-eval.md +++ b/docs/search-eval.md @@ -1,6 +1,6 @@ # Search Decision & Answer Evaluation -Dev-time tooling for measuring three different things about the built-in search pipeline: whether it decides to search at all (`live_classifier_eval.rs`), what it answers once it does (`live_answer_capture.rs`), and whether that answer is actually *correct* (`live_answer_quality_eval.rs`, this doc's newest addition). None of the three is a CI gate; all are `#[ignore]`d integration tests run by hand against a live `llama-server` and the live internet. +Dev-time tooling for measuring three different things about the built-in search pipeline: whether it decides to search at all (`live_classifier_eval.rs`), what it answers once it does (`live_answer_capture.rs`), and whether that answer is actually _correct_ (`live_answer_quality_eval.rs`, this doc's newest addition). None of the three is a CI gate; all are `#[ignore]`d integration tests run by hand against a live `llama-server` and the live internet. Product and pipeline context (not this eval doc): [built-in-web-search.md](./built-in-web-search.md), baked-in constants in [configurations.md](./configurations.md) (Built-in web search), user-facing `/search` and Auto search in [commands.md](./commands.md) and [privacy.md](./privacy.md). @@ -8,13 +8,13 @@ Product and pipeline context (not this eval doc): [built-in-web-search.md](./bui `src-tauri/src/websearch/search_decision_eval.jsonl` is the labelled should-search / should-not-search set: one JSON object per line, no wrapping array (JSONL). Fields: -| Field | Type | Meaning | -|---|---|---| -| `message` | string | The user's latest turn, verbatim. | -| `label` | `"search"` \| `"no"` | Whether this turn should trigger a web search. The measurement target of `live_classifier_eval.rs` and the prefilter soundness tests in `prefilter.rs`. | -| `category` | string | A free-form tag grouping related rows (`weather`, `sports`, `stable_fact`, `followup_current`, ...). Informational; not asserted on directly. | -| `route` | string, optional | The expected retrieval tier (`web`, `news`, `weather`, `sports`, `wiki`) for rows where it's unambiguous. Absent on context-dependent follow-up rows, where the tier depends on prior turns the row doesn't carry. | -| `volatility` | `"never"` \| `"slow"` \| `"fast"` \| `"false-premise"` | How fast the true answer changes, independent of `label`. See below. | +| Field | Type | Meaning | +| ------------ | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `message` | string | The user's latest turn, verbatim. | +| `label` | `"search"` \| `"no"` | Whether this turn should trigger a web search. The measurement target of `live_classifier_eval.rs` and the prefilter soundness tests in `prefilter.rs`. | +| `category` | string | A free-form tag grouping related rows (`weather`, `sports`, `stable_fact`, `followup_current`, ...). Informational; not asserted on directly. | +| `route` | string, optional | The expected retrieval tier (`web`, `news`, `weather`, `sports`, `wiki`) for rows where it's unambiguous. Absent on context-dependent follow-up rows, where the tier depends on prior turns the row doesn't carry. | +| `volatility` | `"never"` \| `"slow"` \| `"fast"` \| `"false-premise"` | How fast the true answer changes, independent of `label`. See below. | ### Volatility categories @@ -60,14 +60,14 @@ Output: `target/eval/answers-.jsonl` (one file per run, timestamped so "question": "weather in Tokyo", "volatility": "fast", "outcome_kind": "answer", - "sources": [{"url": "https://open-meteo.com/", "title": "..."}], + "sources": [{ "url": "https://open-meteo.com/", "title": "..." }], "writer_user_turn": "" } ``` `outcome_kind` is `"answer"`, `"unreachable"` (retrieval produced nothing citable), or `"nosearch"` (not expected given the scripted classifier, but handled rather than panicking). `sources` is url+title only, no vertical-tier label. `writer_user_turn` holds the actual final prompt turn the writer model would see for an `"answer"` outcome, or a fixed marker string otherwise. -This harness captures *what the pipeline answers*, not *whether it decided to search* — that's `live_classifier_eval.rs`'s job. +This harness captures _what the pipeline answers_, not _whether it decided to search_ — that's `live_classifier_eval.rs`'s job. ## Answer-quality eval (`live_answer_quality_eval.rs`) @@ -83,12 +83,12 @@ THUKI_EVAL_PORT= cargo test --test live_answer_quality_eval -- --ignored - Four sources, unified under one schema (`question`, `gold_answer`, `acceptable_answers`, `volatility`), loaded and combined by `load_full_corpus()`: -| Source | File | Rows | License | Gates? | -|---|---|---|---|---| -| SimpleQA-Verified | `tests/j5_corpus/simpleqa_verified.jsonl` | 50 | MIT | **Yes** | -| FreshQA | `tests/j5_corpus/freshqa.jsonl` | 30 | Apache-2.0 | No (tracked) | -| Seal-0 | `tests/j5_corpus/seal0.jsonl` | 15 | Apache-2.0 | No (tracked) | -| Decision corpus | `src/websearch/search_decision_eval.jsonl` (existing, reused) | 103 | Apache-2.0 (this repo) | N/A, not graded | +| Source | File | Rows | License | Gates? | +| ----------------- | ------------------------------------------------------------- | ---- | ---------------------- | --------------- | +| SimpleQA-Verified | `tests/j5_corpus/simpleqa_verified.jsonl` | 50 | MIT | **Yes** | +| FreshQA | `tests/j5_corpus/freshqa.jsonl` | 30 | Apache-2.0 | No (tracked) | +| Seal-0 | `tests/j5_corpus/seal0.jsonl` | 15 | Apache-2.0 | No (tracked) | +| Decision corpus | `src/websearch/search_decision_eval.jsonl` (existing, reused) | 103 | Apache-2.0 (this repo) | N/A, not graded | Total: 198 rows, 95 of them new to this harness. (The task that specified this harness estimated "~150 rows total"; the actual combined total once the existing 103-row decision corpus is included in full is 198. Recorded here rather than silently adjusted, since the estimate and the literal per-source row counts the same spec listed do not reconcile: the per-source counts are what was implemented.) @@ -118,25 +118,14 @@ Deterministic-first, never pairwise, never a rubric score: Before trusting the gate, hand-label a sample of 30-40 rows spanning all three graded sources and volatility buckets with your own CORRECT/INCORRECT/NOT_ATTEMPTED judgment against the same predicted answers the harness generated, then run the live judge (majority-of-3) over that identical sample and compute agreement: the fraction of rows where the judge's majority verdict matches your hand label. This is the only way to know whether the local judge model is reliable enough for `CONFIDENTLY_WRONG_GATE` to mean anything, and is a prerequisite for tightening it. It needs a live model and a human rater (Logan), so it is a documented manual procedure, not code in this repository. -## CI workflows +## Related automation -`.github/workflows/search-eval-capture.yml` is a `workflow_dispatch`-only workflow, triggered by hand from the Actions tab, with a `mode` choice input (`live`, the default, or `hermetic`). - -Two jobs: - -- **`hermetic-corpus-checks`** always runs, regardless of `mode`. It builds the sidecar (needed for the crate to compile at all) and runs `cargo test --test live_answer_quality_eval`, which is every non-ignored test in that file: the corpus-loading, grading-logic, and composition unit tests. No network, no live model; this is the closest thing this repo has to a CI gate on the eval corpus itself. -- **`live-capture`** runs only when `mode` is `live`, and only after `hermetic-corpus-checks` passes. Before building anything, it sends one real request to each keyless engine the pipeline uses (DuckDuckGo's `html` endpoint, Mojeek) and checks for a clean HTTP 200 with no captcha/block markers. GitHub-hosted runner IPs are heavily rate-limited by keyless search engines; if either engine blocks the probe, the job fails immediately with a `::error::` annotation and a step-summary explanation, without spending the rest of the timeout on an engine build that would only produce empty capture rows. If the probe passes, it builds the engine sidecar and runs `cargo test --test live_answer_capture -- --ignored --nocapture --test-threads=1`, then uploads `src-tauri/target/eval/answers-*.jsonl` as a 90-day artifact (uploaded even on a failed test step, so a partial capture is still inspectable). - -A probe failure is not itself a bug: it means the harness has to stay a dev-machine tool for now, run by hand as documented above, rather than something CI can execute unattended. If the probe consistently passes over time, that is the "proves viable" signal this doc's phase 2 note below is gated on. - -A live-model judge step over two capture runs (a baseline and a candidate) is documented as future work, gated on phase 1's reachability probe proving reliable in practice; it is not built in this workflow. See "Judging: pairwise superseded by absolute SimpleQA grading" below for why the judge design for that step, if it is ever automated, would not be the pairwise comparison this doc's history once sketched. - -The domain-credibility list has its own scheduled refresh workflow (`.github/workflows/credibility-list-refresh.yml`); see [built-in-web-search.md](./built-in-web-search.md) for that automation. +These harnesses are dev-machine tools, run by hand as documented above; no CI workflow executes them. The domain-credibility list has its own scheduled refresh workflow (`.github/workflows/credibility-list-refresh.yml`); see [built-in-web-search.md](./built-in-web-search.md) for that automation. ## Judging: pairwise superseded by absolute SimpleQA grading An earlier revision of this doc sketched pairwise, position-swapped LLM-as-judge scoring, following [Brave's published search-eval methodology](https://brave.com/blog/), as the intended next step once two capture runs existed to compare: an LLM judge shown both runs' answers in one order, then the swapped order, with a majority vote across both orderings deciding the winner or a tie. -**That plan is superseded by `live_answer_quality_eval.rs`'s absolute, SimpleQA-protocol grading (see above), and will not be built.** The reasoning: small local judge models (the only kind Thuki's keyless, no-server design can call) are known to be catastrophically position-biased in pairwise comparison. Shown the same two answers in swapped order, a weak judge frequently flips its preference to whichever answer appeared first or second, regardless of content. Position-swapping is a mitigation for that bias, not a cure, and a pairwise judge still produces no notion of *absolute* correctness, only *relative* preference between two specific runs. Grading each answer independently against a fixed gold reference (SimpleQA's CORRECT/INCORRECT/NOT_ATTEMPTED) sidesteps position bias entirely: there is only one item to grade, not two to compare, and it produces a metric (confidently-wrong rate) that means the same thing across every run, not just between the two runs being compared. +**That plan is superseded by `live_answer_quality_eval.rs`'s absolute, SimpleQA-protocol grading (see above), and will not be built.** The reasoning: small local judge models (the only kind Thuki's keyless, no-server design can call) are known to be catastrophically position-biased in pairwise comparison. Shown the same two answers in swapped order, a weak judge frequently flips its preference to whichever answer appeared first or second, regardless of content. Position-swapping is a mitigation for that bias, not a cure, and a pairwise judge still produces no notion of _absolute_ correctness, only _relative_ preference between two specific runs. Grading each answer independently against a fixed gold reference (SimpleQA's CORRECT/INCORRECT/NOT_ATTEMPTED) sidesteps position bias entirely: there is only one item to grade, not two to compare, and it produces a metric (confidently-wrong rate) that means the same thing across every run, not just between the two runs being compared. **This tooling is exempt from Thuki's keyless/no-server product constraint.** The app itself never calls out to a hosted search API or a hosted judge model, which is the whole point of the built-in engine and its keyless `websearch` retrieval pipeline. This doc's harnesses, including the judge in `live_answer_quality_eval.rs`, are dev-time-only measurement tooling, run by a developer's hand against their own llama-server, never shipped or called from the app. From 4fa58aaa4df8338f9e1f1468e86a6db6353eafd7 Mon Sep 17 00:00:00 2001 From: Logan Nguyen Date: Mon, 20 Jul 2026 15:47:25 -0700 Subject: [PATCH 5/6] docs(search): revert formatting-only churn in search docs Signed-off-by: Logan Nguyen --- docs/built-in-web-search.md | 112 ++++++++++++++++++------------------ docs/search-eval.md | 34 +++++------ 2 files changed, 73 insertions(+), 73 deletions(-) diff --git a/docs/built-in-web-search.md b/docs/built-in-web-search.md index d4557c34..cf8b58b3 100644 --- a/docs/built-in-web-search.md +++ b/docs/built-in-web-search.md @@ -135,8 +135,8 @@ These principles drive the design. Everything later is an application of them. | **Evidence over confidence** | “Today” and price questions drop stale archive URLs and numberless marketing pages so the model cannot sound sure on empty chrome. | | **Bounded work** | Fixed budgets: pages, tokens, repair rounds, at most one requery. Fixed stage order in the **orchestrator** (the module that runs stages in order and handles cancel/outcomes). | | **Honest when empty** | If search was needed but nothing usable returned, disclose that; do not fake “current” knowledge. | -| **Privacy of cache** | Conversation page cache and scrape caches live in process memory only; process exit wipes them. | -| **Reuse is gated** | Classifier `cached` only hints; re-rank stored **pages** for the new question and run a buffered writer check; escalate to a full search if pages cannot ground the answer. | +| **Privacy of cache** | Conversation page cache and scrape caches live in process memory only; process exit wipes them. | +| **Reuse is gated** | Classifier `cached` only hints; re-rank stored **pages** for the new question and run a buffered writer check; escalate to a full search if pages cannot ground the answer. | --- @@ -185,24 +185,24 @@ flowchart TD **Stages and helpers** (code under `src-tauri/src/websearch/`): -| Order | Stage | One-line role | -| ------ | ------------------------------- | ------------------------------------------------------------------------- | -| 1 | `prefilter` | Code rules: must search / must not / unsure | -| 2 | `prepass` | Classifier model: `no` / `cached` / `web` + queries (+ optional `lang`) | -| (side) | ForceWeb SERP race | On engine-shaped ForceWeb, raw-query SERP runs **with** the classifier | -| (side) | `clock` | Place-time for clock questions (not a search) | -| 3 | `lang` | Resolve language once from the user message; shape every channel | -| 4 | `cache` / `serp_cache` | Conversation page cache (reuse) + process SERP/page scrape cache | -| 5 | Verticals | Weather, news, Wikipedia, sports APIs | -| 6 | `judge` | Did the vertical actually answer? Else escalate | -| 7 | `engine` | Scrape keyless SERPs, fuse ranks (RRF + credibility list) | -| 8 | `fetch` | Download pages, extract readable text | -| 9 | `rank` + `recency` + `evidence` | Passages, freshness prior, price/freshness filters | -| 10 | `assemble` | Numbered source blocks under a token budget | +| Order | Stage | One-line role | +| ------ | ------------------------------- | ----------------------------------------------------------------------- | +| 1 | `prefilter` | Code rules: must search / must not / unsure | +| 2 | `prepass` | Classifier model: `no` / `cached` / `web` + queries (+ optional `lang`) | +| (side) | ForceWeb SERP race | On engine-shaped ForceWeb, raw-query SERP runs **with** the classifier | +| (side) | `clock` | Place-time for clock questions (not a search) | +| 3 | `lang` | Resolve language once from the user message; shape every channel | +| 4 | `cache` / `serp_cache` | Conversation page cache (reuse) + process SERP/page scrape cache | +| 5 | Verticals | Weather, news, Wikipedia, sports APIs | +| 6 | `judge` | Did the vertical actually answer? Else escalate | +| 7 | `engine` | Scrape keyless SERPs, fuse ranks (RRF + credibility list) | +| 8 | `fetch` | Download pages, extract readable text | +| 9 | `rank` + `recency` + `evidence` | Passages, freshness prior, price/freshness filters | +| 10 | `assemble` | Numbered source blocks under a token budget | | 11 | `writer` | Grounded answer stream; cache tier may buffer for `INSUFFICIENT_EVIDENCE` | -| 12 | `cite_check` | Mechanical citation support + optional repair | -| glue | `orchestrator` | Fixed order, cancellation, outcomes, timings | -| glue | `stage_timing` | Per-stage wall times → stderr + chat trace | +| 12 | `cite_check` | Mechanical citation support + optional repair | +| glue | `orchestrator` | Fixed order, cancellation, outcomes, timings | +| glue | `stage_timing` | Per-stage wall times → stderr + chat trace | Outbound HTTP goes through `src-tauri/src/net/`: SSRF-safe transport (every fetch re-checks that the target is a public internet address, not private/LAN). @@ -447,10 +447,10 @@ Code: `clock.rs`, `prefilter` helpers, `commands.rs`. **What.** Two different in-memory caches. Do not mix them up. -| Cache | Holds | Scope | Why | -| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | -------------------------------------------------------------------------------------- | -| **Conversation page cache** (`cache`) | Full **fetched page texts** from recent successful **engine-tier** searches (up to **4** entries), not pre-assembled `[n]` blocks | Per conversation (epoch scope) + per-entry **TTL** (~10 min) | Follow-ups can re-answer from pages already downloaded without re-hitting the open web | -| **Web scrape cache** (`serp_cache`) | Per-engine SERP hit lists + extracted page bodies keyed by query | Process-wide; SERP ~5 min, pages ~15 min; **FIFO** max entries | Same scrape twice soon: save latency and rate-limit budget | +| Cache | Holds | Scope | Why | +| --- | --- | --- | --- | +| **Conversation page cache** (`cache`) | Full **fetched page texts** from recent successful **engine-tier** searches (up to **4** entries), not pre-assembled `[n]` blocks | Per conversation (epoch scope) + per-entry **TTL** (~10 min) | Follow-ups can re-answer from pages already downloaded without re-hitting the open web | +| **Web scrape cache** (`serp_cache`) | Per-engine SERP hit lists + extracted page bodies keyed by query | Process-wide; SERP ~5 min, pages ~15 min; **FIFO** max entries | Same scrape twice soon: save latency and rate-limit budget | **Rule:** memory only. Process exit wipes both. Queries and page text stay off disk. @@ -794,39 +794,39 @@ How to run: [search-eval.md](./search-eval.md). ## Glossary -| Term | Plain meaning | -| --------------------- | ------------------------------------------------------------------ | -| Auto search | Setting that allows plain turns to open the web when needed | -| Builtin | Thuki’s bundled llama.cpp / `llama-server` provider | -| Pipeline | Fixed multi-step stages run in order | -| Keyless | No user-supplied API key for those sources | -| Classifier / prepass | Short model call that only routes search (JSON form) | -| Prefilter | Code-only search / must-not / unsure rules | -| ForceWeb / ForceNo | Prefilter: force web / force no web | -| Cached (classifier) | Hint to try conversation page reuse; still gated | -| Page cache reuse | Re-rank stored engine pages for a follow-up; escalate if thin | -| INSUFFICIENT_EVIDENCE | Buffered-writer sentinel: reuse failed grounding → fresh search | -| Vertical | Specialized API path (weather, news, wiki, sports) | -| SERP | **S**earch **E**ngine **R**esults **P**age (hit list + snippets) | -| RRF | **R**eciprocal **R**ank **F**usion: merge lists by `1/(k+rank)` | -| BM25 | Classic keyword relevance score (Okapi BM25) | -| DDG | DuckDuckGo HTML search | -| SSRF | **S**erver-**s**ide **r**equest **f**orgery risk; blocked by `net` | -| Nonce | One-time random fence token around untrusted page text | -| TTL | Time-to-live (cache expiry) | -| FIFO | First-in first-out eviction when a cache is full | -| Readability | Extract main article text from HTML | -| num_ctx | Model context window size (tokens) | -| IPC | Inter-process messages (Rust ↔ UI) | -| TTFT | Time to first streamed answer token | -| Punycode | ASCII encoding of international domain names | -| User-Agent | Client identity string on HTTP requests | -| Citation audit | Post-check that `[n]` claims match source text | -| Attribution | Licence/provider credit on a source (UI + metadata) | -| Language parity | Retrieval and answers follow the user’s language | -| ForceWeb race | Parallel raw SERP while the classifier rewrites | -| Evidence filters | Post-rank drops for stale archives / bad prices | -| Orchestrator | Module that runs stages, cancel, and outcomes | +| Term | Plain meaning | +| -------------------- | ------------------------------------------------------------------ | +| Auto search | Setting that allows plain turns to open the web when needed | +| Builtin | Thuki’s bundled llama.cpp / `llama-server` provider | +| Pipeline | Fixed multi-step stages run in order | +| Keyless | No user-supplied API key for those sources | +| Classifier / prepass | Short model call that only routes search (JSON form) | +| Prefilter | Code-only search / must-not / unsure rules | +| ForceWeb / ForceNo | Prefilter: force web / force no web | +| Cached (classifier) | Hint to try conversation page reuse; still gated | +| Page cache reuse | Re-rank stored engine pages for a follow-up; escalate if thin | +| INSUFFICIENT_EVIDENCE | Buffered-writer sentinel: reuse failed grounding → fresh search | +| Vertical | Specialized API path (weather, news, wiki, sports) | +| SERP | **S**earch **E**ngine **R**esults **P**age (hit list + snippets) | +| RRF | **R**eciprocal **R**ank **F**usion: merge lists by `1/(k+rank)` | +| BM25 | Classic keyword relevance score (Okapi BM25) | +| DDG | DuckDuckGo HTML search | +| SSRF | **S**erver-**s**ide **r**equest **f**orgery risk; blocked by `net` | +| Nonce | One-time random fence token around untrusted page text | +| TTL | Time-to-live (cache expiry) | +| FIFO | First-in first-out eviction when a cache is full | +| Readability | Extract main article text from HTML | +| num_ctx | Model context window size (tokens) | +| IPC | Inter-process messages (Rust ↔ UI) | +| TTFT | Time to first streamed answer token | +| Punycode | ASCII encoding of international domain names | +| User-Agent | Client identity string on HTTP requests | +| Citation audit | Post-check that `[n]` claims match source text | +| Attribution | Licence/provider credit on a source (UI + metadata) | +| Language parity | Retrieval and answers follow the user’s language | +| ForceWeb race | Parallel raw SERP while the classifier rewrites | +| Evidence filters | Post-rank drops for stale archives / bad prices | +| Orchestrator | Module that runs stages, cancel, and outcomes | --- diff --git a/docs/search-eval.md b/docs/search-eval.md index 048c9e01..16858bd9 100644 --- a/docs/search-eval.md +++ b/docs/search-eval.md @@ -1,6 +1,6 @@ # Search Decision & Answer Evaluation -Dev-time tooling for measuring three different things about the built-in search pipeline: whether it decides to search at all (`live_classifier_eval.rs`), what it answers once it does (`live_answer_capture.rs`), and whether that answer is actually _correct_ (`live_answer_quality_eval.rs`, this doc's newest addition). None of the three is a CI gate; all are `#[ignore]`d integration tests run by hand against a live `llama-server` and the live internet. +Dev-time tooling for measuring three different things about the built-in search pipeline: whether it decides to search at all (`live_classifier_eval.rs`), what it answers once it does (`live_answer_capture.rs`), and whether that answer is actually *correct* (`live_answer_quality_eval.rs`, this doc's newest addition). None of the three is a CI gate; all are `#[ignore]`d integration tests run by hand against a live `llama-server` and the live internet. Product and pipeline context (not this eval doc): [built-in-web-search.md](./built-in-web-search.md), baked-in constants in [configurations.md](./configurations.md) (Built-in web search), user-facing `/search` and Auto search in [commands.md](./commands.md) and [privacy.md](./privacy.md). @@ -8,13 +8,13 @@ Product and pipeline context (not this eval doc): [built-in-web-search.md](./bui `src-tauri/src/websearch/search_decision_eval.jsonl` is the labelled should-search / should-not-search set: one JSON object per line, no wrapping array (JSONL). Fields: -| Field | Type | Meaning | -| ------------ | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `message` | string | The user's latest turn, verbatim. | -| `label` | `"search"` \| `"no"` | Whether this turn should trigger a web search. The measurement target of `live_classifier_eval.rs` and the prefilter soundness tests in `prefilter.rs`. | -| `category` | string | A free-form tag grouping related rows (`weather`, `sports`, `stable_fact`, `followup_current`, ...). Informational; not asserted on directly. | -| `route` | string, optional | The expected retrieval tier (`web`, `news`, `weather`, `sports`, `wiki`) for rows where it's unambiguous. Absent on context-dependent follow-up rows, where the tier depends on prior turns the row doesn't carry. | -| `volatility` | `"never"` \| `"slow"` \| `"fast"` \| `"false-premise"` | How fast the true answer changes, independent of `label`. See below. | +| Field | Type | Meaning | +|---|---|---| +| `message` | string | The user's latest turn, verbatim. | +| `label` | `"search"` \| `"no"` | Whether this turn should trigger a web search. The measurement target of `live_classifier_eval.rs` and the prefilter soundness tests in `prefilter.rs`. | +| `category` | string | A free-form tag grouping related rows (`weather`, `sports`, `stable_fact`, `followup_current`, ...). Informational; not asserted on directly. | +| `route` | string, optional | The expected retrieval tier (`web`, `news`, `weather`, `sports`, `wiki`) for rows where it's unambiguous. Absent on context-dependent follow-up rows, where the tier depends on prior turns the row doesn't carry. | +| `volatility` | `"never"` \| `"slow"` \| `"fast"` \| `"false-premise"` | How fast the true answer changes, independent of `label`. See below. | ### Volatility categories @@ -60,14 +60,14 @@ Output: `target/eval/answers-.jsonl` (one file per run, timestamped so "question": "weather in Tokyo", "volatility": "fast", "outcome_kind": "answer", - "sources": [{ "url": "https://open-meteo.com/", "title": "..." }], + "sources": [{"url": "https://open-meteo.com/", "title": "..."}], "writer_user_turn": "" } ``` `outcome_kind` is `"answer"`, `"unreachable"` (retrieval produced nothing citable), or `"nosearch"` (not expected given the scripted classifier, but handled rather than panicking). `sources` is url+title only, no vertical-tier label. `writer_user_turn` holds the actual final prompt turn the writer model would see for an `"answer"` outcome, or a fixed marker string otherwise. -This harness captures _what the pipeline answers_, not _whether it decided to search_ — that's `live_classifier_eval.rs`'s job. +This harness captures *what the pipeline answers*, not *whether it decided to search* — that's `live_classifier_eval.rs`'s job. ## Answer-quality eval (`live_answer_quality_eval.rs`) @@ -83,12 +83,12 @@ THUKI_EVAL_PORT= cargo test --test live_answer_quality_eval -- --ignored - Four sources, unified under one schema (`question`, `gold_answer`, `acceptable_answers`, `volatility`), loaded and combined by `load_full_corpus()`: -| Source | File | Rows | License | Gates? | -| ----------------- | ------------------------------------------------------------- | ---- | ---------------------- | --------------- | -| SimpleQA-Verified | `tests/j5_corpus/simpleqa_verified.jsonl` | 50 | MIT | **Yes** | -| FreshQA | `tests/j5_corpus/freshqa.jsonl` | 30 | Apache-2.0 | No (tracked) | -| Seal-0 | `tests/j5_corpus/seal0.jsonl` | 15 | Apache-2.0 | No (tracked) | -| Decision corpus | `src/websearch/search_decision_eval.jsonl` (existing, reused) | 103 | Apache-2.0 (this repo) | N/A, not graded | +| Source | File | Rows | License | Gates? | +|---|---|---|---|---| +| SimpleQA-Verified | `tests/j5_corpus/simpleqa_verified.jsonl` | 50 | MIT | **Yes** | +| FreshQA | `tests/j5_corpus/freshqa.jsonl` | 30 | Apache-2.0 | No (tracked) | +| Seal-0 | `tests/j5_corpus/seal0.jsonl` | 15 | Apache-2.0 | No (tracked) | +| Decision corpus | `src/websearch/search_decision_eval.jsonl` (existing, reused) | 103 | Apache-2.0 (this repo) | N/A, not graded | Total: 198 rows, 95 of them new to this harness. (The task that specified this harness estimated "~150 rows total"; the actual combined total once the existing 103-row decision corpus is included in full is 198. Recorded here rather than silently adjusted, since the estimate and the literal per-source row counts the same spec listed do not reconcile: the per-source counts are what was implemented.) @@ -126,6 +126,6 @@ These harnesses are dev-machine tools, run by hand as documented above; no CI wo An earlier revision of this doc sketched pairwise, position-swapped LLM-as-judge scoring, following [Brave's published search-eval methodology](https://brave.com/blog/), as the intended next step once two capture runs existed to compare: an LLM judge shown both runs' answers in one order, then the swapped order, with a majority vote across both orderings deciding the winner or a tie. -**That plan is superseded by `live_answer_quality_eval.rs`'s absolute, SimpleQA-protocol grading (see above), and will not be built.** The reasoning: small local judge models (the only kind Thuki's keyless, no-server design can call) are known to be catastrophically position-biased in pairwise comparison. Shown the same two answers in swapped order, a weak judge frequently flips its preference to whichever answer appeared first or second, regardless of content. Position-swapping is a mitigation for that bias, not a cure, and a pairwise judge still produces no notion of _absolute_ correctness, only _relative_ preference between two specific runs. Grading each answer independently against a fixed gold reference (SimpleQA's CORRECT/INCORRECT/NOT_ATTEMPTED) sidesteps position bias entirely: there is only one item to grade, not two to compare, and it produces a metric (confidently-wrong rate) that means the same thing across every run, not just between the two runs being compared. +**That plan is superseded by `live_answer_quality_eval.rs`'s absolute, SimpleQA-protocol grading (see above), and will not be built.** The reasoning: small local judge models (the only kind Thuki's keyless, no-server design can call) are known to be catastrophically position-biased in pairwise comparison. Shown the same two answers in swapped order, a weak judge frequently flips its preference to whichever answer appeared first or second, regardless of content. Position-swapping is a mitigation for that bias, not a cure, and a pairwise judge still produces no notion of *absolute* correctness, only *relative* preference between two specific runs. Grading each answer independently against a fixed gold reference (SimpleQA's CORRECT/INCORRECT/NOT_ATTEMPTED) sidesteps position bias entirely: there is only one item to grade, not two to compare, and it produces a metric (confidently-wrong rate) that means the same thing across every run, not just between the two runs being compared. **This tooling is exempt from Thuki's keyless/no-server product constraint.** The app itself never calls out to a hosted search API or a hosted judge model, which is the whole point of the built-in engine and its keyless `websearch` retrieval pipeline. This doc's harnesses, including the judge in `live_answer_quality_eval.rs`, are dev-time-only measurement tooling, run by a developer's hand against their own llama-server, never shipped or called from the app. From 93fbd4f40b472f15ff430f0b78adaf5e074425c7 Mon Sep 17 00:00:00 2001 From: Logan Nguyen Date: Mon, 20 Jul 2026 16:36:32 -0700 Subject: [PATCH 6/6] chore(search): prune three dead hoax domains from drop list Signed-off-by: Logan Nguyen --- src-tauri/src/websearch/credibility_domains.txt | 3 --- 1 file changed, 3 deletions(-) diff --git a/src-tauri/src/websearch/credibility_domains.txt b/src-tauri/src/websearch/credibility_domains.txt index 7cff449f..f20d589e 100644 --- a/src-tauri/src/websearch/credibility_domains.txt +++ b/src-tauri/src/websearch/credibility_domains.txt @@ -16,10 +16,8 @@ # cluster: editorial judgment -- impostor/hoax "news" domains, individually verified active 2026-07-10 via en.wikipedia.org/wiki/List_of_fake_news_websites (cross-cited PolitiFact/NewsGuard/Snopes/Lead Stories); not a bulk copy of that page actionnews3.com -breaking13news.com dailybuzzlive.com news4ktla.com -news4local.com now8news.com channel22news.com channel24news.com @@ -36,7 +34,6 @@ current-affairs.org # cluster: editorial judgment -- 2026-07-14 gold-price smoke (giá vàng hôm nay): SEO scrapes / # free-blog hosts that assembled stale "SJC 80 triệu" / 2020 USD quotes ahead of real VN hubs blogdanica.com -globaleasyforex.com ponselharian.com kulturellen.net