feat(scraper): harden fetch fallbacks, llms.txt parsing, and version resolution

[ahmedhosnypro]

Summary

This PR hardens the scraping pipeline, llms.txt parsing, and version resolution logic. It improves resilience against anti-bot blocks, refines fallback strategies when resolving library versions, and prevents scraping failures on non-critical llms.txt seed errors.

Scraper fetchers & anti-bot resilience

• AutoDetectFetcher: falls back to the browser fetcher on 403/429 anti-bot blocks (not just challenge pages), and honors scrapeMode: "playwright" by going straight to the browser fetcher
• BrowserFetcher / HtmlPlaywrightMiddleware: realistic user agent and viewport, hardened Chromium launch flags (--disable-blink-features=AutomationControlled, new headless mode), and system Chromium path discovery (PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH, common distro locations)

llms.txt handling

• llmsTxtParser: accepts headerless llms.txt files (uses the first non-empty line as the name) and unwraps content delivered inside a <pre> tag when a server renders plain text via HTML
• BaseScraperStrategy: a 404 on an llms.txt-seeded URL no longer aborts the whole scrape (complementary to the fix that landed upstream in fix(scraper): don't abort scrape when only an llms.txt-seeded url 404s #479; adds coverage for initialQueue-seeded items)

GitHub strategy

• Scope check keeps discovered links within the same owner/repo on github.com while still honoring include/exclude patterns

Version resolution

• findBestVersion now supports named/non-semver versions (latest, canary, ...) and falls back to the latest available version for invalid target formats instead of failing

Embeddings

• EmbeddingFactory: passes the OpenAI API key and optional OPENAI_API_BASE explicitly via the supported apiKey/configuration fields

Changes

Scraper and Fetcher Hardening

• AutoDetectFetcher: Fallback to BrowserFetcher on 403 or 429 status codes, or when scrapeMode is set to "playwright". 19
• BrowserFetcher: Added default user agent, viewport size, and fallback paths to locate local Chromium executables. 13, 14, 15
• HtmlPlaywrightMiddleware: Configured default user agent and viewport options for browser contexts. 20
• BaseScraperStrategy: Prevented job failures on depth-0 items when they are seeded from llms.txt. 22, 23, 24
• GitHubScraperStrategy: Overrode shouldProcessUrl to restrict processing to the target repository's owner and name. 12
• WebScraperStrategy: Added subpath candidates for llms.txt and bypassed archive/canonical checks for llms.txt items. 8, 9, 10

Parsing and Versioning Improvements

• llmsTxtParser: Added support for unwrapping HTML <pre> tags, additional bullet characters, and missing H1 headings. 16, 17
• DocumentManagementService: Hardened findBestVersion to fall back to the latest semver version on invalid target formats. 1, 2, 3
• EmbeddingFactory: Explicitly passed apiKey in the configuration object to OpenAIEmbeddings. 7

Verification

Automated Tests

Run the updated test suites to verify the new fallback and parsing behaviors:

npm run test src/store/DocumentManagementService.test.ts
npm run test src/scraper/utils/llmsTxtParser.test.ts
npm run test src/scraper/strategies/BaseScraperStrategy.test.ts

Test plan

• npm run lint — clean (Biome)
• npm run typecheck — clean (tsc --noEmit)
• npm run build — clean
• npm test — all unit/integration suites pass; tests updated where the patches intentionally change behavior (llms.txt headerless parsing, findBestVersion fallback)
• Added coverage for the llms.txt 404 path via initialQueue items

Closes #185

[Copilot]

🟡 Changes recommended

findBestVersion can select the wrong fallback version and can return { bestMatch: null, hasUnversioned: false } without throwing, which can cause incorrect downstream behavior.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR strengthens the scraping pipeline’s resilience (fetcher fallbacks, Playwright realism, llms.txt robustness, GitHub scoping) and expands documentation version resolution behavior, while updating/adding tests to cover the new paths.

Changes:

• Hardened web scraping against anti-bot responses and improved Playwright fingerprinting/context defaults (UA/viewport/Chromium launch/executable discovery).
• Expanded llms.txt support (headerless files, <pre>-wrapped text) and ensured llms.txt-seeded 404s don’t abort scrapes (including initialQueue coverage).
• Broadened version resolution to support named/non-semver versions and new fallback behavior; updated embedding configuration to pass OpenAI credentials/base URL via supported fields.

File summaries

File	Description
src/store/embeddings/EmbeddingFactory.ts	Passes OpenAI apiKey/baseURL via apiKey/configuration for embeddings.
src/store/DocumentManagementService.ts	Updates findBestVersion to support named versions and fallback behavior.
src/store/DocumentManagementService.test.ts	Adjusts tests to reflect new version-resolution fallbacks.
src/scraper/utils/llmsTxtParser.ts	Adds headerless parsing and unwraps <pre>-wrapped llms.txt content; expands bullet parsing.
src/scraper/utils/llmsTxtParser.test.ts	Adds/updates tests for headerless parsing and stricter “invalid content” coverage.
src/scraper/strategies/WebScraperStrategy.ts	Propagates scrapeMode to fetch options; expands llms.txt candidate probing; avoids root-only behaviors for fromLlmsTxt.
src/scraper/strategies/WebScraperStrategy.test.ts	Updates fetch expectations to include scrapeMode.
src/scraper/strategies/GitHubScraperStrategy.ts	Tightens scoping so discovered links stay within the same owner/repo on github.com.
src/scraper/strategies/BaseScraperStrategy.ts	Treats llms.txt-seeded items as non-root for fail-fast and failure-rate counting purposes.
src/scraper/strategies/BaseScraperStrategy.test.ts	Adds coverage ensuring initialQueue llms.txt-seeded 404s don’t abort the scrape.
src/scraper/middleware/HtmlPlaywrightMiddleware.ts	Sets a realistic user agent + viewport on browser contexts.
src/scraper/fetcher/types.ts	Extends FetchOptions with scrapeMode.
src/scraper/fetcher/BrowserFetcher.ts	Adds realistic UA/viewport; hardens Chromium launch args and discovers system Chromium paths.
src/scraper/fetcher/AutoDetectFetcher.ts	Honors scrapeMode=playwright and falls back to browser fetcher on more anti-bot indicators.
.gitignore	Ignores .playwright-mcp/.

Review details

• Files reviewed: 14/15 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.