fix: resolve Walmart/Target/Costco/BestBuy extraction failures via remote browser + improve product image extraction - #37
Closed
tonyfruzza wants to merge 10 commits into
Closed
Conversation
1. scraper.ts: Update URL match regex to accept a.co short links in addition to full amazon.com URLs 2. products.ts: Use voting-based scraper in confirmed-price flow for richer price candidate data 3. Add k8s manifests for deploy via GH Actions
Merges fix/aco-url-support into main
The Sec-* and Cache-Control headers in the axios request config trigger Amazon's bot detection, returning a 5KB blocking page instead of the full product page. This breaks ALL Amazon scraping, not just a.co links. Removed headers: Cache-Control, Pragma, Sec-Ch-Ua, Sec-Ch-Ua-Mobile, Sec-Ch-Ua-Platform, Sec-Fetch-Dest, Sec-Fetch-Mode, Sec-Fetch-Site, Sec-Fetch-User, Upgrade-Insecure-Requests
fix: remove bot-triggering headers from Amazon HTTP requests
…rt, Best Buy, Costco) The scraper was going straight to Puppeteer for JS-heavy sites, but Chromium fails to launch in the container (crashpad issue). These sites actually serve product data via JSON-LD in the initial HTML, so plain HTTP requests work fine. Changes: - Try axios HTTP fetch FIRST for ALL sites (including JS-heavy) - Fall back to Puppeteer only on 403 or empty response - Add size check: if HTML < 50KB and site is JS-heavy, try browser
Replaces the broken local Chromium launch (crashpad handler crash in k8s) with connection to the shared browser-node Chromium via CDP. Changes: - Add getRemoteBrowserWSEndpoint() — fetches WebSocket URL from browser-node's /json/version endpoint, remaps localhost → k8s service - scrapeWithBrowser tries remote browser first, falls back to local launch - Use browser.disconnect() for remote, browser.close() for local - REMOTE_BROWSER_URL env var overrides the default k8s service URL
axios auto-parses JSON responses, so response.data is already an object. Calling JSON.parse on it throws a silent error, preventing the remote browser connection from working.
- Expand genericImageSelectors with 10+ additional CSS patterns - Fix extractGenericImage to check data-src/data-lazy-src on all img tags - Fix Target scraper: add JSON-LD, og:image, and hero-image fallbacks - Fix Best Buy scraper: add image gallery, JSON-LD, and og:image fallbacks - Add extractImageFromBrowser() helper that evaluates JS in the real browser - Use browser JS extraction as last resort in scrapeProductWithVoting
Author
|
Closing this — the remote browser / CDP architecture is a significant enough departure from the current self-contained Puppeteer model that it's better kept as a fork-specific customization for now rather than pushed upstream. Leaving the issue #9 comment in place as a reference for anyone who wants to implement a similar approach. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes extraction failures for JS-heavy retailers (Walmart, Target, Costco, Best Buy) by connecting to an external browser via the Chrome DevTools Protocol (CDP) instead of requiring Puppeteer to launch a local Chromium process inside the Docker container.
It also improves product image extraction for these same sites using browser-executed JavaScript.
Also includes: the OpenAI base URL feature from #36 — if that merges first, this diff will reduce to only the scraper/image changes.
Problem
Issue #9 describes the core symptom:
Even when Puppeteer can launch locally, these sites block headless detection and often return anti-bot pages.
Solution
1. Remote browser via CDP (
scraper.ts)Adds a
getRemoteBrowserWSEndpoint()helper that queriesREMOTE_BROWSER_URL/json/version(Chrome DevTools Protocol). If a remote browser is reachable,scrapeWithBrowser()andextractImageFromBrowser()connect to it viapuppeteer.connect()instead ofpuppeteer.launch().Falls back to local Puppeteer launch if the remote browser is unreachable — zero breaking change for existing setups.
Environment variable:
REMOTE_BROWSER_URL(default:http://browser-node.openclaw.svc.cluster.local:9222)Any Chrome/Chromium with remote debugging enabled (e.g.,
chromium --remote-debugging-port=9222) works.2. Resilience improvements to
scrapeWithBrowser()ERR_HTTP2_PROTOCOL_ERRORwith a fresh pagedomcontentloadedinstead ofnetworkidle2— more reliable on SPAs3. Product image extraction via browser JS (
extractImageFromBrowser())New function that injects a multi-strategy JS payload into the page to extract product images: Open Graph tags → JSON-LD →
itemprop→ common CSS selectors → first non-placeholder<img>. Used as fallback when cheerio-based extraction returns no image.Related Issues
Testing
Tested against: