Add MAGIC scraper

[MrKoga]

What this does

Adds a MAGIC scraper — the MAGICapp platform — to the datasets scraping pipeline, same shape as the NICE scraper: discovery and extraction are separate, and everything outputs a normalized ScrapedDocument.

MAGICapp (https://app.magicapp.org) is a publishing platform for GRADE clinical practice guidelines, so one source covers many institutions — 85 publishing organisations, 50 with English guidelines, including the WHO, the European Stroke Organisation and Cancer Council Australia. About 233 English guidelines today.

Meditron lists MAGIC as a source; its scraper drives the single-page app with Selenium and is marked UNTESTED in the source. MAGICapp has a public unauthenticated JSON API, so this replaces that approach rather than porting it.

How it works

• GET https://api.magicapp.org/api/v1/guidelines returns the whole catalogue in one response; each entry's jsonPath points at the full guideline as structured JSON.
• Sections nest and are rendered as markdown headings; recommendation objects are rendered beneath them with the publisher's own label kept where one exists.
• English is the default (languages=("en",)), as a default rather than a policy — happy to change it, drop it, or expose a flag.

A few behaviors worth knowing, one line each:

• strength: "INFO" objects are editorial callout boxes, not recommendations — emitted as plain content, never labelled.
• Recommendations the publisher marks POSSIBLY_OUTDATED are skipped (2 in the catalogue).
• Root-level recommendations (an older export format, 11 guidelines) are read as well as section-level ones.
• http:// catalogue URLs are upgraded to HTTPS (the file host 403s plain HTTP).
• Embedded base64 images are dropped — they were half the emitted characters.
• Repeated front and back matter (Methods, Glossary, References…) is skipped, with a per-section rescue: any subtree holding a recommendation is kept whatever its heading.
• metadata carries both guideline_id and published_id; version-pinned viewer URLs need the latter.

Shared-helper change included

The first commit adds an opt-in drop_images to html_to_markdown — default False, so NICE output is byte-for-byte unchanged; MAGIC passes True because MAGICapp inlines figures as base64 data URIs (53.9% of emitted characters across a 63-guideline sample). The same commit hardens fragment re-serialization to keep text sitting before a fragment's first tag. Happy to split that commit into its own PR if you prefer shared-code changes reviewed separately.

CLI

uv run amfv-scrape --source magic --documents 1
uv run amfv-scrape --source magic --url https://app.magicapp.org/#/guideline/nyxpZL
uv run amfv-scrape --source magic --documents 3 -f markdown -o ./magic-out/

--documents is an upper bound on a catalogue run: unreadable or placeholder guidelines are logged and skipped.

Files

• datasets/amfv_datasets/scraping/magic.py — scraper module
• datasets/amfv_datasets/scraping/cli.py — --source magic dispatch
• datasets/amfv_datasets/scraping/__init__.py — public names
• datasets/test/test_scraping_magic.py — offline tests, MockTransport, no fixture files
• datasets/amfv_datasets/scraping/html.py + datasets/test/test_scraping_html.py — the shared-helper commit above

Test plan

• uv run pytest datasets/test — 66 passed, no regressions
• uv run ruff check datasets/ && uv run ruff format --check datasets/
• Live smoke test — --documents 3, non-empty markdown, no embedded image data
• reviewer: uv run amfv-scrape --source magic --documents 3 -f markdown -o /tmp/magic-out/

🤖 Generated with Claude Code

[CLAassistant]

All committers have signed the CLA.