perf(common-crawl): parse WARC files with fastwarc instead of warcio

[chethanuk]

Description

Problem

CommonCrawlWarcIterator carried a TODO pointing at #778: use fastwarc
instead of warcio. warcio parses every record's headers in Python and hands them to the iterator,
which then throws away roughly two thirds of them — the warcinfo, request and metadata
records — before doing any work. fastwarc filters by record type in C++, so only response records
ever cross into Python.

Root cause

warc_iterator.py built ArchiveIterator(file_pointer, arc2warc=True) and filtered with
if rec.rec_type == "response". Two more warcio iterators sat in download.py
(_fetch_from_s3 and _read_warc_record), so swapping only the first would have left warcio
imported and the dependency un-removable.

The change

• All three call sites now use fastwarc.warc.ArchiveIterator with
  record_types=WarcRecordType.response, which replaces the explicit rec_type check.
• auto_decode="all" keeps the yielded content equal to warcio's content_stream(), which
  decoded the HTTP body's Content-Encoding. Without it a gzip-encoded body comes back as raw
  gzip bytes.
• strict_mode=False at all three call sites. Without it, a record with an unparseable WARC
  header ends the file: the good records after it are never yielded. warcio kept going, so the
  flag is what preserves that.
• In warc_iterator.py, next() moves into its own try. A stream the parser cannot open at all
  leaves nothing to resynchronize to, so that case is logged once and ends the file. Failures
  while reading an individual record still log and continue, unchanged.
• pyproject.toml: warcio out, fastwarc<1 in. fastwarc was already installed transitively by
  resiliparse, so no new wheel is pulled in. The cap is load-bearing twice over: the two
  packages pin each other exactly so it also holds resiliparse below 1.x, and fastwarc 1.x drops
  the strict_mode resync this iterator now relies on.
• The test_mixed_record_types... fixture now emits Content-Type: application/http;msgtype=response
  on its response record, as real Common Crawl WARCs do. fastwarc only strips HTTP headers from
  records that declare it.

Test evidence

Behaviour locks were added and committed first, green against warcio, so they pin the contract the
swap has to preserve: the yielded content must be the decoded HTTP body (parametrized over plain
and Content-Encoding: gzip), and a corrupt record must not cost the rest of the file
(parametrized over corrupt-header-first, corrupt-header-mid-file, and an unopenable stream).

Suite (tests/stages/text/download/common_crawl/): 73 passed before, 78 passed after.

Against a real WARC — the first 50 MiB of
CC-MAIN-2024-10/segments/1707947473347.0/warc/CC-MAIN-20240220211055-20240221001055-00000.warc.gz,
3,208 response records — both parsers produce byte-identical output. Measured on the versions this
PR pins, fastwarc 0.15.2 / resiliparse 0.15.2 / warcio 1.8.1, with every record's
(url, warc_id, source_id, content) folded into one digest:

warcio    records=3208 sha256=1abd9f3070e1c238793e814357a62cc3c540646ab0235fee2047c0e29beb2ecc
fastwarc  records=3208 sha256=1abd9f3070e1c238793e814357a62cc3c540646ab0235fee2047c0e29beb2ecc

Timing over the same file, best of three, full content read: warcio 1.14 s, fastwarc 0.84 s. The
measuring machine was not idle and the run-to-run spread is a meaningful fraction of that gap, so
read it as "roughly 20% faster here", not as a benchmark. It lands in the range of the ~25%
reported on the issue rather than the ~4x fastwarc advertises.

ruff check and ruff format --check clean at the version .pre-commit-config.yaml pins;
uv lock regenerated (Removed warcio v1.7.5, fastwarc stays at the already-locked 0.15.2).

Limitations

• ARC support is dropped with arc2warc=True; fastwarc has no equivalent. Common Crawl URL
  generation only ever emits warc.paths.gz, so no supported input path is affected.
• Chunked transfer encoding is not decoded. warcio's ChunkedDataReader handled it; fastwarc
  0.15.2 does not, even with auto_decode="all" (1.x does). Its frequency in Common Crawl was
  not measured; the 3,208-record sample shows no divergence.
• A record skipped past by the resync is dropped without a log line. strict_mode=False buys back
  the records after a corrupt WARC header, but the corrupt one itself disappears silently: fastwarc
  0.15.2's ArchiveIterator exposes no skip counter and no callback, emits no warning, and does not
  surface the record even with record_types=any_type, so there is nothing the loop can detect and
  log. warcio was noisier here — it raised ArchiveLoadFailed, which the old loop logged before
  losing the rest of the file. The trade is fewer lost records, but no signal on the one that is
  lost; a short record count is the only symptom. Documented in the iterator's comment.
• The HTTP preamble is only stripped from records that declare Content-Type: application/http.
  warcio inferred HTTP-ness from the record type; fastwarc requires the header. Common Crawl emits
  it, so its records are unaffected, but a hand-built or non-conforming WARC response record now
  yields HTTP/1.1 200 OK\r\n... as part of content. WarcRecord.parse_http() does not override
  this (it is a no-op when is_http is false), so matching warcio would mean reimplementing its
  sniffing in Python; out of scope here.

closes #778

Usage

No public API change: CommonCrawlWarcIterator keeps its signature and the download stages call
it exactly as before. The swap is internal to WARC parsing.

from nemo_curator.stages.text.download.common_crawl.warc_iterator import CommonCrawlWarcIterator

# unchanged - same call, same (url, warc_id, source_id, content) records
for record in CommonCrawlWarcIterator().iterate(path):
    ...

Checklist

• I am familiar with the Contributing Guide.
• New or Existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[greptile-apps]

Greptile Summary

This PR replaces warcio with the already-transitive fastwarc parser throughout Common Crawl WARC processing.

• Filters response records within fastwarc and enables automatic body decoding.
• Configures malformed-header resynchronization and adjusts iterator error handling.
• Replaces the optional dependency, regenerates the lockfile, and adds decoded-body and corruption-recovery tests.
• The primary remaining concern is that the two range-fetch readers do not assert exact parser output in their tests.

Confidence Score: 4/5

The PR appears safe to merge, with a non-blocking request to strengthen exact-output coverage for the two range-fetch readers.

The parser migration is covered for the main iterator and preserves dependency availability across supported extras; the only accepted concern is that two independently changed parsing paths could regress body extraction without their current tests detecting it.

Files Needing Attention: nemo_curator/stages/text/download/common_crawl/download.py; tests/stages/text/download/common_crawl/test_warc_reader.py

Important Files Changed

Filename	Overview
nemo_curator/stages/text/download/common_crawl/download.py	Migrates both range-based WARC readers to fastwarc; exact returned-body behavior remains weakly tested.
nemo_curator/stages/text/download/common_crawl/warc_iterator.py	Migrates whole-file iteration to filtered, automatically decoded fastwarc records with explicit terminal-stream handling.
pyproject.toml	Replaces warcio with a fastwarc pre-1.x constraint in the same text_cpu dependency extra.
tests/stages/text/download/common_crawl/test_warc_iterator.py	Adds decoded-body and malformed-record recovery coverage for the main iterator.
uv.lock	Regenerates optional dependency metadata while retaining the existing fastwarc 0.15.2 resolution and removing warcio.

Sequence Diagram

sequenceDiagram
  participant Pipeline
  participant Storage as fsspec / HTTPS / S3
  participant Decoder as gzip
  participant Parser as fastwarc
  Pipeline->>Storage: Open WARC file or fetch byte range
  Storage-->>Decoder: Compressed or raw WARC bytes
  Decoder-->>Parser: WARC stream
  Parser->>Parser: Filter response records
  Parser->>Parser: Strip HTTP headers and decode body
  Parser-->>Pipeline: URL, WARC ID, source ID, body bytes

Loading

_{Reviews (1): Last reviewed commit: "perf(common-crawl): parse WARC files wit..." | Re-trigger Greptile}

[greptile-apps]

Range Reader Output Untested

The parser migration changes body extraction and decoding in both range-fetch readers, but their tests only check that the result is not None. They do not verify that the returned bytes are the exact HTTP body. Please add exact-body assertions, including a Content-Encoding case, so the tests detect regressions that return WARC or HTTP headers or compressed bytes.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!