fix(validators): fail fast on 0-record manifests + validate NLP text content

[shujaatTracebloc]

Summary

Adversarial end-to-end MLM ingestion (dev cluster, ingestor:0.3.12) fed the validator chain malformed-but-realistic datasets. The tokenizer checks held up, but several input-hygiene gaps let bad data slip past pre-ingestion validation and fail late with misleading messages. This hardens them at the validator level (reproduced via temp CSVs + temp files driving the validators directly), so each is caught before the table is created — no orphan empty table, no misleading "rows already in the database".

Fixes

MUST FIX — "0 ingestable records" (two cases)

Finding: an empty/header-only CSV passes all current validators, creates the table, ingests 0 rows, then fails late at the edge-label step with "…the dataset was NOT registered (its rows are already in the database)" — but there are 0 rows. Same misleading late error when a CSV references files that don't exist and all are missing (every record skipped at transfer → 0 transferred → orphan empty table). PR #250 only caught a totally empty file; a header-only CSV slipped through.

Fix:

• New IngestableRecordsValidator (wired into the NLP factories via _nlp_content_validators) rejects:
  • header-only / empty CSV → "No data rows found in CSV …"
  • file-bearing manifest with every referenced file missing → "No referenced data files could be found; nothing to ingest" (cross-checks the CSV filename column against SRC_PATH/<subdir>/, early-exiting on the first file that resolves)
  • It runs in validate_data before create_table, so a rejected ingest leaves no orphan table (the bug: validator-rejected ingest leaves an orphaned table; no rollback blocks the next ingest #260 guard, at the zero-records gate).
• base.py: _rows_state_clause(inserted_records) rephrases the three registration-failure messages so they can never claim "its rows are already in the database" when 0 rows were ingested ("no rows were ingested, so nothing was left in the database"). The "NOT registered" wording is kept.

Repro (tests): test_header_only_csv_is_rejected, test_empty_zero_byte_csv_is_rejected, test_all_referenced_files_missing_is_rejected, test_mlm_header_only_csv_fails_before_table_creation + …all_files_missing… (assert create_table is never called), test_zero_inserted_registration_failure_message_is_truthful.

SHOULD FIX — NLP text-content validator (binary / empty docs)

Finding: the File Type Validator only checks the extension. A .txt holding non-UTF-8/binary bytes is ingested silently (5/5 "successfully processed" with garbage); an empty .txt is ingested with no warning. The user only finds out at training time.

Fix: new TextContentValidator, gated on text_classification / token_classification / masked_language_modeling. Samples the referenced text files (deterministic strided sample + per-file byte cap) and:

• rejects binary / non-UTF-8 content (NUL byte, or bytes that fail an incremental UTF-8 decode — incremental so a multibyte char split at the byte cap isn't a false positive)
• warns on empty / whitespace-only docs

Repro (tests): test_binary_content_is_rejected, test_non_utf8_bytes_are_rejected, test_empty_file_is_warned_not_rejected, test_whitespace_only_file_is_warned, test_multibyte_char_split_at_byte_cap_is_not_a_false_positive, test_sampling_bounds_files_checked.

NICE TO HAVE — within-CSV duplicate detection

Finding: duplicate rows in the CSV (same filename twice) ingest as separate records; the existing Duplicate Validator only checks against the existing table, not within the incoming CSV.

Fix: DuplicateValidator now also warns (no hard-fail — repeats may be intentional) when a filename appears more than once in the incoming CSV. Bounded (reads only the filename column); no-op for non-CSV / tabular inputs without a filename column.

Repro (tests): test_within_csv_duplicate_filenames_warn, …case_insensitive_column, …no_filename_column_is_noop, …none_input_is_noop.

Implementation notes

• Follows the existing validator framework: new validators wired via map_validators (per-category factories in modalities/validators.py), gated to NLP categories only (no impact on image/tabular).
• Extends the fix(csv): empty CSV must fail fast with clear input-error message #250 empty-CSV guard rather than duplicating it; reuses text_profile._sample and file_transfer._has_extension.
• Shared BaseValidator._match_column centralises the case-insensitive header lookup (also fixed an existing unused typing.List import in duplicate_validator.py).
• Diff is surgical — only the 11 touched files; no repo-wide reformatting.

Tests

• pytest tests/ -q --cov=tracebloc_ingestor --cov-fail-under=95 → 1183 passed, 1 xfailed, total coverage 96.7% (gate 95%).
• Only ADDED tests; no existing tests removed or rewritten.

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes the pre-ingestion validation path and registration error text for NLP categories; behavior is well-covered by new tests but affects a critical ingest gate before table creation.

Overview
Hardens preflight validation for NLP ingests so bad manifests fail before create_table, with clearer errors instead of late registration failures that wrongly claim rows are already in the database.

Zero-record fail-fast: Adds IngestableRecordsValidator for text/token classification and MLM. It rejects header-only or empty CSVs and manifests where every referenced file is missing under SRC_PATH (using the same _safe_join rules as transfer). Wired via _nlp_content_validators in modalities/validators.py (texts/ vs sequences/ per category).

NLP text content: Adds TextContentValidator on the same categories—samples staged files, fails on binary/NUL/non-UTF-8 bytes, warns on empty/whitespace-only docs, with bounded sampling and safe path handling.

Registration messaging: base.py introduces _rows_state_clause(inserted_records) so edge-label, global-meta, and prepare failures describe DB state accurately (e.g. “no rows were ingested…” when count is 0).

Duplicate CSV filenames: DuplicateValidator warns (does not fail) when the same filename appears multiple times in the incoming CSV; adds BaseValidator._match_column for case-insensitive headers.

^{Reviewed by Cursor Bugbot for commit 041ed13. Bugbot is set up for automated code reviews on this repo. Configure here.}

[LukasWodka]

👋 Heads-up — Code review queue is at 32 / 30

Above the WIP limit. The team convention is to review existing PRs before opening new work.

Open PRs currently in Code review (oldest first):

• .github#57 — fix(fr-gate): pass items at or beyond the required stage · author: @aptracebloc · no reviewer assigned
• .github#60 — feat(security): public-repo PII gate (block customer names + secrets in PRs) · author: @LukasWodka · no reviewer assigned
• .github#61 — fix(wip-limit): avoid sort|head SIGPIPE failing the WIP check when queue >10 · author: @shujaatTracebloc · reviewer: @saadqbal
• averaging-service#117 — refactor(averaging): per-framework weight-handling seam (WS-B B1) · author: @divyasinghds · no reviewer assigned
• averaging-service#118 — feat(averaging): trainable-only keyed-dict weight format (WS-C image_validator falsely rejects images when target_size matches exactly #104) · author: @divyasinghds · no reviewer assigned
• averaging-service#119 — fix(averaging): exact federated GaussianNB merge — usable + correct (WS-B B2, docs: fix staging-recipe link branch ref (main → develop) #112) · author: @divyasinghds · no reviewer assigned
• backend#815 — chore(deps): bump cryptography from 47.0.0 to 48.0.1 · author: @dependabot · no reviewer assigned
• backend#829 — feat(#817): failure surfaces a reason end-to-end + stuck-run reaper (WS6 I3/I4/I5/I8) · author: @saadqbal · no reviewer assigned
• backend#833 — fix(deploy): graceful shutdown + Dockerfile hygiene; drop dead EC2 script · author: @LukasWodka · no reviewer assigned
• backend#840 — feat(rfc-0001): namespace-slug rule + validation command (#837) · author: @LukasWodka · no reviewer assigned

Pull from review before opening new work. (This is a nudge from the kanban WIP check, not a block.)

[shujaatTracebloc]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

1 issue from previous review remains unresolved.

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 1c8f80d. Configure here.}

[shujaatTracebloc]

bugbot run

[shujaatTracebloc]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 32b199e. Configure here.}