Conversation
The backend no longer accepts per-row GlobalMeta POST calls or the
separate prepare_dataset / generate_edge_labels_meta steps. Instead,
the ingestor counts labels locally after committing all rows to MySQL
and sends one summary call as its final step, which creates the
UserDataSet on the backend directly.
- `client.py`: remove `send_batch`, `send_generate_edge_label_meta`,
`send_global_meta_meta`, `prepare_dataset`, `create_dataset`; add
`send_ingest_summary` → POST /global_meta/summary/<table_name>/
with `{ingestor_id, labels, dataset_title, data_format, data_intent,
category, schema, samples}`
- `database.py`: add `get_label_counts(table_name, ingestor_id)` and
`get_samples(table_name, ingestor_id, limit)` — query helpers used
by the post-ingest summary step
- `base.py`: remove per-batch `send_batch` call from `_process_batch`;
after all records are committed query DB for accurate label counts,
fetch preview samples, and call `send_ingest_summary`; fall back to
warning and skip summary if no rows were inserted
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat: add embeddings (self-supervised contrastive) modality Enable the NLP embeddings / contrastive use case end-to-end on the ingestor side, mirroring how seq2seq is wired. The on-disk contract is one .txt per sample under texts/, each a tab-separated anchor<TAB>positive pair OR anchor<TAB>positive<TAB>negative triplet (no label column). Wiring (mirrors seq2seq — closest sibling): - utils/constants.py: TaskCategory.EMBEDDINGS. - schema/ingest.v1.json: enum value, texts requirement, and the self-supervised no-label guard (#213). - modalities/registry.py: ModalitySpec (file-bearing, self-supervised, NLP, texts/) — is_nlp gates the #805 data-derived text profile (the ingestor side of the federated tokenizer-alignment check). - modalities/transfer.py + cli/conventions.py: raw-text staging from texts/. What's distinct from seq2seq: the on-disk shape is STRUCTURED, so beyond the shared TextContentValidator (UTF-8/binary hygiene) the embeddings factory adds a centralized ContrastivePairsValidator that rejects any .txt that is not exactly 2 or 3 non-empty tab fields (no plain prose, no empty fields, one record per file). Template fails loudly via the shared run_ingestion helper (no exit-0-on-failure); shared validation lives in the centralized validators (#317). Adds templates/embeddings (script + README + sample pairs/triplets), examples/yaml/embeddings.yaml, and full test coverage (new + updated enumerating tests, e2e params). Suite green; new validator at 100% coverage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: resolve embeddings sidecar paths via _safe_join (Bugbot) ContrastivePairsValidator joined the manifest filename onto SRC_PATH/texts with plain os.path.join, so an absolute / `..` manifest value could be validated and read from outside texts/ even though text_transfer rejects it (#239) — preflight could disagree with copy behavior. Resolve with _safe_join and skip on ValueError, exactly as TextContentValidator does. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Cuts v0.5.4 to ship the embeddings (self-supervised contrastive) modality (#327), which merged after v0.5.3. release-image.yml builds 0.5.4 and repoints the floating 0.5 tag, the first image to contain embeddings. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: add sentence_pair_classification modality Enable the NLP sentence-pair classification use case end-to-end on the ingestor side (Phase-4 cross-repo wiring for engine PR #301). The on-disk contract is one .txt per sample under texts/, each a tab-separated text_a<TAB>text_b sentence pair, plus a class label in the labels CSV. sentence_pair is SUPERVISED text classification — the label travels in the CSV exactly like text_classification (is_classification=True, is_self_supervised=False, staged from texts/) — but its .txt has a STRUCTURED shape (the text_classification layout with a tab separating the pair). Wiring (mirrors text_classification; structural check mirrors embeddings): - utils/constants.py: TaskCategory.SENTENCE_PAIR_CLASSIFICATION. - schema/ingest.v1.json: enum value, texts requirement + description, and the "requires label" rule (supervised — NOT the self-supervised no-label guard). - modalities/registry.py: ModalitySpec (file-bearing, classification, NLP, texts/) — is_nlp gates the #805 data-derived text profile (the ingestor side of the federated tokenizer-alignment check). - modalities/validators.py + transfer.py + cli/conventions.py: same validator set + raw-text staging from texts/ as text_classification. What's distinct from text_classification: the on-disk shape is STRUCTURED, so beyond the shared TextContentValidator (UTF-8/binary hygiene) it adds a structural check that rejects any .txt that isn't exactly 2 non-empty tab fields (no plain prose, no empty side, one record per file). Rather than clone ContrastivePairsValidator, extract the shared scaffold into a new TabSeparatedRecordValidator base (centralized validators, #317); ContrastivePairsValidator (2-or-3 fields) subclasses it with byte-identical messages (its tests pass unchanged) and SentencePairValidator (exactly-2) is a thin subclass. Template fails loudly via run_ingestion (no exit-0-on-failure). Adds templates/sentence_pair_classification (script + README + sample pairs), examples/yaml/sentence_pair_classification.yaml, and full test coverage (new + updated enumerating tests, e2e params). Unit suite green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(release): bump version to 0.5.5 Ships the sentence_pair_classification modality; v0.5.5 will be the first image to contain it. Bumping in this PR so the release tag can be cut straight from develop after merge (no separate bump PR needed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
feat: replace per-row GlobalMeta flow with single ingest summary call
Ships the global-meta client fix (#325): renames the injestor_id -> ingestor_id param/field and replaces the per-row GlobalMeta flow with a single ingest-summary call, matching the deployed backend #900. v0.5.5 still sends the old misspelled injestor_id and is rejected by the backend ("ingestor_id: This param is required"), breaking all ingestion on dev. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…mmary flow The embeddings (#327) and sentence_pair (#329) suites branched before the #325 global-meta refactor, which removed the per-batch send_batch publish in favour of one send_ingest_summary call after commit. Their `*_api_send_failure_counts_every_record` tests still asserted the removed per-record `api_send_failed` accounting (send_batch is never called now), so they failed `assert 0 == 2`. Rewrite both to the post-refactor contract, mirroring the passing sibling test_ingest_summary_failure_raises_out_of_ingest: a failing send_ingest_summary raises out of ingest() so the run still exits non-zero. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
chore(release): bump version to 0.5.6
…332) (#335) The ingest-summary client raised on any status >= 400, but backend #900 returns 409 CONFLICT + {dataset_id, dataset_key} on a duplicate (owner, ingestor_id) — a success signal for a transport-level POST retry after a dropped 201, not an error. That made an otherwise-successful ingest crash non-zero and a manual re-run create a duplicate dataset. Special-case 409 before the >= 400 check: parse and return the existing dataset body. Add a regression test asserting 409 returns the dataset and does not raise. Also clarify the post-commit rollback() comment in BaseIngestor.ingest (it's a no-op on already-committed rows). Co-authored-by: Syed Saqlain <syedsaqlain@MacBook-Pro.local> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
… (#334) - delete examples/custom_ingestor.py and examples/image_ingestor_Divya.py: both were built around the removed processors/ package (and the old DataCategory name) and crashed on import; custom per-record processors have no equivalent in the current programmatic API - rewrite examples/image_ingestor.py against the current API (CSVIngestor + run_ingestion, components constructed inside main() so import is side-effect-free, matching the templates/ convention) - examples/blob_ingestor.py: drop the undefined BlobDataProcessor reference (NameError at runtime) left over from the same removed machinery - examples/yaml/custom_processor.yaml: loud NOT YET EXECUTED header for spec.processors / spec.validators, mirroring the runtime warnings in cli/run.py; the aspirational mechanism is now labelled as planned v1.1 - add tests/test_examples_import.py importing every examples/*.py so a stale example fails CI instead of the first customer who copies it Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: shujaat <shujaat@tracebloc.io>
) Rows commit per batch during the ingest loop, so any failure after the first batch — mid-loop, or a genuine 4xx/5xx from send_ingest_summary — left durable rows whose dataset never registered. Training can't reach them (queries are scoped to REGISTERED ingestor_ids), but they inflate the heartbeat's availability report and occupy disk with no remote delete path (live wound: #336's qa_ic_edge304_train). Now the failure path deletes exactly this run's rows (WHERE ingestor_id = the per-process uuid) before re-raising: - Database.delete_by_ingestor_id: backtick-quoted, bindparam'd DELETE under _execute_with_retry; returns rowcount. - dataset_registered flag flips only after send_ingest_summary returns, so a late failure (e.g. summary rendering) can never delete a REGISTERED dataset's rows — mutation-tested (dropping the flag condition fails the new test). - session.rollback() hardened (a dead connection must not mask the original error or skip the cleanup); cleanup failure logs CRITICAL and preserves the original exception. - Residual two-generals window (backend registered, client raised on an unparseable 2xx after retries) documented in the except block; a re-run re-ingests from source. Staged files stay: keyed by filename, idempotently overwritten on re-run. Crash windows (SIGKILL) remain sweepable by ingestor_id — the log-first sweep is the client-runtime follow-up per backend#818. Decisions + refreshed design: backend#818 (2026-07-07 comment). Closes #227. Tests: +5 unit (delete-on-failure, never-on-success, never-after- registration [mutation pin], cleanup-failure preserves original error, no-insert-no-delete) + 1 e2e against real MySQL (removes only the failed run's rows; idempotent). Full suite 1341 passed, 1 xfailed; e2e_database 6/6 on live MySQL locally. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
) Landed DARK: default stays 'uuid'; opt-in via ingest.yaml data_id: {strategy: content_hash}. Flip-to-default after one release of soak (decision: Lukas, 2026-07-07; privacy sign-off on backend#818). With uuid ids, a retried k8s Job re-inserts every row under fresh ids — the 3x-duplication bug. content_hash derives data_id = sha256(per-table-salt + canonical JSON of the cleaned record + source filename), so a retry reproduces its ids exactly and the data_id UNIQUE upsert RE-CLAIMS the prior attempt's rows (ownership moves to the new ingestor_id, whose scoped label counts then cover the full dataset, and the drained old dataset auto-retires via the heartbeat reaper). Proven end-to-end against real MySQL: same content re-ingested → row count constant, ownership flipped, counts correct. Privacy: the salt is 32 random bytes, created atomically (INSERT IGNORE) in a cluster-MySQL meta table (tracebloc_ingest_meta, name now reserved against dataset use) and never leaves the cluster; per-table salting makes ids unlinkable across tables/clusters. Only 10 opaque {data_id,label} samples ever egress. Salt fetch is deferred to ingest() post-validation (mirrors #260's deferred create_table), so a rejected run leaves no salt row. The source filename participates in the hash — for file-bearing categories the schema-filtered record is just {label, data_intent}, so without it all same-label images would collide and the upsert would collapse the dataset (caught in self-review, pinned by test). Documented semantic: genuinely identical source rows collapse into one stored row (content-level dedup; noted in the schema description). Adversarial review pre-push caught a HIGH: CSVIngestor/JSONIngestor have explicit param lists and dropped the new kwarg — a TypeError on every YAML-driven run, invisible to the unit suite because nothing constructed through _build_ingestor. Fixed + pinned by a 4-way construction matrix test; the full-engine e2e (run.main over all 15 templates, real MySQL) now runs green (36 passed). Note for tracebloc/cli: the vendored ingest.v1.json gains an enum value — re-vendor with the pending category sync when this promotes. Refs #225; design + decisions: backend#818 (2026-07-07 comment). Tests: +17 unit (determinism across instances/salts, filename+label participation, precedence, dark-default, construction matrix, schema acceptance, conventions) + 2 e2e on real MySQL (retry re-claim, per-table salt isolation). Suite: 1354 passed, 1 xfailed; e2e 36/36. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
7 tasks
Validator, cast, and DB error strings embedded customer cell values —
and those strings land in the install log and can egress via the
failure-report log tail: a hole in the data-stays-on-prem guarantee.
New tracebloc_ingestor/utils/redaction.py sets the policy (cell VALUES
never; column/file names, dtypes, counts, row indices OK; classification
labels exempt where label diagnostics is the validator's purpose — they
egress by design in the summary's label counts) and provides:
- row_refs() — WHERE the offenders are (what a customer actually
needs to fix their file)
- mask_shape() — digits→#, letters→x, separators kept (for date-
format errors where structure IS the diagnosis)
- safe_db_error() — exception class(es) + MySQL errno + hint; never the
driver message (MySQL embeds values: "Duplicate
entry 'X'", "Incorrect value: 'X' for column")
Sanitized (16 sites across 11 files):
- DataValidator: numeric ×2, all three BOOLEAN branches (one dumped an
UNBOUNDED set of distinct offending values), non-finite sample
- csv_ingestor: float-overflow, date-cast (also severs the exception
chain — pandas' message embeds the value and rode back in via
__cause__ in logged tracebacks), INT branch (was unwrapped
errors='raise' — reachable un-gated on image-family categories and
category=None), blanket wrapper (non-ValueError → type only, from None)
- json_ingestor: 11 value!r dtype messages → mask_shape; whole-record
{record!r} dump → position + type name
- database.py: hide_parameters=True on both engines (SQLAlchemy
otherwise appends every statement parameter — whole rows — to error
strings), all log/store sites → safe_db_error; batch_writer same
- time_to_event ×2 (+ dropped raw-sample metadata keys),
numeric_columns, time_format (shape-masked), coercion int64-overflow
- record_processor: removed the hot-loop full-record INFO dump; missing-
unique-id warning and missing-columns error no longer print the record
- xml_validator ×2 / keypoint_visibility / bio_label → mask_shape
Deliberately exempt: duplicate_validator's filename listing (filenames
are dataset structure, stored by design) and LabelDiversityValidator's
label values (documented in redaction.py).
Tests: tests/test_error_redaction.py — 14 plant-a-secret tests (a
distinctive fake-PII value planted at every sanitized site, asserted
absent from every message; the severed date-cast chain pinned via
__cause__/__suppress_context__; safe_db_error class/errno/no-message
pins) + 7 existing tests updated from value-assertions to row-ref
assertions. Suite 1350 passed, 1 xfailed; e2e 34/34 on real MySQL.
Adversarial leak-hunt pre-push found 3 HIGH / 2 MEDIUM residual paths
beyond the original scope (JSON per-value errors, SQLAlchemy parameter
dumps, the unbounded BOOLEAN set) — all folded above.
Refs #226 (Phase 2 of the reduced plan on backend#818); sibling PRs
tracebloc/client-runtime#160 (Phase 1), #337, #339.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fix: compensating delete — a failed run no longer orphans its rows (#227)
…ash-data-id # Conflicts: # e2e/test_database_e2e.py
feat: salted content-hash data_id — a Job retry re-claims its rows (#225, landed dark)
…tizer # Conflicts: # tracebloc_ingestor/database.py
fix: errors and logs never carry raw cell content (#226 Phase 2)
1. tests/test_validators_mapping.py — sentence_pair_classification was missing from ALL_CATEGORIES and _CLASSIFICATION (14 entries, not 15), so test_common_validator_frame_composed_for_every_category never checked the shared validator frame for sentence-pair. The registry flags it is_classification=True; both lists now carry it. (Bugbot, low.) 2. examples/blob_ingestor.py — the example documents a BLOB/LONGBLOB schema-driven tabular workflow but constructed the ingestor with TaskCategory.TEXT_CLASSIFICATION + DataFormat.TEXT, which runs text sidecar validators against a workflow that has no sidecar files. Now TABULAR_CLASSIFICATION + DataFormat.TABULAR with label_column=document_type (stripped from the table schema per BaseIngestor semantics) and an honestly-named sample path. (Bugbot, medium.) Suite: 1350 passed, 1 xfailed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fix: resolve Bugbot findings on the #330 promotion surface
LukasWodka
previously approved these changes
Jul 7, 2026
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit e477942. Configure here.
…tually work Bugbot on #330 flagged that examples/blob_ingestor.py points at a sample CSV that doesn't exist (examples/data/ wasn't in the repo at all). Shipping the sample exposed the deeper bug: the example could NEVER have run — BLOB/LONGBLOB columns raise StatementError(TypeError) on every row, because SQLAlchemy wants bytes at bind time and CSV/JSON cells arrive as str. The 'documented BLOB workflow' has been broken end to end, invisible because nothing executed it. - examples/data/blob_documents_sample.csv: 6 documents, 3 label classes (passes LabelDiversityValidator), base64-ish payloads, JSON metadata. - database.insert_batch: str values bound to BLOB/LONGBLOB/LargeBinary columns are utf-8-encoded once, covering both ingestion paths. - Guardrail test: every data path an example references must exist in the repo (this class of dangling-example can't recur). - e2e (real MySQL): BLOB roundtrip from str cells → bytes back out. Proven end to end: the example executed against real MySQL + wire-level registration — 6 rows, 3 labels, 1 summary call, LONGBLOB payloads stored. Suite: 1373 passed, 1 xfailed; e2e 9/9 (database file). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fix: ship the blob example's sample data — and make BLOB ingestion actually work
LukasWodka
approved these changes
Jul 7, 2026
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release promotion:
develop→masterOpened ahead of the staging release week of Mon 2026-07-06 to start Bugbot review early and surface issues before the cutover. Head is
develop, so PRs that merge in before the release will update this PR's head and re-trigger Bugbot automatically.Delta (2 commit(s) ahead of
master)Initial snapshot — will change as more lands on
develop.Note
High Risk
Consolidates dataset registration into one backend API and changes failure semantics (raises, compensating deletes, content-hash upserts), which are critical for data integrity and cluster re-runs; breadth of test updates reduces but does not eliminate integration risk with the backend.
Overview
Release promotion bundling modality work, a backend registration redesign, and several reliability/privacy fixes ahead of the staging cutover week.
New NLP modalities:
sentence_pair_classification(supervised tab-separatedtext_a\ttext_bintexts/+ CSV labels) andembeddings(self-supervised contrastive pairs/triplets). Each ships templates, YAML examples, registry wiring, structural validators (SentencePairValidator,ContrastivePairsValidator), and expanded e2e/characterization coverage.Backend contract: Per-batch
send_batchand the multi-stepsend_global_meta_meta/prepare_dataset/create_datasetflow are replaced by a single post-commitsend_ingest_summary(labels, schema, samples,meta_dataincluding NLP text profile). Failures raise instead of returning false; HTTP 409 is treated as idempotent success when the dataset already exists. Characterization tests now assert the summary payload instead of the old trio of calls.Ingest reliability: Opt-in
content_hashdata_idwith per-table salt so Job retries re-claim rows via upsert instead of duplicating (#225). On registration failure after inserts,delete_by_ingestor_idcompensates unless registration already succeeded (#227). BLOB/LONGBLOB cells accept string CSV values encoded to bytes at insert.Privacy (#226): Validator, cast, and DB failure paths report row references (and date shape masks) instead of raw cell values; DB errors surface exception class/errno only.
Docs/examples: Readme lists the new categories; blob/image examples align with tabular BLOB and framework sidecars; obsolete custom-processor example removed;
custom_processor.yamldocuments v1.1 (processors not executed yet). CI import-checks allexamples/*.pyand referenced data files.Reviewed by Cursor Bugbot for commit 67fbd38. Bugbot is set up for automated code reviews on this repo. Configure here.