Multilingual Grievance Model

What is included

mg_code/
├── src/mgbench/
│   ├── population.py     # freeze eligible sentences and post-grouped splits
│   ├── lexicon.py        # paper-equation lexical anchor
│   ├── sampling.py       # non-circular U/P/N benchmark sampling
│   ├── annotations.py    # panel design, adjudication, aggregation, IAA
│   ├── predictions.py    # validate/import external outputs, no API client
│   ├── inputs.py         # target-only/full-post model inputs
│   ├── encoder.py        # matched Hugging Face encoder training/prediction
│   ├── experiments.py    # complete five-seed matched experiment matrix
│   ├── evaluation.py     # AP/AUROC/Spearman/ECE and fixed construct sets
│   ├── bootstrap.py      # paired post-clustered system comparisons
│   ├── audit.py          # reproduce the completed-pool selection audit
│   └── release.py        # code-only publication gate
├── configs/              # paper-aligned example configuration
├── schemas/              # offline interchange schemas
├── examples/             # invented, non-research smoke-test material
├── tests/                # unit and integration tests
└── docs/PAPER_CODE_MAP.md

Raw social-media posts, human annotations containing redistributable text, model weights, and API credentials are deliberately not copied into this folder.

The package contains no API calls.

Install

From this directory:

python -m pip install -e ".[lexicon,segmentation]"

For contextual encoder experiments:

python -m pip install -e ".[encoder]"

Near-duplicate clustering additionally needs .[near-dedup]. Encoder loading is explicitly local-only. Before a run, stage each approved model revision in a local directory or the Hugging Face cache; the code never contacts the model hub. Pin the immutable revision in the experiment config before a confirmatory run.

End-to-end benchmark construction

1. Freeze the eligible population

Input is JSONL with post_id, text, language, and optionally created_at.

mg-benchmark prepare-population \
  --posts /authorized/path/posts.jsonl \
  --out runs/population/items.jsonl \
  --manifest runs/population/manifest.json \
  --corpus-version meta-content-library-YYYY-MM-DD \
  --near-dedup-threshold 1.0

1.0 performs exact deduplication. A value below 1.0 activates MinHash LSH and requires the near-dedup extra. The manifest records the source digest, time range, normalizer, segmenter, duplicate method, split ratios, seed, and counts.

2. Draw the test strata

The sampler draws U without consulting the anchor, scores the remaining frame, and then draws P and N. Machine scores go to a separate hidden file so annotators cannot see them.

mg-benchmark sample-benchmark \
  --items runs/population/items.jsonl \
  --lexicon en=../data/dictionary_7plus.csv \
  --lexicon nl=../data/nl_dict.csv \
  --lexicon de=../data/de_dict.csv \
  --lexicon it=../data/it_dict.csv \
  --languages en nl de it \
  --n-u 200 --n-p 200 --n-n 200 \
  --out runs/benchmark/items.jsonl \
  --hidden-scores runs/benchmark/anchor_scores.jsonl \
  --manifest runs/benchmark/sampling_manifest.json

French should only be added with an explicitly identified French resource; the seed list used in the audit is not a validated translation.

3. Design, annotate, and adjudicate

mg-benchmark design-annotation-round \
  --items runs/benchmark/items.jsonl \
  --target-only-fraction 0.25 \
  --out-dir runs/annotation/design

This sends every item to the full-context panel and creates stable pair IDs for a language/stratum-balanced, counterbalanced subset sent to a separate target-only panel. The paired rows can therefore be joined without exposing sampling scores or letting one panel see both conditions. Use different rater panels for the two views. Do not let one annotator see both versions of the same item. Use docs/ANNOTATION_GUIDELINES.md.

After at least two independent ratings are returned, build the third-rater queue for disagreements crossing the absent/present boundary:

mg-benchmark adjudication-template \
  --ratings 'runs/annotation/submitted/*.csv' \
  --items runs/benchmark/items.jsonl \
  --out runs/annotation/adjudication.csv

mg-benchmark aggregate-ratings \
  --ratings 'runs/annotation/submitted/*.csv' \
  --adjudication runs/annotation/adjudication_completed.csv \
  --require-adjudication \
  --items runs/benchmark/items.jsonl \
  --out runs/benchmark/gold.jsonl \
  --min-raters 2

mg-benchmark agreement \
  --ratings 'runs/annotation/submitted/*.csv' \
  --out runs/benchmark/agreement.json

mg-benchmark split-file \
  --items runs/benchmark/gold.jsonl \
  --out-dir runs/benchmark

If a file contains one retained panel consensus instead of independent ratings, pass --consensus-size 3; disagreement and IAA are then correctly reported as unavailable rather than reconstructed.

Repeat benchmark sampling and annotation for the declared train and development partitions (or import a separately documented training collection) before running split-file. The confirmatory test quotas should remain frozen and untouched during model development.

4. Train the complete matched encoder matrix

The full-post input is encoded as target [SEP] marked_post with truncation="only_second". The code fails if a target cannot fit instead of silently truncating it.

mg-benchmark run-encoder-matrix \
  --config configs/paper.example.json \
  --train runs/benchmark/train.jsonl \
  --dev runs/benchmark/dev.jsonl \
  --test runs/benchmark/test.jsonl \
  --out-dir runs/encoder_matrix

This runs mDeBERTa and XLM-R in target-only and full-post modes for seeds 13, 17, 23, 29, and 31; selects stopping and operating thresholds on development data; retains item-level predictions; reports strict and sensitivity outcomes; and computes paired post-clustered context contrasts. All model files must already be locally available. The lower-level train-encoder and predict-encoder commands remain available for schedulers. The matrix command rejects any seed list other than the paper's fixed five-seed sequence.

5. Import external reference-system outputs offline

There is intentionally no API client. An approved external system writes JSONL matching schemas/predictions.schema.json, then:

mg-benchmark validate-predictions \
  --predictions runs/external/reference_predictions.jsonl \
  --items runs/benchmark/test.jsonl \
  --out runs/external/validation.json

Register the prompted-system provenance separately. This records the exact model revision, prompt and rubric versions, generation settings, label date, compute/cost metadata, and file digests without storing credentials or making a remote call:

mg-benchmark register-reference-run \
  --metadata configs/reference.example.json \
  --predictions runs/external/reference_predictions.jsonl \
  --items runs/benchmark/test.jsonl \
  --out runs/external/reference_run.json

The metadata format is schemas/reference_run.schema.json. Its system value must exactly match the prediction rows.

Approved provisional labels can be joined only to non-test items:

mg-benchmark attach-labels \
  --items runs/population/items.jsonl \
  --predictions runs/external/provisional_labels.jsonl \
  --partitions train dev \
  --out runs/training/soft_labels.jsonl

The command refuses test even if it is explicitly requested.

6. Evaluate without pooling strata

Generate both lexical comparison systems with the same item IDs:

mg-benchmark score-lexicon \
  --items runs/benchmark/test.jsonl \
  --lexicon en=../data/dictionary_7plus.csv \
  --lexicon nl=../data/nl_dict.csv \
  --lexicon de=../data/de_dict.csv \
  --lexicon it=../data/it_dict.csv \
  --out runs/predictions/lexical_unmodified.jsonl

mg-benchmark score-lexicon \
  --items runs/benchmark/test.jsonl \
  --lexicon en=../data/dictionary_7plus.csv \
  --lexicon nl=../data/nl_dict.csv \
  --lexicon de=../data/de_dict.csv \
  --lexicon it=../data/it_dict.csv \
  --context-modifiers \
  --out runs/predictions/lexical_heuristic.jsonl

mg-benchmark evaluate \
  --gold runs/benchmark/test.jsonl \
  --predictions runs/predictions/full_post.jsonl \
  --positive-threshold 2 \
  --out runs/reports/full_post_strict.json

Repeat with --positive-threshold 1 for the slight-or-ambiguous sensitivity analysis. Development-selected operating thresholds can be supplied with --thresholds.

For a paired contextual contrast:

mg-benchmark compare \
  --gold runs/benchmark/test.jsonl \
  --baseline runs/predictions/target_only.jsonl \
  --comparison runs/predictions/full_post.jsonl \
  --metric average_precision \
  --replicates 2000 \
  --out runs/reports/context_delta.json

Reproduce the completed-pool audit

From the parent repository, using authorized local files:

mg-benchmark audit-pool \
  --gold ../data/gold/gold.jsonl \
  --anchor '../data/gold/*_lexicon_scores.jsonl' \
  --out runs/reports/completed_pool_audit.json

This command reports the P and legacy random halves separately, detects that the latter occupies the lexicon-negative score region, and intentionally emits no pooled model score.

Code-only release gate

The implementation is complete, while data and model artifacts remain pending approval. Before publishing the folder:

mg-benchmark release-check --root .

The gate rejects social-media/annotation/prediction record files, checkpoint formats, generated-output directories, common secret files, and unexpectedly large files. Only the two explicitly invented fixtures in examples/ are allowed.