COMPAS: Feature-Focused Test Generation.

COMPAS implements Feature-Focused Test Generation, an approach that bridges grammar-based testing and software product line engineering. Given a Fandango .fan grammar, COMPAS:

1. Featurizes the grammar into a UVL feature model, mapping semantic alternatives to configurable features and extracting cross-tree constraints
2. Configures the model — the user selects which features to target
3. Prunes the grammar to a subgrammar that generates only inputs matching the selection

By constructing one subgrammar per feature value, COMPAS produces a feature-wise test suite where each input targets a specific feature with full traceability to the code it exercises.

The name is also compas — the rhythmic beat that guides a fandango.

---

Installation

git clone https://github.com/fandango-fuzzer/compas.git
cd compas
make env
source .venv/bin/activate

Requires Python >= 3.11, fandango-fuzzer >= 1.1.0, and flamapy >= 2.5.0.

make env also installs the evaluation runtime dependencies (Pillow, coverage, mutagen, pypdf, and pyelftools) so the evaluation runs out of the box across all 10 subjects.

---

Quick Start

Interactive pipeline

compas pipeline grammar.fan

Step-by-step

# 1. Featurize: grammar -> UVL feature model
compas featurize grammar.fan -o grammar.uvl

# 2. Configure: select features interactively
compas configure grammar.uvl -o grammar.csvconf

# 3. Prune: grammar + config -> targeted subgrammar
compas prune grammar.fan -c grammar.csvconf -o grammar_targeted.fan

# 4. Generate with Fandango
fandango fuzz -f grammar_targeted.fan -n 100

Python API

from compas import analyze, prune
from compas.model import Configuration

# Featurize
fm = analyze("grammar.fan")

# Configure
config = Configuration()
for f in fm.all_features():
    config.select(f.name)
config.deselect("SomeFeature_value")

# Prune
prune("grammar.fan", fm, config, output="targeted.fan")

---

Evaluation

The evaluation/ directory evaluates Feature-Focused Test Generation against baseline Fandango generation across 10 subjects. The harness featurizes each grammar, derives one test slice per feature value, generates a uniform budget per slice and an equal-budget unfocused baseline, measures the real parser's per-input Python-line coverage, and regenerates evaluation/structural/RESULTS.md with every table. See evaluation/structural/README.md for the full run guide.

Running

# Reproducible (pinned environment) via Docker — recommended:
make build        # build the compas-eval image
make smoke        # fast seeded sanity run (~1-2 min) -> RESULTS-smoke.md
make calibrate    # budget sweep N in {1,5,50,100}, seeded
make campaign     # full paper run: N=50, 10 reps, seeded (multi-hour)

# Or locally in the project venv:
make evaluation                       # full run (N=50, 10 reps, seeded)
make evaluation-target TARGET=toml    # single subject
python evaluation/structural/bench.py --n 50 --runs 10 --seed 0   # equivalent
python evaluation/structural/bench.py --calibrate --runs 3        # budget sweep

--seed makes runs bit-for-bit reproducible (default 0). The throughput column is steady-state generation throughput (inputs/sec, excluding one-time setup), on which targeted is at least baseline on every subject. The uniform per-slice budget N=50 is justified by the calibration sweep — metrics are identical at N=50 and N=100, so 50 is the lower bound. The baseline receives N x (number of slices) inputs.

What it measures

For each subject the harness:

1. Featurizes the grammar into feature groups, values, and constraints.
2. Builds one subgrammar per feature value (a slice), with constraint propagation.
3. Generates N targeted inputs per slice, and a baseline corpus of exactly the targeted suite's realized size from the full grammar (equal input budget).
4. Measures validity and per-input Python line coverage of the real parser module.
5. RQ1 Preservation: validity, feature diversity, Python line coverage, exact arc distance, aggregate coverage (equal by design), and steady-state generation throughput, targeted vs baseline.
6. RQ2 Focusing: how often / how quickly targeted slices reach their manually annotated intended regions versus baseline.
7. RQ3 Traceability: manual feature-value to handler-region annotations (bench_annotations.py), scored as Hit/Own/Top/Uniq per slice.
8. RQ4 Minimization: greedy labelled covering set and per-region time-to-coverage speedup.

All tables are regenerated into evaluation/structural/RESULTS.md on every run.

Subjects

Format	Parser	Groups	Values	Excl.	Comp.	Configs
ELF	pyelftools	2	11	0	1	28
GIF	Pillow	7	18	0	0	576
HTML	html.parser	3	13	0	0	48
JPEG	Pillow	5	15	3	0	216
MP3	mutagen	2	9	0	2	20
PDF	pypdf	3	13	1	0	80
PNG	Pillow	10	29	5	2	9,216
TIFF	Pillow	4	17	8	15	240
TOML	tomllib	3	15	0	0	60
ZIP	zipfile	4	8	0	13	32

Groups = feature groups; Values = feature values; Excl. = extracted cross-tree excludes; Comp. = computed-field constraints (CRC, sizes, offsets) that derive dependent fields rather than restrict feature combinations; Configs = valid configurations.

---

Project Structure

compas/
    src/compas/        Core library
        __init__.py           Public API: analyze(), prune()
        model.py              Feature, FeatureModel, Configuration
        analyzer.py           Grammar -> FeatureModel
        pruner.py             Grammar + Config -> Subgrammar
        configurator.py       Interactive configuration
        uvl.py                UVL serialization
        cli.py                CLI entry points
    evaluation/
        structural/    Evaluation harness (see its README)
            bench.py              Benchmark driver + dynamic RESULTS.md
            eval.py               Shared helpers (generation, coverage, validity)
            cover_worker.py       Per-input coverage subprocess
            fmstats.py            Table 1 (feature-model statistics)
            bench_annotations.py  Manual feature-value -> handler regions (RQ3)
            p_<format>.py         Per-subject parse harnesses
            README.md             Run guide
            RESULTS.md            Results, all tables (seed-0 reference; regenerated on run)
        targets/<format>/<format>.fan    One Fandango grammar per subject
    Dockerfile
    requirements.txt   Pinned dependencies for the reproduction image
    tests/             pytest test suite

---

Development

make test            # run full test suite
make lint            # ruff
make typecheck       # mypy
make clean           # remove build artifacts (keeps .venv)

---

License

EUPL-1.2 — see LICENSE.md.