Skip to content

[Test logic] PCM prefix-match fallback silently accepts clients that truncate trailing samples #48

Description

@balloob

Problem

src/conformance/runner.py:544-559 allows a PCM scenario to pass if the client's hash merely matches a prefix of the source PCM:

prefix_hasher = FloatPcmHasher()
prefix_hasher.update_from_pcm_bytes(prefix_pcm, bit_depth=audio["bit_depth"])
prefix_hash = prefix_hasher.hexdigest()
if prefix_hash == received_hash:
    missing_samples = fixture.frame_count * audio["channels"] - received_sample_count
    return True, f"PCM prefix matches; trailing samples omitted={missing_samples}"

This is deliberate tolerance for adapters that disconnect slightly early. But:

  • The passed reason "trailing samples omitted" ships in the report with status "passed", not a distinct yellow state.
  • A client that quietly truncates audio under normal conditions will appear fully green.
  • No threshold exists — a client that receives 1 sample and correctly hashes the first sample would pass.

Proposed fix

  1. Introduce a bounded tolerance (e.g. "fail if truncation exceeds 5% of source frames" or "only allowed if final chunk was cut off mid-transmission").
  2. Surface a distinct result state in the report (e.g. passed_with_truncation) so truncation shows differently than a full match.
  3. Consider removing the prefix fallback entirely if the intent is to prove full-stream reception.

Context

Prefix tolerance was a pragmatic escape hatch when adapters had racy shutdowns. Now that the harness has deterministic stop sequences, the cost of silent truncation > the convenience of tolerating it.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions