tracelint

tracelint checks an agent's execution trace for structural defects, locally or in CI. It can read trace files directly or import traces from your existing observability platform.

It reads what a tool-calling agent actually did — the execution trace — and flags structural bugs deterministically, with the exact evidence and a CI exit code. It runs after the run, on the trace, not on your code, and no second model ever judges it. Model-as-judge detection of these defects is unreliable (published trace-error benchmarks show low localization accuracy); many of them are structurally decidable and need no judge — that is the entire premise of the tool.

Website: tracelint.com — what it catches, how it works, and how to wire it into CI. See a live demo report — the constructed validation suite (one planted instance of every defect, clean controls, and legitimate-but-suspicious cases) plus the robust-vs-buggy recovery scorecard, generated by tracelint demo.

Quick start — capture → lint → CI

The core loop is the whole product:

run agent in tests → capture the trace → run tracelint → fail CI on provable defects

1. Capture a trace. Most people don't have a trace file yet, so tracelint.capture records one. It wraps your framework's stock OpenInference instrumentation, so the captured file lints with no extra conversion:

pip install "tracelint[capture-smolagents]"   # or [capture-langchain] / [capture-crewai]

from tracelint.capture import capture

with capture("trace.json", framework="smolagents"):
    agent.run("...")            # your agent, unchanged

2. Lint it.

tracelint check trace.json --format openinference

Exit codes: 0 clean · 2 a structurally-provable defect (hard_defect) · 3 an input error. Heuristic candidates never fail CI on their own; a suppression (a rule that couldn't run) is disclosed but is not a defect.

3. Fail CI on a defect. check returns 2 on a provable defect, so it gates a build directly:

# .github/workflows/agent-tests.yml
      - run: pip install tracelint
      - run: tracelint check trace.json --format openinference   # exit 2 fails the job

See Add to CI for the ready-made GitHub Action, SARIF code-scanning, and pre-commit.

The in-process capture helper covers the three in-process frameworks — smolagents, LangGraph / LangChain, and CrewAI. Langflow is a running product, not a library you call, so it uses the OTel-to-file recipe instead (point its export at a file, then check it). Already collect traces elsewhere? Any supported format works with no capture step.

Prefer to see it work first, with no API key or agent? The keyless demo runs a validation suite and recovery scorecard end to end:

pip install tracelint
tracelint demo --html demo.html

Limitations (read first)

1. Deterministic rules catch structural defects, not whether the final answer was correct.
2. Hallucinated-argument, loop, and redundant-call findings are candidates unless structurally proven — legitimate value transforms and intentional retries can trip them; each is shown with its evidence for human review, never asserted as a verdict. High-confidence hallucination detection requires the tool schema to declare field origins (x-value-origin).
3. The recovery scorecard needs labeled task outcomes (success oracles); without them it measures behavioral recovery only ("did not crash"), a weaker claim than correctness.
4. A trace is only as complete as its instrumentation. A rule whose required field is missing is suppressed with a stated reason — tracelint never lints a partial trace as if complete.

Supported formats

check reads native tracelint JSON by default; --format points it straight at the trace your stack already emits — no manual schema conversion:

tracelint check trace.json    --format native         # canonical tracelint JSON (default)
tracelint check spans.json    --format openinference   # OTel/OpenInference: Phoenix, OTLP, TRAIL
tracelint check messages.json --format openai          # an OpenAI chat message list
tracelint check trace.json    --format langfuse        # a Langfuse trace export
tracelint check run.json      --format langsmith       # a LangSmith run tree export

OpenTelemetry / OpenInference is a format, not a platform — a set of OTel span conventions. So --format openinference works from any instrumentation that emits them (Arize Phoenix, OpenLLMetry, Langfuse-via-OTel, datasets like TRAIL) with no account, no platform, and no vendor lock — and it's the format the capture helper writes. One shared adapter reaches the whole ecosystem instead of one vendor.

Most rules need no tool schemas, so this works keyless; add --tools tools.json to light up the schema-dependent rules (R1, and R3's high-confidence tier). Don't have one? tracelint init spans.json --format openinference -o tools.json bootstraps a starter contract from the trace — schemas discovered where the telemetry carries them, behavior fields left as placeholders to review. A multi-trace input (a .jsonl file, a JSON array, or an OTLP export carrying several trace_ids) fans out to one report each. From the library, the same one-liner:

from tracelint import lint_otel_trace

report = lint_otel_trace(spans)   # spans: your OpenInference span export (a list of dicts)
print(report.exit_code)           # 0 or 2

The rules

Rule	Finding	Tiers
R1	schema violation — args fail the tool's JSON Schema	hard_defect
R2a	tool returned an error	hard_event (structured signal) / candidate (heuristic)
R2b	an errored result's value reused by a later side-effecting call	hard_defect / candidate
R3	hallucinated argument — value not derivable from provenance	candidate; hard_defect if the field is annotated provided
R4	loop — N identical no-progress calls (polls/retries excluded)	candidate
R5	redundant call — identical call + identical result, no mutation between	candidate
R6	malformed arguments — the emitted tool-call arguments are not valid JSON	hard_defect
R7	unknown tool — a call to a tool absent from the declared toolset (possible hallucinated tool)	candidate

hard_event and hard_defect are orthogonal to the finding kind: a tool-error event is a hard_event from a structured status field but a candidate from an exception-like string in free-form content.

Input format

A native trace is a JSON object (.json, or .jsonl for many):

{
  "run_id": "run-1",
  "steps": [
    {"type": "message", "role": "user", "content": "cancel order 4521 if it hasn't shipped"},
    {"type": "tool_call", "call_id": "c1", "name": "get_order_status", "args": {"order_id": "4521"}},
    {"type": "tool_result", "call_id": "c1", "content": {"status": "processing"}, "status": "ok"},
    {"type": "tool_call", "call_id": "c2", "name": "cancel_order",
     "args": {"order_id": "4521", "reason": "not_shipped"}}
  ],
  "final": "Order 4521 has been cancelled."
}

tools.json supplies the ground truth the rules check against:

{
  "tools": {
    "cancel_order": {
      "schema": {"type": "object", "properties": {"order_id": {"type": "string"}},
                 "required": ["order_id"]},
      "metadata": {"side_effecting": true}
    }
  }
}

A tool can also declare what failure looks like in its result, so a domain failure returned as a transport success (HTTP 200 carrying {"status": "declined"}) is caught structurally instead of slipping through:

{
  "tools": {
    "charge_card": {
      "metadata": {
        "side_effecting": true,
        "failure_when": {"pointer": "/status", "in": ["declined", "failed"]}
      }
    }
  }
}

failure_when is a JSON Pointer into the result plus a match (in / equals / exists); a match is a structured error for R2 (feeding R2a and, on reuse into a side-effecting call, R2b). A side-effecting tool with no failure_when and an unclassifiable result is suppressed with a reason — never counted as a clean pass.

The rules run against one canonical trace schema; a thin adapter translates each source's format into it, so the rules never change: from_openai_messages, from_langfuse_trace, from_langsmith_run, and from_otel_spans. The adapters are validated against live data, not just the spec — from_otel_spans on real TRAIL benchmark traces, where tracelint deterministically localized real tool errors, a malformed call, and excessive-retry loops with no model in the loop. Real exports vary, so a new source may need a small adapter tweak — and when a field a rule needs is absent, that rule suppresses (says so) rather than guessing, so an unhandled quirk degrades safely.

Integrations & recipes

tracelint reads the telemetry your stack already emits — one shared adapter reaches the whole ecosystem. docs/integrations/ has short, reproducible one-pagers, each validated on a real captured trace:

• Frameworks (capture a trace in a test): smolagents · LangGraph / LangChain · CrewAI — each captures via the helper above and lints via --format openinference with no framework-specific code. Langflow is a running product, so it uses the OTel-to-file recipe instead of the in-process helper.
• Platforms (already have traces?): the always-works baseline is to export the trace to a file and check it — Arize Phoenix · Langfuse · LangSmith · OpenLLMetry / Traceloop · OpenAI / ShareGPT message lists.

For Langfuse, pull is convenience sugar on top of that baseline — it fetches a trace straight to a file, so pull → check composes and the file doubles as a saved fixture:

pip install "tracelint[langfuse]"                 # reads your LANGFUSE_* env vars
tracelint langfuse pull <trace-id> -o trace.json   # writes native tracelint JSON
tracelint check trace.json                          # lint it (native is the default format)

These are compatibility validations on real traces, not benchmarks or endorsements.

Straight from a running Arize Phoenix instance, from the library:

import phoenix as px
from tracelint import lint_otel_trace

spans = px.Client().get_spans_dataframe().to_dict("records")
print(lint_otel_trace(spans).exit_code)

Both Phoenix shapes are handled: the span-export JSON (top-level span_kind) and the get_spans_dataframe() records (attributes as attributes.* columns).

Add to CI

tracelint check returns exit 2 on a structurally-provable defect, so it gates a build directly. Point it at the traces your agent test job already produces — a defect fails the job; heuristic candidates never do.

GitHub Actions — the ready-made action:

name: lint-agent-traces
on: [push, pull_request]
jobs:
  tracelint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      # ... your step that runs the agent and writes traces to ./traces ...
      - uses: AshwinUgale/tracelint@v0.8.0
        with:
          traces: "traces/*.jsonl"
          format: "openinference"     # or native / openai / langfuse
          tools: "tools.json"          # optional — lights up R1, R3, R2 predicates

Any CI, without the action — it's one pip install and one command:

pip install tracelint
tracelint check traces/*.jsonl --format openinference --tools tools.json

GitHub code scanning (SARIF) — surface findings in the repo's Security → Code scanning tab and as inline pull-request annotations. --sarif writes a SARIF 2.1.0 file (mapping hard_defect → error, hard_event → warning, candidate → note). It's written before the exit-2 gate, so an if: always() upload step runs even when a defect fails the job:

      - run: tracelint check traces/*.jsonl --format openinference --tools tools.json --sarif tracelint.sarif
      - uses: github/codeql-action/upload-sarif@v3
        if: always()          # upload even when tracelint exits 2 on a hard_defect
        with:
          sarif_file: tracelint.sarif

pre-commit — lint only the trace files a commit touches:

repos:
  - repo: https://github.com/AshwinUgale/tracelint
    rev: v0.4.1
    hooks:
      - id: tracelint
        files: ^traces/.*\.jsonl$
        args: ["--format", "openinference", "--tools", "tools.json"]

Library

from tracelint import lint_trace, default_rules, Trace, ToolRegistry

trace = Trace.load("trace.json")
registry = ToolRegistry.load("tools.json")
report = lint_trace(trace, default_rules(), registry)
print(report.exit_code)          # 0 or 2
for f in report.active_findings:
    print(f.rule, f.tier.value, f.summary)

See examples/lint_openinference_phoenix.py for an offline, keyless end-to-end run (Phoenix-shaped spans → findings, with and without a tool registry).

Advanced

These layers sit around the core loop and are opt-in.

Native test assertion (pytest). Opt in with pytest_plugins = ["tracelint.pytest_plugin"] in your conftest.py; the trace_capture fixture then captures a run and lints it inline, so a hard defect fails the test (candidates never do):

def test_agent(trace_capture):
    with trace_capture(framework="smolagents"):
        agent.run("refund order A100")   # a hard defect in the trace fails this test

Pass assert_clean=False to inspect cap.report yourself instead of auto-failing. It's a thin wrapper over the capture helper, so the same capture-<framework> extra applies.

Recovery scorecard & fault injection. Measure how an agent behaves under injected faults, scored against deterministic success oracles:

tracelint scorecard --demo --faults timeout,error,rate_limit --runs 5

The baseline must satisfy the oracle first (else recovery is not measured). Each fault type reports a correctness-recovery rate with a Wilson confidence interval; with no oracle it falls back to behavioral recovery, labeled as weaker.

Write findings back into Langfuse. Beyond reading a trace, tracelint can write its verdict back into Langfuse's own Score model so it appears beside your normal evals. This is the one part that overlaps with what a platform does itself, so it's an advanced recipe, not the primary path — the baseline (export or pull a trace, then check it) always works without it:

pip install "tracelint[langfuse]"        # v3 SDK; reads your LANGFUSE_* env vars

tracelint langfuse check --trace <trace-id> --tools tools.json               # read-only: prints the plan
tracelint langfuse check --trace <trace-id> --tools tools.json --write-back  # writes Scores

With --write-back, tracelint writes tracelint.passed (BOOLEAN) and tracelint.hard_defects (NUMERIC) at the trace level, plus one score per certain finding attached to the exact offending observation. Heuristic candidates are shown in the terminal but never written back. Scores are keyed by a stable finding id, so re-running updates them in place rather than duplicating.

Development

python -m pytest
ruff check src tests

The core is dependency-light (jsonschema + stdlib) and the whole test suite is deterministic and offline. A real OpenAI trace-generating agent lives behind the opt-in [real-agent] extra and is never part of the linter. Python 3.10–3.12.