[Cookie0103]

What does this PR do?

Fixes #514

Runner (adalflow/components/agent/runner.py) runs call, acall and astream without any reliability handling. Today:

• Nothing can time out. A planner or tool call that never returns blocks the run forever.
• Errors are classified by substring matching. _is_unrecoverable_error checks "400" in error / "429" in error, so any error text that happens to contain those digits is fatal. An HTTP 429, which should back off and retry, ends the run instead.
• A tool exception ends the run. In acall it reaches the outer except Exception and breaks the loop.
• Step history lives only in memory. A crash or a cancelled stream loses every completed step, and running again repeats tools that already had side effects.

This PR adds an opt-in reliability layer, adalflow.components.agent.reliability, and wires it into all three Runner loops. With no ReliabilityConfig passed, the default code path is unchanged; the 44 existing runner and tracing tests pass without modification.

Usage

from adalflow.components.agent import (
    Runner, ReliabilityConfig, RetryPolicy, TimeoutPolicy, ToolPolicy, FileCheckpointStore,
)

runner = Runner(
    agent=agent,
    reliability=ReliabilityConfig(
        retry=RetryPolicy(max_attempts=4),                              # 429 / 5xx / timeouts
        timeouts=TimeoutPolicy(planner_timeout=60, tool_timeout=30, run_timeout=600),
        tools={"search": ToolPolicy(idempotent=True, timeout=10)},      # safe to retry
    ),
    checkpoint_store=FileCheckpointStore("./runs"),
)

result = await runner.acall(prompt_kwargs, run_id="ticket-123")
# ...the process crashes mid-run...
result = await runner.aresume("ticket-123")   # completed tool calls are replayed, not re-executed

Design

Module	Responsibility
classification.py	ErrorClassifier reads status_code / response.status_code, Retry-After (seconds or HTTP date) and the __cause__/__context__ chain, with no provider SDK imports. For GeneratorOutput.error strings it uses anchored regexes instead of bare digit matching. Network OSErrors are told apart from e.g. FileNotFoundError. Users can register their own rules.
policy.py	Frozen, validated BackoffPolicy (exponential with full/equal jitter, honours Retry-After, capped), RetryPolicy, TimeoutPolicy (per-op timeout bounded by the remaining run budget), and per-tool ToolPolicy.
executor.py	ReliableExecutor applies the policies to planner and tool calls, in both async and sync paths (sync timeouts use an owned single-worker pool, released by Runner.close()). CancelledError is never retried, and cancelling during a backoff sleep propagates immediately. Clock, sleep and RNG are injectable so tests run instantly.
checkpoint.py	Versioned RunCheckpoint with a sha256 integrity check and a migration hook. Holds the step history, token usage, a RunFingerprint (agent, model, tool signatures, input), the in-flight pending_function, and a write-ahead tool ledger.
store.py	InMemoryCheckpointStore (deep copies, locked) and FileCheckpointStore (temp file + fsync + os.replace; run IDs restricted to a safe charset or hashed to prevent path traversal; optional history rotation; falls back to the newest valid history file if the primary is corrupted; prune()).
runner.py	Shared helpers (_planner_call / _planner_acall, _execute_tool_reliably[_async], _checkpoint) used by call, acall and astream, including stream_tool_execution. Adds run_id / resume / on_incomplete_tool / allow_fingerprint_mismatch keywords and resume() / aresume().

Engineering trade-offs

• Tools are non-idempotent by default. A tool is only retried if it is marked ToolPolicy(idempotent=True). A tool timeout becomes a ToolOutput(status="timeout") observation that the agent can react to; it does not crash the run.
• The tool ledger is write-ahead and persisted. A started entry is saved before the tool runs, and marked completed with the resolved output afterwards (after coroutine and generator results have been consumed).
  • On resume, a completed call is replayed from the ledger and the tool is not called again.
  • A started entry for a non-idempotent tool raises ToolReplayRefusedError unless the caller passes on_incomplete_tool="rerun" or "skip".
  • Ledger keys include the step index, so calling the same tool with the same arguments at two different steps still executes it both times.
• Resume checks a fingerprint. If the agent, tools, model or input differ from the checkpoint, resume raises ResumeMismatchError, unless allow_fingerprint_mismatch=True.
• Checkpointing never breaks a run. Store failures are logged, not raised.
• Out of scope: PermissionManager behaviour (PermissionManager: a tool registered require_approval=True executes with no approval when no callback is set #501 / fix(permission): deny require_approval tools when no callback is set #503).

Verification

• pytest tests/test_runner.py tests/test_runner_tracing.py tests/test_runner_reliability.py tests/test_runner_reliability_integration.py: 118 passed, on both Python 3.12 and Python 3.9 (the package's minimum).
• The new tests use fake planners and tools, with no network or model calls. They cover:
  • the classifier table, including the "Processed 1400 rows" false positive, Retry-After handling and cause chains;
  • backoff bounds;
  • 429 retried instead of terminating the run, and context overflow stopping it;
  • the existing error-step semantics when retries are exhausted;
  • planner, tool and run timeouts;
  • idempotent vs. non-idempotent retry;
  • cancellation during backoff saving a cancelled checkpoint;
  • checkpoint round-trip, corruption and versioning;
  • atomic writes and path-traversal rejection;
  • resume after a crash, both mid-tool and after the tool finished but before the step was recorded (checked with call counters);
  • fingerprint mismatch, and resuming a completed run without calling the planner;
  • the sync call path;
  • astream reliability events;
  • async-generator tool outputs in the ledger;
  • executor cleanup.
• Full tests/ directory: the failures and errors (40 failed, 3 errors in modules such as test_openai_client.py and _test_optimizer.py, plus collection errors for missing optional dependencies) are identical on a clean checkout of main. This PR introduces no new failures.
• ruff (v0.4.2, repo config) and black (24.4.2) are clean on all new and changed files.

Breaking changes

None. Additive keyword-only parameters, a new "timeout" value in ToolOutput.status, and a new ReliabilityRunItem event type.

Limitations

• A timed-out sync call can be abandoned but not killed, because Python threads cannot be interrupted. This is documented in the executor.
• FileCheckpointStore uses fcntl advisory locks for cross-process safety. On Windows that lock is skipped, though in-process locking still applies.

Before submitting

• Discussed in issue Runner: add opt-in timeouts, retry policy, checkpoints and resume #514
• Did you read the contributor guideline?
• Did you make sure your PR does only one thing?
• Did you write any new necessary tests?
• Did you verify new and existing tests pass locally with your changes?
• Did you list all the breaking changes introduced by this pull request? (none)

AI disclosure: implemented with AI coding assistance (Claude, Cursor). I reviewed and tested every change.

🤖 Generated with Claude Code