Feat/clear mlflow traces pypi 2.0.2

adapters/clear/examples/docs/03-local-run.md

@@ -13,21 +13,31 @@ pip install -U pip
 pip install -r requirements.txt
 ```
 
-IBM CLEAR is installed from **Git** per **`requirements.txt`** (PyPI wheels may omit agentic pieces).
+IBM CLEAR is installed from PyPI (`clear-eval`) per **`requirements.txt`**.
 
 ## 2. Traces
 
-Put one or more CLEAR-compatible **`*.json`** trace files in a directory. A common layout is **`input-trace/`** beside **`main.py`**; this repo also ships **example traces** under **`examples/input-traces/`** (see [02-agent-traces.md](02-agent-traces.md)). In your job JSON, set **`parameters.data_dir`** to that folder’s path (relative to where you run **`python main.py`**, or absolute). The adapter collects **`*.json`** under that directory.
+The adapter supports three trace input sources:
+
+| Source | How to configure |
+|---|---|
+| **S3** | Set `test_data_ref.s3` in the job spec — Kubernetes mounts the bucket contents under `/test_data` or `/data` automatically |
+| **MLflow** | Set `parameters.mlflow_traces_experiment_name` — traces are fetched directly from the MLflow tracking server (requires `MLFLOW_TRACKING_URI` env var) |
+| **Local path** | Set `parameters.data_dir` to a local folder path — for local runs and testing |
+
+For a quick local run, put `*.json` trace files in a directory (e.g. `input-trace/` beside `main.py`) and set `parameters.data_dir`. This repo ships example traces under `examples/input-traces/` (see [02-agent-traces.md](02-agent-traces.md)).
+
+> **MLflow experiment names:** `experiment_name` (top-level in the job spec) is where CLEAR **results** are stored. `parameters.mlflow_traces_experiment_name` is where input **traces** are fetched from. They can be the same or different experiments.
 
 ## 3. Job specification
 
 Copy or start from **`meta/job.json`**. Minimum expectations include:
 
 - **`benchmark_id`**
-- **`parameters.data_dir`** — path to your trace folder
 - **`parameters.eval_model_name`**, **`parameters.provider`**
+- One trace source (see table above); for local runs set **`parameters.data_dir`**
 - **`parameters.inference_backend`**: **`"litellm"`** or **`"endpoint"`**
-- **`model.url`** — OpenAI-compatible API base (often **`…/v1`**)
+- **`model.url`** — OpenAI-compatible API base (often **`...../v1`**)
 
 **How `model.url` is used depends on the backend (`parameters.provider` is required for both):**
 
@@ -36,7 +46,7 @@ Copy or start from **`meta/job.json`**. Minimum expectations include:
 | `"litellm"` (default) | The adapter reads `model.url` and sets it as `OPENAI_BASE_URL` before invoking CLEAR. LiteLLM picks up `OPENAI_BASE_URL` automatically and routes all calls through it. If `model.url` is not set, LiteLLM falls back to the OpenAI SDK default. |
 | `"endpoint"` | The adapter passes `model.url` as `endpoint_url` directly in the CLEAR config. |
 
-For **local** runs with **`litellm`**, you often **delete `model.auth`** from the JSON when no Kubernetes Secret exists, and set **`OPENAI_API_KEY`** in the shell **only if** your endpoint requires it. Many **local** servers (for example some **Ollama** setups) do not require a key.
+For **local** runs with **`litellm`**, you often **delete `model.auth`** from the JSON when no Kubernetes Secret exists, and set **`OPENAI_API_KEY`** in the shell **only if** your endpoint requires it. Many **local** servers (for example some Ollama setups) do not require a key.
 
 ## 4. Run
 
@@ -46,11 +56,18 @@ export EVALHUB_JOB_SPEC_PATH=meta/job.json   # or your edited copy
 python main.py
 ```
 
-**MLflow upload (optional)** — set an experiment and tracking URI if you want artifacts in MLflow:
+**MLflow upload (optional)** — set an experiment and tracking URI if you want results stored in MLflow:
+
+```bash
+export MLFLOW_TRACKING_URI='https://your-mlflow-server.example/'
+# experiment_name in job JSON controls which experiment results go into
+```
+
+**MLflow trace fetch (optional)** — to fetch input traces from MLflow instead of a local path:
 
 ```bash
 export MLFLOW_TRACKING_URI='https://your-mlflow-server.example/'
-# experiment_name in job JSON, or parameters.mlflow_experiment_name
+# set parameters.mlflow_traces_experiment_name in job JSON
 ```
 
 Without **`MLFLOW_TRACKING_URI`** / experiment configuration, the adapter skips MLflow upload as documented in the adapter README.
@@ -63,7 +80,7 @@ After a successful run the **run root** (e.g. `output/` beside `main.py`, or the
 - **`clear_results.html`** — static dashboard; open in a browser.
 - **`clear_results.dashboard_data.json`** — companion data for the dashboard.
 
-Intermediate CLEAR directories (`step_by_step/`, `traces_data/`, etc.) are **removed** after a successful run once their outputs have been preserved at the run root. If you open `step_by_step/clear_results.html` directly and find it missing, that is expected—use the root-level `clear_results.html` instead.
+Intermediate CLEAR directories (`step_by_step/`, `traces_data/`, etc.) are **removed** after a successful run once their outputs have been preserved at the run root. If you open `step_by_step/clear_results.html` directly and find it missing, that is expected — use the root-level `clear_results.html` instead.
 
 Optional styling: **`parameters.clear_dashboard_theme`** — [06-dashboard-theme.md](06-dashboard-theme.md).
 

adapters/clear/examples/docs/05-benchmarks-and-parameters.md

@@ -42,6 +42,17 @@ CLEAR maps traces against this list rather than relying only on open-ended clust
 
 Sample: [`benchmark-jobs/03-predefined-issues.json`](../benchmark-jobs/03-predefined-issues.json)
 
+## Tool-call analysis (SPARC)
+
+Set **`parameters.separate_tools`** to **`true`** to enable SPARC — CLEAR's tool-call analysis mode. When enabled, each span whose response contains a tool call is split into a separate row before the judge sees it, giving finer-grained per-tool scoring.
+
+```json
+"separate_tools": true
+```
+
+- Default is **`false`** (standard agentic mode — tool calls are evaluated in-context with the surrounding agent step).
+- Requires the adapter to be installed with the `[tool-calls]` extra (already included in `requirements.txt`).
+
 ## Inference backend
 
 Set **`parameters.inference_backend`** to **`"litellm"`**. Configure **`model.url`** and credentials per [03-local-run.md](03-local-run.md) and the adapter **`README.md`**.

adapters/clear/main.py

@@ -35,8 +35,9 @@
     OCIArtifactResult,
     OCIArtifactSpec,
 )
+from evalhub.adapter.mlflow import MlflowClient as EvalHubMlflowClient
 from evalhub.adapter.auth import resolve_model_credentials
-from evalhub.adapter.mlflow import MlflowArtifact
+from evalhub.adapter.mlflow import MlflowArtifact, TracesNamespace
 
 from themes import RED_HAT_CLEAR_DASHBOARD_CSS, RED_HAT_DASHBOARD_JS_PATCHES
 
@@ -49,7 +50,8 @@
     from clear_eval.agentic.pipeline.utils import get_run_output_dir
 except ImportError as exc:
     raise ImportError(
-        "Install IBM CLEAR from source (see requirements.txt; PyPI wheels omit agentic)."
+        "Install IBM CLEAR with tool-calls support (see requirements.txt): "
+        "pip install 'clear_eval[tool-calls] @ git+https://github.com/IBM/clear.git@main_with_sparc'"
     ) from exc
 
 logger = logging.getLogger(__name__)
@@ -281,6 +283,135 @@ def _apply_clear_dashboard_theme(html_paths: list[Path], theme: str | None) -> N
             logger.warning("Could not apply dashboard theme to %s: %s", path, exc)
 
 
+def _unwrap_proto_attr_value(v_wrapper: Any) -> Any:
+    """Unwrap a protobuf-style attribute value dict to a plain Python value."""
+    if not isinstance(v_wrapper, dict):
+        return v_wrapper
+    for key in ("string_value", "bytes_value", "int_value", "double_value", "bool_value"):
+        if key in v_wrapper:
+            return v_wrapper[key]
+    if "array_value" in v_wrapper:
+        return str(v_wrapper["array_value"])
+    return str(v_wrapper)
+
+
+def _proto_attrs_to_dict(attrs: Any) -> dict[str, Any]:
+    """Convert protobuf-style span attributes list to a plain dict."""
+    if isinstance(attrs, dict):
+        return attrs
+    if not isinstance(attrs, list):
+        return {}
+    result: dict[str, Any] = {}
+    for item in attrs:
+        if isinstance(item, dict) and "key" in item:
+            result[str(item["key"])] = _unwrap_proto_attr_value(item.get("value", {}))
+    return result
+
+
+def _fetch_mlflow_traces_to_dir(mlflow_client: EvalHubMlflowClient, parameters: dict[str, Any], output_dir: str) -> None:
+    """Fetch full traces (with spans) from MLflow using the eval-hub-sdk client.
+
+    Uses the sdk's TracesNamespace (search + get) to retrieve spans via the MLflow
+    3.x API, then converts each trace to the flat format CLEAR's preprocessor expects:
+    {trace_id, experiment_id, state, timestamp_ms, execution_time_ms, tags, spans[]}
+    """
+    experiment_name = parameters.get("mlflow_traces_experiment_name")
+    experiment_id = parameters.get("mlflow_traces_experiment_id")
+    max_results = int(parameters.get("mlflow_traces_max_results", 100))
+    run_id = parameters.get("mlflow_traces_run_id")
+
+    if not experiment_name and not experiment_id:
+        raise ValueError(
+            "Set parameters.mlflow_traces_experiment_name or mlflow_traces_experiment_id"
+        )
+
+    if not experiment_id and experiment_name:
+        exp = mlflow_client.get_experiment_by_name(experiment_name)
+        if exp is None:
+            raise ValueError(f"MLflow experiment not found: {experiment_name!r}")
+        experiment_id = exp.experiment_id
+
+    filter_string = None
+    if run_id:
+        safe_run_id = run_id.strip().replace("'", "''")
+        filter_string = f"attribute.run_id = '{safe_run_id}'"
+
+    all_trace_summaries = []
+    page_token: str | None = None
+    while len(all_trace_summaries) < max_results:
+        page_size = min(100, max_results - len(all_trace_summaries))
+        traces, page_token = mlflow_client.traces.search(
+            experiment_ids=[experiment_id],
+            max_results=page_size,
+            filter_string=filter_string,
+            page_token=page_token,
+        )
+        if not traces:
+            break
+        all_trace_summaries.extend(traces)
+        if not page_token:
+            break
+
+    if not all_trace_summaries:
+        raise ValueError(
+            f"No traces found in MLflow experiment {experiment_name or experiment_id!r}"
+        )
+
+    Path(output_dir).mkdir(parents=True, exist_ok=True)
+    for summary in all_trace_summaries:
+        # Re-fetch with get() to include full span data (3.x API)
+        trace = mlflow_client.traces.get(
+            request_id=summary.info.request_id,
+            experiment_id=experiment_id,
+        )
+        info = trace.info
+        rid = info.request_id
+
+        spans = []
+        for s in trace.data.get("spans", []):
+            attrs = _proto_attrs_to_dict(s.get("attributes", []))
+            raw_status = s.get("status", {})
+            if isinstance(raw_status, dict):
+                code = raw_status.get("code", "")
+                if code.startswith("STATUS_CODE_"):
+                    code = code[len("STATUS_CODE_"):]
+                span_status = {"status_code": code or info.status, "description": ""}
+            else:
+                span_status = {"status_code": info.status, "description": ""}
+
+            start_ns = s.get("start_time_unix_nano", 0)
+            end_ns = s.get("end_time_unix_nano", 0)
+            duration_ms = (end_ns - start_ns) / 1_000_000 if end_ns and start_ns else None
+            spans.append({
+                "span_id": s.get("span_id", ""),
+                "parent_span_id": s.get("parent_span_id"),
+                "name": s.get("name", ""),
+                "status": span_status,
+                "start_time_unix_nano": start_ns,
+                "end_time_unix_nano": end_ns,
+                "duration_ms": duration_ms,
+                "attributes": attrs,
+            })
+
+        trace_doc = {
+            "trace_id": rid,
+            "experiment_id": experiment_id,
+            "state": info.status,
+            "timestamp_ms": info.timestamp_ms,
+            "execution_time_ms": info.execution_time_ms,
+            "tags": info.tags or {},
+            "spans": spans,
+        }
+
+        safe_rid = rid.replace("/", "_").replace("\\", "_")
+        out_path = Path(output_dir) / f"{safe_rid}.json"
+        with open(out_path, "w") as fh:
+            json.dump(trace_doc, fh, indent=2, default=str)
+
+    logger.info("Fetched %d trace(s) from MLflow experiment %s -> %s",
+                len(all_trace_summaries), experiment_name or experiment_id, output_dir)
+
+
 class ClearAdapter(FrameworkAdapter):
     """eval-hub FrameworkAdapter that runs IBM CLEAR on trace JSON and returns JobResults."""
 
@@ -343,13 +474,38 @@ def run_benchmark_job(self, config: JobSpec, callbacks: JobCallbacks) -> JobResu
                             [c.name for c in data_path.iterdir()],
                         )
 
+            if not data_dir and TracesNamespace.is_source_configured(config.parameters):
+                if os.environ.get("MLFLOW_TRACKING_URI"):
+                    logger.info("Fetching traces from MLflow experiment via SDK")
+                    callbacks.report_status(
+                        JobStatusUpdate(
+                            status=JobStatus.RUNNING,
+                            phase=JobPhase.LOADING_DATA,
+                            progress=0.1,
+                            message=MessageInfo(
+                                message="Fetching traces from MLflow",
+                                message_code="fetching_mlflow_traces",
+                            ),
+                        )
+                    )
+                    mlflow_traces_dir = tempfile.mkdtemp(prefix="clear_mlflow_traces_")
+                    _fetch_mlflow_traces_to_dir(self.mlflow, config.parameters, mlflow_traces_dir)
+                    data_dir = mlflow_traces_dir
+                    logger.info("MLflow traces fetched to %s", data_dir)
+                else:
+                    logger.warning(
+                        "MLflow trace source is configured in parameters but "
+                        "MLFLOW_TRACKING_URI is not set — skipping MLflow fetch"
+                    )
+
             if not data_dir:
                 data_dir = config.parameters.get("data_dir") or config.parameters.get(
                     "traces_input_dir"
                 )
                 if not data_dir:
                     raise ValueError(
                         "No input traces: mount data under /test_data or /data, "
+                        "fetch from MLflow (set parameters.mlflow_traces_experiment_name), "
                         "or set parameters.data_dir (preferred) or parameters.traces_input_dir"
                     )
                 logger.info("Using data_dir from parameters: %s", data_dir)
@@ -524,11 +680,13 @@ def _validate_config(self, config: JobSpec) -> None:
         has_traces_param = (
             "data_dir" in config.parameters or "traces_input_dir" in config.parameters
         )
+        has_mlflow_traces = TracesNamespace.is_source_configured(config.parameters)
 
-        if not has_test_data and not has_data and not has_traces_param:
+        if not has_test_data and not has_data and not has_traces_param and not has_mlflow_traces:
             raise ValueError(
-                "Provide parameters.data_dir (or traces_input_dir) or mount test data under "
-                "/test_data or /data"
+                "Provide parameters.data_dir (or traces_input_dir), set "
+                "parameters.mlflow_traces_experiment_name to fetch from MLflow, "
+                "or mount test data under /test_data or /data"
             )
 
         if "eval_model_name" not in config.parameters:
@@ -595,21 +753,24 @@ def _build_agentic_config(
             # the remaining params are set internally in clear
         }
 
-        if agentic_config.get("inference_backend") == "endpoint":
-            if "endpoint_url" in config.parameters:
-                ep = config.parameters["endpoint_url"]
-            elif "inference_url" in config.parameters:
-                ep = config.parameters["inference_url"]
-            elif config.model.url:
-                ep = config.model.url
-            else:
+        if config.model.url:
+            ep = (
+                config.parameters.get("endpoint_url")
+                or config.parameters.get("inference_url")
+                or config.model.url
+            )
+            if agentic_config.get("inference_backend") == "endpoint" and not ep:
                 raise ValueError(
                     "model.url or parameters.endpoint_url (or inference_url) is required "
                     "when inference_backend is 'endpoint'"
                 )
-            # CLEAR's eval pipeline expects endpoint_url; keep inference_url for compatibility.
             agentic_config["endpoint_url"] = ep
             agentic_config["inference_url"] = ep
+        elif agentic_config.get("inference_backend") == "endpoint":
+            raise ValueError(
+                "model.url or parameters.endpoint_url (or inference_url) is required "
+                "when inference_backend is 'endpoint'"
+            )
 
         if "eval_model_params" in config.parameters:
             agentic_config["eval_model_params"] = config.parameters["eval_model_params"]

adapters/clear/meta/job.json

@@ -3,23 +3,30 @@
   "provider_id": "ibm-clear",
   "benchmark_id": "agentic-evaluation",
   "benchmark_index": 0,
+
   "experiment_name": "clear-agentic-eval-example",
+
   "model": {
     "url": "http://127.0.0.1:8000/v1",
     "name": "example-model",
     "auth": {
       "secret_ref": "your-model-api-key-secret"
     }
   },
+
   "parameters": {
-    "data_dir": "input-trace",
     "eval_model_name": "openai/example-model",
     "provider": "openai",
     "agent_framework": "langgraph",
     "observability_framework": "mlflow",
     "inference_backend": "litellm",
-    "clear_dashboard_theme": "red_hat"
+    "clear_dashboard_theme": "red_hat",
+
+    "mlflow_traces_experiment_name": "your-agent-experiment",
+
+    "separate_tools": false
   },
+
   "test_data_ref": {
     "s3": {
       "bucket": "clear-traces",
@@ -30,5 +37,6 @@
       }
     }
   },
+
   "callback_url": "http://localhost:8080"
 }

adapters/clear/requirements.txt

@@ -1,12 +1,6 @@
-eval-hub-sdk[adapter]==0.4.0
-# IBM CLEAR (agentic): PyPI `clear-eval` (e.g. 1.0.8) does not ship `clear_eval.agentic`—verified
-# the wheel has no agentic tree. Install from IBM/CLEAR until agentic is on PyPI.
-# Tag tracks a released version (bump tag when you need newer upstream fixes).
-# 2.x nests per-agent results under reasoning_eval / tools_eval — see main.py extraction.
-clear-eval @ git+https://github.com/IBM/CLEAR.git@2.0.0-rc.2
+eval-hub-sdk[adapter]>=0.4.1
+# IBM CLEAR with tool-call (SPARC) support. Set parameters.separate_tools: true to enable.
+clear_eval[tool-calls] @ git+https://github.com/IBM/clear.git@main_with_sparc
 
 pandas>=2.0.0
 pydantic>=2.0.0
-
-# Optional (local MLflow): if artifact upload needs the upstream Python client, set
-# EVALHUB_MLFLOW_BACKEND=upstream and: pip install mlflow-skinny