Skip to content

feat(clear): MLflow trace fetch, SPARC support, PyPI clear-eval #104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
williamcaban merged 1 commit into from
Sep 11, 2026
Merged

feat(clear): MLflow trace fetch, SPARC support, PyPI clear-eval #104

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion adapters/clear/Containerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,10 @@ FROM registry.access.redhat.com/ubi9/python-311:latest
WORKDIR /app

USER 0
# clear-eval is installed from git in requirements.txt; pip clone needs git in the image.
# clear-eval[tool-calls] from PyPI may pull compiled dependencies that need build tools.
# (ubi9/python-311 provides dnf; microdnf is not guaranteed.)
RUN dnf install -y git && dnf clean all

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

Expand Down
56 changes: 30 additions & 26 deletions adapters/clear/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ This directory is the **eval-hub community adapter** for **[IBM CLEAR](https://g

| Requirement | Notes |
|-------------|-------|
| **Python 3.13+** | Create a virtualenv from `adapters/clear`: `python3 -m venv .venv && pip install -r requirements.txt` |
| **IBM CLEAR** | Installed from Git by `requirements.txt` (currently **`2.0.0-rc.2`**); PyPI wheels omit the agentic pipeline |
| **Python 3.11+** | Create a virtualenv from `adapters/clear`: `python3 -m venv .venv && pip install -r requirements.txt` |
| **IBM CLEAR** | Installed via `clear-eval[tool-calls]>=2.0.0` from PyPI (includes agentic pipeline and SPARC/tool-call analysis) |
| **OpenAI-compatible inference endpoint** | Used as the **judge model** (`model.url`, e.g. an Ollama, vLLM, or OpenAI-API-compatible server). Supply `OPENAI_API_KEY` only when your endpoint requires it |
| **Agent traces** | One or more `*.json` trace files from a **LangGraph + MLflow-autologged** agent run — see [`examples/docs/02-agent-traces.md`](examples/docs/02-agent-traces.md) |
| **Eval Hub** *(deployed only)* | A running Eval Hub instance with the `ibm-clear` provider registered and traces staged from S3 — see [`examples/docs/04-deployed-eval-hub.md`](examples/docs/04-deployed-eval-hub.md) |
Expand Down Expand Up @@ -68,15 +68,19 @@ Summary:
"experiment_name": "my-mlflow-experiment",
"model": {
"url": "https://your-inference-endpoint.example.com/v1",
"name": "your-model-name"
"name": "your-model-name",
"auth": {
"secret_ref": "judge-llm-api-key"
}
},
"parameters": {
"data_dir": "input-trace",
"eval_model_name": "openai/your-model-name",
"eval_model_name": "your-model-name",
"provider": "openai",
"agent_framework": "langgraph",
"observability_framework": "mlflow",
"inference_backend": "litellm"
"inference_backend": "litellm",
"mlflow_traces_experiment_name": "your-agent-traces-experiment",
"separate_tools": false
},
"callback_url": "http://localhost:8080"
}
Expand Down Expand Up @@ -105,41 +109,33 @@ Example (dummy host, token, and secret names):
```bash
curl -sS -X POST 'https://evalhub.example.com/api/v1/evaluations/jobs' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer eyJhbGciOiJub3QtYS1yZWFsLXRva2Vu' \
-H 'Authorization: Bearer <YOUR_TOKEN>' \
-H 'X-Tenant: your-namespace' \
-d '{
"experiment_name": "clear-agentic-eval-example",
"name": "clear-eval-run",
"experiment_name": "clear-agentic-eval-results",
"model": {
"url": "http://127.0.0.1:8000/v1",
"name": "example-model",
"url": "https://generativelanguage.googleapis.com/v1beta/openai",
"name": "gemini-2.5-flash",
"auth": {
"secret_ref": "my-openai-api-key-secret"
"secret_ref": "judge-llm-api-key"
}
},
"benchmarks": [
{
"id": "agentic-evaluation",
"provider_id": "ibm-clear",
"parameters": {
"data_dir": "input-trace",
"eval_model_name": "openai/example-model",
"eval_model_name": "gemini-2.5-flash",
"provider": "openai",
"agent_framework": "langgraph",
"observability_framework": "mlflow",
"inference_backend": "litellm"
}
}
],
"test_data_ref": {
"s3": {
"bucket": "clear-traces",
"path": "traces/",
"secret_ref": {
"name": "clear-traces-bucket",
"namespace": "your-namespace"
"inference_backend": "litellm",
"mlflow_traces_experiment_name": "your-agent-traces-experiment",
"separate_tools": false
}
}
}
]
}'
```

Expand All @@ -154,7 +150,7 @@ podman build -f Containerfile -t quay.io/evalhub/community-ibm-clear:latest .

From repo root: `podman build -f eval-hub-contrib/adapters/clear/Containerfile -t quay.io/evalhub/community-ibm-clear:latest eval-hub-contrib/adapters/clear`

Bump the CLEAR archive URL in `requirements.txt` when you adopt a newer IBM/CLEAR revision. Pin the container image tag in your eval-hub provider definition when you want immutable pulls.
Bump the `clear-eval` version in `requirements.txt` when you adopt a newer IBM CLEAR release. Pin the container image tag in your eval-hub provider definition when you want immutable pulls.

## `parameters.inference_backend` and credentials

Expand All @@ -170,6 +166,14 @@ Prefer **`model.auth.secret_ref`** on the job so credentials stay in a **Kuberne

Configure **`MLFLOW_TRACKING_URI`** (and any other MLflow env) on the runtime. On the job, set **`experiment_name`** or **`parameters.mlflow_experiment_name`**. When set, the adapter uploads **`clear_results.json`**, **`metrics_summary.json`**, and HTML artifacts when present. **Local runs use the same rules** as cluster runs: upload is skipped if there is no experiment name or **`MLFLOW_TRACKING_URI`** is unset. Longer notes: [`examples/docs/03-local-run.md`](examples/docs/03-local-run.md).

**Fetching traces from MLflow:** Set **`parameters.mlflow_traces_experiment_name`** to the MLflow experiment containing your agent traces. The adapter uses the `eval-hub-sdk` MLflow client to fetch and convert traces into the format CLEAR expects. Optionally limit with **`parameters.mlflow_traces_max_results`**.

**Workspace override (OpenShift AI):** When the EvalHub sidecar sets `MLFLOW_WORKSPACE` to the tenant namespace but traces live in a different workspace, set **`parameters.mlflow_workspace`** to override it. The adapter will bypass the sidecar proxy and talk directly to the MLflow service. Optionally set **`parameters.mlflow_route_url`** to specify the MLflow route explicitly.

## SPARC (Tool-Call Analysis)

Set **`parameters.separate_tools: true`** to enable SPARC mode. In this mode, CLEAR separates tool-call spans from the main LLM reasoning and evaluates tool usage quality independently—covering issues like unnecessary tool calls, incorrect parameters, and missing tool invocations. When `false` (default), tool calls are analyzed inline as part of the standard step-by-step evaluation.

## Layout

| Path | Purpose |
Expand Down
117 changes: 107 additions & 10 deletions adapters/clear/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@

from evalhub.adapter import (
DefaultCallbacks,
ErrorInfo,
EvaluationResult,
FrameworkAdapter,
JobCallbacks,
Expand Down Expand Up @@ -51,7 +52,7 @@
except ImportError as exc:
raise ImportError(
"Install IBM CLEAR with tool-calls support (see requirements.txt): "
"pip install 'clear_eval[tool-calls] @ git+https://github.com/IBM/clear.git@main_with_sparc'"
"pip install 'clear_eval[tool-calls] @ git+https://github.com/IBM/clear.git@main'"
) from exc

logger = logging.getLogger(__name__)
Expand Down Expand Up @@ -382,6 +383,21 @@ def _fetch_mlflow_traces_to_dir(mlflow_client: EvalHubMlflowClient, parameters:
start_ns = s.get("start_time_unix_nano", 0)
end_ns = s.get("end_time_unix_nano", 0)
duration_ms = (end_ns - start_ns) / 1_000_000 if end_ns and start_ns else None

# The SDK's _normalize_spans pops mlflow.spanType/Inputs/Outputs from
# attributes into top-level fields. Re-inject them so CLEAR's
# preprocessor can identify LLM calls and extract I/O.
span_type = s.get("span_type") or attrs.get("mlflow.spanType", "UNKNOWN")
inputs = s.get("inputs") or attrs.get("mlflow.spanInputs", {})
outputs = s.get("outputs") or attrs.get("mlflow.spanOutputs", {})

if "mlflow.spanType" not in attrs:
attrs["mlflow.spanType"] = span_type
if "mlflow.spanInputs" not in attrs:
attrs["mlflow.spanInputs"] = inputs
if "mlflow.spanOutputs" not in attrs:
attrs["mlflow.spanOutputs"] = outputs

spans.append({
"span_id": s.get("span_id", ""),
"parent_span_id": s.get("parent_span_id"),
Expand All @@ -391,6 +407,8 @@ def _fetch_mlflow_traces_to_dir(mlflow_client: EvalHubMlflowClient, parameters:
"end_time_unix_nano": end_ns,
"duration_ms": duration_ms,
"attributes": attrs,
"inputs": inputs,
"outputs": outputs,
})

trace_doc = {
Expand Down Expand Up @@ -425,14 +443,37 @@ def run_benchmark_job(self, config: JobSpec, callbacks: JobCallbacks) -> JobResu

try:
callbacks.report_status(
JobStatusUpdate(status=JobStatus.RUNNING, phase=JobPhase.INITIALIZING)
JobStatusUpdate(
status=JobStatus.RUNNING,
phase=JobPhase.INITIALIZING,
progress=0.0,
message=MessageInfo(
message="Initializing CLEAR for agentic evaluation",
message_code="initializing",
),
)
)

self._validate_config(config)

callbacks.report_status(
JobStatusUpdate(status=JobStatus.RUNNING, phase=JobPhase.LOADING_DATA)
)
# Allow job to override the MLflow workspace (e.g. when EvalHub sidecar

@ruivieira ruivieira Aug 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One concern on the mlflow_workspace / mlflow_route_url bypass: allowing the adapter to override the sidecar-injected workspace and talk directly to MLflow breaks the multi-tenancy contract.

The sidecar exists precisely to ensure jobs can only access resources within their own tenant namespace. Would it be acceptable to restrict MLflow trace fetching to the same workspace as the job? That would keep the adapter within the platform's tenancy boundaries and avoid the need for the bypass entirely.

@suppathak suppathak Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ruivieira .

Here is why we did this: when we deployed MLflow on Red Hat OpenShift AI, the EvalHub sidecar automatically injects MLFLOW_WORKSPACE set to the tenant namespace (in our case redhat-ods-applications). The problem is, MLflow doesn't have a workspace matching that namespace name. There are multiple workspaces available, and we uploaded our traces to the default workspace, but there's just no workspace called redhat-ods-applications.

So when the adapter runs and tries to fetch traces from MLflow, it's looking in a workspace that doesn't exist, and it comes back empty. The traces are there in default, but the sidecar is forcing all requests to go to redhat-ods-applications which has nothing.

The override was the workaround we found to make the end-to-end flow work, telling the adapter to use default instead and bypassing the sidecar's workspace injection.

Happy to hear if there's a better way to handle this on the platform side. for example, having the sidecar pick up the workspace from a job parameter rather than always defaulting to the namespace name.

@suppathak suppathak Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ruivieira, let me know if you have any thought on this comment above. Thank you.

# sets it to the tenant namespace but traces live in a different workspace).
# When overriding, bypass the sidecar proxy (which injects the tenant workspace
# header) and use the external MLflow route directly.
ws = config.parameters.get("mlflow_workspace")
if ws:
os.environ["MLFLOW_WORKSPACE"] = ws
sidecar_mlflow_uri = os.environ.get("MLFLOW_TRACKING_URI", "")
if sidecar_mlflow_uri.startswith("http://localhost"):
mlflow_route = config.parameters.get("mlflow_route_url")
if not mlflow_route:
ns = os.environ.get("POD_NAMESPACE", "redhat-ods-applications")
direct_uri = f"https://mlflow.{ns}.svc:8443"
else:
direct_uri = mlflow_route.rstrip("/")
os.environ["MLFLOW_TRACKING_URI"] = direct_uri
os.environ["MLFLOW_TRACKING_INSECURE_TLS"] = "true"
logger.info("Bypassing sidecar proxy for MLflow workspace=%s uri=%s", ws, direct_uri)

data_dir: str | None = None
test_data_path = Path("/test_data")
Expand Down Expand Up @@ -473,6 +514,17 @@ def run_benchmark_job(self, config: JobSpec, callbacks: JobCallbacks) -> JobResu
if not data_dir and TracesNamespace.is_source_configured(config.parameters):
if os.environ.get("MLFLOW_TRACKING_URI"):
logger.info("Fetching traces from MLflow experiment via SDK")
callbacks.report_status(
JobStatusUpdate(
status=JobStatus.RUNNING,
phase=JobPhase.LOADING_DATA,
progress=0.1,
message=MessageInfo(
message="Fetching traces from MLflow",
message_code="fetching_mlflow_traces",
),
)
)
mlflow_traces_dir = tempfile.mkdtemp(prefix="clear_mlflow_traces_")
_fetch_mlflow_traces_to_dir(self.mlflow, config.parameters, mlflow_traces_dir)
data_dir = mlflow_traces_dir
Expand Down Expand Up @@ -519,6 +571,18 @@ def run_benchmark_job(self, config: JobSpec, callbacks: JobCallbacks) -> JobResu

logger.info("Found %d trace file(s) in %s", len(trace_files), data_dir)

callbacks.report_status(
JobStatusUpdate(
status=JobStatus.RUNNING,
phase=JobPhase.LOADING_DATA,
progress=0.2,
message=MessageInfo(
message="Processing MLflow traces",
message_code="loading_data",
),
)
)

results_dir_param = config.parameters.get("results_dir")
run_name_param = config.parameters.get("run_name")
if results_dir_param and run_name_param:
Expand All @@ -532,7 +596,15 @@ def run_benchmark_job(self, config: JobSpec, callbacks: JobCallbacks) -> JobResu
logger.info("CLEAR output base: %s", output_dir)

callbacks.report_status(
JobStatusUpdate(status=JobStatus.RUNNING, phase=JobPhase.RUNNING_EVALUATION)
JobStatusUpdate(
status=JobStatus.RUNNING,
phase=JobPhase.RUNNING_EVALUATION,
progress=0.4,
message=MessageInfo(
message="Running CLEAR agentic pipeline",
message_code="running_evaluation",
),
)
)

if config.model.url and config.parameters.get("inference_backend") != "endpoint":
Expand All @@ -552,7 +624,15 @@ def run_benchmark_job(self, config: JobSpec, callbacks: JobCallbacks) -> JobResu
logger.info("Results file: %s", json_results_path)

callbacks.report_status(
JobStatusUpdate(status=JobStatus.RUNNING, phase=JobPhase.POST_PROCESSING)
JobStatusUpdate(
status=JobStatus.RUNNING,
phase=JobPhase.POST_PROCESSING,
progress=0.8,
message=MessageInfo(
message="Processing CLEAR results",
message_code="post_processing",
),
)
)

evaluation_results = self._extract_agentic_results(str(json_results_path))
Expand Down Expand Up @@ -608,13 +688,22 @@ def run_benchmark_job(self, config: JobSpec, callbacks: JobCallbacks) -> JobResu

except Exception as exc:
logger.exception("CLEAR evaluation failed")
error_msg = str(exc)
callbacks.report_status(
JobStatusUpdate(
status=JobStatus.FAILED,
error_message=MessageInfo(
message=str(exc),
message=MessageInfo(
message=error_msg,
message_code="failed",
),
error=ErrorInfo(
message=error_msg,
message_code="evaluation_error",
),
error_details={
"exception_type": type(exc).__name__,
"benchmark_id": config.benchmark_id,
},
)
)
raise
Expand Down Expand Up @@ -894,7 +983,15 @@ def _report_artifacts(
return None

callbacks.report_status(
JobStatusUpdate(status=JobStatus.RUNNING, phase=JobPhase.PERSISTING_ARTIFACTS)
JobStatusUpdate(
status=JobStatus.RUNNING,
phase=JobPhase.PERSISTING_ARTIFACTS,
progress=0.9,
message=MessageInfo(
message="Persisting CLEAR artifacts",
message_code="persisting_artifacts",
),
)
)

if self.local_jobs_base_path is not None:
Expand Down
30 changes: 11 additions & 19 deletions adapters/clear/meta/job.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,41 +1,33 @@
{
"id": "clear-agentic-test-001",
"id": "clear-agentic-eval-001",
"provider_id": "ibm-clear",
"benchmark_id": "agentic-evaluation",
"benchmark_index": 0,

"experiment_name": "clear-agentic-eval-example",
"experiment_name": "clear-agentic-eval-results",

"model": {
"url": "http://127.0.0.1:8000/v1",
"name": "example-model",
"url": "https://generativelanguage.googleapis.com/v1beta/openai",
"name": "gemini-2.5-flash",
"auth": {
"secret_ref": "your-model-api-key-secret"
"secret_ref": "judge-llm-api-key"
}
},

"parameters": {
"eval_model_name": "openai/example-model",
"eval_model_name": "gemini-2.5-flash",
"provider": "openai",
"agent_framework": "langgraph",
"observability_framework": "mlflow",
"inference_backend": "litellm",
"clear_dashboard_theme": "red_hat",

"mlflow_traces_experiment_name": "your-agent-experiment",
"mlflow_traces_experiment_name": "your-agent-traces-experiment",
"mlflow_traces_max_results": 10,

"separate_tools": false
},
"separate_tools": false,

"test_data_ref": {
"s3": {
"bucket": "clear-traces",
"path": "traces/",
"secret_ref": {
"name": "clear-traces-bucket",
"namespace": "your-namespace"
}
}
"clear_dashboard_theme": "red_hat",
"overwrite": true
},

"callback_url": "http://localhost:8080"
Expand Down
6 changes: 2 additions & 4 deletions adapters/clear/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
eval-hub-sdk[adapter]>=1.0.2
# IBM CLEAR with tool-call (SPARC) support. Set parameters.separate_tools: true to enable.
clear_eval[tool-calls] @ git+https://github.com/IBM/clear.git@main_with_sparc

eval-hub-sdk[adapter]>=0.4.1
clear-eval[tool-calls]>=2.0.0
pandas>=2.0.0
pydantic>=2.0.0
Loading