=== MLflow Trace Analysis ===
Cluster mode: kind
MLflow URL: http://localhost:8080 (via port-forward svc/mlflow)
Experiment ID: 0
Auth mode: secret
Window: 3h
Current kubectl context: kind-rossoctl
Kind cluster verified (context: kind-rossoctl)
Starting port-forward for MLflow (rossoctl-system/svc/mlflow:5000 -> localhost:8080)...
Checking if MLflow pod is ready...
✓ MLflow port-forward established (PID: 46586)
Connecting to MLflow...
✓ Connected to MLflow
Obtaining OAuth token via mlflow-oauth-secret...
✓ OAuth token obtained
Fetching traces from MLflow (experiment_id=0, window=3h)...
Traceback (most recent call last):
File "/Users/snible/src/workload-harness/exgentic_a2a_runner/download_mlflow_traces.py", line 283, in <module>
sys.exit(main())
^^^^^^
File "/Users/snible/src/workload-harness/exgentic_a2a_runner/download_mlflow_traces.py", line 209, in main
all_traces = fetch_trace_list()
^^^^^^^^^^^^^^^^^^
File "/Users/snible/src/workload-harness/exgentic_a2a_runner/download_mlflow_traces.py", line 100, in fetch_trace_list
response = mlflow_get(url)
^^^^^^^^^^^^^^^
File "/Users/snible/src/workload-harness/exgentic_a2a_runner/download_mlflow_traces.py", line 64, in mlflow_get
with urllib.request.urlopen(req, timeout=30, context=_SSL_CONTEXT) as resp:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/snible/miniforge3/lib/python3.12/urllib/request.py", line 215, in urlopen
return opener.open(url, data, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/snible/miniforge3/lib/python3.12/urllib/request.py", line 521, in open
response = meth(req, response)
^^^^^^^^^^^^^^^^^^^
File "/Users/snible/miniforge3/lib/python3.12/urllib/request.py", line 630, in http_response
response = self.parent.error(
^^^^^^^^^^^^^^^^^^
File "/Users/snible/miniforge3/lib/python3.12/urllib/request.py", line 559, in error
return self._call_chain(*args)
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/snible/miniforge3/lib/python3.12/urllib/request.py", line 492, in _call_chain
result = func(*args)
^^^^^^^^^^^
File "/Users/snible/miniforge3/lib/python3.12/urllib/request.py", line 639, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Unauthorized
Traceback (most recent call last):
File "/Users/snible/src/workload-harness/exgentic_a2a_runner/analyze_traces.py", line 773, in <module>
sys.exit(main())
^^^^^^
File "/Users/snible/src/workload-harness/exgentic_a2a_runner/analyze_traces.py", line 760, in main
raw = json.load(sys.stdin)
^^^^^^^^^^^^^^^^^^^^
File "/Users/snible/miniforge3/lib/python3.12/json/__init__.py", line 293, in load
return loads(fp.read(),
^^^^^^^^^^^^^^^^
File "/Users/snible/miniforge3/lib/python3.12/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/snible/miniforge3/lib/python3.12/json/decoder.py", line 338, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/snible/miniforge3/lib/python3.12/json/decoder.py", line 356, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The script should succeed or print an understandable error message.
Describe the bug
When I run
analyze-run.shI see a backtrace.Steps To Reproduce
env OPENAI_API_KEY=${SNIBLE_LITELLM_TOKEN} OPENAI_API_BASE=https://ete-litellm.ai-models.vpc.res.ibm.com ./deploy-and-evaluate.sh --benchmark gsm8k --agent tool_callinganalyze-run.shIt is failing for me with this message:
Expected Behavior
The script should succeed or print an understandable error message.
Rossoctl Version
v0.7.0
Additional Context
No response