Skip to content

🐛 analyze-run.sh backtraces #68

Description

@esnible

Describe the bug

When I run analyze-run.sh I see a backtrace.

Steps To Reproduce

  1. env OPENAI_API_KEY=${SNIBLE_LITELLM_TOKEN} OPENAI_API_BASE=https://ete-litellm.ai-models.vpc.res.ibm.com ./deploy-and-evaluate.sh --benchmark gsm8k --agent tool_calling
  2. analyze-run.sh

It is failing for me with this message:

=== MLflow Trace Analysis ===
Cluster mode: kind
MLflow URL: http://localhost:8080 (via port-forward svc/mlflow)
Experiment ID: 0
Auth mode: secret
Window: 3h

Current kubectl context: kind-rossoctl
Kind cluster verified (context: kind-rossoctl)

Starting port-forward for MLflow (rossoctl-system/svc/mlflow:5000 -> localhost:8080)...
Checking if MLflow pod is ready...
✓ MLflow port-forward established (PID: 46586)

Connecting to MLflow...
✓ Connected to MLflow

Obtaining OAuth token via mlflow-oauth-secret...
✓ OAuth token obtained

Fetching traces from MLflow (experiment_id=0, window=3h)...
Traceback (most recent call last):
  File "/Users/snible/src/workload-harness/exgentic_a2a_runner/download_mlflow_traces.py", line 283, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/snible/src/workload-harness/exgentic_a2a_runner/download_mlflow_traces.py", line 209, in main
    all_traces = fetch_trace_list()
                 ^^^^^^^^^^^^^^^^^^
  File "/Users/snible/src/workload-harness/exgentic_a2a_runner/download_mlflow_traces.py", line 100, in fetch_trace_list
    response = mlflow_get(url)
               ^^^^^^^^^^^^^^^
  File "/Users/snible/src/workload-harness/exgentic_a2a_runner/download_mlflow_traces.py", line 64, in mlflow_get
    with urllib.request.urlopen(req, timeout=30, context=_SSL_CONTEXT) as resp:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/snible/miniforge3/lib/python3.12/urllib/request.py", line 215, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/snible/miniforge3/lib/python3.12/urllib/request.py", line 521, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/snible/miniforge3/lib/python3.12/urllib/request.py", line 630, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/Users/snible/miniforge3/lib/python3.12/urllib/request.py", line 559, in error
    return self._call_chain(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/snible/miniforge3/lib/python3.12/urllib/request.py", line 492, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/Users/snible/miniforge3/lib/python3.12/urllib/request.py", line 639, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Unauthorized
Traceback (most recent call last):
  File "/Users/snible/src/workload-harness/exgentic_a2a_runner/analyze_traces.py", line 773, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/snible/src/workload-harness/exgentic_a2a_runner/analyze_traces.py", line 760, in main
    raw = json.load(sys.stdin)
          ^^^^^^^^^^^^^^^^^^^^
  File "/Users/snible/miniforge3/lib/python3.12/json/__init__.py", line 293, in load
    return loads(fp.read(),
           ^^^^^^^^^^^^^^^^
  File "/Users/snible/miniforge3/lib/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/snible/miniforge3/lib/python3.12/json/decoder.py", line 338, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/snible/miniforge3/lib/python3.12/json/decoder.py", line 356, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Expected Behavior

The script should succeed or print an understandable error message.

Rossoctl Version

v0.7.0

Additional Context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugSomething isn't working

    Type

    No type

    Projects

    Status
    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions