[scheruku-rh]

What and why

Implements SDK and CLI support for Hugging Face Hub test data, matching the server-side changes from https://github.com/eval-hub/eval-hub/pull/996/changes

• Add HFTestDataRef model (repo_id, revision, sub_path, secret_ref) to api.py
• Extend TestDataRef validator to accept exactly one of s3 / pvc / git / hf
• Export HFTestDataRef from models package
• Add --test-data-hf-repo-id / --test-data-hf-revision / --test-data-hf-sub-path / --test-data-hf-secret CLI flags to eval run
• Update validation to reject mixed sources across all four types
• Add CLI usage examples for HF source
• Add model, client submit, and CLI unit tests

Closes # https://redhat.atlassian.net/browse/RHAISTRAT-2059

Type

• feat
• fix
• docs
• refactor / chore
• test / ci

Testing

• Tests added or updated
• Tested manually

Testing:

HF_REPO_ID = "SobhaCh/evalhub-offline-testdata"
HF_REVISION = "main"
HF_TEST_DATA_SUB_PATH = "staging_sub_path"
HF_CREDS = "sobha-hf-secret"

job = client.jobs.submit(JobSubmissionRequest(
    name="quickstart-eval",
    model=ModelConfig(
        url="https://vllm-llama-prabhu.apps.rosa.prabhu-comhub.xqmp.p3.openshiftapps.com/v1",
        name="meta-llama/Llama-3.2-1B-Instruct",
        auth = ModelAuth( secret_ref = "vllm-api-key")
    ),
    benchmarks=[
        BenchmarkConfig(
            id="truthfulqa_mc1",
            provider_id="lm_evaluation_harness",
            parameters={
               "limit":5,
                "num_examples":5,
                "tokenizer":"/test_data/tokenizer"
            },
            test_data_ref=TestDataRef(
                hf=HFTestDataRef(
                    repo_id=HF_REPO_ID,
                    revision=HF_REVISION,
                    sub_path=HF_TEST_DATA_SUB_PATH,
                    secret_ref=HF_CREDS,
                ),
            ),
        )
    ]
))

Job completed successfully

2026-09-16 12:27:49,159 - lm_eval.evaluator - INFO - Setting random seed to 42 | Setting numpy seed to 42 | Setting torch manual seed to 42 | Setting fewshot manual seed to 1234
2026-09-16 12:27:49,159 - lm_eval.evaluator - INFO - Initializing local-completions model, with arguments: {'model': 'meta-llama/Llama-3.2-1B-Instruct', 'base_url': 'http://localhost:8080/v1/completions', 'tokenizer_backend': 'huggingface', 'tokenizer': '/test_data/tokenizer', 'tokenized_requests': False, 'num_concurrent': 1, 'batch_size': 1, 'timeout': 300}
2026-09-16 12:27:49,159 - lm_eval.models.api_models - INFO - Using max length 2048 - 1
2026-09-16 12:27:49,159 - lm_eval.models.api_models - INFO - Concurrent requests are disabled. To enable concurrent requests, set `num_concurrent` > 1.
2026-09-16 12:27:49,159 - lm_eval.models.api_models - INFO - Using tokenizer huggingface
2026-09-16 12:27:49,688 - lm_eval.api.task - INFO - Offline: loading DatasetDict bundle at /test_data/truthful_qa--multiple_choice
2026-09-16 12:27:49,780 - lm_eval.evaluator - INFO - num_fewshot has been set to 0 for truthfulqa_mc1 in its config. Manual configuration will be ignored.
2026-09-16 12:27:49,780 - lm_eval.api.task - INFO - Building contexts for truthfulqa_mc1 on rank 0...
100%|██████████| 5/5 [00:00<00:00, 995.33it/s]
2026-09-16 12:27:49,786 - lm_eval.evaluator - INFO - Running loglikelihood requests
Requesting API: 100%|██████████| 24/24 [06:48<00:00, 17.00s/it]
fatal: not a git repository (or any of the parent directories): .git
2026-09-16 12:34:38,790 - httpx - INFO - HTTP Request: POST http://localhost:8080/api/v1/evaluations/jobs/0fa569db-ffd2-4f9b-965c-2374f24f50d3/events "HTTP/1.1 204 No Content"
2026-09-16 12:34:38,877 - httpx - INFO - HTTP Request: POST http://localhost:8080/api/v1/evaluations/jobs/0fa569db-ffd2-4f9b-965c-2374f24f50d3/events "HTTP/1.1 204 No Content"
2026-09-16 12:34:38,878 - __main__ - INFO - No OCI exports configured; skipping artifact persistence
2026-09-16 12:34:38,879 - __main__ - INFO - ================================================================================
2026-09-16 12:34:38,879 - __main__ - INFO - Evaluation completed successfully
2026-09-16 12:34:38,879 - __main__ - INFO - Overall score: 0.6
2026-09-16 12:34:38,879 - __main__ - INFO - Examples evaluated: 5
2026-09-16 12:34:38,879 - __main__ - INFO - Duration: 412.97s
2026-09-16 12:34:38,879 - __main__ - INFO - ================================================================================
2026-09-16 12:34:38,885 - evalhub.adapter.callbacks - INFO - Environment Card auto-captured (completeness: 15%)

Breaking changes

Summary by CodeRabbit

• New Features

  • Added Hugging Face Hub repositories as a test-data source for evaluation runs.
  • Added options for repository revision, subpath, and secret configuration.
  • Added validation to ensure valid repository details and prevent conflicting test-data sources.
  • Hugging Face test-data references are supported in benchmark configurations and submissions.

• Tests

  • Added coverage for CLI options, validation, serialization, and synchronous/asynchronous submissions.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 7fdab8c6-bbfe-4b11-a29b-a5ec8bb820e1

📥 Commits

Reviewing files that changed from the base of the PR and between 593ac9a and 63f4ce1.

📒 Files selected for processing (6)

• src/evalhub/cli/main.py
• src/evalhub/models/__init__.py
• src/evalhub/models/api.py
• tests/unit/test_cli_eval.py
• tests/unit/test_evalhub_client.py
• tests/unit/test_models_api.py

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The SDK adds Hugging Face Hub repositories as eval run test-data sources. It adds the HFTestDataRef model, CLI options and validation, public exports, submission handling, and unit coverage.

Changes

Hugging Face test data

Layer / File(s)	Summary
Hugging Face test-data model contract src/evalhub/models/api.py, src/evalhub/models/__init__.py, tests/unit/test_models_api.py	Adds HFTestDataRef with repository, revision, subpath, and secret fields. Adds blank repository validation, integrates the hf source into TestDataRef, exports the model, and tests serialization and source exclusivity.
CLI Hugging Face source integration src/evalhub/cli/main.py, tests/unit/test_cli_eval.py	Adds four CLI options and matching eval_run parameters. Validates repository-dependent options, enforces single-source selection, constructs HFTestDataRef, and tests the supported configurations and help output.
Hugging Face submission behavior tests/unit/test_evalhub_client.py	Tests synchronous and asynchronous submissions that preserve Hugging Face fields and omit resolved_sha.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Feature

Suggested reviewers: ruivieira

Sequence Diagram(s)

sequenceDiagram
  participant CLI as eval run
  participant Model as HFTestDataRef
  participant Client as EvalHub client
  CLI->>Model: Create repository reference
  Model-->>CLI: Validate hf test-data source
  CLI->>Client: Submit test-data reference
  Client-->>Client: Omit resolved_sha from submission

Loading

Merge Risk: ⚪ Minimal · up to 63f4c

The Hugging Face test-data feature is wired through the SDK and CLI with validation and unit coverage; no actionable merge risk is identified.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 7.69% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 26 functions across 6 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the main change: adding Hugging Face Hub test data source support to the SDK. The trailing issue marker is slightly awkward but does not obscure the change.
Description check	✅ Passed	The description covers the change, motivation, type, testing, related issue, implementation details, and manual test result. The Breaking changes section remains empty, but the description is otherwis…

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[scheruku-rh]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[julpayne]

LGTM