Skip to content

Add ClawBench evaluation framework - #2315

Open
reacher-z wants to merge 1 commit into
huggingface:mainfrom
reacher-z:add-clawbench-eval-framework
Open

Add ClawBench evaluation framework#2315
reacher-z wants to merge 1 commit into
huggingface:mainfrom
reacher-z:add-clawbench-eval-framework

Conversation

@reacher-z

@reacher-z reacher-z commented Jul 26, 2026

Copy link
Copy Markdown

Summary

  • register clawbench-eval as a supported benchmark evaluation framework
  • link the canonical ClawBench repository

The ClawBench benchmark dataset already declares this identifier in its root eval.yaml; adding it here satisfies the documented framework-enum requirement for native Hub benchmark registration.

Validation

  • pnpm --filter @huggingface/tasks format:check
  • pnpm --filter @huggingface/tasks check

Disclosure

I help maintain ClawBench. This focused contribution was prepared with assistance from OpenAI Codex.


Note

Low Risk
Single registry entry with no runtime or security impact; only extends the allowed framework list for eval.yaml.

Overview
Registers clawbench-eval in the EVALUATION_FRAMEWORKS registry in eval.ts, alongside existing frameworks like claw-eval and researchclawbench. The entry includes a short description of ClawBench (web-agent evaluation with isolated runs and behavioral traces) and links to https://github.com/TIGER-AI-Lab/ClawBench.

This aligns the tasks package with benchmark datasets whose root eval.yaml already reference this framework identifier for native Hub registration.

Reviewed by Cursor Bugbot for commit 87e3439. Bugbot is set up for automated code reviews on this repo. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant