Skip to content

Clarify code_eval language support #769

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pgpg-chokoro wants to merge 1 commit into
base: main
Choose a base branch
from
+9 −1
Open

Clarify code_eval language support #769

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion metrics/code_eval/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,14 @@ The Code Eval metric calculates how good are predictions given a set of referenc

`timeout`: The maximum time taken to produce a prediction before it is considered a "timeout". The default value is `3.0` (i.e. 3 seconds).

### Language support

`code_eval` executes candidate programs and reference tests with the Python HumanEval-style runner.

* `pass@k` itself is language-agnostic.
* The current `code_eval` implementation only supports Python out of the box.
* Evaluating other programming languages would require a language-specific execution runner, test harness, and appropriate sandboxing or isolation.

```python
from evaluate import load
code_eval = load("code_eval")
Expand All @@ -51,7 +59,7 @@ N.B.
This metric exists to run untrusted model-generated code. Users are strongly encouraged not to do so outside of a robust security sandbox. Before running this metric and once you've taken the necessary precautions, you will need to set the `HF_ALLOW_CODE_EVAL` environment variable. Use it at your own risk:
```python
import os
os.environ["HF_ALLOW_CODE_EVAL"] = "1"`
os.environ["HF_ALLOW_CODE_EVAL"] = "1"
```

## Output values
Expand Down