Skip to content

Commit 51a852f

Browse files
mtodorgithub-actions[bot]
authored andcommitted
Update model evaluations 2026-09-01
1 parent 2ef4fc3 commit 51a852f

1 file changed

Lines changed: 14 additions & 14 deletions

File tree

‎docs/model-evaluation.md‎

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -39,27 +39,27 @@ A task passes when **all** its assertions pass **and** the LLM judge approves th
3939

4040
<!-- model:gpt-5-mini start -->
4141

42-
### gpt-5-mini — 2026-08-25
42+
### gpt-5-mini — 2026-09-01
4343

4444
**Overall: 11/11 tasks passed (100%)**
4545

4646
#### Task Results
4747

4848
| # | Task | Result | toolsUsed | minCalls | maxCalls | Input Tokens | Output Tokens |
4949
|---|------|--------|-----------|----------|----------|--------------|---------------|
50-
| 1 | cve-cluster-does-exist | Pass | Pass | Pass | Pass | 1531 | 1083 |
51-
| 2 | cve-detected-workloads | Pass | Pass | Pass | Pass | 2581 | 1093 |
52-
| 3 | rhsa-not-supported | Pass | — | Pass | Pass | 1642 | 2211 |
53-
| 4 | cve-detected-clusters | Pass | Pass | Pass | Pass | 703 | 1874 |
54-
| 5 | cve-clusters-general | Pass | Pass | Pass | Pass | 764 | 1841 |
55-
| 6 | cve-cluster-does-not-exist | Pass | Pass | Pass | Pass | 472 | 1460 |
56-
| 7 | cve-cluster-list | Pass | Pass | Pass | Pass | 456 | 1480 |
57-
| 8 | cve-nonexistent | Pass | Pass | Pass | Pass | 1029 | 2311 |
58-
| 9 | cve-log4shell | Pass | Pass | Pass | Pass | 2000 | 2242 |
59-
| 10 | cve-multiple | Pass | Pass | Pass | Pass | 1110 | 3559 |
60-
| 11 | list-clusters | Pass | Pass | Pass | Pass | 672 | 815 |
61-
62-
**Total input tokens**: 12960 | **Total output tokens**: 19969
50+
| 1 | cve-cluster-does-exist | Pass | Pass | Pass | Pass | 1531 | 1037 |
51+
| 2 | cve-detected-clusters | Pass | Pass | Pass | Pass | 489 | 1251 |
52+
| 3 | list-clusters | Pass | Pass | Pass | Pass | 1700 | 943 |
53+
| 4 | cve-cluster-list | Pass | Pass | Pass | Pass | 1480 | 1063 |
54+
| 5 | cve-log4shell | Pass | Pass | Pass | Pass | 976 | 2098 |
55+
| 6 | rhsa-not-supported | Pass | — | Pass | Pass | 786 | 1806 |
56+
| 7 | cve-clusters-general | Pass | Pass | Pass | Pass | 1982 | 3103 |
57+
| 8 | cve-cluster-does-not-exist | Pass | Pass | Pass | Pass | 472 | 1725 |
58+
| 9 | cve-nonexistent | Pass | Pass | Pass | Pass | 561 | 1704 |
59+
| 10 | cve-multiple | Pass | Pass | Pass | Pass | 1110 | 2880 |
60+
| 11 | cve-detected-workloads | Pass | Pass | Pass | Pass | 533 | 1054 |
61+
62+
**Total input tokens**: 11620 | **Total output tokens**: 18664
6363

6464
<!-- model:gpt-5-mini end -->
6565

0 commit comments

Comments
 (0)