@@ -15,6 +15,8 @@ This table shows the percentage of immediate ("1️⃣") successes per task.
15
15
| :-----------:| :-----------:| :-----------:| :-----------:| :-----------:| :-----------:| :-----------:| :-----------:| :-----------:| :-----------:| :-----------:| :-----------:| :-----------:| :-----------:| :-----------:| :-----------:|
16
16
| claude-3.5-sonnet (anthropic/claude-3.5-sonnet) | 100.0% | 100.0% | 100.0% | 0.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | N/A | N/A | N/A |
17
17
| claude-3.7-sonnet (anthropic/claude-3.7-sonnet) | 100.0% | 100.0% | 100.0% | 0.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | N/A | N/A | N/A |
18
+ | claude-opus-4 (anthropic/claude-opus-4) | 100.0% | 40.0% | 100.0% | 90.0% | 100.0% | 100.0% | 100.0% | 60.0% | 70.0% | 50.0% | 60.0% | 100.0% | N/A | N/A | N/A |
19
+ | claude-sonnet-4 (anthropic/claude-sonnet-4) | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | N/A | N/A | N/A |
18
20
| deepseek-chat-v3-0324 (deepseek/deepseek-chat-v3-0324) | 100.0% | 30.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | N/A | N/A | N/A |
19
21
| deepseek-r1 (deepseek/deepseek-r1) | 70.0% | 0.0% | 20.0% | 40.0% | 30.0% | 40.0% | 60.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | N/A | N/A | N/A |
20
22
| gemini-2.0-flash-001 (google/gemini-2.0-flash-001) | N/A | N/A | 0.0% | 0.0% | 100.0% | 0.0% | 0.0% | N/A | N/A | N/A | N/A | N/A | 100.0% | 0.0% | 100.0% |
@@ -41,6 +43,8 @@ For each task the cell shows the percentage of trials (out of 10) that ultimatel
41
43
| :-----------:| :-----------:| :-----------:| :-----------:| :-----------:| :-----------:| :-----------:| :-----------:| :-----------:| :-----------:| :-----------:| :-----------:| :-----------:| :-----------:| :-----------:| :-----------:|
42
44
| claude-3.5-sonnet (anthropic/claude-3.5-sonnet) | 100.0% | 100.0% | 100.0% | 0.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | N/A | N/A | N/A |
43
45
| claude-3.7-sonnet (anthropic/claude-3.7-sonnet) | 100.0% | 100.0% | 100.0% | 0.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | N/A | N/A | N/A |
46
+ | claude-opus-4 (anthropic/claude-opus-4) | 100.0% | 40.0% | 100.0% | 90.0% | 100.0% | 100.0% | 100.0% | 60.0% | 70.0% | 50.0% | 60.0% | 100.0% | N/A | N/A | N/A |
47
+ | claude-sonnet-4 (anthropic/claude-sonnet-4) | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | N/A | N/A | N/A |
44
48
| deepseek-chat-v3-0324 (deepseek/deepseek-chat-v3-0324) | 100.0% | 30.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | N/A | N/A | N/A |
45
49
| deepseek-r1 (deepseek/deepseek-r1) | 70.0% | 0.0% | 20.0% | 40.0% | 30.0% | 40.0% | 60.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | N/A | N/A | N/A |
46
50
| gemini-2.0-flash-001 (google/gemini-2.0-flash-001) | N/A | N/A | 0.0% | 0.0% | 100.0% | 0.0% | 0.0% | N/A | N/A | N/A | N/A | N/A | 100.0% | 0.0% | 100.0% |
@@ -65,6 +69,8 @@ For each task the cell shows the percentage of trials (out of 10) that ultimatel
65
69
| -----------| :---------------------:| :------------------:| :---------------:| :-----------------:| :----------------:| :--------------:| :--------------:| :--------------:|
66
70
| claude-3.5-sonnet (anthropic/claude-3.5-sonnet) | 110 | 110 | 0 | 0.00% | 10 | 1.458 | 6 | 1 |
67
71
| claude-3.7-sonnet (anthropic/claude-3.7-sonnet) | 110 | 110 | 0 | 0.00% | 10 | 1.425 | 6 | 1 |
72
+ | claude-opus-4 (anthropic/claude-opus-4) | 97 | 97 | 0 | 0.00% | 23 | 1.050 | 2 | 1 |
73
+ | claude-sonnet-4 (anthropic/claude-sonnet-4) | 120 | 120 | 0 | 0.00% | 0 | 1.000 | 1 | 1 |
68
74
| deepseek-chat-v3-0324 (deepseek/deepseek-chat-v3-0324) | 113 | 113 | 0 | 0.00% | 7 | 1.175 | 5 | 1 |
69
75
| deepseek-r1 (deepseek/deepseek-r1) | 26 | 26 | 0 | 0.00% | 94 | 1.000 | 1 | 1 |
70
76
| gemini-2.0-flash-001 (google/gemini-2.0-flash-001) | 30 | 30 | 0 | 0.00% | 50 | 3.500 | 6 | 1 |
0 commit comments