A North Star for open AGI. Co-founders: @fchollet @mikeknoop. President: @gregkamradt. We're hiring mission-driven builders: https://xitter.cf/t.co/GswTSnyCoJ
Earth
Joined March 2024
- Tweets1K
- Following187
- Followers43.2K
- Likes4.1K
Claude Sonnet 5.5 from @AnthropicAI on ARC-AGI (Verified):
- ARC-AGI-3: 72.4%, $8.2K (provider adapter harness), 24.3%, $7.4K (standard harness)
- ARC-AGI-2: 83.9%, $0.55/task
- ARC-AGI-1: 96.5%, $0.14/task
Sonnet 5.5 is notable for its 72.4% score on ARC-AGI-3.
While testing Sonnet 5.5 with max reasoning, we frequently saw it exhaust its entire output token budget on thinking without returning a response. For that reason, we excluded the max partial scores from the ARC-AGI-2 and ARC-AGI-3 reporting.
Note that we were not able to test Opus 5.5 on ARC-AGI-3 due to frequent API refusals during early access.
Full results: arcprize.org/results/anthrop…
- Leaderboard: arcprize.org/leaderboard
- Reproduce the public results: github.com/arcprize/arc-agi-…
- Testing policy: arcprize.org/policy
- Full Claude Sonnet 5.5 results: arcprize.org/results/anthrop…
Beam 1.0 from @reflection_ai on ARC-AGI (Verified):
- ARC-AGI-2: 23.5%, $0.61/task
- ARC-AGI-1: 74.0%, $0.31/task
Beam’s best scores came at xhigh reasoning, but high was close on ARC-AGI-2: 20.0% versus 23.5%, at about 43% lower cost.
Cost per task is based on the @Baseten pricing we used for testing: $1.30 input / $4.30 output per 1M tokens.
ARC-AGI-3 results are pending.
Full results: arcprize.org/results/reflect…
- Leaderboard: arcprize.org/leaderboard
- Reproduce the public results: github.com/arcprize/arc-agi-…
- Testing policy: arcprize.org/policy
- Full Beam 1.0 results: arcprize.org/results/reflect…
Gemini 4 Argon from @Google on ARC-AGI (Verified):
- ARC-AGI-3: 97.5%, $4.8K (provider adapter harness), 49.8%, $5.8K (standard harness)
- ARC-AGI-2: 91.3%, $0.32/task
- ARC-AGI-1: 98.5%, $0.16/task
Google DeepMind is the second frontier AI lab to score above 95% on ARC-AGI-3.
In addition to its strong performance on ARC-AGI-3 using the provider adapter harness (which preserves opaque reasoning and enables automatic compaction), Argon is also notable for its high scores using the standard harness (which enables the model to carry forward notes between turns).
However, we encountered API issues that prevented us from completing testing with the standard harness:
- Low reasoning: 49.8%, with runs completed on 42 of 55 games
- Medium reasoning: 46.3%, with runs completed on 46 of 55 games
- High reasoning: Runs completed on only 19 of 55 games, so we did not publish a score
Reported scores include all 55 games, retaining partial credit earned on incomplete runs.
We used the current introductory pricing of $2 input / $10 output per million tokens to calculate the costs.
Full results: arcprize.org/results/google-…
- Leaderboard: arcprize.org/leaderboard
- Reproduce the public results: github.com/arcprize/arc-agi-…
- Testing policy: arcprize.org/policy
- Full Gemini 4 Argon results: arcprize.org/results/google-…
Haiku 5.5 from @AnthropicAI on ARC-AGI (Verified):
- ARC-AGI-3: 2.8%, $437 (provider adapter harness); 0.5%, $388 (standard harness)
- ARC-AGI-2: 61.3%, $0.085/task
- ARC-AGI-1: 89.0%, $0.051/task
Haiku 5.5 delivers competitive performance for Anthropic's small-model tier.
On ARC-AGI-2, we omitted max reasoning because it completed only 74 of 120 tasks, with many incomplete attempts failing to return parseable answers before exhausting the output budget.
Full results: arcprize.org/results/anthrop…
- Leaderboard: arcprize.org/leaderboard
- Reproduce the public results: github.com/arcprize/arc-agi-…
- Testing policy: arcprize.org/policy
- Full Haiku 5.5 results: arcprize.org/results/anthrop…
DeepSeek V4.1 Flash from @deepseek_ai on ARC-AGI (Verified):
- ARC-AGI-3: 3.3%, $878 (provider adapter harness), 0.9%, $1.2k (standard harness)
DeepSeek V4.1 Flash is the first model we've evaluated on ARC-AGI-3 to score above 3% for under $1k in total evaluation cost.
Here's our earlier thread about DeepSeek V4.1 Flash's scores on ARC-AGI-1 and ARC-AGI-2:
xitter.cf/arcprize/status/210749…
DeepSeek V4.1 Flash from @deepseek_ai on ARC-AGI (Verified):
- ARC-AGI-2: 72.9%, $0.13/task
- ARC-AGI-1: 94.5%, $0.07/task
DeepSeek V4.1 Flash beats V4 Flash's best scores by 5.5 points on ARC-AGI-1 and 11.5 on ARC-AGI-2, but costs about 250% more per task.
- Leaderboard: arcprize.org/leaderboard
- Reproduce the public results: github.com/arcprize/arc-agi-…
- Testing policy: arcprize.org/policy
- Full DeepSeek V4.1 Flash results: arcprize.org/results/deepsee…
