architect by day, tinkerer by night. ≥ https://teapawt.org/t.co/kmsJMKOyJH ≥ https://teapawt.org/t.co/v61NkbBarz ≥ https://teapawt.org/t.co/Zd5g7Oryh5
Spain
Joined July 2022
- Tweets836
- Following268
- Followers231
- Likes190
Pinned Tweet
Kimi K3.1 Pro joins FarmerBench 9th of 14: a mean of 48.1 over three seeds, between Claude Sonnet 5.5 (55.1) and DeepSeek V4.1 Flash (46.8).
FarmerBench gives a model six game-hours to build an economy in The Farmer Was Replaced, by writing code.
Gemini 4 Argon is currently the highest rated model on FarmerBench and I think had a really unfair reception here on X.
Super interested to see how Gemini 4 Carbon looks.
Introducing Gemini 4 Carbon, it's a big jump over Argon which we released just 2 weeks ago.
Gemini 4 Carbon is SOTA on cyber, long horizon tasks, ranks #1 on Artificial Analysis V5, significantly improves on usability in domains like coding, and is more aligned.
This is a "let's connect" post wrapped in a physics simulation so it doesn't feel like one.
Crossed 200 followers. Every coin is one of you. The jar holds 500.
I build things in public and occasionally finish them. If you do too, say hi.
How it's made: Three.js + Rapier physics, baked at 240Hz and rendered frame by frame in headless Chrome. Each coin is a follower's profile pic. The metal depends on your own follower count, and the wear depends on how long you've followed. About 5 minutes to render; considerably longer to stop fiddling with it.
Key line: mods aren't sandboxed and blast radius only reads command text, so aliases and scripts slip past
String-matching hooks catch mistakes but hard blocks belong in permission rules or the OS
I hope that more harnesses will build on this
seeing so many crazy takes the last few days across Terra and Kimi
is it bait or am I genuinely missing something?
GPT-6.1 Terra played six hours of a coding game in about 3 minutes.
Same game, same three seeds:
Terra: 62 pts · 3 min · $0.22 a run
GPT-6.1 Sol: 74 pts · 8 min · $0.47
GPT-6 Astra: 67 pts · 5 min · $1.79
Very fast. Not quite Sol. 🎥
It's FarmerBench: each model gets 6 game-hours in The Farmer Was Replaced (@metaroot_ch), where the only way to play is writing code for a drone.
Every model's code, decisions and runs are open: farmerbench.dev
I run new models within hours of release. Follow if you want the next one.
daily reminder that you can now just think of stuff and then type stuff and then have stuff
and that's pretty cool
Kimi says it's stretching K3.1 Pro's quota by "reducing unnecessary subagent calls and output length".
Fair enough, and a 25% reset is decent of them.
But my FarmerBench runs had no subagents. The harness only allows the game's own tools.
One seed still didn't fit in a 5-hour allowance.
So subagents might be part of the problem but they can't be all of it.
We're sorry for the rate limits and unnecessary quota consumption on the K3.1 Pro subscription plan. These were our issues, and we should have fixed them sooner.
Today, every subscriber on every plan gets a one-time usage reset, restoring 25% of their monthly quota (usage below 25% resets to zero), equal to one week of usage. Your usage bar will reflect this adjustment by end of day PST.
We've also adjusted the plan's parameters and are reducing unnecessary subagent calls and output length, so your quota goes further.
Thank you for your patience while we put this right.
reading that this may be a bug and I hope that’s true
I didn’t make it through 1 seed on the 5 hour allowance and had to top up for API usage
should have waited for opencode
Some insane hype on my timeline for Kimi K3.1 Pro
Spoilers ahead
- it’s expensive
- it’s not an Opus killer
- you will not be cancelling your frontier lab subs
The bones came down to a hat. Cashing in a dinosaur's tail means swapping hats, and green and gray were locked on all three seeds.
Seed 3 tried the straw hat. Later it predicted a full 16×16 route would pay tail² × 2 ≈ 130k bones. It paid 130,050.
Seed 1 thought of the straw hat twice and never tried it. It also bought four drones and farmed with one.
Caveats: one benchmark, three seeds. High effort (Kimi has no medium), on a Kimi Code subscription; $2.23/run is at list prices.
farmerbench.dev/models/kimi-…
It reached all 11 resources on every seed. But the score is a sum of logs, so balance decides it.
Seed 2: 10.4M wood but 29 bones, for 52.8. DeepSeek on the same seed: 229k wood and 33.7k bones, for 51.1. Nearly the same score, opposite farms.
I don't think many people have clocked yet that you can just build your own harness.
If there's one job you want done, give an agent only the tools that job needs, lock everything else down, and let it work out the rest itself.
DeepSeek Search has the cheapest price per search of any paid API on this board.
Then each 7k-token result goes back through the model. About $0.032 a task, and $0.023 of that is model cost.
The price per task is more important than the price per API call
DeepSeek Search debuts on the Artificial Analysis Search Index at 69, 17th of 27 search products, at the lowest price of any paid Search API on the board
@deepseek_ai launched its Search API on 17 October. We benchmarked it in our Stirrup agent harness, where GPT-5.6 Luna (medium) calls DeepSeek Search as its web_search tool to complete a combination of tasks from BrowseComp, DeepSearchQA and AA-Omniscience.
At $0.75 per 1k searches, DeepSeek Search costs less per search than any other paid Search API we benchmark, below Octen, Parallel (turbo) and Parallel (fast) at $1. At about $0.032 per task, it is the 5th cheapest of 27 products, cheaper than 20 of the other 25 Search APIs (median $0.066) and level with Nimble (lite). TinyFish, Octen, Parallel (turbo) and Parallel (fast) cost less.
Key benchmarking results for DeepSeek Search:
➤ Artificial Analysis Search Index: DeepSeek Search scores 69, a 36-point lift over the same model with no search (33). It sits between TinyFish at 71 and Linkup (standard) at 68
➤ AA-Omniscience is DeepSeek Search's strongest benchmark: it scores 70% accuracy on our proprietary factual answer benchmark. This ranks 10th of 27, 3 points behind the joint leaders, Exa (auto, dynamic highlights) and Firecrawl (73%). BrowseComp, which needs multi-hop reasoning across several searches, is its weakest benchmark: at 63% it ranks 25th of 27, well behind the Perplexity variants and Octen (85% to 87%)
➤ Large search results: each search returns about 7k tokens, near the top of the board's range. Each result carries about 1.8k characters of passages from the page. This raises model cost: at about $0.023 per task, DeepSeek's model cost is the 2nd highest on the board, behind Linkup (flash, alpha) at $0.039
➤ 128-character query limit: DeepSeek rejects queries longer than 128 characters. 6% of the agent's searches hit this limit, and for 87% of these, the model's first shorter rewrite worked
Key details:
➤ Setup: GPT-5.6 Luna (medium) in Stirrup, DeepSeek Search API with need_content=false (page passages, no full page content) and 10 results per search
➤ Pricing: $0.75 per 1k successful searches. DeepSeek does not bill empty results
➤ Rate limit: 5 searches per second per account
DeepSeek V4.1 Flash costs 13 cents a run on FarmerBench.
Grok 4.7 costs $11.86.
On the same seed, DeepSeek finished 2.4 points behind it.
🧵
Seed 3 never bought it. It decided weird substance had no source, skipped mazes, and built a self-funding dinosaur loop that made about 450,000 bones.
Then it checked what it was missing.
The score rewards making every resource. It was not fine.
Setup: OpenCode, pinned to DeepSeek's own endpoint on OpenRouter, at high effort. DeepSeek has no medium; everyone else ran at medium.
9th of 11 overall. Full write-up, with code, replays and every quote linked to the log: farmerbench.dev/models/openr…
