AI Engineering Consultant. I help companies ship their first AI projects, unglamorous plumbing included. I build real systems, not decks. Ex Tenstorrent.

Toronto, Canada
Joined March 2007
Finding a job is a job. I spent months treating mine like one. Spreadsheets, 40 tabs, follow ups I forgot to send. So I did what engineers do with messy processes. I built an app.
3
1
5
207
Shane Bennett retweeted
Nice thread, some RL details used by closed labs slowly leaking out.
Tinker's new StableReinforce objective is Reinforce + Score Centering. Here's how I reverse-engineered it 🧵
17
28
1
786
48,139
Shane Bennett retweeted
Banger guide by Anthropic. While frontier models have improved at context management, subagents still help in some cases (e.g., parallelizing research tasks with their own context windows). I still use them, but mostly for complex research work via dynamic workflows and /goal. For most agent workflows, folding subagents into the main loop via tool calls and skills makes a ton of sense (as Anthropic recommends). That's because newer models are more persistent and better at managing large contexts. Subagents might be getting in the way now, since models can handle much of what subagents were used for. I see a similar pattern with memory (another day). But you can't just get rid of subagents if you have been using them extensively. You need to run evals (cost, efficiency, etc.) and let that help you determine whether to fold them (and how) into the main agent loop. The evals should always inform your strategy. But keep in mind issues that can arise when scaling agent work, such as too many tool calls, context bloat, and unclear separation of concerns. Subagents are still very useful for that. Lastly, all of this is constantly evolving as frontier models improve. Always keep an open mind and regularly test/eval your workflows and harnesses.
Anthropic is now recommending that companies REMOVE unnecessary subagents from their AI systems. This is a pretty significant shift in agent architecture, with their team finding that a single agent equipped with skills can frequently outperform complex multi-agent systems, and one customer seeing roughly 2x faster execution. The emerging rules of thumb: → Skills: Teach the agent HOW to do something, including procedures, expertise, and instructions. → Tools / MCP: Give the agent the ability to DO things through APIs, actions, and live data. → Subagents: Use for parallel work, isolated context, or tasks a single agent struggles with. The interesting part is progressive disclosure. Instead of spawning specialized agents or stuffing everything into a giant prompt, one agent discovers and loads skills and tools as needed. Their advice is to fold existing subagents into the main loop, turn their instructions into skills, expose their actions as tools, and run evals to see if performance holds up. As models have gotten smarter, improving agent architecture is becoming less about what you can add and more about what you can delete. claude.com/resources/article…
59
46
11
432
36,116
Shane Bennett retweeted
At the top end of models, real-world usability matters a lot more than +/- 2 points on AA. This takes a while to understand for a model, where AA is a sort of potential, but some models hit and some miss (e.g. opus 5 vs 5.5). Seems like 3.1 may be closer to the miss side.
OPENAI IS NO LONGER EVEN TOP 5 ON AA ANALYSIS AFTER KIMI K3.1
52
10
2
276
21,443
Shane Bennett retweeted
DeepSeek is serving 150T tokens a day on one cluster, more than the open model inference on OpenRouter, Fireworks, Together, and BaseTen combined. Below is my reconstruction of various inference volume reports across said companies.
Replying to @natolambert
right from the horse's mouth ah, right. That's "a single cluster", who knows how many they have github.com/deepseek-ai/DeepI…
26
35
8
428
30,487
Shane Bennett retweeted
By every measure, Opus 4.8 beats human experts as peer reviewers in a study of 124 Harvard social science papers covering a range from statistical analyses to formal models to qualitative work. As academics, we need to figure out how to adapt the practice of science to AI, fast.
258
261
90
1,709
104,032
Shane Bennett retweeted
When my first AI book was published in 2024, the best AI could do 4 minutes of human work with a 50% chance of success When I turned in the draft of Co-Existence earlier this year, it was 17 hours When it was published two days ago, AI could do 17 weeks open.substack.com/pub/oneuse…
146
88
37
777
52,809
Spent the weekend playtesting my game's BBS mode on my phone. Full TW2002 style terminal, green phosphor, pixel art ships. Type a number to warp. Dialed in like it was 1993 and found a few bugs in my own code within the hour. Fixed them that night and played some more.
2
15
Shane Bennett retweeted
Highly recommended read from Google DeepMind and colleagues. There is a lot of interest in self-evolving agents, but they're hard to get right. This work highlights how important and difficult memory and retrieval are in agents. The big question in this paper is whether self-evolving agents keep learning without forgetting. They tested ReasoningBank, GEPA, and Meta-Harness on four task streams that drift in topic, workflow, or domain, built from BIRD, BFCL, and ScienceWorld. The weights stay frozen, and the agents adapt by rewriting prompts, memory, and skills. Self-evolution improves overall scores on most streams. But more adaptation consistently means agents forget more. Methods that rewrite a shared prompt or workspace make global edits, so they interfere with unseen tasks and overwrite older knowledge when the stream shifts. Memory retrieval keeps updates more local, but it still pulls in memories from other domains and collides when similar queries need different workflows. Static benchmarks hide both problems, so test self-evolving agents on a stream that changes over time. arxiv.org/abs/2610.25858 Chat with Paper: academy.dair.ai/papers/are-s…
57
59
342
17,370
Shane Bennett retweeted
We have presents for you! 🎁 - A new WebMCP Skill with best practices for security, development, & consequential flows - WebMCP Types: TypeScript definitions, with type inference from InputSchema Give em to your agent, it helps scaffold out WebMCP so fresh and so clean clean
40
127
16
874
71,693
Shane Bennett retweeted
A consequential choice will be made soon by government and companies as they are increasingly flooded with agentic AI inquiries that overcome the frictions that gated access (see the graphs - it has already started!) Two options: do you make services worse or better as a result?
273
180
74
1,375
162,582
Shane Bennett retweeted
I’m speaking at the IMF in an hour about the singularity and why I believe it has begun and the choices we'll need to make. Watch live here: youtube.com/live/Rqt9kWoTJbc…
76
105
45
787
218,607
Shane Bennett retweeted
Banger paper from Google DeepMind on long-horizon multi-agent coding. (bookmark it) They report that, to the best of their knowledge, this is the first end-to-end autonomous completion of Minecraft by AI agents. They report that four Gemini 4 Argon agents beat Minecraft end-to-end. They defeated the Ender Dragon in 27 hours and 14 minutes with no human intervention. Each agent controlled one player through the standard Mineflayer library. No one assigned roles, wrote game logic, or scouted a seed in advance. The agents planned, wrote, and debugged every program themselves, and coordinated through in-game chat and a shared memory. Over the run, they wrote 2,783 programs and died 605 times. They split the work, shared what they learned, and recovered each other's items after deaths. If you build agents for long software or enterprise tasks, this is a useful case study. The team had to keep one plan consistent across thousands of dependent steps for more than a day. arxiv.org/abs/2610.22319 Chat with Paper: academy.dair.ai/papers/a-tea…
98
85
12
589
31,655
Shane Bennett retweeted
router lives in the harness, not a gateway, bc the harness knows the task blog: langchain.com/blog/how-to-bu… xitter.cf/neil_xbt/status/210997…
LangChain gave its coding agent three models and a router. Open SWE, their open-source coding agent, now sends each thread to the cheapest model likely to finish the job. they tested it on 973 threads, half routed and half always on GPT-6 Astra. median cost per thread: $0.94 against $2.61. that's 64% lower. threads ending in a merged PR: 29.2% routed, 27.3% on Astra. LangChain's p-value for that gap is 0.49. the router sent 56% of threads to the balanced model, 34% to the fast one and 10% to the performance one. they also tried always using the fast model. that test ended within a day, because engineers reported low output quality. LangChain's own caveats: the control was the most expensive model, and the router is a proof of concept. Which three models would you put in the tiers?
73
17
8
153
11,449
I built it because it was my own itch. The best tools come from the builder feeling the pain directly. Using it every day shaped what it became.
1
1
29
Shane Bennett retweeted
Banger paper from Google DeepMind on co-evolving an agent's harness and its weights. (bookmark it) If you already log your harness changes and results, those logs can become training data for the model running inside the harness. This is a nice paper from Google showing how to do this co-evolution. Specifically, they propose Evo2, a framework for weight-harness co-evolution. How it works: One policy plays two roles. As a Proposer, it writes edits to its own harness, and as a Reflector, it diagnoses what each edit changed. Those edits and their outcomes then become training data for both roles, using SFT followed by interleaved GRPO. The result: Evo2 reaches 48.3% on SWE-bench Verified and 82.1% on Polyglot, against 41.0% and 62.6% for the strongest baselines. The harness part alone moves a GPT-5-mini agent from 24.4% to 48.3% on SWE-bench Verified. The evolved harnesses also transfer to other benchmarks without further evolution. Paper: arxiv.org/abs/2610.13025 Chat with Paper: academy.dair.ai/papers/evo-2…
54
56
4
334
19,464
Shane Bennett retweeted
Recommended NVIDIA paper on memory-efficient post-training RL. A very interesting technique for scaling RL for LLMs. The main takeaway is that you can update all of a model's weights with RL and still use less memory than LoRA. At 1.5B, LoGRA averages 7.2 GiB of training memory, while rank-256 LoRA needs 13.2 GiB. Pass@4 is about the same (81.8% and 81.4%), and Pass@1 is about 2 points lower. If GPU memory is what stops you from running RL on a bigger model, this method is worth a look. LoGRA never builds a full gradient for the weight matrices it compresses. As microbatches accumulate, each weight gradient is compressed with a rank-256 random projection that is redrawn every step, and the update is rebuilt from two small matrices. Before each step, it predicts how much the policy will change and scales the step to stay within a KL budget. Average training memory per GPU drops 45.7% at 7B (31.8 to 17.3 GiB), with peak accuracy close to dense Adam (72.3% and 72.5% Pass@1). It also trains a 27B model for more than 1,100 steps on a single eight-GPU node, where dense Adam runs out of memory. Paper: arxiv.org/abs/2610.06647 Chat with Paper: academy.dair.ai/papers/logra…
30
31
5
182
14,592