Pinned Tweet
Sharing my latest initiative - the Nerfwatch Index
What is Nerfwatch Index?
Like many, I have often questioned whether models from the big AI labs are dialed back (i.e. 'nerfed') over time. I presume this might be to practically manage ongoing model delivery commitments within finite compute. Maybe it's just to reduce costs over time? Either way- it's challenging to prove, but we sometimes just sense things 'back off a bit' as we use them.
I'm contributing some of my subscription capacity ongoing for a while to test for this phenomena (started just before Sonnet 5.5 launched), and will see whether it is indeed true or not.
How does it work?
Nerfwatch Index tracks whether closed AI models maintain their original performance over time. Twice daily, models face the same 20 tests, with the outputs judged against strict quality standards (measurable / empirical), despite challenging constraints and conditions.
Link: nerfwatch.googz.com
Do you believe the big labs nerf their models over time?
Does your local model actually need to think?
I plotted PA Core (my multi-step personal agent tests) with thinking off vs on for 9 models on my RTX3080.
Most models gain from thinking, and the biggest jumps were Spark X2.6 1.7B (+66) and Groot 19B-A3B (+61).
SenseNova-Loop 9B stayed flat, and MiniCPM-V 4.7 surprisingly scored 12 lower with thinking on.
But the score isn't the whole story, because thinking costs time. On multi-step tasks it can add thousands of tokens and turn seconds into minutes.
For a personal assistant agent, responsiveness matters. If a model is strong with thinking off, fits on my device and runs at a similar speed, I'll choose it every time. A quick, good answer beats a slow, slightly better one.
So test both modes. Your model might not need to think as much as you'd assume.
Re-ran Spark X2.6 4B, this time on my RTX3080. My earlier Macbook review had setup issues on my end, so @SparkLLM deserved a fairer test.
Reasoning on, 51K context, 3 runs each:
- A1 (light agent) 10/10
- T1 (tools protocol) 30/30
- C1 (context retrieval) 10/10 at 8K and 16K
- 48/48 tasks finished, zero failures. Perfect!
That's a lot cleaner than what I showed before. Some of the gap was my setup and scoring, not the model.
PA Core (multi-step agent tasks): 175/300. On the same protocol that's a step up from Spark's own 1.7B at 172, with Qwen3.5 4B still ahead at 206.
Rock solid on the fundamentals, room to grow on longer 'personal admin' agent work with unstructured data.
Lesson for me: when results look off, check your own setup first.
My thanks to @Tylerwtzhang from the @SparkLLM team who reached out with some advice about this.
Specifically- Spark X2.6 4B is built to work better with thinking mode on. And, it is mostly improved.
I re-baselined my personal agent testing (X2.5 vs X2.6, thinking off and on), and also comparing them to an incumbent (Qwen 3.5 4B).
Some thoughts:
- Reasoning improved some personal-admin capabilities, but it was not a consistent upgrade, and importantly, substantially increases waiting time (this is partially why my testing and interest in how good models are with thinking off, because it makes them much more responsive.
- Enabling reasoning improved tool reliability for both Spark models and improved Qwen’s context handling and daily organization. Other outcomes regressed or failed to complete: Spark X2.5 introduced scope errors, while X2.6 entered a reasoning loop on the A1 agent-context test (@Tylerwtzhang I'll share more diagnosis about this offline if interested, will DM you).
For a personal admin/assistant agent, the trade-off is therefore task-dependent: extra thinking sometimes delivers a better result, but it also brings longer waits and a completion failure with one case.
Background of my test suite here: teapawt.org/PaulGug/status/2097943…
Zero-trust security, people. That’s the goal you aim for. Don’t just secure your environment for the accidental- you secure it assuming that there could be malicious intent.. because the accident becomes malicious once it’s in.
Congrats to @ttrimoreau. 9 months of daily posting and $15.9K is real, earned success. 🎉
A gentle note however for anyone thinking "I'll just do that":
Work out your effort vs reward. ~$57/day sounds great but at 3 hrs/day avg of posting and replying, that's less than $20/hr.
And under the newer content rewards model, the latest fortnight paid ~$26/day. Same 3 hrs/day, that's closer to $8.50/hr (Lots of assumptions here btw.. just to give an idea).
It's also one platform's money and that’s a risk. X can change the payout rules, shrink the pool or demonetise an account overnight. I've seen it happen to great creators like @nxhaaa19, @elixir_art and others.
Use it as a bonus while you build an audience and skills you own. Don't make it your only income.
Congrats again Thomas! 😊
I started posting on X every single day on January 27, 2026.
I had no audience. No experience. Just a goal to grow.
9 months later, X has paid me $15,898.75.
What actually made the difference:
Posting every single day.
Replying to hundreds of people.
Building genuine connections.
Staying consistent even when nothing worked.
Some days I spent 6+ hours on X.
Some posts got 200 views. Others reached millions.
But the biggest lesson?
You don’t need to go viral every day.
You just need to keep showing up.
Would you post every day for 9 months to get these results
After a few weeks of local model reviews on my 10GB RTX 3080, the data is starting to tell a story.
PA Core score (my personal-agent tests, /300) vs GGUF file size. Each point is the best result I got for that model on my setup.
What stands out:
• The 2 to 7GB zone is crowded and competitive. Gemma 4 12B hits 235 from 6.4GB, and the 9Bs cluster around 225 to 231.
• The top scores come from MoE models that spill past 10GB of VRAM. MiniCPM-V 35B-A3B (273) and Groot 19B-A3B (269) still run well with partial CPU offload.
• Reasoning on vs off can move a model 30 to 60 points, in either direction.
• 1.7B models scoring 190+ still blow my mind.
Not a strict leaderboard, since quants and modes vary. It shows the best each model could do on my consumer GPU.
More points coming as the reviews keep rolling.
#dataisbeautiful
Sharing a completely non-AI/SI related shameless plug for my mates’ film discussion podcast ‘Lead Smoke’ - I feature as a guest on this one episode, where we discuss The Shawshank Redemption.
Despite my attempts to ruin it, Pete and Dat really know their stuff and they have covered a few interesting films already - please consider a listen and a follow 👍
open.spotify.com/episode/4uc…
Agents-A1-4B (InternScience) vs its base, Qwen 3.5 4B. Same 3080, both Q4_K_M, reasoning off.
Impressive! The agent fine-tune earns its keep where it counts: staying on task over multiple turns.
• PA Core (personal agent) : 197 vs 172 (+25)
• PA2: 92 vs 51. Qwen hit 86 on one run but ended early on the other two. That gap is nearly the whole lead.
Where the base holds:
• PA1: 68 vs 54
• C1 (context search) at 16K: 9.0 vs 8.3
• T1 (tool discipline) and A1 (agent light): ties (30/30 and 9.33)
Agents is also a bit faster (~144 vs ~131 tok/s generation, ~5.4K vs ~4.6K prompt).
One caveat: with thinking on, Agents-A1-4B overthinks simple tool calls in LM Studio, on both Q4 and Q8. Stick to reasoning off for now.
Takeaway: the fine-tune beats its base on multi-turn agent work. Use it with thinking off.
RTX 3080 10GB, LM Studio, Q4_K_M both, 51.2K context, temp 0, PA output cap 2048. 3-run means, seeds 1234 / 2026 / 42. A1 manually graded. Reasoning off.
Background on my testing: teapawt.org/PaulGug/status/2097943…
Model card: huggingface.co/InternScience…
The 10GB RTX3080 is an awkward in-between GPU. 35B MoEs only just squeeze in with heavy CPU offload, and 9Bs leave room unused.
Groot 19B A3B (a pruned Qwen3.6-35B-A3B) fills that gap nicely. I put it up against the top tier, @OpenBMB MiniCPM-V 4.7 35B-A3B (IQ3_M), on the same card.
Quality:
• Groot, reasoning on: 269/300
• MiniCPM-V, reasoning off: 273/300
That's 35B-class quality from a 19B. Groot does need thinking on, since it scores 208 without it.
Speed:
• 52 vs 20 tok/s at 51K (~2.6x)
• 42.5 vs 17 tok/s at 128K (~2.5x)
Groot kept the same offload at 128K. MiniCPM-V had to push 3 more layers to CPU.
If you're on a GPU between 8 and 12GB, a ~19B MoE might be the sweet spot. You get near-35B results, much faster generation, and real room for long context.
Quants differ (Q4_K_M vs IQ3_M) as this compares the best each can do on 10GB, not a matched A/B test.
Notes: RTX 3080 10GB, 32GB RAM, LM Studio, Q8 KV. Quality is 3-run means at 51.2K ctx. 128K speed is a single run each (Groot at 18 CPU layers, MiniCPM-V at 26).
Model card: huggingface.co/GrootTeam/Gro…
Testing info: teapawt.org/PaulGug/status/2097943…
Understandable, but this is another reason why I have my reservations about going all-in on a personal assistant that is in the cloud.
The main one being data privacy, but the other is this - so I’m keeping the control at arm’s length.
I prefer Hermes for this reason- and I’ll plug in whatever ‘brain’ that is most capable/viable at any given time.
I'm considering an overhaul/consolidation of my agent builder setup. Throw me some ideas please!
Current setup:
- ChatGPT/Codex: Project planning, building, execution, & maintenance.
- Hermes Agent: Personal admin, research & ideation. Some building as well.
- Grok bot: X platform activity/analysis/news.
- OpenClaw: Legacy personal admin (migrating off this).
Subs used: GPT Plus and X Premium. Some credits with Deepseek OpenRouter for testing and backup.
The issues:
I dont like how codex locks me down to GPT models for example, so am looking for an agent for building that allows different models/subs. I'm also looking to continue with Hermes Agent for everything else.
Other capabilities I'm after:
- I have a home lab server and also my desktop. I want to be able to reach them from other devices (i.e. the potential replacement for codex needs the ability to be remotely accessible).
- Going for value here - I'm a part time enthusiast, at best. $100-$200 subs will be wasted on me sadly.
- Ideally would consolidate around the one LLM provider sub almost everything, and PAYG for the rest.
Thoughts?
I consider this to be an informative ‘insider’ perspective on what makes harnesses work well. Comparing with my own suite of reviews and testing to see how I can make it a more informed/accurate predictor of personal agent efficacy.
Thanks for sharing @witcheer.
a 12 GB card already gets Hermes Agent through 27 of 30 agent tasks on a local model. a 32 GB card gets 30.
30 agent tasks, run offline in Hermes Agent: renaming code across files, solving a git merge conflict, finding the bad commit in a 14-commit history, fixing five bugs behind a 40-test suite.
TL;DR, the best file that fits each card at 64K context:
> 8 GB: Spark-X2.6-4B, 25 of 30
> 12 GB: Ornith 1.5 9B, 27 of 30
> 16 GB: Qwen3.8-27B GSQ-RCO IQ2_XS, 27 of 30
> 24 GB: Qwen3.8-27B Q4_K_M, 29 of 30
> 32 GB: Qwen3.8-27B Q6_K, 30 of 30
the difference is how they get there. the 9B made 514 tool calls for its 27, Qwen3.8-27B Q4_K_M made 257 for its 29. the 9B generates about 2.6x faster, so the whole set took 34 min against 29. the Q6_K file made 253 calls for its 30 and took 43 min.
full results: github.com/notwitcheer/llm-b…
This lines up with what I see on a 10GB card. On lower-end hardware the harness overhead isn't free: every bundled tool, skill and long system prompt gets prefilled again whenever the cache misses.
Leaning out your agent profile to just what you actually use is one of the cheapest speed-wins locally.
Impressive to see Hermes at 93% cache reuse here, the highest of the lot. Its token count looks big, but most of that is cheap reused prefix, which is exactly what you want on modest hardware.
Need more evidence harnesses matter? Here is an example.
Same local model on the same Mac. One coding agent reported nearly 12× the token usage of another. 👀
We spend plenty of time choosing a local model. What about the software running it?
A Reddit user tested 10 coding-agent harnesses using
🧠 Qwen3.8-27B Q4
🍎 M5 Max with 64GB
⚙️ llama.cpp, thinking disabled
🧪 Five small coding tasks, sandboxed
Results:
Median time and token usage
Pi: 5/5 passed · 38 seconds · 9.2K tokens
DeepSeek Harness: 5/5 · 60 seconds · 35.3K
Codex CLI: 5/5 · 94 seconds · 36.3K
Qwen Code: 4/5 · 183 seconds · 107.9K
That’s an 11.7× difference in reported median tokens between Pi and Qwen Code. Pi also finished faster and passed all five tasks.
🔗 Original test linked below.
ALT https://www.reddit.com/comments/1xj182d