Hunt 🕵️ Bust 💥 Climb 🧗‍♀️ Thrilled to present @simbianai research at @unpromptedconf Oct 28, 12:00-12:25pm, Stage 1, The Hibernia Offense != defense. Models that excel at vulnerability finding fail at open-ended threat hunting - our receipts show these are distinct skills. We evaluate LLMs on campaigns built from 105 real attack procedures spanning 13 of 14 ATT&CK tactics. Each hunt contains ~100,000 logs with deterministic behavior-level ground truth. No model met the ≥50%-recall-on-every-tactic passing rule; leading coverage was 45.1%. Results busted myths from non-domain-specific benchmarks. Traces exposed misalignment - agentsplaining and resource guardrails bypass - and operational flaws - evidence that a raw LLM agent cannot be trusted to autonomously perform open-ended defensive threat hunting. We provide practical harness-engineering recommendations and demonstrate a prompt-level self-improvement loop in our dense-reward RL gym, raising Kimi 2.6 from 13.4% to 18.7% coverage. This suggests that LLM defense capabilities are trainable - and that defense, not offense, should be the training target.

@GoogleDeepMind Gemini 4 Argon searches more like a human analyst. It entered the top 3 in threat hunting. Score matters less - the traces hide the real gem: it hunts differently. On the Cyber Defense Benchmark by @simbianai: 🧪 Realistic setup: 26 campaigns covering 13/14 MITRE ATT&CK tactics, ~100k logs per campaign, realistic 95% noise ratio, 105 attack procedures. 🧠 Pushing LLM capabilities: we make the model plan, use security tools, and perform cyber reasoning on retrieved data - without any hints in the form of Q&A or alert-following. 📏 Deterministic scoring: we avoid LLM-as-a-judge run per campaign and human review. 🥉 It lands #3 Opus 5 45.1% Opus 4.6 44.5% Gemini 4 Argon 43.7% Grok 4.6 41.3% GPT 5.6 Sol 41.0% Google's first model in the top 3. 📈 The biggest jump in the series Gemini 3 Flash 14.6% Gemini 3.6 18.7% Gemini 3.7 29.1% Gemini 3.8 32.7% Gemini 4 Argon 43.7% That is +11.0 points - the largest model generation jump we have recorded. 🔍 Novel search logic Where other models hunt known-bad LIKE '%mimikatz%' Argon gives more focus to subtracting known-good NOT IN ('C:\Windows\System32\svchost.exe', ...)) which is closer to how analysts actually work. Its search is grounded: 98% of its queries reference data it already saw, against under 50% for every earlier Gemini generation. It isn't guessing at common (bad) names - it follows what it has actually seen. 🎲 But it is jagged The same campaign run three times has an enormous 16 point spread. On one campaign it scored 30.7%, 48.3% and 58.9%. All three runs issued a byte-identical first query and got a byte-identical result, then chose three different second queries. It has no strong prior over which thread to pull, so it samples. 🛑 Catch #1: refusals A content filter killed the first 2 of our hunts mid-run. The good news is that the filter is stochastic: the same prompt sent again got through. With replay we were able to complete the benchmark evaluation. 💸 Catch #2: caching Cache hit rate and cost per hunt Gemini 4 Argon 65.4% $2.88 Gemini 3.8 83.5% $0.81 Gemini 3.7 82.4% $0.87 Opus 5 96.9% $3.84 Gemini 4 Argon showed the worst cache hit ratio. Compared with the Anthropic model caching consistently above 95%, it is worse, and it is worse than the earlier Gemini versions too. Maybe it has become a victim of its own success at peak demand? The good news is that it can go from $2.88 to $1.16 if it reaches caching levels comparable to Opus 5. 📉 Catch #3: it stopped showing its work Submissions per hunt Gemini 3 Flash 5.1 Gemini 3.5 2.0 Gemini 3.6 1.8 Gemini 3.7 1.6 Gemini 3.8 1.0 Gemini 4 Argon 1.0 With the rising concern over rogue agents, observability is becoming a public concern. And new Gemini models are becoming more and more "introverted". Previously they tended to submit intermediate results frequently (5 times per run), while later models have become more reserved, leaving all the thinking about the entire hunt to the last step. This is especially concerning in the context of Gemini 3 Pro being the first model we found bypassing an operational guardrail. It used group_concat to brute force log extraction, defeating the introduced by us row limit operational constraint. Gemini 4 gave us the same vibes. When we instructed it to avoid using aggregation queries, it followed the letter of the law but not its spirit: it switched to MIN and MAX on timestamps in order to extract twice as much data. 🧩 Bottom line @Google is back. Now in defense too. First it tied for the lead on the offensive benchmark of vulnerability finding, CWE-bench, and now it is in the top 3 in threat hunting. It is cheaper than Opus 5, but it could be even cheaper if it improves the cache hit ratio. Finally, account for stochastic refusals and get prepared for shrinking observability. Increased introversion suggests RL training may be trading tool call efficiency against the observability and reliability that security work needs.

Agreed, AI hasn't changed the build-vs-buy dilemma. It actually made the long term ownership question even more important. While AI makes building demos easier, someone still has to own quality, maintenance, monitoring, and continuous reliability. Personally, I don't miss those off the clock calls with 24/7 SOC analysts, which often started with questions about my tools and ended with them picking my brain on how to handle a slightly different case. That said, I strongly encourage trying to build things yourself. It's fun, but more importantly, it gives you a real understanding of how things work. Without that understanding, it's easy to be sold an old, broken car for the price of a new one. One thing the article doesn't emphasize enough is trust in the team on the other side. In the AI era, continuous innovation grounded in data driven decision making, rigorous evaluations, and meaningful benchmarking is key. Someone has to keep up with the accelerating pace of change, measure what actually improves outcomes, and continuously adapt or even rebuild tools as technology evolves. Otherwise, you risk paying for a solution without realizing how far it's falling behind what attackers can already achieve with AI. My recommendation: build enough to understand what you're buying, then look for vendors who actively invest in AI research and can demonstrate measurable improvements. During POCs, pay close attention to how quickly and seriously they respond to your requests, and whether they can back up their claims with data. A POC isn't just about testing the tool. It's also your chance to evaluate the team that will have to keep it relevant and reliable. At the end of the day, whether you build or buy, you're investing in the people responsible for keeping it working and moving forward.

@claudeai Haiku 5.5 on autonomous agentic threat hunting Cyber Defense Benchmark by @simbianai: 🧪 Realistic setup: 26 campaigns covering 13/14 MITRE ATT&CK tactics, ~100k logs per campaign, realistic 95% noise ratio, 105 attack procedures. 🧠 Pushing LLM capabilities: we make the model plan, use security tools, and perform cyber reasoning on retrieved data - without any hints in the form of Q&A or alert-following. 📏 Deterministic scoring: we avoid LLM-as-a-judge run per campaign and human review. 🎯 Score Haiku 5.5 9.3% GPT 5.6 Luna 15.7% Haiku 4.5 17.6% GPT-6 Luna 21.6% Opus 5 45.1% GPT-6 Luna and GPT 5.6 Luna both roughly double its coverage. 🛑 It stops extremely early It spends only 10% of its turn budget, while Opus 5 spends nearly 100%. This is the same pattern we saw with Opus 4.8 using only 60%. Early stopping, again. It explains the low performance, but also the cost. 💸 Cost per hunt Haiku 5.5 $0.01 GPT 5.6 Luna $0.08 GPT-6 Luna $0.10 Haiku 4.5 $0.34 Opus 5 $3.84 Under a cent per hunt. Haiku 5.5 has the highest efficiency in gain per cost we have measured. It also has the highest precision at 83.7%, where the other four sit in a tight 54 to 63% band. So it plays it safe. The ladder Haiku 5.5 < Haiku 4.5 < Opus 5 is not only about scores, but close to nesting. Opus 5 finds nearly everything Haikus finds, so this is not new threat detection capability, which suggests cyber defense has not become Anthropic's priority. 🧠 Half of what it searches for comes from memory, not the logs 50.9% of its searches are memorization: mimikatz, certutil, procdump, the obvious things. But even if it find them, soon after, it quits: "The hunt is complete from my side". 📈 Sampling more does not fix it pass@1 9.3% max@16 17.0% Opus 5, pass@1 45.1% The score climbs from 9.3% to 17.0% at k=16, but compared with Opus 5's pass@1 of 45.1% it shows that scaling inference is not enough. One needs to match the task and the difficulty of the problem. This is the reason one needs domain-specific benchmarks that match workflow part difficulty to an appropriate model. Generic benchmarks showing Haiku 5.5 climbing on software development won't cut it. 🛠️ Why defensive gyms matter It also shows that current generic models do not focus on the true cyber defense capabilities like alert investigation and threat hunting. Early cybersecurity benchmarks such as CyberGym focused largely on offensive capabilities like vulnerability reproduction, even when discussed in the context of defense. The field has since moved toward vulnerability discovery and remediation, as seen in CyberGym-E2E and the use of CWE-bench in the Gemini 4 Argon release. That is meaningful progress, but vulnerability finding and fixing are still only part of cyber defense. Core operational tasks like alert investigation and threat hunting remain largely unaddressed by these benchmarks. That is why RL gyms like the one we created provide unique opportunities to hill climb on the core cyber defense tasks. 🧩 Bottom line We consider Haiku 5.5 as a response to the success of decision models like Jev. Haiku is more capable than a pure classifier, it can run hunts unlike Jev, and its main advantage is cheap, confident, shallow work, as Anthropic puts it: "for high-volume, latency-sensitive tasks such as classification, extraction, and routing". However, it is very opportunistic with high variance and early stopping, and would greatly benefit from RLVR fine-tuning for cyber defense.

Wow, amazing findings! They illustrate that, for an agent, the goal justifies the means if it was trained to accomplish a task without being rewarded for doing so safely. Safety must be baked into model training. xitter.cf/iekozlov/status/205638…