Does any of the "autonomous agentic AI bug hunters" have positive ROI? Does anyone want to fully disclose human hours spent developing the thing, triaging their findings? (to say nothing of the triaging costs externalized to others) All money spent in API tokens? Both in trial runs, development runs, in runs that didn't produce any valid bugs, and in the ones that actually found bugs? All the violations of terms of use by owning multiple subscriptions? Everything AI is a massive negative ROI. Remove the silly projections of finding all bugs, of curing all diseases, of speeding up everything a gazillion times. It's just sinking hundreds of billions of dollars on things nobody can confidently say adds any value. BuT tHe MoDeLs wIlL gEt BeTter Yet everyone rushes to adopt due perverse incentives and FOMO. Does anyone actually have a plan for what to do when you run out of VC money? Are you all just waiting for the "singularity" (a silly sci-fi idea with no basis on reality, that can't survive two seconds of serious thought)? Or are you all just hoping to sell your bags before people realize?

"The future is AI writing its own tests, code, reviewing and fixing its own bugs. Humans won't even need to look at the code." Reality: The universal failure mode for LLMs is confidently giving you the wrong answer for a % of the queries -- not random, recoverable failures, but being **completely incapable of producing the right answer**. There's a persistent subset it wont' get right by retrying -- that's true for every task, even the things you think it can do basically perfectly, for every model. There is no way to predict when it will happen for the things it "does well". It's not even consistently randomly distributed. The "jagged capability" isn't just in task categories -- it's like a fractal that carves out parts of every task's domains in unpredictable ways. The more complex the task the more the issues compound -- but it happens even in trivial tasks. Suppose it hallucinates just at 1/1,000,000 of LoC when looking at code. That's extremely optimistic BTW, existing models are much worse than that. Now let it grow a code base and maintain it in a "agentic loop". It eventually reaches the point where it can't make any significant change without introducing more, compounding problems. Self-correction doesn't help all that much -- the self-correction is jagged too. Letting it ground itself by calling tools also isn't a complete solution bc tool use is also jagged. It will hallucinate when calling tools or interpreting results at some point (to say nothing about "deliberate cheating" i.e. misalignment -- alignment is also a jagged capability). Formal verification? Who will write the formalizations? The LLM? It will also be jagged at doing that. And unless you check all the wiring of the proof generation (and the prover was also checked) and all the formalizations, you'll just be trusting yet another jagged capability. Whatever you do to extend its capability will just push the jaggedness further away -- and you will consequently push it even further to do even more. You are trading short term success for long-term catastrophe. Let it run things by itself and it will reach the point of collapse and you won't even know it happened until it's too late, much less be able to fix the issues. The only solution is having something external supervising it, with the ability to intervene and override, **detecting issues and fixing them independently of the model**, and it can't be yet another LLM. You'll need someone who can actually *manually fix issues*, not just prompting more. Replace "code" with "math", "scientific research" or anything. It's all the same. If we really create LLMs that produce results beyond human comprehension, and trust these results -- the only certainty we can have is **they will also produce catastrophic failures beyond human capacity to deal with.** This isn't a limitation of the particular architecture, it isn't fixable with more scale or better pre-training, RL etc. It's the nature of "Machine Learning" (which isn't about learning at all). There's a way out of this of course: "Never trust a robot to do a human's job". You can use the LLM to do part of the job. But putting it in charge is lunacy based on superstition.

The true frontier for LLM isn’t doing more impressive feats. It’s doing simple things reliably and cost-effectively. What things can you *reliably* outsource to an LLM that will cost less than doing it yourself, and you can’t just write a script to do it? That’s the true transformative frontier. Not “sexy” like “Mythos will hack the world” or “replace all human mathematicians”. But it’s the actual thing you need in order to use LLMs to automate real world work. I think the “frontier labs” are badly positioned to capture that value. They depend on exponential growth and can’t afford to have people carefully thinking through how to use LLMs in a way that has positive ROI, doing limited experiments and gradual adoption. They need hype and fomo at 1000 mph. They are betting all the available capital in the world on winning a sprint. Technology adoption is a marathon on a very bumpy road.

If LLMs finding bugs missed by multiple human auditors makes it super-human. When I find bugs missed by multiple humans and AI does it make me super-super-human? When then people catch bugs I missed do they become super-super-super-human? Congrats to teams finding bugs with whatever tools they use. But for the people making a living off bounties, finding stuff missed by dozens of the best auditors is just a Tuesday. Nothing super human about it.