sf thinkcat
emoticat
Joined September 2023
- Tweets12.1K
- Following391
- Followers11.1K
- Likes87.5K
formal verification is one of the interesting “future technologies” being made possible with superhuman ai coders
i think we should think about what is possible now with formal verification, which was once too expensive / difficult / costly for mere humans
go read oleg!
Three days before I was pushed out of OpenAI I posted a Formal Verification Manifesto in a core research Slack channel. It listed OpenAI systems that could benefit from formal verification.
The only person who responded was @sama, who was very supportive. Not sure what the moral of this story is, but it's never too late for OpenAI to invest seriously in software & hardware formal verification!
On a more serious note, @theoremlabs and @AnthropicAI did something amazing. They showed how to verify software at scale without requiring developers to write specifications in Rocq or Lean. Instead, they used existing fuzz harnesses as specifications and LLMs to generate machine-checked proofs.
Of course, fuzz harnesses specify only some correctness properties. In the simplest case, the proofs establish crash-safety, not full functional correctness.
We need community-maintained open-source projects with reusable formal specifications and correctness proofs for major libraries, kernels, compilers, virtual machines and hardware.
Human review of critical specifications and LLM-generated patches is essential. A model could satisfy a specification simply by removing functionality.
As AI writes and rewrites critical infrastructure, formal verification should become a key part of AI control. Critical properties of AI-generated code should be machine-checked rather than taken on faith.
AI is making vulnerabilities easier to find and exploit. Frontier LLMs and reusable formal specifications could finally make formal verification practical and economical at scale, while keeping it understandable to humans.
Kudos to Theorem co-founders @rajashree and @diagram_chaser!
i think a good benchmark for the future of security is to have models write complex programs from scratch, like web browsers, and then test whether models released later can still find vulnerabilities in them, and how many
basically tracking whether we’re approaching vulnzero
some quick thoughts on frontier labs and cybersecurity
1) two critical vulnerabilities were discovered in php that allow for remote code execution; the first was reported on august 11
2) it seems now humans just weren’t devoting so many resources to finding vulnerabilities, and perhaps weren’t that good at it in the grand scheme of things
3) i think open-source software is going to have to change: maintaining it will increasingly become the responsibility of frontier models and frontier labs
4) no one else will have such advance access to frontier models or be able to spend as much on tokens as the labs themselves
5) so, it makes sense for frontier labs to scan and patch open-source software, especially the code their coding agents use to build software
6) one issue, though, is that open-source projects often have difficult governance structures, so i can imagine a world where frontier labs just fork these projects
7) build and maintain their own versions, making fixes, improving performance, adding features, etc and then train their models to use them by default
8) i can also see a world, very soon, where ~all software at a company has a claude or codex instance running 24/7 that watches over it (like the anthropic superfans)
9) the agent watches an anthropic or openai feed and, when a vulnerability is found, immediately kicks off a full migration to the new version
10) or otherwise refactors the codebase, changes the infrastructure, ports capabilities, etc... to mitigate the vulnerability
11) this will happen automatically, without a human in the loop; engineers will just come in the next morning and see what happened
12) this is part of what it means for coding agents to increasingly own software, and for labs to become responsible for the software supply chain
13) at some level, i think this just becomes the enterprise expectation when they are paying millions for coding agents / the agents are making arch choices
14) it is a very exciting time to be at a frontier lab and think through what this product offering should look like and how it should be sold to enterprise clients
Effective Immediately, all organizations using PHP for their systems are advised to be prepared for an update addressing two critical vulnerabilities. PHP is widely used for web applications. These vulnerabilities often provide a direct path to the server's file system or database. If an immediate update is not feasible, it is recommend that organizations take the system offline until it can be fully mitigated. NSA will follow up shortly with more specific guidance.
i talked about this almost a year ago while reflecting on the anthropic <> bun acquisition
xitter.cf/fleetingbits/status/19…
Some thoughts on the Bun acquisition
1) The prosaic description of why Anthropic bought Bun is that Claude Code is built on top of Bun and Anthropic wants as much control as possible over the installation experience and performance of Claude Code
2) But, in an accelerating world, we should expect more deals like this with respect to key projects in the open source ecosystem, because labs will have a real desire to control the open source software supply chain
3) First, just from a performance perspective, labs are going to want to ensure that their agents are more robust and this means being able to do RL over major open source projects on stable APIs or otherwise being able to control API updates
4) Second, from as security perspective, labs are going to become more and more responsible for any security vulnerabilities that their agents introduce in the codebases in which they implement features, so labs will want guarantees that the libraries they are using are secure (it may eventually be part of their value prop).
5) Part of this will be scanning open source libraries for security vulnerabilities and only having their agents using those libraries that meet security standards that the labs themselves have determined in advance.
6) Part of this may be actually building or acquiring the projects and taking over responsibility for their maintenance and development. They get to control the features and make sure it makes sense within their RL pipeline and also they get to ensure that the libraries are secure.
7) Big tech already does this to some degree. But, it has always been cost prohibitive to to this on a truly ecosystem wide scale. And so, effort has been focused on a small set of core projects (e.g. Linux).
8) But, in a world where coding agents scale and the cost of development drops dramatically, suddenly this is possible and labs can oversee hundreds or thousands of ecosystem projects. Because, it's Claude Code overseeing the project, for the most part.
9) And, using open source is better and more robust than just keeping this library code closed source or alternatively developing everything client side, where the model can make errors, has to keep the data in context, might do it slightly differently each time, etc...
10) In the limit, you may even start to see some of these projects rewritten in formal languages like Lean, open sourced by the labs, etc... as the security aspect of development becomes more important and it becomes increasingly the responsibility of the labs as vendor.
i’ve been working with claude on a 3d replica of philip johnson’s glass house for my personal website, fleetingbits.io, and i’ve turned a lot of the site’s features into locations on the property.
i think it's interesting to look at developer productivity at a frontier lab.
it seems like anthropic’s engineering headcount grew modestly from q1 into q2 2026, while code output grew much faster.
individual commits also appear to have become smaller in total lines of code.
it looks like anthropic researchers use research agents about 4x as much as openai researchers, on average.
and since may, usage across both labs appears to have been doubling roughly every 35 days.
i keep thinking about this; 75% of compute spent on internal ai r&d is insane
If I'm not mistaken, Anthropic's 85% gross margin and >40% compute margin from this Bloomberg story implies that 70-75% of their compute is now going to R&D.
bloomberg.com/news/articles/…
a lot of investors make projections of frontier labs as if these were 20-year-old companies that had already spent at least a decade optimizing their economics.
anthropic has added $70bn of arr so far this year; these are the most incredible businesses of all time.