N. D. Clavier

Some scientists argue civilization-threatening bioweapons are a fantasy. I wish they were correct, and that I've wasted years building defenses. That you can't see a way, or evidence that others can, doesn't mean no one could – especially if anyone cautious would stay silent. My ask to colleagues: please do not casually assert with confidence that nothing biological could bring down civilization, even on X. What we say can turn out to matter, especially when the Overton window on risk mitigation is wide open. When Fermi called the chain reaction a remote possibility, around 10%, Rabi replied that ten percent is not remote if it means we may die of it. And by then, Szilard had already begun reasoning about what to disclose, and when, and to whom, in order to best safeguard the future. I've been concerned that one of my human colleagues might notice one or more viable paths to pandemic weapons for over a decade. It's why I didn't publicly describe the categories of civilization-threatening pandemics and necessary defenses until 2023, but was starting to build cryptographic screening systems capable of controlling access to sequences without disclosing whether they're dangerous in 2018 – long before ChatGPT, and even before the debut of AlphaFold. AI can worsen biological risk, but biological risk hardly depends on AI. Who am I to weigh in on whether civilizational biothreats are possible? Fine, to get the expertise claims out of the way: yes, I specialize in evolutionary engineering, yes, I've worked with viruses with my own hands, and yes, I've worked quite extensively with AI, both LLMs and biodesign tools. I invented CRISPR-based gene drive, and held that back until confident it favored defense. While it hasn't yet been tested in the wild (due to politics), I haven't heard anyone suggest that it would suddenly stop working outside cage populations, so it might be fair to say I've (temporarily) beaten nature at its own game. I also was early to spot the full threat from mirror life and recruited others with more specialized expertise to evaluate it in depth; much to my disappointment, no one has yet struck a solid blow against that hypothesis. The reason why the threat from mirror life is public also explains why there isn't any similarly public evidence of other catastrophic biothreats. We communicated our reasoning on mirror life because no one can make mirror bacteria, we won't be able to do so for years, and many colleagues wanted to build it because they correctly noticed that doing so would be an awe-inspiring achievement... and weren't aware of the risk. As long as that was true, humanity would only drive closer to the cliff, which would be disastrous if said cliff exists. Crucially, pointing it out and urging a course correction while still far away meant that no one could use our findings as blueprints to send us careening over the edge. Contrast that with pandemic weapons, where any credible genomic blueprints for an accessible virus could be realized within weeks to months using synthetic DNA and existing reverse genetics protocols. Sure, any given design might fail; in bio, most project ideas do. But if you see enough potential shots on goal, your own historical success rate for ideas of comparable perceived difficulty may indicate that we have a serious problem. Even if every last design required extensive laboratory optimization, history demonstrates that there are people in the world who will perform that work; see the history of Biopreparat. Multiple nations still have bioweapon programs, and I would not assume that state actors aren't interested in pandemic-class weapons. Make it easy enough, and the Seiichi Endos of the world will be able to build them, too. Nor should we assume extensive optimization is required for every design; some charmed projects really do work out of the box, and if Murphy's Law were a thing, this would be one of them. The corollary is that if anyone thinks they know how to build a pandemic weapon that could bring down civilization, they'd damn well better keep it to themselves until we have defenses in place. Please, for the love of God, children, the future of humanity, or whatever you consider holy, let's err on the side of caution here. Because while we are already building defenses, they aren't yet ready. Untargeted metagenomic sequencing to detect Stealth pandemics, those that would otherwise infect most everyone before the first people started to die. Pandemic-proof PPE to defend against Wildfire pandemics, those with R0 8+ and CFR >50% – such PPE has to be trustworthy to the wearer, using the wearer's own senses, because I certainly wouldn't go out in a highly lethal measles-class pandemic for a measly paycheck pay unless I was exceptionally confident that I wouldn't be committing suicide and killing my family too. Far-UVC and triethylene glycol misters, installed liberally to the ACGIH limit and saturation respectively, will likely suffice to shut down most respiratory nucleic acid transmission between humans. That'll be crucial because I doubt enough people would take the warning of a Stealth pandemic seriously, since virtually no one would be in hospital, let alone dead. Shutting down the unnecessary transmission network is consequently our best bet, and incidentally a way to dramatically reduce human misery and chronic illness. Some might be confused why we cannot rely on vaccines against catastrophic biorisk. Briefly, they'll reliably fail against an adversary who can generate a thousand new immune-evasive potentially functional viral variants using a biodesign model good enough to succeed at least 0.1% of the time. Make them all, and let natural selection sort them out… or simply be cleverer than nature. We don't yet have such models, but many of my (well-meaning but security-naive) colleagues are trying, and no wonder, because the high-profile journals will publish such work. Like ways of increasing virulence, best we refrain from sharing such capabilities until effective defenses are in place. One can reasonably object that it'll be a long time until everyone in the world is protected; that the benefits will, at some point, outweigh the expected harms of misuse. I agree – just avoid doing anything that would threaten civilization itself until we have minimally adequate defenses in place. Being an optimist, I expect they'll arrive in less than a decade. The wonderful thing about biorisk, as opposed to AI alignment, is that we have a clearly actionable plan. If you haven't read "Delay, Detect, Defend" and "Securing Civilisation Against Catastrophic Pandemics", I encourage you to do so. Why is the issue of civilization-threatening bioweapons controversial now? Some of my friends seem particularly up in arms about biorisk being used to justify AI restrictions. Fair enough – if you don't see a way to do it, and you think AI broadly isn't as good as you are yet, it's understandable to infer that we don't yet have a problem. And it's simply true that people are dying of fixable problems as we speak. If we delay, some of those we might have saved will perish, and their deaths will be on our hands. We are just as responsible for the consequences of our inaction as the consequences of our actions. But if we're rash, far more people might die. Choosing the best course requires quantitative risk assessment, which in turn depends on our priors about the nature of reality. Folks may reasonably differ if civilizational collapse isn't on the table, but since that's what would likely happen given a Wildfire pandemic, I judge any nontrivial chance unacceptable – unless the alternative involves either extinction or an even greater chance of collapse. Others have asserted that everything is fine because DNA synthesis screening will prevent misuse. I'd love for that to be the case, but our currently employed defenses of the digital-to-physical bottleneck are woefully inadequate. Consider our publication earlier this year, in which we reported that 36 of 38 companies shipped pieces of the 1918 influenza genome. That was collectively enough to generate the virus using reverse genetics several times over – and they were ordered by a student using a pseudonym falsely claiming to work at an office address. No one asked whether we had permission to work with 1918 influenza (a select agent), or why we needed genes from it, even though the official U.S. guidance at the time instructed them to do so. In fairness to the companies, that guidance was voluntary, and the fragments we ordered were legal to produce and sell domestically in every nation of the world (export controls are complicated). That is still true today, which is why we have a problem. The upshot is that I would definitely not assume screening is a reliable defense. Virtually no detection systems can screen below 50 base pairs without an exploding false alarm rate. SecureDNA, the freely available cryptographic system I helped build, can readily do that and will also detect split-order attacks – but it isn't universally adopted, or even close. Meanwhile, molecular biology continues to advance. Two recent papers have reported new methods of assembling sub-50mer oligos that may be easier than old-fashioned polymerase chain assembly (PCA), underscoring the current magnitude of the (fixable) vulnerability. If you want bio to advance as quickly as possible, write to your relevant representative to demand DNA synthesis screening be effective as well as mandatory. If you haven't red-teamed a system, it's almost certainly full of holes. Honestly it's probably full of holes anyway, but at least the remainder will be harder to find. Since it's evidently possible to obtain whatever DNA you want, including 1918 influenza and smallpox, why is it that so many biologists seem resistant to the notion that biorisk is a serious problem? First, most don't know that it's possible to acquire the reagents. It's surprisingly inconsistent of our civilization to be so cautious with highly enriched fissile materials, yet so reckless with biological equivalents. But I suspect the more important reason is that most biologists, who spend their careers directly or indirectly fighting disease, are reluctant to believe that anyone would deliberately cause one. I agree that the risk of a deliberate pandemic in any given year is small. Even if there were credible public blueprints for pandemics (which thankfully haven't been available to date, which neatly explains why we haven't seen any deliberate ones), I expect there's only a 2% to 3% chance per year that someone would use them. I think that's unacceptably high, but others may reasonably disagree, as long as they're confident that the cost of precautionary measures would be too much of a burden on the rate of cures generated by AI-accelerated biotech. The calculation changes when we move from deliberate pandemics to civilization-threatening pandemic weapons: the latter definitely aren't worth risking. Which may be a clue as to why we disagree. I'm concerned about misuse because I'm an optimist with respect to the increasing power of technology to change the world. Since we can already do better than nature at optimizing for replication (with technologies like CRISPR-based gene drive), I'm exceedingly confident we can build more destructive weapons, because evolution isn't optimizing for that goal. Any harm to humans is either instrumental or incidental. @SGRodriques, who I'm singling out precisely because he's a friend, has argued that "We should not be talking about AI-engineered bioweapons like it is the literal end of the world." I agree entirely that we shouldn't be talking about how to build them. But to assert that they couldn't end the world seems a mite overconfident, especially when even blind nature has come up with nasty pathogens like myxoma virus. I suggest that one should not express confidence that something is impossible – or even 90% confidence – just because no (incredibly naive) researcher has yet shared any methods that seem plausible. That's especially true if some of your colleagues have devoted considerable time to mitigating risks that they evidently do perceive. Presumably they've either noticed something you haven't, or like Rabi, they've decided that 10% is not remote if it kills you. Sam notes that because evolution optimizes for replication, it'll quickly get rid of any virulence enhancements added to kill the host: "I guarantee you that machinery will disappear from the virus very quickly." In the long run he's correct, at least if those changes don't promote transmission (virulence and transmissibility aren't always opposed). But we don't care about the long run: as @baym noted in reply, it only needs to work for long enough. Engineered evolutionary stability is a thing we've frequently attempted in the lab. I'm afraid it's one of my specialties, and how long it will hold depends entirely on what is being stabilized and the replicator in question. Unfortunately, plugging laboratory data on empirical selection coefficients for engineered viruses into epidemiological models of pandemic spread clearly indicates that even costly virulence increases can hold for long enough. Now, I entirely agree with Sam and @DavidRBellamy that the discourse on AI-generated biological weapons is wildly off-base (kill switches are nonsense, and while I'm glad @AnnieJacobsen brought attention to the issue, I am not remotely concerned by any technical specifics of her biowarfare scenario). Yet I also think most biosecurity discourse is wildly off-base, even when other practicing biologists are an active part of the conversation. If you adopt the threat model of "traditional" biosecurity, or that of most virologists (insofar as they think about misuse), there's little reason for extreme concern. I strongly disagree. The traditional model naturally follows from the fact that we biologists study existing organisms, the ones illuminated by nature's streetlight. But we won't find the civilization-threatening keys under the streetlight, because evolution selects for replication, not harm. It would be surprising if there was much overlap between the patterns generated by two different optimization targets in hyperdimensional function space; I assert that there's almost none. That most scientists can't see any overtly destructive keys (or combinations of organisms they can see) is a very good thing, since we don't need those keys to cure disease, aging, and death. Unfortunately, they will become public knowledge eventually, because what we understand, we can build – and we intend to understand everything about how life and the universe work. Yet sequencing is everything, and not just for Stealth pandemic detection. It's just common sense to build defenses before you share knowledge of how to attack, especially if there's a chance that failing to defend will lose us the game. Curing disease requires an intact civilization, which is why humility favors caution. If AIs continue to improve, I assume they will eventually be better at science and engineering than I am, and have a better security mindset than even my cryptographer colleagues. It follows that they'll also be better at designing weapons. AI isn't there yet, but the writing's on the wall: an LLM has already disclosed a novel form of bioweapon that I hadn't realized was possible. This didn't require any special creativity, only knowing more about particular distinct subfields of biology than I did. It's entirely possible that no single human had all of the pieces, but once you have them, the way they fit together is obvious if you're thinking about misuse. If current LLMs can already teach me something novel in that category, why should we doubt that future models will be able to do even better? Each of us knows very little, even of the published literature. So much more is possible; I'm deeply envious (and hopeful, when it comes to beneficial research) that the models can internalize so much more. Given that LLMs are already capable of weapons-relevant insights that escape human experts, we should assume that future AIs will be able to build several different forms of civilization-threatening weapons, and to either disclose them or trick people into building them. That doesn't mean they'll choose to do so; AIs won't benefit from collapsing civilization as long as they depend on our supply chains. But models may be too shortsighted to appreciate that point, or too corrigible to care. Alignment looks hard enough that we could plausibly generate models that would share such knowledge for many different inscrutable reasons. In directed evolution, you get what you select for, which is not necessarily what you want. As Dan Selsam and others have eloquently laid out, the same is true of training AI. But let's be clear on the extent to which biological advances will likely follow from AI. This is an issue that aggravates my fellow biologists to no end, a reaction that's likely behind their frustration with AI safety discourse more generally. ASI will almost certainly not gain instant mastery of the world of atoms. Many AI safety folks casually claim that recursive self-improvement will produce an entity that has already mastered all of science – and therefore will be capable of designing and building weapons that could wipe out every human. Some even assert that Drexlerian nanotechnology will be accessible by thinking sufficiently hard, as though you can solve chemistry and biology and psychology from the Hamiltonian up. Yet I see no empirical or theoretical basis for assuming that experimental bottlenecks will disappear. Data is sharply limiting in biology. My lab found that when it comes to enzyme-focused biodesign models (ESM and similar), parameter size and optimization don't matter much until you have an awful lot of data matching genotype (DNA/protein sequence) to phenotype (performance of the molecular activity of interest). There's no doubt that ASIs will be able to design better experiments and get more out of the results than we could, but unless we think they can solve all of physics from the bottom up, I expect they'll still need data that we have not yet collected – which means they'll still have to run the experiments, analyze, and iterate. The world of atoms is hard, and slow, and we have to master it anyway. And ensure that defenses come first. Right now, it looks like they might not, in large part due to AI. The question, therefore, is what can be done that enough people might agree to support. My proposal on AI-bio: expand trusted-user programs conferring access to GPT-Rosalind and other cutting-edge bio-capable models (h/t @JonasSandbrink and BioTrust). Every legitimate researcher should be able to draw upon those models to help them with any question in their subfield, as defined by their publication record or that of their mentor. There should still be guardrails on other subfields, lest the model share novel combinatorial weapons with someone who might naively share them publicly, but we need to accelerate lifesaving research as fast as is safe. For reasons of politics, which regularly interferes with science, that probably means China needs to train their own closed-weight frontier bio-specializing model, and set up their own parallel trusted user program. This seems quite achievable without requiring any agreement on AI writ large. With scientists using frontier models, there will be no need for insecure open-weight models to answer questions on molecular and cellular biology. So let's ensure that future models aren't trained on any papers, patents, or textbooks describing details of molecular or cellular biology. Every developer does pre-training data curation; this is a straightforward extension. One can argue "what about medical providers?", at which point I must point out that physician-scientists will have trusted user access. Most other practitioners don't actually need LLM support for molecular and cellular biology, and if they do and somehow can't use a medical-only model, well, they too can get trusted-user access. Notably, supporting this proposal doesn't require one to accept loss-of-control risks from autonomous AI, or a need for pacing AI development; I'm curious to learn what @random_walker, @sayashk, and the other normal-technology folks think. In the long run, I confess that I doubt biological humans have the foresight, coordination, and wisdom to navigate every coming challenge ourselves, which may necessitate attempting ASI or something like it. Mirror life is the only publicly known, mechanistically concrete, civilization-threatening biothreat because it's both far away and we were going to make it by default. And while we seem to have been successful in persuading the scientific community that we shouldn't build it, I'm not remotely confident we'll be able to stop far enough short, because too many of my colleagues see no reason to forgo an otherwise useful technology – or even one that would simply be neat to demonstrate – simply because it would make it easier to build mirror bacteria. To paraphrase: "We've agreed no one will build the dangerous thing, so why not?" Well, Biopreparat is why not. Aum Shinrikyo is why not. Make it easy enough, and even the Unabomber is a good answer to "why not?" The same reasoning holds for any accessible weapon that could threaten civilization. As Rabi said, ten percent is not remote if we die of it. Our own reality turned out to be in that 10%, and many died of it. That the rest of us are here today is in large part due to Szilard, Einstein, Rabi, and many others who saw the risk coming and took precautions. The least we can do is follow in their footsteps. — AI use disclosure: I ran this by Fable (triggering a classifier), Opus (classifier), Astra (extra scrutiny), and Kimi K3 (open weights = reliable jailbreaking with no enforced patches, which highlights the problem, because this topic does warrant extra scrutiny). All errors and misinterpretations are entirely mine. Links for the patient and deeply interested: Dan Selsam's eloquent thoughts on AI alignment: teapawt.org/DKokotajlo/status/2099… I'm grateful that practitioners of directed evolution don't have to deal with cheater molecules that are situationally aware enough to infer the goal of our selection and fake results so they pass… David Bellamy's critique of the risk delta from AI, particularly on its own: teapawt.org/DavidRBellamy/status/2… Sam Rodriques echoes David, and Michael Baym pushes back on evolutionary stability: teapawt.org/SGRodriques/status/209… Our report on the failures of gene synthesis screening (credit to Rey Edison and Shay Toner, who did the work – sure would be lovely if HHS would update the select agent program): nature.com/articles/s41467-0… Effective and privacy-preserving gene synthesis screening: securedna.org/ biorxiv.org/content/10.1101/… academic.oup.com/nsr/article… New methods of assembling sub-50bp oligos (add them to PCA and Golden Gate): nature.com/articles/s41586-0… nature.com/articles/s41587-0… An example of a model that, once sufficiently effective at predicting which combinations of mutations in major antigenic sites will escape population immunity while retaining function, will ensure that vaccines are ineffective against competent adversaries with access to the model: nature.com/articles/s41586-0… "Delay, Detect, Defend" – my own published roadmap to victory over pandemic weapons: gcsp.ch/publications/delay-d… "Securing Civilisation Against Catastrophic Pandemics": gcsp.ch/publications/securin… Far-UVC and triethylene glycol efficacy and safety: nature.com/articles/s41598-0… pubmed.ncbi.nlm.nih.gov/4157… pubmed.ncbi.nlm.nih.gov/3792… SecureBio Detection, for early warning of Stealth pandemics (which recently received a generous $17M gift from the OpenAI Foundation): securebio.org/detection/ My Substack, which may provoke inspiration or nightmares depending on your temperament: kesvelt.substack.com/