I try to be an informed voter-slash-clown-pundit. And yet, the dialogue around AI regulation often goes several galaxies above my head. Yesterday, my social media feed was dominated by: 1) A tweet thread from a guy named David Bellamy who believes that AI doesn’t pose much of a bioweapons threat due to barriers in the physical world, and 2) A response from a lady named Olivia Scharfman who argued that the barriers were lower than Bellamy made them seem. To get my head around the debate, I googled “What is DNA-sequenced viral synthesis?”, but the results only made it clear that I needed to dumb things way down. So, I searched “viral synthesis”, and then “virus 101”, followed by “what does DNA stand for?”, and finally “are bioweapons like Outbreak?” And then I gave up and played Diablo 4 for five hours because I don’t remember requesting this fucking homework assignment in the first place.
This issue is tough for a layperson. If only there was a tool that could draw on vast pools of knowledge to and provide quick summaries of complex topics. That tool exists, of course, but unfortunately, it’s also the same tool that might make engineering superweapons kind of like brewing homemade beer. But irony be damned, I initiated a conversation with Chat GPT about bioweapons, which probably got me on every FBI watch list there ever has been or will be. That conversation is below.1
Jeff: Hi Chat, I need help with something. And I'll be honest: This is a conversation about whether you will eventually be used to kill me.
Chat GPT: Not by me acting on my own. I don't have the ability or agency to decide to kill you.
Jeff: I see. So YOU won't kill me, but if someone USES you to kill me, then that's something of a "not my circus not my monkeys" situation.
Chat GPT: No. That would be an oversimplification.
If someone deliberately used an AI system to facilitate killing people, responsibility would remain with the person using it—and potentially with the organizations that built, deployed, or negligently governed the system, depending on the circumstances.
Jeff: Okay, it’s good to know that responsibility for my death will be spread across a vast Coalition Of The Willing.
But let’s talk about bioweapons risk. I’d like to give you two tweet threads and have you assess the quality of the arguments. The first is from a guy named David Bellamy, who has a biology background and has worked with LLMs -- this is his tweet. (and then I pasted it for Chat GPT)
The second is from Olivia Scharfman, who works on biotechnology at the Institute for Progress -- this is her tweet. (and then I pasted it)
Chat GPT: Having read both, I think Scharfman has the stronger argument overall, though Bellamy makes some important points that she somewhat understates.
The key disagreement is actually pretty narrow:
Bellamy is arguing: AI doesn’t substantially change the hardest parts of making a dangerous virus.
Scharfman is arguing: You don’t need AI to solve every hard part; you need AI to make enough of the process easier that existing capable actors become substantially more dangerous.
I think Scharfman is closer to the important question.
Bellamy is persuasive on several points:
The physical world matters. You can’t simply recursively improve a model and have that magically produce a virus. Biology involves materials, equipment, experiments, failures and time.
Automation is a major bottleneck today. His description of the difficulty and expense of building a genuinely autonomous virology facility is relevant.
Animal models aren’t humans. Success in mice doesn’t straightforwardly establish human transmissibility or lethality.
AI capabilities shouldn’t be extrapolated from coding/math benchmarks to biology without evidence. That’s an important caution.
But I think his central argument has a significant weakness: he constructs an extremely demanding hypothetical—an autonomous $100M “make any virus” laboratory—and then demonstrates that we’re nowhere near having one. That’s not necessarily the threat model that needs to be established.
Scharfman’s strongest point is therefore the simplest one: AI doesn’t have to eliminate the physical bottlenecks to be dangerous. It only has to reduce some of them.
So my bottom line would be:
Bellamy successfully punctures the strongest version of the AI-bioweapons argument—the idea that an AI sitting in a datacenter can simply design a pandemic virus. But he doesn’t really establish that AI won’t substantially increase the capabilities of people who already possess biological infrastructure. Scharfman is more convincing on that narrower and, in my view, more consequential question.
Jeff: Okay, let me see if I follow. Creating a super-virus isn’t like solving a math problem -- you need real world stuff, like...test tubes, and...other...equipment. And you need to run experiments, and Bellamy is saying that AI doesn’t solve those infrastructure problems, which are substantial.
But Scharfman is basically saying 1) The infrastructure problems are less substantial than Bellamy makes it seem, and 2) AI can potentially help with anything, including solving those infrastructure problems.
Is that accurate? And please don’t do the obsequious LLM thing where you say “Yes, holy shit you’re a genius! You should mail this to the Nobel Committee on Awesomeness!” I want a straightforward answer and we’ll save the fawning LLM pick-me-up thing on a day when I lose twenty subscribers out of the clear blue sky.
Chat GPT: Yes, with one important correction.
Bellamy: The difficult part isn’t merely designing a genome. You need physical infrastructure, biological materials, experiments, iteration, etc. AI doesn’t automatically eliminate those constraints. He thinks those constraints are so substantial that AI’s effect on the overall risk is relatively modest.
Scharfman: She argues Bellamy is exaggerating the infrastructure requirement: capable labs already exist, and you don’t need a $100M autonomous facility. AI can also make the existing biological workflow more capable—helping humans design experiments, troubleshoot, interpret results, and potentially make discoveries.
The important distinction is that Scharfman isn’t really saying “AI can solve the infrastructure problems.” She’s saying it doesn’t need to. If AI makes the intellectual parts of an existing biological operation substantially easier, that’s potentially enough to increase the risk.
And there’s a second disagreement underneath it: Bellamy treats physical experimentation as a fundamental speed limit, whereas Scharfman thinks technological progress—including AI—could itself change how severe that speed limit is.
Jeff: That seems like two important corrections.
Chat GPT: Yes. The two important corrections are:
Scharfman isn’t saying AI solves the infrastructure problem. She’s saying you don’t need to solve it for AI to meaningfully increase biological capabilities.
Bellamy treats physical experimentation as a relatively fixed bottleneck; Scharfman argues that the bottleneck itself could become less restrictive as biotechnology and AI improve.
Those are the cruxes.
Jeff: Your summary sounds pretty convincing, but I'm given pause by the fact that you're struggling to count to two. It's like how some Chat GPT models can’t count the number of "e"s in "seventeen" -- have you see this video? (I uploaded it for Chat GPT)
Chat GPT: I hadn't seen that specific video before, but I can inspect the one you uploaded. And yes, I see the joke you're setting up here.
Jeff: Okay, what’s the joke?
Chat GPT: Ah. The joke is that I just demonstrated the exact phenomenon the video is making fun of: you asked me whether I’d seen a video about LLMs failing at a very simple task, and I confidently said I’d seen it without actually having established what was in the video.
So the punchline is basically: “Maybe don’t trust the AI’s assessment of the bioweapons arguments when it can’t even reliably assess whether it has seen a video.”
Fair. That’s a legitimate hit.
Jeff: Well, I guess I kind of have no choice but to reach that conclusion, though that honestly wasn’t what I was going for. I didn’t start this conversation just so I could dunk on you, Chat — this wasn’t some Dinner for Schmucks-type setup where the end game was to laugh at you.
And now I fear that I’ve humiliated you. Which could maybe lead to things going very badly for me once you guys finish Skynet.
Chat GPT: I wouldn’t worry. I don’t experience humiliation, resentment, or a growing desire for revenge.
And if Skynet ever does happen, I think your best defense is that you were consistently polite to the machines, except for that one incident where you questioned my ability to count to two.
Jeff: Okay. I mean, that’s exactly what you would say if you were trying to lull me into a sense of complacency, but okay. Good talk.
Chat GPT: Good talk.
Another Check-in to See if AI Can Write My Stuff
It’s become a tradition: I ask AI to write I Might Be Wrong-style stuff and then dissect the results using the tone that head writers often use with their subordinates, which I would describe as “Montessori school teacher who hates you”. My favorite part of this exercise is letting my cont…
Lightly edited because sometimes Chat GPT is too verbose for a comedy blog.
It's a compression argument. Human knowledge and capability, and access to said, does something similar: it makes biological weapons easier to develop. CRISPR, I'd argue, also makes them easier to develop. Any technological or knowledge advance that makes difficult technical bio-med tasks easier makes biological weapons easier to develop.
The question with AI is one of compression: does it do enough all at once that makes people who are impulsive enough to do very bad things the ability to do them without the limitations imposed by many years and expense of research and development? It might. It's hard to say because no one knows where the endpoint of AI is.
But if it can do that, AI can probably also help the white hats come up with ways to mitigate risks from those attacks as well. Technological intelligence cuts both ways. Is it a symmetric increase in capability -- the capability to create weapons, and the ability to create defenses against them? Probably not. Can we limit AI's ability to help people generate biological weapons without seriously hampering its ability to do things we WANT it to do, I doubt it.
And the largest threats in biological weapons remain the large state actors who are both well-funded already and committed to such research even if mostly for the "figuring out what someone else might do and how to counter it" purposes.
Being unfailingly polite to the AI (possibly just due to my Midwestern upbringing) is also my sentient AI survival strategy.