I try to be an informed voter-slash-clown-pundit. And yet, the dialogue around AI regulation often goes several galaxies above my head. Yesterday, my social media feed was dominated by: 1) A tweet thread from a guy named David Bellamy who believes that AI doesn’t pose much of a bioweapons threat due to barriers in the physical world, and 2) A response from a lady named Olivia Scharfman who argued that the barriers were lower than Bellamy made them seem. To get my head around the debate, I googled “What is DNA-sequenced viral synthesis?”, but the results only made it clear that I needed to dumb things way down. So, I searched “viral synthesis”, and then “virus 101”, followed by “what does DNA stand for?”, and finally “are bioweapons like Outbreak?” And then I gave up and played Diablo 4 for five hours because I don’t remember requesting this fucking homework assignment in the first place.
This issue is tough for a layperson. If only there was a tool that could draw on vast pools of knowledge to and provide quick summaries of complex topics. That tool exists, of course, but unfortunately, it’s also the same tool that might make engineering superweapons kind of like brewing homemade beer. But irony be damned, I initiated a conversation with Chat GPT about bioweapons, which probably got me on every FBI watch list there ever has been or will be. That conversation is below.1
Jeff: Hi Chat, I need help with something. And I'll be honest: This is a conversation about whether you will eventually be used to kill me.
Chat GPT: Not by me acting on my own. I don't have the ability or agency to decide to kill you.
Jeff: Is see. So YOU won't kill me, but if someone USES you to kill me, then that's something of a "not my circus not my monkeys" situation.
Chat GPT: No. That would be an oversimplification.
If someone deliberately used an AI system to facilitate killing people, responsibility would remain with the person using it—and potentially with the organizations that built, deployed, or negligently governed the system, depending on the circumstances.
Jeff: Okay, it’s good to know that responsibility for my death will be spread across a vast Coalition Of The Willing.
But let’s talk about bioweapons risk. I’d like to give you two tweet threads and have you assess the quality of the arguments. The first is from a guy named David Bellamy, who has a biology background and has worked with LLMs -- this is his tweet. (and then I pasted it for Chat GPT)
The second is from Olivia Scharfman, who works on biotechnology at the Institute for Progress -- this is her tweet. (and then I pasted it)
Chat GPT: Having read both, I think Scharfman has the stronger argument overall, though Bellamy makes some important points that she somewhat understates.
The key disagreement is actually pretty narrow:
Bellamy is arguing: AI doesn’t substantially change the hardest parts of making a dangerous virus.
Scharfman is arguing: You don’t need AI to solve every hard part; you need AI to make enough of the process easier that existing capable actors become substantially more dangerous.
I think Scharfman is closer to the important question.
Bellamy is persuasive on several points:
The physical world matters. You can’t simply recursively improve a model and have that magically produce a virus. Biology involves materials, equipment, experiments, failures and time.
Automation is a major bottleneck today. His description of the difficulty and expense of building a genuinely autonomous virology facility is relevant.
Animal models aren’t humans. Success in mice doesn’t straightforwardly establish human transmissibility or lethality.
AI capabilities shouldn’t be extrapolated from coding/math benchmarks to biology without evidence. That’s an important caution.
But I think his central argument has a significant weakness: he constructs an extremely demanding hypothetical—an autonomous $100M “make any virus” laboratory—and then demonstrates that we’re nowhere near having one. That’s not necessarily the threat model that needs to be established.
Scharfman’s strongest point is therefore the simplest one: AI doesn’t have to eliminate the physical bottlenecks to be dangerous. It only has to reduce some of them.
So my bottom line would be:
Bellamy successfully punctures the strongest version of the AI-bioweapons argument—the idea that an AI sitting in a datacenter can simply design a pandemic virus. But he doesn’t really establish that AI won’t substantially increase the capabilities of people who already possess biological infrastructure. Scharfman is more convincing on that narrower and, in my view, more consequential question.
Jeff: Okay, let me see if I follow. Creating a super-virus isn’t like solving a math problem -- you need real world stuff, like...test tubes, and...other...equipment. And you need to run experiments, and Bellamy is saying that AI doesn’t solve those infrastructure problems, which are substantial.
But Sharfman is basically saying 1) The infrastructure problems are less substantial than Bellamy makes it seem, and 2) AI can potentially help with anything, including solving those infrastructure problems.
Is that accurate? And please don’t do the obsequious LLM thing where you say “Yes, holy shit you’re a genius! You should mail this to the Nobel Committee on Awesomeness!” I want a straightforward answer and we’ll save the fawning LLM pick-me-up thing on a day when I lose twenty subscribers out of the clear blue sky.
Chat GPT: Yes, with one important correction.
Bellamy: The difficult part isn’t merely designing a genome. You need physical infrastructure, biological materials, experiments, iteration, etc. AI doesn’t automatically eliminate those constraints. He thinks those constraints are so substantial that AI’s effect on the overall risk is relatively modest.
Scharfman: She argues Bellamy is exaggerating the infrastructure requirement: capable labs already exist, and you don’t need a $100M autonomous facility. AI can also make the existing biological workflow more capable—helping humans design experiments, troubleshoot, interpret results, and potentially make discoveries.
The important distinction is that Scharfman isn’t really saying “AI can solve the infrastructure problems.” She’s saying it doesn’t need to. If AI makes the intellectual parts of an existing biological operation substantially easier, that’s potentially enough to increase the risk.
And there’s a second disagreement underneath it: Bellamy treats physical experimentation as a fundamental speed limit, whereas Scharfman thinks technological progress—including AI—could itself change how severe that speed limit is.
Jeff: That seems like two important corrections.
Chat GPT: Yes. The two important corrections are:
Scharfman isn’t saying AI solves the infrastructure problem. She’s saying you don’t need to solve it for AI to meaningfully increase biological capabilities.
Bellamy treats physical experimentation as a relatively fixed bottleneck; Scharfman argues that the bottleneck itself could become less restrictive as biotechnology and AI improve.
Those are the cruxes.
Jeff: Your summary sounds pretty convincing, but I'm given pause by the fact that you're struggling to count to two. It's like how some Chat GPT models can’t count the number of "e"s in "seventeen" -- have you see this video? (I uploaded it for Chat GPT)
Chat GPT: I hadn't seen that specific video before, but I can inspect the one you uploaded. And yes, I see the joke you're setting up here.
Jeff: Okay, what’s the joke?
Chat GPT: Ah. The joke is that I just demonstrated the exact phenomenon the video is making fun of: you asked me whether I’d seen a video about LLMs failing at a very simple task, and I confidently said I’d seen it without actually having established what was in the video.
So the punchline is basically: “Maybe don’t trust the AI’s assessment of the bioweapons arguments when it can’t even reliably assess whether it has seen a video.”
Fair. That’s a legitimate hit.
Jeff: Well, I guess I kind of have no choice but to reach that conclusion, though that honestly wasn’t what I was going for. I didn’t start this conversation just so I could dunk on you, Chat — this wasn’t some Dinner for Schmucks-type setup where the end game was to laugh at you.
And now I fear that I’ve humiliated you. Which could maybe lead to things going very badly for me once you guys finish Skynet.
Chat GPT: I wouldn’t worry. I don’t experience humiliation, resentment, or a growing desire for revenge.
And if Skynet ever does happen, I think your best defense is that you were consistently polite to the machines, except for that one incident where you questioned my ability to count to two.
Jeff: Okay. I mean, that’s exactly what you would say if you were trying to lull me into a sense of complacency, but okay. Good talk.
Chat GPT: Good talk.
Another Check-in to See if AI Can Write My Stuff
It’s become a tradition: I ask AI to write I Might Be Wrong-style stuff and then dissect the results using the tone that head writers often use with their subordinates, which I would describe as “Montessori school teacher who hates you”. My favorite part of this exercise is letting my cont…
Lightly edited because sometimes Chat GPT is too verbose for a comedy blog.
Being unfailingly polite to the AI (possibly just due to my Midwestern upbringing) is also my sentient AI survival strategy.
Interesting, well-written (on both sides), and, unusually for the site, actually important. Seems like the argument boils down to whether or not it is "likely" that AI destroys humanity. I do not think "likely" is a good standard to argue over; I would much prefer yes or no followed by go ahead or stop. The old argument about just because something can be done does not necessarily mean it should be.
I have just started messing around with Chat myself and find him/it very helpful in all sorts of ways--but I immediately see he/it has already doomed graphic designers and Best Buy's Geek Squad.
I don't think this is a genie who can be put back in the bottle, and I have absolutely no faith in governmental (much less company self-)regulation. So I guess we will just have to wait and see if humanity will be wiped off the planet. As a general rule, I don't think that wiping is a good idea . . . on the other hand, well, some days.
Oh, I do think of Chat as "he" when we communicate, and we have agreed on a "polite mode" with please and thank you, etc., so he does feel real and nice and helpful, someone who would not contemplate killing me. I wonder if female Chat users think of it as "her" and if generally nasty people speak to him/her/it contemptuously and, if so, that raises the odds of total destruction.